Knights, Knaves, Protected and Internal
When you override a virtual method in C# you are required to ensure that the stated accessibility of the overridden method - that is, whether it is public, internal, protected or protected internal(*) – is exactly re-stated in the overriding method. Except in one case. I refer you to section 10.6.4 of the specification, which states:
an override declaration cannot change the accessibility of the virtual method. However, if the overridden base method is protected internal and it is declared in a different assembly than the assembly containing the override method then the override method’s declared accessibility must be protected.
What the heck is up with that? Surely if an overridden method is protected internal then it only makes sense that the overriding method should be exactly the same: protected internal.
I’ll explain why we have this rule, but first, a brief digression.
A certain island is inhabited by only knights and knaves. Knights make only true statements and only answer questions truthfully; knaves make only false statements and only answer questions untruthfully. If you walk up to an inhabitant of the (aptly-named) Island of Knights and Knaves you can rapidly ascertain whether a particular individual is a knight or a knave by asking a question you know the answer to. For example “does two plus two equal four?” A knight will answer “yes” (**), and a knave will answer “no”. Knaves are prone to saying things like “my mother is a male knight”, which is plainly false.
It might seem at first glance that there is no statement which could be made by both a knight and a knave. Since knights tell the truth and knaves lie, they cannot both make the same statement, right? But in fact there are many statements that can be made by both. Can you think of one?
.
.
.
.
.
.
.
Both a knight and a knave can say “I am a knight.”
How does that work? The reason this works is because the pronoun “I” refers to different people when uttered by different people. If Alice, a knight, makes the statement “I am a knight”, she is asserting the truth that “Alice is a knight”. If Bob, a knave, makes the statement “I am a knight”, he is not asserting the true statement “Alice is a knight” but rather the false statement “Bob is a knight”. Similarly, both Alice and Bob can assert the statement "My name is Alice," for the same reason.
And that’s why overriding methods in a different assembly aren’t “protected internal”. The modifier “internal” is like a pronoun; it refers to the current assembly. When used in two different assemblies it means different things. The purpose of the rule is to ensure that a derived class does not make the accessibility domain of the virtual member any larger or smaller.
An analogy might help. Suppose a protected resource is a car, an assembly is a dwelling, a person is a class and descendent is a derived class.
- Alice has a Mazda and lives in House with her good friend Charlie.
- Charlie has a child, Diana, who lives in Apartment.
- Alice has a child, Elroy, who lives in Condo with his good friend Greg.
- Elroy has a child – Alice’s grandchild -- Frank, who lives in Yurt.
Alice grants access to Mazda to anyone living in House and any descendent of Alice. The people who can access Mazda are Alice, Charlie, Elroy, and Frank.
Diana does not get access to Mazda because she is not Alice, not a descendent of Alice, and not a resident of House. That she is a child of Alice’s housemate is irrelevant.
Greg does not get access to Mazda for the same reason: he is not Alice, not a descendent of Alice, and not a resident of House. That he is a housemate of a descendent of Alice is irrelevant.
Now we come to the crux of the matter. Elroy is not allowed to extend his access to Mazda to Greg. Alice owns that Mazda and she said "myself, my descendents and my housemates". Her children don't have the right to extend the accessibility of Mazda beyond what she initially set up. Nor may Elroy deny access to Frank; as a descendent of Alice, Frank has a right to borrow the car and Greg cannot stop him by making it "private".
When Elroy describes what access he has to Mazda he is only allowed to say "I grant access to this to myself and my descendents" because that is what Alice already allowed. He cannot say "I grant access to Mazda to myself, my descendents and to the other residents of Condo".
-----------------
(*) Private virtual methods are illegal in C#, which irks me to no end. I would totally use that feature if we had it.
(**) Assuming that they answer at all, of course; there could be mute knights and knaves.
Comments
Anonymous
March 24, 2010
Yes, why are private virtual. methods illegal? I know of some experts in the C++ communithy that assert that all vritual methods should be private! Another irksome thing is that overrides can't make a member more accessible. There certainly are reasonable designs where a member that was an internal detail in a base class is a sensible part of the public API in a derived class. Making overrides more visible doesn't violate LSP or break any important scenarios that I can think of, so why is this not allowed in C#?Anonymous
March 24, 2010
I sometimes wish that the accessibility of the overriding method could be either wider or narrower than the accessibility of the overridden method. Then I could avoid creating (and coming up with a name for) a forwarding method when I want to change the accessibility of either an overriding or an overridden method from protected to public.Anonymous
March 24, 2010
There's a much simpler explanation. Making an external override protected internal would make it internal to the overriding assembly, which (outside the inheriting type) has no business seeing the method. I don't see why you need the whole analogy. I think you'll find that in general it is rather easy to explain things to people who already understand in depth what you're attempting to explain. Successfully explaining things to people who don't already have a deep understanding of the topic is rather more difficult. -- EricAnonymous
March 24, 2010
Is that a nitpicker's corner developing at the bottom there? ;) I do wish you could sometimes make the scope narrower on overridden methods. Sometimes I find myself with a hierarchy two or three levels deep where I want to turn off one little bit of functionality for any derived classes and I have to end up throwing a bool in and checking for that feature. I'm not terribly sure how you'd implement this so callers expecting a certain visibility didn't get shafted when something came by that hid that method.Anonymous
March 25, 2010
@Carl D: Making a method more accessible could end up allowing a caller to break the invariants of a class by having it called in unexpected ways or at unexpected times. For example, suppose there is a BankAccount class with a protected internal method SetBalance. The protected portion of the modifier communicates that derived classes are trusted to 1. Alter the default behavior of setting the balance as necessary (override) 2. Decide when the balance can be set (call) The internal portion of the modifier communicates that classes in the same assembly as BankAccount should be able to call the SetBalance method as necessary. Obviously, even these trusted classes should not be able to alter the behavior of the SetBalance method. If a derived class were able to declare the method more accessible, it would be possible to allow untrusted callers (outside of BankAccount's assembly) to call the method, potentially altering the balance in unforeseen ways. Derived classes are already able to do the same thing, which is a strong argument for private virtual methods.Anonymous
March 25, 2010
The comment has been removedAnonymous
March 25, 2010
OK, I'm lost on this whole "private virtual" thing. Tell me where I'm wrong here: A "private" method is a method that only the base class knows about; a derived class cannot call that method and can create its own method by that name without needing "new" to avoid a warning. A "virtual" method is a method which has an implementation in the base class, but for which a derived class can provide an overriding implementation if it wants to. Logically, a "private virtual" method would be one which the derived class can override but not know about. How does that make sense? You've forgotten that the derived class can be a nested class; nested classes have access to the private methods of their containers. I quite like the pattern of making an abstract base class with a private constructor so that the only concrete classes possible are those declared within the base class itself. But if you want to create a virtual method that only precisely those derived classes you know about can override, the best you can do is make it internal. -- Eric I'm going to guess that people actually want something else, like a method that can be overridded in a derived class but hidden from children of the derived classes. This would entail a method being visible to a class's parent but not said class's children. If that's the case, maybe what you want is more like "semiprivate virtual" or "overprotected virtual" or use a different definition of "virtual" from what I understand it to be. As for external assemblies not being able to declare functions "override internal protected", why not? It seems like I can easily create "override protected Car GetMazda() { return Mazda; }" and then "internal Car ShadowGetMazda() { return Mazda; }", so why force me to pollute my class's namespace like that?Anonymous
March 25, 2010
Knights and Knaves can both say, "The time is 12:15". I don't believe this has any relevance.Anonymous
March 25, 2010
Ah, so "protected internal" means protected "union" internal, not protected "intersection" internal. <i>Private virtual methods are illegal in C#, which irks me to no end. I would totally use that feature if we had it.</i> Why are they illegal? IOW, what were the language designers thinking? Would it break existing code to make them legal?Anonymous
March 25, 2010
> Making an external override protected internal would make it internal to the overriding assembly, which (outside the inheriting type) has no business seeing the method. But the assembly can already expose a protected internal method as internal, by wrapping it in a method declared as internal. It just requires one extra step of indirection from the guy who's doing it in his class, but the result is precisely the same. There is an old adage regarding "protected" modifier, which goes as follows: "Remember that 'protected' really means 'public'!". And that is very true. Sure, you only expose the member to derived types - but they can expose it to anyone they want! The only exception is when you control the entire hierarchy by e.g. making the constructor private or internal. In light of that, what is the purpose of restriction of widening visibility on override? Just an extra hoop to jump through to ensure that whoever is doing it, really know what they want?Anonymous
March 25, 2010
> OK, I'm lost on this whole "private virtual" thing. Tell me where I'm wrong here: A "private" method is a method that only the base class knows about; a derived class cannot call that method and can create its own method by that name without needing "new" to avoid a warning. A "virtual" method is a method which has an implementation in the base class, but for which a derived class can provide an overriding implementation if it wants to. Logically, a "private virtual" method would be one which the derived class can override but not know about. How does that make sense? Private virtual methods exist in C++, and there is a school of thought (to which I subscribe) which advocates their use in lieu of protected virtual methods. The problem with understanding that you have comes from overly narrow definition of "private". At least in C++, "private" doesn't mean "something that only the class knows about". It means "something that can only be referenced by name from within that class". There are various consequences for this rule - for example, in C++, a public class method can return a private struct declared within the type, and the method would still be callable from outside; furthermore, members of the struct it returns would also be accessible, so long as they themselves are public. You can pass the returned value to a function template using template argument inference, and the private type will be inferred; from there, you actually have an alias to it, and can declare variables of that type etc. Or, in C++0x, you can just use auto/decltype. The important consequence, though, is that, while a private virtual method cannot be called from outside the class, it can be overridden in classes derived from that class, since an overriding method does not in any way references the method it overrides by name; it just happens to be a method with the same name, and that is it. The reason why you'd want this is when virtual methods are used strictly as hooks (e.g. like Collection<T>.AddItem in .NET). In this case, they are provided solely for derived classes to override, but they already have a strict contract on when, exactly, they will be called. If they are protected, derived classes can break that contract by calling them themselves. By making them private, you ensure that only the base class can call them, and thus guarantee that contract always holds, for the entire hierarchy (i.e. you don't have to trust any intermediate base classes in the chain to not break things for you). One could argue that "private" is, perhaps, a misnomer for it in general. Nonetheless, due to the peculiar definition of "private" in C++, and the established use of "private virtual" in C++ as a consequence of that, it is what it is.Anonymous
March 25, 2010
> Why are they illegal? IOW, what were the language designers thinking? Would it break existing code to make them legal? I suspect that there is some simple rule of thumb, such as "private members of the class are never part of its API surface", that they wanted to preserve. Private virtual methods aren't really private in that sense - now you have to document them, all class versioning problems apply to them, etc.Anonymous
March 25, 2010
"When Elroy describes what access he has to Mazda he is only allowed to say "I grant access to this to myself and my descendants" " But if Elroy decides to replace the Mazda with a newer faster Mazda 3 (ie, override the Mazda), he may be doing so with the intention (based on his declaration) that only his relatives (descendants by virtue of 'protected' and ancestors/siblings by virtue of 'virtual override') can use it. But little does he know, Charlie can actually use his brand new M3, because Charlie just thinks it's Alice's car... I can't see any situation where this could really be a problem, because Elroy has no way of providing visibility of anything back to Alice without also providing visibility to Charlie, but is this actually the case, or is there something else that could happen? PS: Where's Bob? Why'd Eve get a gender reassignment?Anonymous
March 25, 2010
Chris B.: > Making a method more accessible could end up allowing a caller to break the invariants of a class by having it called in unexpected ways or at unexpected times. [...] I don't see the logic in this. If a method is declared protected, then a subclass is able to call that method at will, or to forward another (more accessible) method call to it. Disallowing making an overridden method more accessible does nothing to maintain invariants. The subclass can break invariants in any number of other ways if it chooses to do so. I understand why narrowing accessibility is disallowed, but I still fail to understand the reasoning behind disallowing widening accessibility. If A.f() is protected, and B.f() overrides A.f() and makes the method public, B has done nothing that it couldn't do by simply exposing B.g() publicly and forwarding that to A.f(). Or to put it in terms of Elroy and Greg, it seems that Elroy can easily give Greg access to Mazda. They just call it Hyundae instead.Anonymous
March 25, 2010
@Derek: indeed, the CLR permits increasing the visibility of overridden methods. It's C# that's lacking.Anonymous
March 25, 2010
Eric, I have also found it annoying that you can't create private virtual methods for the case when you have a nested derived class that you want to be able to override the behavior of it's base. An alternative (albeit a hacky one) that I have use in place of private virtual methods is a delegate-based dispatch to the overriding method. Essentially, I'm creating my own virtual function dispatch table. There are numerous improvements that can be made to the code below, but I think it illustrates the pattern: class Program { static void Main(string[] args) { var animal1 = new Animal.Cat(); var animal2 = new Animal.Lion(); Console.WriteLine("animal1 is a " + animal1); Console.WriteLine("animal2 is a " + animal2); } } abstract class Animal { private Func<string> WhatAmI; protected Animal() { WhatAmI = ImAnAnimal; } public override string ToString() { return WhatAmI(); } private string ImAnAnimal() { return "animal"; } public class Cat : Animal { public Cat() { WhatAmI = ImACat; } private string ImACat() { return "cat"; } } public class Lion : Cat { public Lion() { WhatAmI = ImALion; } private string ImALion() { return "lion"; } } }Anonymous
March 25, 2010
That's not surprising. The CLR seems more general than C# in a lot of ways (which I suppose makes sense). I'm just not clear what the logic is in C#'s restriction, with the possible exception that it might be easier to implement this way. Although with the introduction of the "protected internal" rule that Eric describes above, it's probably not any simpler any more.Anonymous
March 25, 2010
@Pavel "Private virtual methods aren't really private in that sense - now you have to document them, all class versioning problems apply to them, etc." But if in C# private virtual methods can only be overridden by nested classes, versioning problems are restricted to the code you're already changing anyway. Since all the relevant code is under your control, I don't think there is much of a versioning problem. Documentation, too, seems only necessary to the extent that you document all your other private members.Anonymous
March 25, 2010
Derek, There IS a big difference. If a derived class overrides a method, then invocations of that method FROM THE BASE CLASS will use the overriden implementation. If a derived class "shadows" a method so that funcationallity can be accessed from outside the class, it does NOT change the behaviour of the BASE class.Anonymous
March 25, 2010
@Derek, I agree. I was including the overriding class in the set of potential callers. They can either call the protected method directly or create a proxy method to expose it publicly. The only thing that is missing in C# currently is the ability to bypass the proxy method. Although this is a very flimsy defense for invariants, its better than nothing, so I still feel it should be preserved. Its kinda like protecting a resource with a pole. I guess I'll have to walk around before I take it. What I really should have said is that it allows invariants to be broken more easily. I think a lot of this comes from access modifiers determining both who can alter the behavior (override) and who can invoke the behavior (call). AFAIK, the use case for most protected methods is to make them overridable in derived classes. They are not generally intended to be called from arbitrary locations in the derived classes. The only call from those derived classes is usually to go back up the inheritance chain to invoke the base implementation before/after doing pre/post processing. What would be ideal is if there was a mechanism by which we could express the need to override the method in any derived class, but only call from the current assembly.Anonymous
March 25, 2010
@David, I think we're talking about separate concerns. I wasn't discussing overriding vs hiding/shadowing. I'm talking about opening accessibility when overriding. Whether the overriding class changes the behavior of the method is a separate issue from whether the accessibility changes. @Chris, I can kind of understand the logic behind restricting the access to exactly that of the base class. I just think the logic is weak, because I'm not sure it's got any real benefits. As you pointed out, accessibility addresses multiple concerns. But it seems pretty trivial to work around the small restriction the same-accessibility rule imposes. I was really hoping someone could provide either a real-world case that this rule helps, or possibly some other reason for this rule that I'm not aware of.Anonymous
March 25, 2010
Narrowing access: this is not really possible, anyway. One can always cast your instance to the base class and use the access modifiers initially granted. So if class A has a public method M, and class B tries to restrict it to a private M, calling code could always use ((A)new B()).M(). Widening: sure, this is actually possible, since a class could expose your protected method through a new public method, delegating the call. But exposing "inner workings" of another class when they weren't meant to is rarely a good idea. So I think it's safer if the language doesn't allow that too easily. Private virtual: actually, you have them in C# (with a very slight twist). You can always put the sealed keyword on an override. So this is your private virtual pattern (see remark below, though): public class Outer { private Outer() {} protected virtual M() {} public class Inner : Outer { sealed protected override M() {} } } The only "restriction" is that you can't inherit directly from Outer (as that would allow a class to override M). This can be worked around be creating an inner class for that purpose. In all other respects, this is your private virtual pattern: you define some method and only the inner classes can override it. Other classes can be further inherited, but they can't modify M anymore.Anonymous
March 25, 2010
"But if you want to create a virtual method that only precisely those derived classes you know about can override, the best you can do is make it internal." If something is internal, then only classes in the contained assembly know about it, right? And if it's in my assembly, then I know about, right? So how would "private virtual" give me any expressive abilities that I don't already have?Anonymous
March 25, 2010
@Derek, but if you open the accessability, then you provde the capability for OTHER classes to ALSO CHANGE THE BEHAVIOR [by having them create an additional override!Anonymous
March 25, 2010
Dear Eric, I just came by to say that this Blog is amazing, I have been following your posts for more than a year, and found it really interesting. each post has something for me to learn, and always described in an easy way. This post was also one of them that made me smile when I finished reading it.Anonymous
March 25, 2010
@Gabe "f something is internal, then only classes in the contained assembly know about it, right? And if it's in my assembly, then I know about, right? So how would "private virtual" give me any expressive abilities that I don't already have?" Suppose: class Parent { private Parent() {} private virtual Method() {...} class NestedInheritedClass: Parent { private override Method {...} //Valid as its accessible } } class ExternalInheritedClass: NestedInheritedClass { private override Method {...} //No cigar. Method is not accessible. } What you can do through this mechanism is allow the nested inherited classes to override some virtual methods while you dissallow all external classes of the same assembly from ever being able to override said method. Right now you can't do that. You can dissallow classes from other assemblies of ever doing that, but not classes that belong to the same assembly. At least thats what I think Eric is saying.Anonymous
March 25, 2010
> And if it's in my assembly, then I know about, right? Not really. Dump some stock assembly from BCL using ildasm sometime, and count the number of types within.Anonymous
March 25, 2010
Isn't making the constructor private sufficient to restrict only nested classes from overriding protected methods (marking those overrides sealed)?Anonymous
March 25, 2010
@David > but if you open the accessability, then you provde the capability for OTHER classes to ALSO CHANGE THE BEHAVIOR [by having them create an additional override! Unless the function is declared internal only, this capability already exists. If class A declares function f() as protected, B (extending A) can override it, and C (extending B) can also override it. This would be the case regardless of whether B makes the permissions on f() more open or not. In fact, this is the case even in the case of an internal-only function. e.g. A declares f() as internal. B (extending A, same assembly) declares g() virtual, and overrides f() to just call g(). C (extending B, but in a different assembly), can now change the behavior of f() by overriding g(). If a class allows subclassing and overriding, it is trusting those subclasses to do the right thing. But it cannot force them to.Anonymous
March 25, 2010
> Isn't making the constructor private sufficient to restrict only nested classes from overriding protected methods (marking those overrides sealed)? It is, but it also prevents outside assemblies from deriving in general, which may not be what is desired. (though, to be honest, I've never actually ran into a case where private constructor + nested wouldn't be enough)Anonymous
March 25, 2010
So "private virtual" methods are only needed by developers who work with such large assemblies that they can't trust developers writing code in the same assembly to respect their invariants? That doesn't seem like a terribly good reason to add a feature to a language.Anonymous
March 25, 2010
The comment has been removedAnonymous
March 25, 2010
@Gabe - Designing so that the computer enforces the devlopers intentions is always a good thing. It is not even a matter of "trusting others"; it also lowers the chances that a single developer will make a mistake [we ae all human].Anonymous
March 25, 2010
The comment has been removedAnonymous
March 25, 2010
@gabe "So "private virtual" methods are only needed by developers who work with such large assemblies that they can't trust developers writing code in the same assembly to respect their invariants? That doesn't seem like a terribly good reason to add a feature to a language." Then why do we have private at all? We could all get by perfectly with internal...after all we trust our developers don't we?Anonymous
March 26, 2010
Pavel wrote: <quote>There is an old adage regarding "protected" modifier, which goes as follows: "Remember that 'protected' really means 'public'!". And that is very true. Sure, you only expose the member to derived types - but they can expose it to anyone they want!</quote> This couldn't be more wrong. Derived classes can give away access only to protected members of instances of the derived class. Derived classes get exactly zero access to protected members of instances of other descendants of the base class. And since C# actually enforces the runtime type cast, you can't fool the compiler into giving you access you shouldn't have.Anonymous
March 26, 2010
@Ben Voigt I think what Pavel was implying is that when you design an extensible class, marking a method as protected does not ensure that you are limiting access to a derived class: Class A { protected void SomeMethod() {} } Class B: A { public void SomeOtherMethod() { SomeMethod() } } So essentially anyone consuming B can have public access to a theoretically protected member of A even if it was meant to be protected. Horrible practice but possible and thus Pavel's adage.Anonymous
March 26, 2010
The comment has been removedAnonymous
March 26, 2010
@Aaron G I agree completely...I'm not saying it should be done, I'm just stating the fact that it can be done in response to Ben's post and clarifying what I think Pavel meant to say with his adage.Anonymous
March 26, 2010
The comment has been removedAnonymous
March 26, 2010
The comment has been removedAnonymous
March 28, 2010
"Elroy can still give the Mazda to Greg - it's just that he's not ALLOWED to. Nothing is physically preventing him from doing it, but when he does it, he's being a bad boy." But the "not ALLOWED to" is part of the specific analogy, which is questionable as to whether it's applicable to all possible situations in C#. The inability to widen access in derived classes is a (poor) attempt to physically prevent it. And it only even reflects the intent of the author of the base class because of the existence of this restriction - if the restriction did not exist, then there would be no reason to think that the developer of the base class did not intend for it to be possible to widen access.