Iterator Blocks Part Seven: Why no anonymous iterators?
This annotation to a comment in part five I think deserves to be promoted to a post of its own.
Why do we disallow anonymous iterators? I would love to have anonymous iterator blocks. I want to say something like:
IEnumerable<int> twoints = ()=>{ yield return x; yield return x*10; };
foreach(int i in twoints) ...
It would be totally awesome to be able to build yourself a little sequence generator in-place that closed over local variables. The reason why not is straightforward: the benefits don't outweigh the costs. The awesomeness of making sequence generators in-place is actually pretty small in the grand scheme of things and nominal methods do the job well enough in most scenarios. So the benefits are not that compelling.
The costs are large. Iterator rewriting is the most complicated transformation in the compiler, and anonymous method rewriting is the second most complicated. Anonymous methods can be inside other anonymous methods, and anonymous methods can be inside iterator blocks. Therefore, what we do is first we rewrite all anonymous methods so that they become methods of a closure class. This is the second-last thing the compiler does before emitting IL for a method. Once that step is done, the iterator rewriter can assume that there are no anonymous methods in the iterator block; they've all be rewritten already. Therefore the iterator rewriter can just concentrate on rewriting the iterator, without worrying that there might be an unrealized anonymous method in there.
Also, iterator blocks never "nest", unlike anonymous methods. The iterator rewriter can assume that all iterator blocks are "top level".
If anonymous methods are allowed to contain iterator blocks, then both those assumptions go out the window. You can have an iterator block that contains an anonymous method that contains an anonymous method that contains an iterator block that contains an anonymous method, and... yuck. Now we have to write a rewriting pass that can handle nested iterator blocks and nested anonymous methods at the same time, merging our two most complicated algorithms into one far more complicated algorithm. It would be really hard to design, implement, and test. We are smart enough to do so, I'm sure. We've got a smart team here. But we don't want to take on that large burden for a "nice to have but not necessary" feature.
Comments
Anonymous
August 24, 2009
That just sounds scary. Seems perfectly reasonable why it wasn't done.Anonymous
August 24, 2009
What would be really cool would be for the compiler error messages to link to Eric's blog posts. They already link to MSDN, so adding "This error is explained on Eric Lippert's blog" hyperlinks automatically to topics matching specially formatted blog keywords ("keyword: CSnnnn" anyone) would be really neat. Even someone adding the links by hand would be really helpful.Anonymous
August 24, 2009
Eric:Thanks for devoting an entire post to my question(comment) about Anonymous iterators from Part5 :-)Anonymous
August 24, 2009
First, I believe that Microsoft should commit itself to achieving the goal, before this decade is out, of allowing anonymous iterator blocks and having them nest safely. No single feature in this period will be more awesome to mankind, or more important for the future of c#; and none will be so difficult or expensive to accomplish. We propose to accelerate the development of the appropriate compiler. We propose to develop alternate iterator anonymous method rewriter, much smarter than any now being developed, until certain which is superior.Anonymous
August 24, 2009
@Eric: One more approach to get this done, you could have some Sequence expression pattern as in F#(using Monads), that allows you to wrap a IEnumerable on the return values. Of course, this is not totally applicable for C#, but that could be one way to get this done for anonymous methods? What do you say? Seq(()=>{yield return x;}) Adding up, this would be a great benefit for LINQ too,Anonymous
August 24, 2009
The comment has been removedAnonymous
August 24, 2009
Any thoughts on allowing anonymous types to implement interfaces? That would be a really useful feature.Anonymous
August 25, 2009
C# is a great language and it seems to get significantly better with each version. For the most part, I really like the direction your team has taken the language. If there is one criticism I would level, it is that I feel that I often bump into what seem like arbitrary limitations of a given language feature. To put it more precisely, I feel that a lot of the language features are not very isomorphic. The word you're looking for is "orthogonal", not "isomorphic". "Orthogonal" means that two things can vary independently of each other. By analogy, "orthogonal" features are those where there are no weird interactions with other features. "Isomorphic" means "having the same shape". -- Eric I guess I'm trying to say that when you folks are doing your cost/benefit analysis, I hope that a lot of consideration is given to trying to make the language natural and consistent. Perhaps I was not sufficiently clear. The whole purpose of this series of articles was to justify why it is that sometimes we choose to make language features nonorthogonal: orthogonality is a goal, but sometimes it is too expensive. Sometimes it is better to get a feature that works well with 95% of the language. -- EricAnonymous
August 25, 2009
Anonymous iterators is simply a must do for the compiler team. More and more async frameworks are discovering that iterators can be used to implement coroutine-like functions. However, in the absence of anonymous iterators, it puts additional effort and complexity on the developer. Given that we're in the age of concurrency, there will be more need for async/concurrent/coordinated code. Do we really want to leave this responsibility to each and every developer instead of the compiler and/or a framework? It may be a complex endeavor, but it's also one that will yield many rewards! No pun intended.Anonymous
August 25, 2009
I would like to add my voice to the plea for anonymous iterators. We've built a coroutine framework on top of yield and do a lot of iterator work. But we are basically stuck in .NET 1.0 land in many ways, since we cannot take advantage of closure capturing to simplify our code. In short, adding anonymous iterators would bring us folks into .NET 2.0 land and get all the benefits of anonymous methods and I'm sure that nobody is challenging that anonymous method weren't hugely beneficial!Anonymous
August 25, 2009
Well I'll add my voice to the chorus of folks saying that anonymous iterators would be nice to have, but given a finite budget I would much rather see the brains at Microsoft implement deep const and isolation semantics into the CLR so that languages targeting it can then expose them.Anonymous
August 25, 2009
@Tom, I agree with you! [eric: I also know how difficult "const" is to properly implement]Anonymous
August 25, 2009
The comment has been removedAnonymous
August 26, 2009
Of course you can do anonymous iterators in F#: let y = 10 seq { yield y; yield 10*y } |> Seq.iter (fun x -> printfn "%A" x ) (and you can even have "inner iterators" via "yield!".)Anonymous
August 26, 2009
unacceptable! This isn't a "would be nice to have" feature. This feature is absolutely required for writing the highest readable code possible. Code readability and thus maintainability are the number one concerns when writing. Having to hunt down some generator method somewhere because I could not write it inline reduced readability and increases maintenance cost. Well then, if C# does not meet your absolute requirements then you shouldn't use it. Though it is disappointing to not delight every customer, I do not think it reasonable to expect that we'll meet all the requirements of every person. Try F# instead. -- EricAnonymous
August 26, 2009
@Jay - of course this is a subjective discussion, but I have to disagree. Too many times I have found even anonymous methods to be a MAJOR source of testability issues. With very few exceptions, I have chosen to go with well named, well factored methods. Inlining, does not [IMHO] prove readability or understandability. Well factored code where a given item performs one well defined function does improve understanding of the code, and allows a person to focus on just the specific area instead of being bound into a given context.Anonymous
August 31, 2009
Hi Eric. You say that the two algorithms for rewriting anonymous methods and iterator blocks would interact too much if anonymous iterators were allowed. I would be very interested in seeing a more detailed explanation of this. Why can't the algorithm for rewriting anonymous methods simply rewrite anonymous methods irrespective of whether they contain 'yield' or not? In other words, does the anonymous methods rewriter really need to care whether any particular anonymous method is an iterator block? The way I see it, it doesn't; you already have the algorithm in place to handle nested anonymous methods, so an anonymous iterator block nested inside an anonymous method doesn't seem like a barrier. Once anonymous methods have been rewritten into actual (top-level) methods, the iterator-block rewriter can work its magic, and at this point does not need to care whether any particular iterator block used to be an anonymous method and whether it used to be nested. Thus, to me the two features seem completely orthogonal. So orthogonal, in fact, that the relevant rewriting algorithms should be orthogonal too. What is the detail I am missing that makes them interact in complex ways?Anonymous
September 01, 2009
The comment has been removedAnonymous
September 02, 2009
The comment has been removedAnonymous
September 02, 2009
(things I forgot in the prior post) "You are probably smarter than me and find these sorts of things simple, but I do not." More like, I have a tendency to frequently forget that the C# language team is, in fact, made up of mortals. Naturally, from the results of your work, I tend to assume you are all extreme programming gods ;) (quoting myself) "make code nicer to read without precluding doing smarter, optimized codegen when the research problem is solved, wouldn't it?" - Not to mention that, presuming the research problem will eventually be solved, having a naive implementation now would mean that when that solution is found, C# can take advantage of it to produce drastically better performing binaries from existing code. If you wait for the solution to the research problem before adding the language feature that could take advantage of it, then everyone will need to scan their code for "foreach" loops that could benefit. Or the compiler's optimizer will need to be a lot cleverer to identify yields in foreach blocks that could also be transformed in the same way.Anonymous
September 14, 2009
@Steve Bjorg: Eric requested in a comment a year ago (http://blogs.msdn.com/oldnewthing/archive/2008/08/15/8868267.aspx) that programmers not make "clever use" of iterators.Anonymous
August 12, 2010
This feature isn't a big loss. 99% of the cases where I'd wanted to use it, new[] { a } is more appropriate... especially considering that any iterator will be a new object anyway. On the C# side, the language has essentially everything one could desire at this point (except A<T> : T, which opens up a whole world of great possibilities for generics, but...). .Net has bigger problems -- for example, if a struct or array of structs has no object pointers, I should be able to modify it at a pointer level without going to unsafe code. This is also the appropriate way to allow access to SIMD features...Anonymous
August 12, 2010
"I should be able to modify it at a pointer level without going to unsafe code" If you modify/access it at pointer (as in c style) then you are being unsafe as you have no bounds checking and can read/write anywhere. How is that not unsafe in managed world. Or did you mean pointers with bounds checking (which sound rather tricky to integrate as well as raw pointers and keep the syntax sane) I would like SIMD support, perhaps Mono's testing of the waters would be an option (I haven't personally tried it) but actaully what I really want is intrinsics. If I'm running on a machine with crc32 I really want to be able to get to it without having to do a managed/unmanaged transition. Having the JIT transparently give me a software based implementation when it's not present is nice, but I'd even accept it throwing NotSupportedException.Anonymous
February 15, 2011
The comment has been removedAnonymous
April 10, 2011
Thank you for writing this article. It is easy to forget the huge complexity underneath. From the face of it, it seemed trivial to just have the compiler wrap it into a private helper method just because I wasn't using any other lambda features (didn't access any variable from external scope). Your two initial points about iterator blocks and anonymous methods being the two most complicated language features, compiler wise, quickly set me straight. From proudly professing my newfound functional highground to being humble. Let's just take it as a positive signal that more and more laypeople are endeavouring into functional paradigms. And as a reminder that even C# will at some time be replaced. I hope to see you at the NDC 2011.Anonymous
February 29, 2012
It's disappointing that VB 5 supports anonymous iterators and C# 5 doesn't, while C# has supported named iterators since C# 2. I understand that it's not a generally useful feature that outweighs all of its associated development costs, but why promise language parity and then decide that this feature outweighs its costs for VB only? I'd really like to use anonymous iterators in my reactive parsers library, and I'll bet I'd find lots more unrelated uses if it were available in C#.Anonymous
February 29, 2012
Just thought of another great use for anonymous iterators: Rx Experimental contains an overload of Observable.Create that accepts an iterator block. It provides a way to write Async-like code that generates an observable sequence (as opposed to a scalar-valued Task.) The fact that I have to use a named method for the iterator argument takes away from the "flow" of a reactive LINQ query.Anonymous
July 04, 2012
Thanks for the post, that's a nice explanation. It would definitely be "nice to have" and I hope it's on the list somewhere (probably near the bottom), but there are many other features I'd consider more important :)