The Roslyn Preview Is Now Available

  • Artykuł

I am super excited to announce that the Roslyn project code is now sufficiently coherent that we can start showing it to customers!

But I am getting ahead of myself somewhat. What is this "Roslyn" project?

Here's the deal. We've got these great premiere languages for .NET development, C# and Visual Basic. Obviously the compilers need to do considerable lexical, syntactic and semantic analysis of the code in order to first off, produce IL out the back end of the compiler, and second, produce all of that on-the-fly analysis needed for syntax colourization, IntelliSense, automatic refactorings, and so on. We do all this work to produce an analysis of the code, but we do not let you, the customer, take advantage of that analysis engine directly. Rather, you just get to see how the compiler and IDE teams make use of that analysis engine.

This is unfortunate. When I look around just Microsoft I see a dozen little C# and VB language lexers, parsers and semantic analyzers that different teams have written to meet their own needs. There have got to be many more out there in the wild. This is a problem we can solve once for you, and then let you make use of the analysis engine for your own purposes.

Furthermore, it became clear as we were designing the next versions of C# and VB that the existing compiler infrastructure built in unmanaged C++ was not going to meet our needs without a major overhaul at some point. We want a compiler infrastructure that supports both new interesting language features, and more dynamic ways to interact with your code as you're developing it.

To achieve all those goals we decided to join the C# and VB teams together, and then split the resulting team apart into two subteams: one concentrating on upcoming C# and VB features like async/await, and one concentrating on the long-term future of the compiler and tools infrastructure. The latter team is codenamed "Roslyn", and that's the team I've been working on for some time now.

We are now at the point where the Roslyn codebase is sufficiently fully-featured and coherent that it makes sense to start getting feedback from customers. You can download the Roslyn Community Technology Preview as of right now; it installs as an extension to Visual Studio 2010 SP1. We would love to get as much constructive feedback from you as possible; rather than leaving comments here, please leave comments on the Roslyn Forum. That way we'll be sure that our crack team of program managers sees the feedback and can aggregate it all appropriately. We're making a huge investment here and want to know that we're heading in the right direction to make something insanely great for you guys.

What we're looking for feedback on primarily is the set of code analysis APIs we're exposing to you guys. The APIs themselves as a set of classes with methods and so on, are now reasonably complete; we do not anticipate making major changes to the infrastructure for Symbols, SyntaxNodes and so on unless we hear loud feedback that we have gone with the wrong model. Does this set of APIs meet your needs? What parts do you like, and what would you like to be different?

I want to make sure that we're setting expectations appropriately here. This is a very early pre-beta-quality release of some extremely complicated software. Keep that in mind as you are using it. The lexical and syntactic analysis engines that underlie those APIs are pretty much complete. The semantic analysis engine that sits behind the semantic analysis APIs is right now nowhere near done; on the C# side we are still missing semantic analysis of major feature areas like query comprehensions, attributes, iterator blocks, optional arguments, dynamic, async/await and unsafe as well as a bunch of control flows; VB is in a similar state. Extension methods, method type inference, lambdas, and generics are working, so LINQ queries in the "method call" syntactic form should work.

Also, the performance of the entire analysis engine is not as high as we would like; we have done some performance tuning but there is more to come.

In short, you're getting the earliest possible build we could reasonably show to people to get good feedback; we are still a long way from being done. We are absolutely not announcing any dates or "ship vehicles" for a final release; even if I knew - which I don't - I couldn't tell you, so don't ask.

We are also as a part of this release shipping a preview of a new "interactive scripting" environment for C# that will allow you to play around with code in a more experimental and free-form way, like you can do in F# and other "Read-Eval-Print Loop" languages. Give it a spin and let us know what you think! (Sorry, no VB version of this feature is available with the CTP.)

Over the next few months I'll do some more posts describing the interesting technology we've built behind-the-scenes to make the C# and VB Roslyn code analysis engines work. For now, you can get more information on Roslyn at the following places:

Some helpful blog posts:

Fun stuff:

Comments

  • Anonymous
    October 19, 2011
    Quick question: is there issue with installing Roslyn on a system with the async CTP already installed? I would guess not, but it would be nice to be sure before I hose things...

  • Anonymous
    October 19, 2011
    This is very nice to hear! It's good to know you're making the progress you were hoping for and it sounds like it's going to be part of .Net v.next (though, of course, you can't promise anything). Suppose I'm building my own .Net language. Will the existence of Roslyn help me integrate it better with VS, e.g. ease syntax colouring because I could use some of the same framework C# does? Or will it be pretty much the same as integrating a new language is today? Roslyn is about code analysis for C# and VB; it is not a set of APIs for creating new language services. The new C# and VB language services will be built on top of Roslyn. Roslyn will be a consumer of the Visual Studio APIs that allow you to create language services, not a producer of those APIs. -- Eric

  • Anonymous
    October 19, 2011
    Jon: it should be fine side-by-side with AsyncCTP. We've tested this configuration.

  • Anonymous
    October 19, 2011
    The comment has been removed

  • Anonymous
    October 19, 2011
    Will be possible to extend the language, with macros and new keywords, like Boo language? Nope. Roslyn is about code analysis, not about extending the C# or VB languages. -- Eric

  • Anonymous
    October 19, 2011
    Will it be integrated in VS2012? What part of 'we are not announcing ship dates or vehicles' was unclear? -- Eric  

  • Anonymous
    October 19, 2011
    This looks fantastic! can't wait to see what great use will come out of it. One question, will there be any (partial) backward compatibility with VS2010 and possibly lower for using this? We don't have plans for that at this time. -- Eric ?

  • Anonymous
    October 19, 2011
    Big thanks for adding an interactive scripting environment !!!

  • Anonymous
    October 19, 2011
    Will Roslyn have any effect on the day-to-day programming of your traditional business developer (like System.Linq.Expression did) or will the effects mostly be felt through improved tooling?

  • Anonymous
    October 19, 2011
    Congrats on getting it out! Just out of curiosity, why do many of Syntax's collections contain an extra property that doesn't seem to add anything?  For example, if I have a MethodDeclarationSyntax (representing a method), and I want its parameters, I have to type  method.ParameterList.Parameters.  It seems like method.Parameters would have sufficed and produced slightly cleaner code, as ParameterList adds nothing. I love the use of the "Opt" suffix on things that are optional!  It's the next best thing to non-nullable types.

  • Anonymous
    October 19, 2011
    So, is this like LLVM for the CTS/CLR? Absolutely not. Low-Level Virtual Machine (LLVM) is a tool that takes code in an intermediate form from the semantic analysis of a compiler, optimizes that intermediate form, and then emits optimized machine code. Roslyn takes in C# and VB source code and produces a lexical, syntactic and semantic analysis of it for you, and then emits the code in an intermediate form. The thing in .NET that most closely resembles LLVM is the jitter. -- Eric

  • Anonymous
    October 19, 2011
    Are you aware that Roslyn.Compilers.CSharp.dll is not verifiable due to a compiler bug?

  • Anonymous
    October 19, 2011
    @Jason: "So, is this like LLVM for the CTS/CLR?" no. The number of people asking this in so many different ways suggests a real discontinuity in expectations Eric. I'm pretty sure that's not a good thing, but I'm stumped if I can see how you could possibly have made it any clearer in the previous comments.

  • Anonymous
    October 19, 2011
    Can I ask why you didn't reuse the expression-trees from System.Linq.Expressions? Good question; complicated answer. First off, expression trees do not solve all of the problems we want to solve in Roslyn; they do not represent lexical or syntactic state. Second, it is hard to see how we would build an expression tree to represent, say, "this expression tree represents the addition operator on a delegate where the left-hand side contains an overload resolution error and the right hand side is a lambda whose types cannot be inferred". We need to consider "broken code" scenarios at a deep level in Roslyn because most of the code it processes will be broken. Code processed as you type it is usually incorrect code in some way. Third, we've decided to not expose the "bound trees" directly; rather, we expose the syntax tree and then provide an API through which you can query the semantic analysis engine for facts about that syntax location. Fourth, expression trees are deeply tied to Reflection. That then of course raises the question of why we created our own Symbol types rather than using Type, MethodInfo and so on. Again, it is because reflection types were designed for scenarios in which the code is already in an assembly, correct and ready to run. We need to represent error situations like "this type inherits from itself", and so on. -- Eric   And in the same vein, what is the compilation mechanism for Roslyn: CodeDom, Lightweight Code Generation, Reflection.Emit or something new? We use a slightly hacked-up version of CCI to do metadata emitting and code generation. Naturally we are working with Microsoft Research on better integrating our code analysis and generation efforts. It is unclear at this time exactly what software, if any, will result from this effort so it would be premature to comment on it further. -- Eric

  • Anonymous
    October 20, 2011
    Is there any alternative to play with Roslyn if I don't have Visual Studio 2010? Maybe Visual C# Express? Can I get just a set of dlls? No, no and no. Sorry! The Roslyn CTP requires Windows 7 or Windows Server 2008 R2 with Visual Studio 2010 SP1 Professional, Premium, or Ultimate. And you also have to have the VS SDK installed. -- Eric

  • Anonymous
    October 20, 2011
    Is there a plan to reimplement the existing tools to use roslyn once it's done(such as intellisense) or is that too large of a customer impacting project/too far into the future to tell? No; the plan is to re-implement the existing tools like IntelliSense to use Roslyn before it's done. There is no reason why we have to wait for the whole thing to be finished before we start using it internally! In fact, that would be a bad idea; by using it internally we are discovering very early what parts of the APIs work well and which ones need improvement. We have been re-implementing important tools like the "extract method" refactoring to use Roslyn for some time now; we are now at the point where we are satisfied that our APIs will meet our needs to build tools on top of Roslyn. The relevant question that motivates the CTP is are the APIs good enough for you to build the code analysis tools you want with them? -- Eric

  • Anonymous
    October 20, 2011
    Great work to all of you involved.  I am actually at a conference at Microsoft today and got the chance to see Alex Turner demonstrate the capabilities of Roslyn as a part of his talk on C# today.  Really impressive stuff and I am excited to see what direction Roslyn and the other related parts (especially meta programming, which Alex explained rather well why there is some hesitation over the subject) take C# in the future. I can definitely see this having advantage for development groups being able to do things like build their own static analysis tools for enforcing coding standards specific to their team.  I really can't wait to get my hands on the CTP and start prodding it and seeing what sort of things I can do with it.

  • Anonymous
    October 20, 2011
    Will there be some documentation on MSDN or discussion here about how C# works in "scripting"/top-level expression mode and how the incremental submissions in interactive mode trample on or succeed each other? I read the original request for comments here a few years ago and I'd appreciate a look into what has since transpired. That would be a great blog topic. I do not understand this area very well myself; I've been working on the semantic analyzer mostly, not on the interactive code. It would help me to learn it if I wrote about it, so perhaps I shall do so. -- Eric Additionally, will it be possible to ask for the expanded syntax tree of some code, where (for example) iterators are turned into classes, foreach loops into whiles and usings into try-finallys, or will these conversions continue to only happen from C# directly to IL? I have a theory it could make flow analysis more consistent, although if you're already supporting the special functionality of all these things (I didn't even mention hoisting locals for closures or the await state machine), I guess the extra logic for this is just a few more drops in the ocean. We considered exposing the "lowered" forms but decided to not do so. The reason is that we decided to only expose the "syntax" trees and not the "semantic" trees. If you want semantic information about a syntax tree, just ask for an analysis of it. Most of the "lowerings" you talk about are lowerings on the semantic tree; only query comprehension rewrites are purely syntactic. -- Eric

  • Anonymous
    October 20, 2011
    As an addition to my previous questions, thank you all very much for doing this. I've been hoping for something like this to happen for years even before the word started going around back in 2008. You are pushing the state of the art forward for statically-typed languages and you are entrusting the community with the same smarts Microsoft has. Besides also making it open source, which I know you want to happen, I can think of no better sign of respect.

  • Anonymous
    October 20, 2011
    Could I use this as a replacement for the missing parts of the Microsoft.CSharp namespace, so I can more easily do dynamic code generation?

  • Anonymous
    October 20, 2011
    So, is my understanding that Roslyn code analysis features are not and won't be available outside Visual Studio correct? Or is it true only for this CTP? The code analysis APIs are just a library; you can new up a Compilation, start plugging code and references into it, and query the compilation for facts about the analyzed code. Obviously the infratstructure that does stuff like adding smart tags to the VS editor requires VS. -- Eric

  • Anonymous
    October 20, 2011
    Is there any Roslyn API documentation online? Full API documentation is not yet written; the complete set of APIs is enormous and still somewhat in flux, so it would be premature to spend a lot of resources on documenting them. I'm not sure what all is included in the CTP as far as documentation goes. -- Eric Based on my reading of the overview material, it seems like Roslyn isn't just a code analysis tool (compiler front-end), it's also an IL emitter (compiler back-end). This means it should allow you to create your own language (possibly very much like C#) rather more easily than before, even if it does not allow you to extend the C# language itself. For example, you can use Roslyn to parse your C#-like program, invoke some macros (think of them as declarative refactorings) that you define to rewrite the syntax tree, and then use Roslyn to generate assemblies from your output -- or convert the rewritten syntax tree back into C# to be compiled by ordinary means. Previously this sort of thing was performed only by complex IL manipulation tools like Cecil or PostSharp. Now you can write your own "PreSharp" in a day! At some point in the future, the C# compiler itself could even have extensibility points that you can hook into with Roslyn APIs. Did I get any part of this wrong? Though it might be possible to do all that with Roslyn, that would be using the tools "off label". They are not designed for those scenarios. The point of Roslyn is to enable analysis of C# and VB programs or program fragments, not to enable analysis of languages that are in some way similar to C# and VB. I agree that it would be very nice to have an IL-emitting compiler back-end that took as its inputs a form "lower" than C# or VB parse trees, but "higher" level than IL. We're kicking around ideas in that space and discussing them with the Microsoft Research guys that work on CCI, but like I said before, these ideas are still too inchoate to discuss further. -- Eric

  • Anonymous
    October 21, 2011
    Gabe: It's curious that Anders has previously opposed tools like PostSharp on the grounds that IL weaving is messy. I wonder what his opinion would be of Roslyn-powered precompile-conversion, official extensibility points or not.

  • Anonymous
    October 21, 2011
    Ah, so this is not anything like LLVM, but it is exactly like Clang http://clang.llvm.org/ .  From it's web page, Clang:

  • Modular library based architecture
  • Support diverse clients (refactoring, static analysis, code generation, etc)

  • Anonymous
    October 21, 2011
    Will the interactive window let me run statements in the context of a running code around a breakpoint while debugging? (Just like the watch window lets to do it with lots of limitations)

  • Anonymous
    October 21, 2011
    Artificial intelligence programming. when will  microsoft  create in .net

  • Anonymous
    October 22, 2011
    It would be interesting to implement something like that gist.github.com/1306491 in roslyn

  • Anonymous
    October 22, 2011
    The comment has been removed

  • Anonymous
    October 23, 2011
    @Stuart Your example rules can be easily verified using mere reflection. Just write a verification app and invoke it in postbuild event (or get more fancy and do an msbuild task).

  • Anonymous
    October 24, 2011
    @Stuart & @Martin: If something more standard than a fully custom verification system is desired, You can use tools like NDepend. NDepend in particular is explictly designed to use custom rules. The standard ules are mere suggestions for you to edit or diable as you see fit. The real power is adding new rules like the ones you mention, or rules to enforce layering, or similar.

  • Anonymous
    October 24, 2011
    It's a nice achievement but I'm wondering who the target audience is? C# already has some nice refactoring features and this whould merely allow some third party developers to come with their own plug in like resharper. Few developers would have an interest developing these features. Maybe others will make use of a different analysis when dealing with certain third party types, but that's breaking the familiar user experience. Your question is basically "who is the target audience for your library of code analysis tools?" The answer is "people who like to write programs that analyze code". If you're not interested in writing programs that analyze code then Roslyn is probably not the library for you. -- Eric

  • Anonymous
    October 24, 2011
    Eric, this is a great CTP which I was awaiting for a long time. I am a promoter of using pragmatic aspect oriented approaches with dynamically created proxies and interceptors. However, in the .NET area (unlike Java) up to now I have to fall back to 3rd party tools (e.g. Springframework) or use System.Reflection.Emit directly. The 3rd party tools often generate proxies which use reflection at runtime etc. This makes it inappropriate for performance reasons at times. Now with Roslyn I am able to very easily generate my dynamic proxies myself in the way I want them. I know that this may be not the originally intented use but it works quite nicely so far... Thanks again! Sven

  • Anonymous
    October 25, 2011
    Probably publish the top 10 use-case scenario of Roslyn would help to understand why we should use it. Or what is it and what is not would be even  better.

  • Anonymous
    October 27, 2011
    Any response to the comment on comparing Roslyn to Clang? How are they different? How are they similar? Thanks!

  • Anonymous
    October 30, 2011
    Duplicate code detection would be a helpful outcome and use of the new API.  Find all token sequences over X long that match treating an indentifier as 'identifier' and literals as 'number literal' would be great.

  • Anonymous
    November 21, 2011
    Just wrote my first VS plugin using roslyn. It shows a warning when a pure method is used as a statement. github.com/.../Chaos.PureMethodStatement Unfortunately the contract reference assemblies aren't redistributable, and as a non lawyer I don't know if I may distribute a list of pure functions with my plugin, so for now the full functionality will only be available when Code Contracts is installed.

Didn't come across any big issues in the API. I missed an equivalent of Type.FullName, but that wasn't hard to work around. Loading the contract reference assemblies for the framework libraries was a bit more work. Reflection-only assembly loading is very annoying(typeload exceptions,...), so I had to use MonoCecil instead.