SerialSebMicrosoft and the Magic sweet spot
As I was coding away on the train today, for a few minutes my mind blanked while watching the landscapes Virgin offer to us regular travelers. The did an amazing job at decorating the length of the track, to the point where you'd nearly get fooled into thinking there are actually villages, cows and other things between London and Manchester. As it happens sometimes, I started thinking about my early days as a fresh and learning developer, many many years ago.
Against the normal academic training some IT people receive, I evolved as a developer from the top down. I didn't do University and got no degree in IT. I actually got as far as studying law. I learnt everything by starting to write HTML web sites back in 95, then started JavaScript in 96 and VB in 97, and discovered JScript.net and C# when I got through a friend an early preview of what was to become .net (NGWS anyone?). And I'll admit that while I understand how x86 works and how pointers work, I am absolutely unable to write a single like of C++ or any other esoteric close to the metal languages (I hope esoteric here is not taken as a challenge of C++ usefulness, as I'm sure the gurus amongst you will have understood). And it's only after all these years of coding that I start understanding calling conventions, Win32, etc.
It all got me thinking about how you understand a language and a platform. Until you reach the thorough knowledge that only years of experience will bring you, you often ignore the why question, preferring the answer to the how: you care about how you achieve a result, with the acceptance that you do not understand why it needs to be done in a specific way.
For example, in JavaScript, you'd add a method to an object type by adding it to its prototype property. You know that instance methods and type methods are two different beasts intuitively, because var $cube = new Cube(); $cube.getValue = function () { return 2; } will not work for another Cube you'd create later on. This empirical knowledge is surface magic for however long you'll take to dig deeper, learn and discover what a prototype is and how it relates to objects. At this stage, if things go wrong you'll probably have to learn more things. And it's an open ended confusing thing because both $cube.getValue and $cube.prototype.getValue can exist.
Another example comes from the WPF world.. Whenever you create a custom library of Controls for WPF, you learn about themes and how styles and templates get applied to your controls. You know that the themes being selected for a control are the end of the line customizations you provide, and you know you have to declare an attribute at the assembly level, and we're still very much in the domain of empirical knowledge. But you also know that the fallback when you don't provide themes is a file in the themes folder called general.xaml. You know the how but the why has no answer. It is a decision the framework designers took, and you have no way of changing it or influencing it in any way. This by design rule is crystal magic. The behavior cannot be changed, and once you learn how to do it, the why question is irrelevant. And if it doesn't work, there's no additional knowledge cost, because you don't need to learn anything else.
My point in all that is that these two magical elements have conflicting developer audiences. More Junior developers (and some senior developers I've worked with in the past), let's call them Mortimer, are happy with crystal magic, because it massively reduces the surface of the API you need to learn. More senior developers that have a will to understand (anyone that uses reflector as his main documentation tool would fall in this category), let's call them Einstein, tolerate surface magic because they can crack the surface, dig deeper and know all the intimate details of how the surface is achieved. They don't necessarily like crystal magic because it just works and they have no control over it, but as it's a closed ending, there's nothing that can be done about it, so they move on.
Everything would be for the best if surface magic worked for Mortimer. APIs like WCF provide a very simple surface magic, with only ServiceContract, OperationContract and DataContract concepts to learn. With the tooling support for WCF services in visual studio, the surface is small enough for most developers to start being productive very quickly. The reality strikes as soon as something unexpected happens: passing too big of an object graph, or worse, a cyclic reference within the DataContracts. Suddenly, Mortimer faces understanding notions such as bindings, behaviors, configuration files, and many other concepts that multiplies exponentially the API surface they need to understand before they can fix a problem. You just designed an API that targets Mortimer when everything works, but that requires Einstein to fix if anything goes wrong. That said, tooling support is appropriate for being productive, which should ensure WCF adoption, so we're pretty close to the sweet point.
WPF on the other hand has a bit of both. Xaml is easy enough for Mortimer: it's understandable and can be fixed by Mortimer, because most of the bindings between DependencyObject and Xaml follows a crystal magic design. The property store will map Xaml and DependencyProperties based on the name, and nothing you want or can do will change this. You also very much pay as you go when you learn Xaml, because you can start with simple object graphs, and discover attached properties at a later time. You can use them without understanding them, and often you'll be able to fix issues by learning a bit about the Binding syntax. On the other hand, it's a very large platform and requires much more time than Mortimer would spend learning a new technology. Areas like Bindings are complex when they don't work, outputting errors to the debug window is confusing when you don't know where to look, the coexistence of logical and visual trees is difficult to grasp for winforms developers, etc. Tooling support should help, but from what I see in beta 2, I don't believe the sweet spot will be hit with this version of Visual Studio.
And finally PowerShell is a deliciously efficient crystal magic solution. Things work as expected, pipes and formats work by magic, using a defined behavior that will not change. The framework is static enough that people are confident that the knowledge they learn using the script engine will be valid whichever Commandlet they call. The built-in variables work automatically. $_ will always have the same meaning, $$ too.
What Microsoft doesn't have anymore is a completely crystal magic solution for development. ASP was very much crystal magic, ColdFusion was too, back in the CFML days. Early days VB had few extensibility points which let developers grasp the platform more easily. PHP still has a bit of that feel, just like many other frameworks that limit choice and extensibility to increase the pace at which their developers become experts. When most of your developers know 80%+ of your API, you know you're on the right track. Anything under that and you face a problem: You're segmenting your market between Mortimer and Einstein. Your Mortimer has to get more knowledgeable to survive your technology update, your Einstein will just keep doing his expert job. The low end of the Mortimer market is gone to join the ranks of other easier and simpler technologies.
The sweet spot is not to try and provide one API for everyone, but to provide one crystal magic API for the quick and easy, and a completely different solution for Einstein and upgraded Mortimer.
Getting dynamically typed languages with the DLR is going the right direction. But now, this is a call for action to you Microsoft: Provide us with simple tools with a lot of magic to gain back the people gone to LAMP. Give us a forms design tool that maps any control on a form if the form is named after a table and a control is named after a field in that table. Give us a web tool that simply easy work with no mvc, no presenter, and a simple script language to inject a bit of server side code where it makes sense.
And remember that a crystal magic API is one that you *do not extend*. Not everyone needs a framework.
P.S.: This is just my opinion on why we're loosing so many people to php and other simpler technologies. I have a lot of designer friends that just plain refuse to touch any Microsoft technologies, because PHP is so simple. I'm probably not the target audience for any of this. Any resemblance with current or previous coworkers is purely intentional. No guarantees implied or granted. Do not use in a nuclear facility. Etc.