TechEd: Metropolis - Interchangeability of Operations

[EDIT: Unbelievable but you have to sneak in the conference rooms to get access to the one power point that's hidden under traps. Hopefully no one will ask me to unplug myself because of Health and Safety but you never know... An IT conference without power plugs, what a great idea!]

I've used the term Metropolis for a set of talks. Look at Cities, Transportation, manufacturing and try and find some trends. From looking at cities we can learn a lot about what will happen in IT.

At a high level, IT shops map to cities. You have evolution that occurs in isolation, and then things get connected, which causes change. Factories and buildings are comparable to applications. Each application is part of an IT shop. Retail maps to business process. It's about tying together deliveries that come from different origins.

Manufacturing goods and structured data and operations are similar. That what we see as data structures and operations are made to fit better together.

I don't believe it's an analogy, it's not helping to think about a subject, it's the same drive behind both technologies.

What are the challenges in SOA? Explicit boundaries; Autonomy - you can replace your service without impacting the other services.

How do you perform actions across these boundaries. What trust do you use? SOAs are behaving the way you see people in business behave in the real world. Its an opening up of the autonomy and explicit boundaries.

Those that know of me know I've been doing transactions for a long time; but I've grown to believe that transactions across boundaries are very fragile. I don't think services as boundary of trusts will engage into transactions with other services. Some say two-phase transactions will happen across services, some say they shouldn't because of trust issue. For this talk, each service will have a transaction but will not hold locks for another service.

When you look at manufacturing, there was a transition from hand-crafted to automated. A machine creates the part. The shape and form was so crude with the machine that you had to manually change them before assembling. Interchangeability was driven by being able to exchange pieces between two broken guns on a battlefield to make one that work.

In computing we're massively stuck because we don't do very well on how to exchange services. The important points that you need to take from this transaction:

  • Autonomy
  • Agreement is different: Attempt and confirm/cancel
  • Interchangeability: One operation is as good as the next. Fullfill my requirement, be it that you use A or B. You execute your operation on a pool of services.
  • Semantics: How do you make the request to rearrange and reorder. You need Precision but you don't want the Intricacy.
  • Variety: It can be made out of interchangeable of the operations. If you put different goods in WalMart, the combination is astonishing, but they come from a limited number of pools.

Let's see the history of manufacturing. If you go back to the early nineteenth century, you'd order one thing and you'll get one gun. You have a lot of craftsmen, each of them builds a complete item. But each person made a different gun because it was based on their own approach. Completing anything needed fitting. Soft parts were stamped because they couldn't be exchanged.

The American way of manufacturing comes from shortage of labour. They started building machines to do the work. The problem was that the machine created parts were still inaccurate. Because of that, they need adjustment. Fitting was still required.

With the armory system of manufacturing. LeBlanc proposed interchangeable small parts but it was dropped, but it was brought back in America, or at least the idea. By creating Jugs, Gauges, and the precision work John Hall, the price of manufacturing the pieces increased because you had to be picky with the pieces you get. You had interchangeability but the economics didn't work.

From armories to bicycles to sewing machines, all the way to cars with Ford, impacts what we do today. The disassembly plant: the meat arrived in Chicago, and each butcher was responsible for cutting one piece of meat. There was very high throughput. In assembly, Ford moved the parts to the people rather than the people to the parts.

The Ford factory made sure there was no fitting, every part had to be fully designed, custom machines were made, so all the pieces could be interchangeable. The machine was giving the precision and the accuracy. Even the building was shaped to optimize the production. But it became impossible to change the car design as you would now have to knock the building down.

Interchangeability is great but there are other requirements! Which brings us to General Motors. GM gradually introduced the concept of a yearly model, which introduced the requirement of being agile. You suddenly need multi-purpose machine tools. For this you had to be able to re-arrange the components of the machine to create a new piece. Mechanization, interchangeability and rapid changeover to new models.

Let's talk about the American system of transaction processing. You start your activities, be it a human or a machine, and it goes to your database, you calculate ad you return your answer. But you have to do fitting inside the transaction. There's no boundaries when you update two databases within a transaction. Within each transaction you need to fit the work. But it gives you volume.

We need transactional "machine tools". No precision tools: our current tools can be used in different ways. Different programmers get different results. They're wonderful for labour savings, but not to separate their behaviour. Today, we have applications that have external behavior that are similar but the inside is different. If we are to take our applications apart, we need to make the internals of our applications interchangeable.

Services are connected by messaging. You don't know how the operation is executed, but you know the contract of the messages and the contract of the order in which messages are exchanged. Interaction is based on business functionality. The services share operations, not data. Sometimes there's reference data. It's like a department store catalog, you use that data to connect to your business.

What about optimistic concurrency control? Can you do optimistic concurrency work with your bank? No, because there's a security boundary. You need to think about trust.

Autonomy means independent control. My local business logic decides how i change my local data. I decide what changes. If you want a change, ask me to do a business operation.

Long-running work. Services don't share transactions. How do they cooperatively make decisions? how do independent businesses take decisions? There's tentative operations, with reservations, cancelable orders, or confirmation. If you buy a house in the states, as a buyer you give everything to the Escrow company. The bank, the buyer, the seller, everyone puts their trust in the escrow company. However the sell is only a reservation until you know if you will get it. The only guarantee is that you'll get your money back if the deal doesn't work out.

Coordinating n-systems requires that at least n-1 accept uncertainty. You system needs to be in a confused state. In 2-phase commit, the database maintains locks. With cancelable operations, uncertainty is the reservation you have for a hotel room. There's an intrinsic conflict between consistency and availability. Two Phase commit is the anti-availability protocol.

Because operations are cancelable, you can reorder them. Your cancellation is not an undo but an action you receive to cope with the failure. So what are the semantics for cancellation / confirmation?

Cancellation is about coping with the operation not being done. You accept the right to cancel. Confirmation is the guy that has the right to cancel saying that he won't, for example hotel room confirmations in the morning. Airline companies clear up their overbooking once the plain leave.

To make your operation cancelable, you want to reserve its effect until the confirmation. If an operation is unique, you must lock its effect until you cancel it. You need to make a decision between provisioning and overbooking. How do you manage the pool of resources?

One operation in a class is the same as another. If you reserve a king-size room, it could be any of the rooms. Interchangeability comes by increasing commonality.

It's a pain to offer cancelable operations. But you want to do it because your customers need that in order to function correctly. Its annoying that the customer can be fickle but it's the reality of a complex application.

We used to make decisions atomically by committing or rolling back your transactions. You want to now think about how things compose, but for that you need to have each service being part of a long running work.

In the armory system of transaction processing, you have a complete solution when you assemble small services. In manufacturing you need precision. You need to understand the constraints of your operation. We've made components that were multi-purpose. What you need to do is to have the least amount of functionality to make them interchangeable.

Resource-oriented data lives longer but is changed by long-running operations, like bank records. You have activity-oriented data, that gets created when you start doing a job, and retires when the operation is complete, like an order.

These two classes of data have different characteristics.

In both cases, the data is encapsulated within the service, and there's no optimistic concurrency control externally. You mediate access through business logic.

Let's take a Foo operation. First case, the data is isolated. If I change something about someone's order, I only change a bit of data. But for Joe's order, that may modify my resource-oriented data by changing my inventory. Activity oriented data is used for a single long-running activity and won't span several. You can create a tentative operation, it's easy to track down in case the operation is canceled, but nothing is shared so there's no impact. What happens if you share data?

It becomes harder in resource-oriented services. What is interacting with what? They make commitments and these commitments need to be tracked. You have an association between activity data, the reservation, and the resource oriented data, the list of hotel rooms that are available every day.

Specificity is not interchangeability, Variety is the enemy of interchangeability.

There's a lot of work around DSLs. You express the intent of the business domain with a domain language. That's a great foundation to think about the issues that have been raised.

This is heavy-duty reconfigureable manufacturing machines. You don't need much skills to use DSLs if they're constrained enough.

Service operations are like standard parts, whereas tentative operations compose. That lets you do standard software components and use them across several products. All the products look different on the outside but they're all the same inside.

We should learn about that manufacturing part, and do some rambling philosophy.

Atomic transactions are singularities. New challenges happen as you spread work across space and time. Interchangeability helps relax space and time. You can interleave and reorder. Variety means you have lots of options and choices, but you can't keep stocks. Interchangeability needs precision rather than variety.

EDIT: Some form of question about Wal-Mart using Child labour and how it relates to services. Sometimes reality is better than science fiction!

Update 26th November 2007:

Technorati Tags: ,


Even on Keynotes people don't set their fonts properly

I know it's not the aim to get Visual Studio or Blend visible, but for heaven's sake, LET ME SEE WHAT YOU DO and increase your font size. Silly!

Technorati Tags:


TechEd - Keynote

Just arrived in the room for the keynote. Of course as it usually happens, all the devices I carefully put on charge last night were not charged, which include camera and camcorder. Expect multimedia content tomorrow instead :(

The room features two screens on which bee videos are being broadcasted, as well as live video from two artists doing some real-time tagging. It all sounds very social network. The theme is orange and yellow, very cyber-opera! And for one the wireless network seems to be sustaining the load quite well!

[13:59] S. Somasegar gets on stage.

TechEd gives a chance to interact with some of you. A lot of you have had a crucial role in delivering .net 3.5, through feedback, technology previews, bugs, scenarios you highlighted and taking a bet on those technologies. For this I'd like to take the opportunity of saying thank you.

When I went for dinner on Saturday my colleagues organized, two things made it an experience: the food of course, but also the environment and how it was presented. Alike, Differentiated user experiences are what will drive the next generation of applications.

It's two years ago that we've launched Visual Studio 2005. As of today, there are 1 million professional developers using vs2005, and that's a phenomenal number. As well, we had over 17 million downloads, a mind-blogging response. At the other hand of the spectrum, we wanted to get into the lifecycle tool space. Over a fourth of our developers are using the Team System tools. But our developers are a community. A few years ago, if you asked a question you would get 30 or 40% chance of getting an answer. Today it's 80%.

Microsoft has always been since 1975 a platform company. Every piece of software is always thought about as having a platform component and a user interface component. The best thing we can do is to create a modular and extensible platform. We want to make sure we have the most vibrant developer and partner communities.

My 11yo does not know programming but spend a fair amount of time online, she's on facebook, she wants to customize her home-page, and share what she's creating. She wants to create an online experience even though she's not coding. My wife is an accountant, she doesn't code as part of her day to day work, but to automate some tasks she sometimes code. You have different experiences for different platforms for different users.

That's why we have different tools and different platforms.

As we speak, the product team is putting the final touches to .net Fx 3.5, which is the next big step for productivity. You should be able to use your .net programming when you do client, services, cross-browser web through silverlight. We already have .net 3.0 that we released in Vista with WPF, WCF, WF... AddIns in WPF, WCF protocol additions, designers being integrated in Visual Studio.

But my favorite is Language Integrated query, LINQ. You had the data world, and the programming world, and they were different things. You have to understand the data language, the constraints and understand the semantics to write your application. We said, let's reuse your skills, what if you could use the same query system as a first class citizen for your objects, for your data.

Silverlight is the new platform. We shipped the first version earlier this year. and we'll push further with the version integrating .net

While we finish the framework, we also finish Visual Studio. For the first time we're supporting multi-targeting. For the first time yu can have one toolset and build applications that target 2.0, 3.0 and 3.5. Looking at the feedback people are very happy we're going that path.

If you're building an AJAX platform, it's integrated in VS2008. If you want Vista technologies, .net 3, we have designers. It's also easier to target applications for office 2007. You used to have to buy VSTO and you get all the tools. We got feedback that it was more complicated than it needed to be, so we took the feedback and integrated VSTO in VS2008 Professional.

We have a good story for VS with server, client, office, cross-browser. We've integrated database tools as well.

And finally we released earlier this year Expression Studio to enable a smooth workflow between Expression and Visual studio.

We also want to continue delivering more prescriptive guidance. Patterns & Practices will continue and we added the S+S (Software+Services) solutions, so it's easier for you to build yoru applications.

MSDN used to be a publishing platform where we pushed articles. It was a good start but msdn should be a community platform. The community can come together through forums and be self-sufficient. We now have MSDN Wiki, took our documentation and published it and added a wiki to annotate content and add content. Just like we've translated visual studio in many more languages. Using automated translation we provide a base for content that the community can build on. Were working on the Portuguese documentation.

Currently we're working on MSDN Code Gallery. There's no easy way to share a sample code. MSDN Code gallery will let you do that. We'll release a beta sometime in the next month, and Live in the next year, so you can come and share your code samples Code gallery. I want to turn MSDN into a community platform and we're well on the way.

Today we're making available a technology preview for Microsoft Sync Framework. Take data synchronization to a new level: it's a framework to enable synchronization for your application, for peer2peer and offline capabilities. Take a look at it and give us a feedback. [EDIT: Is that the return of the sync work from WinFS?]

Software+Services Blueprints is the first guidance we release for standard scenarios. I'm a big user of Outlook 2007. My wedding anniversary is coming next month. Would it be nice if there could be an add-in that integrates the wedding date, buy on its own the present based on analyzing my previous spending, on its own without no intervention buy and deliver the present? [EDIT: What the frell?]

The Blueprint is a bunch of prescriptive guidance and building blocks

As we speak today, later this month we'll release the RTM [EDIT: I Wonder where the RC1 went? Or do they think beta 2 was good enough?], and we'll ship soon after that.

Tony Goodhew is working for the Prodct marketing team is going to do a demo of the cool stuff you can do with Visual Studio 2008.

[EDIT: I won't retranscript the demo, I doubt there's much inherent value for my readers in describing someone showing technology that's already available to people].

I hope you had a play with the beta. We've shown you what you can do with it. Let's see a video of what really happened to get Visual Studio 2008.

[EDIT: Video is slightly funny, I'll let to others the pleasure of commenting on it. Let's just say it emphasize the fact that VS2008 was built by VS2008.]

In 2005 we took a bit longer than we wanted to deliver, in hindsight, you did a good choice delivering it. On the way there was a whole bunch of tide overs, and we only managed to do what we did because we had visiblity on our project, where things needed help, more resources or scoping.

We have more than 200 partners devliering mroe than 2000 products. We're making changes to our partner programs. Some of our partners wnat to target other platforms, we're removing the restriction on using our tools to write for other platforms. We're also making the Visual Studio source code to our partners, and debug your add-in. Dan Fernandez will show you some of the products that can be done.

[EDIT: Demo of an open source project for World of Warcraft add-ons for designing worlds, it was built in a few weeks by a few developers. Show demo of LUA editing].

With Express editions we wanted to expand our audience. Popfly is a visual mashup system: as an open beta last motnh, there's been 100k customers.

With Popfly, you can publish your web page to popfly. Popfly explorer and visual studio integration will help with that.

[DEMO: Visual Web Developer and Popfly explorer beta: It's available at

Technorati Tags: ,