Todd Biske: Outside the Box

Two recent posts that were completely unrelated have prompted me to write a little bit about the human interaction side of SOA and BPM. First, in response to the debate on maturity levels between myself and David Linthicum, Lori MacVittie posted this entry on the F5 DevCentral blogs. She didn’t get into the debate on maturity levels, but rather brought up a point about the use of the term orchestration. She states:

Orchestration of applications is a high level automation mechanism that can’t really be completed until there is a solid underlying SOA infrastructure and set of common services in place. It’s the pinnacle of SOA achievement, the ultimate proof that an organization SOA can indeed provide the benefits touted by pundits. But orchestration of services should also be the mechanism by which applications are constructed.

The second post that caught my eye was Ismael Ghalimi’s post, “What is Wrong with BPM.” In this post, he talks about the problems customers face in selecting a BPM product and some of the things that customers run into after the purchase has been made and they try leveraging the solution on one of their real business problems. He states:

Then comes the really fun part: the business folks want a different user interface for their workflow. The one you got out of the box seems to be working pretty well, and you could display your company logo at the top left, but somehow the suits have something different in mind, and they want it now. They paid $300,000 for some magic pixie dust that gives them business agility, and they expect it to make you a contortionist worthy of a full-time job with Cirque du Soleil. So you end up spending the next six months writing massive amounts of JavaScript code that will hardcode the customer’s process deep into the user interface. You will be late, over budget, and won’t benefit from future software upgrades, for what you have now is built upon a completely different codebase. Great…

The two things I want to call out are Lori’s phrase “orchestration of applications” and Ismael’s laments about the quality of the user interface. I believe both of these posts are hitting on an element that is frequently forgotten around SOA, which is the human interaction. Regardless of how many services you build, some user is still going to need a front end, and there are inherently different concerns involved. Ismael’s absolutely right that some bare bones web form creation tool slapped onto the ugly schemas that represent the process context just don’t cut it. While 5-10 years ago, you may have been able to limp by with basic HTML forms, today’s web UIs involve AJAX, CSS, JavaScript, Flash, and much more. The tools that are calling themselves orchestration engines excel at what Lori calls orchestration of services (the BPEL space), but I don’t know that there are many that are really excelling at business process orchestration. I’m using my definition of business process orchestration here, which is both the human activities and the automated activities. I’m guessing that this is what Lori meant by orchestration of applications, however, I try not to use the term application anymore. It implies a monolithic approach where everything is owned end-to-end, and that simply won’t be the case in the future. If I do use the term, it’s reserved for the human facing component of the technology solution.

True business process orchestration that includes the human element, is not one that we’re seeing a lot of case studies on, but it’s where we need to set our sights. The problem is quite difficult, as the key factor is context. When I was working with a team on a reference solution architecture for BPM technology, one of the challenges was how and when to bring in context. If you rely on events to trigger state transitions, should those events carry references to information, or the contextual information itself? If it contains references, then you need access to all of the associated information stores, and you need to figure out what information is relevant for the problem at hand. It’s hard enough to get this right for an automated system where the information required is probably well defined. Now try getting it right when the events are tied back to a user interface. The problem is that every scenario may require a different set of information. As humans, we’re good at determining correlations and understanding where to go. Systems are not. Our goal should be to creating solutions that support the flexible context required for true business process orchestration. I think this will keep many of us gainfully employed for years to come.

James McGovern asks, “How come Enterprise Architects don’t embrace agilism?” He asked a number of us to share our experiences about how frequently we talk about agile methods. The first question I asked myself after reading his entry was, “What are we even talking about when we say agile methods for EA?” Agile methodologies have been associated with software development, and while the domain of enterprise architecture does include software development, it is not solely limited to that. So, I went to the agile manifesto and revisited its four values. They are:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiations
• Responding to change over following a plan

So, what does this mean in the context of EA practices? Let’s look at them one at a time. Keep in mind that the original authors of this manifesto state these as preferences, not absolutes. All components have value, the authors simply believe that the items on the left have more value.

Individuals and interactions over processes and tools. This could easily be rephrased as “don’t become an ivory tower.” This is certainly something that all EA teams should keep in mind. If the only meetings you have are with other enterprise architects, there’s a problem. I’m a firm believer that information is power. Too often, we prevent the sharing of information because of fiefdoms that have been set up in the enterprise. Some are formal, some are not. It’s frequently a matter of trust. If you trust that the people you share information with will use it wisely, it gets shared. If you don’t, you don’t share it. Sometimes the lack of trust is justified, sometimes it is not. As far as the manifesto goes, however, the key word is interactions. Interestingly, in James’ post, he states, “Why not optimize the organization so as to reduce the amount of folks one needs to interact with in order to get the job done?” While you may think he’s dismissing interaction, which would be contrary to the manifesto values, I think the message is that you also need to understand your role in the organization. Interaction is important, to a point. As an individual, you need to make good decisions on when to interact and who to interact with. If you don’t, you lose trust and credibility. It’s like the Emotional Bank Account® from Seven Habits of Highly Effective People. As far as how this impacts EA, it really means that you need to get out and work with the people using your reference architectures and your strategic plans.

Working software over comprehensive documentation. Since most of what EA does is create documentation, the thing to look at here is whether the right documentation is being created. Again, the authors of the manifesto don’t say that documentation has no value. Documentation does have value, and EAs need to ensure that their own documents are targeted with purpose, as are the documents they receive. The EA needs to concerned about the items that will live on and be of importance to the enterprise.. The project architect needs to be concerned about the items that make the project successful. The two may not necessarily overlap. This ties back into my comments about roles and responsibilities. Without a clear understanding of roles and responsibilities, how will the project know when to interact with the EA and vice versa. Forcing a formal review process to mandate that some interaction occur is not a solution when the roles, responsibilities, and goals are not well understood. Formal reviews can be powerful, but only where they complement the interactions taking place, rather than replacing them.

Customer collaboration over contract negotiation. No arguments here. Unfortunately, we’re still in the state where the IT / business relationship is seen as customer / supplier, rather than working as partners. It ripples all the way back up to the information sharing, and ultimately, that can have a very negative effect on the solutions. If, as an EA, I’m not aware that the company’s strategy is to grow by acquisition, my focus may be in the wrong area, resulting in the integration costs being much higher than they would have been had I been in the loop.

Responding to change over following a plan. I had someone ask me once, “How do I make sure this service is an enterprise service?” I told that person, “You don’t. Instead, assume that your service will change and understand how you’re going to manage it.” The real problem is that we don’t plan for change to begin with. We think things can be etched in stone once and we’re done. Technology solutions in the enterprise, software development projects in particular, simply don’t work that way. The organization needs to figure out how to include change in their planning processes and accommodate it accordingly. This doesn’t mean throw out your project plans, but it does mean that a single annual review and funding cycle won’t work.

My final comments: Adopting agile methods is kind of like losing weight. Going to an extreme doesn’t work. You need to have moderation, finding the right amounts of each food and each type of exercise for you to be successful. The authors of the manifesto clearly state that they are articulating preferences, not absolutes. You may read James’ post and think that the we need less communication. I don’t believe that. I also don’t believe that more communication is the answer, either. I believe that we need more of the right kind of communication, and less of the wrong kind of communication. If you’re an individual in an enterprise, always ask the questions: Who should I be communicating with and should be communicating with me? Don’t wait for them to come to you and ask for something, and don’t wait for them to come and give you something you think you need. Get out of your chair and do it.

I’m seeing more and more articles about complex event processing (CEP) these days. If you’ve followed by blog, you’ll know that I’m a big fan of events, so I try to read these when they come across my news reader. One of the challenges I see, however, is that event-driven thinking is not necessarily the norm for businesses. Yes, insurance companies may deal with disaster events, and financial services companies may deal with “life” events like weddings, births, kids going to college, but largely, the view is very service-based. It is reactionary in nature. You ask me for something, and I give it to you.

This poses a challenge for event processing to gain mindshare. While event processing is certainly the norm in user interface processing and embedded systems, it’s not in your typical business IT. Ask yourself- if you were to install a CEP system in your enterprise today, what events would it see?

The starting point that I see for events should merely be publication. Forget about doing anything but collecting statistics at the beginning. Since events don’t align with how we’re normally thinking, perhaps we should let them show us how we should be thinking. This gets into the domain of business intelligence. The beauty of events, however, is that they can make the intent explicit, rather than implicit. If I’m only performing analysis based on database changes, am I seeing the right thing, or am I only seeing symptoms of the event? Not all events may result in a database change, and that’s where the important correlations may lie. If some companies shows up on page one of the Wall Street Journal, it could result in increased trading activity for that company. My databases may record the increased trading, I may not have a record of the triggering event- the news story.

Humans are very good an inferring relationships between events, sometimes better than we think. But without any events, how can we infer any relationships? We don’t want to overwhelm the network with XML messages that no one ever looks at, but we shouldn’t be at the opposite extreme either. Starting with new applications, I’d make sure that those systems are publishing some key events based upon their functionality. Now, we can start doing some analysis and looking for correlations. We then start to encourage event-driven thinking about the business, and as a result, have now created the potential for CEP systems to be used appropriately.

As an example of how far we still have to go, let’s look at Amazon. They certainly leverage business intelligence extremely well. Unfortunately, as far as I can tell, it’s largely based upon tracking the event that impacts their bottom line directly- purchasing. If I were them, I’d be looking at wish list activity much more strongly. Interestingly, my wife gets more recommendations on technical books that I do. Why? Because she’s purchasing them as gifts for me. I put them on my wish list, she buys them. Because they’re looking at the wrong event, they now make an inference that she’s interested in them when she isn’t. I am. They need to track the event of me adding it to my wish list, along with someone purchasing it for me, and then turn around and make recommendations back to me. Of course, that’s part of the challenge, though. There is simply a ton of information that you could collect, and if you collect the wrong stuff, you can waste a lot of time. Start with a small set of information that you know is important to your business and build out from there.

I just finished giving a webinar on the importance of SOA pilots with Alex Rosen, and I hope the attendees found it informative. One of the things that I discussed in the webinar was the scope of SOA adoption. Given the recent attention to my last post, I thought I’d discuss it a bit more, since it’s one of the two dimensions of the maturity matrix. It’s also what makes the effort more than just a “search and replace” on the SEI CMMI models as one commenter over on InfoWorld thought.

The last post introduced the levels of maturity, which are:

0. Ad Hoc
1. Common Goals
2. Pilot
3. Extend
4. Standardize
5. Optimize

Those levels are a pretty straightforward way of describing the maturation process of just about anything. So what’s really import is the other dimension which defines exactly what we’re maturing.

In the case of this model, we’re discussing SOA adoption maturity. SOA adoption is not simply about purchasing technology. No one can sell you an SOA, although there was someone selling “SOA in a box” back around Christmas on eBay in Australia. SOA adoption does involve new technologies that can provide support in service development and hosting, such as orchestration engines or web service frameworks, service connectivity, such as SOA appliances or ESBs, and service management. SOA adoption also involves organizational changes. If your organization is structured around application development, which team is responsible for building a service that spans multiple groups? SOA adoption involves governance, whether it be funding models, design time policies, or run time policies. SOA adoption involves new processes designed around the consumer/provider interaction. SOA adoption involves training and communication. How do we market services that have been created to ensure their reuse? Clearly, SOA adoption involves architecture. Enterprise architecture must provide appropriate reference architectures and reviews to ensure both tactical and strategic success. SOA adoption involves Operational Management. Services can’t be dumped into production and forgotten, we must take a proactive approach to monitoring and metric collection and feed that information back into the machine for continuous improvement.

SOA is not easy. If it were, we’d all be done by now. Every company will have different drivers, and different technology needs. An assessment of their maturity in SOA adoption should examine all of dimensions required.

David Linthicum recently re-posted his view on SOA maturity levels and I wanted to offer up an alternative view, as I’ve recently been digging into this subject for MomentumSI.

Interestingly, Dave calls out a common pattern that other models he’s seen define their levels around components and not degrees of maturity. He states:

While components are important, a maturity model is much more important, considering that products will change over time…

I completely agree on this. Maturity is not about what technologies you use, it’s about using them in the right way. Comparing this to our own maturity, just because you’re old enough to drive a car, doesn’t mean you’re mature. Just because you’ve purchased an ESB, built a web service, or deployed a registry doesn’t mean you’re mature.

Dave then presents his levels. I’ve cut and paste the first sentence that describes each level here.

• Level 0 SOAs are SOAs that simply send SOAP messages from system to system.
• Level 1 SOAs are SOAs that also leverage everything in Level 0 but add the notion of a messaging/queuing system.
• Level 2 SOAs are SOAs that leverage everything in Level 1, and add the element of transformation and routing.
• Level 3 SOAs are SOAs that leverage everything in Level 2, adding a common directory service.
• Level 4 SOAs are SOAs that leverage everything in Level 3, adding the notion of brokering and managing true services.
• Level 5 SOAs are SOAs that leverage everything in Level 4, adding the notion of orchestration.

While these levels may be an accurate portrayal of how many organizations leverage technology over time, I don’t see how they are an indicator of maturity, because there’s nothing that deals with the ability of the organization to leverage these things properly. Furthermore, not all organizations may proceed through these levels in the order presented by Dave. The easiest one to call out is level 5: orchestration. Many organizations that are trying to automate processes are leveraging orchestration engines. They may not have a common directory yet, they may have no need for content based routing, and they may not have a service management platform. You could certainly argue that they should have these things in place before leveraging orchestration, but the fact is, there are many paths that lead to technology adoption, and you can’t point to any particular path and say that is the only “right” way.

The first difference between my efforts on the MomentumSI model and Dave’s levels is that my view is targeted around SOA adoption. Dave’s model is a SOA Maturity Model, and there is a difference between that and a SOA Adoption Maturity Model. That being said, I think SOA adoption is the right area to be assessing maturity. To get some ideas, I first looked to other areas, such as CMMI and COBIT. If we look at just the names of the CMMI and COBIT levels, we have the following:

So how does this apply to SOA adoption? Quite well, actually. COBIT defines a level 0, and labels it as “non-existent.” When applied to SOA adoption, what we’re saying is that there is no enterprise commitment to SOA. There may be projects out there building services, but the entire effort is ad hoc. At level 1, both CMMI and COBIT label it as “Initial.” Again, applied to SOA adoption this means that the organization is in the planning stage. They are learning what SOA is and establishing goals for the enterprise. Simply put, they need to document an answer to the question “Why SOA?” At level 2, CMMI uses “Managed” and COBIT uses “Repeatable.” At this level, I’m going to side with CMMI. Once goals have been established, you need to start the journey. The focus here is on your pilot efforts. Pilots have tight controls to ensure their success. Level 3 is labeled as “Defined” by both CMMI and COBIT. When viewed from an SOA adoption effort, it means that the processes associated with SOA, whether it be the interactions required, or choosing which technologies to use where, have been documented and the effort is now underway to extend this to a broader audience. Level 4 is labeled as “Quantitatively Managed” by CMMI and “Managed” by COBIT. If you dig into the description on both of these, what you’ll find is that Level 4 is where the desired behavior is innate. You don’t need to handhold everyone to get things to come out the way you like. Standards and processes have been put in place, and people adhere to them. Level 5, as labeled by CMMI and COBIT is all about optimization. The truly mature organizations don’t set the processes, put them in place, and then go on to something else. They recognize that things change over time, and are constantly monitoring, managing, and improving. So, in summary, the maturity levels I see for SOA Adoption are:

0. Ad hoc: People are doing whatever they want, no enterprise commitment.
1. Common goals: Commitment has been established, goals have been set.
2. Pilot: Initial efforts are underway with tight controls to ensure success.
3. Extend: Broaden the efforts to the enterprise. As the effort expands beyond the tightly controlled pilots, methodology and governance become even more critical.
4. Standardize: Processes are innate, the organization can now be considered a service-oriented enterprise.
5. Optimize: Continued improvement of all aspects of SOA.

You’ll note that there’s no mention of technologies anyway in there. That’s because technology is just one aspect of it. Other aspects include your organization, governance, operational management, communications, training, and enterprise architecture. SOA adoption is a multi-dimensional effort, and it’s important to recognize that from the beginning. I find that the maturity model is a great way of assessing where an organization is, as well as providing a framework for measuring continued growth. That being said, your ability to assess it is only as good as your model.

I recently had the opportunity to discuss the positioning of SOA appliances and it caused me to revisit my convergence model, as shown here:

This diagram was intended to show the challenges that enterprise face in choosing vendors today, as there are solutions from multiple product spaces. The capabilities frequently associated with activities “in the middle” can come from network appliances, application servers, ESBs, service management systems, etc. My original post talked about the challenges that organizations face in trying to pick a solution. A key factor that must be weighed is the roles in the organization. My view is that the activities in the middle should beconfigured, not coded. That’s a topic for another post, however.

I started thinking about the future state, and realized that while the offerings from vendors overlap today, that shouldn’t be the long term trend. Even vendors that can cover the entire space do it with multiple products. The right model that we should be shooting for is one that looks like this:

In this model, we have four distinct components. Service hosting is concerned with the development and execution of services. Service connectivity consists of the capabilities in the middle: routing, mediation, etc. Service management provides management facilities over both of these domains. All of these systems rely on a set of information resources which provide both the information to process, as well as the policy and meta-information required for the appropriate execution of the systems. A registry/repository, therefore, would fall into the information resource domain.

What we’d like is for all of these domains to be integrated in an open, standards-based manner. Unfortunately, we’re still a ways off from that day. There have been some proprietary efforts to create integrated solutions that look like this, such as the Governance Interoperability Framework effort by Systinet, but there’s still a long way to go. None of the vendors associated with GIF are the big players in the service hosting space (IBM, BEA, Oracle, Microsoft, etc.), and the integration standards are not open. When we have open, integration standards, we can now begin to create the feedback loop.

One short-term issue that needs to improve is the tight coupling of management consoles to the platforms. In the model, service management is loosely coupled. It integrates with the other domains through loosely coupled services, all of the best practices of SOA. Today, service hosting platforms and service connectivity systems all come with their own management consoles. In order to enable this model, the management architecture of those systems must be built on SOA principles. That means that all of the capabilities that can be managed should be exposed as services. You want to deploy a new application to the application server? Call the application deployment service. This creates a great situation for automated build systems. Out of the box, the build process could be executed in your favorite BPEL engine, with controls for compilation, automated testing, source code tagging, and deployment all orchestrated through web service interactions. Now add in a feedback loop by which metrics cause additional provisioning, or even where an uncaught exception results in a tag being placed on the source files associated with the stack trace to aid in debugging. It all begins with having the services available. Ask your vendors one simple question: are all the capabilities available through the management console also available as services?

Vilas posted a response to some of the postings (here and here) I made regarding the relationship of city planning to EA/SOA. He provides an example of a business sponsor that promotes a program that can add million dollars to the bottom line, but has an extremely short timeline, one that requires the existing architectural guidelines, principles, and processes to be short-circuited, or more likely, completely ignored. He compares this effort to a slum getting developed in a nice city.

I’m not going to argue that this situation doesn’t happen. It does. What I will argue, however, is that the fact that it allowed to be built can be a case of ineffective governance. The governance policies and processes have to be about encouraging the desired behavior. If the policies and processes aren’t consistent with the desired behavior, it’s a case of bad governance. In this example, this is likely a rapid growth opportunity. If the enterprise as a whole is in a cost cutting mode, I have a hard time believing that this rapid growth scenario would pass the governance checks and be viewed as a “solid business case.” If the corporate leaders have decided that the best direction for the company is to cut costs, odds are that a project such as this will never make it out of the governance process to begin with. If the company is focused on increasing revenue and growth, odds are it has taken more of a federated governance model, and allows individual business units to make decisions that are in their own best interest, sometime introducing redundant technology in order to meet the schedule demands of the growth cycles. If the enterprise architects in this model are instituting technical governance that constrains that growth, again, they’re acting in a way that is inconsistent with the goals of the organization, a case of bad governance. In either case, that mismatch will eventually cause problems for the organization. In the case of city, it may bring in crime, lower property values, and cause prosperous businesses and their revenues to move elsewhere. In the business world, it could cause a lack of focus on core capabilities, cost overruns, and fragmentation within. None of these risks were probably included in the business plan.

This is the dilemma of the enterprise architect or really anyone with some authority in the governance process. Growth is usually something that is important and achievable in the short term, but difficult to sustain in the long term. Growth has to occur in other areas, while cost cutting measures must be introduced in the former areas. Cost cutting leading to the elimination of redundancy, and if the technology wasn’t planned for that eventual occurrence from the beginning, the effort to reduce costs may eat away any potential savings. This is where the service abstraction is extremely important. Correctly placed services can position a company to consolidate where appropriate down the road. It will pay benefits when a merger and acquisition must occur by providing an analysis point (the service portfolio) from both a business and technology perspective to better estimate the cost of the integration activities.

Alex Rosen and I will be giving a webinar next Friday on the role of pilots in achieving SOA success. I haven’t blogged on SOA Pilots in quite some time (March 23rd of last year, to be exact). It’s always interesting to go back and read some of my past posts to see how my thinking has evolved. I had quoted the ZapThink guys, as well as Miko Matsumura in that entry, stating:

Miko stated that the only ones getting it right were ZapThink, who state that “the things you do in a pilot are the exact opposite of what you need to do to get to enterprise scale.” For the record, I agree. This all comes down to defining the pilot properly. In their book, “Service Orient or Be Doomed!” Jason and Ron call out three SOA Pilot essentials: an architectural plan (the pilot will cover some portion of it), a specific scope, and clear acceptance criteria.

There shouldn’t be much controversy over these, but yet, the case studies and whitepapers that I see presented don’t have these elements, and it’s usually because the study is equating web services usage with SOA. Taking a user-facing customer portal and extending it by allowing customers to integrate their systems directly can be a good thing, but is it really an SOA pilot?

I went on in the entry to lock in on the subject of culture change, stating: “a proper SOA pilot is to pick a problem that will require the organization to see the cultural changes that are necessary to become a service provider.” I still think that this is the case, however, I would also say that I was being just as narrow as the teams that strictly focus on using Web Services for the first time.

What you’ll find in the webinar is that SOA adoption involves many dimensions. One of those dimensions is technology based. Another dimension is cultural. I’ve been working with my colleagues on a maturity model that outlines these dimensions and the stages that an organization goes through across all of them. Pilot efforts should cover all of these. It may be done in one large program, or there may be several pilot projects. Every organization is different, therefore, there is not a one size fits all project that every organization should embrace.

If this sounds interesting to you, then I encourage you to sign up here, and listen in on Friday the 16th, at 1pm Eastern Time (Noon central, 10AM Pacific).

I want to turn on Donald Trump next Sunday night and see him task the teams with the successful creation and marketing of SOA within an enterprise. Okay, so it can’t be done within the day or two that they normally have, and outside of some Dilbert-esque quotes, it probably wouldn’t make for good TV. What it would do, however, is allow IT to see what their culture needs to be like in the future.

There’s a discussion just getting started in the Yahoo SOA group that raises some questions about the importance of marketing in SOA. A frequent complaint in the boardroom on “The Apprentice” is that the marketing strategy didn’t cut it and as a result, the person responsible for marketing on that task is fired. IT isn’t made up of bunch of people with MBA’s from Harvard, Wharton, or even Trump University. I have two degrees from the College of Engineering at the University of Illinois. During my stay there, I was not required to take any marketing courses, although the Computer Science department did require students in their undergraduate engineering program to take 4 courses outside of the department (independent of other electives) to which CS could be applied. The most popular area was business, with my choice, psychology, being second. The typical techie does not have formal training in marketing or other aspects of running a business, so it’s no surprise that we have a hard time with it.

We need to bring some business savvy into the IT department. I’m not talking about an understanding of the business being supported by IT (although that’s important too), I’m talking an understanding of how to succeed in business. Marketing, sales, product development, research, etc. A service provider needs to think of themselves as a vendor. They need to have a customer centric focus, with an understanding of the market trends (i.e. the business goals), customer needs, product lifecycles, resource availability, etc. to be successful. IT cannot simply be order takers in the process, because technology usage within an enterprise is not a commodity. The business side can’t simply go to IT Depot at the nearest shopping zone and pick up what they need. There are elements of technology that can be, and this will continue to fuel SaaS and other managed services, but here and now, the need for the IT department still exists. It’s time to change the IT culture, and get the development teams thinking about Service Management and a more business-like approach to their efforts.

Update: While the whole idea of bringing MBAs in was somewhat in jest, this is exactly what IBM did. Joe McKendrick’s eBizQ SOA in Action blog brought it to my attention, here’s a link to the original eBizQ story.

Recently, there’s been a few posts that have discussed the role of the individual in an SOA effort. David Margulius of InfoWorld posted an article about a doctor that took it into his own hands to create an electronic records system for outpatient services. Joe McKendrick of ZDNet and eBizQ followed on with some commentary. Prior to this, there was an article by a developer that was particularly critical of Enterprise Architecture.

Both of these articles raise some interesting points, and for the discussion, we need to go back to the city planning analogy that I recently discussed. If we were to equate these two technology related activities back to the city planning analogy, what would we have? In the case of the doctor that built his own electronic outpatient records system, let’s look at a recycling program. It’s entirely possibly that a homeowner’s organization could use some of the dues collected and implement a recycling program for an individual subdivision. It’s even possible for an individual homeowner to take their recycling to some provider elsewhere. Does it work? Absolutely. Is it cost-effective at the city level? Well, maybe not. Every individual getting in their car and driving to the nearest recycling center will be more expensive than a few trucks coming from the center for curbside pickup. The homeowners organization may be slightly more cost effective, but if each subdivision picks their own provider, you could have problems with far more recycling trucks on the street than are really necessary. The real concern, however, comes back to the problems in the first place. Why didn’t the city council put a recycling program in place? The fact that the city failed to act on this in a timely manner for its citizens is what causes a homeowner or homeowner’s organization to take matters into their own hands.

Likewise, let’s take the case of the frustrated developer. Here, the author called out:

Projects are driven by business deadlines, and not by the time it will take to get things right. So the foundation starts out weak…

This sounds like a case of being doomed from the start. In the city planning analogy, there are many scenarios, ranging from fixing a pothole in a street, to building highway interchanges. How many stories have been there about some worker that follows the letter to the T? They’re told to fix a pothole, so they fix that pothole, and only that pothole, regardless of whether or not the street looks like the surface of a golf ball. Some major development project is put under way, and winds up having huge cost overruns because an over-aggressive schedule.

While some may think that the core of the problem is too much governance, it’s not. The problem is ineffective governance. There needs to be an understanding from top to bottom of the roles and responsibilities associated with decision making and all need to be held accountable. A developer that isn’t following the guidelines isn’t a good thing, nor is an architect that establishes the wrong guidelines. An enterprise architect that obsesses about particular lines of code in a project is focused on the wrong thing. The city council shouldn’t be mandating what color a homeowner paints their bathroom. At the same time, there will always be gaps, and there will always be people with time on their hands to address them, like the doctor in the InfoWorld article. A key to being successful is the creation of the appropriate framework so those people can fill those gaps, but in a way that leads not only to short term success, but long term success, as well. I’m just as guilty as anyone else of jury-rigging a solution for a problem that I had, and then telling a few people about it, who in turn told their friends about it, etc. and the time I spent on it starting going up and up. The problem was a lack of understanding of either the need or the value that would achieved from the solution.

SOA is just another example of this. I recently had a discussion with Phil Windley about an upcoming article. In our discussion, one of the things I mentioned was that I felt that the technology governance (i.e. EA) needs to get in sync with the traditional IT governance (i.e., project scoping and funding). Project establish boundaries, and if those boundaries make it difficult to build services that have broader benefit, the odds are already stacked against you. We need to find a way to have those services be built the right way, and that may be more about changing the way that IT operates than it is about using a particular technology.

In short, the groups that comprise IT Governance in my opinion, EA and Management, need to establish the right guidelines and the right funding models for SOA, and really IT as a whole, to be successful. For some larger efforts, it will involve significant planning and effort from those areas. The smaller efforts will still occur, however, and they can’t be ignored either. An environment that encourages the individual to contribute to the overall success, and helps them to contribute in a constructive, rather than destructive way, will be the most successful. Sometimes it is necessary to deny a building permit for the greater good. The best situations, however, are where it can be turned into a win-win situation for both.

Dan Foody of Progress Software has a great post entitled “The SOA Glass Ceiling” that brings up a familiar topic- the service lifecycle. He states:

If you only take away one thing from this blog its that in SOA you don’t build services, you deliver services.

What was particularly interesting to me was the discussion around product delivery. I’ve always used the analogy of a product manager when discussing the service lifecycle, and it would seem that Dan’s article shoots a big hole in this analogy. In reading into it, however, it doesn’t. Dan describes product delivery as this:

…you roll out a service, and at a later date (6, 9, 12 months later) you roll out the next version, and the software development lifecycle continues.

Inside of most enterprises, I’d actually argue that this is an improvement. Because what’s really happening is that all of the efforts are disjoint. Most likely, there may not be a next version, and when there is, it’s a completely new set of people that lack the context of the original team. The management was project management, not product management. Dan is in agreement, as he states:

IT organizations are used to building applications for the business, shipping them, and going on to the next (typically) unrelated thing to build.

I like the notion of Service Delivery. Obviously, building and deploying the service is a part of it. But to do it properly, the entire Service Lifecycle must be managed, and it’s not strictly about project management of the development efforts for each version. It’s also about observing how customers are using the service, marketing the service out to the enterprise, providing excellent support when expectations are not met, establishing contracts to ensure expectations are clear from the beginning, and much more.

Brenda Michelson of Elemental Links just posted a great excerpt of an unpublished paper from Annie Shum of BEA that discusses the city planner analogy to SOA.

Interestingly, my recent post on facilitating the service lifecycle spawned an email conversation on this topic. The gist of the conversation was that many of the tools out there in the SOA space are targeting the Building Inspector, rather than the City Planner or the Developer/Landlord (developer in the city planning sense, not in the software development sense).

Annie’s paper points out that:

City planning is not concerned with the design and construction of individual buildings. Rather, it is concerned with the multi-aspect relationship of individual buildings to one another… city planning is not about building architecture; it’s about meta-architecture for designing communities of buildings by focusing on common infrastructure, governance and cohesion.

This is a great statement, and very useful when trying to explain the role of an enterprise architect. James McGovern recently brought up the need for architects to work downward to the developers. I think the city planning analogy can be a useful tool in helping to create an understanding of the roles involved. Interestingly, recent events in my town demonstrate how well this analogy works, even in providing some guidance to how collaboration should occur. The town I live in is growing. Years ago, it would have been considered a rural farming community. Now, with the normal suburban sprawl, its close proximity to downtown St. Louis is making it more suburban and less rural. The biggest event associated with this growth is about to happen: a major tract of farmland is going to be turned into a major shopping and industrial complex. The opinions page in our local paper has become very active on the subject. The individuals expressing their opinions are akin to the individual developer in the enterprise. The city council is akin to the enterprise architecture team. I would not want to leave this effort in the hands of the individuals who are expressing their opinions. They simply don’t have the right perspective to make these types of decisions. The city council has been working on this for at least 5 years, studying the implications of tax incentives, infrastructure development, impact on local businesses, etc. The roles are clear. What I do believe in, however, is the ability of the individual to participate and contribute to the process, and the responsibility of the city council to solicit input and communicate their direction. After all, this entire process may have begun with an individual land owner selling their acreage to the developer to begin with.

The same holds true for the corporate enterprise architect. While enterprises are not democracies, so your EA can’t be recalled or replaced at the next review cycle by a vote of the developers, the best practices of a democratic style of government should be followed. The Enterprise Architect must be producing documents for upward communication (business and IT executives), lateral communication (other architects), and downward communication (developers, operators, engineers). Likewise, they should be soliciting input from all three directions, as well. Communication should never be one-way, nor should it be one size fits all. Some of the best ideas may come from people in the trenches, and some of those people may be a future architect. I was one of those people in an organization some 7 years ago. Annie’s paper points the importance of communication and collaboration with this statement:

…the entrprise architects’ role extends well beyond software coding specifications. Instead they must also act as the chief enablers of borderless collaboration by coordinating and prioritizing the input from disparate groups with different needs, interests, and views, including business stakeholders, software architects, developers, and DBAs, as well as external partners, suppliers, providers, customers, auditors, and so on.

Brenda pointed out that this document hasn’t been published yet. Based upon this excerpt, I hope it will be published. Add me to the list of interested parties.

I’m listening to the latest Technometria podcast from IT Conversations where Phil Windley and his gang have a discussion with Ben Galbraith. At 53:45 into the podcast, Phil asks Ben about his home automation efforts that he’s incorporating into his house under construction. While there’s no mention at all of SOA, it was an interesting discussion nonetheless. He’s planning on leveraging RFID, webcams, etc. It sounds really cool, and I’d be interested in hearing an after-the-fact case study on the integration problems he ran into. Ben’s a great guy, I had the opportunity to meet him at a Microsoft Technology Summit in 2005 (he probably doesn’t remember me, but who cares), and knows a tremendous amount about the presentation side of development technologies.

Do you have ERS? Jeff Schneider had a great post outlining the symptoms of Empty Registry Syndrome. One of the things that struck home with me was his last suggestion on how to cure it.

Verify that your registry doesn’t stink. Here’s the test: If you search for a service that doesn’t exist does it return with:
A) No results found
or
B) No results found, would you like to request a new service?
If the answer is “A” please call your vendor and let them know that their software is spreading a disease known as ERS.

Prior to becoming a consultant, I was at an enterprise where we were just beginning to explore the registry/repository space. We had one vendor come by and give a demo, and I asked them, “Is there any way to put services into the registry/repository that are in a planned state?” “Umm…. that’s a great idea!” A couple months prior to this, a project team contacted me about a service they would need. No one had implemented it yet, but we sent out a notification to the development teams about this request, and in came flooding all sorts of requirements, with no more effort than an email. It’s amazing what a little bit of communication can do, and think of how that effort could have been facilitated by some tools. The enterprise was embracing workflow technologies. I’m starting to see some potential here.

1. The project architect is building out the candidate architecture, and as that happens, the service library is searched.
2. If there’s a match, the architect can immediately register interest in the service. There may be tasks spun out of this to either the project architect or the responsible service manager.
3. If there isn’t a match, the architect has two option. Request that a new service be created, or create a planned new service.
4. If they request a new service, this results in tasks for a governance group to determine who the right owner should be, and schedule out a project. The project architect of that new task would update the service to a “planned” state from a “requested” state.
5. For services that enter the “planned” state, notifications would go out to the appropriate development teams with pertinent information, allowing them to contribute to the service requirements.

This scenario that I’m painting is an area that I’m going to call “Service Lifecycle Management.” Governance is a part of it, as shown, but there’s much more to it than that. It’s also not SDLC management, either, although that too is part of it. The service lifecycle doesn’t end when the service has been developed. The service lifecycle only ends when the service is decommissioned.

I think this area is one that is ripe for opportunity. I commented about this last year, but this topic still hasn’t received much attention from the vendor community. Mindreef was heading down this path back in Dec. 2005 with their Coral product, but there’s no mention of it on their website anymore (perhaps it was incorporated into SOAPscope Server). The challenge with lifecycle support for vendors is that it has to be about integration. Any product that tries to own the entire lifecycle will probably struggle. I see this space as being a combination of workflow/bpm, registry/repository, service management, and a service lifecycle management tool, at a minimum.

Look for a more in-depth blog on the Service Lifecycle soon. In the meantime, if you’re interested in an introduction to my thoughts on Service Lifecycle and being a Service Manager, a good starting place is this podcast I did with ZapThink last year.

James McGovern posted a question for me to answer:

“…crisply define the distinction between consulting firms who offer services that are consumed by enterprise architecture teams vs. the actual act of practicing enterprise architecture?”

In thinking about this, my first thoughts were pretty simplistic. Unless the consultant is in a long-term staff augmentation role, the consultant assists, the corporate EA does. That’s painfully obvious, however, and I don’t think it’s what James was really asking about. As I thought about this, it really came back to collaboration. In order for a consulting firm to be successful in the long term, it has to leverage the breadth of exposure it has. Within an enterprise, the opportunities for cultural breadth (corporate culture) are minimal. It may happen when a change in management is made, or when a merger or acquisition occurs, but in general, you settle into one way of doing things. A consulting firm, on the other hand, gets exposure to many corporate cultures, and has to find a way to make an organization successful within that culture, potentially being an agent for change. An 8 week or 12 week assignment is typically not long enough to introduce significant cultural change, however.

Given that the consulting firm has the breadth of exposure, it is critical that the consulting firm collaborate on engagements so that the best solutions can be offered. Even if I’m the only person assigned to a particular engagement, I will bounce ideas off of my peers to get their thoughts. By seeing what different organizations do, and understanding which approaches work best with a particular corporate culture, the engagements are much more likely to be successful.

Within an enterprise, I don’t think the notion of collaboration is as important as it should be. All too often, it’s about turf battles and clear lines of responsibility. When I ask my teammates a question, I don’t have any fear that they’re going to get recognition by my boss before I do. I ask them because I want the best solution for my client. In the enterprise world, internal advancement and recognition is on the mind of many individuals, and may come at the expense of creating a collaborative, team approach.

All of this is not specific to enterprise architecture, but I think it’s even more important at this level. In my experience as part of an EA team prior to becoming an enterprise, it became very clear that architecture couldn’t be done in a vacuum. Yet, the corporate culture makes it very difficult to foster the type of collaboration that was needed to make it successful. Too many meetings, too many fire drills, and too many time-dependent activities (i.e. top priority is some date in a project plan, not necessarily getting the best answer). This isn’t a knock on my former employer, it’s probably true of most enterprises. Enterprise architecture, by its very nature, requires collaboration. The EA focused on Security has to be working closely with the EA focused on Information and the EA focused on networking technologies, etc. Ironically, what often causes that collaboration to occur quickly? Bringing in a consultant. The consultant needs to quickly understand where an organization is, and it meets bringing all of the right people together and having the right discussions.