Todd Biske: Outside the Box

Archive for the ‘IT’ Category

Nortel and IBM recently announced joint technology for integrating business applicaitons with communication services (InfoWorld article, SearchSOA article). Personally, I’m glad to see this announcement. I first had conversations around communications services with a colleague back in early 2006. He was looking at the future state communications infrastructure and came to me wanting to know how to make sure it would fit in with SOA. I had never thought about this before, but it made perfect sense. Communications, clearly, is a capability, so why shouldn’t those technical capabilities be exposed as services? Kudos to Nortel for having a press release about this and really emphasizing how this can play in a company’s SOA is a win in my book. While I’m sure other vendors in the communications space also have these capabilities, they’re not emphasized. As a result, it creates an atmosphere for more silo-based thinking around point-to-point integrations, rather on how this capability fits into the broader collection of enterprise capabilities.

The field of communications would also probably make a great case study or research project. If someone were to try to define communications services 10 years ago versus today, you’d have a very different collection. Would a presence service even be mentioned? Would it have been voice only, or would it have involved text/instant messaging, email, and/or video as well? It certainly makes the case for active service lifecycle management versus defining the services once and then moving on to the next project. As you define your service domains, you have to recognize that the definitions of those domains and even the collection of domains themselves will change.

I recently had a conversation with Ron Schmelzer of ZapThink and we started talking about how the nature of the entry point for enterprise users to interact with the information technology will change in the future. You’ll notice that I didn’t use the term “application” in that sentence and there’s a reason for that. Personally, I want to get rid of it. To me, it implies a monolith. It’s a collection of functionality that by its very nature goes against the notion of agility. When I look at a future state where we’re leveraging BPM, SOA, and Workflow technology, I see very small, lightweight entry points that are short and to the point. I’ve mentioned this before in connection with Vista Gadgets or MacOS X Dashboard Widgets.

Ron brought up a ZapFlash that came out over a year ago that he wrote called “SOA: Enabling the Long Tail of IT.” I didn’t make the connection at the time, but it makes perfect sense now. In the ZapFlash, Ron describes the “Long Tail” this way:

The Long Tail, a term first coined and popularized by Chris Anderson, refers to the economic phenomenon where products that are of interest to only small communities, and thus result in low demand and low sales volume, can collectively result in a large aggregate market. This large collection of small markets can significantly exceed the more traditional market that the most popular and high volume sales items can generate. For example, Amazon.com generates more business in aggregate from its millions of books that each only sell a few copies than they do from the top 100 best sellers that might each sell tens of thousands of units.
One quick way of summing up the Long Tail is by saying that thereís more opportunity in catering to a mass of niche markets than a niche of mass markets. Large enterprises in particular are composed of masses of such niches, operating in different geographies and business units, catering to specific demographics with tailored solutions to meet the needs of all constituents. And yet, the centralized IT organization that serves the needs of the entire organization is typically woefully unprepared to serve these masses of niches: large numbers of users with widely varying IT needs. How, then, can IT support the needs shared in common with all the business groups without overextending its centralized resource to meet the specific needs of each of the individual groups?

Fundamentally, we’re both talking about the same thing. What I describe as very lightweight user-facing entry points are the “long tail” of applications. They’re small, niche solutions that get the job done. Underlying all of this is a robust SOA that are the enablers of these solutions which is loosely-coupled from the user-facing needs. If you think about it, the long tail of application development today is the business user using Excel because they could get done what they needed quickly. I’ve even done this myself, and even progressed up to getting a simple database setup to do a bit more. We shouldn’t be on a quest to squash these out, but rather to figure out how to enable it in a manageable way. The problem is not that somebody’s Excel macro pulling data out of Oracle exists, the problem is that we’re not aware that it exists. Clearly, someone had a need to put it together, and if we can find a way to enable this to where we’re aware of it and our systems support it easily, even better. Personally, I think the technologies we have at our disposal today are on track for making this a reality.

In the latest BriefingsDirect SOA Insights Edition, Dana Gardner, Jim Kobielus, Neil Macehiter, and Joe McKendrick discussed, among other things, Microsoft’s recent announcements. The conversation started very similar to some of my own comments on the subject with this sense of deja vu. Neil Macehiter made a great point, however, that shows that this isn’t simply a rehash of model-driven architecture. He stated:

…they are actually encompassing management into this modeling framework, and they’re planning to support some standards around things like the service modeling language (SML), which will allow the transition from development through to operations. So, this is actually about the model driven life cycle.

This reminded me of my trip to Redmond in 2005 for the Microsoft Technology Summit. At the summit, we were shown an internal tool, I think from the Patterns & Practices group, that presented a deployment model of a solution. I recall a number of us going, “we want that.” If Microsoft has taken steps to integrate these models into the development and run-time management tooling, this is an excellent step, and certainly something beyond the typical model-driven development of the BPM suites. At a minimum, these capabilities should be enough for people to at least track the ongoing progress of the Oslo effort.

The second thing that came up, which again was consistent with some recent blogs of mine (see Registries, Repositories, and Bears, oh my! and Is Metadata the center of the SOA technology universe?), was the discussion around the metadata repository at the heart of Microsoft’s strategy. Dana pointed out that “there really aren’t any standards for unifying modeling or repository for various models” with some comments from Neil that this is very ambitious. First, I’d have to say that Microsoft trumped IBM on this one. Remember when IBM announced WebSphere Registry Repository and stated that they’d be coming out with their own standards for communication with it? They were slammed by many analysts. Microsoft, rather than trying to operate in the narrow space of the SOA registry/repository, are talking about the importance of metadata in general. The breadth of models and associated metadata when talking about full IT product lifecycle (development and management), is far broader than what is typically discussed in the SOA space. AS a result, there are no standards that cover this completely, so the lack of standards-based integration is a non-issue, and Neil nails it by saying Microsoft is trying to get out in front of the metadata federation problem and drive others to comply with what they do.

Give the entire podcast a listen here, or read the transcript here.

Jason Bloomberg of ZapThink, in their latest ZapFlash, put a new spin on their old concept of the SOA champion and put forth a job description for VP of SOA. While he certainly suggested a good base salary for the position, I question whether a permanent, executive position around SOA makes sense?

If you look at the job responsibilities listed, the question I ask is not whether these tasks are needed, but rather, whether a dedicated person is needed for all of them. Let’s look at a few of them:

Provide executive-level management leadership to all architecture efforts across the enterprise. The directors of Business Architecture, Enterprise Architecture, Technical Architecture, Data Architecture, and Network Architecture will all be your direct reports.
Don’t we already have this? It sounds like a Chief Architect to me.

Drive all Business Process Management (BPM) initiatives enterprisewide. Coordinate with process specialists across all lines of business, and drive architectural approaches to business process.
Again, to me, this sounds like the responsibility of the COO.

Establish and drive a SOA governance board that incorporates the existing IT governance board.
This is the only one that I simply disagree with. If we’re speaking in terms of IT governance as defined in Peter Weill’s book, I think the IT Governance Board should be factoring SOA strategy and governance into their decisions. That is, IT Governance subsumes SOA governance, not vice versa. Of course, there are also aspects of SOA governance that are implemented at a much lower level within projects to ensure consistency of service definitions, etc. Once again, however, this should be handled by the existing technology governance processes. We’re merely adding some additional criteria for them to be applying to projects.

Establish and lead the SOA Center of Excellence across the enterprise to pull together and establish core architectural best practices for the entire organization. Develop and enforce a mandate for conformance to such best practices.
This gets into the technical governance I just mentioned. I certainly agree that a SOA COE can be a good thing, but you certainly don’t need a new position just to manage in it. Why wouldn’t the Chief Architect or a delegate lead the COE as part of their day to day responsibilities?

Manage a budget that is not tied to individual line-of-business projects, but is rather earmarked for cross-departmental SOA/BPM initiatives that drive business value for the enterprise as a whole.
The question here is whether or not the current organizational structure prevents cross-departmental initiatives from being funded and managed properly. It implies that the individual LOBs will be more concerned about their own needs, and not that of the broader enterprise. If the corporate governance principles and goals have stated that the use of more shared, cross-cutting tehcnologies are needed, then it’s really a governance problem. While an organizational change can assist, you still need to ensure that LOB managers are making decisions in line with the corporate goals, rather than that of their individual LOBs.

Work closely with the VP of Project Management to insure close cooperation between architecture and project management teams, and to improve project management policies.
Once again, why can’t this be done by the existing architectural leadership?

There were additional items, but my thoughts kept coming back to “shouldn’t someone already be doing this?” If the answer to this is “no,” then you must ask yourself whether or not you’re really committed to SOA adoption. If you feel that you are, but don’t have anyone assigned to these responsibilities, does the creation of a new position make sense? For example, if the organization struggles with getting LOB managers to produce cross-departmental solutions, will throwing one more person into the mix fix the problem, or just add another point of view?

As with many of the ZapThink ZapFlashes, there’s always a bit of controvery but lots of goodness. In this case, the articulation of the responsibilites associated with managing SOA adoption are excellent. Do you need a new position to do it? As with any organizational decision, it depends. If you are committed to SOA, but can’t make it happen simply because all of the possible candidates are simply to busy to take on these new responsibilities, then it might make sense (although you’ll probably need to add staff elsewhere, too, if people really are that busy). If you’re trying to use the position to resolve competiting priorities where there isn’t universal agreement on what the right thing to do for the enterprise is, a new position may not resolve it.

InfoWorld published a collection of end-user stories on SOA this week, and the discussion on Leapfrog Enterprises caught my attention. In the article, Galen Gruman states that:

Leapfrog had many of the same goals that typify a typical SOA initiative: greater reuse of code, faster development time, and easier integration. But the company did not want to approach SOA as simply a changing of the guard for development tools and integration platforms. Instead it wanted to free its developers from conforming to a platform’s idea of best practices so they could focus on the applications’ functionality and use a wide range of development technologies as best for each job.

[Eugene] Ciurana [director of systems infrastructure for Leapfrog] did not want to force developers to all use the same transport. “The transport doesn’t matter,” he says. He chose to use the open source Mule ESB as a messaging backbone, relying on it to deal with transport interfaces. That way, “developers could focus as little as possible on the implementation of services,” he explains. Instead, their focus is on the functionality they are trying to achieve. The result is that developers tend to use HTTP as their transport mechanism, but some use REST (Representational State Transfer) and SOAP — “whatever works best or they’re most comfortable in.”

This caught my attention because it appears to be contrary to what I’ve seen at other organizations. Typically, the organization wants to constrain the platform to ease the integration problem and get away from the notion of “any-to-any” integration hubs. This may just be my misinterpretation, but it does raise an interesting question. How many constraints should be put in place? Interestingly, I’ve yet to run into an organization that has had to drive adoption of XML-based integration via enterprise architecture. The developers have slowly migrated away from whatever distributed object technology they were using and toward XML. The bigger challenge has been whether or not the XML contained the right information for broad consumption, not on whether or not XML was used. That being said, many EA teams are focused on the latter constraint (when to use XML or not). Knowing that there’s only so much that can be governed, what are the critical factors that EA teams should be making sure we get right as compared to the broad spectrum of things that we could be governing? Is it worth the pain to create policies regarding SOAP/HTTP versus XML/HTTP? The approach draws parallels to the big government/small government discussions that go on between the Democrats and the Republicans. The right answer is very dependent on the culture within IT, as I’ve stated often in this blog. Personally, I’m not a big fan of the integrate any-to-any approach. That being said, I also recognize that not everything is going to adhere to the constraints. I think it’s important to differentiate between your connectivity (mediation) infrastructure and your service enablement activities. Connectivity is about connecting two parties that adhere to the constraints for communication that have been adopted by the organization. Unless there’s only one way to communicate, there will be a need for mediation, for example, HTTP to JMS bridging. It’s important that the set of technologies be small, however. Service enablement is about taking something that doesn’t adhere to the standards and exposing it in a way that it does. We should strive to reduce the amount of service enablement over time, but the need for connectivity and mediation will always be there.

Okay, no bears, sorry. I read post from my good friend Jeff Schneider regarding SAP’s Enterprise Service Repository (ESR). He states:

At the core of the SAP SOA story is the Enterprise Service Repository (ESR). It is actually a combination of both registry and repository. The registry is a UDDI 3.0 implementation and has been tested to integrate with other registries such as Systinet. But the bulk of the work is in their repository. Unlike other commercial repositories, the first thing to notice is that SAP’s is pre-populated (full, not empty). It contains gobs of information on global data types, schemas, wsdl’s and similar artifacts relating to the SAP modules.

This now brings registry/repository into the mix of infrastructure products that SAP customers must make decisions regarding adoption and placement. Do they leverage what SAP provides, or do they go with more neutral products from a pure infrastructure provider such as BEA, HP, SOA Software, or SoftwareAG/WebMethods? The interesting thing with this particular space is that it’s not as simple as picking one. Jeff points out that the SAP ESR comes pre-populated with “gobs of information” on assets from the SAP modules. Choose something else, and this metadata goes away.

I hope that this may bring some much needed attention to the metadata integration/federation space. It’s not just a need to integrate across these competing products, but also a need to integrate with other metadata systems such as configuration management databases and development lifecycle solutions (Maven, Rational, Subversion, etc.). I called this Master Metadata Management in a previous post.

Back when Gartner was pushing the concept of the ESB heavily, I remember an opening keynote from Roy Schulte (I think) at a Web Services Summit in late 2005. He was emphasizing that an organization would have many ESBs that would need to interoperate. At this point, I don’t think that need is as critical as the need for our metadata systems to interoperate. You have to expect that as vendors of more vertical/business solutions start to expose their capabilities as services, they are likely to come with their own registry/repository containing their metadata, especially since there’s no standard way to just include this with a distribution and easily import it into a standalone RR. It would be great to see some pressure from the end-user community to start making some of this happen.

Both Collin Smith and Rob Eamon responded to my post regarding my participation in an upcoming panel discussion at the Gartner EA Summit asking for my thoughts on SOA, EA, and EA Frameworks, so I thought I’d oblige.

First off, I can be considered a “big SOA” advocate. That is, I think it needs to be applied at something larger than a project scope. I think it needs to be an initiative that is not tied to any one particular implementation project. This implies that it needs to be driven by a group in the organization that is not predominantly consumed with project activities. One obvious candidate, therefore, is an Enterprise Architecture organization. In fact, I can’t think of any other organization that is as good of a fit. Individual managers may embrace it, but they are typically not positioned to guide the organizational and cultural changes required, except at the highest levels. Their primary concern (and rightly so) is typically keeping their stakeholders happy. Enterprise-wide, or even department-wide adoption of SOA can be very disruptive in the short term. So, if I had to pick a group to drive SOA adoption, it will almost always be enterprise architecture.

As for whether “SOA folds into EA,” I agree with Rob’s comments. SOA doesn’t replace Enterprise Architecture, it’s simply one view of the enterprise. Today, one could argue that most organizations view the IT landscape as a collection of applications. Efforts like application rationalization or application portfolio management reinforce this notion. So, you could also say that today we have application oriented architectures. The unit of composition is the application. This isn’t flexible enough, as it is too coarsely defined. If we break these applications into smaller uniits, we now get to service oriented architecture, which I feel is a better way of describing things. Is it the only way? Certainly not. There may be value in a process-oriented view. We still need deployment-centric views that simply show physical, and now virtual, servers. We may need a network-centric view. These are all tools in the toolbox of the Enterprise Architecture team, and depending on your specific responsibilities within that team, some may be more important than others. As I’ve mentioned before, I have a background in human-computer interaction going all the way back to my college days, and one thing that I’ve always believed is that it is very unlikely that one view, whether it be a diagram, a user interface, will meet the needs of everyone. This is why I’m also not a huge fan of EA Frameworks. I think EA frameworks can be of great value when you’re starting out. The scope of EA can be daunting, and if you’re tasked with establishing an EA practice in an organization, it never hurts to begin with an established framework. When those frameworks become too focused on trying to make everything fit into a one-sized fits all approach, rather than on actually making the effort successful is where things can become problematic. Within EA, I don’t think it’s necessarily the fault of the frameworks, but more due to EA being an immature practice. While the concepts have been around for more than a decade, there are still many large organizations (at least in the area where I live) that don’t have an EA practice at all, or have only been doing it for 2 or 3 years. While my sample base is relatively small, my experiences have been that every organization does it differently. Some EA groups have significant authority, some have virtually no authority. Some groups spend all of their time engaged on projects, some have no engagement with projects. Some are committees, some are standing organizations. Some are exclusively focused on managed the technology footprint, some are actively involved with business strategy and business architecture. With this much variation, it’s hard for any framework to achieve wide adoption, because they’re simply not a good fit for the short term needs that the EA team needs to accomplish. When the primary artifact of EA tends to be intellectual capital (i.e. thought leadership, future state models, etc.), you need to have flexibility in how that capital is represented, because consumption is the number one factor, not standardization.

I’ll be part of two panel discussions at the upcoming Gartner Application Architecture, Development and Integration and Enterprise Architecture Summits. These are being held at the Rio Casino and Conference Center in Las Vegas the week of Dec. 3-7. In the App Arch summit, I’ll be part of a Power Breakfast discussing funding SOA on Tuesday morning at 7:30 am. In the EA summit, I’ll be part of a panel discussion jointly moderated by Gartner and The SOA Consortium discussing the relationship between EA and SOA on Wednesday at 3:30 pm.  I’ll be at the two summits from beginning to end (Monday – Friday), so feel free to find me and say hi.  One of the more enjoyable parts of these conferences for me is the networking opportunities.

Dan Foody of Progress Software had an interesting blog recently called UDDI in a Web 2.0 World. In it, he asks:

SOA What? With all of this Web 2.0 development, it’s clear that internet scale folksonomies work far better than taxonomies. On the other hand enterprises are, for the most part, stuck with UDDI-related SOA governance tools and their strict taxonomy and categorization mechanisms. The open question though… is this really a problem?

Aside: I love the use of SOA What? That’s exactly why I try to always say S-O-A. On the subject, however, I think Dan raises an interesting question. One of the questions I’ve asked some of the registry/repository vendors is “Can you be indexed by a Google Appliance?” Admittedly, I’m not a huge fan of taxonomy-based searching. At the same time, however, a typical enterprise asset repository may not have enough critical mass to get appropriate metadata for folksonomy based searching. The Web is filled with hyperlinks. How many links to a service detail page am I going to have inside a typical enterprise?

Personally, I’d rather try to find a way to build up the metadata than go crazy building taxonomies to support direct navigation. First off, you can quickly get into taxonomy hell where there are so many variations that you try to support that it becomes difficult to present to the user. Second, people are so used to using Google, Desktop Search, Spotlight, etc. Universal search is going to be a standard part of the office toolset, and we need to find a way to ensure relevant results get returned. This will likely require analysis of software development artifacts (including source code) and building up those relationships based upon presence within project repositories and the role of the user performing the search. A developer performing a search will want to see very different results when searching on “Customer service” than a business manager.

The challenge we face is that the documents and their metadata are scattered all over the place. I previously asked if metadata should be the center of the SOA universe. Neil Ward-Dutton replied that it the center of the universe, and is inherently federated. We need intelligent crawlers that can infer the appropriate relationships and feed this into the universal search engine. Is anyone out there leveraging a Google appliance or other universal search option to facilitate searching for services and other IT assets? If so, like Dan, I’d love to hear about your experiences.

Mike “The Mad Greek” Kavis had an interesting post on SOA Lessons Learned over at IT Toolbox. First off, a big thank you to Mike for sharing some experiences about an effort that didn’t go the way it was originally planned. We can learn as much from these as we can from success stories.

The part of the post that caught my attention began with this line:

As we started the second wave of projects, I mandated that all code should be delivered with test harnesses, the build process should be 100% automated, and testing automation should be part of the project deliverables.

Mike went on to discuss how automation and governance were quickly forgotten when the schedule began to slip. The surprising thing to me was not that these aspects were dropped when the schedule began to slip, but that the implementation of things like continuous integration and testing automation were tied to the SOA effort to begin with.

To me, this was indicative of something that I’ve seen at a number of places which is to use “SOA” as the umbrella to fix all things in need of improvement within the software development process. Continuous integration is a great thing and everyone should be practicing it. But if you organization hasn’t adopted it or created a standard, repeatable way of doing it, don’t target your pilot for another key initiative (like SOA) to try to make it happen. SOA adoption is not dependent on having a continuous integration system. If you have it, will it make SOA easier? Yes, probably, but more so because it’s making all development easier, regardless of whether you’re practicing SOA or not. Give it its own pilot where it can be successful in a very managed fashion, and roll it out to the rest of the enterprise.

Part of the problem is finding ways to adopt these improvements in the development process. Is the business going to care about continuous integration? You can argue that they should, but it’s really about internal IT processes. All too often, IT is left to take the congressional lawmaker approach and find some big project that will be sure to be funded and then push everything but the kitchen sink into it under the radar. This creates additional risk and often results in the project team biting off more than they can chew. It’s unfortunate that IT frequently has little ability to improve on its internal processes due to the project-centric nature of its work. Take another support organization like HR. While I’m sure some work in HR work is project-driven, a lot of the work is day-to-day operations. My suspicion is that organizations that are focused more on fixed-cost day-to-day work like this probably have more ability to take on internal improvement initiatives.

My point of all this is that an organization has to be careful on what they try to take on. There’s always opportunities for improvement, and the point should be quality, not quantity. Trying to take on everything is unlikely to lead to success on any of it. Taking on a smaller set of goals and ensuring that you do them well is a safer approach.

I just read about three blog entries that all had to do with Joe McKendrick’s recent post titled, “Enterprise SOA concept falls out of favor.” Others commenting on this post include Jeff Schneider and this one (author unknown).Personally, I’m tired of these articles that are lamenting SOA. If you read between the lines, the real story is that there are very few case studies of “big SOA” in the way it was originally hyped. Therefore, we need to start hyping “small SOA” to keep it relevant. Neither one of these is the right thing. If “Big SOA” isn’t succeeding, we should be asking why? Odds are, it’s because the organizations involved are simply trying to do IT the same way that they’ve always been doing IT, and merely hoping that by creating some “services” things will suddenly be better. Sorry, won’t happen. In this scenario, what’s really happening is “Small SOA.” Because we haven’t changed anything about the way IT defines its activities, the only hope of achieving any of the hyped goals of SOA is to find some existing projects that are of sufficient enough scope to create that potential. If they don’t exist, now it’s a pure bottom up approach where people are throwing arbitrary services out there and hoping for the best. Certainly having some services is better than no services, but you’d be hard pressed to claim that the organization has adopted SOA and shown value that a CIO would appreciate.Coming back to “Big SOA”, it also doesn’t do any good to say, “it must be enterprise” and try to get funding for some all-encompassing, top-down enterprise initiative. That’s unlikely to happen. What needs to happen is “right SOA.” In order to do it, however, you need to have some knowledge of where you’re going to get the most bang for your buck. I point back to my post on horizontal versus vertical thinking for this. If you don’t have an idea on how to break down your functional capabilities into this, you need to take the time to better understand your business. Do the analysis, and then make the decisions on where to apply it. If you only focus on projects at hand, you may have success on a project with appropriate scope, but then it all falls apart when you don’t have the next target area. This doesn’t have to be some huge, all-encompassing analysis effort, but it does need to be enough to ensure that the techniques you’re going to apply are relevant, will add value, and will be successful. If we don’t have this knowledge, how can we have any confidence that any new technology approach, whether it is SOA, REST, BPM, EDA, AJAX, Web 2.0, etc., will be successful?

I’m a frequent listener of Biotech Nation with Dr. Moira Gunn, available through IT Conversations, and have begun learning more about the world of biotechnology. In doing so, I realized that there are some good analogies between the practices in agriculture and what we need for governance from IT. I’m not the first one to use an agriculture analogy, as Neil Macehiter and Neil Ward-Dutton described the IT technology landscape as a garden in their book, “The Technology Garden: Cultivating Sustainable IT-Business Alignment.”

When we grow something, it starts with the seed. The seed needs to germinate to grow into something, be it corn, wheat, soybean, etc. There’s plenty of things that can go wrong at this stage, so seeds are normally pre-treated with some form of protection, such as an pesticide. Once the seed germinates and the plant begins to grow, the farmer or gardener now needs to worry about weeds and insects. In the past, this involved heavy doses of chemicals, be it pesticides or herbicides. The problem with these is that these chemicals tend to work selectively on particular bugs or particular weeds. If you had a different bug or a different weed, you threw more chemicals on it. Not only is there risk that these chemicals stay on the food that the plants produce, but also that the runoff does harm to the environment. Today, there are non-specific herbicides that will kill just about anything that is green and leafy. The problem with this is that is would normally kill the crop. To that end, biotechnology has allowed crops to be grown that have a resistance to these non-specific herbicides. The net result is that less chemicals are necessary. Rather than continual spraying of selective herbicides, a single spray of a non-specific herbicide can be used.

So how does this relate to governance? The seed is the typical IT project. This is the thing that we’re trying to grow. Pick the wrong seed and you have problems. If you don’t pre-treat the seed, it may never germinate. The theme here is that projects need to be positioned for success from the beginning. There are activities that take place at the inception (or even before) of a project that can make or break your efforts. Given that most IT shops are still very tactical in their activities, SOA adoption is still predominantly a bottom-up effort. If you don’t properly scope your projects, as well as watch it carefully at the beginning so that any service development efforts are properly scoped, your chances of SOA success (or long-lasting IT success for that matter), will be less.

Step two in the growing process was the use of biotechnology. These plants have genetics that create a resistance toward the bad things that could come along. In the IT project sense, this is all about making the right thing the path of least resistance. If you’re successful with doing this through education, tooling, and mentoring, you won’t need to worry about review committees and rigid processes, because your seed will naturally do the right thing.

Finally, there is still a need for some general oversight. Weeds can still grow in your garden, and those weeds can consume vital resources that your plants need to live. In the IT sense, this isn’t about the specific decisions on the projects, it’s about the resources allocated to the project. Even if your architecture is sound, if your resources aren’t empowered to do the right thing, or if they’re pulled in many directions by multiple projects, you’re going to have problems.

So, while many people have frequently used the city planning analogy for architecture and governance, I’m beginning to like the gardening/farming analogy even more. Create an environment that positions things for success, make the “right thing” the path of least resistance for the project, and finally, keep the weeds out, allowing the resources allocated to focus on making the project successful.

It’s about 6:20 in the morning, and I’m presently on a 5 hour bus ride with a bunch of IT staff headed toward some facilities of my employer to learn more about their business. Before I get into the topic for this entry, I certainly want to give kudos to my employer for setting up this opportunity for IT to learn more about the business. With the long bus ride, I’m trying to get caught up on some podcasts. I’m presently listening to a discussion on SOA Management with Jason Bloomberg of ZapThink and Dana Gardner of Interarbor Solutions.

In his intro, Dana Gardner used the oft-mentioned and maligned term, silo. For whatever reason, it suddenly occurred to me that we always work within silos. Organizational structures create silos. Projects create silos. Physical locations can create silos. So, perhaps the right discussion shouldn’t be how to eliminate silos, but rather, how to choose silos correctly, work appropriately within them, and know when to redefine them. I’ve commented a bit on how to organize things, specifically in my post on horizontal and vertical thinking. For this entry, I’d like to discuss the appropriate way to operate within a horizontal silo.

A horizontal silo is one where the services being provided from that silo have broad applicability. A very easy example most organizations should be familiar with is servers. While there are multiple products involved based on the type of processing (e.g. big number crunching versus simple transactional web forms), you’d be hard pressed to justify having every project select its own servers. This area isn’t without change, one only needs to look at the space of VMWare and its competitors to see this.

In many IT organizations, when a centralized group is established, the focus can often be on cost containment or reduction. For many horizontal domains, this makes a lot of sense, but there’s a risk associated with this. When a group focuses on cost reduction, this is frequently done at the expense of the customer. Cost reduction typically means more standardization and less customer choice. To an extreme, the service team can wind up getting too focused on their own internal processes and expenses and completely forget about the customer. This is neither good nor bad, only something that must be decided by the organization. I can do a lot of my household shopping at either Target or Walmart. I’m probably going to have a better experience at Target, however, it will probably cost more than Walmart. What’s most important to you?

Within the enterprise, I’ve seen this dilemma occur in areas where some specialized technical knowledge is needed, but where the services themselves ore not standardized/commoditized. The most frequent example is that of Data Services. Many organizations create a centralized data services group to retain oversight over all SQL written. The problem with this is that while we may have a team that can write great SQL, we may not have a team that really understands what information the business needs from those queries. The team may try to minimize the amount of services available rather than give their customers what they need. What’s the right answer?

When establishing a service team, you need to think about your engagement model. Are you going to provide an outsourcing model, or a consulting model? In an outsourcing model, the service is largely provided without customer input. Customers simply pick from a list of choices and the burden is completely on the service team to provide. Getting a server or a network connection may be much better suited for this category. In a consulting model, there’s a recognition that some amount of input from the customer is still needed to be successful. I can’t create a good data service without knowledge of the customer’s information needs. A consulting model is going to be more expensive than an outsourcing model. If an organization is judging the success of this team based on cost, however, that’s a problem. This is an issue with the success criteria, however. First and foremost, we want to be sure that the right solution gets built. When the services being provided are provided in a cookie-cutter approach, the emphasis can be on cost. When each service requires customization, the focus needs to shift a little bit more toward providing the right solution, with cost as a secondary concern. It may not be about creating reusable services, but usable services that are less likely to cause problems than a custom-built solution by staff without the proper expertise in a given area.

More and more, we’re seeing articles that are now questioning whether SOA is just the next overhyped thing that has come out of IT that has failed to deliver on its promises. Based on a conversation with colleagues at the SOA Consortium, I think at least part of the reason for this is that many organizations still don’t really understand what they’re getting into. At the same time, there’s been a debate on the Yahoo SOA group on SOA as a business thing versus an IT thing. All of this seems to point back to a lack of a clear definition and picture of what we’re talking about.

As I started forming my ideas to blog on this, I was listening to a Software Engineering Radio podcast, which just happened to be a discussion on SOA with Nico Josuttis, who has just authored a book called, “SOA in Practice: The Art of Distributed Systems Design”. In the discussion, I thought Nico put it very well when he stated that SOA is essentially distributed system design where the distributed components have distinct owners. He then said the essentially, there won’t be any projects where a single team has full end-to-end control. It was that second statement that is the key message.

There’s no shortage of organizations that are simply taking the same projects that they’ve been doing for years and simply producing services now as part of them. Is this SOA? It is possible that the project could have been of significant enough scope (more likely a program) to where it included the development of services along with multiple consumers of those services, potentially yielding immediate benefits. The problem with judging SOA from this standpoint, however, is that it is unlikely to scale. Not all projects have this broad of scope. Secondly, it didn’t adhere to the second statement by Nico. By being underneath a common project or program, that effort had complete control over the entire domain. Yes, that project probably had multiple teams involved, but it’s likely that ultimate authority rested with the project architect and project manager, and if they said to do something, it was going to be done that way. Funding decisions and other governance issues don’t even come into play, because it’s all controlled by the project umbrella. An easy litmus test for this is to ask what the organization will do when a new consumer outside of the project comes along and wants to use one of the services that was (or is being) built, but needs some modifications. If it occurs while the project is still in flight, the project manager is likely to raise their hand and say, “Sorry, scope creep, can’t do it.” If it occurs after the project is complete, the new project may have no idea who to even talk to, because the whole structure around that service was based upon the project, and now that the project is over, it’s now being ignored, save for bug fixes/production problems. This isn’t a good situation.

I’m a believer that if you are adopting SOA, you’re committing to a fundamental change in the way IT operates. Of course, this assumes that there are opportunities for improvement within IT. If your IT department is delivering the right things to the business at the right time and for the right cost, great, you’re in the top 5% of organizations and have probably already embraced the changes, whether due to SOA or not. If you’re in the remaining 95% of organizations, you need to give some thought to where you want to go with IT. Is the use of shared services a goal? What are the domains where you know you’ll get some sharing? Is it more about business agility and being able to change the way things are wired in a more efficient manner? How will you tackle the problem of competing priorities when two projects want to leverage a new service, but have competing timelines, or even worse, where one project has control over one consumer and the service, and the other consuming project is at its mercy? How does the organization need to be changed to support this model?

Rather than try to paint my own picture of what this future state could be, I’m going to point all my readers to Nick Malik’s series on Joe Freeflier, part 1, part 2, and part 3. This does a great job in doing this. While this may represent an extreme example, and one that could take 5-10 years to reach, it does a great job in introducing the concepts and ideas that you need to be thinking about as you go forward with SOA.

Pete Lacey had a very interesting post entitled, “What is SOA?” I encourage you to go to his blog and read the whole thing, but I wanted to call out a couple of nuggets here for some comments. First:

“But wait,” I hear my SOA-loving readers say. “SOA is not about exposing business logic on the network. That’s just a technology thing. SOA is about the business! CxOs and business units don’t care about technology, they will only pay for business solutions.” Which always makes me scratch my head. What exactly does IT ever do that’s not about the business? Do they not work for the same company as the other BUs?

This is very reminiscent of the recent discussion on the business case for SOA, which I argued should be a business case for services, not SOA. Pete hits the nail on the head with his comment, “What exactly does IT ever do that’s not about the business?” Of course, it’s probably better stated, “Shouldn’t everything IT does be about the business?” After all, there’s probably some things going on most large IT shops whose business value is debatable. Anyway, what’s really interesting about this is how many activities within IT don’t go far enough to be about the business. IT is just as guilty of putting up walls as the business partners are. Recently, I had a discussion regarding instrumentation of services and collection of usage metrics. I argued that anyone who is acting in the capacity of a service manager should be looking at those metrics on a regular basis. Think about this. How frequently do you run into someone within IT who has recently put a solution into production who is actively monitoring the usage of it to determine if it is a business success? More often than not, I see project teams that quickly dissolve, allocated off to other projects, and no one paying attention to run-time usage unless some red light goes off because something’s broken. This type of monitoring should not strictly be the domain of “the business”. As IT, you are PART of the business, and you should be proactive in your monitoring and understanding of how your solutions are supporting the business. If you throw it over the wall into production, then the business-IT relationship is going to look very similar.

Pete also states:

“Well, it’s not just about business alignment,” another group of SOA advocates claim. “It’s really about governance.” Again, I’m scratching my head here. Everything is about governance! Software development (network-oriented or not) is about governance: ‘You must use a version control system; you must write unit tests.’ Moving systems into production is about governance. Updating a routing table is about governance. Hiring a new employee is about governance. Buying a plane ticket is about governance.

“No, no.” They go on. “With SOA there’s new things to govern.” That’s true, there are. But really, is it that much different than any other governance process? Not really.

I certainly agree that the need for governance is not just limited to SOA. What I’ve found in my work is that long term SOA success requires a fundamental change in the way IT operates. It’s not limited to SOA, however. SOA is merely the trigger that is bringing attention to the flaws in the system. As IT grows, there will be a need for governance, period, whether it’s for SOA or not. This doesn’t mean that SOA isn’t important, but it also means that there’s a risk that SOA becomes the banner under which all things wrong with IT are destined to be cured, and that’s a dangerous path. I’ve done work where the “SOA deliverables” contained a whole bunch of stuff that arguably were not specific to SOA at all. At the same time, if that’s what it takes to bring some attention to some things that need to be fixed, I’m not going to argue. Just be cautious in how much falls under that banner, because you can inadvertently jeopardize your SOA efforts on things that have nothing to do with it.