Todd Biske: Outside the Box

I’ll be part of a customer panel in Phil Windley’s SOA Governance session at the upcoming InfoWorld SOA Executive Forum. This event, as well as recent publications on SOA Governance (Phil’s article in the Jan. 23rd print edition of InfoWorld, Jason Bloomberg’s Butterfly Effect ZapFlash), have me thinking more and more about this topic.

One very simple analogy to make regarding SOA Governance is to compare it to traditional government. What does Congress do? What do your local city leaders do? This is actually a very good analogy, because it works for all definitions of SOA Governance, whether applied to strategic planning, design-time governance, or run-time governance. Recently, I’ve started thinking about more intricate details of this analogy. It is certainly true that SOA Governance has to do with setting and enforcing policy, just as does traditional governance. What’s interesting, however, are the types of policies being set and the communication involved in those processes.

A great illustration is my own experience. From mid-1994 to late 1997, I lived in California. During that timeframe, California had ballot initiatives on discrimination in state university admission processes (Proposition 209) and on basic services (i.e. health care) for illegal immigrants (Proposition 187). As part of the voting process, California would provide a sample ballot, as well as statements on the proposed initiatives and candidates. There would typically be a statement for the initiative/canditate along with a rebuttal, as well as a statement against the initiative/candidate along with a rebuttal. I found this to be extremely valuable in my own processes to be an educated voter.

In late 1997, I moved to Missouri. The most highly contested initiative for my first ballot as a Missouri resident was cock fighting restrictions. Missouri politicians are only now beginning to debate things like smoking bans in restaurants. Thankfully, I’ve moved to southern Illinois where we at least have the influence of Chicago on statewide issues to keep things a bit more progressive. Neither state, however, provides anything to help me be an educated voter. I have to rely on my own research, in addition to the normal media.

What this demonstrates is that in addition to figuring out what SOA means to your company, you also need to figure out what SOA Governance means to your company. Should you be progressive about governance and focus on cutting edge concerns, or do you still need to cover the basics? What information do you need to provide as part of the governance process, and how much do you involve the people who will ultimately be governed by the policies being set? Ultimately, success will largely depend on how well your approach to these topics match your corporate culture. If your company is conservative, adopting a progressive stance on governance may be too much shock for the company to take right off the bat. Likewise, A conservative approach on governance may put a halt to the rapid innovation occurring at a progressive company. The same holds true in traditional government. The focus of media attention in the middle east has been on the fact that voting is occurring, rather than on what topics are on the ballot. Simply having a vote for the highest levels of government is enough of a culture change. Anything more would have been a failure.

As with any change, and SOA is most definitely a change, the key is understanding where you want to go, creating a plan to get you there, and then executing that plan. SOA Governance is simply another element to be addressed. Take the time to understand what your needs are and the pace that they can be addressed. If you’re currently using punch cards, hanging chad and all, but voting on things that have minimal impact on society, switching to an electronic voting machine isn’t going to help.

Tags: SOA, Governance

WebServices.org created a marketing opportunity for several vendors by asking a number of industry visionaries to tell uswhat to expect for 2006. Personally, I’d love to see Joe do the same thing with technology analysts. Most of them have posted their 2006 predictions, although I haven’t seen anyone roll them all up with appropriate analysis.

Back to the vendors predictions: my first reaction was that most of them (but not all) made statements consistent with their marketing messages. What’s good about this is that now we can compare them for consistency, inconsistency, etc. and see what conclusions can be drawn. Let’s start with the buzzwords.

• ESB: Three people mentioned ESBs: Paul Patrick (BEA, they sell one), Paul Lipton (CA), and Bob Brauer (StrikeIron). All three extremes are covered on this one. Paul Patrick sees “an increasingly important role for ESBs” emerging. Paul Lipton dismisses ESB as an “excessive and inappropriately used buzzword.” Bob Brauer comes in the middle, merely stating that emphasis will shift from the infrastructure components to “actual solution building.”
• Governance: This one had more press. Paul Lipton also classified “governance” as an “excessive an inappropriately used buzzword” but then said the new “excessive and inappropriately used buzzword” will be policy, which has clear ties to governance. Bob Brauer included governance in the list of items that had previously been emphasized. Andrew Nash (Reactivity) never used the word governance, but did discuss the need for standard semantics for policy-based management (we need semantics for the actual service definitions, too). Inge Cheng (Layer 7), stated that there will be “questions of how services can be governed consistently across application and organizational silos.” Miko Matsumura (Infravio) wins the award by titling his comments, “It’s About Governance, Governance, Governance.”

Looking for common threads across all of these, I see the following:

• We’ve only begun our understanding of Governance. My own recent blog called out the confusion around development governance versus operational management or run time governance. Paul Lipton (CA) called out the same thing. Over half of the comments had something to do with governance, so it’s clear that there’s still plenty of life left in that topic.
• Enterprise SOA adoption will still be few and far between. Andrew Nash (Reactivity) made some very conservative comments regarding SOA adoption, but stated that there “have been some real projects with real money behind them.” Wayne Ariola (ParaSoft) hit on a topic that hasn’t received much discussion, which is trust between service provider and consumer. The “not done here” mentality of many developers is a barrier to enterprise adoption, and until that culture changes, progress will be slow. Inge Cheng (Layer 7) calls out that most implementations have been at the project level, not the enterprise level. Miko’s (Infravio) comments on governance stressed that the barriers are “organizational, political, social, and regulatory.” These don’t change overnight. Paul Lipton’s (CA) comments are that there will be “an increased realization that SOA is more than an IT initiative.” This doesn’t say that the business side will get involved with SOA adoption in 2006, only that they’ll begin to understand that they need to be involved. All of these things point to a slow, conservative uptake. Andrew Nash phrased it well: “I’m not going to say that 2006 is the year where it will all happen. Rather, this is the start of a long-term, steadily increasing use of SOA.”

What would I like to see? I’d like to see some case studies on companies who are starting to show results from an SOA adoption that was formalized no earlier than 2003. Most case studies I’ve seen fall into two categories:
1. Projects that were well-suited for SOA and Web Services, but existed solely within a single domain of the business, eliminating many of the governance and management issues inherent to an enterprise adoption.
2. Enterprise scale initiatives that began prior to 2003, where the direction was clearly laid out by a CxO level executive to embrace enterprise architecture, system reuse, and elimination of redundancy. These typically only mention Web Services at the end of the study when they state that they are incorporating the use of Web Services and associated infrastructure into their environments.
Three years should be long enough to show some preliminary results across the enterprise, even if this was going on while business still churned along.

I think there’s going to be a butterfly effect of blog postings from the latest ZapFlash. As always, the ZapThink guys (in this case Jason) have managed to take a hot topic (SOA Governance), look at it from an unexpected angle (chaos theory), touch on many of the key aspects of it, get people thinking about the possibilities down the road, and introduce just enough controversy to get people talking about it (like me).

The first thing that struck me was when Jason introduced two views of SOA Governance. He states:

There are two faces to SOA governance, however. On the one hand, SOA governance simply means governing a SOA implementation initiative — for example, communicating corporate policies to developers implementing Services, and giving them the tools they need to follow those policies as they assemble the various elements of the SOA implementation. On the other hand, there’s a broader, more strategic definition of SOA governance: IT governance in the context of SOA.

The example of IT governance that he uses later in the article is that of a business executive changing the SLA associated with a corporate report to require a one-day turnaround rather than a one-week turnaround. My first question is whether or not this constitutes SOA governance. This isn’t the first discussion around SOA governance that has broached into this territory, and the truth is that there is no solid definition of what is SOA governance and what isn’t.

The source of the confusion, as I see it, is that both areas Jason describes involve policies. Does this mean that the act of setting policies is always an act of governance? The concern I have with Jason’s example is that the act of the executive changing the policy is an example of poor governance. How many governments enable this kind of legislative power? This sounds like a dictatorship. I’m not implying that executives shouldn’t be empowered, but policies shouldn’t be enacted or changed without a thorough understanding of the consequences.

The second thing that struck me about this article is that in typical ZapThink fashion, this scenario is absolutely possible. It may not be the business executive, but it certainly could be an operations staffer, or even worse, it could be the system itself! I recently replied to a query on a mailing list about applying SOA in areas outside of business applications with an operational management scenario that was fully automated. The automation was controlled by policies that could be performed by intelligent infrastructure with operational interfaces exposed as web services, orchestrated through a BPM platform. All aspects of this are declarative in nature.

The real risk that Jason is pointing out has more to do with automation of processes that it does with governance. With more and more systems being configurable through declared policies, this leads to automation possibilities, which in turn can lead to exactly the possibility that Jason describes.

The takeaway, as I see it, is that the industry is continuing to move in the direction of empowering the user, all driven by a desire to have our systems be more flexible and responsive to business change. We have to think about what the impact of that future state is. If the business users have the power, what governance has to be in place around the changes that they can now make?

James McGovern posted some comments on what “the real problem” is in adopting SOA.

The most important thing that he called out is that “SOA is a paradigm shift.” I couldn’t agree with this more. While he went on to apply this to IT staffers that have worked in legacy environments, I would go even further than that. Anyone who hasn’t had the opportunity to work on a large scale program that impacted a significant portion of the enterprise falls into this category. Why? They simply haven’t had to look outside of the boundaries of the project they were handed. I think this probably impacts the average client-server developer far more than the legacy developer. A VB6 developer has probably built a lot of point solutions for a small set of users. The legacy developer has been the one having to stretch their system to integrate with the demands from the changing business.

Independent of any individual developer, an enterprise wide SOA implementation absolutely requires a paradigm shift. It is a culture change, plain and simple. As long as we continue to define projects in such a way that it is difficult to see outside of the boundaries of that project, we won’t be successful. Think about the case studies that have been published regarding SOA. How many of them are service-oriented applications versus service-oriented applications? Don’t get me wrong, there is nothing wrong with service-oriented applications, and it absolutely is a step in the right direction. It is not service oriented architecture, however. If you look at a case study for service oriented architecture, you will find that the culture change was directed from the top down. Without this culture change, the path to a service oriented architecture will be far longer, and probably far more painful.

I’ve said this before, and I’ll keep saying it. In order for SOA to achieve the goal of accommodating business change, IT has to be willing to change first.

Hey, maybe Phil Wainewright reads my blog. Yesterday, I posted some comments about his blog on LooselyCoupled.com, and some of the excellent comments he made regarding SOA Repositories. I pointed out the importance of marketing and communication to reuse, and how the social networking aspects of Web 2.0 was a perfect fit for this. I called out Verizon’s SOA as a perfect example. Well, lo and behold, today, Phil posted on his ZDNet blog today:

Learning to love software reuse by ZDNet‘s Phil Wainewright — Only when you move to a truly loosely coupled services architecture do you get to reuse software without the temptation of changing the code. But there are still obstacles.

If you follow this, you’ll find that Phil posted on Verizon’s SOA. That certainly sounds familiar. It was either pure coincidence (which is entirely possible, he could have planned the split of content between the two blogs all along) or maybe he read my posting. Phil, drop me a comment if you did!

Phil Wainewright, of ZDNet and LooselyCoupled, posted an article on SOA repositories at LooselyCoupled. I thought it did a great job in calling out the phases that we’ve gone through in realizing that a system facing registry was not enough, and that tools were needed at design and development time that were human facing, i.e. repositories. Joe McKendrick of ZDNet also posted a summary of this.

No one will argue that one of the reasons for embracing SOA is reuse. Reuse has been somewhat of a holy grail in software development, however. My first introduction to the challenges associated with reuse came at the Software Development Expo East (SD Expo) in 1998. I can’t remember who the presenter was (and unfortunately, the archives at http://www.sdexpo.com/ only go back to 1999), but he stated that the number one factor that determined success or failure of an enterprise-wide reuse effort was marketing and communication.

Therefore, it’s no surprise to me that the industry is starting to rally around the notion of a repository, rather than a registry. In reality, it goes far beyond that. If the key to success is marketing and communication, this seems to be a great fit for everything that is Web 2.0. The example of Verizon’s IT Workbench Portal is a great case study. While Verizon’s approach did begin with a CIO mandate, their approach was not strong-handed, but rather a non-intrusive process that integrated seamlessly into the developer’s world. The term repository almost seems too limiting in this context. Perhaps this is the first big example of how the social networking aspects of Web 2.0 can be utilized productively in the corporate world.

Yesterday, I was driving my oldest daughter to school (she’s in Kindergarten) and she was very excited. On Thursdays, her class goes to the computer lab, and “computers are her favorite subject!” So, that night at dinner, I asked her what she did in computers. She told me that they went to a new web site that had a doodle pad. They were all supposed to draw a picture of something (I don’t remember what), and then they could draw whatever they wanted. I asked her what she drew, and she said, “A three-dimensional apartment building.” Not the answer I was expecting.

Anyway, what does this have to do with SOA? Well, first off, the fact that my 5 1/2 year old daughter is drawing “three-dimensional apartment buildings” in Kindergarten tells me that the pace of things, and the need for information for this generation is only going to grow. If this isn’t a reason to start building agility into our systems, I don’t know what is.

Second, a common analogy for enterprise architecture these days is that of city planning. How many of us can look at the “doodle pad” of our enterprise architecture and know where to place those three dimensional apartment buildings? Or, is your architecture have more parallels to the hot political potato of eminent domain? Are you having to bulldoze applications that were only built a few years ago whose residents are currently happy for the greater good? What about old town? Any plans for renovation? If you aren’t asking these questions, then are you really embracing enterprise SOA?

In talking to the computer teacher this morning, she said that my daughter had drawn a cube. A box. To her, it wasn’t just a box. It was a three-dimensional apartment building. One of the pieces of advice I had in my talk at the IQPC SOA conference in Chicago last October was “Think Outside the Box.” My 5 1/2 year old daughter is doing this. To be successful with SOA, someone in your organization has to be doing this. If you’re still thinking in terms of the application at hand, you won’t get there. Someone needs to have the responsibility to be looking for these opportunities. Not everyone can look at a vacant lot in the city, and envision what can happen by bringing the appropriate business to that lot. Not everyone can look at the open land on the outskirts of town and envision what it can become. Every successful city has someone who has done this, working with developers and the community at large to make it happen. Use this analogy, and find those planners in your organization who are not simply looking at one building or one subdivision. Find the ones who can look outside of the box, and build your services blueprint that you need to chart the future course.

There have been a number of blogs recently discussing Web 2.0 and SOA. In his IT Toolbox blog, David Linthicum quotes Dion Hinchcliffe, essentially saying that Web pages are becoming less important, and Web services are becoming more important.

One thing that I’ve spoken with some people about is applying the concepts of SOA to the Presentation Tier. I think portlets are to the presentation tier what Web Services are to the business tier. In the same way that the business tier is headed toward orchestration and composition of services, the presentation tier should be headed toward orchestration and composition of presentation services, i.e. portlets. This tier is far more challenging than the business tier, simply because this is the interface that deals with people, and people are complicated. The number of things that can be (and need to be) communicated through a user interface is far greater than what needs to be communicated from a WSDL file.

That being said, I think we can really get somewhere if some of the best thinkers on server-side SOA got together with some leading usability / user interface gurus and tried to come up with a solution. If we had a solution that captured the dynamic behaviors and flexibility required on the user interface side, this could in turn be utilized to enable dynamic behaviors and flexibility on the server side even more.

Jeff Schneider, of MomentumSI, posted his predictions for 2006. One of them was this:

The Business Process Platform becomes the accepted standard as the foundation for enterprise software.

This was very interesting. One thing I’ve commented on in the past is the tooling support for contract first development, and how the BPM tools work better when taking a contract first approach. One thing that I don’t think too many people will argue with is that with advances in frameworks and code generation, the opportunity for more visual development environments on the server side is increasing. You can look at BEA Workshop, M7 (now also owned by BEA), or Microsoft’s Whitehorse as examples of this. Therefore, there is a convergence that is happening between these two environments for developing services. While this convergence continues, what do we do in the short term? What things are best suited for development on a business process platform, using schema-driven tools, and what things are still best suited for development in a traditional OO programming language? Certainly, problem number one is that each tool has its own custom development “language.” While most of the tools either have or plan to have BPEL support, the development language is usually a graphical model from which BPEL can be generated/exported. What about the model for reuse? Do we need OO concepts like inheritance in these visual models? What other programming concepts are being rendered unnecessary for typical corporate IT development because all of the source code is generated behind the scenes, if at all (do these tools generate code or are they tightly coupled with an execution container that interprets the models on the fly… I suspect the latter)? I’m very interested in hearing what other practitioners are doing- where are they getting big wins from the graphical model-based business process platforms and where are they continuing to utilize programming languages.

“SOA is about breaking the silos in a company… A successful SOA requires that departments and teams collaborate with each other not bilaterally but well in a federation of service providers and consumers. Small central teams will have to be created to manage that growing spaghetti of interactions at the federation level. Companies that don’t create these central teams will fail in getting any return from their SOA.”

The above quote is from Robin Mulkers. I couldn’t agree with it more. One of the key challenges in adopting SOA across an enterprise is determining what the right organizational approach is. Clearly, the old way of developing applications will not work in the SOA future. These applications must be decomposed into business services, each of which must be developed and maintained independently of the other services. This means that the “traditional” project will involve more development groups, and the releases of each of these groups must be coordinated to not create a project management nightmare. This is a huge challenge for the enterprise, and must be done as a planned transition rather than as a big bang. Start with a few dedicated service development teams, and slowly transform the development organization from an application development organization and into a service development organization.

A colleague of mine was showing me an RSS feed that the USGS is now publishing.  Within each item, in addition to the normal title, description, and link elements, it also contains latitude and longitude values, among other things.  This spun its way into a conversation around subscribing to events, and whether we need an EDL (Event Description Language) that would be analagous to WSDL.  Essentially, at the time a subscription is established, the EDL could be pulled in, allowing the subscriber to integrate the data from those events in a much simpler fashion.  XML Firewalls could also use that to perform schema validation (among other things), ensuring that incoming event feeds don’t do bad things to the subscribers.  The question is, does this exist?  Does WS-Notifications handle this?  If we had an EDL, we could have standard Web Services for querying the EDL and subscribing to the event feed.  In the case of the USGS, I could have an application that queries the EDL, and sets up a subscription to earthquake events that occur within some range of lat/lon values of the user’s current location. 
It’s suprising to me how EDA has not received anywhere near the press that SOA and BPM have.  To me, they represent the triumverate that must exist.  Services are the processing logic that get work done.  Business processes control the sequencing of that logic, with events triggering the transitions. 

Phil Howard of Bloor Research recently wrote a series of articles after attending IBM’s annual analyst conference. In this one, he does a comparison of an ESB-based SOA to EAI-based hub-and-spoke. It was this comparison that caused me to give some thought to why ESB is such a strange beast. It starts with the name: Enterprise Service Bus. Clearly, there are products that are marketed as an ESB that truly implement a bus. There’s also many products that are marketed, and even named, as an ESB that don’t. Perhaps they use ESB to stand for Enterprise Service Broker, but no one’s ever asked them. It would be very interesting if the products had more representative names using the ES* prefix. We could have Enterprise Service Integrator, Enterprise Service Orb, Enterprise Service Broker, Enterprise Service Hub, Enterprise Service Router, Enterprise Service Appliance, etc. That’s not the point I’m trying to make, however.

The point where we can actually have some meaningful discussion in the space, in my mind, is what truly belongs in the middle, and what belongs on the endpoints/nodes. The most controversial point of most of the ESBs for me, is when they begin to include orchestration. To me, an orchestration engine is an endpoint, not something that sits in the middle. It can act as a service provider (typically kicked off through a fire-and-forget style service request, more on service invocation styles for another blog), and certainly acts as a service consumer, in response to events occurring in the enterprise. Taking orchestration out of the picture, the question is all about what technology is at the endpoints, and what capabilities you need in the middle to best handle the communication between those endpoints. Once you know what capabilities are needed, make the call as to whether you need a bus architecture, a hub-and-spoke architecture, or a fabric/network. There are products that provide all of these approaches in a variety of form factors. One thing you can always assume, however, is that standards will win out. Heterogeneity is here to stay, and the major vendors are committed to supporting interoperable standards.

The grey area is in the transformation space, specifically in the semantic space. When it’s as simple as applying an XSL stylesheet, putting it “in the middle” makes sense. What if that semantic translation requires calls out to multiple back end systems, though? Making calls out to multiple systems and then combining the results to pass along sounds much more like either a composite service or some automated orchestration to me. Per my previous argument, this belongs on the endpoints. Simply put, there is no right answer. The beauty of IT systems is that there are many viable choices, and every one will have different contexts that may point in a different direction. The debate around ESBs, intelligent networks, etc. is often simply about moving the location where processing occurs, not introducing new processing. My advice? Take it all into account. Understand how the processing rules can change over time, who uses them, who changes them, and look at the vendor space to determine products that have the right vision and the right fit for your organization. Those products that try to define the space in a way that doesn’t match up with the viewpoint of too many consumers won’t survive. Some will find a niche will enough demand to sustain the product for a healthy amount of time, and others will find the space with the most growth potential. And then it will all change again.

I recently attended a conference with a colleague who is new to the world of SOA. We had some conversations after a couple of case studies about whether what was presented was SOA or not. This reminded me of the IQPC conference in Chicago where I spoke. Interestingly, nearly every presenter there began with a definition of SOA. Why is this? I’ve given it some thought, and realized that one of the most critical steps in implementing SOA is actually defining what it means to your enterprise.

One of the classic examples for Web Services is the supply chain model. In this case, services are exposed to partner organizations. This is an example of what David Linthicum calls “Outside-In SOA.” In looking just at one partner’s services, what are the reasons for this service? Clearly, ease of integration is one. How about reuse? That’s a common goal for SOA, although in this case, it would be reuse by multiple partners rather than multiple internal applications. What’s interesting then, is that for this company, the definition of the enterprise must be extended to include at least some portion of the external partners’ enterprises as well.

Continuing to extend this same example, let’s look inside. It is entirely possible that these services at the edge may yield the benefits of SOA, but that the internal view of the enterprise is not service-oriented at all. The implementations behind these services could each be their own stovepipe solution without any elements of service composition whatsoever. In many of these case studies, we simply don’t know.

What I told my colleague was that these case studies were indeed examples of service-oriented architecture applied to a particular problem space, or put another way, a service-oriented application. Everyone has to start somewhere, and these types of problems are well-suited to a service-oriented approach. Achieving a service-oriented architecture, or as Jeff Schneider’s has titled his blog, the service-oriented enterprise, is a far more daunting challenge, and one that will take careful planning driven from the business strategy. Until then, continue to look for the opportunities to apply service-oriented approaches where you can.

If you haven’t figured it out, I switched from a Blogger-managed blog, to managing it myself. Sorry for the inconvenience, since it probably means the existing feed is now broken. The new feed is this.

The ZapThink guys put on a great ZapForum today on SOA Governance with Andrew Nash from Reactivity and Sean Kline from Systinet. You may have even heard me on it if you stuck around for the Q&A portion, since Ron and Jason were kind enough to take one of my questions for about the 5th forum in a row, I think. While my last post mentioned that the key to governance was process, and not technology, I also pointed out that technology can make governance easier. If you want to get a good picture of how it can, check out the replay of this when it’s available. There were some really great nuggets in this one. Two of them that I’d like to point out:
• Ron and Jason set a goal for the future of using SOA principles to assist governance, not just governing SOA. I’ve said the same thing about Systems Management.
• Andrew and Sean actually had elements of this in their talk when they suggested using BPM/Workflow to assist in the policy setting process associated with a service invocation. This was one of the first times I’ve seen this recognized in a presentation, although I’ve had many a conversation about it.

My question for this forum was on the lack of standard policy assertion languages. To that end, OASIS has announced that they are considering forming a TC on Domain Independent Policy Assertion Languages. Progress, progress, progress…