Todd Biske: Outside the Box

Archive for the ‘SOA’ Category

“SOA is about breaking the silos in a company… A successful SOA requires that departments and teams collaborate with each other not bilaterally but well in a federation of service providers and consumers. Small central teams will have to be created to manage that growing spaghetti of interactions at the federation level. Companies that don’t create these central teams will fail in getting any return from their SOA.”

The above quote is from Robin Mulkers. I couldn’t agree with it more. One of the key challenges in adopting SOA across an enterprise is determining what the right organizational approach is. Clearly, the old way of developing applications will not work in the SOA future. These applications must be decomposed into business services, each of which must be developed and maintained independently of the other services. This means that the “traditional” project will involve more development groups, and the releases of each of these groups must be coordinated to not create a project management nightmare. This is a huge challenge for the enterprise, and must be done as a planned transition rather than as a big bang. Start with a few dedicated service development teams, and slowly transform the development organization from an application development organization and into a service development organization.

A colleague of mine was showing me an RSS feed that the USGS is now publishing.  Within each item, in addition to the normal title, description, and link elements, it also contains latitude and longitude values, among other things.  This spun its way into a conversation around subscribing to events, and whether we need an EDL (Event Description Language) that would be analagous to WSDL.  Essentially, at the time a subscription is established, the EDL could be pulled in, allowing the subscriber to integrate the data from those events in a much simpler fashion.  XML Firewalls could also use that to perform schema validation (among other things), ensuring that incoming event feeds don’t do bad things to the subscribers.  The question is, does this exist?  Does WS-Notifications handle this?  If we had an EDL, we could have standard Web Services for querying the EDL and subscribing to the event feed.  In the case of the USGS, I could have an application that queries the EDL, and sets up a subscription to earthquake events that occur within some range of lat/lon values of the user’s current location. 
It’s suprising to me how EDA has not received anywhere near the press that SOA and BPM have.  To me, they represent the triumverate that must exist.  Services are the processing logic that get work done.  Business processes control the sequencing of that logic, with events triggering the transitions. 

Phil Howard of Bloor Research recently wrote a series of articles after attending IBM’s annual analyst conference. In this one, he does a comparison of an ESB-based SOA to EAI-based hub-and-spoke. It was this comparison that caused me to give some thought to why ESB is such a strange beast. It starts with the name: Enterprise Service Bus. Clearly, there are products that are marketed as an ESB that truly implement a bus. There’s also many products that are marketed, and even named, as an ESB that don’t. Perhaps they use ESB to stand for Enterprise Service Broker, but no one’s ever asked them. It would be very interesting if the products had more representative names using the ES* prefix. We could have Enterprise Service Integrator, Enterprise Service Orb, Enterprise Service Broker, Enterprise Service Hub, Enterprise Service Router, Enterprise Service Appliance, etc. That’s not the point I’m trying to make, however.

The point where we can actually have some meaningful discussion in the space, in my mind, is what truly belongs in the middle, and what belongs on the endpoints/nodes. The most controversial point of most of the ESBs for me, is when they begin to include orchestration. To me, an orchestration engine is an endpoint, not something that sits in the middle. It can act as a service provider (typically kicked off through a fire-and-forget style service request, more on service invocation styles for another blog), and certainly acts as a service consumer, in response to events occurring in the enterprise. Taking orchestration out of the picture, the question is all about what technology is at the endpoints, and what capabilities you need in the middle to best handle the communication between those endpoints. Once you know what capabilities are needed, make the call as to whether you need a bus architecture, a hub-and-spoke architecture, or a fabric/network. There are products that provide all of these approaches in a variety of form factors. One thing you can always assume, however, is that standards will win out. Heterogeneity is here to stay, and the major vendors are committed to supporting interoperable standards.

The grey area is in the transformation space, specifically in the semantic space. When it’s as simple as applying an XSL stylesheet, putting it “in the middle” makes sense. What if that semantic translation requires calls out to multiple back end systems, though? Making calls out to multiple systems and then combining the results to pass along sounds much more like either a composite service or some automated orchestration to me. Per my previous argument, this belongs on the endpoints. Simply put, there is no right answer. The beauty of IT systems is that there are many viable choices, and every one will have different contexts that may point in a different direction. The debate around ESBs, intelligent networks, etc. is often simply about moving the location where processing occurs, not introducing new processing. My advice? Take it all into account. Understand how the processing rules can change over time, who uses them, who changes them, and look at the vendor space to determine products that have the right vision and the right fit for your organization. Those products that try to define the space in a way that doesn’t match up with the viewpoint of too many consumers won’t survive. Some will find a niche will enough demand to sustain the product for a healthy amount of time, and others will find the space with the most growth potential. And then it will all change again.

I recently attended a conference with a colleague who is new to the world of SOA. We had some conversations after a couple of case studies about whether what was presented was SOA or not. This reminded me of the IQPC conference in Chicago where I spoke. Interestingly, nearly every presenter there began with a definition of SOA. Why is this? I’ve given it some thought, and realized that one of the most critical steps in implementing SOA is actually defining what it means to your enterprise.

One of the classic examples for Web Services is the supply chain model. In this case, services are exposed to partner organizations. This is an example of what David Linthicum calls “Outside-In SOA.” In looking just at one partner’s services, what are the reasons for this service? Clearly, ease of integration is one. How about reuse? That’s a common goal for SOA, although in this case, it would be reuse by multiple partners rather than multiple internal applications. What’s interesting then, is that for this company, the definition of the enterprise must be extended to include at least some portion of the external partners’ enterprises as well.

Continuing to extend this same example, let’s look inside. It is entirely possible that these services at the edge may yield the benefits of SOA, but that the internal view of the enterprise is not service-oriented at all. The implementations behind these services could each be their own stovepipe solution without any elements of service composition whatsoever. In many of these case studies, we simply don’t know.

What I told my colleague was that these case studies were indeed examples of service-oriented architecture applied to a particular problem space, or put another way, a service-oriented application. Everyone has to start somewhere, and these types of problems are well-suited to a service-oriented approach. Achieving a service-oriented architecture, or as Jeff Schneider’s has titled his blog, the service-oriented enterprise, is a far more daunting challenge, and one that will take careful planning driven from the business strategy. Until then, continue to look for the opportunities to apply service-oriented approaches where you can.

If you haven’t figured it out, I switched from a Blogger-managed blog, to managing it myself. Sorry for the inconvenience, since it probably means the existing feed is now broken. The new feed is this.

The ZapThink guys put on a great ZapForum today on SOA Governance with Andrew Nash from Reactivity and Sean Kline from Systinet. You may have even heard me on it if you stuck around for the Q&A portion, since Ron and Jason were kind enough to take one of my questions for about the 5th forum in a row, I think. While my last post mentioned that the key to governance was process, and not technology, I also pointed out that technology can make governance easier. If you want to get a good picture of how it can, check out the replay of this when it’s available. There were some really great nuggets in this one. Two of them that I’d like to point out:
• Ron and Jason set a goal for the future of using SOA principles to assist governance, not just governing SOA. I’ve said the same thing about Systems Management.
• Andrew and Sean actually had elements of this in their talk when they suggested using BPM/Workflow to assist in the policy setting process associated with a service invocation. This was one of the first times I’ve seen this recognized in a presentation, although I’ve had many a conversation about it.

My question for this forum was on the lack of standard policy assertion languages. To that end, OASIS has announced that they are considering forming a TC on Domain Independent Policy Assertion Languages. Progress, progress, progress…

Governance is one of my favorite topics.  If someone asked me the thing that will influence the success of an SOA initiative the most, it would be governance. A very easy way to look at governance it is to compare it to a traditional government.  A government has to legislate, provide infrastructure, maintain strategic plans, enforce laws (police force), etc.  These are all activities that an IT organization must do to govern an SOA.  In reality, these are all things that an IT organization should have been doing, regardless of whether SOA is being done or not. 
 
The same challenges that municipalities face in their strategic growth are faced by IT organizations.  Urban centers grew through a very centralized approach, but have had to become more and more decentralized due to suburban sprawl.  As rural communities have grown, they have had to work more and more with their neighboring communities, possibly sharing common infrastructure and services.  The same is true of IT organizations. The urban center can be thought of as the mainframe or legacy systems.  Due to the web, web services, etc., portions of the legacy logic needs to be decentralized to meet the demands of the future.  At the same time, silo’d application development represents the rural communities.  These applications have grown, and the world of business processes is requiring them to work together seamlessly, rather than through inefficient handoffs and redundant processing. 
 
There are now tools out claiming to provide “SOA Governance.” The truth is that there is no tool or technology that will provide SOA Governance.  There are tools and technologies that can make governance easier, but ultimately, it will come down to process and communication.  If the process and communication isn’t there, the governance won’t be either.  At the same time, we can’t govern by process alone.  The things being enforced (i.e. the legislation) must be documented for all to see.  Herein lies the real challege with regards to SOA or, more broadly, applying governance to IT.  SOA is about looking horizontally while others are looking vertically.  How do you document the rules associated with making something an enterprise service versus an application-specific service?  Yes, we can have rules around WS-I compliance and naming conventions, but this often comes down to semantics and a strategic vision (i.e. business service blueprint).  This is akin to a business applying for a business license in a city.  There will be guidelines for the application that must be followed, but there is still a judgement that must be done by a city council as to whether they want the business in their city.  There may be general guidelines in the city master plan, and the opinions of the council members are exposed through the political process, but largely, things will be handled on a case by case basis by a set of people given the responsibility for making those decisions.  If you have the wrong people in place, you won’t be successful.
 

Most people are advocates of taking a WSDL-first/Schema-first/Contract-first approach to Web Services. I think most people would also agree that when implementing the service using Java or C#, the tooling support for this approach leaves a lot to be desired. That being said, we still need to do this.

The scenario gets more complicated when you take into account that many industry groups have produced data standards built using XML Schema. Many of these were established well before Web Services.

What I’ve been surprised by is the lack of documentation on how to bridge these two worlds. Maybe I haven’t looked hard enough, but I haven’t come across any best practices around XML Schema design when the intent is to use them within Web Services, especially when you know you’re going to be using Java or C# for the implementation. Anyone who’s gone the code first approach knows that class members map to child elements and not to attributes of an element. So, does this mean that when designing XML schemas for use within Web Services we should avoid using attributes of elements? If we don’t, does it mean that the developers writing the service are going to have to deal with low level XML APIs? Of course, a lot of this could be avoided by using a schema-oriented development environment typically associated with BPM tools, but that platform may not be appropriate for all scenarios.

Anyone with any pointers to some best practices or documentation in this space?

I was just reading this on MSDN. The author defines a contract as follows:
        •        A contract is everything about the behavior of the service that is engraved in code. They are drawn up at design time.
        •        Policies are everything that can be changed without coding. They are typically detailed later in the process and they are easier to change.

I agree with everything said here, except the use of the term contract. When I’ve heard discussions about service contracts, it goes beyond the functional interface as this article implies. Service contracts are by nature, defined not just by the functional aspects, but the non-functional aspects as well. Think about legal contracts in the real world. They not only say what is going to be done, but they say how it will be done, what the stipulations are if something is done incorrectly, etc. Applying that to system interaction, my tendency would be to rephrase the above definition as follows:
        •        The interface is everything about the behavior of the service that is engraved in code. They are drawn up at design time.
        •        Policies are everything that can be changed without coding. They are typically detailed later in the process and they are easier to change.
        •        The sum of the interface and the policies that dictate its usage constitute the service contract between a service consumer and a service provider.

All of this aside, this is a very good article detailing the importance of the interface, and the challenges associated with taking either a schema first approach or a code first approach to service creation. While I may not agree with his use of the term contract, I do agree with the fundamentals of the discussion.

Miko Matsumura writes in his blog:

I think as SOA “Crosses the chasm”, there’s a need for more people to think in terms of the architecture side. The architecture side tends to better keep an eye out for the bigger organizational goals. If SOA is truly to meet a larger vision including:
1) Aligning business and IT
2) Making IT strategic and a key part of competitive advantage
3) Enhancing business agility
4) Creating dramatic new forms of resource reuse
then folks need to be mindful of the architecture, the “A” side.

I agree with this completely. A favorite line of mine is that the “A” in SOA stands for architecture, not application. SOA is all about thinking outside of the box, that is, looking beyond the boundaries of the problem/project at hand. That’s something that doesn’t come easy.

At the same time, we can’t forget about the “S” either. The best solution will leverage the tactical “S” opportunities at hand to achieve the strategic “A” direction. There are risks in going too far in either direction. I consider myself more of “big picture” thinker and have always found that the most successful projects I’ve been involved with have had someone who was very detail oriented working with me. The “S” and the “A” must complement each other to achieve success.

While I won’t be there for this, I think Jon Udell has got some very interesting topics for the discussion, outlined in his blog here and here. For day one, a topic listed is “migration to policy-driven intermediation.” He mentions security and compliance policies, but I’d also add in operational (i.e. routing rules, caching, monitoring) and what I term integration (transformation, especially in support of versioning) policies. While externalizing these policies is problem one, what will quickly follow is a big management problem. Right now, policy management is typically provided as part of an enforcement solution. When those policies start to branch out beyond security, can the management solution still be sold as part of the enforcement solution? Is that what we want? Ultimately, we’d want the policy owners to be the ones setting these policies which could mean at least four departments: Information Security, Compliance, Operations, and the development team. How many tools currently support this scenario well?

He also suggests some topics around the “evolving role of the repository” including the human factors. I think this will be an interesting discussion as we’re starting to see some convergence between the human-oriented tools from vendors like Flashline and LogicLibrary and the system-facing UDDI registries from vendors like Systinet and Infravio.

One topic that I’ve been interested to hear about (I keep meaning to call David Linthicum’s voicemail line for his <a href=”http://www.soaexpertpodcast.com/”>podcast</a> on this one) is how to make Web Services with a “pull” invocation model a first class citizen. A service is a function that is performed on behalf of some external party. There are really two styles for this: a push model, where the consumer “pushes” the request to the service provider directly (easily implemented as SOAP/HTTP(S) and may or may not involve a response message) and a pull model, where the service provider “pulls” the request from some form of queueing system, i.e. JMS, MQ Series, or MSMQ. Think of this as the difference between a session bean and a message-driven bean in the J2EE world. The problem with the “pull” model is that it’s not a first class citizen in the Web Services world. This isn’t a notification scenario, so WS-Notification isn’t appropriate. Notifications tells you something happened, but the source of the notification has no expectation that some action will be formed.

The current state-of-the-art for handling this would be to use some form of intermediary (ESB, WSM broker, etc.) to perform HTTP-JMS bridging. Until we have an open wire protocol like HTTP for talking to queueing systems, that’s about all we can hope for. The problem with that is we now move out of the Web Services and into the MOM world. Depending on the implementation platform (especially if it’s not from the same vendor as your messaging backbone), the fact that the payload is a SOAP envelope may be completely invisible to the platform framework, resulting in a lot more work for the service developer. This topic doesn’t really fit with John’s other topics, but it’s of particular interest to me, so if you’ve got thoughts on this, blog away.

Overall, I think this forum should be very interesting. I look forward to following the blogs from it.

While I certainly haven’t been an active blog poster, I do have a lot to say around SOA and Web Services and now I’ve got my chance. I’ll be speaking at the ITPQ (IT Productivity & Quality) Conference on SOA and Web Services Best Practices in Chicago on October 18th and 19th. Here’s a link to a conference brochure. If anyone out there in the blogosphere will be there, let me know and I’d be happy to sit down and chat with you.

Aaron Skonnard, a proponent of contract first development, recently posted on the disconnect between what the vendors are doing and what developers want. He makes the claim that vendors don’t see the value of the contract-first approach. On this statement, I disagree. I don’t think the problem is that the vendors don’t see the virtues of it, I think it’s more the case that it is such a conceptual shift, that it’s a very difficult problem to solve when catering to an audience of C#, VB.NET, or Java coders. This is further justified by the fact that tools like BizTalk, as he mentions, make it so easy to do contract first development. The key is that BizTalk and most tools that fall into the Business Process Orchestration space are designed from the ground up for schema-oriented development. This makes working with XML messages and XSD much more natural. C# and Java are clearly for object-oriented development, so there’s a natural disconnect. There needs to be a bridge between the two worlds, and unfortunately, it began with an effort around traditional RPC mechanisms and as a result, wound up with hard, restrictive bindings between XSD and a C# or Java class. In my opinion, the toolsets need to bring some schema-oriented approaches to the tool sets for C# and Java. This usually involves code generation, and tools that do this have had a hard time gaining adoption. It also brings up memories of the whole object-relational mapping space, so the vendors are proceeding with caution. I expect that in 5 years we still won’t see advances in this area, but we’ll see the majority of services built using tools like BizTalk. For services that still need to be written in Java or C#, the developer will still have to deal with things using a DOM object, just as a database developer still often works with a RowSet. These will be relegated to more complex operations where low-level DOM access is a requirement.

Phil Windley recently asked in his blog, “Is WS-MetadataExchange really necessary?” I would argue that it is, and is actually one of the cornerstones of really making system integration easier. Why? Well, someone else posted in the comments that we should focus on other standards, like WS-Security. WS-Security is exactly the reason why we need WS-MetadataExchange. Suppose I put up a web service and lock it down. I’ve chosen to support SAML assertions embedded within WS-Security headers. I advertise this service out, but only provide a reference to the WSDL. How will any client application know that WS-Security headers are required, and that only SAML assertions should be sent? Wouldn’t it be great if a client system only needed to get the WSDL file, and then through metadata exchanges at the time the service is invoked, the client endpoint would first ask for WS-Policy information, and find out that the service is secured and requires SAML? While that may not guarantee access if the right assertions don’t exist, the systems would have at least included the proper SOAP headers rather than rejecting the request for not having enough information. This is even more important as the policies around the service change. Suppose I now need to require that some data in the message be encrypted. While the functional interface doesn’t change, the policies associated with the message interchange have. With WS-Mex, the runtime engines would pick up the new policy and make the appropriate changes without having to rebuild the Web Services client.

If we want to move toward policy-driven computing, and enable policies to be quickly enacted without requiring a code/build/deploy cycle, WS-Mex and WS-Policy are necessary.