Todd Biske: Outside the Box

Scott Mark posted some comments on his blog about ESBs versus Smart Routers. This is one of my favorite subjects, and I posted some extensive comments on Scott’s blog. I wanted to go a bit further on the subject, so I decided to post my own entry on the subject.

First off, full disclosure. My personal preference is toward appliances. That being said, I don’t consider myself an ESB basher, either. If there’s one takeaway from this that you get, it should be that the selection of a product for intermediary capabilities is going to be different for every organization, based upon their culture and who does what. I spoke with a colleague at a conference last June who had both an appliance and software-based broker, each performing distinct functions, largely due to organizational responsibilities. While both products could have provided all of the capabilities on their own, there were two different operational teams responsible for subsets of the capabilities.

Let’s start with the capabilities. This is the most important step, because as soon as you throw ESB’s into the mix, the list of capabilities can become volatile. The area of concern that I believe is the right set of capabilities are those than belong “in the middle.” The core principle guiding what belongs “in the middle” are things that developers shouldn’t be concerned about. Developers should be concerned about business logic. Developers should not have to code in security (in the Authentication/Authorization sense), routing, monitoring, caching. These are things that should be externalized. J2EE and .NET try to do this, but they still make the policies part of the developers domain, either by bundling into an archive file or by using annotations in source code. In a Web Services world, these capabilities can be completely externalized from the execution container. My list of capabilities include:

• Routing
• Load Balancing
• Failover
• Monitoring and Metrics
• Caching
• Alerting
• Traffic Optimization
• Transport Mapping
• Standards Mediation (e.g. WS-Security 1.0 to WS-Security 1.1)
• Credential Mapping / Mediation
• Authorization
• Authentication
• Encryption
• Digital Signing
• Firewall
• Auditing
• Content-Based behavior, including version management
• Transformation (not Translation)

Note that one area commonly associated with ESBs these days is missing from my list: Orchestration. I don’t view orchestration as an “in the middle” capability. Orchestration does represent an externalization of process, just as we are externalizing these capabilities, however, an orchestration engine is an endpoint. It initiates new requests in response to events or incoming service requests. Adapter-based integration ala EAI is also not on my list. Many cases fall into the category of translation, not transformation. There is business logic associated with translating between an incoming request, and one or more requests that need to be directed at the system being “adapted.” The grey areas that are tougher to draw a line include queueing via MOM and basic transformation capabilities.

Now, if we take this list, you’ll see that it can be covered by products from three major areas: Networking and Security (normally appliances), Software Infrastructure (ESBs, Application Servers, EAI/BPM), and to a lesser extent, Systems Management (most management systems employ an agent based architecture. Most Web Services Management providers include gateways and agents that perform these). My own tracking and discussion with various vendors that deal in this space yields a diagram like this:

As you can see, these three areas overlap. There’s a convergence of capabilities in the middle, but not complete overlap. Also, note that this is more about the target space of the product versus the form factor. There are some surprises on the diagram. SOA Software’s Network Director (formerly BlueTitan) and perhaps IBM’s WebSphere XD are much more like network devices than software infrastructure, even though they are software based.

So how do you choose one of these providers? First off, I believe all of the products that control the capabilities I listed should be policy-driven, utilizing distributed enforcement points with centralized management. The notion of policy-driven is extremely important, because we need to externalize policy management from policy enforcement. Why? If you look at those capabilities, who sets the policies associated with them? It likely should be multiple groups. Information Security may handle authorization and authentication policies, Compliance may handle auditing, Operations will handle routing, and the development team may specify transformations in support of versioning. Right now, the management tools are all tightly coupled to the enforcement points. Therefore, if you choose an ESB for all capabilities, your Information Security team may need to use Eclipse to set policies, and they may have visibility into other policy domains. This could be a recipe for disaster. You need to match the tools to the groups that will be using them. Software infrastructure tools will likely be targeted at developers. Network and Security tools will likely be targeted toward network and security operations, and may be in an appliance form factor. Systems Management will also be operations focused, but may have a stronger focus on monitoring and a weaker focus on active enforcement. It may involve the management of agents. Think about who in your organization will be using these tools and whether it fits their way of working. When evaluating products, see how far they’ve gone in externalizing policy. Can a policy be reused across multiple services via reference? Can policies be defined independent of a service pipeline? Is there clear separation of the policy domains? Do you have to be a developer to configure policies?

In short, I think if you nail down the capabilities, eliminating the grey area in integration and transformation, and have a clear idea of who in the organization will use the system, the product space should be much easier to navigate, for now. This convergence will continue, as smaller vendors get gobbled up and registry/repository becomes a more integrated part of the solution as a policy repository. It will begin to converge with the Configuration Management database space as I’ve previously discussed. Network devices continue to converge with MPLS and the addition of VoIP capabilities. If you have an intermediary today, continue to watch the space every 6 months to a year to see what’s happening and know when it might make sense to converge.

9 Responses to “Converging in the middle”