Todd Biske: Outside the Box

A previous post discussed how no two organizations seem to define the role of the architect the same way. It just occurred to me that the same thing holds true for another common deliverable of the enterprise architect: the reference architecture. I’ve had the opportunity to work on a few reference architectures with a few different companies and I can certainly say that no two were the same, although there was certainly commonality between them.

There are at least two questions that have come up in multiple efforts. The first is whether or not a reference architecture should recommend specific technologies (e.g. “if you meet these conditions, you should use vendor product MagicFooBar”) or only describe the capabilities required, leaving implementations teams to choose a technology that fits (e.g. “the services tier must be exposed using XML-based interfaces”). The second question is one of depth. How much guidance should a reference architecture give? Should it go so far as to place constraints on how individual classes are named, or how the file hierarchy is set up in the source code repository? Or should it remain at a level somewhere above the design of individual classes? It may seem simple at first glance, but I can speak from experience that once you start providing some guidance, it’s very easy to get pulled into the depths.

What’s the right answer to these questions? I don’t think there is one. Why? Because reference architecture is about guidance, and the guidance needed in any one particular organization is going to be dependent on the skills of the staff, the organizational structure, the technologies involved, the problems being solved, etc. So what do you do? In my opinion, a good course of action is to focus on delivering guidance that is singular in purpose, at least within a single deliverable. What does this mean? It means that you should avoid trying to answer all of the questions within a single (and likely very large) document. Rather, each deliverable should start with one simple question, and focus on answering that question clearly. For example, rather than having one big reference architecture that attempts to cover all possible business solutions which could easily include web UIs, desktop UIs, mobile UIs, workflows, services, automated processes, applications portals, content portals, collaboration portals, and much more, consider having a separate document for each one, forming essentially a hierarchy of documents. The earlier documents discusses things more broadly, intending to answer the question of what collection of technologies are needed for a solution, but not necessarily guidance on the appropriate way to use that technology. For that, once the decision to use a particular technology has been made, a separate document goes into added depth, and the process continues. The reference material will eventually work its way down to development and operational guidelines (or hook up with those that may already exist).

Of course, all of this is easier said than done. It’s not easy to determine the “right” hierarchy up front. Again, the litmus test to apply is whether the document has clarity and singularity in purpose. If you find yourself with a document that starts getting onerous to use, consider breaking it apart.

What are others thoughts on this? As always, this is simply my opinion (after all, this is my blog, so that’s what you’re going to get), but I also am always interested in the thoughts of others and striving for a more collective wisdom. What has been successful for you when trying to provide reference guidance to teams? What hasn’t been successful? Use comments or trackback.

Update: I was just checking my feeds and there were two others blogs that discussed reference architectures: this one from George Ambler and this one from Simon Brown. Enjoy.

Wow, I was very surprised to come into work to find out that Oracle had agreed to acquire BEA and that Sun had agreed to acquire MySQL. Based on the reaction of many in the blogosophere, like Miko Matsumura whose Facebook status said he was “amazed by sun buying MySQL and yawning at Oracle buying BEA” not many people were surprised by the Oracle/BEA deal. I actually was a little bit surprised, simply because it felt like BEA would stick it out as an independent, and it didn’t appear that Oracle was taking its “we will get what we want no matter how long it takes” approach as it did with PeopleSoft. That’s about all the comment I’ll make on this as I try to avoid too much speculation in the vendor space in this blog, but this was too big of an event to not mention. While I chose not to do 2008 predictions this year, all of those people who picked the no-brainer of more vendor acquisitions in the SOA space can check this one off your list only 16 days into the year.

The other acquisition was one that was very surprising in so much as not many people probably saw it coming. While not in the SOA space, it certainly shows that open source is becoming more and more ingrained in Sun. This is one that’s going to take some time to digest in the blogosphere to see what people will think about it. I think being associated with a bigger name can only help MySQL. As for the benefits to Sun, we’ll see. I’ll be interested to see what others have to say about it.

Phil Windley couldn’t have summed up my own thinking better:

In all the years I’ve owned mobile phones, not one ever got better as it aged. The iPhone has gotten better three times now and promises to do so in the future.

My upgrade was seamless and two new features were ones that have a significant impact on usability. The first was the pseudo-GPS feature. I was very surprised that it worked from my house Where I was connected via WiFi, since my house isn’t a public hot spot. Is it GPS? No. Is it good enough? Absolutely. The second feature is the ability to have web sites show up as buttons on the home page. There are three sites that I visit regularly on my iPhone: Gyminee, SportsTap, and Facebook. While navigating through bookmarks wasn’t that difficult, navigating to them with at most two touches (home button and an icon tap) is superb. It was only about two weeks ago that the thought of having buttons on the home page for web sites occurred to me, and now great that Apple’s provided it.

The only thing I’d still like to have is a native (not web-based) instant messaging client. I rarely use SMS, so multiuser SMS isn’t a big deal for me. With the SDK due out in a month, I’m sure an “approved” IM client will appear in short order. There’s only two reasons I can think of for why it doesn’t exist. First, SMS generates revenue for AT&T, IM will not. Second, IM could be a drain on battery life. I don’t know the first thing about the IM protocols and how often it would require the phone to be sending data for an “always on” approach, but I do know that my old Motorola V360 had it, so it can’t possibly be that significant.

The other interesting announcement was the Apple TV Take Two. I’m hoping to get HD into my house in the spring, and with the new announcements, I’m now including AppleTV into my budget. I had previously said that the key to success would be adding video rentals, and it’s now happened. I’ll admit that I was a bit concerned about the 24 hour rental time limit, but the caveat that it’s 24 hours from when you start watching it helps a bit. I’d still like to see this longer, as I’ve watched movies in 30-minute segments during my daily workout, but at least I can rent 2 movies before a trip somewhere and watch one on the flight out and one on the flight back. I still think it would be cool if Apple would embed the Apple TV into an actual TV, but that’s a pretty crowded market right now. I also wish they would have lowered the price of the 160GB model. But, they’ve made just enough changes to convince me to get one in the near future.

One thing that I have noticed over the past few years of my career is that no two companies define the role of the architect the same way. Personally, I view this as neither good nor bad, and this isn’t going to be a post on the whole architect certification efforts that exist out there. While it may make it more challenging to find qualified candidates when the definition of the role changes from company to company, once you’re in a company, clarity of the specific responsibilities at that company are certainly more important than whether or not you’re doing the same work as Joe or Jane Architect down the street. As another aside, I also think that those job responsibilities should serve as a guide, but not a barrier. All organizations require some flexibility in what people do if they expect to get things done versus devolving into a finger-pointing episode of people saying, “That’s not my responsibility.” Which brings us to the first type of role.

The Gluer

In this role, the person with the lucky fortune to have the architect title is expecting to hold everything together from a technical perspective. A typical software development project involves far more technical tasks than just writing code. Servers may need to be built, software installed, routers configured, etc. Put simply, the person playing this role was expected to be the glue that holds everything together from a technical perspective. You’re probably asking, “Isn’t this the job of the technical lead?” Well, the problem with the technical lead is that in most organizations I’ve seen, it was purely a role, and not a job title. Therefore, it was difficult to actually know who the technical leads were in an organization, get coverage across all projects, etc. Titles like senior developer or lead developer don’t cut it, because the emphasis is still on development, and not the other tasks involved with getting something into production. Unfortunately, giving them the title of architect is problematic. While it does allow an organization to establish some clear technical leadership, it’s likely that the individuals with this title will be so consumed with tactical project decisions, that very little of their time will actually be spent on architecture.

The Scheduler

While I never would have guessed this one, I heard it mentioned at two different organizations that architects are supposed to act as project managers for technical activities, normally working in conjunction with the “real” project manager. While I do think that all leaders should have some ability to do basic project management activities. That is, break down a goal into some set of constituent tasks on a timeline. There are certainly ties back to the gluer role, as this essentially takes the technical leadership a step further. In addition to identifying all of the technical activities, the person now has to manage all of them, rather than delegating this back to the project manager. Of all the roles, this one is the least desirable to me because I don’t consider project management a strong point for me. I’m admittedly a big picture thinker. Most good project managers I know are extremely detail-oriented. The best working scenario for me personally, is not to have me be the project manager, but work side-by-side with the project manager.

You’ll notice that both of these roles are very project-specific. If all of the architects in your organization are on project assignments, that’s a problem. Project, or solution, architecture is important, but you also need architects outside of projects, hence the next two roles.

The Decision Maker

This architect is normally not assigned to projects, but is still involved with the decision making process within projects. This person is likely viewed as the top of the technical hierarchy within some organizational or technical domain. If it’s associated with software development, I typically see it following organizational boundaries (e.g. an architect for all solutions developed by one development organizationy), while on some of the other activities, it follows technical domains, such as a Security Technology Architect, a Networking and Communications Technology Architect, etc. Projects go to these architects when decisions need to be made or approved. The challenge I’ve seen with this role is the flood gate, especially when the title is first established. Many organizations don’t have a technical hierarchy in their organization, and as a result, it can be unclear as to who has the technical decision making responsibilities. As a result, when someone is granted the title, the flood gates open, and every technical decision, even those that should be no brainers, wind up coming to those deemed “architects.” The challenge here is that the architect winds up having all their time consuming by decision making, with no time left over for establishing a strategy and direction.

The Strategist/Policy Maker

At the opposite end of the spectrum from the decision maker. Architects acting in this capacity are the legislative branch of the government, focused on established reference architectures, policies, roadmaps, etc. I thought about breaking this into two roles, because there are plenty of architects who don’t do strategy, but in general, the perception of the enterprise is similar. There’s a significant risk of becoming an ivory tower in this role. Just as the decision maker gets sucked back into project activities, the strategist can become disconnected from the reality of projects.

So what’s right? Personally, I think an organization needs gluers, decision makers, and strategists. We already have project managers, and as I previously stated, I don’t think we need to break out “technical” project management as a separate discipline. Should the remaining roles all have the “architect” title? In my mind, there’s really only debate about one role, the gluer. Clearly, not all of the activities associated with technical leadership have something to do with architecture. At the same time, however, if it’s not clear whose responsibility it is, these technical concerns will bubble their way up to the “decision makers.” If it’s necessary to bless that role with a title like “Solution Architect” to avoid this scenario, then do it.

What other roles and responsibilities have others seen with architecture? What’s missing from the list?

The IT as a service provider discussion was brought back up by Joe McKendrick of ZDNet in this blog. In it, Joe made reference to a past blog of mine in which I stated my opinion that a customer/supplier relationship between IT and their end users in the business was a bad thing, and I still believe that. Joe’s post brought to light some small nuances on that opinion that need clarification.

In my original post, I stated that IT moving solely to a supplier model to the business is an invitation to be outsourced. If you’re simply an order taker, there’s no reason that someone else can’t take those orders. The value-add that an internal IT department provides is not technology expertise, but technology expertise combined with knowledge of the company. Any SaaS provider or outsourcing agency must provide services to a mass market, or at least a market with more than one customer.

Getting back to Joe’s post, the inference that could be made is that IT shouldn’t be a service provider. This is not the case however. The IT-Business relationship should be a partnership, but you can’t be a partner if you’re not providing good service. Understanding the services that you bring to the table and doing them well is critical to the relationship. The difference is that those services do not define the boundaries of the relationship. Instead, they bring structure and foundation to it, on which partnership can be built. If your foundation is weak, the relationship will crumble. Therefore, adopting principles of service management within IT is a good thing, however, don’t approach it from the standpoint of competing against outside service providers. The decision to use outside providers should be made by the business, which includes IT. IT should be the one driving the discussion to say, “some aspects of our technology are really becoming commoditized and we can achieve some significant cost benefits through an external provider” rather than being told, “you’re a commodity and we’ve outsourced you.” In this sense, IT is no different than other business support organizations. Take HR as a good example. One could certainly argue that HR could be outsourced as it provides commodity services that all companies need. Every large company I’ve been at still has an HR department, though. What is more the norm is to have HR working as part of the business to make good business decisions on what aspects of HR to outsource, and what aspects of HR should remain within the company because there is value add. Only someone within the company can really understand the corporate culture which is critical to attracting and retaining talented individuals.

Be a partner in your business, but ensure that your partnership is on a solid foundation.

I received a couple comments on my previous post, and rather than respond in the comments themselves, I thought I’d use another blog post to address them since I don’t think many people subscribe to the comments feed. Before I get into them, I also wanted to call out this blog post from Anne Thomas Manes of the Burton Group. Out of a discussion on the Yahoo SOA Group about the relationship between 3-tier and SOA, she posted this entry which included this statement:

I also expect that the concept of “application” is likely to go away. Why is it that we as systems users have to be constrained by the limits of this artificial boundary called an application? Why do I have to shift my focus among multiple applications? Why do I have to copy data from one application to another? Why do I have to launch this stupid browser or that stupid application to get my work done? Why isn’t everything just accessible to me from my desktop (via widgets and gadgets) or from within my preferred operating context (e.g., email)?

It was this that prompted me to put together my original post, since I couldn’t find an entry in my blog that was specifically dedicated to this topic, although I had mentioned it as part of other entries. I had meant to include a link to Anne’s post in my first entry and forgot.

The first comment came from Brian “Bex” Huff who stated:

People understand the term “application,” but the word “solution” is a bit too nebulous.
Applications stand alone… what good is a widget to view an Excel doc, without the Excel application to create the doc in the first place?
I agree that IT should always *think* in terms of dynamic, evolving “solutions”… but the basic building blocks still include “applications”… as well as toolkits, frameworks, libraries, etc.

Bex actually made a statement that is indicative of the current culture, which was “Applications stand alone.” My opinion is that applications shouldn’t stand alone. Why shouldn’t we have the ability to present a UI component that can manipulate spreadsheets anywhere? Yes, there will always be cases where spreadsheet manipulation is the only thing we want to do, but there’s also plenty of cases where embedded spreadsheet manipulation would be better for users. What will enable this is thinking in terms of capabilities, rather than in terms of applications. My opinion is that an application is the result of packaging, rather than a unit of composition.

The second comment came from Rob Eamon. He wrote:

There will always be a collection of components/services that have been designed to work together to provide some set of functionality. And there will always be multiple sets of these. And there will always be a need for these sets to interact, possibly through mediation.
Disaggregating application components and making them independently accessible in many contexts seems very appealing.
But take the issues that the typical application development/support team faces and blow that up to the enterprise level. The fragile nature of such an approach will inherently stop it from becoming a reality.
IMO, the application is still a useful and reasonable building block and allows us to break down the enterprise solution space into manageable chunks.
Some of the application components might be useful as independent entities in a larger setting, but I donít think approaching everything that way is a wise course. IMO, it would inhibit flexibility rather than promote (counter-intuitive such that may be).

I wish Rob would have went into more detail on the “issues” that the typical application development/support team faces, because I can only guess what they may be. My first guess is that application teams currently have to deal with ever changing requirements and needs. If you increase the number of parts, you now have smaller components with new relationships associated with their use, and if we don’t manage it well, chaos will ensue. It should be noted that I’ve never said that this type of shift would be easy, and if anything I’d say it’s going to incredibly difficult. I’ve reflected on this in the past, specifically in this post, and wonder if it will take some company approach IT like this from the beginning, without any baggage to create the impetus to change.

Rob went on to state that in his opinion, the application is still a useful and reasonable building block. Here’s where I disagree, for the same reasons I used in Bex’s comments. An application is typically a “packaging” exercise, and those packages aren’t what we want for composition. The only part of an application that still has significant potential for being a “stand-alone” entity is the UI. I’d be happy to see an IT organization that makes an organizational/funding separation between development of UI code from development of services that are used by the presentation tier, much as Jeff Schneider suggested in this post from early last year.

Where I’ll agree with Rob is that this is not a change for the weak of heart. If a new CIO came in and reorganized the organization along these lines, the chaos that might ensue could certainly result in the perception that IT is even less agile than they were before. So, this isn’t a change that will occur in a year. This is a gradual, evolutionary change that will take time, but it will only happen if we’re committed to making it happen. I think a key to that is to get away from the application mindset.

Thanks to Rob & Bex for the great comments, and I’d love to hear from others whether you agree or disagree with my opinions.

I was very surprised that in ZapThink’s 2008 predictions, they predicted that they will be acquired:

Our prediction for 2008 is that one of these firms will acquire ZapThink, as well as other SOA thought leadership firms, because we can establish the winning acquirer as a global SOA leader

I wasn’t so much surprised that ZapThink would eventually be acquired, but I was surprised at the public announcement. Now, I’ve never been an entrepreneur, have never been involved with a startup, and have never gone through an acquisition, but a statement like this in an open forum sent out to ZapThink’s mailing list (beyond paying ZapThink clients) would seem to indicate one of two things:

1. An acquisition is already in the works.
2. The company is struggling and has just raised a huge banner saying, “We’re for sale. Please come save us.”

I’d be very surprised if it was the latter case, as a public statement like this would seem to kill any negotiation position. So, I’m going to assume that the former is the case and that it won’t be long before this “prediction” is reality.

It certainly brings up an interesting topic for companies in the SOA consulting space, one area where I do have past experience. Obviously, consulting companies make money by having billable resources. If you’re a consultant and you’re not billable, the revenue is not coming in to pay your salary. The owners of the company ultimately want your salary to be paid by clients, not by them. At the same time, this makes the development of intellectual property very difficult when the consultants are 100% billable. While intellectual property is certainly an indirect source of revenue, as it can help close deals, it’s not a direct source of revenue. So what’s a consulting firm to do? It would seem that bringing in a set of experts in intellectual property development that also have experience in creating non-consulting revenue streams (training, vendor marketing) could be a very potent combination that gets around the revenue challenge. The risk here, however, is that the “vendor neutrality” that an independent ZapThink has provided becomes challenging, given that many consulting firms get deals through their vendor partnerships, and it’s difficult to be a partner to competing vendors. Even ZapThink themselves have recognized the challenges of consulting and vendor relationships in Dave Linthicum’s recent post on his InfoWorld blog. We’ll just wait and see what the future holds for ZapThink. I wish my friends Ron, Jason, and Dave all the best in 2008.

Normally I’m not the guy to stir up a lot of controversy, but there’s one topic that I’ve mentioned on mailing lists, here, etc. that usually attracts a few opposing comments. That topic is the use of the term application in the enterprise. I’d like to see that term go away, or at least be limited to just the presentation layer of a solution. Why is that? It’s because it’s the boat anchor of IT, in my opinion. Think about it. How do we get out of the habit of thinking about monolithic point solutions for one small user group and into the habit of thinking about business capabilities that have potential for broader applicability when we have one hundred-plus organizations with the title “application development?” How do we deal with the organizational concerns when applications that were built for one group with one intent are now suddenly critical components for many other solutions with different user bases, each with their own set of priorities?

If nothing else, a switch from using the term “application” to the term “solution” would be a symbolic gesture that represents the change that must occur within IT. IT shouldn’t be producing applications, we should be producing solutions. Those solutions may still be self-contained like yesterday, or they may be a composite of pieces from many previous solutions, some new development, some off the shelf solutions, and some Internet-hosted offerings. The boundary of what constitutes “one application” are all but impossible to draw because of the many necessary interdependencies. Even if we limit the use of the term to just the presentation layer, it’s still debatable whether we should be using it. For example, are widgets on the OS X Dashboard applications? We don’t call them applications, we call them widgets. There’s some pretty sophisticated widgets, though. But wait a minute, won’t Word and Excel be around forever as applications? Well, the latest version of OS X introduced Quick Look, where with a press of the space bar, I can see the contents of an Excel spreadsheet without launching “an application.” The ability to present information in a tabular format is merely a capability. So, even in the sacred ground of the desktop, there are arguments that this notion of application launching may become less important to where the right thing simply happens.

I’m not a fool, and I certainly don’t think the use of the term application is going to go away in 2008, or anytime soon, but I do think that to continue the innovative use of technology, we need to make sure that the terms of the past aren’t locking us in to a single way of thinking about how computers are used. The continued need for better integration and more context-awareness in our technology solutions will only increase, and that pressure will continue to challenge any organization that is stuck with the application-centric view of technology as their boat anchor.

To all my readers, have a Merry Christmas and Happy New Year!

Last year, like many other pundits, I decided to have a go at some predictions for 2007. Let’s revisit them.

• Vendors: Surprise, surprise, the consolidation will continue. There aren’t too many niche SOA vendors left, and by the end of 2007 there will be fewer.
  Okay, so this was a no-brainer, but the devil is in the details. We had a very big deal in the SOA space between SoftwareAG and webMethods, and the deal that hasn’t happened between Oracle and BEA. There really wasn’t much that happened with the smaller SOA vendors (AmberPoint continues to chug along in their narrow niche), so maybe that category has already reached its threshold. There were far more dealings outside of the SOA space, clearly around business intelligence.
• Operational Management: At least one of the major Enterprise Systems Management providers will actually come out with a decent marketing effort around SOA management along with a product to back it up…
  My wishful thinking didn’t pan out. I’m still astonished at the lack of activity around this from the big players in Enterprise Systems Management. While I understand that management software is an extremely difficult sell, you’d think that a connection could be made into the visibility provided by an SOA management system and the growing interest and importance in business intelligence. Overall, the marketing message is still louder from companies like AmberPoint, SOA Software, and Progress Actional than from the big players.
• Registry/Repository: At least one players in the CMDB space will enter into the Registry/Repository space, most likely through acquisition. Thereís simply too much overlap for this not to occur.
  Once again, this didn’t pan out, however, in my conversations this past year I ran into far more people who now understand the relationship between CMDB and the SOA Registry/Repository, although I think some of the lower level marketing and sales people at the conferences need to educate themselves some more on this, because many of them didn’t get it. At best, they only understood a basic need to grab some list of services via UDDI from a registry.
• CMDB: At least one of the super-platform vendors will see the overlap between CMDB and Registry/Repository and begin to incorporate it into their offerings, either through partnership, or through an acquisition …
  I won’t be as harsh on myself on this one. While none of the super-platform vendors have done what I’ve said, the majority of them have begun to show the importance of the metadata repository in their overall infrastructure ecosystem. Whether it is Oslo, IBM’s WSRR, BEA’s ALRR, SoftwareAG’s CentraSite, or any of the others, expect to see more on this one.
• Events: … I think weíll see renewed interest in event description, as I see it as a critical tool for enabling agility. Services provide the functional aspect, but there needs to be a library of triggers for those functions. Those triggers are events. Along with this, the players in the Registry/Repository space will begin to market their ability to provide an event library just as they can provide a service library.
  I should have re-read my previous postings on the slow uptake of CEP and other event technologies, even though it stirred the pot with some CEP vendors. I haven’t seen much of anything on the event description front or any discussion of event libraries from the SOA Registry/Repository space.
• Service Development and Hosting: … While itís too early to proclaim the application server dead in 2007, I think the pendulum will begin to swing away from flexibility (i.e. general purpose servers that host things written in Java or C#) and toward specialization. A specialized service container for orchestration can provide a development environment targeted for orchestration with an execution engine targeted for that purpose …
  Thanks to Microsoft and Oslo, I think I can claim a winner on this one. The pace of adoption may be slower than the wording in my prediction, but I do think it’s safe to say that more companies are trying to leverage model-driven BPM-based technologies for development than last year. The key question is whether they’re seeing the success they desire or not. A subject for another blog entry is the current reality of model-driven development with today’s BPM tools.

Overall, I think my predictions are about typical of my thinking. I freely admit that I’m a forward thinker, and as a result, I’m usually guilty of thinking things will happen faster than they actually do. I haven’t decided whether or not to make 2008 predictions yet, and given that the bulk of my 2007 predictions are still off in the future, I don’t want to do a rehash of the same thing. We’ll see. If I feel inspired after seeing others publish their own predictions, perhaps I’ll do it again. One prediction I didn’t make was how many blog posts I’d have, and I was surprised to see that my 2007 predictions was post 103, and this post will be number 345. While it’s not as easy to find time to post now that I’m back in the corporate world, hopefully you found some of those 200+ posts enlightening and you’ll do the same in 2008. I’ve seen a pretty significant uptick in the number of subscribers over the year from FeedBurner, so I’m thankful for all of you continuing to read and sharing the links with others.

Just when you thought it was safe to build your SOA, the ESB has returned. A whitepaper from Paul Fremantle of WSO2 seems to have stirred the ESB pot once again, with a number of pundits chiming in on the future of the ESB. I was surprised at how little conversation there was about at the Gartner AADI Summit in comparison to the previous Gartner event I had attended two years prior, but that’s quickly changed.

I read Paul’s paper, and from my perspective, not much has changed from the views that I expressed two years ago back in a Burton Group Catalyst presentation and then in this followup blog post. The bulleted list of capabilities provided in that post aren’t ones that are going to go away, and it surprises me that there continues to be so much debate in this space.

Joe McKendrick recently summarized a number of recent comments in a blog entry, including ones from Roy Schulte of Gartner, Lief Davidson of IBM, ZDNet blogger Dana Gardner, and Lorraine Lawson of IT Business Edge (who has given a couple of my recent blogs some publicity, thanks for that). There were a couple things in Joe’s entry that just rubbed me the wrong way.

First is this notion of federated ESBs. Both Roy and Dana made comments around this, and I simply don’t agree. I’ll admit, there are a certain class of organizations that will have multiple ESBs. They’re the same organizations that have one of just about every technology because they’re so huge. Take them out of the equation, because I don’t believe they represent the masses. For the rest of us, what does “federated ESBs” mean? Does it mean that I have multiple ESBs performing redundant capabilities? Or does it mean that I partition up the capabilities across multiple products/devices, but yet with no two products providing the same capability, even though they may be capable of it? The latter is tolerable and likely, the former is not. For example, Roy mentioned security policies and quality of service. I may rely on an XML appliance for service security at the perimeter, and then rely on app server capabilities or some security agent within the data center. For simplicity sake, if we define quality of service as some form of intelligent routing and traffic shaping, I may rely on an ESB, a high-end XML appliance or network device, or advanced clustering capabilities of an app server. This type of partitioning makes sense. What doesn’t make sense is having one ESB (pick your favorite) providing security, QoS, etc. for services by development group A and having another ESB providing the same for services from development group B. To put it in context, routing is a core capability according to my definitions. Would you let development group A put a Cisco router in front of their services and development group B put a Juniper router in front of their services? You certainly wouldn’t do this if all of the services are hosted in the same data center. Yes, if your company has grown through acquisition and has lines of business all over the place, you may have different routers, but now we’re getting back to those super-large conglomerates I mentioned at the beginning. For those of you now envisioning the “big honking ESB,” don’t think that way. Think of it more like the network. I may have many deployments of a single ESB product, each handling a portion of the services in the enterprise, and their consumers know to direct to the appropriate ESB from design time rather than some uber-ESB. Applications point at specific databases or hostnames, there’s no reason that services can’t work the same way. Again, I don’t view this as federation.

What I believe we need to strive for in this space is centralized policy management and distributed enforcement, with standards based communication between the policy manager and the enforcement points (if only those standards existed). For example, one could make the argument that SAP could provide a second ESB that deals with services provided by the SAP infrastructure. I don’t view this as federation, however. From my perspective, I don’t even care whether SAP has an ESB or not. What I do care about is whether the entry points exposed by SAP can enforcement my QoS policies, my security policies, my versioning policies, etc. Let me centrally manage the policies, push them out, and have it just work. The challenge with this is not the enforcement architecture and federation there, but rather the metadata repository that will hold all of this policy information. This is where federation is important, because I may be getting third party products that come with their own pre-populated registry/repository of services that I need to manage.

The second statement that I disagreed with was Paul’s comment that ESBs discourage the shared ownership of services. If it does, then I think ESB ownership is the problem. Most web applications require the configuration of a load balancing farm before they can be accessed. No one considers the keeper of the farms in network operations the “owner” of those web applications, so why would the team responsible for the configuration of the ESB be considered the owner of services? Personally, I think a lot of this stems from ESBs being targeted at developers, rather than at operations teams. As I’ve commented in the past, I think the inclusion of service development capabilities like orchestration and the leftovers from the EAI space messed up the paradigm and caused confusion. Keep mediation separate from development.

So, as I climb down from my soap box, what are my parting words? I still believe that the capabilities in my post from last year are necessary, and an ESB is one way of providing those capabilities. Intermediaries are almost always used in web architectures, so there shouldn’t be such a strong aversion to having them as a front end to a service. That being said, too many intermediaries is a bad thing, because we haven’t gotten to the single pane of glass management that I think is necessary. Rather, each intermediary has its own management console, and the chances of something getting missed or fat-fingered goes up. Focus on an approach for centrally managing the policies, minimizing the number of places where policies are enforced, and keeping operational activities separated from service development activities.

Mike Kavis asks the question, “Why are you still generating reports?” In his blog, he states that “we should empower the users to create their own reports.” This brings up an interesting discussion. Anyone who has worked in IT for a few years know that there’s a significant amount of work that is done using Excel. Excel is the empowered user’s tool of choice. Is this a good thing or a bad thing? On the one hand, individual users are empowerd. On the other hand, is the analytics being performed of value to more than just that user? Could someone with a computational background perform those same analytics in a much more efficient manner?

The statement that Mike makes for which there is absolutely no argument against was this one:

When users ask for a report, the business analyst must ask the user, “What problem are you trying to solve?”

Ultimately, IT and the power user should be collaborating on what the best solution is. The user may have a need for ad hoc analytics, but there should also be a way that those analytics come back into the fold and are available for broader use. If the power user wants IT to simply get out of the way or IT simply wants to maintain tight controls on information, that’s a sign of an unhealthy relationship. Rather, both parties should be concerned with leveraging each other’s strengths to the fullest extent to create the best solution.

James Taylor (not the singer) posted a followup to Mike’s blog that makes similar points. He also suggests that we dig into the reasons behind the request for information. Given his focus on enterprise decision management, he suggests focusing on the decision that the person was trying to make, rather than on the information used to support that decision. The key similarity between my message, Mike’s, and James’ is that we need to go beyond the initial request and understand the purpose behind it. Odds are that there’s something else that IT can be helping out with.

James McGovern called out that he hasn’t seen much discussion on the topic of reference architectures, and called me out for my thoughts. I’m never one to pass up a good blog topic, especially when I don’t have to come up with on my own.

First, some background on my experience. My current job responsibilities include the development of reference architectures, my engagements with clients while I was a consultant all included the development of either a deliverable that included “reference architecture” in the title, or was clearly some form of reference material, and my job prior to consulting, included the development of reference architectures within the last 12 months of my time there. So, I’m no stranger to this space.

Reference architectures, and reference materials (since the need doesn’t stop at architecture) in general are an interesting beast. Personally, I view them as part of the overall governance process, mainly because they’re created to document a desirable pattern or approach that the authors would like (or will ensure that) others follow. At the same time, a document alone does not create governance, just as buying an SOA Registry/Repository doesn’t create SOA governance. Reference materials are a tool in the arsenal and the degree to which they are used is dependent on how you architects work with the end consumer of the reference material. Organizations are all over the spectrum on this. Some architects live in an ivory tower with virtually no interaction with teams on active projects, some architects are the exact opposite, with their time completely consumed by day-to-day project activities. Most organizations fall somewhere in between.

My opinion is that reference material is absolutely necessary, if nothing else but to prevent the organization from tribal operations. If none of the standards and guidelines ever get written down, and decisions are solely based on tribal knowlege, the organization can quickly break down into the haves and the have-nots. If you’re part of the tribe, you have the knowledge. If you’re not, all you can do is make your best guess until you have to show up to tribal council and get lambasted. Trying to gain the knowledge from the outside is a very difficult process.

The next question, however, is what information belongs in the reference material? Does it do any good to document something in the reference architecture that everyone should already know, or should you assume that no one knows anything, and document it all? The problem is that EA has limited resources, just like everyone else, so you have to give consideration to the bang for the buck in the reference material. Once again, what’s “right” is very dependent on the end consumer of the material (which is why having a consumer focus is important). If you have an organization of seasoned Java programmers, how much reference material is needed on developing good web applications? If you have an organization with lots of VB6 and COBOL developers, they may need lots of reference material on web applications. So, know your audience, and make sure that the reference material is relevant and valuable for them.

I had the opportunity recently to learn more about the role of internal audit in an organization. It was a very interesting and educational experience, and got me thinking a lot about the relationship between the two.

What’s the visual that comes to mind when we hear the word audit? People in the USA probably think of an audit by the Internal Revenue Service. They would also rather go to the dentist and have a root canal done without anestesthia than be audited. So, you can certainly argue that the internal auditors have their work cut out for them. The presenter pointed out, however, that the role of internal audit is changing with time. While a few years ago, they may have been viewed as a reactive police force, today, there’s a shift toward a proactive consulting organization. Rather than coming in after the fact and telling organizations whether they’re compliant or not, they’re now being asked at the beginning, “What do we need to do to make sure we’re compliant?”

There are strong parallels to what goes on in the world of enterprise architecture. First off, many organizations have the dreaded architectural review board, the reactive police force of architectural governance. Projects teams dread them. Somehow, we need to move from this model to the latter model where projects teams know they need to be architecturally compliant and are actively seeking out the input of enterprise architecture to ensure this is the case from the beginning.

Unfortunately, the challenge for Enterprise Architecture is that there is no corporate mandate for EA in the same way that there is for Internal Audit. While I personally thought David Linthicum’s posts on EA as a corporate responsibility were a big stretch, you could certainly argue that if enterprise architecture was a corporate responsibility in the same way as Sarbanes-Oxley, then there would be no debate on whether an organization needed Enterprise Architecture. I found it very sad that at the Gartner EA Summit closing session, when Gartner posted a predication that 40% of EA programs will be stopped by 2012, about 40% of the audience agreed. Note that prediction didn’t say “changed” or “restarted,” it said “stopped.” A publicly listed organization on the NYSE can’t stop the Internal Audit program, it is required.

Overall, my takeaway from this session was that EA and Internal Audit need to be best friends. If Internal Audit has an IT audit group, which most do, it needs to be working closely with the EA group, as both are providing governance. In one of my panel discussions at the Gartner event, I made the comment that EA is certainly about governance. It could be argued that EA activities are basically centered around two major activities: strategic planning and governance. While Internal Audit probably has less of a role in strategic planning (except where governance issues are necesseary), clearly, there’s significant overlap in the governance function. Determine how both groups can work together to ensure that projects aren’t bombarded with governance from multiple groups. The view of the governed is already very negative, we need to do what we can to change that view.

James Urquhart brought to my attention the public review of SOA Patterns, as authored in the forthcoming book, “SOA Design Patterns,” by Thomas Erl. You can see the press release from Prentice-Hall here.

My first reaction when I received the email, prior to visiting the SOAPatterns.org site was one of skepticism. While I think patterns can add a lot of value, the immediate problem I saw stems from the fact that I’m very much a believer in business-driven SOA. In order to reach a broader audience across multiple verticals, you have to be more business agnostic. As we get more business agnostic, we naturally move deeper into the technology stack, and things at that level of granularity may not be the best service candidates, although they may be great candidates for reusable frameworks. If we’re talking about patterns inside of the service implementation, then we’re really talking about general design patterns, building on the original work of the Gang of Four, not really SOA Design Patterns.

So, with my bias set, I visited the web site. The first thing I hoped to see was some classification by business industries, such as “SOA Patterns for Insurance” or “SOA Patterns for Health Care” but I didn’t find them. Bummer, but I also didn’t expect this. Something like that would be of significant value as intellectual property to a consulting firm, and I think they’d make a lot more money keeping it to themselves and leveraging it on their engagements. What was on the site was four chapters: Basic Service Inventory Design Pattern Language, Architectural Design Patterns, Basic Service Design Pattern Language, and Service Design Patterns.< ?)>

In looking at the first chapter, Basic Service Inventory Design Pattern Language, my first reaction was again one of skepticism. The first page begins with “Inventory Context Design Patterns,” “Inventory Boundary Design Patterns,” “Inventory Structure Design Patters,” and “Inventory Standardization Design Patterns.” It also introducted a phrase- “service inventory architecture” -which I had never heard before. Looking at this page, nothing was making a connection with me. As I drilled into each section, I did find some goodness, but it could be argued that what is really being presented in this section is really a description of a step in a methodology, rather than a pattern. For example, one pattern listed is the “Enterprise Inventory” pattern, which lists the problem as:

Delivering services independently via different project teams across an enterprise establishes a constant risk of producing inconsistent service and architecture implementations, compromising recomposition opportunities.

The solution is:

Services for multiple solutions can be designed for delivery within a standardized, enterprise-wide inventory architecture wherein they can be freely and repeatedly recomposed.

This doesn’t feel like a “pattern” to me, but it’s certainly something that should be done. I don’t think anyone would argue that having an enterprise service inventory is a bad thing. Another pattern I looked at was the “Vendor-Agnostic Context” pattern. Again, what was presented in the “pattern” was goodness, however, it felt more like a principle rather than a pattern. This particular one did do a good job in demonstrating how this principle does lead to the use of specific techniques, such as leveraging the “Canonical Protocol” and “Decoupled Contract” patterns.

Overall, what did I think? Well, it certainly didn’t meet my original hope of seeing industry-specific business patterns. By that, I mean I didn’t find something that said “Order to Cash” with guidance on the types of services that should make up that process. I didn’t find something that said, “here are services that all organizations with an HR department should have.” Nothing business-specific whatsoever. Of course, I didn’t expect to see this, it’s just what I was hoping to see, just as I hoped the (defunct?) SOA Blueprints effort from OASIS a couple years ago might produce something along these lines.

Putting that bias aside, the more I dug into the site, the more I found things that provided good guidance, even though I’d say the use of the “pattern” moniker was a bit liberal. If you want to get an idea of what principles and factors should be considered in creating good services, versus just slapping WSDL or XML schemas in front of some existing logic, there’s a lot of good material here that is freely available. While some of the earlier pages read too much like a college textbook, a couple drilldowns brought me to things that were applicable to my daily work and made sense. So, based on that, I would recommend that people at least visit the patterns site, drill down into it, and see what nuggets you can leverage in providence guidance to your service development teams. If you really like it, then perhaps Thomas’ book can become part of your standard library for your developers.