Todd Biske: Outside the Box

Archive for the ‘SOA’ Category

Chris Haddad of the Burton Group recently had a post entitled, “What is a service?” on their Application Platform Strategies blog. In it, he points out that “it is useful to first categorize services according to the level of interaction purpose.” He goes on to call out the following service categories: Infrastructure, Integration, Primitive (with a sub-category of Data), Business Application, and Composite.

First off, I agree with Chris that categorization can be useful. The immediate question, however, is useful for what? Chris didn’t call this out, and it’s absolutely critical to the success of the categorization. I’ve seen my fair share of unguided categorization efforts that failed to provide anything of lasting value, just a lot of debate that only showed there any many ways to slice and dice something.

As I’ve continued to think about this, I keep coming back to two simple goals that categorizations should address. The first is all about technology, and ensuring that is is used appropriately. Technologies for implementing a service include (but certainly aren’t limited to) Java, C#, BPEL, EII/MDM technologies, EAI technologies, and more. Within those, you also have decisions regarding REST, SOAP, frameworks, XML Schemas, etc. I like to use the term “architecturally significant” when discussing this. While I could have a huge number of categories, the key question is whether or not a particular category is of significance from an architectural standpoint. If the category doesn’t introduce any new architectural constraints, it’s not providing any value for this goal, and is simply generating unnecessary work.

The second goal is about boundaries and ownership. Just as important as proper technology utilization, and probably even more important as far as SOA is concerned, is establishing appropriate boundaries for the service to guide ownership decisions. A service has its own independent lifecycle from its consumers and the other services on which it depends. If you break down the functional domains of your problem in the wrong way, you can wind up making things far worse by pushing for loose coupling in areas where it isn’t needed, and tightly coupling things that shouldn’t be.

The problem that I see too frequently is that companies try to come up with one categorization that does both. Take the categories mentioned by Chris. One category is data services. My personal opinion is this is a technology category. Things falling into this category point toward the EII/MDM space. It doesn’t really help much in the ownership domain. He also mentions infrastructure services and business application services. I’d argue that these are about ownership rather than technology. There’s nothing saying that my infrastructure service can’t use Java, BPEL, or any other technology, so it’s clearly not providing guidance for technology selection. The same holds true for business application services.

When performing categorization, it’s human nature to try to pigeonhole things into one box. If you can’t do that, odds are you’re trying to do too much with your categories. Decisions on whether something is a business service or an infrastructure service are important for ownership. Decisions on whether something is an orchestration service or a primitive service are important for technology selection. These are two separate decisions that must be made by the solution architect, and as a result, should have separate categorizations that help guide those decisions to the appropriate answer.

Back in February, Jason Bloomberg of ZapThink posted a ZapFlash entitled “Competitive SOA.” I didn’t blog about it at the time, but this topic was brought back to the forefront of my mind by this post from Ian Thomas, with some follow-on commentary from Joe McKendrick.

While I’m not one to take one side or the other strongly, I must admit that I have significant reservations about a competition model, whether it is internal competition as suggested in the initial ZapThink article, or it is competition between IT and outside providers of services. First, let’s get the easy part of this out of the way. Part of Ian’s article is about simply running IT as a business and having good cost accounting. I’m certainly not going to argue about this. This being said, there’s a big difference between being a division or department of a company versus a supplier to a company.

I believe strongly that a customer/supplier relationship between IT and the end users of IT in the business is a bad thing, in most cases. If IT moves to exclusively to that model, the business leaders should clearly always be considering outsourcing IT completely. In doing so, it definitely sends a clear message that technology usage is not going to be a competitive advantage for this company. I believe that outsourcing can make sense for horizontal domains, where cost management is the most important concern.

The right model, in my opinion, is to have IT be part of the business, not a supplier to the business. To be part of a business, you need to be a partner, not a supplier. Brenda Michelson posted some excerpts from a Wall Street Journal interview with three CIO’s: Meg McCarthy of Aetna, Inc., Frank Modruson of Accenture Ltd., and Steve Squeri of American Express Co. Some great quotes from this:

Ms. McCarthy: At Aenta, the IT Organization is critical to enabling the implementation of our business strategy. I report to the chairman of our company and I am a member of the executive committee. In that capacity, I participate in all of the key business conversations/decisions that impact the company strategy and the technology strategy.

Mr. Squeri: I believe that over the next 10 years, the CIO will get more involved in the overall business strategy of the company and see their role expand in importance. The CIO will be increasingly called upon not only to translate business strategies into capabilities but to become even more forward-looking to determine what capabilities the business will need in the future.

The days of tech leaders as relationship managers and “order takers” will go by the wayside and they will be called upon to create and drive technology strategies that drive business capabilities.

It’s great to hear these leaders calling out how IT is becoming a partner, rather than a supplier. While our business leaders are certainly more tech-savvy than they have been in the past, there is still significant value in having people that specialize in technology adoption and utilization on your leadership team, just as you have people who specialize in sales, marketing, operations, etc.

Ian suggests letting “self interest flourish within the bounds set by the organisational context as long as it delivers cost-effective services but punish it by outsourcing where it doesn’t.” Cost reduction is just one factor in a complex decision. Holding the threat of outsourcing over IT may certainly in a more efficient operation, but applying those principles to areas where the decision shouldn’t be based on cost efficiency, but strategic impact to the business is a risky proposition. Let the business, which includes IT, decide what’s right to outsource and what isn’t. It shouldn’t be a threat or a punishment, but a decision that all parties involved agree makes good business sense.

The latest Briefings Direct: SOA Insights podcast is now available. In this episode, we discussed semantic web technologies, among other things. One of my comments in the discussion was that I feel that these technologies have struggled to reach the mainstream because we haven’t figured out a way to make it relevant to the developers working on projects. I used this same argument in the panel discussion at The Open Group EA Practitioners Conference on July 23rd. In thinking about this, I realized that there is a strong connection in this thinking and SOA. Simply put, it is all about the consumer.

Back when my day-to-day responsibilities were programming, I had a strong interest in human-computer interaction and user interface design. The reason for this was that the users were the end consumer of the products I was producing. It never ceased to amaze me how many developers designed user interfaces as if they were the consumer of the application, and wound up giving the real consumer (the end user) a very lousy user experience.

This notion of a consumer-first view needs to be at the heart of everything we do. If you’re an application designer, it doesn’t bode well if you consumer hate using your application. Increasingly, more and more choices for getting things done are freely available on the Internet, and there’s no shortage of business workers that are leveraging these tools, most likely under the radar. If you want your users to use your systems, the best path is make it a pleasant experience for them.

If you’re an enterprise architect, you need to ask who the consumers of your deliverable are? If you create a reference architecture that is only of interest to your fellow enterprise architects, it’s not going to help the organization. If anything, it’s going to create tension between the architecture staff and the developers. Start with the consumer first, and provide material for what they need. A reference architecture should be used by the people coming up with a solution architecture for projects. If your reference architecture is not consumable by that audience, they’ll simply go off and do their own thing.

If you are developing a service, you need to put your effort into making sure it can be easily consumed if you want to achieve broad consumption. It is still more likely today that a project will build both service consumer and service provider. As a result, the likelihood is that the service will only be easily consumable by that first consumer, just as that user interface I mentioned earlier was only easily consumed by the developer that wrote it.

How do we avoid this? Simple: know your consumer. Spend some time on understanding your consumer first, rather than focusing all of your attention on knowing your service. Ultimately, your consumers define what the “right” service is, not you. You can look at any type of product on the market today, and you’ll see that the majority of products that are successful are the ones that are truly consumer friendly. Yes, there are successful products that are able to force their will on consumers due to market share that are not considered consumer friendly, but I’d venture a guess that these do not constitute the majority of successful products.

My advice to my readers is to always ask the question, “who needs to use this, and how can I make it easy for them?” There are many areas of IT that may not be directly involved with project activities. If you don’t make that work relevant to project activities, it will continue to sit off on an island. If you’re in a situation where you’re seen as an expert in some space, like semantic technologies, and the model for using those technologies on project is to have yourself personally involved with those projects, that doesn’t scale. Your efforts will not be successful. Instead, focus on how to make the technology relevant to the problems that your consumers need to solve, and do it in a way that your consumers want to use it, because it makes their life easier.

As I’ve mentioned previously, I’m a member of the SOA Consortium. In a recent conference call for them, the subject of maturity models came up, and I discussed some of the efforts that I’ve done on MomentumSI’s Maturity Model. Anyway, the end result of the discussion was an email exchange between myself and Ron Schmelzer of ZapThink discussing the whole purpose of maturity models. In my prior life working in an enterprise, I’ve seen my fair share of maturity models on various subjects and I know there were certainly some that I dismissed as marketing fluff. At the same time, there were others that peaked my interest, and even more interesting, I also spent time creating my own maturity models. So what gives? Are they good or bad?

The purpose of a maturity model is pretty straightforward. It’s an attempt to try to quantify where you fall on some continuum. Obviously, there are many ways that this can be problematic, but it usually falls into two categories. First, you may not even think the subject at hand is worth measuring, therefore, a maturity model is a waste of time. Second, you may agree that subject is worth measuring, but you may disagree with the measuring scale. Frequent readers of my blog know that I’m no stranger to that category, as I started a bit of stir with some comments regarding the maturity model that David Linthicum had posted on his blog.

For this entry, the real question to ask is whether or not SOA adoption is something worth measuring? If you answer yes, then you’re going to need some yardstick to do so. I certainly think that any SOA adoption effort must include some way of assessing your efforts. If you don’t have this, how can you ever claim success? A maturity model is one way of assessing it. A roadmap with goals attached to points in time can certainly be another way of doing it, as well. I believe that you’ll need both. A maturity model approach is good because it’s not based upon reaching a point in time, it’s based on more qualitative factors, such as how you do things. You don’t become more mature with SOA buy purchasing a product, for example. You become more mature by using a product appropriately, consistently, successfully and in a repeatable fashion. The challenge with maturity models, however, is that they are often created as a unit of comparison, rather than as a unit of assessment. This has everything to do with the yardstick of measurement. During the SOA Consortium discussion, someone brought up that some maturity models aren’t valuable because an organization may not be concerned with reaching the upper levels of maturity. When you create a maturity model that is intended as a comparison tool, you need a yardstick that works for all. For organizations that are content at level one or level two on your scale, there may not be much value. If you focus solely on a maturity model as an assessment tool, step one is to establish criteria for the levels that make sense for your organization. If you do this, now you can use it as a means of measuring your progress on your journey. You won’t be able to use it to compare yourself to your competitors, but that’s okay, because you’re measuring your activities, not theirs.

Roadmaps are good as well, because they are very execution-oriented. While a maturity model may call out some lofty, qualitative goals, it doesn’t articulate a path to get you there. Roadmaps, on the other hand, consist of activities and deadlines. At the same time, roadmaps can run a risk of being too focused on the what and when, and not on the how.

The important takeaway from this is that if you’re going to attempt to adopt SOA, you need a way of measuring it. As many have said, SOA is a journey. It’s not one project. My recommendation it consider using both a maturity model and a roadmap with concrete milestones as tools for measuring your efforts. The roadmap will keep you focused on the execution, and the maturity model will ensure that you can capture the more qualitative aspects, such as service consumer/service provider interactions in the development lifecycle. What are your thoughts? Are maturity models a good things or a bad thing? If they can be either, what makes for a good maturity model versus a lousy one?

The panel discussion I was involved with at The Open Group Enterprise Architecture Practitioner’s Conference went very well, at least in my opinion. We (myself, our moderator Dana Gardner, Beth Gold-Bernstein, Tony Baer, and Eric Knorr) covered a range of questions on the future of SOA, such as when will we know we’re there, will we still be discussing it 5 years from now or will it be subsumed by EA as a whole, etc.

In our preparations for the panel, one of the topics that was thrown out there was how SOA will play with BPM, EDA, BI, etc. I should point out that our prep call only set the basic framework of what would be discussed, we didn’t script anything. It was quite difficult biting my tongue on the prep call as I wanted to jump right into the debate. Anyway, because it didn’t get the depth of discussion that I was expecting, I thought I’d post some of my thoughts here.

I’ve previously posted on the integration between SOA, BPM, Workflow, and EDA, or probably better stated, services, processes, and events. There are people who will argue that EDA is simply part of SOA, I’m not one of them, but that’s not a debate I’m looking to have here. It’s hard to argue that there are natural connections between services, processes, and events. I just recently posted on BI and SOA. So, it’s time to try to bring all of these together. Let’s start with a picture:

In its simplest form, I still like to begin with the three critical components: processes, services, and events. Services are explicitly invoked by sending a service invocation message. Processes are orchestrated through a sequence of events, whether human-generated or machine generated. Services can return responses, which in essence are a “special” event directed solely at the requestor, or they can publish events available for general listening. So, we’ve covered SOA, BPM, EDA, and workflow. To bring in the world of EDW (Enterprise Data Warehouse), BI (Business Intelligence), CEP (Complex Event Processing), and even BAM (Business Activity Monitoring, although not shown on the diagram), the key is using these messages for purposes other than which they were intended. CEP looks at all messages and is able to provide a mechanism for the creation of new events or service invocations based upon an analysis of the message flow. Likewise, take these same messages and let them flow into your data warehouse and allow your business intelligence to perform some complicated analytics on them. You can almost view CEP as a sort of analytical engine operating on a small window, while business intelligence can act as the analytical engine operating on a large window. Just with CEP, your EDW and BI system can (in addition to report) generate events and/or service invocations. Simply put, all of the technologies associated with all of these acronyms need to come together in a holistic vision. At the conference, Joe Hill from EDS pointed out that when many of these technologies solved 95% of the problem they were brought in for. Unfortunately, when your problem space is broadened to where it all needs to integrate, the laws of multiplication no longer apply. That is, if you have two solutions that solved 95% of their respective problems, they don’t solve 0.95 * 0.95 = 90.25% of the combined problem. Odds are that combined problem falls into the 5% that neither of them solved on their own.

It is the responsibility of enterprise architecture to start taking the broader perspective on these items. The bulk of the projects today are still going to be attacking point problems. While those still need to be solved, we need to ensure that these things fit into a broader context. I’m willing to bet that most service developers have never given thought to whether the service messages could be incorporated into a data warehouse. It’s just as unlikely that they’re publishing events and exposing some potentially useful information for other systems, even where their particular solution didn’t require any events. So, to answer the question of whether SOA will be a term we use 5 years from now, I certainly hope we’re still using it, however, I hope that it’s not still as some standalone initiative distinct from other enterprise-scoped efforts. It all does need to fall under the umbrella of enterprise architecture, but that doesn’t mean that the EA still doesn’t need to be talking about services, events, processes, etc.

Update: I redid the picture to make it clearer (hopefully).

Andres Carvallo is the CIO for Austin Energy. He was just speaking on how the Internet has changed the power industry. He brought up the point that we’ve all experienced, where we must call our local power company to tell them that the power is out. Take this in contrast to the things you can do with package delivery via the Internet, and it shows how the Internet age is changing customer expectations. While he didn’t go into this, my first reaction to this was that IT is much like the power company. It’s all too often that we only know a system is down because an end user has told us so.

This leads to discussion of something that is all too frequently overlooked, which is the management of our solutions. Visibility into what’s going on is all too often an afterthought. If you exclusively focus on outages, you’re missing the point. Yes, we do want to know when the .001% of downtime occurs. What makes things more important, however, is an understanding of what’s going on the other 99.999% of the time. It’s better to refer this as visibility rather than monitoring, because monitoring leads to narrow thinking around outages, rather than on the broader information set.

Keeping with the theme of the power industry, clearly, Austin Energy needs to deal with the varying demands of the consumers of their product. That may range from some of the major technology players in the Austin area versus your typical residential customer. Certainly, all consumers are not created equal. Think about the management infrastructure that must be in place to understand these different consumers. Do you have the same level of management in your IT solutions to understand different consumers of your services?

This is a very interesting discussion, especially given today’s context of HP’s acquisition of Opsware (InfoWorld report, commentary/analysis from Dana Gardner and Tony Baer).

Joe Hill from EDS is on the stage now. The title of his talk is on SOA and Outsourcing, although it’s far more broad than just an outsourcing discussion. It would be nice to have a copy of his slides, as he had some really good ways of visualizing SOA adoption along a timeline and the differences on how it should be approached. For example, using a chart reminiscent of Clayton Christensen’s “The Innovator’s Solution”, he showed that at the early phases where we have “performance undersupply,” the typical solution may need to be more tightly coupled, fewer vendors, and more use of proprietary techniques. The discussion around SOA is typically on the right hand side of the chart, emphasizing loose coupling, multiple vendors, open standards, etc. The problem that I felt Joe was emphasizing was that you need to recognize where you are and apply the right techniques at the right time, rather than on focusing too heavily on the end state on the right hand side of the diagram.

Rob High of IBM is on the stage now with a presentation titled “SOA Foundation” which runs the gamut of topics associated with SOA. One thing he took a lot of time to discuss was the notion of coherency and the importance of semantics to SOA. It was nice to hear some emphasis on this point, as I believe that an understanding of the semantics is a critical component in moving from SOA applied to applications to SOA applied to the enterprise. Just as with the rest of SOA, make sure you understand the semantics first before throwing any semantic technology at it. While there are evolving specs and tools in this space, none of that will do you any good if don’t first understand the semantics themselves and how that information can be leveraged in your project efforts.

I’m here at The Open Group Enterprise Architecture Practitioner’s Conference, and will try to blog where I can on the presentations. Right now, I’m sitting in David Linthicum’s keynote, which is on EA & SOA. He had an interesting quote, which was:

Five years from now, we won’t be talking about SOA… It will all be folded back into EA.

There’s some truth to this, but there’s also a lot of risk. One of the issues with EA (and many other efforts within IT), is that it can become disconnected from the project efforts that are going on. The term most frequently used with this is “ivory tower” where enterprise architects are simply viewed as paper pushers that know how to create a lot of PowerPoint slides. One of the side benefits of SOA is that it relates very well to the world of the development projects, with “service” being the point of common language. The enterprise architects may be modeling the enterprise in terms of the services that are needed, and projects can now utilize these models in their project architecture. This is easier said than done, however, as you need to ensure that the reference material containing these models (e.g. a reference architecture) is consumable at the project level. If the only group that can understand your models are fellow enterprise architects, that’s a problem.

So, will SOA be folded into EA as a whole? If EA can ensure that the artifacts it creates are consumable at the project level, then absolutely, SOA will be folded into EA. If EA is not creating artifacts that are consumable at the project level, then we have a problem. You’ll likely still have tension between EA and SOA, and likely not achieve the levels of success that organizations that have successfully bridged the world of EA and the project space. This doesn’t apply solely to SOA. This applies equally to any architectural domain. You may have information architects working on canonical or enterprise data models, performing data quality analysis, etc., but if it doesn’t find a way to become relevant to project efforts, it will exist on an island with continual struggles to achieve the objectives that were set out.

Joe McKendrick’s blog alerted me to Microsoft’s efforts around an “Internet Service Bus,” essentially a hosted version of BizTalk. Details are available in this Redmond Developer News article from Chris Kanaracus.

After reading both Joe’s blog and Chris’ article, this initially didn’t make much sense to me. I did the smart thing, and didn’t write a knee jerk reaction blog (only because I received a phone call), so now I’ve had time to ponder it a bit.

When I read Joe’s blog, my first thought was whether this would be a competitor to hosted integration players like Sterling Commerce. My limited understanding of one of Sterling’s offerings was essentially that it outsourced the mapping effort normally associated with B2B exchanges. They act as an external intermediary, connecting to your partners and then providing you with data in your preferred format(s) and integration technology. I then read Chris’ article and realized that this wasn’t at all what they were talking about. Quoting from a Microsoft whitepaper, Chris’ article stated:

“For example, when school is closed due to weather, a workflow kicks off. As part of that workflow, the system can notify parents, teachers, and bus drivers, as well as food service vendors, snow plow operators, and local police, using the ISB to traverse networks across these disparate organizations.”

Initially when I read this, I had my enterprise hat on and thought, “Why does this need to be hosted?” There’s absolutely no reason that I’d want hosted orchestration of services that exist inside my firewall, and there’s no reason that an internal orchestration engine can’t access services hosted outside of the firewall. Finally, however, the light bulb has come on, and it has to do with the specific example Microsoft used: a school.

I’ve previously blogged about SOA for schools (here and here). My father-in-law is a grade school principal, so I have the occasional conversation about the use of technology in school administration. Your average school is not going to be able to invest in BizTalk or any other orchestration engine, yet, as the example calls out, there’s certainly opportunities to apply orchestration. What this strategy really is a competitor to is something like Yahoo Pipes. There’s probably a broad market where significant efficiency gains can be made, but the cost of the infrastructure is not worth the investment. Is a school really going to buy BizTalk to automate a workflow that maybe occurs once or twice a year (depending on where you live)? No. This seems much better suited to a pay-per-use model. In this manner, the provider of the hosted workflow can have many, many workflows, any one of which is used infrequently at best. Think of it as the long tail of workflow. This model actually makes some sense to me. What are your thoughts?

Finally, it needs a new name. Internet Service Bus would be a disaster, because it’s not a bus, and it conveys this image of all service traffic on the Internet having to flow through it. Hosted integration doesn’t capture it either, because that’s already taken by the Sterling Commerce’s of the world. What we’re really talking about is Hosted Workflow or Hosted Orchestration. The latter would make a very bad acronym, however, just ask Don Imus. So, I’ll call it Hosted Workflow.

Joe McKendrick asked the question in his blog, “How will we look back on SOA in 2020?” This is actually something I’ve been thinking about (well, maybe not 2020, but certainly the future) in preparation for the panel discussion at The Open Group EA Practitioners Conference next week.

One of the things that I’m very interested in is the emerging companies of today, and what their technology will look like in the future. So many companies today are having to deal with existing systems, focusing on activities such as service enablement. Largely, these exercises are akin to turning a battleship on a dime. It’s not going to happen quickly. While I’m fully confident that in 2020 many of these companies will have successfully turned the battleship, I think it will be even more interesting to see what companies that had a blank slate look like.

Dana Gardner recently had a podcast with Annrai O’Toole of Cape Clear Software, where they discussed the experiences of Workday, a Cape Clear customer. Workday is a player in the HR software space, providing a SaaS solution, in contrast to packaged offerings from others. Workday isn’t the company I’d like to talk about however. The companies that I’m more interested in are the ones that are Workday customers. Workday is a good example for the discussion, because in the podcast, Dana and Annrai discuss how the integration problems of the enterprise, such as communication between HR and your payroll provider (e.g. ADP), are now Workday’s problems. By architecting for this integration from the very beginning, however, Workday is at a distinct advantage. I expect that emerging companies with a clean IT slate will likely leverage these SaaS solutions extensively, if for no other reason than the cost. They won’t have a legacy system that may have been a vertical solution 20 years ago, but is now a horizontal solution that looks like a big boat anchor on the bottom line.

One thing that I wanted to call out about the Workday discussion was their take on integration with third parties. As I mentioned, they’ve made that integration their problem. This is a key point that shouldn’t be glossed over, as it’s really what SOA is all about. SOA is about service. The people at Workday recognized that their customers will need their solution to speak to ADP and other third parties. They could easily have punted and told their customers, “Sorry, integration with that company is your problem, you’re the one who chose them.” Not only is that lousy service, but it also results in a breakdown of the boundaries that a good SOA should establish. In this scenario, a customer would have to jury rig some form of data extract process and act as a middleman in an integration that would be much better suited without one. You have potentially sensitive data flowing through more systems, increasing the risk.

The moral of this story is that there are very few times in a company’s history where the technology landscape is a blank slate. Companies that are just starting to build their IT landscape should keep this in mind. I’ve blogged in the past on how the project-based culture where schedule is king can be detrimental to SOA. An emerging company is probably under even more time-to-market pressure, so the risk is even greater to throw something together. If that’s the case, I fear that IT won’t look much different in 2020 than it does today, because the way we approach IT solutions won’t have changed at all. Fortunately, I’m an optimist, so I think things will look significantly different. More on that at (and after) the conference.

I’ll be giving a webinar on Policy-Driven SOA Infrastructure with Mike Masterson from IBM DataPower next week on Thursday at 1pm Eastern / 10am Pacific, and probably could find a way to tie in today’s Dilbert to it. Give it a read.

As for the webinar, it will discuss themes that I’ve previously blogged about here, including separation of non-functional concerns as policies, enforcing those policies through infrastructure, and the importance of it to SOA. Mike will cover the role of SOA appliances in this domain. You can register for it here.

InfoQ recently posted an article on Business Intelligence (BI) and SOA and it made some good points in the later half of the article. Hopefully readers get there, however, because the initial discussion around ETL and getting data originally gave me the impression that the article wasn’t going to get to the really important factors on the relationship between SOA and BI. It began the discussion by stating:

The road to BI usually starts with extract transform and laod (ETL).

I don’t have any disagreement with that statement, however, the next part of the article goes on to talk about the potential problems introduced by SOA to ETL. In essence, it states that because SOA is all about isolating internal data behind interfaces, it becomes problematic for BI systems rooted in ETL because of the need to intimately understand the data. The article seems to suggest that companies may try to use SOA to pull data into the data warehouse rather than an ETL approach. This would be a mistake, in my opinion. If the existing ETL processes work just fine, why change it? Typically service usage is more about transactional processing and not about the bulk data movement associated with ETL.

Where the discussion gets interesting is when we start looking at the future of BI. There are two areas for improvement. The first is in the timeliness of the information. BI processing that is dependent on ETL processing is typically going to happen as part of some scheduled job that occurs each night. The second is in the information itself.

To address the first area, a move toward an event-driven architecture for pushing information into the business intelligence system is necessary. The article does a very good job in addressing this, and even correctly calls out that advancing technologies in event stream processing such as CEP (complex event processing) will play a key role in this.

The second area is not as clearly addressed in the article. I am by no means an ETL and BI expert, but my limited experience with it indicated that it was largely based on the end results of processing. That is, for any given processing chain, you typically have nodes that represent and endpoint where that particular processing chain ends. Users typically begin the chain, and some resource (relational database, content management system, file system) typically ends the chain, if there’s any permanent record associated with the processing. Graphically, it looks like this:

Today, BI and its associated ETL jobs may focus only on relational databases that make a permanent record of a transaction. That’s only one resource in the chain. What about all of the intermediate steps along the way? Think of what Amazon does by not only looking at your purchase history but also in looking at the things you (and others) browse for, independent of whether or not you actually purchase it. Arguably, this represents a far better picture of your interests that your orders. This is where I feel SOA can really provide benefits. Adopting SOA should result in more individual components for any given solution, but by accessing those components in a standards-based manner, we’re still able to manage the associated increase in complexity. If we have more components, we have greater visibility into the information flowing in and out. Employ some standard service management technology, and you can easily extract the message flows in real time and centrally store them for extraction into your BI environment. Or, if you’re ready to do it all in real time, those same tools can simply publish those service messages out to an event bus for real-time extraction into your BI environment. This is where I think the potential lies. It’s not about getting at the same old information in an easier way, it’s about getting new information that can yield better intelligence. Consistency of service schemas is important to SOA, and if done right, the incorporation of this wealth of information sources into your BI system shouldn’t become a complex nightmare. This is yet another example of how we need to think outside the box, and realize that information can always be used in novel ways beyond the immediate consumer-provider interaction that is driven by the project at hand.

Joe McKendrick recently had a post regarding bottom-up approaches to SOA in response to this post from Nick Malik of Microsoft, which was then followed up with another post from Nick. I was going to post solely on this discussion, when along came another post from James McGovern discussing bottom-up and top-down initiatives. While James’ post didn’t mention any of the others, it certainly added to the debate. Interestingly, in his post, James put SOA in a “bottom-up” list. Out of this, it seems like there are at least three potential definitions of what bottom-up really is.

Joe’s discussion around “bottom-up” has a theme of starting small. In his followup post, Nick actually concurs with using this approach to SOA, however, he points out that this approach is not what he considers to be “bottom-up.” For the record, I agree with Nick on this one. When I think of bottom-up in the context of SOA, I think of individual project teams building whatever services they want. This puts the organization at great risk of just creating a bunch of services. While you’re guaranteed that your services will have at least one consumer by simply identifying them at the time they’re needed within a project, this is really the same way we’ve been building systems for years. If you change the approach to how you build the solution from the moment the service is identified, such as breaking it out as its own project with an appropriate scope increase to address broader concerns, you can still be successful with this approach, but it’s not easy. When I think of top-down, like Nick, it also starts to feel like a boil the ocean approach, which is probably at just as big of a risk of being unsuccessful.

Anyway, now bringing James’ post into the discussion, he had given a list of three “top-down” IT initiatives and three “bottom-up” initiatives. Top-down ones were outsourcing, implementing CMMi and/or PMI, and Identity Management within a SoX context. The bottom-up initiatives were SOA, Agile Methods, and Open Source. By my interpretation, James’ use of the terminology refers more to the driving force behind the effort when mapped to the organization chart. His top-down efforts are likely coming from the upper echelons of the chart, while the bottom-up efforts (although SOA is debatable) are driven from the lower levels of the chart. I can certainly agree with this use of the terminology, as well.

So is there any commonality? I think there is and it is all about scope. The use of the term top-down is typically associated with a broad but possibly shallow focus. The use of the term bottom-up is typically associated with a narrow but possibly deep focus. What’s best? Both. There’s no way you can be successful with solely a bottom-up or top-down approach. It’s like making financial decisions. If you live paycheck to paycheck without any notion of a budget, you’re always going to be struggling. If you develop an extensive budget down to the penny, but then never refer to it when you actually spend money, again, you’re always going to be struggling. Such is the case with SOA and Enterprise Architecture. If you don’t have appropriate planning in place to actually make decisions on what projects should be happening and how they should be scoped, you’re going to struggle. If the scope of those projects is continually sacrificed to meet some short term goal (typically some scheduling constraint), again, you’re going to struggle. The important thing, however, is that there does need to be a balance. There aren’t too many companies that can take a completely top-down approach. Even startups with a clean IT slate are probably under such time-to-market constraints that many strategic technology goals must be sacrificed in favor of the schedule. It’s far easier to fall in the trap of being overweighted on bottom-up efforts. To be successful, I think you need both. Enterprise architects, by definition, should be concerned about breadth of coverage, although not necessarily breadth of technology as specializations do exist, such as an Enterprise Security Architect. An application architect has narrower breadth than an enterprise architect, but probably more depth. A developer has an even narrower breadth, but significant depth. The combination of all of them are what will make efforts successful both in the short term and in the long term.

I was listening to an Inflection Point podcast from the Burton Group entitled Enterprise 2.0: Overcoming Fear of Blogs and thought that this was a subject that I hadn’t posted anything about. The topic of the podcast was the use of blogs and wikis internally within the enterprise, rather than enterprise workers blogging to the outside world, which is a different subject.

The interviewer, Mike Gotta, made the comment that blogs within the enterprise need to be purposeful, and that there’s a fear that they will simply be a soap box or a place to rant. To any manager or executive that has a fear about them becoming a place for employees to rant, you’re looking at the wrong issue. If employees feel a need to rant, they’ll be doing it in the hallways, break rooms, and cafeteria. If anything, blogging could bring some of this out in the open and allow something to be done about it. This is where I think there is real value for blogging, wikis, etc. in the enterprise. I recently had a post titled Transparency in Architecture that talked about a need for projects to make their architectural decisions transparent throughout the process (and the same holds true for enterprise architects and their development of reference architectures and strategies). Blogs and wikis create an opportunity to increase the transparency in enterprise efforts. I’ve worked in environments that had a “need to know” policy for legitimate reasons, but for most enterprises, this shouldn’t be the case. When information isn’t shared, it may be a symptom of lack of trust in the enterprise, which can be disastrous for SOA or anything else IT does.

Blogs and wikis are about communication and collaboration. Communication and collaboration help to build up trust. That being said, you do need to be respectful of the roles and responsibilities within the organization. There’s a time and place for debate, and there’s a time and place for adherence to policies. I’ve been in the trenches and had my fair share of times where I didn’t agree with a decision that was made by people above me in the organization. I also understood that it wasn’t my decision to make. In general, however, I believe that things would have been better if there were transparency behind those decisions. That’s important because the principles that guide those decisions wind up influencing the decisions that are made further down the line. If the staff has no visibility to those principles, how can they be expected to make good decisions? The end result of that situation is increased distrust on both sides. It’s not easy to do, because once you expose that information, it leaves you open for debate, which is where mutual trust must come into play. Betrayals of trust either by not disclosing information or by misusing information that was disclosed can be detrimental.

In general, I’m an optimist and I give people the benefit of the doubt. As a result, I’m all for transparency and the use of blogs and wikis in the enterprise. Will blogs and wikis dramatically change the way IT works? Probably not, but it certainly has the potential to improve morale which certainly have a role in improving productivity.