Todd Biske: Outside the Box

Archive for the ‘Governance’ Category

I had the opportunity to do a review of a book, and then discuss it in a podcast with the author and Dana Gardner. The book is entitled, “Succeeding with SOA: Realizing Business Value Through Total Architecture” and is written by Dr. Paul Brown of TIBCO Software.

You can view a transcript here, listen to the podcast here, or I’ve also added it as an enclosure on this entry.

I enjoyed this book quite a bit, and have to point out that it’s not your typical technology-focused SOA book. It presents many of the cultural and organization aspects behind SOA, and does a pretty good job. It tries to offer guidance that works within the typical project-based structures of many IT organizations. While I personally would like to see some of these project-based cultures broken down, this book offers practical advice that can be used today and eventually lead to the cultural changes necessary. Overall, I recommend the book. I found myself thinking, “Boy, if I were writing a book on SOA, these are things that I’d want to cover.” Give the podcast a listen, and check out the book if you’re interested.

Full disclosure: Outside of receiving a copy of the book to review, I did not receive any payment or other compensation for doing the review or participating in the podcast.

A conversation that I’ve seen in many organizations is around the notion of an “enterprise” service. Personally, I think these conversations tend to be a fruitless exercise and are more indicative of a resistance to change. I thought I’d expound on this here and see what others think.

My arguments against trying to distinguish between “enterprise” services and “non-enterprise” services are this:

Classifications are based upon knowledge at hand. Invariably, discussions around this topic always come back to someone saying, “My application is the only one that will use this service.” The correct statement is, “My application is the only one I know of today that will use this service.” The natural followup then is whether or not the team has actually tried to figure out whether anyone else will use that service or not. Odds are they haven’t, because the bulk of projects are still driven from a user-facing application and are constrained from the get-go to only think about what occurs within the boundary of that particular solution. So, in the absence of information that could actually lead to an informed decision, it’s very unlikely that anything will be deemed “enterprise.”

What difference will make? To someone that makes the claim that their service is not enterprise, does it really give them tacit permission to do whatever they want? A theme around SOA is that it approaches things with a “design for change” rather than a “design to last” philosophy. If we follow this philosophy, it shouldn’t matter whether we have one known consumer or ten known consumers. I think that good architecture and design practices should lead to the same solution, regardless of whether something is classified as “enterprise” or “not enterprise.” Does it really make sense to put a service into production without the ability to capture key usage metrics just because we only know of one consumer? I can point to many projects that struggled when a problem occurred because it was a big black box without visibility into what was going on. If there’s no difference in the desired end result, then these classifications only serve to create debate on when someone can bend or break the rules.

What I’ve always recommended is that organizations should assume that all services are enterprise services, and design them as such. If it turns out a service only has one consumer, so what? You won’t incur any rework if only one consumers uses it. The increased visibility through standard management, and the potential cost reductions associated with maintaining consistency across services will provide benefits. If you assume the opposite, and require justification, then you’re at risk of more work than necessary when consumer number two comes along.

It’s certainly true that some analysis of the application portfolio can help create a services blueprint and can lead to a higher degree of confidence in the number of consumers a service might have. This can be an expensive process, however, and projects will still be occurring while the analysis takes place. Ultimately, the only thing that will definitively answer whether a service is “enterprise” or not is time. I’d rather set my organization up for potential success from the very beginning than operate in a reactionary mode when it’s definitive that a service has multiple consumers. What do others think?

I was a panelist for a discussion on the Future of SOA at The Open Group Enterprise Architecture Practitioner’s Conference in late July. The session was recorded and is now available as a podcast from Dana Gardner’s BriefingsDirect page. Please feel free to followup with me on any questions you may have after listening to it.

A fellow member of the SOA Consortium, Surekha Durvasula, EA Manager for Kohl’s, recently posted her perspective on service versioning on the SOA Consortium’s blog. At the end, she asked five questions, to which I thought I’d respond.

1. How many versions of a service should be “live” at any given time? How many versions are too many?
   I’ve actually blogged on this one, so make sure you read this entry before reading the rest of my answer. I’ve yet to see this put into practice, however, for two reasons. First, many organizations are still focused on getting version one out the door, so the pressure isn’t on to solve this problem. Second, organizations are still learning how to manage the service consumer and service provider relationship. The approach I outline requires significant maturity in managing consumers and product lifecycles. In the typical project-based culture in IT, this level of maturity is hard to find. In the absence of maturity, I typically see an arbitrary number set (usually 3). Setting a limit certainly recognizes that versioning must be dealt with, but only time will tell whether that number is right or not.
2. Do you use the service mediation layer to achieve backward compatibility?
   I’ve certainly advised companies that this can be done, but again, haven’t seen it put into practice yet. If we have a policy that says two or more versions must be able to co-exist in production, that implies that we have the ability to route to either of them. If we can do this, there shouldn’t be a need to leverage a mediation layer unless some consumer can’t change in time. This is why I think lots of companies are picking 3 as the “right” number of versions to support. It assumes that when version 2 comes out, consumers should be able to move to v2 by the time v3 is ready. If you choose to only have one version running in production (the latest), and leverage the mediation layer, you may eventually run into a type of change that isn’t easily mediated at which point you need to fall back to having multiple versions running at once. Knowing this, I’d make sure I knew how to run multiple versions at the same time first, and then try to leverage mediation where it makes sense.
3. Where do you determine the service version – the mediation layer, the service consumer, or a service provider?
   Let’s handle the easy one. If you handle it at the service consumer side, you’re effectively saying that consumers must explicitly specify a version somehow. If this is the case, versioning effectively goes away, the explicit specification would treat a different version as if it were a different service. You could certain leverage a mediation layer and apply some transformations to send a V1 request to V2, but eventually you won’t be able to mediate for backward compatibility. If the version desired is not explicit, now we have to handle it through a mediation layer or at the service provider. My personal preference is to leverage the mediation layer for this routing. Otherwise, your service implementation will now be littered with logic to determine what version the request represents, map that to the appropriate implementation, etc. I just don’t think that belongs in code. That belongs in a policy repository. If you externalize this, the service developer can just focus on building services.
4. Do you have governance models and policies around transitioning existing consumers to new versions? Who is responsible for executing that transition?
   Again, I’ve yet to personally see this in practice from an SOA standpoint. I have seen it put into practice from a shared infrastructure standpoint, however, such as rolling out a new version of an app server. Governance is critical. When the governance processes did not allow appropriate priority for these versioning efforts, it was a nightmare. It’s not just about funding for the upgrade, but it’s also about funding the impacted consumers and ensuring they have the resources to do their part. Second, the service provider must make it a consumer-friendly process as possible. Before that new version is ever put into production, the service team should have figured out exactly how their consumers will be impacted. In the infrastructure example, the team responsible for the app server worked with every single application hosted on the platform to make the process as painless as possible. They presented this approach when they requested funding for the effort, and the IT governance process made sure the impacted teams were onboard before it was approved. As a result, it went very, very smoothly. Planning these upgrades is tricky however, as unless things are properly coordinated and bundled, the entire IT project portfolio can get bogged on with independent versioning efforts. There does need to be some master planning over the whole portfolio.
5. What is the most graceful way of “retiring” or “decommissioning” a service?
   If you’ve tackled the first question of how many versions, this shouldn’t be an issue. If you’re enforcing your policy, all consumers of V1 will have migrated to V2 or V3 before V4 goes in. V4 goes live, V1 is shut down. Of course, this is where management must come into play. Whether you perform authorization or not, all service requests must have identity attached to them. If you don’t, you’ll never know whether all the known consumers have migrated. There shouldn’t be any unknown consumers, and if there are, some process wasn’t followed long before this decommissioning effort. In addition to identity, you also need to know the usage profile. I had one consumer of a service that was only run once a quarter, as it was part of quarterly financial processing. If I relied on daily usage counts of 0 to judge when to decommission, this could be devastating for a consumer that is run once every 90 days.

I look forward to the day where these conversations are applicable to the mainstream. Clearly, there are thought leaders and some companies that are already facing these issues. The majority will be following in the future, so now is the time to start thinking about this.

Chris Haddad of the Burton Group recently had a post entitled, “What is a service?” on their Application Platform Strategies blog. In it, he points out that “it is useful to first categorize services according to the level of interaction purpose.” He goes on to call out the following service categories: Infrastructure, Integration, Primitive (with a sub-category of Data), Business Application, and Composite.

First off, I agree with Chris that categorization can be useful. The immediate question, however, is useful for what? Chris didn’t call this out, and it’s absolutely critical to the success of the categorization. I’ve seen my fair share of unguided categorization efforts that failed to provide anything of lasting value, just a lot of debate that only showed there any many ways to slice and dice something.

As I’ve continued to think about this, I keep coming back to two simple goals that categorizations should address. The first is all about technology, and ensuring that is is used appropriately. Technologies for implementing a service include (but certainly aren’t limited to) Java, C#, BPEL, EII/MDM technologies, EAI technologies, and more. Within those, you also have decisions regarding REST, SOAP, frameworks, XML Schemas, etc. I like to use the term “architecturally significant” when discussing this. While I could have a huge number of categories, the key question is whether or not a particular category is of significance from an architectural standpoint. If the category doesn’t introduce any new architectural constraints, it’s not providing any value for this goal, and is simply generating unnecessary work.

The second goal is about boundaries and ownership. Just as important as proper technology utilization, and probably even more important as far as SOA is concerned, is establishing appropriate boundaries for the service to guide ownership decisions. A service has its own independent lifecycle from its consumers and the other services on which it depends. If you break down the functional domains of your problem in the wrong way, you can wind up making things far worse by pushing for loose coupling in areas where it isn’t needed, and tightly coupling things that shouldn’t be.

The problem that I see too frequently is that companies try to come up with one categorization that does both. Take the categories mentioned by Chris. One category is data services. My personal opinion is this is a technology category. Things falling into this category point toward the EII/MDM space. It doesn’t really help much in the ownership domain. He also mentions infrastructure services and business application services. I’d argue that these are about ownership rather than technology. There’s nothing saying that my infrastructure service can’t use Java, BPEL, or any other technology, so it’s clearly not providing guidance for technology selection. The same holds true for business application services.

When performing categorization, it’s human nature to try to pigeonhole things into one box. If you can’t do that, odds are you’re trying to do too much with your categories. Decisions on whether something is a business service or an infrastructure service are important for ownership. Decisions on whether something is an orchestration service or a primitive service are important for technology selection. These are two separate decisions that must be made by the solution architect, and as a result, should have separate categorizations that help guide those decisions to the appropriate answer.

I’ll be giving a webinar on Policy-Driven SOA Infrastructure with Mike Masterson from IBM DataPower next week on Thursday at 1pm Eastern / 10am Pacific, and probably could find a way to tie in today’s Dilbert to it. Give it a read.

As for the webinar, it will discuss themes that I’ve previously blogged about here, including separation of non-functional concerns as policies, enforcing those policies through infrastructure, and the importance of it to SOA. Mike will cover the role of SOA appliances in this domain. You can register for it here.

Joe McKendrick recently had a post regarding bottom-up approaches to SOA in response to this post from Nick Malik of Microsoft, which was then followed up with another post from Nick. I was going to post solely on this discussion, when along came another post from James McGovern discussing bottom-up and top-down initiatives. While James’ post didn’t mention any of the others, it certainly added to the debate. Interestingly, in his post, James put SOA in a “bottom-up” list. Out of this, it seems like there are at least three potential definitions of what bottom-up really is.

Joe’s discussion around “bottom-up” has a theme of starting small. In his followup post, Nick actually concurs with using this approach to SOA, however, he points out that this approach is not what he considers to be “bottom-up.” For the record, I agree with Nick on this one. When I think of bottom-up in the context of SOA, I think of individual project teams building whatever services they want. This puts the organization at great risk of just creating a bunch of services. While you’re guaranteed that your services will have at least one consumer by simply identifying them at the time they’re needed within a project, this is really the same way we’ve been building systems for years. If you change the approach to how you build the solution from the moment the service is identified, such as breaking it out as its own project with an appropriate scope increase to address broader concerns, you can still be successful with this approach, but it’s not easy. When I think of top-down, like Nick, it also starts to feel like a boil the ocean approach, which is probably at just as big of a risk of being unsuccessful.

Anyway, now bringing James’ post into the discussion, he had given a list of three “top-down” IT initiatives and three “bottom-up” initiatives. Top-down ones were outsourcing, implementing CMMi and/or PMI, and Identity Management within a SoX context. The bottom-up initiatives were SOA, Agile Methods, and Open Source. By my interpretation, James’ use of the terminology refers more to the driving force behind the effort when mapped to the organization chart. His top-down efforts are likely coming from the upper echelons of the chart, while the bottom-up efforts (although SOA is debatable) are driven from the lower levels of the chart. I can certainly agree with this use of the terminology, as well.

So is there any commonality? I think there is and it is all about scope. The use of the term top-down is typically associated with a broad but possibly shallow focus. The use of the term bottom-up is typically associated with a narrow but possibly deep focus. What’s best? Both. There’s no way you can be successful with solely a bottom-up or top-down approach. It’s like making financial decisions. If you live paycheck to paycheck without any notion of a budget, you’re always going to be struggling. If you develop an extensive budget down to the penny, but then never refer to it when you actually spend money, again, you’re always going to be struggling. Such is the case with SOA and Enterprise Architecture. If you don’t have appropriate planning in place to actually make decisions on what projects should be happening and how they should be scoped, you’re going to struggle. If the scope of those projects is continually sacrificed to meet some short term goal (typically some scheduling constraint), again, you’re going to struggle. The important thing, however, is that there does need to be a balance. There aren’t too many companies that can take a completely top-down approach. Even startups with a clean IT slate are probably under such time-to-market constraints that many strategic technology goals must be sacrificed in favor of the schedule. It’s far easier to fall in the trap of being overweighted on bottom-up efforts. To be successful, I think you need both. Enterprise architects, by definition, should be concerned about breadth of coverage, although not necessarily breadth of technology as specializations do exist, such as an Enterprise Security Architect. An application architect has narrower breadth than an enterprise architect, but probably more depth. A developer has an even narrower breadth, but significant depth. The combination of all of them are what will make efforts successful both in the short term and in the long term.

Brandon Satrom posted a blog last week asking the question, “Is Enterprise Architecture Declarative or Imperative?” In clarifying the question, he pointed out that declarative programming is where a developer instructs a system what to do, but leaves it to the system to decide how best to implement that instruction. Imperative programming is where the system is told both the what and how. He goes on to discuss that his thoughts are that EA should be declarative, but working closely with a group of Solution Architects (much like I presented in my People! Policies! Process! post) that take the visions and translate them into execution.

I will agree that for the most part, EA is a declarative activity, although I wouldn’t go so far as to say that the path to execution falls to solution architects. If EA’s merely define the future state, but don’t define the executable actions (albeit at a higher level, they’re not providing project plans), they’ll likely become an ivory tower. Personally, I think the model of increasing constraints is probably a more accurate depiction of what goes on. Through reference models and architectures, there’s a certain set of constraints established that projects must now adhere to. Within the project, there are additional constraints that are established based upon the solution architecture, and then the class design, etc. You could argue that each set of constraints represents a declaration of what things should be.

As soon as you establish constraints, the subject of governance now comes up. I had an interesting discussion with someone today regarding this topic and he made the point that the projects that are intended to operate within the constraints established by EA must be transparent. Rather than assign someone from the security force to watch over the project team with an iron fist, simply make the architectural decisions visible so that at any point, someone can take a look. This made perfect sense to me. I’ve worked at a company that had taken a similar approach for financial governance on projects, where reports were generated every week and the budget watchers could easily determine whether things were compliant or not. It struck me that this would be a good approach for architectural governance, as well. Of course, this does imply that there needs to be a way to actually see the architecture from the outside, which is another story. Visibility to the source code does not equate to visibility of the solution architecture. What are your thoughts on this?

There was a question on the Yahoo SOA group on the notion of SOA compliance, jokingly introduced in this blog from January of 2006 discussing SOA compliant toasters and cell phones. The important question was whether there is a notion of “SOA compliance” or not.

I fall in the camp of saying yes. There won’t be any branding initiative associated with it, however, because it’s not about compliance with some general notion of what SOA is (such as having a product that exposes Web Services or REST interfaces), it’s about a solution’s ability to fit within the SOA of its customers.

It is certainly true that many enterprises are moving toward a “buy” strategy instead of a “build” strategy. This doesn’t mean that they don’t have any work in planning out their SOA, however. Organizations that simply accept what the vendors give them are going to have difficulty in performing strategic architectural planning. The typical IT thinking may be that this is simply an integration problem, but it’s deeper than that.

Previously, I had a post on horizontal and vertical thinking. In this discussion, I called out that an organization needs to understand what domains of functionality are horizontal, and what domains are vertical. When looking at vendors, if their model doesn’t match up with the organization’s model, conflict will occur. It may not be at an integration level, it may be at a higher semantic level where the way the business wants to use a tool just doesn’t match with the way the vendor chose to provide those capabilities. Having a domain model of the business functionality provides a way to quantitatively assess (or at least get a step closer to it) the ability of a product to be successful in the enterprise. This is the notion of SOA compliance, in my opinion.

On the vendor side of the equation, the vendor needs to be able to express their capability in the form of functionality model that can easily demonstrate how it would fit into a company’s SOA. This applies whether it’s a hosted solution or a shrink-wrapped product. Any vendor that’s using SOA in its marketing efforts but can’t express this should be viewed with caution. After all, architectural compliance is only one factor in the equation, however I think it’s one that can have strategic, rather than short-term implications.

In my experience, neither side is very well positioned to allow a quantitative assessment of SOA compliance. Both parties lack the artifacts necessary, and there certainly aren’t any standards around it. Even just having Visio diagrams with rectangles labeled “such-and-such service” would be an improvement in many cases. More often than not, the vendors win out for a simple reason: they have a marketing department, whereas the typical enterprise architecture organization does not. You can assume that a vendor has at least created some collateral for intelligent conversations with Enterprise Architects, however, you can’t assume that an enterprise has created collateral for intelligent conversations with vendors. It needs to go both ways.

Sorry, I need to rant, although, not as much as I originally thought. I just read the interview with Gartner’s Paolo Malinverno on SOA Governance at SearchWebServices.com. I read his answer to the question “How do you define SOA governance” and decided I needed to blog on this, although he made up for it a bit in his answer discussing “phases.”

Anyway, in his definition of SOA governance, he split it into two halves. The first half is about making sure that important decisions go through to appropriate people and that these people have appropriate input to make those decisions. The second half is making sure those decisions are followed. I wanted to scream out, “Where are the policies???” But, he did mention in his answer to the next question that there is a “phase which defines what the policy is.”

Unfortunately, I really didn’t feel like this article made the whole picture crystal clear, so let me do that here.

1. People.
2. Policies.
3. Process.

Step one is to figure out who your governors are. Before you can have governance, you need to have someone responsible for creating the policies to be governed. Paolo made the statement, “This set of decisions, I wanted them covered.” I think it’s more vague than that. I doubt any organization already knows all of the decisions that need to be made. It’s more likely that they know the desired outcome, and now need someone to work backwards to figure out what policies will enable that behave. For example, there’s the holy grail of IT, reuse. If the desired outcome is reuse of services, you need someone to work backward and figure out where policies are needed to make that outcome a reality.

Step two, therefore is about establishing the policies. The key to this is making sure the policies exist is some form other than the memories of the governors. Put them on paper in the form of patterns or reference architectures. Codify them and rely on tools to automate it. If the only place the policies exist are in the brains of your governors, all you’ve created is a bottleneck. This results in a situation like this:

On the one side up in the ivory tower sits the governors, pontificating away. On the other side are their constituents (the developers) busily working away in the trenches. Only on a rare occurrence are the constituents able to gain audience with the governors in an exercise known as a review. At such review, the constituents demonstrate their mind-reading abilities in predicting what things the governors care about, and praying that they don’t reach the dreading situation of disagreeing governors who hadn’t even talked to each other about their own opinions.

Clearly, we need some documented policies here.

Well, that helped a bit, as the mind reading isn’t necessary anymore, but as is most often the case, the constituents either disagree with the policy, or don’t bother to look at it until it’s time for their review, only then to find out that they are out of compliance. Now it’s time for the filibuster! At this point, the best course of action from the constituent’s standpoint is to simply put it off until the cost of making it compliant is far more expensive than the perceived cost of leaving it non-compliant. It usually entails telling the business sponsor that the project can be released in two weeks or it can made compliant and released in 3-6 months. Thus ensues the debate between the business leaders and the elected officials, yada, yada, yada. And this doesn’t even touch on the subject of policies which the constituents don’t like, and claims that the governors are being overly influenced by lobbyists and out of touch with the people.

To conquer this one, and get the right process in place, I recommend the following model:

In this model, we introduce a third party that bridges the gap and is the key to successful policy enforcement. Keeping policies up to date can easily be a full time job, as is building solutions. What we need is a third role that does a bit of both. I’ll call it the solution architect. These individuals are able to participate in the policy setting process, but they also are the key enforcer on projects. They may have the role of technical lead, but at a minimum, they have the authority to set direction on a project. In this way, they help projects apply policy at the right time, and they help the governors define relevant policy. Formal reviews may still take place, but hopefully, they’re much faster and focused on increasing communication and collaboration instead of being a potentially belittling exercise and demonstration of authority, which isn’t good for the health of the organization.

Most importantly, people must know their role. The governors should be judged on their ability to produce relevant policy in a timely manner. Constituents should be judged on their ability to produce solutions in a timely manner. Solution architects should be judged on a combination of the two.

The Dilbert for Tuesday, June 12th was certainly appropriate for any conversation on Governance. In all seriousness, however, if you governance processes resemble this, you’ve got problems.

Back in January of 2006, I had a conversation with Phil Windley on governance for an article in InfoWorld. One of the points that I made was that you do need to get your policies documented and communicated. If the only way someone knows whether the decisions they make are right or wrong is by attending a meeting and hoping someone with authority (if they even know who that is) agrees with them, that’s not good governance, albeit it is governance.

The Dilbert cartoon is titled, “CEO Visit.” Clearly, no one will question the authority of the CEO. So, you could argue that step one of governance has been handled, which is establishing the authority of the people who will govern. It’s not just the CEO, but the CIO, CTO, Chief Architect, etc. The problem doesn’t normally exist with authority at these levels, but at the levels in the middle. The chief architect may be signing off on all policies, but they’re probably not setting every policy. There needs to be a hierarchy of decision making, so that decisions are made at appropriate levels, and only where conflicts arise do things bubble up. The problem with this is that if the policies and guiding principles at the higher levels are not established, everything breaks down at the lower levels, because no one knows who has authority to set direction in which area. As a result, people are constantly pushing the boundaries and trying to gain power and influence, rather than actually creating good solutions. It’s not about doing the right thing, it’s about doing my thing. If no one (with authority) has stated what the “right thing” is, individuals will always think that their thing is the right thing. Step two is about establishing and communicating policy.

In a bit of irony, the CEO in the Dilbert makes the statement, “Opinions are treason.” I hadn’t thought about it at the time, but opinions are the downfall of the enforcement portion of governance. Opinions are good when policies are being established. Opinions are bad when policies are being enforced. Think about it. It’s not going to do a bit of good to tell the police officer, “Sorry, I was speeding because I feel the speed limit should be 20 miles per hour higher on this road because of the light traffic and the limited number of entrances, exits, and intersections.” When that highway is being constructed, however, and speed limits are being established, bring all the opinions you want (although backing them up with facts is even better).

We can’t forget about the process (step 3), however. If you employ architecture reviews, what happens all too frequently is that at the time of a review, the discussion turns in a debate of opinions, which will normally go unresolved unless someone in the room is in a position of authority recognized by all. A review needs to demonstrate that policies have been followed, and where they have not, focus on the reasons why an exception is necessary, not on whether it’s a bad policy. In addition, the review must also address areas where better guidance was needed, so new policies can be created. Ask the question, “In coming up with the solution architecture, where did you have multiple options for which the reference architecture and patterns did not provide sufficient guidance? What factors led you to choose the option that you did?”

So, as I’ve said before, governance is about people, policies, and processes, in that order. Without clear understanding of authority, documented and communicated policies, and processes to keep improving things, you’ll wind up going to your CEO far too many times for decisions until he starts calling you “Doofy.”

As another example of innovative thinking and how we should look “outside the box”, take the recent entry by Vinnie Mirchandani entitled “People! More Government is Not Good!” While by the title you might think it is a political commentary (which it partially is), it begins with a discussion on compliance IT. Vinnie further uses real world examples of government activities versus activities in the private sector, such as Wal-Mart’s response to Hurricane Katrina as quoted from Fortune:

That was Hurricane Katrina, when government at nearly every level looked utterly incompetent while businesses became the heroes. FedEx delivered 440 tons of relief supplies, mostly at no charge. Wal-Mart meteorologists informed managers that Katrina was headed for New Orleans more than 12 hours before the National Weather Service told the public; the company later hauled millions of dollars of supplies into the worst-hit areas days before FEMA showed up.

It is posts like these that keep things interesting for me. By looking at how governance operates in non-IT scenarios, there are lots of lessons that can be learned on how to apply it within IT scenarios. SOX was intended to bring a new level of compliance and accountability to businesses. Will SOX bring back credibility and trust or is some form of self-regulation more important? How does this apply to the context of SOA governance? One key difference is that businesses already had governance prior to SOX, especially for public companies. On top of that, there is the notion of business ethics. In contrast, the more people I talk to, the more I think that the majority of organizations have little to no technical governance. So what’s the right approach? While businesses may not have needed more governance, it’s hard to argue that for IT. In a culture that lacks governance, can they be trusted to self-regulate? One thing is for sure, it won’t happen if there isn’t a shared understanding of what the right thing is. Perhaps the message should begin with one of SOA ethics and morality and then move on to SOA governance. Ethics and morality establish the unbreakable principles of the society as a whole. Communities that agree on these principles can be very successful with self-regulation and less governance. Communities that do not are unlikely to be successful with self-regulation. Where does your IT department stack up? Are there unbreakable principles that they all agree on?

I was recently having a conversation regarding data services and my recent posts (here and here) on horizontal versus vertical thinking put it in a new light. My experience has shown me that data services are one of the most controversial topics within an enterprise.

Nearly all solutions, somewhere along the line, are going to need to access some form of structural data, typically from one or more relational databases. The debate around data services begins with a discussion on whether or not all access to the data should go through a service or not. I’m not going to get into that part of the debate in this post. The point of this discussion lies more with the organizational challenge of creating a data services layer.

One reason (but certainly not the only one) that is sometimes used to justify a data services layer is to maintain control over how the databases are used. All it takes is one bad SQL statement to cause a significant impact to all systems using that database. When organizations see this problem, the instinct is to centralize the development of data services. By creating a team with expertise in database access technologies, the assumption is that you’ll get high quality services that will prevent system failures due to data unavailability. The challenge with this approach is that this assigns ownership and service boundaries along a technical domain: data access. Per my previous postings, assigning ownership based on technical boundaries implies a horizontal domain. Horizontal domains are ones where the capabilities are becoming commodities and can be done in a standard fashion. Service teams in a horizontal domain typically strive to minimize the number of services involved. This is where the challenge occurs. Data access is anything but a commodity. While the techniques used to access data may be standardized (SQL), the structure of data and the combinations of data desired by consumers is certainly not.

In his book Human Interactions,
Keith Harrison-Broninski pointed out that everyone deals with information differently. We file it differently, we take notes differently. Trying to standardize information access is inherently problematic. Why is this? Because information retrieval operates in a vertical domain. We’re retrieving information for the purpose of doing a particular task. So, we have a services team that is viewing the world in a horizontal fashion, with their primary goal being to minimize the number of services they provide. On the other side of the equation, we have vertical thinking consumers that want data for a particular purpose in a format that is optimized for what they need to do. Clearly, if the consumers have the most influence, the goal of the service team to minimize the number of services is not going to happen. If the service team has the most influence, the goal of the consumers to have optimized services is not going to happen. We simply have a situation where the two (or more) teams involved have competing goals. You can’t say that either one is right or wrong, only that they are competing with each other. There can be sound justification on both sides, so the organization must mediate and try to create a win-win situation.

I think this subject is a great example of the concepts of horizontal and vertical thinking. Things can go very smoothly when both the consumer and provider are consistent with their understanding of a particular domain as either horizontal or vertical. When there is disagreement, and one group sees things as horizontal while another sees things as vertical, the relationship is going to be problematic. If you can make it a win-win situation, that is the best scenario. If one party or the other simply has a view that is inconsistent with the view of enterprise architecture, that’s a communications and education issue for EA. EA needs to set the policies for what constitutes good architecture, and if a particular functional domain is deemed to be horizontal (or vertical) that information should be well communicated so the individual solution architect approaches its usage properly.

James McGovern made the following comments regarding my Service Adverse Architecture post:

Todd Biske is right on the money by echoing the fact that companies who have mastery of SOA also have forward thinking management. I wonder if him and Joe McKendrick would define a litmus test so that others can characterize their own enterprise in terms of the management team?

While I’ll have to give some thought to the entire litmus test, the first thing that immediately came to mind was trust, and it’s not limited to management. If I’m a project manager, I want to have as much control over my own destiny as possible. When I now become dependent on other groups and other projects outside of my control and my management to be successful, that’s a big leap of faith. Unfortunately, it’s part of human nature to remember the one project where a team had problems rather than on the times the team was successful. As a result, parties may go into the relationship with the expectation that someone is going to screw up, spend all their time looking for it ready to point blame, and as a result, that’s exactly what happens. If the groups instead focused on what is necessary to be successful, they’d probably be successful.

We live in a culture where making a mistake is unacceptable. One only needs to look at the current political process to understand how much our culture is based on avoiding failure rather than achieving success. A legislator is not allowed to change a stance on any issue, even though we as individuals do it all the time because we learn more as we go along. Cultures that are based on avoiding failure are, in general, going to have a difficult time doing more than maintaining the status quo. Innovating and advancing means taking risks, and when you take risks, some of them don’t pan out. Teams should not be penalized when they don’t pan out, unless it’s clear that they made very bad decisions based upon the information available at that time. I can look back at many decisions I’ve made in the past and recognize that some of my assumptions didn’t come true. As long as I’ve learned from that and don’t repeat those same decisions, there should be no penalty associated with it.

I suspect that organizations that are more forward-thinking have a greater level of trust within the organization. There is more collaboration than competition, and people all understand that everyone in the organization has the best interests of the organization as a whole in mind, not the best interests of themselves, their team, or their manager. Furthermore, management works effectively with the individuals in the trenches to help them understand what levels of risk the organization will tolerate and what the boundaries for innovation are within each group in the organization (essentially around roles and responsibilities). I’d much rather have an application developer creating innovative applications rather than trying to leverage a new Java framework. At the same time, the team responsible for development frameworks needs to be open to receiving input from the application developer. If they have mutual trust, the development framework team will be open to hearing about new alternatives, and the application developer will not throw a fit if the development framework team decides that it’s not in the best interests of the company to leverage the new framework at this time.

SOA will create more moving parts associated with the delivery of IT solutions. More moving parts means more ownership and hence, more interaction among teams. If we don’t trust each other, the chances of success are greatly reduced. That being said, trust must be earned and maintained, it can not be established by edict. Service providers must do all they can to build trust. Management must ensure that the organization takes an “innocent until proven guilty” approach, rather than the opposite, with actions that back it up.

John Wu commented on my horizontal and vertical thinking post earlier in the week, and I felt it warranted a followup post. He stated:

Enterprise Architecture is a horizontal thinking while application development is a vertical thinking.

While I understand where John was coming from, I think that this approach is only appropriate at the very early stages of an EA practice. The first problem that an organization may face is that no one is thinking horizontally. This may go all the way down to the infrastructure level. Projects are always structured as a top-to-bottom vertical solution. While there may be individuals that are calling out some horizontal needs, unless the organization formally makes it someone’s responsibility to be thinking that way, it will have difficulty gaining traction.

Unfortunately, simply creating an EA organization that thinks horizontally and still letting the project efforts think vertically is not going to fix the problem. If anything, it will introduce tension in the organization, with developers claiming EA is an ivory tower and EA claiming that developers are a bunch of rogues that are doing whatever they want in the name of the project schedule.

If we characterize where the organization needs to go, it’s where both EA and the development organization are thinking both vertically and horizontally. This does come back to governance. Governance is about decision making principles to elicit the desired behavior. The governance policies should help an organization decide what things are horizontal, what things are vertical, and then assign people to work on those efforts within those architectural boundaries. Right now, many organizations are letting project definitions establish architectural boundaries rather than having architectural boundaries first, and then projects within those boundaries. Project boundaries are artificial, and end when the project ends. Architectural boundaries, while they may change over time, should not be tied to the lifecycle of a project.

So, EA should be concerned with both the vertical and the horizontal nature of IT solutions. Based upon the corporate objectives, they should know where it is appropriate to leverage existing, horizontal solutions and where it is appropriate to cede more control (although maintaining architectural consistency) to a vertical domain. Two systems that have some redundant capabilities but are architecturally consistent at least create the potential for a consolidation at some later point in time, when the corporate objectives and policies change. In order to do this, the project staff must also be aware of both the vertical and horizontal nature of IT solutions.