Killer, Gorilla, Guerilla SOA

The Proposition

Just read an interesting post by Jim Webber about a lighter weight technique for SOA implementation that he calls Guerilla SOA.  This spurred a number of comments from people such as Joe McKendrick, Tony Baer and ZapThink’s Jason Bloomberg.  Generally the comments were positive excepting the usual concerns about bottom up implementations leading nowhere.

Guerilla Campaigns Need Direction

As I’ve discussed previously I’m a big fan of thinking broadly but starting small, since with difficult to grasp concepts like SOA we have to rapidly and continually deliver benefit to generate buy in from our business colleagues.  As a result I am intrigued and broadly supportive of the concept of guerilla SOA implementation.  The key thing in my view, however, is to start in the right way by concentrating initially on getting a broad view of the business capabilities needed by the enterprise through a rapid approximation and then use these as a governing framework for the implementation of services within the organisation. I’m a firm believer in mixing systematic design with a dash of chaos - essentially govern the top level business capabilities required by the organisation but then delegate responsibility for the realisation of these downwards (and since this is a fractal process there are a limited number of capabilities at each level and as such the decision making process is made simpler by restricting the number of abstractions that any one group of people need to manage).  As a result we allow capability owners to realise services any way that they want - potentially meaning that many replicated services spring up across the organisation within the control of different capabilities - since the only thing we care about is that we have the ‘right’ capabilities and that these perform to the cost and quality metrics that we require.  In this context sharing of services at lower levels of granularity will be an emergent property caused by attempts to improve the cost effectiveness and quality of capabilities rather than something that is planned for from day one (which requires a deep and complex understanding of how everything in the organisation works from top to bottom - an impossible task despite the best efforts of many current enterprise architecture approaches).  Equally importantly in this context, attempting to force capability owners to share smaller granularity services by looking for ‘reuse’ often damages these business capabilities in subtle ways that are impossible to guess ahead of time - as a result independence with future emergent sharing opportunities based on metrics is a more appropriate mechanism IMHO (which now that I’ve read that sounds a bit like the agile principle of only building what you need - hadn’t really thought about it within a business context like that before - gives a whole different spin to the hackneyed term ‘agile business’).

Executing Your Campaign

Now the question is how this relates to the discussion of guerilla SOA?  I would argue that the notions of business capabilities and guerilla SOA are orthogonal and complementary.  Essentially you map out the broad ’shape’ of the organisation (your campaign goals) using business capabilities - consider this the horizontal dimension - but then concentrate on federated projects (guerilla operations) that maximise value in the improvement of these capabilities and fill them out using agile implementation styles, essentially rapidly implementing elements of capabilities with focused projects - consider this the vertical dimension.  The reality is that no business design (or indeed guerilla campaign) can ever be correct and pre-planned from top to bottom at any point in time and so the key is to rapidly approximate the systematic capabilities needed by the organisation and then get on with implementing value in a stepwise fashion.  Your governance process will need to be able to cope with changes - e.g. versioning of capabilities, business messages and everything else - as your transition to SOA proceeds but there is no way to nail all of this up front and for ever and so you’d better get used to that and move on.  Essentially anyone who comes to you and suggests a top-down process re-engineering exercise should be politely shown the door, since such exercises take years, are always out of date and never actually deliver any sustainable value.  Let’s call this top down approach “Killer SOA” as it is guaranteed to go nowhere and will quickly lead to a morass of paperwork that scuppers your attempts at SOA.  Conversely however just pitching in like a gorilla without any context will make you look busy and actually deliver some stuff but it won’t really add up to anything that makes a significant difference to your organisation - the sum of the parts will really be just the parts (and it’ll be more complex and expensive to deliver, operate and manage).  Let’s call this bottom up approach “Gorilla SOA” as we can imagine it as a chase for the next banana without any concern for what such plundering might do to the environment that sustains us.  To me Guerilla SOA takes the best approach possible, relying on a combination of a broad plan and lots of loosely coordinated and federated cells with the power to implement in a situationally appropriate way.  

Enable Guerillas to Improvise in Realising Goals

This approach to business capability realisation is also applicable in the technology space as well.  I am glad to see Jim stating (and an increasing number of other people realising) that technology as architecture (e.g. ESBs/application servers/whatever) is both meaningless and dangerous.  As I’ve discussed in the past trying to force centralised architectural approaches onto an essentially heterogeneous business environment (heterogeneous from a requirements perspective (e.g. QoS or cost requirements) as well as a technical one) leads to an environment that is fit for nobody.  Architectural approaches should concentrate on the space between capabilities (i.e. standards that support interoperability (or an overall loose coordination of cells)) and on the governance of the required outputs (i.e. capability metricisation and performance management (or realisation of strategic operations)) and enable capability owners (and guerillas) to maximise existing assets or new approaches that best map to their goals and aspirations rather than on trying to use technology as an excuse for architecture (I have to also admit to having a particular distaste for ‘ESBs’ which in most incarnations are just hub-and-spoke message hubs with proprietary transport which undermine the very notions of SOA - i.e. federation, standardisation and interoperability - but I guess that’s another post, grrr ;-)).

It’s the Difference Between Killer, Gorilla and Guerilla that Matters

In the broader context the question is whether Jim’s comments reflect an anarchic and directionless dead-end of random service implementation or a dynamic and manageable way to break the SOA deadlock within organisations.  To me the question you have to ask yourself is whether you are a Killer that drowns your organisation in paperwork, a Gorilla who just wades into SOA in an elemental fashion with little thought beyond the moment or whether you are a Guerilla who is part of a larger group of loosely federated fighters striving to realise a shared vision. 

P.S.

I realise that many people refer to ‘Guerilla SOA’ as ‘Middle Out’ (see Nick Malik for an example).  To me the two terms describe the same thing but whereas ‘Guerilla’ makes me think of glamour, adventure and action ‘Middle Out’ makes me think of the male aging process and brings to mind increasing flabbiness around the midriff and balding.  Just sounds less exciting doesn’t it, especially when we’re trying to convince people to follow us into action, LOL

:: :: :: :: :: :: :: :: :: :: ::

SCA = Scary Component Architecture?

Only just catching up with SCA/SDO as part of some work with a current client and have to say that there seems to be a great deal of confusion around these essentially useful specs.  First of all read the threads over at Joe McKendrick’s blog where a conversation raged for a while over the utility of SCA but then also came across this blog post by Dana Gardner over at FASTForward complaining about Microsoft’s unwillingness to play ball.  This latter post confused me deeply.  Dana (and others that he references) seem disturbed by the fact that Microsoft are not involved in this effort and that this represents a blow to ‘SOA interoperability’ and an unwillingness from Microsoft to be open.  This seems like a strange statement to me as from what I can glean from the specifications for SCA and SDO - and from great resources such as those provided by Dave Chappell - these proposals seem to be more about how you create components that sit within a particular realisation environment - realistically Java ones given the spread of vendors involved - and they seem to say nothing about interoperability per se.  Interoperability across realisation technologies (even across specific containers in fact) is left - very sensibly in my view - to existing web service standards.  So far so good in my view as the Java community have badly needed some support in simplifying the creation of lighter weight services of the sort needed in a federated world, since Java EE’s component roots are based on a previous generation of architectural model, IMHO, and as a result it has become way too complex.  Suggestions that Microsoft are not ‘open’ however on the basis of the fact that they only provide support for interoperability standards seems perverse to me.  Whether you are using SCA to compose applications that will realise services within a specific Java vendor’s environment, using WCF within the Microsoft world - or indeed ignoring all of these and resorting to simpler, more dynamic environments such as Ruby - surely having a range of programming models and platforms competing to realise services most efficiently is a healthier model than trying to force them to create service realisations in the same way?

The suggestion that we would have a common programming model that transcends implementation language, however, (as seems to be implied by Dana’s post and indeed others that I’ve read) seems a bit oxymoronic to me as surely our existing XML standards (of whichever flavour you prefer) take care of the multi-language aspect of the problem space, leaving composition as the main issue to be addressed (since BPEL only partially addresses this space).  I’m all for interoperability as a way of enabling competition in the provision of implementations but less ecstatic about the idea of trying to arrive at a converged programming model with a confusing mix of service and implementation metadata.  To be honest I feel that trying to mix the interoperability and implementation spaces by pushing a converged realisation model onto vendors introduces unnecessary complexity and so would not be a good approach to solving the issues around service composition.  Trying to talk SCA up as a way of enabling interoperability across implementation technologies (which again I have to stress is not something I believe the spec authors are doing) is a shame as it confuses an area around which we absolutely need more research and guidance, namely specific ways of modelling composite services - whether UML-based via the OMGs recent call for a services profile, as part of the STP project within Eclipse or via DSLs such as those supported by Microsoft. In each case, however, I feel that these should be at a level of abstraction above the individual technologies being composed since surfacing this information makes consumption a much more complex proposition by putting the onus on the consumer to take responsibility for integration rather than forcing the provider to expose their services using existing standards.

I guess I’m also influenced by the fact that I lived through the CORBA period where we tried to implement a standard programming model and - realistically - failed.  When I compare the subsequent success that the industry has had by concentrating on simplicity and interoperability suggestions of conformity of approach across realisation  technologies feels - to me at least - like a backward step out of touch with the current zeitgeist; currently we’re increasingly looking at simple, open, web based technologies that are ‘good enough’ and rail against the complexity of WS-*.  Such pragmatism has unleashed a wave of innovation from companies both large and small and anything that could introduce constraints and confusion to this model by trying to develop an uber-model of implementation feels to me like a bad idea. One of the great drivers of recent success has been federation - moving responsibility for service enablement to the endpoints (i.e. providers) to enable consistent and simplified consumption of these by consumers.  Trying to deliver a cross-language rather than web based composition model feels to me very much like the old school EAI approach that we have only just recovered from, moving information about endpoints back into the consumer space and increasing the complexity of the composition space.  it all makes me feel vaguely like I’ve returned to the mid-90s (not something you want to do if you’re Welsh as it was probably one of the worst periods in the checkered history of our rugby team).  At the end of the day I would rather see standardisation at the ‘what’ level (i.e. interoperability) than at the ‘how’ level (i.e. programming model) since it feels to me that this is the sweet spot for having your cake and eating it - an ability to work simply and consistently with anyone whilst allowing them maximum scope for innovation on your behalf. 

Whilst being uncomfortable with the idea that SCA represents some kind of uber-programming model, however, I do absolutely recognise the need to simplify service realisation in the Java space and therefore am heartily on side with it’s core aims.  I guess I’m more interested in the next level of composition, however - i.e. composition across standards-based services - rather than just on improvements in the way that one or more particular technology stacks supports developers in the creation of services that execute within their environment.

:: :: :: :: :: :: :: :: :: :: ::

Where To Start With SOA

Introduction

I was talking to one of my colleagues a few days ago about some of the issues he’s having helping his customers make the scary transition to a more adaptable organisational style based on business capabilities and service-orientation.  I thought I would briefly capture some of the questions and my feelings on the answers.

Questions, questions

1 Looking for a clear start position (“where do we start to eat the elephant”)

I firmly believe that structural changes driven by increasing interconnection across organisations require organisations to atomise - essentially break into their component parts and then specialise in those that will enable them to excel whilst offloading other capabilities onto partners.  Business capability modelling allows us to set the scene for this transition by considering an organisation in terms of what it does and then placing commitments and metrics around these capabilities.  Considering business capabilities therefore allows us to understand the shape of an organisation in terms of what it does and then ask questions about the performance of those capabilities in the here and now.  This gives us an excellent start point since any initial capability-based view of the organisation is likely to highlight many misalignments and performance issues based on that particular organisations journey to it’s current state; surfacing these issues with some well crafted value questions can really help us to pick a good area to begin the transition whilst simultaneously giving us the strategic backdrop to prepare for atomisation.  Anybody looking to take this journey - and this should really be everybody - should adopt a philosophy of “get the (broad) big picture, identify and prioritise improvements, start small, deliver incremental value and then broaden the scope”.  A recent presentation I saw (somewhere - sorry lost the reference) on SOA adoption from a US government agency said “think big, start small, fail, fail, fail, succeed, scale rapidly”. I think this would summarise my preferred approach.  My particular interest is in how we use industrialisation to reduce risk during the ‘fail, fail, fail’ part of the story.  One key way to do this is to look at shared service platforms that take care of the technology end of the issue for you.

2 How do we encourage our staff on this journey (The carrot not the stick)

Business capabilities (particularly when supported by industry metrics from a framework like APQC) give us the ability to set people ‘challenges’ by highlighting the gaps between their performance and external averages. This is not really a carrot or a stick specifically but it’s an important aspect of starting the journey, since focussing pressure onto people within the business through highlighting performance gaps helps to reduce the pressure on us in justifying why they should engage.  In many cases the surfacing of metrics that are actually aligned to organisational needs turns the conversation around such that customers are asking us for help to meet their newly minted metrics rather than us trying to sell some notional concepts such as ‘agility’ or ‘cost effectiveness’.  All of these may be important aspects of the way we do things but they need a proper context in which to be governed and delivered; essentially we need to shift away from the notion of ‘general’ improvement across everything due to technology (e.g. 10% reduction in applications delivery time) to specific, governable and business focused metrics (e.g. £50 reduction in the cost of widget procurement).  Again in this context industrialisation - rather than SOA per se - allows the pitch to be about the greater reliability and lower risk of transitioning to a capability-based approach (and hence aligning their delivery capabilities with their metrics) to improve their performance rather than on trying to sell the abstract notion of SOA.

In the longer term the aim should be to support capability owners in the monetisation of services and thereby give rise to much more innovative compensation schemes based on known, concrete and defensible metrics.  It is only through having commitments and rewards properly aligned and - equally importantly - the means of collecting and surfacing the information needed to assess against these metrics - that we truly develop a culturally holistic organisation. 

If the question being asked is more about how customers encourage developers to share, however, then the question is at wholly the wrong level; essentially we should be considering how you give capability owners a sufficiently high leverage view that they can make sourcing decisions for themselves.  At the end of the day the end state requires that procurement decisions shift focus away from IT and onto capabilities (which will include the IT specific to their implementation).  In that context the capability owner is making a decision between buying or building a capability and developers will never be involved until after  those decisions have been made (potentially not even then if the capability owner decides to contract the work to an external provider or buy a SaaS proposition).

3 How do we gain traction and momentum (early wins)

One of the key tenets of any successful approach IMHO is that although we take an initial (rapid) view of the breadth of capabilities within the organisation to support ongoing governance, we then quickly zero in on specific projects that deliver attributable, high value improvements.  We should always pick sensible early projects based on high impact but low risk, but then have the ability to rapidly scale out based on successes, since successes breed their own momentum.  In particular, if we have metricised the capabilities that the organisation has then we’ll potentially  have two very favourable conditions:  a backlog of performance gaps within the capability map (and thus people looking for help to meet their commitments) and demonstrable early successes in helping other parts of their organisation succeed in this aim.   As a result momentum will build organically - and potentially rapidly - if we get the early project decisions right.  One of the key benefits of the capability approach as scale builds, however, is that it enables much more parallel working on smaller improvement programmes, since commitments are scoped to providers as a ‘black box’ and thus the amount of cross enterprise coordination is reduced (not eliminated, just reduced).

4 How do we make this sustainable (culturally imbedded)

The key aspect of a sustained change is the realignment of commitments around systematic boundaries (i.e. the capabilities).  The use of capabilities and metrics to specify the required outputs enables us to gradually transition behaviours and organisational structures - business capabilities can be ‘logical’ during the early stages of a transition to service-orientation and layered over the top of existing hierarchies.  Concentration on required outcomes will tend to gradually shift organisational structures to best realise commitments, however and to maximise specialisation - as structures that do not align well to commitments (or capabilities that try to do too much) will struggle to realise their cost and performance metrics.  Equally fundamental to sustainability, however, is the need to ensure that service provision within the enterprise becomes  monetisable in order to ensure that capability owners have the scope to:

• Invest in building and extending their services based on their commitments and the needs of their consumers;
• Clearly see the impacts of building instead of sourcing (since it will have to go onto their cost base rather than come from some nebulous central benefactor);
• Can compare the costs of internal capabilities against those from external providers (whether those capabilities that they leverage as services such as recruitment or those that they leverage in order to improve their own capabilities such as IT service provision);
• Generate reward schemes based on achievement of the metrics that are important to the organisation.

Summary

In order to genuinely realise the potential benefits of service-orientation in the short term and - more importantly - prepare for the coming wave of atomisation and capability unbundling, organisations need to find a digestible and sustainable route through the transition process.  Such a route needs to be incremental (but holistic), foster decentralisation (but systematic design) and to ensure sustainability through the alignment of metrics, rewards and monetisation strategies.  Using capabilities as a logical framework enables much more coherent decisions about organisational specialisation, improvement and delivery, and - equally importantly - generates a pull for the requisite assistance as people struggle to realise their commitments rather than a befuddled ‘huh?’ from people whose metrics encourage them to maintain the status quo.

:: :: :: :: :: :: :: :: :: :: ::

Summer Hiatus

Well it’s been a long break since my last post as I’ve been coping with the long summer break - both my own (well deserved) and others (totally undeserved) holidays.  In the former case I obviously left everything in good order for my colleagues but in the latter they obviously left me a pile of stuff to deal with in their absence. 

In any case I’m back now and have made a number of resolutions:

1. To post more often
2. To post less lengthily (if that is a word)
3. To comment more on other people’s posts.

In deference to the second aim I’ll stop there and go and post something short and more useful; although it’ll take a train journey or something before I’m able to post about other people’s stuff

:: :: :: :: :: :: :: :: :: ::