SOA, Web 2.0 and the End of Drudgery

The Growth of Services

One of the great boons of the growth of XML has been the way in which it has transformed integration - increasingly we’re seeing information flowing around the Internet as a set of self-describing documents that contain their own context.  Many of the services that process these documents may be automated - like many SOA or Web 2.0 services that we see today - but many will actually be routes to people (like Amazon’s Mechanical Turk – genius).  In this post I wanted to just explore some of the implications of this, since I believe that human-provided services will come to have very different characteristics from those provided by machines. 

The Growth of Drudgery

Over the last twenty years we have seen increasing concentration on the automation of transactional tasks (i.e. things which can be ‘proceduralised’ and repeated).  This has yielded enormous cost and efficiency benefits but we’re increasingly finding ourselves passing those benefits onto consumers.  As a result cost-cutting and efficiency are becoming a way to remain in the game rather than a source of competitive advantage.  In fact many people - most famously Nick Carr - argue that the use of IT to automate rote processes no longer yields any opportunity for differentiation, since all companies have access to the same technical capabilities and the work being automated is itself highly proceduralised and therefore by it’s nature easy to replicate.  Part of the rush to automate and proceduralise, however, has seen organisations place considerable emphasis on ‘workflow’ technologies that attempt to box people into specific tasks within an overall process, with little chance for innovation or creativity.  Such tools generally emphasise control and repeatability over contextual decision making - in a Taylorist mould - and therefore limit the freedom of the people and teams receiving work requests.  As a result we have seen a huge increase in the amount of work that is fixed, detailed and monitored, resulting in what I would call drudgery. 

Talent as the New Competitive Differentiation

In parallel with the realisation that increasing automation is reducing the scope for differentiation in transactional processes we have started to realise that - unsurprisingly - real, hard to recreate differentiation is found in the knowledge and talent of our people.  Most organisations make little or no attempt to recognise and promote the value of talent, however, and in fact encase people within industrial age processes that attempt to control their every response irrespective of context, reducing the scope for them to creatively tackle problems and find solutions whilst simultaneously reducing the adaptability and initiative of the enterprise as a whole.  Despite the continued belief in many companies that ‘the centre’ knows best, we’re actually seeing a movement towards increasing federation; as the world becomes a more uncertain and dynamic place, so the ability to compose the ‘perfect’ plan from the centre and then just get people to execute it breaks down completely.  As a result we need to move decision making to the edges of process boundaries through the coordination of loosely coupled services.  The benefit of this federated approach is that it moves the decisions on how to meet goals much closer to the people that have the knowledge to best decide what needs to be done whilst maintaining governance through concentration on outputs and KPIs in place of tight control of the tasks completed.  This is a major shift in management philosophy, relying as it does on service contracts, trust and a belief in people in place of absolute control of every stage of a process.  The return, however, is that we get more appropriate, adaptable and innovative services that leverage the wealth of human talent available to us in place of smothering it. 

As a result one of the major tasks facing all industry sectors is to yes, continue to automate rote tasks to drive efficiencies but also to realise that real, difficult to replicate competitive advantage will come from the way in which we allow creative people to use information at the edge in providing services both to customers and back into the organisation.  As transactional jobs are increasingly automated, outsourced and commoditised (since it is difficult to create non-replicateable differentiation in these due to the fact that such processes can be easily codified and replicated) we will need to break down our old notions of control and embrace trust based models to support creative, customer relationship or problem resolution processes where real differentiation will be delivered which is difficult to recreate (due to the fact that it is dependent on talent, contextual awareness and experience – three things which cannot be codified).

The Alternative Growth of Extreme Federation

Whilst much of this work has been going on we’ve seen an alternative approach to command and control workflow systems growing on the Internet.  Increasingly complex applications, businesses and social networking sites have sprung up from sets of loosely coupled people and systems with the emphasis much more firmly placed on expectations and trust.  In such extreme federation people concentrate on the required outcomes whilst wholly devolving the way in which those outcomes are achieved to friends, partners or 3rd parties.  In solving particular problems, people come together within newsgroups, on wikis, via blogs or (even - shock) via email, collaborating to solve problems together within their sphere of expertise.  One of the most extreme examples of such community based activity is the open source software movement, where high levels of trust and collaboration yield levels of innovation, creativity and quality that often outperform traditional command and control methods of management within product development companies.  One of the key aspects of this movement, however, is the leveraging of human capital to take advantage of the well of creativity and talent that this represents instead of merely seeing people as drones to effect some part of a pre-planned process.  Many of these changes have resulted in the emergence of the group of concepts known as Web 2.0, with key elements that support people, their collaborations and their access to services in a personalised way.  By supporting such collaboration these technologies are ushering in an age of accelerated innovation, by bringing together many experiences and perspectives and allowing the people involved to interact, argue and learn, building new value faster than ever before.

Human Vs Machine

In considering these two different models we see two alternative needs:

• Codification and automation of repeatable processes to gain cost and efficiency benefits;
• Leverage of the talent and expertise of people to create differentiation.

These two dimensions represent very different kinds of problem areas; as automation increasingly erodes our ability to differentiate through cost and efficiency so the ability to best leverage the talent of our people - which is impossible to codify and replicate - becomes paramount.  As a result we need to concentrate on approaches to service implementation that use technology to automate codified processes whilst delivering collaborative and supportive tooling to enable creative problem solving by people in place of drudgery.  In short we need to have the conceptual notion of services as a unifying set of abstractions around which to build loosely coupled systems, backed up by SOA platforms on which to build automated service realisations and the human-centric view promoted by Web 2.0 to implement those services that are realised by people.

The New People-Oriented Workflow

As a result of these drivers I believe that we will rapidly see much greater emphasis on improving the way that we carry out complex, uncertain tasks which only people can perform.

Whilst transactional tasks will require technology to receive and automatically process the information in well-defined ways, I feel that the main role of technology in human-based services will increasingly be to ‘get out of the way’ and enable people to work creatively to meet their obligations rather than direct them through a fixed set of tasks. I see documents (which is all the contents of a SOAP message or suchlike are) increasingly being surfaced to people in ways that enable them to work directly with the data in a meaningful way; this could be as simple as displaying it using forms technology (rendered at the point of contact using metadata external to the document) or as complex as placing it into a collaboration platform (social software like wikis for example) that enable groups of people to work together. In either case we need tools that enable people to exercise discretion and creativity in responding to service requests whilst supporting collaboration and policy adherence.  This will allow people to work freely on complex problems that require judgement, experience and discretion without the kinds of constraints placed on them by traditional workflow tools and by extension enable a more adaptable, fit for purpose enterprise.

When we look at the majority of current ‘workflow’ tools that dominate the market, however, we see that they are anathema to this model – tightly proceduralised transfer of information through a codified workflow assumes exactly the sort of tasks that should be automated and it doesn’t readily support open, collaborative work between creative individuals; rather it forces them to perform ‘work’ or ‘tasks’ within a set procedure. Such procedures are often overly rigid, quickly overtaken by events and therefore mostly moribund; most importantly, however, they remove the element of human judgement and creativity from the enterprise - increasingly our major source of differentiation.

As a result of these tensions I see Web 2.0 technologies increasingly permeating the enterprise to deliver more lightweight, human-centric support to service realisation, with wikis, presence information, tagging and blogs coming to the fore in an enterprise process and knowledge management context in place of current expensive and ineffectual workflow and knowledge management products.  Whilst SOA technologies will be used to implement both automated services and the macro processes that bring services together, we will use Web 2.0 to deliver a more flexible way of surfacing work to human beings that allows them to meet their obligations in a way that is supportive, easy to use and collaborative.

In addition to these emerging uses within the enterprise such collaborative spaces will become ever more important as customers and partners become codevelopers of our products and services.  The old four walls of the enterprise are rapidly breaking down in the face of decreasing collaboration costs, increasing specialisation and the benefits of what John Hagel and John Seely Brown call ‘productive friction’.  Web 2.0 collaboration technologies can therefore also amplify the benefits of the human talent we have available by linking it to other talent across the whole value chain, leading to more rapid improvement and innovation through diverse experiences and perspectives.

The Death of Process?

On googling around on this subject whilst writing this post I came across a conversation a year ago where Ross Mayfield hypothesised that the types of collaboration that I’m talking about could destroy the whole basis for ‘business process’ by allowing people to rapidly link to the expertise and information necessary to flexibly respond to the demands placed upon them.  This generated a whole slew of comments both for and against and to be clear I just want to state that I do not believe this to be the case.

I feel that far from being less relevant business processes are becoming ever more visible and relevant in the highly connected world that we’re creating - as we design enterprises that leverage business capabilities both within and without the boundaries of the traditional four walls, so we need systematic ways of controlling this coordination to achieve known outputs.  Rather than removing the need for process I think that collaboration technology merely gives us an alternative way of enabling the better leverage of human talent within this systematic framework; at the end of the day we still want to maximise the levels of automation that we can bring to the codifiable elements of our capabilities.  As a result I think we need to look at three main elements in order to understand the connection between these things: 

• Macro processes that describe how capabilities are coordinated in order to realise a value chain;
• Codifiable micro processes that may be used to implement the automatable capabilities; and
• Open, collaborative service realisation technologies - as previously discussed - that enable people to receive service requests in a way which maximises their flexibility and discretion.

Even in this latter category we are not wholly free of the need to adhere to organisational policies - whilst we want to give people the freedom to tackle problems in the best way possible, we also still need to ensure that they do so in a way that is consistent with the policies of the organisation.  Whilst many of these policies can be codified within the service contract that describes the outputs, there will likely also be a need for post-condition type policy evaluation, something discussed frequently by Keith Harrison-Broninski in his work on Human Interaction Management, for an example see here.

As a result I believe that rather than lead to the death of process, collaborative technologies will enhance our ability to deliver flexible processes that fully leverage our available talent by placing human work within the context of a wider value chain in a way that is supportive rather than directive.

Summary

In this post I’ve discussed some of my thoughts around the need to support innovation within human processes by leveraging flexible, collaborative and lightweight tools in place of rigid task-based, automated workflows imposed from the centre.  Such an approach recognises the increasing need for adaptability at the edge and the increasing importance of amplifying human talent as a source of competitive differentiation.  I have also stressed that such tools are a component part of a wider services architecture, however, that has been systematically designed in order to deliver loose organisational coupling; Web 2.0 tools are not a replacement for processes but rather a means of increasing the flexibility of those capabilities within a systematic process that are delivered by people.  In either case my feeling is that we need to move away from an emphasis on tight control and towards a trust and output based model of governance that uses SOA and Web 2.0 to maximise the use of our available talent by allowing people to collaboratively do what needs to be done at the point of need.  Like I said, an end to drudgery.

:: :: :: :: :: :: :: :: :: :: ::

The Folly of BPM

Just come across a post by Steve Jones that discusses the folly of equating BPM (or even business processes) as the top level of engagement that any IT provider can have with a customer.  The phrase he uses to debunk this is ‘Bollocks’ (a lovely British phrase).

That’s hilarious. I couldn’t agree more. My whole view of the world aligns very definitely with Steve’s in that I see surfacing what an organisation does through the business capabilities that it has (in his parlance the business service architecture) and then deciding how you want to go about realising these. To be honest BPM (and processes in general) are a dangerous distraction at the top level of dialogue since they immediately dive into the detail of ‘how’ to do things before people have even decided whether it’s a good idea by understanding the need for - and the value of - the capability that the process will support. When you’re looking at the detail of ‘how’ something is done it’s all too easy to feel that something is both important and necessary and thereby waste an absolute fortune creating or improving pointless processes. As a result I wholeheartedly agree with Steve.

I’ve discussed my view on this stuff a lot, with perhaps the most relevant post being here

:: :: :: :: :: :: :: :: :: ::

Building the Internal SOA Marketplace

The Lack of Cost Transparency

One of the drivers for looking towards increased service-orientation is the pursuit of transparency of cost. There are two issues that currently lead to a lack of transparency:

1. Businesses are not designed as systems and thus we have functional or product based silos that make it difficult to understand the cost of providing business capabilities; and
2. Implementation processes and supporting IT systems are delivered as individual applications within these mis-aligned business silos, resulting in even greater opacity.

This situation means that most organisations have no idea how much any given process or capability actually costs them and hence no worthwhile metrics to help them improve.

Service-Orientation for Business Design

As I’ve discussed previously, it is interesting to note that the concept of service-orientation has nothing to do with technology but is rather a structural abstraction.  As a result you can use the ideas of loose coupling, defined contracts and quality of service within business design as well as within IT - in fact service-orientation makes no sense when this business context is absent.  Service-orientation thus delivers a coherent method for the structuring of a business as a system, with services offered by business capabilities forming the system elements that are coordinated in the pursuit of value.

Service-based Costing

If we use these concepts to redesign the enterprise as a set of services realised by a combination of people, technology and processes then the costs of a given business service are the combined costs of these elements as they interact to deliver to the specified contract. This means that costs are now available at the business service level; i.e. for each service that the enterprise wishes to use they have a specific cost associated with it. When a number of business services are linked together, the cost of the aggregating service needs to be pitched by the owner at such a level that they can profitably aggregate the consumed services in order to deliver to their contract. 

Procuring Services

The expression of an enterprise as a number of collaborating business capabilities allows the organisation to consider how it procures the necessary functions; one option is to deliver a single service that supports the necessary function for the whole enterprise, a second might be to outsource the execution of the service to a specialised provider, whilst a third could be the use of multiple internal and external providers that compete for individual service fulfillment requests on the basis of cost and quality of service. In reality there are likely to be a variety of models applicable.

Utility Business

In order to enable the final transparency, however, and deliver utility style costs to the enterprise, each business capability owner will need to be able to charge for the usage of their services on a per transaction basis. For example, human resources will need to be able to charge for recruitment, induction or dismissal services. This will enable consuming capabilities within the enterprise to be billed for the specific services that they consume and thus see where their major costs lie. Such a model will enable capability owners to decide whether internal service providers represent a good deal or whether they should look to leverage the services of external specialised providers.  In addition service providers will have clear metrics for the costs that they charge to the rest of the enterprise, a view on the competitiveness of their services and a clear sphere of control within which to innovate in order to increase cost-effectiveness. Some service providers may even begin to charge a markup to reinvest in the improvement of their services rather than simply charge at cost.  In point of fact I believe that this will increasingly be the case (even within a given enterprise), since such a market approach drives the sustainability (or not) of a service by forcing it to be competitive or driving it out of the organisation and into the hands of an external provider.  For those capability owners who do succeed in making the transition, the opportunity will be there for them to offer their capabilities to external partners and grow their services as an independent business, taking advantage of greater economies of scale and scope.

Governing the Utility Business

As we increasingly conceive of an enterprise as a set of business capabilities so the focus of governance moves away from inappropriate concentration on technology and onto the portfolio of services that the enterprise needs to realise its goals.  In this scenario the implementation of business capabilities is wholly federated to the capability owners who have the right to decide how they implement the services that they offer to the rest of the organisation, resulting in a cascading (and greatly simplified) ‘business architecture’ where organisations only scope and govern services at the level at which they consume them, leaving onward decisions within the remit of their service providers.  This might seem to fly in the face of traditional notions of SOA where people bemoan the fact that most organisations are ‘project-centric’ and that there should be less ’selfishness’ but the fact of the matter is that if we charge people with delivering certain value - and we judge them on their performance -then the natural tendancy of human beings is to exclude anything that is orthogonal to the task at hand.  This means that rather than force people into situations where they come into conflict over differing priorities we need to realign the way in which people are judged - by charging them with the delivery of a service that meets the needs of the organisation as a consumer - whilst giving them total autonomy over how that service is delivered.  This delivers the best of both project and enterprise thinking by more appropriately aligning people’s responsibilities around system boundaries whilst satisfying their need to control their own destiny (this, incidentally, is a subject that was explored more lucidly by Todd Biske here!).  Such a model also helps to remove the tensions traditionally associated with inter-departmental integration - discussed by Joe McKenrick here - by realigning departments around the services that they provide, again moving the focus to system boundaries in place of overlapping hierarchies.

This will be a significant shift away from current, immature models of governance which focus on forcing shared IT services, applications and infrastructure onto business capability owners, thereby locking them into potentially inappropriate solutions and preventing them from changing independently.  In the business capability model we concentrate on discovering and managing those services that have value to the business and rely on market forces to drive capability owners to share (or not) services that can be conceived of and offered by third parties in a way which is monetisable and thus sustainable.  This covers the other points raised by Joe around governance given that the answer to all of the questions about ownership, charging, change, etc. are answered by the fact that there is a commercial (or at least pseudo-commercial) agreement between the consumers and providers of service within the enterprise, again driving a market approach in search of transparency and sustainability.  In either case sustainability is the key concern here, as current notions of reuse based on small granularity services are not sustainable given that they cannot generate sufficient revenue to ensure their own survival whilst they simultaneously place capability owners into lockstep for little value gained.  In this scenario the IT department is essentially levying a tax on the business in order to unfairly prop up a collection of services that result in an inflexible environment for the business capability owners,

Utility Platforms for Utility Business

In order to enable a more sustainable model based on market forces, service realisation platforms will need to demonstrate a number of characteristics:

1. They must be cost-effective:  As capability owners start to surface the costs that they incur in offering their services, so the cost of the supporting IT systems will come to light.  In many organisations IT has been seen as an unpleasant but necessary evil with the costs dispersed across the organisation in such a way as to make it feel like an ungovernable black hole.  If you suddenly find yourself tasked with delivering a set of services competitively, however, and find that your specific share of the IT costs makes you uncompetitive in doing so you are going to be very concerned.  This will force capability owners to take a long hard look at the costs that they incur and look for ways of reducing IT expenditure.  In this scenario it is likely that they will begin to look at shared platforms from utility computing providers in place of expensive, home grown, inflexible infrastructures and architectures that don’t support them in delivering, robust, monetisable services;
2. They must support service provision: As well as requiring increased quality and robustness, providing services to consumers requires that there be a host of supporting functions in place, including management, reporting and monetisation.  As a result any service delivery platform  must include the ability to capture complex service events for service level management, audit and metering and billing.  Without such functions service providers will be unable to scalably serve their consumers and ensure that they are invoiced for the functionality that they use.  Current enterprise IT infrastructures (and most middleware products) do not support any notion of service delivery at this level of maturity and thus the likely platforms for such service enablement will be SOA-aware SaaS platforms - like the kind described frequently by Phil Wainewright (for example see his discussion of OpSource here) - delivered either centrally by specialised utility computing providers or within the enterprise data centre by such providers as a special service.

Summary

Service-orientation represents a new way to describe the way in which an enterprise functions. If pursued successfully a service-oriented business design can open the way for increased transparency of costs, greater competition and new business opportunities. Any vendor who wishes to help companies with this transition will need to ensure that they have adequately catered for the needs of service delivery within their SOA platform, however.  Of particular concern are a coherent monetisation strategy along with underlying charging and invoicing capabilities, as I believe that ensuring the provision of monetisable services is the only sustainable way of creating and maintaining a portfolio of business services that are exposed to market drivers and hence forced to be competitive. In addition such a market-led approach introduces more transparent governance, focusing on business capabilities and forcing capability owners to deliver clear and competitive services against which they are judged, hiding the complexity of implementation - where most people try to address governance today - and governing more appropriately on the outputs required.

I think I’ll return to the areas of both governance and sustainability in the near future as I believe that they are areas of significant weakness in most current SOA proposals and thus worthy of further discussion.

:: :: :: :: :: :: :: :: :: ::

Capabilities vs ERP

Just picked up on a post by Clive Keyte (here) that discusses the selling of SAP as a set of services.  The post was in response to one by Jeff Kaplan, where Jeff argues that SAP needs to learn some lessons about service enablement and the fostering of an ecosystem from more agile companies - more specifically Salesforce.com (here).

There were two points that I found interesting in the ensuing discussion - 1) that Jeff believes that the value lies in services that can be leveraged to build better end propositions rather than in monolithic applications with outdated licensing models (my emphasis given my biases ;-)) and 2) (and perhaps more interesting) Clive’s response about TDS building a set of services that ‘enwrap’ SAP.  I was initially quite excited about this as I thought that this scenario might mirror some of my comments a couple of weeks ago (here) where I was considering how the ability to buy services from specialised providers impacts companies like SAP who attempt to sell applications that are a) tightly integrated (and thus do not promote process agility) and b) representative of the commodity processes that people don’t focus on.  I thought that TDS had taken SAP and were selling commodity services (i.e. complete propositions) whilst using SAP as the backend to enable them to deliver.  Turns out, however, that TDS are essentially selling an ASP proposition into the mid-market based on a set of applications that they have created that use SAP as the backend.  Nothing wrong with that but it isn’t the big deal I initially thought.

On idly looking at the SAP website for BPO propositions, however, I did discover a list of providers who use SAP within their offerings rather than sell SAP into end organisations.  This is much more like the shift I had in mind and effectively changes the whole consumption model from many-installation to single-installation.  Essentially SAP disappears behind people who provide commodity services and is no longer needed in the end enterprise.  To me this makes much more sense than attempting to sell applications into end organisations (whether on a SaaS or on-premise basis) since as I’ve argued before such applications represent commoditised capability that I may as well buy from someone else if it’s not an area of concern for me.  Currently these services are mostly based around the usual BPO suspects like Human Resources - which have a broad horizontal appeal - but I believe that we’re rapidly going to see a shift towards the outsourcing of much more specialised (and potentially differentiating) capabilities, with complex value chains spanning multiple providers.

If this model proliferates - as I believe it will - then how long will it be before specialised providers decide that they too need more flexibility in how they deliver?  Which other modules - beyond HR - are good candidates as a starting point for the creation of specialised services?  Are ERP packages - given the breadth of their function and the difficulty of breaking them up - too expensive and inflexible to play within a specialised service provider market?  How will a shift towards service provision affect the revenue streams of ERP providers given a consolidation of the supporting applications into fewer end organisations?  All interesting stuff I guess - any ideas would be gratefully received

:: :: :: :: :: :: :: :: :: ::