Evolution and IT

3 Jun

This is a subject that has been on my mind a lot lately as I recently read an astounding book by Eric D. Beinhocker called “The Origins of Wealth”.  It was astounding to me for the way in which Beinhocker imperiously swept across traditional economic theories based on equilibrium systems, critiqued the inherent weaknesses of such theories when faced with real world scenarios and then hypothesised the use of the evolutionary algorithm as a basis for a fundamental shift to what he called ‘complexity economics’.  I’m going to return to discuss some of the points from this book – and the way in which they resonated with my own thoughts around business design, economic patterns and technology change – but for today I just wanted to comment on a post by Steve Jones where he raises the issue of evolution in the context of IT systems.

Steve’s question was whether we should “reject evolution and instead take up arms with the Intelligent design mob”.  His thoughts have been influenced by the writing of Richard Dawkins, in particular the oft-times contrast between the apparent elegance of the external appearance of an animal (including its fitness for its environment) with the messy internals that give it life.  Steve suggests that he sees parallels in the IT world and brings this around to issues with the way in which a shift to service-based models often creates unfounded expectations on internal agility:

“The point is that actually we shouldn’t sell SOA from the perspective of evolution of the INSIDE at all we should sell it as an intelligent design approach based on the outside of the service. Its interfaces and its contracts. By claiming internal elements as benefits we are actually undermining the whole benefits that SOA can actually deliver.”

In the rest of the post and into the comments Steve then extends this argument to call for intelligent design (of externals) in place of evolution:

“The point I’m making is that Evolution is a bad way to design a system the whole point of evolution is that of selection, selection of an individual against others. In IT we have just one individual (our IT estate) and so selection doesn’t apply.”

My own feeling is that there isn’t a direct 1:1 relationship in thinking about evolution and the difficulties of changing the internals of a service in the way that Steve suggests.  I believe that evolution is a fractal algorithm whose principles apply equally to the design of business capabilities, service contracts and code.  To think about this more specifically I’d like to consider a number of his points after first considering evolution and how we frame it more broadly from a market and enterprise context.

What is evolution?

Evolution is an algorithm that allows us to explore large design spaces without knowing everything in advance.  It allows us to try out random designs, apply some selection criteria and then amplify those characteristics of a design that are judged as ‘fit’ by the environment (i.e. the selection criteria).  In the natural world evolution throws up organisms that have many component traits and success is judged – often brutally – by how well these traits enable an animal to survive in the environment in which it exists.  Within an individual species there will be a particular subset of traits that define that species (so traits that govern size, speed or vision for instance).  Individuals within a species who have the most desirable instances of these traits will be better equipped to survive, the mating of these individuals will merge their desirable traits and over time the preponderance of the most effective traits will therefore increase in the population overall.  As a result evolution creates a number of designs and uses a selection algorithm to more rapidly arrive at designs that are ‘good enough’ to thrive within the context of the environment in which they exist.  It is a much more rapid method of exploring large design spaces than trying to think about every possible design, work out the best combination of traits and then create the ‘perfect’ design from scratch (i.e. “intelligently” design something without a full understanding of the complexities of the selection criteria and hence what will be successful).

Enterprises and evolution

In Beinhockers book he uses a ‘business’ as the unit of selection that operates within the evolutionary context of the market.  Those businesses with successful traits are chosen by consumers and thus excel.  These traits – whether they be talent strategies, process strategies or technology strategies – are then copied by other businesses, replicating and amplifying successful traits within the economic system. 

I believe that this is the best approximation that we can use in the – rather unsystematic – businesses that exist today but that we can use systematic business architecture to do better.  I have often written about my belief in the need for companies to become more adaptable by identifying and then reforming around the discrete business capabilities they require to realise value.  Such capabilities would form a portfolio of discrete components with defined outcomes which could then be combined and recombined as necessary to realise systematic value streams. 

Such a shift to business capabilities will allow an enterprise to adapt its organisation through the optimisation and recombination of components; whilst at this stage of maturity Beinhocker’s hypothesis of the ‘business’ as the element of selection remains sound (since capabilities are still internal and not individually selectable as desirable traits) we can at least begin to talk about capabilities and the way in which we combine them as the primary traits that need to be ‘amplified’ to increase the fitness of the design of our business. 

Inside Out 

Whilst realigning internal capabilities is a worthwhile exercise in its own right, evolutionary systems also tend to exhibit long periods of relative stability punctuated by rapid change as something happens to alter the selection criteria for ‘fitness’.  The Web and related techniques for decomposition – such as service-orientation – have made it possible to consume external services as easily as internal services.  Business capabilities can thus be made available by specialised providers from anywhere in the world in such a way that they can be easily integrated with internal capabilities to form collaborative value webs.  We can therefore view the current convergence of business architecture, technology and a mass shift to service models as a point of ‘punctuated equilibrium’. 

In this environment continuing to execute capabilities that are non-differentiating will cease to be an attractive option as working with specialised providers will deliver both better outcomes and more opportunities for innovation.  From an evolutionary perspective our algorithm will continue to select those organisations that are most fit (as judged by the market) and those organisations will be those with the strongest combination of traits (i.e. capabilities).  Specialised, external capabilities can be considered to be more attractive ‘traits’ due to their sharp focus, shorter feedback loops and market outlook; they will thus be amplified as more organisations close down their own internal capabilities and integrate them instead, a kind of organisational mutation caused by the combination of the best capabilities available to increase overall fitness.    Enterprises working with limited, expensive and non-differentiating internal capabilities will risk extinction.

Once this shift reaches a tipping point we discover that business capabilities become the unit of market selection since they are now visible as businesses in their own right.  Whilst this could be considered pedantry – as a ‘business’ is still the unit of selection even though what we consider a business has become smaller – there is an important shift that happens at this point.  Essentially as business capabilities become units of selection in their own right the ‘traits’ for selection and amplification of their services become a combination of their own internal people, process and technology capabilities plus the quality of the external capabilities they integrate.  Equally importantly they have to act as businesses – rather than internal, supporting organisations – and support the needs of many customers – and hence support mass customisation.  This will mean that they will have many more consumers than internal support functions would ever have had and the needs of these consumers could be both very different and impossible to guess in advance; there will be new opportunities to rapidly improve their services based on insight from different industries, orthogonal areas and new collaborations.  An ability to respond to these new opportunities by changing their own capabilities or finding new partners to work with will be a significant factor in whether these capabilities thrive and are thus judged as ‘fit’ by the selection criteria of the market.  An ability to evolve externally to provide the ‘right’ services will thus be a core competency required in the new world. 

What has this got to do with services?

The basic points I’m making here are that evolution acts at the scale of markets and is a process that we participate in rather than a way of designing.  We design our offers using the best knowledge that we have available but the market will decide whether the ‘traits’ we exhibit fit the selection criteria of the environment.  Business capabilities can become the ‘traits’ that make particular market offers (or businesses) fit for selection or not by having a huge influence over the overall cost, quality and desirability of a particular offer.  From a technology perspective such capabilities will in reality need to offer their services as services in order for them to be easily integrated into the overall value webs of their customers and partners; in many cases there may be a 1:1 mapping between the business capability and the service interface used to consume it.  In that sense services are just as much a driver of fitness in the overall ecosystem and their interface and purpose will inevitably need to change as the overall ecosystem evolves.  Hence it is not simply a question of ‘fixing’ interfaces and ‘evolving’ internals; the reality is that the whole market is an evolutionary system and businesses – plus the services they offer for consumption –will need to continually evolve in order to remain fit against changing selection criteria.

Intelligent design or evolution

The core question raised by Steve is whether ‘evolution’ has any place in our notion of service design.  In particular:

“The point I’m making is that Evolution is a bad way to design a system the whole point of evolution is that of selection, selection of an individual against others. In IT we have just one individual (our IT estate) and so selection doesn’t apply.”

Is evolution a ‘bad’ designer?

I do not believe that evolution is either a good or a bad designer but it is a very successful one.  Evolution is an algorithm that takes external selection criteria, applies them and amplifies those traits that are most successful in meeting the criteria.  It is brilliant at evaluating near-infinite design spaces (such as living organisms or markets) and continually refining designs to make them fit for the environmental selection criteria in play.

If I read Steve’s post correctly, however, he actually isn’t objecting to the notion of evolution per se – since at the macro level it is a market process in which we are all involved and not a conscious way of designing our services – but rather to a lack of design being labelled an ‘evolutionary’ approach. 

What is ‘evolutionary’ design?

In the majority of cases when people talk about ‘evolution’ in the context of systems they really mean that they want to implement as quickly and cheaply as possible and then ‘evolve’ the system.  Often vendors encourage this behaviour by promising that new technologies are so rapid as to make changes easy and inexpensive.  Such approaches often eschew any attempt at formal design, choosing instead to implement in isolation and then retro-integrate with anything else on a case by case basis.  I have often seen Steve talk about the evils of generating WSDL from code and I imagine that this is the sort of behaviour that he is classifying as ‘evolutionary’ changes to internals.

Is this a good or a bad approach?  From an evolutionary perspective we can say that we do not care.  Given that we are talking about evolution in its true sense the algorithm would merely continue to churn through its evaluation of services, amplifying successful traits.  It is just that such behaviour might have some unrecognised issues: firstly evolution would have to work for longer to bring a service to a point at which it is ‘fit’, secondly the combination of all of these unfit services means that there is a multiplier effect to evolving the ecosystem of services to a point at which it is fit overall and thirdly whilst all of this goes on at a micro level the fitness of the enterprise against the selection criteria of the market might be poor due to the unfitness of some of its major ‘traits’.

Intelligence in design

Whilst a lack of design might extend the evolutionary process to a point at which it is unlikely that a business could ever become fit before it became extinct, an assumption that we can design service interfaces that are fixed also ignores the reality of operating in a complex evolutionary system (like a business). 

Creating a ‘perfect’ service from scratch is a very difficult thing to do as even within the bounds of a single organisation we cannot know all of the potential uses that might come to pass.  We can however use the best available data to create an approximation of the business capabilities and resulting services required in order to try and speed up the evolutionary process by reducing the design space it has to search.  Hence the notion that we use an evolutionary process of service design (a bit like I discussed here) is an important one; often people will not know what good looks like until they see something.  Whilst I therefore accept that we can start with an approximation of the capabilities (and services) we believe we will need we have to accept that these will evolve as we gain experience and exposure to new use cases.  

From this perspective I don’t agree with the literal statement that Steve has made; it is not about intelligent design vs evolution but rather about intelligence of design to support the evolutionary process.  As I stated previously markets are fundamentally evolutionary systems and therefore our businesses – and the business capabilities and services that represent their traits – are assessed by the evolutionary algorithm for fitness against market selection criteria.  We are not dumb observers in this process, however, and must fight to create offers that are attractive to the market along with supporting organisations that enable us to do it at the right price and service levels.  We can apply our intelligence to this process to increase our chances of success but a key element will be to understand that our enterprises will increasingly become a value web of individual capabilities, that it is the combination of our capabilities that is judged and that we must therefore design our organisations to evolve by adopting successful traits to improve our overall fitness.  As a result we should not expect the evolutionary process to do our work for us – by choosing not to apply any intelligence in design – but we should also not assume that evolution has no place in design given that meeting its demands is becoming the primary requirement of business architecture.

Macro evolution in the economy

Stepping back and taking an external perspective leads us to realise that it is also untrue to say that we only have one individual (in terms of a single IT estate) and that there is nothing to select against; in reality even today we are competing for selection against businesses with other IT estates and thus our ‘traits’ (in the form of our IT) are already a major factor in deciding our fitness (and thus our ability to be ‘selected’ by the evolutionary algorithm of the market).  If we factor in the emerging discontinuities we see as part of the ‘punctuated equilibrium’ process it only makes things worse; the specific IT we have within specific business capabilities will have a large impact on the fitness of these capabilities to survive.  In that context continually evolving our business capabilities (and with them the IT and software services that enable them) is the only way to ensure future success.

More importantly as we look at the wider picture of the position of our business capabilities within the market as a whole so our unknowns become more acute and the more that we can only rely on selection and amplification (i.e. evolution) to guide us in shaping them.  Looking beyond the boundaries of our single organisation we have to consider the fact that all of our services will exist in a market ecosystem many of whose needs and usage we are even less equipped to know in advance.  There will often be new and novel ways in which we can change our services to meet the emerging needs of customers and partners and in this way the overall ecosystem itself will evolve.  As a result selection is the only way in which design can occur in an ecosystem as complex as a market where there are many participants whose needs are diverse.  Nobody can ‘intelligently design’ a whole market from top to bottom.  Furthermore the market – as an evolutionary system – will be subject to a process of ‘punctuated equilibrium’ meaning that sudden changes in the criteria used to judge fitness can occur.  From an IT perspective the shift towards service models such as cloud computing could be considered to be one such change, since it changes the economics of IT from one based on differentiation based on ownership to one based on universal access.  Such changes could be considered ‘revolutionary’ as the carefully crafted and scaled business models created during a period of relative stability cease to be appropriate and new capabilities have to be developed or integrated to be successful.  This is one area where I disagreed with Steve’s comment about the relationship between revolution and evolution:

“The point of revolutionary change is that it may require a drop back to the beginning and starting again. This isn’t possible in an evolutionary model.”

Essentially revolutionary change often happens in evolutionary systems – evolution is always exploring the design space and changes in the environment can lead to previously uninteresting traits becoming key selection criteria.  In this case ‘revolutionary change’ is a side-effect of the way in which evolution starts to amplify different traits due to the changes in selection criteria.  In the natural world such changes can lead to catastrophic outcomes for whole species whose specialisations are too far removed from the new selection criteria and this can also happen to businesses (it will be interesting to see how many IT companies survive the shift to new models, for instance).  Evolution also allows the development of new ‘traits’ that make us sustainable, however, and therefore can support us in surviving ‘revolutionary’ changes if we have sufficient desirable ‘traits’ to prevent total collapse.  The trick is to understand how you can evolve at the macro level to incorporate the changes that have occurred in the selection criteria of your market and to realign your capabilities as appropriate.  Often the safest way to do this is to have different services on offer that try different combinations of traits, hence keeping sensors within the environment to warn you of impending changes. 

Summary

As a result there is no question that both evolution and intelligence in design have a place in the creation of sustainable architectures (whether macro business architectures or micro service architectures).  We have to be precise in the ways in which we use this language, however; it is not sufficient to label a lack of design as ‘evolution’ (which I believe was Steve’s core point).  Evolution is a larger, exogenous force that shapes systems by highlighting and amplifying desirable traits and not something that we can rely on to reliably fix our design issues without an infinite amount of time and change capability.  We therefore need to apply intelligence to the process of design – even when there is great uncertainty – to try and narrow down the design space to minimise the amount we have to rely on evolution to arrive at a viable ‘system’; even once we get to this point, however, we need to be aware of the fact that evolution is an ongoing process of selection and amplification and design our business architectures with the flexibility necessary to recognise this fact.

More broadly I believe that we can also look at the application of ‘intelligence’ and ‘evolution’ as a matter of scale;  we can design individual services with a fair degree of intelligence, we can design our business capabilities with some fair approximations and then rely on evolution to improve them but we can only rely on evolution to shape the market itself and thus the selection criteria that define our participation.  For this reason strategies that stress adaptability (i.e. an ability to evolve in response to changing selection criteria) have to take precedence over strategies that stress certainty and efficiency.

Advertisements

3 Responses to “Evolution and IT”

Trackbacks/Pingbacks

  1. Enterprise Architecture Top to Bottom « IT Blagger 3.0 - December 2, 2010

    […] have spoken on this issue in the past – albeit in a slightly different context (and also in discussing evolution and IT, actually).  I also feel strongly that EA practitioners should be focused far more on enabling a deeper […]

  2. Will CIOs Fail on Cloud? « IT Blagger 3.0 - July 4, 2011

    […] any kind of business service from external providers in order to integrate successful services and increase the ‘fitness’ of the overall organisation.  Establishing the right to do this first requires the CIO to take a leadership position in early […]

  3. Evolution and the IT Industry – Part I | IT Blagger 3.0 - October 25, 2013

    […] few years ago I wrote a (rather long) post about evolution in the context of business and in particular the use of emerging business […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: