I’d like to start by thanking the GigaOM team once again for inviting me to participate in the panel session: ‘The Gap Between Applications and Infrastructure’. It is the first time I’ve attend a GigaOM ‘Structure’ event; and it proved a worthwhile experience.
There were many topics, but the re-occurring theme focused upon the on-going shift towards ‘Cloud Computing’:
- Cloud ‘Business Models’: How ‘Agile’ organisations should off-loading non-critical IT.
- SaaS Use Cases: Again selling the concept of ‘Agile’ SaaS services for all those ‘Agile’ organisations.
- Cloud API’s – which is more open? Which is closest to Amazon? The next great lock-in?
- The role of the current dominant Cloud and Virtual Machine vendors and the rise of ‘Software defined Data Centres’.
All interesting topics. Yet I left the event further convinced that the IT industry continues to be too ‘fashion driven’. Too much evangelism, too little substance, a near criminal overuse of the word ‘Agile’. Beneath the sound-bites, an absence of vision from the dominant Cloud and Virtual Machine vendors.
However there were some important gems.
The analogy between data and mass was touched upon in one panel session: this a fundament, real-world, constraint. Quite simply; the more data you amass, the more energy / cost is required to manage it and move it. Frequently information has a time-to-live and only has value within a specific context. Hence aggregating vast amounts of data to drive centralised decision making processes is not necessarily the greatest of ideas: worth considering before embarking on your companies next ‘must have’ BigData project. For a good popular read which touches upon this area – try ‘Adapt‘.
So what does this imply for ‘BigData’ / ‘Cloud’ convergence?
- If information can be extracted from data in flight, then do so! When possible use CEP rather than Hadoop!
- It will frequently make sense to move processing to the data: not data to the processing.
- Sometimes there is no alternative. A centralised approach which aggregates data from remote sources may be the only approach.
- Analysis must be on an system by system basis. Data may flow from the Core to the Edge; or the data flow / interaction / may oscillate between core and edge.
It seems inevitable that data locality will be a primary forcing factor which will drive the shape of next generation ‘Data Clouds‘ solutions.
A second panel session, ever so briefly, touched on the importance of system modularity. In an animated, amusing and good natured argument between three Cloud Infrastructure vendors, points were attempted to be scored via a modularity argument. One of the solutions consisted of multiple independent sub-projects rather than a monolithic whole. Unfortunately that is as far as the argument went, the fundamental importance of modularity, at all structural levels was not discussed. Meanwhile the Java vendors who were branded ‘monolithic’; didn’t realise that the most powerful modularisation framework in existence – the OSGi framework – could have provide them – if they were using it – with a devastating response to the criticism.
‘times they are a-Changin’
Back-stage conversations were more interesting. There was an increasing awareness of an imminent inflection point in the industry. Environmental complexity is rapidly increasing; this as business systems evolve towards an increasingly intertwined ecosystem of services, resources and highly modular / maintainable components. It was increasingly understood that in this new world order; runtime dependency management would be an essential enabler.
The idea that evolvable, adaptive Cloud platforms will be highly modular and must have powerful runtime dependency management. So enabling:
- Applications to be dynamically assembled as required from fine grained components.
- In a manner influenced by the characteristics of the local host environment
- With middleware services provisioned as required.