Agility and Structural Modularity – part III

The first post in this series explored the fundamental relationship between Structural Modularity and Agility. In the second post we learnt how highly agile, and so highly maintainable, software systems are achievable through the use of OSGi.

This third post is based upon a presentation entitled ‘Workflow for Development, Release and Versioning with OSGi / Bndtools: Real World Challenges‘ (http://www.osgi.org/CommunityEvent2012/Schedule), in which Siemens AG’s Research & Development engineers discussed the business drivers for, and subsequent approach taken to realise, a highly agile OSGi based Continuous Integration environment.

The Requirement

Siemens Corporate Technology Research has a diverse engineering team with skills spanning computer science, mathematics, physics, mechanical engineering and electrical engineering. The group provides solutions to Siemens business units based on neural network technologies and other machine learning algorithms. As Siemens’ business units require working examples rather than paper concepts, Siemens Corporate Technology Research engineers are required to rapidly prototype potential solutions for their business units.

Siemens1

Figure 1: Siemens’ Product Repository

To achieve rapid prototyping the ideal solution would be repository-centric, allowing the Siemens research team to rapidly release new capabilities, and also allowing Siemens Business units to rapidly compose new product offerings.

To achieve this a solution must meet the following high level objectives:

  1. Build Repeatability: The solution must ensure that old versions of products can always be rebuilt from exactly the same set of sources and dependencies, even many years in the future. This would allow Siemens to continue supporting multiple versions of released software that have gone out to different customers.
  2. Reliable Versioning: Siemens need to be able to quickly and reliably assemble a set of components (their own software, third party and open source) and have a high degree of confidence that they will all work together.
  3. Full Traceability: the software artifacts that are released are always exactly the same artifacts that were tested by QA, and can be traced back to their original sources and dependencies. There is no necessity to rebuild in order to advance from the testing state into the released state.

Finally, the individual software artifacts, and the resultant composite products, must have a consistent approach to application launching, life-cycle and configuration.

The Approach

OSGi was chosen as the enabling modularity framework, this decision was based upon the maturity of OSGi technology, the open industry specifications which underpin OSGi implementations, and the technology governance provided by the OSGI Alliance. The envisaged Continuous Integration solution was based upon the use of Development and Release/Production OSGi Bundle Repositories (OBR). As OSGi artefacts are fully self-describing (Requirements and Capabilities metadata), specific business functionality could be dynamically determined via automated dependency resolution and subsequent loading of the required OSGi bundles from the relevant repositories.

The Siemens AG team also wanted to apply WYTIWYR best practices (What You Test Is What You Release). Software artefacts should not be rebuilt post testing to generate the release artefacts; between the start and end of the test cycle the build environment may have changed. Many organisations do rebuild software artefacts as part of the release process (e.g.1.0.0.BETA –> 1.0.0.RELEASE); this unfortunate but common practice is caused by dependency management based on artefact name.

Finally from a technical perspective the solution needed to have the following attributes:

  • Work with standard developer tooling i.e. Java with Eclipse.
  • Have strong support for OSGi.
  • Support the concept of multiple repositories.
  • Support automated Semantic Versioning (i.e. automatic calculation of Import Ranges and incrementing of Export Versions)- as this is too hard for human beings!

For these reasons Bndtools was selected.

The Solution

The following sequence of diagrams explain the key attributes of Siemens AG solution.

repo1

 

Figure 2:  Repository centric, rapid iteration and version re-use within development.

Bndtools is a repository centric tool allowing developers to consume OSGi bundles from one or more OSGi Bundle Repositories (a.k.a OBR). In addition to the local read-write DEV OSGi bundle repository, developers may also consume OSGi bundles from other managed read-only repositories; for example, any combination of corporate Open Source repositories, corporate proprietary code repositories and approved 3rd Party repositories. A developer simply selects the desired repository from the list of authorised repository, the desired artefact in the repository, dragging this into the Bndtools workspace.

Developers check code from their local workspaces into their SVN repository. The SVN repository only contains work in progress (WIP). The Jenkins Continuous Integration server builds, tests and pushes the resultant OSGi artifacts to a shared read-only Development OBR. These artefacts are then immediately accessible by all Developers via Bndtools.

As developers rapidly evolve software artefacts, running many builds each day, it would be unmanageable – indeed meaningless – to increment versions for every development build. For this reason, version re-use is permitted in the Development environment.

repo2

Figure 3:  Release.

When ready, a software artefact may be released by the development team to a read-only QA Repository.

repo3

Figure 4:  Locked.

Once an artefact has been released to QA it is read-only in the development repository. Any attempt to modify and re-build the artefact will fail. To proceed, the Developer must now increment the version of the released artefact.

repo4

Figure 5:  Increment.

Bndtools’ automatic semantic versioning can now be used by the developer to ensure that the correct version increment is applied to express the nature of the difference between the current WIP version and its released predecessor. Following the Semantic Versioning rules discussed in previous posts:

  • 1.0.0 => 1.0.1 … “bug fix”
  • 1.0.0 => 1.1.0 … “new feature”
  • 1.0.0 => 2.0.0 … “breaking change”

we can see that the new version (1.0.1) of the artifact is a “bug fix”.

The Agility Maturity Model – Briefly Revisited

In the previous post we introduced the concept of the Agility Maturity Model. Accessing Siemens’ solution against this model verifies that all the necessary characteristics required of a highly Agile environment have been achieved.

  • Devolution: Enabled via Bndtools’ flexible approach to the use of OSGi repositories.
  • Modularity & Services: Integral to the solution. Part and parcel of the decision to adopt an OSGi centric approach.

As discussed by Kirk Knoernschild in his DEVOXX 2012 presentation ‘Architecture All the Way Down‘, while the Agile Movement have focused extensively on the Social and Process aspects of achieving Agile development, the fundamental enabler – ‘Structural Modularity’ – has received little attention. Those of you that have attempted to realise ‘Agile’ with a monolithic code base will be all to aware of the challenges. Siemens’ decision to pursue Agile via structural modularity via OSGi provides the bedrock upon which Siemens’ Agile aspirations, including the Social and Process aspects of Agile development, can be fully realised.

Bndtools was key enabler for Siemens’ Agile aspirations. In return, Siemens’ business requirements helped accelerate and shape key Bndtools capabilities. At this point I would like to take the opportunity to thank Siemens AG for allowing their work to be referenced by Paremus and the OSGi Alliance.

More about Bndtools

Built upon Peter Kriens‘ bnd project, the industries de-facto tool for creation of OSGi bundles, the Bndtools GITHUB project was created by Neil Bartlett early 2009. Bndtools roots included tooling that Neil developed to assist students attending his OSGi training course and the Paremus SIGIL project.

Bndtools objectives have been stated by Neil Bartlett  on numerous occasions. The goal, quite simply is to make is easier to develop Agile, Modular Java applications, than not. As demonstrated by the Siemens’ project, Bndtools is rapidly achieving this fundamental objective. Bndtools is backed by an increasing vibrant open source community with increasing support from a number of software vendors; including long term commitment from Paremus. Current Bndtool community activities include support for OSGi Blueprint, stronger integration with Maven and the ability to simply load runtime release adaptors for OSGi Cloud environments like the Paremus Service Fabric.

Further detail on the rational for building Java Continuous Integration build / release chains on OSGi / Bndtools can be found in the following presentation given by Neil Bartlett to the Japan OSGi User Forum, May 2013: NeilBartlett-OSGiUserForumJapan-20130529. For those interested in pursuing a Java / OSGi Agile strategy, Paremus provide in-depth engineer consultancy services to help you realise this objective. Paremus can also provide in-depth on-site OSGi training for your in-house engineering teams. If interest in ether consulting or training please contact us.

The Final Episode

In the final post in this Agility and Structural Modularity series I will discuss Agility and Runtime Platforms. Agile runtime platforms are the area that Paremus has specialised in since the earliest versions of our Service Fabric product in 2004 (then referred to as Infiniflow), the pursuit of runtime Agility prompted our adoption of OSGi in 2005, and our membership of the OSGi Alliance in 2009.

However, as will be discussed, all OSGi runtime environments are not alike. While OSGi is a fundamental enabler for Agile runtimes,  in itself, the use of OSGi is not sufficient to guarantee runtime Agility. It is quite possibly to build ‘brittle’ systems using OSGi. ‘Next generation’ modular dynamic platforms like the Paremus Service Fabric must not only leverage OSGi, but must also leverage the same fundamental design principles upon which OSGi is itself based.

Agility and Structural Modularity – part II

In this second Agility and Structural Modularity post we explore the importance of OSGi™; the central role that OSGi plays in realising Java™ structural modularity and the natural synergy between OSGi and the aims of popular Agile methodologies.

But we are already Modular!

Most developers appreciate that applications should be modular. However, whereas the need for logical modularity was rapidly embraced in the early years of Object Orientated programming (see http://en.wikipedia.org/wiki/Design_Patterns), it has taken significantly longer for the software industry to appreciate the importance of structural modularity; especially the fundamental importance of structural modularity with respect to increasing application maintainability and controlling / reducing  environmental complexity.

Just a Bunch of JARs

In Java Application Architecture, Kirk Knoernschild explores structural modularity and develops a set of best practice structural design patterns. As Knoernschild explains, no modularity framework is required to develop in a modular fashion; for Java the JAR is sufficient.cover-small-229x300

Indeed, it is not uncommon for ‘Agile’ development teams to break an application into a number of smaller JAR’s as the code-base grows. As JAR artifacts increase in size, they are broken down into collections of smaller JAR’s. From a code perspective, especially if Knoernschild’s structural design patterns have been followed, one would correctly conclude that – at one structural layer – the application is modular.

But is it ‘Agile’ ?

From the perspective of the team that created the application, and who are subsequently responsible for its on-going maintenance, the application is more Agile. The team understand the dependencies and the impact of change. However, this knowledge is not explicitly associated with the components. Should team members leave the company, the application and the business are immediately compromised. Also, for a third party (e.g. a different team within the same organisation), the application may as well have remained a monolithic code-base.

While the application has one layer of structural modularity – it is not self-describing. The metadata that describes the inter-relationship between the components is absent; the resultant business system is intrinsically fragile.

What about Maven?

Maven artifacts (Project Object Model – POM) also express dependencies between components. These dependencies are expressed in-terms of the component names.

A Maven based modular application can be simply assembled by any third party. However, as we already know from the first post in this series, the value of name based dependencies is severely limited. As the dependencies between the components are not expressed in terms of Requirements and Capabilities,  third parties are unable to deduce why the dependencies exist and what might be substitutable.

It is debatable whether Maven makes any additional tangible contribution to our goal of application Agility.

The need for OSGi

As Knoernschild demonstrates in his book Java Application Architecture, once structural modularity is achieved, it is trivially easy to move to OSGi – the modularity standard for Java. 

Not only does OSGi help us enforce structural modularity, it provides the necessary metadata to ensure that the Modular Structures we create are also Agile structures

OSGi expresses dependencies in terms of Requirements and Capabilities. It is therefore immediately apparent to a third party which components may be interchanged. As OSGi also uses semantic versioning, it is immediately apparent to a third party whether a change to a component is potentially a breaking change.

OSGi also has a key part to play with respect to structural hierarchy.

At one end of the modularity spectrum we have Service Oriented Architectures, at  the other end of the spectrum we have Java Packages and Classes. However, as explained by Knoernschild, essential layers are missing between these two extremes.

diag6

Figure 1: Structural Hierarchy: The Missing Middle (Kirk Knoernschild – 2012).

The problem, this missing middle, is directly addressed by OSGi.

diag7

Figure 2: Structural Hierarchy: OSGi Services and Bundles

As explained by Knoernschild the modularity layers provided by OSGi address a number of critical considerations:

  • Code Re-Use: Via the concept of the OSGi Bundle, OSGi enables code re-use.
  • Unit of Intra / Inter Process Re-Use: OSGi Services are light-weight Services that are able to dynamically find and bind to each other. OSGi Services may be collocated within the same JVM, or via use of an implementation of OSGi’s remote service specification, distributed across JVM’s separated by a network. Coarse grained business applications may be composed from a number of finer grained OSGi Services.
  • Unit of Deployment: OSGi bundles provide the basis for a natural unit of deployment, update & patch.
  • Unit of Composition: OSGi bundles and Services are essential elements in the composition hierarchy.

Hence OSGi bundles and services, backed by OSGi Alliance’s open specifications, provide Java with essential – and previously missing – layers of structural modularity. In principle, OSGi technologies enable Java based business systems to be ‘Agile – All the Way Down!’.

As we will now see, the OSGi structures (bundles and services) map well to, and help enable, popular Agile Methodologies.

Embracing Agile

The Agile Movement focuses on the ‘Processes’ required to achieve Agile product development and delivery. While a spectrum of Lean & Agile methodologies exist, each tends to be; a variant of, a blend of, or an extension to, the two best known methodologies; namely Scrum and Kanbanhttp://en.wikipedia.org/wiki/Lean_software_development.

To be effective each of these approaches requires some degree of structural modularity.

Scrum

Customers change their minds. Scrum acknowledges the existence of ‘requirement churn’ and adopts an empirical (http://en.wikipedia.org/wiki/Empirical) approach to software delivery. Accepting that the problem cannot be fully understood or defined up front. Scrum’s focus is instead on maximising the team’s ability to deliver quickly and respond to emerging requirements.

Scrum is an iterative and incremental process, with the ‘Sprint’ being the basic unit of development. Each Sprint is a “time-boxed” (http://en.wikipedia.org/wiki/Timeboxing) effort, i.e. it is restricted to a specific duration. The duration is fixed in advance for each Sprint and is normally between one week and one month. A Sprint is preceded by a planning meeting, where the tasks for the Sprint are identified and an estimated commitment for the Sprint goal is made. This is followed by a review or retrospective meeting, where the progress is reviewed and lessons for the next Sprint are identified.

During each Sprint, the team creates finished portions of a product. The set of features that go into a Sprint come from the product backlog, which is an ordered list of requirements (http://en.wikipedia.org/wiki/Requirement).

Scrum attempts to encourage the creation of self-organizing teams, typically by co-location of all team members, and verbal communication between all team members.

Kanban

‘Kanban’ originates from the Japanese word “signboard” and traces back to Toyota, the Japanese automobile manufacturer in the late 1940’s ( see http://en.wikipedia.org/wiki/Kanban ). Kanban encourages teams to have a shared understanding of work, workflow, process, and risk; so enabling the team to build a shared comprehension of a problems and suggest improvements which can be agreed by consensus.

From the perspective of structural modularity, Kanban’s focus on work-in-progress (WIP), limited pull and feedback are probably the most interesting aspects of the methodology:

  1. Work-In-Process (WIP) should be limited at each step of a multi-stage workflow. Work items are “pulled” to the next stage only when there is sufficient capacity within the local WIP limit.
  2. The flow of work through each workflow stage is monitored, measured and reported. By actively managing ‘flow’, the positive or negative impact of continuous, incremental and evolutionary changes to a System can be evaluated.

Hence Kanban encourages small continuous, incremental and evolutionary changes. As the degree of structural modularity increases, pull based flow rates also increase while each smaller artifact spends correspondingly less time in a WIP state.

 

An Agile Maturity Model

Both Scrum and Kanban’s objectives become easier to realize as the level of structural modularity increases. Fashioned after the Capability Maturity Model (see http://en.wikipedia.org/wiki/Capability_Maturity_Model – which allows organisations or projects to measure the improvements on a software development process), the Modularity Maturity Model is an attempt to describe how far along the modularity path an organisation or project might be; this proposed by Dr Graham Charters at the OSGi Community Event 2011. We now extend this concept further, mapping an organisation’s level of Modularity Maturity to its Agility.

Keeping in step with the Modularity Maturity Model we refer to the following six levels.

Ad Hoc – No formal modularity exists. Dependencies are unknown. Java applications have no, or limited, structure. In such environments it is likely that Agile Management Processes will fail to realise business objectives.

Modules – Instead of classes (or JARs of classes), named modules are used with explicit versioning. Dependencies are expressed in terms of module identity (including version). Maven, Ivy and RPM are examples of modularity solutions where dependencies are managed by versioned identities. Organizations will usually have some form of artifact repository; however the value is compromised by the fact that the artifacts are not self-describing in terms of their Capabilities and Requirements.

This level of modularity is perhaps typical for many of today’s in-house development teams. Agile processes such are Scrum are possible, and do deliver some business benefit. However ultimately the effectiveness & scalability of the Scrum management processes remain limited by deficiencies in structural modularity; for example Requirements and Capabilities between the Modules usually being verbally communicated. The ability to realize Continuous Integration (CI) is again limited by ill-defined structural dependencies.

Modularity – Module identity is not the same as true modularity. As we’ve seen Module dependencies should be expressed via contracts (i.e. Capabilities and Requirements), not via artifact names. At this point, dependency resolution of Capabilities and Requirements becomes the basis of a dynamic software construction mechanism. At this level of structural modularity dependencies will also be semantically versioned.

With the adoption of a modularity framework like OSGi the scalability issues associated with the Scrum process are addressed. By enforcing encapsulation and defining dependencies in terms of Capabilities and Requirements, OSGi enables many small development teams to efficiently work independently and in parallel. The efficiency of Scrum management processes correspondingly increases. Sprints can be clearly associated with one or more well defined structural entities i.e. development or refactoring of OSGi bundles. Meanwhile Semantic versioning enables the impact of refactoring is efficiently communicated across team boundaries. As the OSGi bundle provides strong modularity and isolation, parallel teams can safely Sprint on different structural areas of the same application.

Services – Services-based collaboration hides the construction details of services from the users of those services; so allowing clients to be decoupled from the implementations of the providers. Hence, Services encourage loose-coupling. OSGi Services‘ dynamic find and bind behaviours directly enable loose-coupling, enabling the dynamic formation, or assembly of, composite applications. Perhaps of greater import, Services are the basis upon which runtime Agility may be realised; including rapid enhancements to business functionality, or automatic adaption to environmental changes.

Having achieved this level of structural modularity an organization may simply and naturally apply Kanban principles and achieve the objective of Continuous Integration.

Devolution – Artifact ownership is devolved to modularity-aware repositories which encourage collaboration and enable governance. Assets may selected on their stated Capabilities. Advantages include:

  • Greater awareness of existing modules
  • Reduced duplication and increased quality
  • Collaboration and empowerment
  • Quality and operational control

As software artifacts are described in terms of a coherent set of Requirements and Capabilities, developers can communicate changes (breaking and non-breaking) to third parties through the use of semantic versioning. Devolution allows development teams to rapidly find third-party artifacts that meet their Requirements. Hence Devolution enables significantly flexibility with respect to how artifacts are created, allowing distributed parties to interact in a more effective and efficient manner. Artifacts may be produced by other teams within the same organization, or consumed from external third parties. The Devolution stage promotes code re-use and efficient, low risk, out-sourcing, crowd-sourcing, in-sources of the artifact creation process.

Dynamism This level builds upon Modularity, Services & Devolution and is the culminatation of our Agile journey.

  • Business applications are rapidly assembled from modular components.
  • As strong structural modularity is enforced (isolation by the OSGi bundle boundary),  components may be efficiently and effectively created and maintained by a number of small – on-shore, near-shore or off-shore developement teams.
  • As each application is self-describing, even the most sophisticated of business systems is simple to understand, to maintain, to enhance.
  • As semantic versioning is used; the impact of change is efficiently communicated to all interested parties, including Governance & Change Control processes.
  • Software fixes may be hot-deployed into production – without the need to restart the business system.
  • Application capabilities may be rapidly extended applied, also without needing to restart the business system.

Finally, as the dynamic assembly process is aware of the Capabilities of the hosting runtime environment, application structure and behavior may automatically adapt to location; allowing transparent deployment and optimization for public Cloud or traditional private datacentre environments.

diag8

Figure 3: Modularity Maturity Model

An organization’s Modularisation Migration strategy will be defined by the approach taken to traversing these Modularity levels. Mosts organizations will have already moved from an initial Ad- Hoc phase to Modules. Meanwhile organizations that value a high degree of Agility will wish to reach the endpoint; i.e. Dynamism. Each organisation may traverse from Modules to Dynamism via several paths; adapting migration strategy as necessary.

  • To achieve maximum benefit as soon as possible; an organization may choose to move directly to Modularity by refactor the existing code base into OSGi bundles. The benefits of Devolution and Services naturally follow. This is also the obvious strategy for new greenfield applications.
  • For legacy applications an alternative may be to pursue a Services first approach; first expressing coarse grained software components as OSGi Services; then driving code level modularity (i.e. OSGi bundles) on a Service by Service basis. This approach may be easier to initiate within large organizations with extensive legacy environments.
  • Finally, one might move first to limited Devolution by adoption OSGi metadata for existing artifacts. Adoption of Requirements and Capabilities, and the use of semantic versioning, will clarify the existing structure and impact of change to third parties. While structural modularity has not increased, the move to Devolution positions the organisation for subsequent migration to the Modularity and Services levels.

diverse set of choices and the ability to pursue these choices as appropriate, is exactly what one would hope for, expect from, an increasingly Agile environment!

Agility and Structural Modularity – part I

Introduction

Agile development methodologies are increasingly popular. Yet most ‘Agile’ experts and analysts discuss agility in isolation.  This oversight is surprising given that ‘Agility’ is an emergent characteristic; this meaning a property of the underlying entity. For an entity to be ‘Agile’ it must have a high degree of structural modularity.

Perhaps as a result of this, many organisations attempt to invest in ‘Agile’ processes without ever considering the structure of their applications. Alongside the question, ‘How might one realise an Agile system?’, one must also ask, ‘How might one build systems with high degrees of structural modularity?’.

We start this series of blog articles by exploring the relationship between structural modularity and agility.

 

Structure, Modularity & Agility 

Business Managers and Application Developers face many of the same fundamental challenges. Whether a business, or a software application serving a business, the entity must be cost effective to create and maintain. If the entity is to endure, it must also be able to rapidly adapt to unforeseen changes in a cost effective manner.

If we hope to effectively manage a System, we must first understand the System. Once we understand a System, manageable Change and directed Evolution are possible.

Yet we do not need to understand all of the fundamental constituents of the System; we only need to understand the relevant attributes and behaviors for the level of the hierarchy we are responsible for managing.

Services should be Opaque

From an external perspective, we are interested in the exposed behavior; the type of Service provided, and the properties of that Service. For example is the Service reliable? Is it competitively priced relative to alternative options?

 

diag1

Figure 1: A consumer of a Service.

As a consumer of the Service I have no interest in how these characteristics are achieved. I am only interested in the advertised Capabilities, which may or may not meet my Requirements.

To Manage I need to understand Structure

Unlike the consumer, the implementation of the Service is of fundamental importance to the Service provider. To achieve an understanding, we create a conceptual model by breaking the System responsible for providing the Service into a set of smaller interconnected pieces. This graph of components may represent an ‘Organization Chart’ , if the entity is a business, or a mapping of the components used,  if the entity is a software application.

A first simple attempt to understand our abstract System is shown below.

 

diag2Figure 2: The Service provider / System Maintainer

From this simple representation we immediately know the following:

  • The System is composed of 15 Components.
  • The names of the Components.
  • The dependencies that exist between these Components; though we don’t know why those dependencies exist.
  • While we do not know the responsibilities of the individual Components; from the degree on inter-connectedness, we can infer that component ‘Tom’ is probably more important than ‘Dick’.

It is important to note that, we may not have created these Components, we may have no understanding of their internal construction. Just as the consumers of our Service are interested in the Capabilities offered, we, as a consumer of these components, simply Require their Capabilities.

Requirements & Capabilities

At present, we have no idea why the dependencies exist between the Components, just that those dependencies exist. Also, this is a time independent view. What about change over time?

One might initially resort to using versions or version ranges with the named entities; changes in the structure indicated by version changes on the constituents. However, as shown in figure 3, versioned names, while indicating change, fail to explain why Susan 1.0 can work with Tom 2.1, but Susan 2.0 cannot!

Why is this?

diag3

Figure 3: How do we track structural change over time? The earlier System functioned correctly; the later System – with an upgraded Component – fails. Why is this?

It is only when we look at the Capabilities and Requirements of the entities involved that we understand the issue. Tom 2.1 Requires a Manager Capability, a capability that can be provided by Susan 1.0. However, at the later point in time  Susan 2.0, having reflected upon her career, decided to retrain. Susan 2.0  now no longer advertises a Manager Capability, but instead advertises a  Plumber 1.0 Capability.

This simple illustration demonstrates that dependencies need to be expressed in terms of Requirements and Capabilities of the participating entities and not their names.

These descriptions should also be intrinsic to the entities; i.e. components should be self-describing.

diag4

Figure 4: An Organizational Structure: Defined in terms of Capabilities & Requirements with the use of Semantic versioning.

As shown, we can completely describe the System in terms of Requirements and Capabilities, without referencing specific named entities.

Evolution and the role of Semantic Versioning

Capabilities and Requirements are now the primary means via which we understand the structure of our System. However we are still left with the problem of understanding change over time.

  • In an organization chart; to what degree are the dependencies still valid if an employee is promoted (Capabilities enhanced)?
  • In a graph of interconnected software components; to what degree are the dependencies still valid if we refactor one of the components (changing / not changing a public interface)?

By applying simple versioning we can see that changes have occurred; however we do not understand the impact of these changes. However, if instead of simple versioning, semantic versioning is used (see http://www.osgi.org/wiki/uploads/Links/SemanticVersioning.pdf), the potential impact of a change can be communicated.

This is achieved in the following manner:

  • Capabilities are versioned with a major.minor.micro versioning scheme. In addition, we collectively agree that – minor or micro version changes represent non-breaking changes; e.g.  2.7.1 2.8.7. In contrast major version changes; e.g.  2.7.1. 3.0.0. represent breaking changes which may affect the users of our component.
  • Requirements are now specified in terms of a range of acceptable Capabilities. Square brackets ‘[‘ and ‘]‘ are used to indicate inclusive and parentheses ‘(‘ and ‘)‘ to indicate exclusive. Hence a range [2.7.1, 3.0.0) means any Capability with version at or above  2.7.1 is acceptable up to, but not including 3.0.0.

Using this approach we can see that if Joe is substituted for Helen, Tom’s Requirements are still met. However Harry, while having a Manager Capability, cannot meet Tom’s Requirements as Harry’s 1.7 skill set is outside of the  acceptable range for Tom i.e. [2,3).

Via the use of semantic versioning the impact of change can be communicated. Used in conjunction with Requirements and Capabilities we now have sufficient information to be able to substitute components while ensuring that all the structural dependencies continue to be met.

Our job is almost done. Our simple System is Agile & Maintainable!

 

Agile – All the Way Down

The final challenge concerns complexity. What happens when the size and sophistication of the System increases? An increased number of components and a large increase in inter-dependencies? The reader having already noticed a degree of self-similarity arising in the previous examples may have already guessed the answer.

The Consumer of our Service selected our Service because the advertised Capabilities met the consumers Requirements (see figure 1). The implementation of the System which provided this Service is masked from the consumer. This pattern is once again repeated one layer down. The System’s structure is itself described in-terms of the Capabilities and Requirements of the participating components (see figure 4). This time, the internal structure of the components are masked from the System. As shown in figure 5; this pattern may be re-repeated at many logical layers.

diag5

Figure 5: An Agile Hierarchy: Each layer only exposes the necessary information. Each layer is composite with the dependencies between the participating components expressed in-terms of their Requirements and Capabilities.

All truly Agile systems are built this way, consisting of a hierarchy of structural layers. Within each structural layer the components are self-describing: self-describing in terms of information relevant to that layer, with unnecessary detail from lower layers masked.

This pattern is repeated again and again throughout natural and man-made systems. Natural ecosystems build massive structures from nested hierarchies of modular components:

  • The Organism
  • The Organ
  • The Tissue
  • The Cell

For good reason, commercial organizations attempt the same structures:

  • The Organization
  • The Division
  • The Team
  • The Individual

Hence we might expect a complex Agile software systems to also mirror these best practices:

  • The Business Service
  • Coarse grained business components.
  • Fine grained micro-Services.
  • Code level modularity.

This process started in the mid/late 1990‘s as organizations started to adopt coarse grain modularity as embodied by Service Oriented Architectures (SOA) and Enterprise Service Buses (ESB’s). These approaches allowed business applications to be loosely coupled; interacting via well defined service interfaces or message types. SOA advocates promised more ‘Agile’ IT environments as business systems would be easier to upgrade and/or replace.

However, in many cases the core applications never actually changed. Rather the existing application interfaces were simply exposed as SOA Services. When viewed in this light it is not surprising that SOA failed to deliver the promised cost savings and business agility: http://apsblog.burtongroup.com/2009/01/soa-is-dead-long-live-services.html.

Because of the lack of internal modularity, each post-SOA application was as inflexible as its pre-SOA predecessor.

 

To be Agile?

We conclude this section with a brief summary of the arguments developed so far.

To be ‘Agile’ a System will exhibit the following characteristics:

  • A Hierarchical Structure: The System will be hierarchical. Each layer composed from components from the next lower layer.
  • Isolation: For each structural layer; strong isolation will ensure that the internal composition of each participating component will be masked.
  • Abstraction: For each layer; the behavior of participating components is exposed via stated Requirements and Capabilities.
  • Self-Describing: Within each layer the relationship between the participating components will be self-describing; i.e. dependencies will be defined in terms of published Requirements and Capabilities.
  • Impact of Change: Via semantic versioning the impact of a change on dependencies can be expressed.

Systems built upon these principles are:

  • Understandable: The System’s structure may be understood at each layer in the structural hierarchy.
  • Adaptable: At each layer in the hierarchy, structural modularity ensures that changes remains localized to the affect components; the boundaries created by strong structural modularity shielding the rest of the System from these changes.
  • Evolvable: Components within each layer may be substituted; the System supports diversity and is therefore evolvable.

The System achieves Agility through structural modularity.

In the next post in this series we will discover how OSGi™ – the Java™ Modularity framework – meets the requirements of structure modularity, and thereby provides the necessary foundations for popular Agile Methodologies and ultimately, Agile businesses.

OSGi – The Business Drivers

Software modularization should not be considered in isolation, but rather seen within the context of several inter-related trends that have occurred over the last decade. Indeed, software modularity is merely the latest visible facet of a much larger and more fundamental technology shift, with an impact at least equivalent to the move from mainframes to client server computing in the 1980’s.

These related trends include:

  • Service Oriented Architecture (SOA) – Enabling previously rigidly coupled business systems with proprietary protocols to be expressed as “services” which can be accessed via common protocols. However the business systems themselves remained opaque and monolithic.
  • Cloud Computing –  Decoupling applications from the underlying compute resources upon which they run, allowing more efficient resource utilization and scaling.
  • Software Modularization – Most recently, replacing the opaque monolithic applications and stove-piped middleware with dynamically assembled alternatives composed from re-usable software components.

As commented on in “The Rise of the Stackless Stack” (see http://www.redmonk.com/jgovernor/2008/02/05/osgi-and-the-rise-of-the-stackless-stack-just-in-time/); these trends collectively shift the industry away from rigidly coupled, static, opaque environments; towards adaptive, loosely coupled systems which are dynamically assembled from well-defined software components that run across a fluid set of compute resource.

.

.

Modularity and Assembly – It’s not a new idea!

The concept of assembling a product from a set of well-defined re-usable components is not new. Indeed its roots can be traced back to at least 250BC with emperor Qin Shi Huang and his commissioning of the Terracotta Army (see http://en.wikipedia.org/wiki/Assembly_line). Whatever the product, the driver for modularity and subsequent assembly is to; increase and maintain quality, reduce cost and increase output. The modern archetype for modularity and assembly is the automotive industry, where extensive use of standardization, modularity and assembly results in affordable automobiles.

Likewise, the computer industry already extensively uses hardware modularization in the form of standardised CPUs, memory and disk subsystems; leading to affordable computer hardware. These concepts are also well understood by software engineers. Extracts from http://en.wikipedia.org/wiki/Programming_in_the_large_and_programming_in_the_small include:

“Small programs are typified by being physically small in terms of their source code size, are easy to specify, quick to code and typically perform one task or a few very closely related tasks very well”,

… programming in the large, coding managers place emphasis on partitioning work into modules with precisely-specified interactions…”

 

and

“… one goal of programming in the large involves setting up modules that will not need altering in the event of probable changes. This is achieved by designing modules so they have high cohesion and loose coupling.”

Coarse grained modularization of business processes (via SOA) and the subsequent re-assembly (via BPEL) has been underway for some time, however the individual applications have remained monolithic and opaque. Further progress was not possible until a strong, widely endorsed, industry standard for enterprise software modularity was available. OSGiTM provides this modularization standard.

Since 1999 the OSGi Alliance (see http://www.osgi.org) has provided standards, reference implementations and guidance for modularization and assembly best practices. Recently, the OSGi Alliance has succeeded in:

  • Recruiting the majority of enterprise software vendors.
  • Encouraging those vendors to migrate their product portfolios to OSGi.
  • Producing the OSGi Enterprise standard which provides mechanisms to integrate OSGi based applications with legacy JEE & Spring.

Today OSGi is rapidly becoming a cornerstone for any successful business system re‑engineering effort.

.

.

The Dilemma

While software modularity and dynamic application assembly are inevitable industry trends; OSGi faces the usual adoption challenges: see Innovators Dilemma for an exploration of this theme – http://www.businessweek.com/chapter.christensen.htm.

‘The Business’ will never ask for OSGi based systems, it’s an implementation detail. Yet in the fullness of time ‘The Business’ will complain when the internal IT organization is seen as; too inefficient, too expensive and /or no longer agile enough to meet tactical, let alone strategic, business objectives.

Yet, from an IT management perspective revisiting a complex but working enterprise software stack is an immediate cost and quantifiable risk.[1] Something to be avoided. The alternative, a slow death seems almost preferable, or at least somebody else’s problem. As the environment degrades, make-shift tactical solutions are implemented to ease immediate pain points caused by strategic issues. This ongoing tactical ‘reactive’, behaviour results in an explosion in operational complexity and associated OPEX. The organisation eventually becomes paralysed by its own management’s inability to ‘grasp the nettle’ and identify and address the fundamental issues.

It is suggested that this organizational behavior parallels the well known ‘Tragedy of the Commons’ (see http://en.wikipedia.org/wiki/Tragedy_of_the_commons).

Developer attitudes and/or their working environment may also act as an impediment:

  • Just Get It Done. At each point in time, the developer is pursuing, or is forced to pursue, the shortest / lowest effort / route to the immediate deliverable at the expense of longer term maintainability.
  • The Artisan: The software developer “knows better” and pursues his own approach in defiance of industry standards and accepted best practices.
  • The Luddite: The current opaque, undocumented spaghetti of code within the organisation ensures employment. Increased efficiency, self-describing dependencies and code re-use sound like a recipe for smaller more flexible development teams.

Each can be a powerful brake on change.

Yet history tells us, those who effectively embrace and leverage change, succeed. OSGi migration is neither free, nor in itself, a quick ‘silver bullet’. However OSGi is of fundamental strategic importance to any organisation whose core business involves the processing of information.

.

.

The Benefits

Quite simply, modularity localizes the impact of change, which directly leads to increased maintainability.

  • As long as the module boundaries don’t change, one can change the functionality of the module freely, without concern for breaking the wider system; i.e. the impact of any local change is prevented from leaking into the wider system.
  • Modules that perform a few, or a single function, are much easier to test exhaustively than an entire monolithic system.
  • Smaller pieces of the system can be independently versioned.
  • Specific knowledge is only required for the particular module being worked upon, along with its relationship to other modules in the system; other modules can be treated as “black boxes” that perform specific functions, without worrying about how they perform them.

The large scale structure of the migrated application, which in all likelihood was previously unknown, is now completely defined by the dependencies between the set of versioned modules. Hence:

  • It is understood which modules are and are not required!
  • By replacing or re-wiring modules, the behavior of a composite system may be rapidly changed.

From an ongoing maintenance perspective it is now possible to re-factor individual modules, or even the overall system, to systematically drive out accidental complexity (see http://theit.org/publishing/books/prof-app/19261.cfm) and so contain, or even reverse, design rot.

While these arguments are understood, many organizations require demonstrable real world examples. The challenge is that those organisations that are first movers have no real incentive to act as references, thereby losing competitive advantage.

For several years, Paremus have been working with a number of organizations who are at various stages of OSGi migration. While Paremus cannot discuss specific details, or disclose the organizations by name, we are able to reference some common themes that demonstrate real world benefits for OSGi in general, and the Paremus Service Fabric in particular.

.

.

Resource Utilisation

When working with traditional applications, a lack of information concerning required libraries results in developers having to load every possible library into their IDE. From experience, Paremus have seen this drive memory requirements as high as 40GB per developer desktop. However, once dependencies are understood, and mechanisms are in place so that only the required components are loaded, Paremus have also seen the number of artefacts reduced by an order of magnitude with corresponding machine memory savings.

For an organization with several hundred developers the saving is considerable. The saving in reduced memory footprint in Production is of course correspondingly larger.

.

.

Developer Efficiency

Developing against a monolithic application requires the developer to work and test against the complete code-base. Yet for large applications it may not be possible to test changes in the local IDE, as compile times may be hours. As a result, developers are forced to rely upon unit and integration tests that run during the nightly build cycle. The result is that the whole bug detection and rectification cycle can take days, with an increased likelihood that some issues are not found and leak into production.

In contrast, rapid testing of OSGi modules is easily achieved in the developer IDE, and Paremus have seen this test / fix cycle reduced from days to minutes.

A modular system also lends itself well to many hands being involved in its development and maintenance. It’s not necessary to understand the whole system inside-out, each individual can independently work on  small well-defined and decoupled modules. This directly translates to Increased project delivery success rates as smaller, well-contained projects have a higher success rate than larger and more poorly constrained projects.

.

.

Definitive Dependencies

Most modern build systems allow build-time dependencies to be specified and obtained from various repositories. However, different types of  build system repository specify dependencies with different formats; e.g. pom.xml, ivy.xml.

OSGi provides runtime dependency meta-data in its Require-Bundle and Import-Package bundle manifest headers. This provides definitive  industry standard dependency information that can be consumed regardless of the build system you use. Hence OSGi decouples your runtime from the source build systems; thereby avoiding meta-data lock-in and allowing  different types of build repository to co-exist.

In addition, OSGi supports package-level dependencies. These are finer-grained than the artefact-level dependencies supported by most build systems and can be programmatically determined directly from source code using tools such as Apache Felix Sigil . This avoids the errors that occur when trying to manually maintain dependency data.

Finally, OSGi supports its own repository API: OBR. A current priority for the OSGi Alliance EEG working group; OBR allows searching for package-level dependencies as well as the artefact-level dependencies used by other repositories.

.

.

Production Stability, Availability & Agility

Unless you happen to own applications that never change; stability, availability and agility are closely related concerns. The more agile the business service, the easier it is to change from one well-defined state to the next. This may be to introduce new business functionality, apply a patch, or roll back to a previously good version.

The internal structure of monolithic opaque applications is poorly understood; hence upgrades are complex and high risk. This, coupled with the long developer diagnostic / fix cycle, can result in repeated production outages and instability that spans several working days. Operations typically respond by maintaining a number of isolated horizontal silos; attempting to ensure service availability by releasing new software one silo at a time. This issue isn’t just common, its almost universal.

In contrast; with an OSGi-based runtime like the Paremus Service Fabric, each application is self‑describing; meaning that the application structure is now fully described in-terms of dependencies between versioned modules. A system may be deployed in seconds. A running system may be upgraded on-the-fly (i.e. within seconds); and may be returned to a previous well known-state just as rapidly.

.

.

Operational Risk & Governance

The implication for operational risk should be immediately apparent:

  • From regulatory and business continuity perspectives, the structure of an application is precisely known at each point-in-time; allowing all versions of the application to be rapidly re-constituted.
  • Key structural information is no longer locked within key members of staff. In principle the organisation can employee any OSGi literate developer / systems architect who can rapidly navigate the structure of the organisation’s software systems.

From a governance perspective, it is now simply task to rapidly answer the following types of question:

  • Which software license types are used within which production applications?
  • Which production applications use third party modules with an identified security vulnerability?

This information is readily available courtesy of the metadata embedded in each OSGi module and the dynamic deployment & assembly mechanisms provided by the OSGi runtime environment.

.

.

An OSGi ROI?

Anne Thomas Manes (Gartner) estimates that ongoing maintenance accounts for 92% of the total lifetime cost (TCO) of each application. Whilst hardware accounts for <10% of TCO, software maintenance accounts for ~70% of TCO; the remainder being the initial cost of developing the application; see slides 9 & 10 – SOA Symposium: Berlin, October 2010.

Given this, the current fashion for virtual machine based Cloud computing seems somewhat perplexing. The deployment of traditional opaque software stacks as virtual machine images does  NOTHING to address the issue of application maintainability.

Closer to home, a group of Financial Services engineers recently attempted to quantify the potential return realised by migrating their production environment to the Paremus Service Fabric. They conclude that OPEX savings of 60% were possible. While Paremus do not know the details of this analysis, it is likely that the following three contributing factors were considered.

  • The Paremus Service Fabric is a cost effective replacement for Application Servers, Compute Grids, CMDBs and provisioning tooling and requires less operational resource to manage it.
  • The Paremus Service Fabric, being a Private Cloud, achieves the resource utilisation and efficiency savings alluded to by many virtualisation vendors, but without the management overhead and operational risk associated with ‘virtual machine sprawl’.
  • However, given the Gartner TCO analysis, the bulk of the identified OPEX saving most likely results from the ongoing maintainability of OSGi based business applications running upon the Service Fabric runtime.

A large Financial Service organisations may have 1000+ applications, each with an average annual running cost of ~$1,000,000. This equates to an potential annual saving of $600,000,000! A significant medium term bottom line saving that surely warrants investing in a multi-year OSGi based application transformation program?

.

.

Migration Strategy

Hopefully a compelling set of arguments for adopting OSGi have been presented, but how do you actually migrate an organisation with a decade of legacy applications to a new OSGi based world?

Organisations considering this invariably start with the following questions:

  • How do we determine and then untangle dependencies in our current environment?
  • How do we move to an OSGi-centric build/test/release cycle within minimum impact on the majority of developers?
  • What level of modularity / granularity should we pursue?

In response, Paremus advise an iterative approach; the precise details dependent upon each organisation’s starting point and business objectives.

Stage I

 

1. Assemble a small high-caliber team of engineers with appropriate OSGi / modularity skills. Ensure that this team have senior management backing and representation.

2. Set up tools to determine and monitor dependencies within the organisation’s existing code base. Provide automated reporting, remove superfluous, and fix unsatisfied, dependencies.

At this early stage, the organization may have already achieved as much as an order of magnitude simplification in the code base dependencies. This in itself will improve developer productivity by reducing compile times and help increase the success of production releases.

Stage II – Tooling & Metadata

 

3. Review organisation’s standards for IDE tooling. Does current tooling support OSGi metadata and enable simple management of exposed dependencies? Review organisation’s current repository standards; will these support the OSGi Alliance OBR standard? If required, select new tooling.

4. Set up OSGi metadata for all organisations projects; this to be maintained by project developers. This metadata need not yet be used at runtime, but it can be used during the build process to monitor the progress of the modularisation effort.

 

It should now be possible to run OSGi and standard Java variants of an application side-by-side. This allows migration to progress without creating large parallel branches in source control, which are difficult to subsequently re-merge. CAPEX savings may be realised at this point, as only the required artifacts will be loaded into the developers IDE, decreasing the amount of resources required and increasing testability further.

Stage III – Runtime

5. Select candidate applications for migration based upon agility and re-usability considerations:

  • A set of applications that share a high degree of functionality / code and require frequent functional updates are excellent candidates for early migration.
  • A standalone application, which seldom, if ever changes, and shares little or no functionality with other applications, is a very low priority – and may never be migrated.

6. Create working runtime bundles using existing libraries as the level of granularity.

7. Create integration tests for use during and after migration to assert that parallel development streams do not break any modularisation efforts as the code is migrated.

8. Test OSGi version of candidate application in an OSGi runtime environment; e.g. Paremus Service Fabric.

9. Create integration tests to ensure fixes for hidden gotchas are caught during development rather than in production; i.e. Development and UAT fabrics.

The organization should now be in a good position to deploy OSGi based applications to production and accelerate OSGi migration. The new development, build, test. release lifecycle will enable considerable improvements in developer productivity via further reductions in time to run unit and integration tests.

Stage IV – Iterate and Reward

10. For each application take an iterative approach to modularisation; break down the deliverable into smaller modules (improving modular boundaries), test, deploy, release to production. Then re-visit modularity, break down further if appropriate, test, release.

11. For each migrated application, using the dependency information now available, publish ‘composition reports’; listing degree of re-used of in-house modules, certified third part modules and alerting on uses of non-certified modules.

12. The modularisation process will now be running across many candidate applications in parallel; during which the core OSGi team will continue to advise each participating application group on appropriate levels of modularity and opportunities for re-use.

13. Reward application teams; not just for meeting an initial delivery objectives, but also on:

  • Use of agreed third party OSGi libraries, where appropriate.
  • Achieve a high degree of re-use in a re-factored applications.
  • Delivering and maintaining in-house modules which are re-used in many in-house applications.

The ‘composition reports’ and associated incentives provide the development teams with powerful feedback mechanisms.

In this manner, the organisation typically starts with an isolated application, building the required skills & processes. Driven by initial successes and cost savings, adoption naturally flows from this initial seed point through the organisation.

.

.

To Conclude

Whether OSGi is destined to be the next IT ‘fashion bubble’ (once the current hysteria on ‘Cloud’ has waned) or it will grow organically via initial adoption by the most sophisticated IT organizations, is unclear.

However, the software industry is a manufacturing industry; no more and no less than manufacturers of disk drives, steel or automobiles.  While the raw materials are low cost, the ongoing effort required to craft and maintain flexible, high quality software, is not.

Necessity will drive the software industry, and those organisations with large in-house development teams, towards modularization, dynamic application assembly and so OSGi.


[1]The author has had direct experience of this behaviour in an enterprise environment.

A backup solution that was initially designed for a hand full of servers was nearing collapse. Each time a new backup server was added to address over-runs the reduction in backup time was less than the 1/n expected as loads could not be exactly balanced across backup real estate. The situation degraded to the point where the backups extended to a full 24 hours. The correct solution was to refresh the enterprise network and consolidate backup servers into centralized high speed silos. Yet, despite the operational risk, some management refused to sign off on what was perceived to be a high risk project. Only when no ‘last’ silver bullet could be found was sign-off achieved to progress the correct strategic solution. The situation had by then become so serious that this solution had to be implemented as rapidly as possible.

Happy New Year

2009 was an interesting year for Paremus. Despite the bitter economic climate, OSGi finally captured the imagination of the software industry: with this, in conjunction with the intensifying ‘Cloud Computing‘ drum-beat, came an increased appreciation of capabilities and benefits brought by the Paremus Service Fabric.

In the closing months of 2009, Paremus released Nimble as a stand-alone product. A state-of-the-art dependency resolution engine,  Nimble’s mantra ‘Making Modularity Manageable’, struck a chord with many of you; and I’d like to thank you all for the extremely positive reception Nimble received.

For those interested in private, public and hybrid ‘Clouds’, the year also closed with a interesting series of Christmas posts (or lectures 🙂 ) by Chris Swan charting his successful attempt to create a hybrid cloud based ‘platform as a service’ by combining the Paremus Service Fabric with Amazon EC2.

So what can we expected in 2010? From a software industry perspective, be prepared for some fundamental shifts as the industry really starts to grapple with modularisation, cloud computing and when used in combination, what they really mean! As usual Kirk Knoernschild captures the moment in his latest post ‘A New Year’s Declaration

Whilst not wanting to give the game away, I can say that Paremus will be making a number of interesting announcements over the coming months. Those of you who are interested OSGi should keep a close eye on Nimble via the Nimble blog. Meanwhile, those interested in low latency computing, private Cloud Computing and innovative implementations of OSGi EEG standards work, should keep a watching brief on the Paremus Service Fabric.

Enough said.

Announced in 2005, the Newton project was, by several years, the industries first distributed SCA / OSGi distributed runtime platform! Since then, Newton has indirectly influenced industry standards, and many of our competitors roadmaps. However, since early 2008, our commercial Service Fabric has rapidly evolved beyond the initial concepts encapsulated within Newton. Given this, we start 2010 by announcing that that Newton will be archived and the CodeCauldron open source community closed.

In due course, a number of new OSS projects will be announced on the new Paremus community site. The CodeCauldron OSS experience and these future initiatives will be the subject of subsequent posts.

In the meantime, I wish Paremus customers, partners and friends all the very best for 2010.

Richard

How Nimble is your OSGi runtime

Hands up all of you managing OSGi dependencies via an editable list of bundles. Easy isn’t it! It just works right!?

Well actually – it ‘just works‘ for a single application running in a small number of containers. From an enterprise perspective you are unintentionally contributing to an impending complexity meltdown; an explosion of dependency and configuration management issues. And if you are unfortunate enough to end up supporting your own composite creations, you may well end up envying the fate of Prometheus and rueing the day you learnt to code.

Possible harsh? But I’m not alone voicing this concern!

In his recent article Reuse: Is the Dream Dead?, Kirk Knoernschild continues his efforts to educate the industry on the tensions between code ‘re-use‘ and ‘simplicity of use‘. Kirk argues that as you increase potential re-use via lightweight fine-grained components, the complexity of dependencies and necessary environmental configurations corresponding increase, so making these same components harder to use.

A simple concept, yet if unaddressed, an issue that will make your life as an enterprise developer increasing uncomfortable and help edge OSGi closer to that seemingly inevitable ‘trough of disillusionment‘.

Yet, from a development perspective the issue of dependency management is well understood.

sigilWhilst initially found wanting, a number of projects now exist to address this; including the SIGIL eclipse plug-in which Paremus recently contributed to the Apache Felix project, (SIGIL leveraging Peter Krien’s BND tool).

In contrast, the issue of dependency management in Production is less immediately obvious, its impact more profound and generally ignored.

  • Will aspects of the runtime environment affect the runtime dependencies within the application?
  • Will applications be isolated from each other, or might they run within the same JVM?
  • How are the released artifacts subsequently managed in the production environment with respect to ongoing bundle dependency and version management?

Echoing Kirk’s concerns, Robert Dunne started his presentation at OSGi DevCon Europe with the observation that; ‘whilst modularity was good, its benefits are often undermined by dependency and configuration complexity‘. The subject of Robert’s presentation? The Paremus Nimble Resolver, which is our response to the concerns posed by Kirk.

Nimble is a high performance runtime dependency resolver. To deploy a composite application to a Nimble enabled runtime (i.e. the Paremus Service Fabric) one specifies:

  • The root component of the artifact.
  • And a set of associated policies and constraints.

Nimble then does the rest.

Presented with the ‘root‘, Nimble dynamical constructs the target composite; ensuring that the structural dependencies are resolved in a manner consistent with both organizational policies and the runtime environment within which it finds itself.

Nimble’s OSGi capabilities include:

  • Fragment attachment policies.
  • Optional import policies.
  • Import version range narrowing.
  • The ability to resolve dependencies on extender bundles (DS, ‘classic’ Spring, Spring DM, iPOJO).

With Nimble policies allowing:

  • The configuration of selected extensions.
  • Flexible constraint requirement -> capability matching.
  • The ability to configure optional dependency resolution behaviors.

Not just OSGi, Nimble is a generic artifact resolver with a plug-able architecture. Any artifact type may be supported, with support currently available for:

  • OSGi Bundles
  • POJO’s, ‘classic’ Spring & Spring DM
  • WAR
  • Configurations.

A Nimble enable runtime quite literally dynamically assembles all required runtime application and infrastructure service dependencies around the deployed business components! Specify a WAR artifact and Nimble will instantiate the appropriate Servlet engine dictated by runtime policy attached to the WAR; i.e. Tomcat or Jetty Sir? Specify a ‘Configuration‘, and Nimble responds by installing the target of the configuration, and of-course its dependencies.

Nimble not only directly addresses Kirk’s concerns, but goes on to radically transforms our understanding of the responsibilities and capabilities of next generation composite aware Service Platforms. But most importantly, Nimble was created to enable effect re-use whilst making life simpler for you and the organizations you work for.