Scaling Up Software Development

Total global demand for software is projected to increase by an order of magnitude over the next decade, driven by new forces in the global economy like the emergence of China and the growing role of software in social infrastructure, by new application types like business integration and medical informatics, and by new platform technologies like web services, mobile devices and smart appliances. Without comparable increases in capacity, total software development capacity seems destined to fall far short of total demand by the end of the decade. Of course, if market forces have free play, this will not actually happen, since the enlightened self interest of software suppliers will provide the capacity required to satisfy the demand. This begs the question of what will change to provide the additional capacity. Our analysis suggests that the answer is not likely to come from increasing the number of developers or their levels of competency. Instead, software development methods and practices will have to change dramatically to make developers much more productive.

Industrializing Software Development

Other industries added capacity by moving from craftsmanship, where whole products are created from scratch by individuals or small teams, to manufacturing, where a wide range of product variants is rapidly assembled from reusable components created by multiple suppliers, and where machines automate rote or menial tasks. They standardized processes, designs and packaging, using product lines to facilitate systematic reuse, and supply chains to distribute cost and risk. Some are capable of mass customization, where product variants are produced rapidly and cheaply on demand to satisfy the specific requirements of individual customers.

What will industrialization look like in the software industry? We cannot know with certainty until it happens, of course, but we can make educated guesses based on the way the software industry has evolved, and on what industrialization has looked like in other industries. The key is to leverage the skills and talents of experienced developers by giving them ways to encapsulate their knowledge as reusable assets that others can apply. Patterns demonstrate limited but effective knowledge reuse. The next step is to move from documentation to automation, using languages, frameworks and tools to automate pattern application.

Automating Software Development

Can we automate software development? Of course, we can, and we have already. Database management systems, for example, automate data access using SQL, providing benefits like data integration and independence that make data driven applications easier to build and maintain. Similarly, widget frameworks and WYSIWYG editors make it easier to build and maintain graphical user interfaces, providing benefits like device independence and visual assembly. Looking closely at how this was done, we can see a recurring pattern.

After developing a number of systems in a given problem domain, we identify a set of reusable abstractions for that domain, and then we document a set of patterns for using those abstractions.
We then develop a run time, such as a framework or server, to codify the abstractions and patterns. This lets us build systems in the domain by instantiating, adapting, configuring and assembling components defined by the run time.
We then define a language and build tools that support the language, such as editors, compilers and debuggers, to automate the assembly process. This helps us respond faster to changing requirements, since part of the implementation is generated, and can be easily changed.

How Do We Move Forward?

Some readers will recognize this as the Language Framework pattern by Roberts and Johnson. The key to a transition from craftsmanship to manufacturing is applying critical innovations to make the Language Framework pattern less expensive to develop. All of these innovations exist today, and have displayed notable potential in commercial products, although most of them are not yet fully mature. They fall into four areas: systematic reuse, development by assembly, model driven development, and process frameworks. Let’s consider each of these areas in turn.

Systematic Reuse

One of the most important innovations in software development is defining a family of software products, whose members vary, while sharing many common features. A family provides a context in which problems common to the family members can be solved collectively. This enables a more systematic approach to reuse, by letting us identify and differentiate between features that remain more or less constant over multiple products and those that vary. A software product family may consist of either components or whole products.

Software product lines exploit product families, identifying common the features and recurring forms of variation in specific domains to make the production of family members faster, cheaper, and less risky. Products developed as family members reuse requirements, architectures, frameworks, components, tests and many other assets.

Figure 1 describes the key tasks performed and artifacts produced and consumed in a product line. Product line developers build production assets applied by product developers to produce family members in much the same way that platform developers build device drivers and operating systems used by application developers. A key step in developing the production assets is to produce one or more domain models that describe the common features of problems in domains addressed by the product line, and the recurring forms of variation. These models collectively define the scope of the product line, are used to qualify prospective family members. Requirements for family members are derived from them, providing a way to map variations in requirements to variations in architecture, implementation, executables, development process, project environment, and many other parts of the software life cycle.

Model Driven Development

Model-driven development (MDD) uses models to capture high level information, usually expressed informally, and to automate its implementation, either by compiling models to produce executables, or by using them to facilitate the manual development of executables. This is valuable because this information is currently lost in low level artifacts, such as source code files, where it is difficult to track, maintain or enhance consistently.

Some development activities, such as building, deploying and debugging, are partially or completely automated today using information captured by source code files and other implementation artifacts. Using information captured by models, MDD can also provide more extensive automation of these activities, and more advanced forms of automation, such as debugging with models and automatically configuring tools.

Domain Specific Languages

For the purposes of MDD, we are not interested in closed end languages like 4GLs, nor in using a single high level language for all aspects of development. The weakness of those strategies has been amply demonstrated. We are also not interested in models rendered by hand on white boards, or on note pads. We are interested in models that can be processed by tools, and we propose to use them in the same way that we currently use source code. Models used in this way cannot be written in languages designed for documentation. They must be precise and unambiguous. In order to raise the level of abstraction, a modeling language must therefore target a narrower domain than a general purpose programming language. We find that the Unified Modeling Language (UML), in particular, is suitable for sketching, but not for the capture of high fidelity metadata used to generate models, code and other software development artifacts. A detailed discussion of the issues around using UML as language for MDD is beyond the scope of this article, but Cook provides a cogent analysis, and Fowler adds several insights on his blog page.

A language that meets these criteria is called a Domain Specific Language (DSL), because it models concepts found in a specific domain. A DSL is defined with much greater rigor than a general purpose modeling language. Like a programming language, it may have either textual or graphical notation.

Two examples of DSLs are illustrated in Figure 2. The model on the left describes components, such as web services. It is used to automate component development and configuration. The model on the right describes logical host types in a data center. It is used to design and implement data center configurations.

An important aspect of MDD is taking advantage of relationships between models. While each model shown in Figure 2 provides significant value on its own, the relationship between the two allows a form of validation that is not practical in most development shops today. Web service deployments can be described by dragging service components onto logical servers. Differences between the resources they require and the resources services available on the logical servers onto which they are deployed are then flagged as validation errors, as illustrated.

Incremental Code Generation

The key to effective code generation is to generate small amounts of code that span only small abstraction gaps. This allows the tool to take advantage of platform features, and therefore to produce focused, efficient, platform specific implementations.

One way to make code generation more incremental is to move the models closer to the platform, as illustrated in Figure 3. For example, a modeling tool for a programming language specific can achieve higher fidelity by using the programming language type system, instead of a type system defined by a general purpose modeling language.

Of course, this reduces the power of the model, by limiting it to abstractions already available on the platform, or only slightly higher level, such as programming idioms. How then, do we work at higher levels of abstraction? We use more abstract models, and move the platform closer to the models with either frameworks or transformations, as illustrated in Figure 4. Let’s look at each of these in turn.

We can use a framework to implement higher level abstractions that appear in the models, and use the models to generate snippets of code at framework extension points. Conversely, the models help users complete the framework by visualizing framework concepts, and exposing its extension points in intuitive ways. A pattern language can be used instead of a framework, as described by Hohpe and Woolf. This requires the tool to generate the pattern implementations, in addition to the completion code. This approach is illustrated in Figure 4 (a).
Instead of a framework or pattern language, we can generate to a lower level DSL. We can also use more than two DSLs to span a wide gap, leading to progressive transformation, where models written using the highest level DSLs are transformed into executables through a series of refinements, as shown in Figure 4 (b).

Development By Assembly

Critical innovations in the areas of platform independent protocols, self description, variable encapsulation, assembly by orchestration and architecture driven development are required to support development by assembly.

Platform Independent Protocols

Web services technologies succeed where earlier component assembly technologies failed by separating the technologies used to specify and assemble components from the technologies used to implement them. Because XML is a technology for managing information, not a technology for constructing components, web services require the use of wrappers to map web method invocations onto local method invocations based on the underlying component implementation technology.

Self Description

Unstated and conflicting assumptions about components create architectural mismatches, making assembly difficult or impossible, or degrading the operational qualities of the resulting system. Self description reduces architectural mismatch by improving component packaging to make assumptions, dependencies, behavior, resource consumption, performance, and certifications explicit. The metadata it provides can be used to automate component discovery, selection, licensing, acquisition, installation, adaptation, assembly, testing, configuration, deployment, monitoring and management.

Variable Encapsulation

Variable encapsulation also reduces architectural mismatch by enabling the publication of partially encapsulated components that can be adapted to new contexts by selecting and weaving appropriate non-functional aspects with their functional ones. The form of a component in a specific assembly can then be determined by the context in which it is placed. This makes it easy to adapt mismatched components at assembly time. It also allows functional aspects to be refactored across component boundaries by removing non-functional assumptions from them. Valid adaptations can be identified in advance, and in some cases can even be performed automatically by tools, given a formal description of the context into which the finished component will be placed.

Assembly by Orchestration

Given adequate contractual mechanisms, web services can be assembled using an orchestration engine, such as Microsoft BizTalk Server, to manage the sequences of messages exchanged between them. Assembly by orchestration makes development by assembly much easier, since web services have far fewer dependencies on one another than binary components. Unlike classes, web services do not have to reside within the same executable. Unlike components that use platform specific protocols, they can be assembled across platform boundaries. Two web services can interact if their contracts are compatible. They can be developed and deployed independently, and then assembled at run time using orchestration. If appropriate interception and redirection services are available, they can even reside in different administrative or organizational domains. In other words, assembly by orchestration eliminates design, compile and deployment time dependencies between components.

Figure 5. Assembly By Orchestration

Architecture Driven Development

While preventing the assembly of mismatched components is better than constructing invalid assemblies, it does not necessarily promote the availability of well matched components. This is the purpose of architecture. According to Shaw and Garlan, a software architecture describes the components to be assembled, the interactions that can occur among them, and the acceptable patterns of composition. This reduces the risk of architectural mismatch by imposing common assumptions and constraining design decisions.

An IEEE standard, 1471, Recommended Practice for Architectural Description, provides some guidelines for architectural description. According to these guidelines, a system has one or more stakeholders. A stakeholder has specific concerns or interests regarding some aspect of the system. In order to be useful to a community of stakeholders, an architectural description must have a format and organization that the stakeholders understand. An Architectural Description Standard (ADS) is a template for describing the architecture of a family of systems.

Process Frameworks

The key to process maturity is preserving agility while scaling up to high complexity created by project size, geographical distribution, or the passage of time. Experience suggests that a small amount of structure increases agility by reducing the amount of work that must be done. This principle can be applied in the context of a software product family to manage complexity without reducing agility using process frameworks.

A process framework provides guidance, and imposes constraints on development artifacts and activities, instead of prescribing a process. It decomposes a process into micro processes attached to viewpoints in an architectural description. Each micro process describes the process for producing a view based on the viewpoint to which it is attached. Constraints are added to describe preconditions that must be satisfied before an artifact is produced, postconditions that must be satisfied after it is produced, and invariants that must hold when the artifacts have stabilized. We call this structure a process framework because it defines a space of possible processes that could emerge, given the needs and circumstances of a specific project, instead of prescribing one general purpose process for all projects.

Once a process framework is defined, micro processes can be stitched together to support any work flow that the project requires, including top down, bottom up, outside in, test then code, code then test, in any combination, or a totally ad hoc work flow, as long as the constraints are satisfied. The work flow can also be resource driven, allowing work to be scheduled to maximize overall throughput using methods like PERT and CPM, and to be started when the required resources become available. This fosters agility by letting a work flow arise dynamically and opportunistically within the constraints, adapting to the innumerable variations in circumstances that will inevitably occur, while incorporating lessons learned to minimize the amount of time and money wasted on rediscovery.

A New Software Development Paradigm

While each of these critical innovations holds significant promise, none of them is sufficient, on its own, to catalyze a transition from craftsmanship to manufacturing. The key is to integrate the critical innovations into a coherent methodology. This is the goal of Software Factories. Before we can talk about Software Factories, we must introduce software factory schemas and software factory templates.

software factory schemas

A software factory schema is a document that categorizes and summarizes the artifacts used to build and maintain a system, and that defines relationships between them, so that we can maintain consistency among them.

A common approach is to use a grid, like the one illustrated in Figure 6. The columns define concerns, while the rows define levels of abstraction. Each cell defines a perspective or viewpoint from which we can build some aspect of the software. For example, for a 3-tiered application, one cell might define an abstract viewpoint on the presentation tier, a second might define an abstract viewpoint on the data tier, and a third might define a concrete viewpoint on the data tier, used for developing database schemas. Once the grid has been defined, we can populate it with views that comprise the development artifacts for a specific software product.

Of course, a grid can be used to build more than one software product. Before it has been populated with specific views, it defines the bill of materials required to build the members of a software product family. Taking a cue from software product lines, we can go one step further and add information to each cell, identifying production assets used to build the views required by each viewpoint, including DSLs, patterns, frameworks, tools, and micro processes.

The grid is not an innovation. What is novel is applying the grid to a product family, identifying production assets for each cell, and defining mappings between and within the cells that can be used to fully or partially automate development tasks. We must use first class development artifacts based on high fidelity languages, like XML, C# and SQL in order to provide this automation. For models, this means using DSLs, not general purpose modeling languages designed for documentation. In some cases, the artifacts described by viewpoints are models, but this is often not the case. They can be any source artifacts based on a formal language, such as high level work flow scripts, general purpose programming language source code files, WSDL files, or SQL DDL files.

Viewpoints define not only the languages used to develop views, but also requirements on views, usually expressed as constraints or patterns. For example, a software factory schema might contain two viewpoints that both use the same class modeling DSL. We might require all of the classes modeled from one of these viewpoints to inherit, either directly or indirectly, from classes supplied by a user interface widget framework associated with the viewpoint. In a real software factory schema, the same languages are typically used by multiple viewpoints, especially viewpoints close to the platform that use general purpose programming languages.

software factory schemas as Graphs

Of course, a software factory schema is better represented as a graph whose nodes are viewpoints and whose edges are computable relationships between viewpoints called mappings. This allows nodes that would not be adjacent in a grid representation to be related. Also, it relaxes the artificial constraint imposed by a grid that the viewpoints must fit into neat classification schemes, creating rows and columns. Finally, and most importantly, it allows the schema to reflect the software architecture. So, for example, a schema for a family of business applications might contain several clusters of viewpoints, one for each subsystem like customer management, catalog management, or order fulfillment. The viewpoints in each cluster might then be further grouped into subsets reflecting the layered architecture of each subsystem, as illustrated in Figure 7.

A software factory schema essentially defines a recipe for building members of a software product family. Clearly, the viewpoints describe the ingredients and the tools used to prepare them, but where is the process of preparing them described? Recall that a process framework is constructed by attaching a micro process to each viewpoint, describing the development of conforming views, and by defining constraints like preconditions that must be satisfied before a view is produced, postconditions that must be satisfied after it is produced, and invariants that must hold when the views have stabilized. Clearly, there is a resemblance between a process framework and a software factory schema. The viewpoints of a software factory schema already define micro processes for producing the artifacts they describe. Adding constraints to a software factory schema to govern the order of execution makes it a process framework, as well. We now have a recipe for the members of a product family.

Customizing A software factory schema

An important aspect of using a software factory schema is customization. A raw software factory schema is a recipe for building a family of software products. However, for any given project, of course, we only care about one software product. We must therefore customize the software factory schema to create a recipe for building that specific member of the product family. This is similar to what cooks do when they modify a recipe before using it, taking into account special requirements, such as the number of servings to be made, the actual ingredients or tools available, or the amount of time that will elapse before the meal is served.

software factory schema customization is actually quite straight forward. In order to support it, a software factory schema must contain both fixed and variable parts. The fixed parts remain the same for all members of the product family, while the variable parts change to accommodate the unique requirements of each specific member. A software factory schema for Online Order Entry Systems, for example, might include viewpoints that describe the artifacts and tools used to build personalization subsystems. For some members of the family, however, a personalization subsystem may not be required. Before we set out to build such a system, we would first remove the viewpoints related to personalization, configuring our project structure and tools accordingly. Different parts of the customization can be performed at different times, according to the needs of the project. Some parts, such as adding or dropping whole viewpoints, and relationships between them, might be performed up front, based on major variations in requirements, such as dropping personalization. Other parts, such as modifying viewpoints and the relationships between them, might be performed incrementally, as the project progresses, based on more fine grained variations in requirements, such as deciding what mechanisms to use to drive the user interface.

software factory templates

Now, all of this is quite appealing, but at this point, it is all on paper. If all we have is the software factory schema, then we can describe the assets used to build family members, but we do not actually have the assets. Before we can build any family members, we must implement the software factory schema, defining the DSLs, patterns, frameworks and tools it describes, packaging them, and making them available to product developers. Collectively, these assets form a software factory template.

A software factory template includes code and metadata that can be loaded into extensible tools, like an Interactive Development Environment (IDE), or an enterprise life cycle tool suite, to automate the development and maintenance of family members. We call it a software factory template because it configures the tools to produce a specific type of software, just as a document template loaded into a tool like Microsoft Word or Excel configures it to produce of a specific type of document.

Like a software factory schema, a software factory template must be customized for a specific family member. While customizing a software factory schema customizes the description of the software factory for the family member, however, customizing a template customizes the assets used to build the family member. The software factory template is generally customized at the same time as the software by adding, dropping or modifying the assets associated with those viewpoints.

Examples of software factory template customization include creating projects for the subsystems and components to be developed, populating a palette with patterns to be applied, setting up references to libraries to be used, and configuring builds. For example, enabling content personalization for an online commerce application might cause the following things to happen:

The viewpoint used to configure personalization is added to the schema.
A personalization configuration tool appears in the IDE menu.
A folder for the personalization subsystem is added to the project for the application.
A framework and patterns for personalization are imported into the project.
The Front Controller pattern is applied automatically in the transformation between the user interaction model and the web front end design model, and appears in the web front end designer, instead of the Page Controller pattern, so that the application will vector to different pages for different users, instead of showing the same content to all users.
Policy on the project folder for the presentation layer is modified so that we cannot create classes that derive from PageController.

Software Factories

We can now define a software factory as:

A software product line whose production assets form a development environment configured to support the rapid development of the target product family members.

The development environment is typically provided by customizing extensible tools using a software factory template based on a software factory schema. A schematic of a Software Factory is shown in Figure 8.

Software Supply Chains

In a mature software factory, application development consists primarily of component selection, customization, adaptation, extension and assembly. Instead of writing large amounts of new code, application developers obtain most of the required functionality from existing components. Case studies documented by Clements and Northrop, Bosch, Weiss and Lai, and others suggest that less than a third of each product is developed from scratch, and that even higher levels of reuse can be sustained in some cases.

One of the keys to implementing software factories cost effectively is not trying to build everything in house. As product developers increasingly depend on external suppliers, supply chains will emerge, as they have in other industries. According to Lee and Billington, a supply chain is a network that starts with raw materials, transforms them into intermediate goods, and then into final products delivered to customers through a distribution system. Each participant consumes logical and/or physical products from one or more upstream suppliers, adds value, usually by applying them to develop more complex products, and supplies the results to downstream consumers.

In other words, suppliers are connected, so that outputs from upstream suppliers become inputs to downstream suppliers. Upstream suppliers can supply implementation assets, such as components, or process assets, such as tools and process documentation, to downstream ones. There are important differences between these two types of assets in a supply chain. Since products produced by downstream suppliers incorporate implementation assets supplied by upstream ones, the downstream suppliers have narrower scope and produce larger products than the upstream ones. For example, a supplier of Online Commerce Systems that consumes a user interface framework has a narrower focus and produces larger products than the supplier of the user interface framework. This may not hold for process assets. The user interface framework supplier, for example, may consume configuration management and defect tracking tools that are both larger and narrower in scope than the user interface frameworks its supplies.

One of the keys to success in a software supply chain is aligning the architectures and processes of suppliers. As we have seen, one of the main impediments to component composition is architecture mismatch. If components are not designed with compatible assumptions about how they will interact with each other and with the platforms on which they execute, then they cannot be assembled without extensive modification, adaptation or mediation. The less compatible these assumptions, the greater the level of architecture mismatch, and the greater the amount of adaptation required to assemble the components. At some level of architecture mismatch, it may not be possible to assemble the components successfully. Similar alignment issues apply to the development process for suppliers that seek to consume or supply process assets. For example, a supplier that uses testing tools from one supplier may not be able to consume defect tracking tools from another supplier, if they cannot be integrated with the testing tools. Integration may require the tools to make compatible assumptions about the development process.

Software Mass Customization

Some industries, such as the web based PC business, produce product variants on demand cheaply and quickly for individual customers today. While software product mass customization will not appear for some time, the broad adoption of Software Factories will make it possible. This occurs in other industries today when businesses optimize their internal value chains. According to Porter, a value chain is a sequence of value adding activities within a single organization, connecting its supply side with its demand side. Value chain optimization requires integrating business processes like customer relationship management, demand management, product definition, product design, product assembly and supply chain management, as illustrated in Figure 9. Software product mass customization will appear when software suppliers optimize their value chains. Imagine ordering a customized financial services application from a web site, the same way customized PCs are ordered today, with the same level of confidence and comparable delivery times

Conclusion

This article has described the inability of the software industry to meet projected demand using current methods and practices, some of the problems that are holding us back, critical innovations that can help us overcome these problems, and a methodology that integrates those critical innovations called Software Factories.

Software Factories are based on the convergence of key ideas in systematic reuse, model driven development, development by assembly and process frameworks. Many of these ideas are not new. What is new is their synthesis into an integrated approach that lets organizations with domain expertise implement the four part Software Factory pattern, building languages, patterns, frameworks and tools to automate development in narrow domains.

We think key pieces of the Software Factory vision will be realized quickly, and some over several years. Commercial tools that can host Software Factories are available now, including Microsoft Visual Studio .NET and IBM WebSphere Studio. DSL technology is much newer than most of the other technologies, and relies on families of extensible languages. DSL development tools and frameworks are currently under development, however, and have already started to appear in commercial form.

References

[Bos00] J. Bosch. Design and Use of Software Architectures. Addison-Wesley, 2000.

[CN01] P. Clements and L. Northrop. Software Product Lines : Practices and Patterns. Addison-Wesley, 2001.

[Coo04] S. Cook. Domain Specific Modeling and Model Driven Architecture. MDA Journal. http://www.bptrends.com/publicationfiles/01- 04%20COL%20Dom%20Spec%20Modeling%20Frankel-Cook.pdf.

[Fow04] M. Fowler. http://martinfowler.com/bliki/UnwantedModelingLanguage.html.

[GS04] J. Greenfield and K. Short. Software Factories: Assembling Applications with Patterns, Models, Frameworks and Tools. John Wiley and Sons, 2004.

[HW02] G. Hohpe and B. Woolf. Enterprise Integration Patterns, Addison-Wesley, 2004.

[LB94] H. Lee and C. Billington. "Designing Products and Processes for Postponement" in S. Dasu and C. Eastman (eds.) Management of Design: Engineering and Management Perspectives (Kluwer, Boston 1994), pp. 105-122.

[Por85] M. Porter. "Competitive Advantage". The Free Press. New York, 1985.

[RJ96] D. Roberts and R. Johnson. Evolving Frameworks: A Pattern Language for Developing Object- Oriented Frameworks. Proceedings of Pattern Languages of Programs, Allerton Park, Illinois. 1996.

[SG96] M. Shaw and D. Garlan. Software Architecture: Perspectives On An Emerging Discipline. Prentice-Hall, Upper Saddle River, 1996.

[WL99] D.M. Weiss, C.T. Robert Lai. Software Product-Line Engineering, Addison-Wesley, 1999.

Authors

Jack Greenfield is an Architect for Enterprise Frameworks and Tools at Microsoft. He was previously Chief Architect, Practitioner Desktop Group, at Rational Software Corporation, and Founder and CTO of InLine Software Corporation. At NeXT, he developed the Enterprise Objects Framework, now called Apple Web Objects. A well known speaker and writer, he also contributed to UML, J2EE and related OMG and JSP specifications. He holds a B.S. in Physics from George Mason University.

Site Map