Saltar al contenido principal
CargarIdioma (ES)Atención al cliente
EmpresarialesMóvilRedes socialesMarketingTecnologíaArte y fotografíaEmpleoDiseñoEducaciónPresentaciones y charlas públicasGobierno y org. sin ánimo de lucroAtención sanitariaInternetDerechoLiderazgo y gestiónMotorIngenieríaSoftwareReclutamiento y RR. HH.Venta al por menorVentasServiciosCienciasPYMES y liderazgoAlimentaciónMedio ambienteEconomía y finanzasDatos y análisisRelaciones con inversoresDeportesMeditaciónNoticias y políticaViajesDesarrollo personalSector inmobiliarioEntretenimiento y humorSalud y medicinaDispositivos y hardwareEstilo de vida
Subido porssiliveri
176 vistas

Soa In The Real World

This document discusses service oriented architecture (SOA) and its application in real world systems. It begins with an introduction to SOA concepts like services, reuse, and loose coupling. It then discusses common architectural capabilities like messaging, workflow, data management and user experience that are important in SOA. The document provides an abstract reference model for SOA and shows how the common capabilities relate to the model's phases of expose, compose and consume. Later chapters discuss specific capabilities like messaging and workflow in more depth and provide examples.

Descargar para leer sin conexión
e Service Oriented ARC Architecture (SOA) in the Real World
Table of Contents Chapter 1: Service Oriented Architecture (SOA) ............................................................................. 7 Reader ROI .................................................................................................................................. 7 Acknowledgements ...................................................................................................................... 7 Introduction to SOA ...................................................................................................................... 8 The SOA Elephant ................................................................................................................... 8 A simple definition for SOA ...................................................................................................... 9 SOA Myths and Facts ............................................................................................................ 11 The Evolution of SOA ............................................................................................................. 12 Why should I care about SOA? .............................................................................................. 14 Understanding Services ............................................................................................................. 16 The Tenets of Service Design ................................................................................................ 18 Tenet 1: Boundaries are Explicit ............................................................................................ 18 Tenet 2: Services Are Autonomous ....................................................................................... 20 Tenet 3: Services share schema and contract, not class ...................................................... 21 Tenet 4: Service compatibility Is based upon policy .............................................................. 23 An Abstract SOA Reference Model ........................................................................................... 24 Expose.................................................................................................................................... 25 Compose ................................................................................................................................ 25 Consume ................................................................................................................................ 26 Recurring Architectural Capabilities ........................................................................................... 27 Messaging and Services ........................................................................................................ 27 Workflow and Process ............................................................................................................ 28 Data ........................................................................................................................................ 28 User Experience ..................................................................................................................... 28 Identity and Access ................................................................................................................ 28 Management........................................................................................................................... 29 Supporting the Common Architectural Capabilities ............................................................... 29 Common Architectural Capabilities and the Abstract SOA Model ............................................. 30 Expose.................................................................................................................................... 30 Compose ................................................................................................................................ 33 Consume ................................................................................................................................ 34 Summary .................................................................................................................................... 36 References: ................................................................................................................................ 38 Chapter 2: Messaging and Services ............................................................................................ 39 Reader ROI ................................................................................................................................ 39
Acknowledgements .................................................................................................................... 40 Understanding Services ............................................................................................................. 41 A SOA Maturity Model (another one??) ................................................................................. 41 A Service Taxonomy .............................................................................................................. 45 A Services Lifecycle ................................................................................................................... 55 Service Analysis ..................................................................................................................... 55 Service Development ............................................................................................................. 56 Service Testing ....................................................................................................................... 56 Service Provisioning ............................................................................................................... 56 Service Operation ................................................................................................................... 56 Service Consumption ............................................................................................................. 57 Service Change Management ................................................................................................ 57 Service Decommission ........................................................................................................... 57 SOA Scenarios........................................................................................................................... 58 Information Integration ........................................................................................................... 58 Legacy Integration .................................................................................................................. 58 Process Governance .............................................................................................................. 58 Consistent Access .................................................................................................................. 59 Resource Virtualization .......................................................................................................... 59 Process Externalization .......................................................................................................... 59 Other Scenarios ..................................................................................................................... 59 SOA and the End User .............................................................................................................. 60 What are Composite Applications? ........................................................................................ 62 What does a Composite Application look like? ...................................................................... 65 Expected Benefits of Composition, and How to Achieve Them ............................................. 67 Conclusion ................................................................................................................................. 67 SOA Case Study: Commonwealth Bank of Australia ................................................................ 69 References: ................................................................................................................................ 71 Chapter 3: Workflow and Process ................................................................................................ 72 Reader ROI ................................................................................................................................ 72 Acknowledgements .................................................................................................................... 73 Understanding Workflow ............................................................................................................ 74 What is Workflow? .................................................................................................................. 74 Workflow Terminology ............................................................................................................ 74 Why Workflow? ...................................................................................................................... 75 A Workflow Model .................................................................................................................. 76 Workflow Contracts ................................................................................................................ 77
Problem-Resolution Collaboration ......................................................................................... 78 Scripted Operations ................................................................................................................ 80 Rule and Policy ...................................................................................................................... 81 Workflow Platform Value ........................................................................................................ 83 More Semantic Exploitation .................................................................................................... 85 Platform Characteristics ......................................................................................................... 86 A Common Workflow Runtime ............................................................................................... 87 Attacking the Problems .......................................................................................................... 88 A Workflow Manifesto ................................................................................................................ 89 Agility ...................................................................................................................................... 89 Abstraction.............................................................................................................................. 90 Workflow is Everywhere ......................................................................................................... 90 Workflow is Expressive .......................................................................................................... 95 Workflow is Fluid .................................................................................................................... 96 Workflow is Inclusive .............................................................................................................. 97 Workflow is Transparent ......................................................................................................... 97 Understanding the Relationship between BizTalk Server and WF ............................................ 98 Conclusion ............................................................................................................................... 100 SOA Case Study: Dollar Thrifty Automotive Group ................................................................. 101 References: .............................................................................................................................. 102 Chapter 4: Data ........................................................................................................................... 103 Reader ROI .............................................................................................................................. 103 Acknowledgements .................................................................................................................. 103 Data Challenges Facing SOA .................................................................................................. 104 Overview............................................................................................................................... 104 Data Integration Issues ........................................................................................................ 104 Database Scalability ............................................................................................................. 107 Master Data Management (MDM) ........................................................................................... 109 What is MDM? ...................................................................................................................... 110 Customer Data Integration (CDI) ......................................................................................... 111 Product Information Management (PIM) .............................................................................. 111 Master Data Management (MDM) Hub Architecture ............................................................ 111 Hub Architecture Styles ........................................................................................................ 112 Architectural Issues .............................................................................................................. 116 Versions and Hierarchies ..................................................................................................... 117 Population and Synchronization ........................................................................................... 122 Publishing Updates .............................................................................................................. 128
Data Integrity and Reliability................................................................................................. 130 Metadata............................................................................................................................... 130 Stewardship and Governance .............................................................................................. 131 Data Profiling ........................................................................................................................ 132 Export ................................................................................................................................... 132 Reporting .............................................................................................................................. 132 Workflow and Business Rules .............................................................................................. 132 Tools ..................................................................................................................................... 133 Conclusion ............................................................................................................................... 133 SOA Case Study: London Stock Exchange ............................................................................. 134 References: .............................................................................................................................. 135 Chapter 5: User Interaction ........................................................................................................ 136 Reader ROI .............................................................................................................................. 136 Acknowledgements .................................................................................................................. 136 What is Architecture? ............................................................................................................... 137 Introducing a Framework for UX .............................................................................................. 138 Interface................................................................................................................................ 139 Interaction ............................................................................................................................. 146 Infrastructure ........................................................................................................................ 151 SOA Case Study: Zurich Airport .............................................................................................. 162 References: .............................................................................................................................. 163 Chapter 6: Identity and Access................................................................................................... 164 Reader ROI .............................................................................................................................. 164 Acknowledgements .................................................................................................................. 165 Identity and Access .................................................................................................................. 166 Overview............................................................................................................................... 166 Trusted Subsystem Design ...................................................................................................... 168 Current Practices .................................................................................................................. 169 Trusted Subsystem Design .................................................................................................. 175 Trusted subsystem process extensions ............................................................................... 177 Trusted Subsystem Policies ................................................................................................. 178 Flowing an Original Caller‘s Identity Claims ......................................................................... 179 Identity/credential mapping .................................................................................................. 182 Benefits of the Design .......................................................................................................... 182 An Identity Metasystem ............................................................................................................ 183 What is the Identity Metasystem? ........................................................................................ 184 Identities Function in Contexts ............................................................................................. 185
The Laws of Identity ............................................................................................................. 186 Roles within the Identity Metasystem ................................................................................... 186 Components of the Identity Metasystem .............................................................................. 187 Benefits of the Identity Metasystem ..................................................................................... 189 An Architecture for the Identity Metasystem: WS-* Web Services...................................... 190 Implementing the Identity Metasystem ................................................................................. 191 Conclusion ............................................................................................................................... 194 SOA Case Study: OTTO .......................................................................................................... 195 References: .............................................................................................................................. 196
Chapter 1: Service Oriented Architecture (SOA) “SOAs are like snowflakes – no two are alike.” - David Linthicum Consultant Reader ROI Readers of this chapter will learn about some of the general concepts generally associated with Service Oriented Architectures (SOA). The chapter provides several analogies for understanding service oriented concepts and some high level recommendations for designing services. This chapter provides an abstract model for describing SOA and introduces a set of architectural capabilities that will be explored in subsequent chapters of this book. Acknowledgements The vast majority of this book consists of content from a wide variety of people. Some of this content is new while other content may have appeared in other formats. Each chapter in this book will begin with an ―Acknowledgements‖ section to thank the authors of the content that comprises each chapter. Many of the concepts in Chapter One are based upon earlier published efforts. We wish to thank the following individuals for their work in this area: Don Box (Four Tenets), John deVadoss (Recurring Architectural Capabilities), and Kris Horrocks (Expose/Compose/Consume).
Introduction to SOA The SOA Elephant SOA has become a well-known and somewhat divisive acronym. If one asks two people to define SOA one is likely to receive two very different, possibly conflicting, answers. Some describe SOA as an IT infrastructure for business enablement while others look to SOA for increasing the efficiency of IT. In many ways SOA is a bit like John Godfrey Saxe‘s poem about the blind men and the elephant. Six blind men from Indostan encounter an elephant – each of the men then describes the elephant a bit differently because they are influenced by their individual experiences: The man touching the trunk believes it to be a snake The man touching the tusk believes it to be a spear The man touching the ear believes it to be a fan The man touching the elephant‘s side believes it to be a wall The man touching the tail believes it to be a rope The man touching the legs believes they are trees. Figure 1: Saxe's Elephant The blind men then engage in a series of debates about what they believe are facing them: “…And so these men of Indostan Disputed loud and long, Each in his own opinion Exceeding stiff and strong,
Though each was partly in the right, And all were in the wrong!” In many ways Mr. Saxe‘s poem has become a prophecy for SOA. Industry analysts, pundits, bloggers and reporters engage each other in an ongoing, never-ending debate about what is or isn‘t SOA. Like Mr. Saxe‘s blind men, people have correctly identified many of the capabilities of SOA but largely fail to communicate the concept as a whole. The challenge of defining SOA has become so important that various vendor consortia and standards organizations have launched initiatives to try and answer the question ―What is SOA?‖ A simple definition for SOA For the purposes of this book, we will define SOA as: A loosely-coupled architecture designed to meet the business needs of the organization. At first glance this definition seems far too simplistic – where is SOAP, web services, WSDL, WS- * and other related standards? A SOA does not necessarily require the use of Web Services – Web Services are, for most organizations, the simplest approach for implementing a loosely coupled architecture. In the past, loosely coupled architectures have relied upon other technologies like CORBA and DCOM or document-based approaches like EDI for B2B integration. Many of these technologies are still in widespread use and are being augmented, replaced or extended with Web Services. Our definition works for us not because the focus here is not on the technology of SOA but upon meeting the needs of the organization. In simpler terms, one organization‘s SOA may look like nothing more than a bunch of Web Services (or other technologies) to another. There may be some common infrastructure capabilities such as logging and authentication, but for the most part a SOA for one organization will be quite different from the SOA used by another. Many analysts and industry pundits have confused the concept of Service Oriented Architecture with Service Oriented Implementations. This has only added to the confusion associated with SOA and its related concepts. This can lead to disastrous results. The Winchester Mystery House is an intriguing tourist attraction in the USA near San Jose, CA. The Winchester Mystery House was the home to the heiress of the Winchester fortune (amassed from the sales of Winchester rifles). According to the legend, the heiress went to see a fortune teller and learned she was cursed to be haunted by the spirits of everyone ever killed by a Winchester rifle. The only way to avoid the curse was to build a mansion – as long as she kept building the spirits would leave her alone. She promptly hired 147 builders (and 0 architects), all of whom began working on the mansion simultaneously. The builders worked on the mansion
until the heiress passed away, 38 years later. The result of their efforts is a classic example of implementation without architecture: The mansion contains 160 rooms, 40 bedrooms, 6 kitchens, 2 basements and 950 doors Of there 950 doors, 65 of them open to blank walls, 13 staircases were built and abandoned and 24 skylights were installed into various floors. No architectural blueprint for the mansion was ever created. Figure 2: The Winchester Mystery House Confusing architecture with implementation generates chaotic and unpredictable results – much like the Winchester Mystery House. Articles that try to explain SOA and jump into a tutorial for building Web Services are providing guidance for coding, not architecture. This is one of the many reasons that SOA is so misunderstood today – the rush to promote loosely coupled architectures focuses on the trees instead of the forest. The architectural concepts associated with SOA are not new – many have evolved from ideas originally introduced by CORBA, DCOM, DCE and others. Unlike these previous initiatives, the key promise of SOA is to enable agile business processes via open, standards-based interoperability. While these standards are important we must remember that standards are not architecture and architectures are not implementations. At the end of the day it is the implementation of a well-designed architecture that will generate business benefits, not the architecture itself. SOA is an architectural approach to creating systems built from autonomous services. With SOA, integration becomes forethought rather than afterthought - the end solution is likely to be composed of services developed in different programming languages, hosted on disparate platforms with a variety of security models and business processes. While this concept sounds incredibly complex it is not new – some may argue that SOA evolved out of the experiences associated with designing and developing distributed systems based on previously available
technologies. Many of the concepts associated with SOA such as services, discovery and late binding were associated with CORBA and DCOM. Similarly, many service design principles have much in common with earlier OOA/OOD techniques based upon encapsulation, abstraction and clearly defined interfaces. Does the buzz around SOA and services mean that IT wasn’t service-oriented in the past? No - IT (outsourced or not) exists solely to enable the business. Without IT businesses will have tremendous difficulty in both execution and competition. However, if IT cannot respond to business needs and opportunities fast enough then IT is perceived as a constrainer to the business instead of an enabler. SOA promises to help IT respond to market conditions in a much timelier manner. SOA, however, is an architectural philosophy and is not necessarily an implementable concept. Many analysts and trade magazines have confused architecture with implementation – this leads one to believe that an implementation is, in fact, an architecture which can lead to disastrous results. Organizations have different requirements and expectations for SOA because of vastly different business needs and objectives. This simple fact is one of the reasons that describing SOA can be such a challenge. SOA, like any initiative, must provide some level of value to the organization – otherwise there would be no use in ever considering it. The best way to ensure that SOA investments will provide a return to the organization is to align SOA with the organization‘s business drivers. Despite this obvious fact there is still a lot of confusion about SOA. SOA Myths and Facts There are several myths associated with SOA which are very important to understand before digging deeper into it. The table below describes some of the top myths surrounding SOA and the facts to help debunk them. Myth Fact SOA is a technology SOA is a design philosophy independent of any vendor, product, technology or industry trend. No vendor will ever offer a ―complete‖ SOA ―stack‖ because SOA needs vary from one organization to another. Purchasing your SOA infrastructure from a single vendor defeats the purpose of investing in SOA. SOAs require Web Services SOAs may be realized via Web services but Web services are not necessarily required to implement SOA
Myth Fact SOA is new and revolutionary EDI, CORBA and DCOM were conceptual examples of SO SOA ensures the alignment of IT and business SOA is not a methodology A SOA Reference Architecture reduces implementation risk SOAs are like snowflakes – no two are the same. A SOA Reference Architecture may not necessarily provide the best solution for your organization SOA requires a complete technology and business SOA should be incremental and built upon your current processes overhaul investments We need to build a SOA SOA is a means, not an end Focus on delivering a solution, not an SOA. SOA is a means to delivering your solution and should not be your end goal. The Evolution of SOA Service Orientation (SO) is the natural evolution of current development models. The, 80s saw object-oriented models; then came the component-based development model in the 90s; and now we have service orientation (SO). Service orientation retains the benefits of component- based development (self-description, encapsulation, dynamic discovery and loading), but there is a shift in paradigm from remotely invoking methods on objects, to one of passing messages between services. Schemas describe not only the structure of messages, but also behavioral contracts to define acceptable message exchange patterns and policies to define service semantics. This promotes interoperability, and thus provides adaptability benefits, as messages can be sent from one service to another without consideration of how the service handling those messages has been implemented. Figure 3: Simple SOAP-based communications between Web Services
Service orientation provides an evolutionary approach to building distributed software that facilitates loosely coupled integration and resilience to change. With the advent of the WS-* Web Services, architecture has made service-oriented software development feasible by virtue of mainstream development tools support and broad industry interoperability. Although most frequently implemented using industry standard Web services, service orientation is independent of technology and its architectural patterns and can be used to connect with legacy systems as well. Unfortunately, the benefits offered by service orientation and SOA have been obscured by the hype and confusion that increasingly surround the terms. As awareness and excitement around SOA have swelled, the clear lines that once defined service orientation have been blurred. However SO does offer some specific benefits when utilized for the right purpose. There are three important observations about SO: 1. It’s evolutionary: The principles of service-oriented development build on decades of experience in building real world distributed applications. SO incorporates concepts such as self-describing applications, explicit encapsulation, and dynamic loading of functionality at runtime – principles first introduced in the 1980s and 1990s through object-oriented and component-based development. What changes with SO is the metaphor with which developers achieve these benefits. Instead of using method invocation on an object reference, service orientation shifts the conversation to that of message passing – a proven metaphor for scalable distributed software integration. 2. It’s not a product or technology: It is a set of architectural principles expressed independently of any product. Just as development concepts such as polymorphism and encapsulation are independent of technology, so is service orientation. And while Web services have in recent years facilitated the development of service-oriented applications, they are not required to do so. 3. It’s incremental: Finally, service orientation can and should be an incremental process – one that can often be done in-house. Customers should not be required to dramatically re- engineer their businesses to attain the benefits of service orientation. Rather, they should be able to leverage existing IT assets in doing so. Service-oriented development can often be achieved using the skills and technologies customers already have today. The fundamental building block of service-oriented architecture is a service. A service is a program that can be interacted with through well-defined message exchanges. Services must be designed for both availability and stability. Services are built to last while service configurations and aggregations are built for change. Agility is often promoted as one of the biggest benefits of SOA—an organization with business processes implemented on a loosely-coupled infrastructure is much more open to change than an organization constrained by underlying monolithic applications that require weeks to implement the smallest change. Loosely-coupled systems
result in loosely-coupled business processes, since the business processes are no longer constrained by the limitations of the underlying infrastructure. Services and their associated interfaces must remain stable, enabling them to be re-configured or re-aggregated to meet the ever-changing needs of business. Services remain stable by relying upon standards-based interfaces and well-defined messages— for example using SOAP and XML schemas for message definition. Services designed to perform simple, granular functions with limited knowledge of how messages are passed to or retrieved from it are much more likely to be reused within a larger SOA infrastructure. Service Orientation does not necessarily require rewriting functionality from the ground up. Following the four tenets (see below) enables reuse of existing IT assets by wrapping them into modular services that can be plugged into any business process that you design. The goals for doing this should be: Connect into what is already there - Layer business process management, collaborative workflows, and reporting on top of existing IT assets. Extract more value from what is already there - Enable existing applications to be re-used in new ways. Extend and evolve what we already have - Create IT support for new cross-functional business processes that extend beyond the boundaries of what the existing applications were designed to do. One of the key benefits of service orientation is loose coupling. No discussion of Web services seems complete without some reference to the advantages of looser coupling of endpoints (applications) facilitated by the use of Web service protocols. The principle is that of using a resource only through its published service and not by directly addressing the implementation behind it. This way, changes to the implementation by the service provider should not affect the service consumer. By maintaining a consistent interface, the service consumer could choose an alternative instance of the same service type (for example change service provider) without modifying their requesting application, apart from the address of the new instance. The service consumer and provider do not have to have the same technologies for the implementation, interface, or integration when Web services are used (though both are bound to use the same Web service protocols). Why should I care about SOA? Service Orientated Architecture is important to several stakeholders:  To developers and solution architects, service orientation is a means for creating dynamic, collaborative applications. By supporting run-time selection of capability providers, service orientation allows applications to be sensitive to the content and
context of a specific business process, and to gracefully incorporate new capability providers over time.  To the IT manager, service orientation is a means for effectively integrating the diverse systems typical of modern enterprise data centers. By providing a model for aggregating the information and business logic of multiple systems into a single interface, service orientation allows diverse and redundant systems to be addressed through a common, coherent set of interfaces.  To the CIO, service orientation is a means for protecting existing IT investments without inhibiting the deployment of new capabilities. By encapsulating a business application behind capability-based interfaces, the service model allows controlled access to mission-critical applications, and creates the opportunity for continuous improvement of the implementation behind that interface. Service orientation protects investments from the swirl of change.  To the business analyst, service orientation is a means of bringing information technology investments more in line with business strategy. By mapping employees, external capability providers and automation systems into a single model, the analyst can better understand the cost tradeoffs associated with investments in people, systems and sourcing.  To Microsoft, service orientation is a crucial prerequisite to creating applications that leverage the network to link the actors and systems that drive business processes. This application model transcends any single device, crossing boundaries respectfully, and rejecting the restrictions of synchronicity. SOA-enabled solutions pull together a constellation of services and devices to more effectively meet your business challenges than the disconnected applications of the past. The architectural concepts associated with SOA enable loose coupling. Loose coupling is the fundamental principle behind SOA, enabling us to summarize up the benefit of SOA in a single word: agility. As traditional application architectures evolved from the mainframe application through client server to multi-tier web applications, the applications have remained to a large extent tightly coupled. In other words, each of the subsystems that comprise the greater application is not only semantically aware of its surrounding subsystems, but is physically bound to those subsystems at compile time and run time. The ability to replace key pieces of functionality in reaction to a change in a business model, or of distributing an application as individual business capabilities, is simply not possible.
With a Service Oriented Architecture the application‘s functionality is exposed through a collection of services. These services are independent and encapsulate both the business logic and its associated data. The services are interconnected via messages with a schema defining their format; a contract defining their interchanges and a policy defining how they should be exchanged. Messages Service Policy Schema and Contact Figure 4: Anatomy of a Web Service An application‘s services are designed to last with the expectation that we cannot control where and who consumes them. This is one of the key differences between SOA and traditional application architectures. Traditionally applications are designed to interact with humans, and the application vendor would provide the user-interface, the underlying business components and data stores. While good engineering disciplines separated the business components from the user interface, the only consumers of this business logic were the user interfaces delivered as part of the application. These UI and business components were traditionally deployed and versioned as a single entity. With service orientation, the business functionality exposed through the service can be used by any consumer outside of the control of the application. These consumers can be other services (incorporating the encapsulated business process) or alternatives to the user interfaces provided by the application itself. Therefore, contracts for these services, once published, must remain constant as we have no idea who is consuming them, nor when. In addition to offering services and exploiting its own services, an application itself should have the flexibility to adapt to new services offered after deployment. The availability and stability of these services therefore becomes a critical factor. Understanding Services The first step in any SOA undertaking is to clearly identify the critical business problems or challenges. The more precisely these can be defined the easier it will be to determine the scope and direction of each SOA project. By setting clear vision and direction from the top, it will be easier to obtain buy in on projects that are cross-functional in nature. Once the business drivers of the organization are defined, the service analysis process can begin. Service analysis is one of several steps that comprise a Service Lifecycle (Chapter Two provides more information about
the Service Lifecycle). The Service Lifecycle explains the necessary steps an organization must go through to define, develop, deploy and operate a service. Services are commonly used to expose IT investments such as legacy platforms and Line of Business applications. Services can be assembled (or ―composed‖) into business processes, and be made available for consumption by users, systems or other services. The process is an iterative one of creating (―exposing‖) new services, aggregating (―composing‖) these services into larger composite applications, and making the outputs available for consumption by the business user. Figure 5: An incremental, business-driven approach to SOA Fundamental to the service model is the separation between the interface and the implementation. The invoker of a service need only (and should only) understand the interface; the implementation can evolve over time without disturbing the clients of the service. Several key benefits of service orientation derive from this abstraction of the capability from how the capability is delivered. This means that, the same interface can be offered by many implementations, or conversely, that implementations can change without affecting the aggregate application. At its most abstract, service orientation views everything — from the mainframe application to the printer to the shipping dock clerk to the overnight delivery company — as a service provider. The service model is ―fractal:‖ the newly formed process is a service itself, exposing a new, aggregated capability. What is a service? In this book we will avoid using the term Web Services simply because all services are not necessarily Web Services. A service might also be manifested as an OS- specific process like a Unix daemon or a Windows Service. A service might also be an application that uses a well-defined contract that may or may not be based upon Web Services. Regardless of how the actual services are developed, they must be capable of participating within a loosely coupled architecture. There are four distinct principles (sometimes referred to as ―Tenets‖) that can help ensure a loosely-coupled architecture. These Tenets of Service Design are defined below:
The Tenets of Service Design The acronym SOA prompts an obvious question – what, exactly, is a service? Simply put, a service is a program that can be interacted with via well-defined message exchanges. Services must be designed for both availability and stability. Services are built to last while service configurations and aggregations are built for change. Agility is often promoted as one of the biggest benefits of SOA – an organization with business processes implemented on a loosely- coupled infrastructure is much more open to change than an organization constrained underlying monolithic applications that require weeks to implement the smallest change. Loosely-coupled systems result in loosely-coupled business processes, since the business processes are no longer constrained by the limitations of the underlying infrastructure. Services and their associated interfaces must remain stable, enabling them to be re-configured or re-aggregated to meet the ever-changing needs of business. Services remain stable by relying upon standards- based interfaces and well-defined messages – in other words, SOAP and XML schemas for message definition. Services designed to perform simple, granular functions with limited knowledge of how messages are passed to or retrieved from it are much more likely to be reused within a larger SOA infrastructure. As stated earlier, recalling basic OO design principles regarding encapsulation and interface design will serve us well as we design and build reusable Web Services. We can extend these OO principles into the world of Web Services by further understanding the frequently cited ―four tenets‖ of Service Orientation: Tenet 1: Boundaries are Explicit Services interact through explicit message-passing over well-defined boundaries. Crossing service boundaries may be costly, depending upon geographic, trust or execution factors. A boundary represents the border between a service‘s public interface and its internal, private implementation. A service‘s boundary is published via WSDL and may include assertions dictating the expectations of a given service. Crossing boundaries is assumed to be an expensive task for several reasons, some of which are listed below: The physical location of the targeted service may be an unknown factor. Security and trust models are likely to change with each boundary crossing. Marshalling and casting of data between a service‘s public and private representations may require reliance upon additional resources – some of which may be external to the service itself.
While services are built to last, service configurations are built to change. This fact implies that a reliable service may suddenly experience performance degradations due to network reconfigurations or migration to another physical location. Service consumers are generally unaware of how private, internal processes have been implemented. The consumer of a given service has limited control over the performance of the service being consumed. The Service Oriented Integration Pattern tells us that ―service invocations are subject to network latency, network failure, and distributed system failures, but a local implementation is not. A significant amount of error detection and correction logic must be written to anticipate the impacts of using remote object interfaces.‖ While we should assume that crossing boundaries is an expensive process, we must also exercise caution in the deployment of local methods designed to minimize such boundary crossings. A system that implements monolithic local methods and objects may gain performance but duplicate functionality of a previously defined service (this technique was referred to as ―cut and paste‖ in OOP and shares the same risks regarding versioning of the service). There are several principles to keep in mind regarding the first Tenet of SO: Know your boundaries. Services provide a contract to define the public interfaces it provides. All interaction with the service occurs through the public interface. The interface consists of public processes and public data representations. The public process is the entry point into the service while the public data representation represents the messages used by the process. If we use WSDL to represent a simple contract, the < message> represents the public data while the <portType> represents the public process(es). Services should be easy to consume. When designing a service, developers should make it easy for other developers to consume it. The service‘s interface (contract) should also be designed to enable evolving the service without breaking contracts with existing consumers. (This topic will be addressed in greater detail in future papers in this Series.) Avoid RPC interfaces. Explicit message passing should be favored over an RPC-like model. This approach insulates the consumer from the internals of the service implementation, freeing service developers to evolve their services while minimizing the impact on service consumers (encapsulation via public messages instead of publicly available methods). Keep service surface area small. The more public interfaces that a service exposes the more difficult it becomes to consume and maintain it. Provide few well-defined public interfaces to your service. These interfaces should be relatively simple, designed to
accept a well-defined input message and respond with an equally well-defined output message. Once these interfaces have been designed they should remain static. These interfaces provide the ―constant‖ design requirement that services must support, serving as the public face to the service‘s private, internal implementation. Internal (private) implementation details should not be leaked outside of a service boundary. Leaking implementation details into the service boundary will most likely result in a tighter coupling between the service and the service‘s consumers. Service consumers should not be privy to the internals of a service‘s implementation because it constrains options for versioning or upgrading the service. Tenet 2: Services Are Autonomous Services are entities that are independently deployed, versioned, and managed. Developers should avoid making assumptions regarding the space between service boundaries since this space is much more likely to change than the boundaries themselves. For example, service boundaries should be static to minimize the impact of versioning to the consumer. While boundaries of a service are fairly stable, the service‘s deployment options regarding policy, physical location or network topology is likely to change. Services are dynamically addressable via URIs, enabling their underlying locations and deployment topologies to change or evolve over time with little impact upon the service itself (this is also true of a service‘s communication channels). While these changes may have little impact upon the service, they can have a devastating impact upon applications consuming the service. What if a service you were using today moved to a network in New Zealand tomorrow? The change in response time may have unplanned or unexpected impacts upon the service‘s consumers. Service designers should adopt a pessimistic view of how their services will be consumed – services will fail and their associated behaviors (service levels) are subject to change. Appropriate levels of exception handling and compensation logic must be associated with any service invocation. Additionally, service consumers may need to modify their policies to declare minimum response times from services to be consumed. For example, consumers of a service may require varying levels of service regarding security, performance, transactions, and many other factors. A configurable policy enables a single service to support multiple SLAs regarding service invocation (additional policies may focus on versioning, localization and other issues). Communicating performance expectations at the service level preserves autonomy since services need not be familiar with the internal implementations of one another. Service consumers are not the only ones who should adopt pessimistic views of performance – service providers should be just as pessimistic when anticipating how their services are to be consumed. Service consumers should be expected to fail, sometimes without notifying the
service itself. Service providers also cannot trust consumers to ―do the right thing‖. For example, consumers may attempt to communicate using malformed/malicious messages or attempt to violate other policies necessary for successful service interaction. Service internals must attempt to compensate for such inappropriate usage, regardless of user intent. While services are designed to be autonomous, no service is an island. A SOA-based solution is fractal, consisting of a number of services configured for a specific solution. Thinking autonomously, one soon realizes there is no presiding authority within a service-oriented environment - the concept of an orchestration ―conductor‖ is a faulty one (further implying that the concept of ―roll-backs‖ across services is faulty– but this is a topic best left for another paper). The keys to realizing autonomous services are isolation and decoupling. Services are designed and deployed independently of one another and may only communicate using contract-driven messages and policies. As with other service design principles, we can learn from our past experiences with OO design. Peter Herzum and Oliver Sims‘ work on Business Component Factories provides some interesting insights on the nature of autonomous components. While most of their work is best suited for large-grained, component-based solutions, the basic design principles are still applicable for service design, Given these considerations, here are some simple design principles to help ensure compliance with the second principle of SO: Services should be deployed and versioned independent of the system in which they are deployed and consumed Contracts should be designed with the assumption that once published, they cannot be modified. This approach forces developers to build flexibility into their schema designs. Isolate services from failure by adopting a pessimistic outlook. From a consumer perspective, plan for unreliable levels of service availability and performance. From a provider perspective, expect misuse of your service (deliberate or otherwise),expect your service consumers to fail – perhaps without notifying your service. Tenet 3: Services share schema and contract, not class As stated earlier, service interaction should be based solely upon a service‘s policies, schema, and contract-based behaviors. A service's contract is generally defined using WSDL, while contracts for aggregations of services can be defined using BPEL (which, in turn, uses WSDL for each service aggregated).
Most developers define classes to represent the various entities within a given problem space (e.g. Customer, Order and Product). Classes combine behavior and data (messages) into a single programming-language or platform-specific construct. Services break this model apart to maximize flexibility and interoperability. Services communicating using XML schema-based messages are agnostic to both programming languages and platforms, ensuring broader levels of interoperability. Schema defines the structure and content of the messages, while the service‘s contract defines the behavior of the service itself. In summary, a service‘s contract consists of the following elements: Message interchange formats defined using XML Schema Message Exchange Patterns (MEPs) defined using WSDL Capabilities and requirements defined using WS-Policy BPEL may be used as a business-process level contract for aggregating multiple services. Service consumers will rely upon a service‘s contract to invoke and interact with a service. Given this reliance, a service‘s contract must remain stable over time. Contracts should be designed as explicitly as possible while taking advantage of the extensible nature of XML schema (xsd:any) and the SOAP processing model (optional headers). The biggest challenge of the Third Tenet is its permanence. Once a service contract has been published it becomes extremely difficult to modify it while minimizing the impact upon existing service consumers. The line between internal and external data representations is critical to the successful deployment and reuse of a given service. Public data (data passed between services) should be based upon organizational or vertical standards, ensuring broad acceptance across disparate services. Private data (data within a service) is encapsulated within a service. In some ways services are like smaller representations of an organization conducting e-business transactions. Just as an organization must map an external Purchase Order to its internal PO format, a service must also map a contractually agreed-upon data representation into its internal format. Once again our experiences with OO data encapsulation can be reused to illustrate a similar concept – a service‘s internal data representation can only be manipulated through the service‘s contract. Given these considerations, here are some simple design principles to help ensure compliance with the third principle of SO:
Ensure a service‘s contract remains stable to minimize impact upon service consumers. The contract in this sense refers to the public data representation (data), message exchange pattern (WSDL) and configurable capabilities and service levels (policy). Contracts should be designed to be as explicit as possible to minimize misinterpretation. Additionally, contracts should be designed to accommodate future versioning of the service via the extensibility of both the XML syntax and the SOAP processing model. Avoid blurring the line between public and private data representations. A service‘s internal data format should be hidden from consumers while its public data schema should be immutable (preferably based upon an organizational, defacto or industry standard). Version services when changes to the service‘s contract are unavoidable. This approach minimizes breakage of existing consumer implementations. Tenet 4: Service compatibility Is based upon policy While this is often considered the least understood design tenet, it is perhaps one of the most powerful in terms of implementing flexible web services. It is not possible to communicate some requirements for service interaction in WSDL alone. Policy expressions can be used to separate structural compatibility (what is communicated) from semantic compatibility (how or to whom a message is communicated). Operational requirements for service providers can be manifested in the form of machine- readable policy expressions. Policy expressions provide a configurable set of interoperable semantics governing the behavior and expectations of a given service. The WS-Policy specification defines a machine-readable policy framework capable of expressing service-level policies, enabling them to be discovered or enforced at execution time. For example, a government security service may require a policy enforcing a specific service level (e.g. Passport photos meeting established criteria must be cross-checked against a terrorist identification system). The policy information associated with this service could be used with a number of other scenarios or services related to conducting a background check. WS-Policy can be used to enforce these requirements without requiring a single line of additional code. This scenario illustrates how a policy framework provides additional information about a service‘s requirements while also providing a declarative programming model for service definition and execution. A policy assertion identifies a behavior that is a requirement (or capability) of a policy subject. (In the scenario above the assertion is the background check against the terrorist identification system.) Assertions provide domain-specific semantics and will eventually be defined within
separate, domain-specific specifications for a variety of vertical industries (establishing the WS- Policy ―framework‖ concept). While policy-driven services are still evolving, developers should ensure their policy assertions are as explicit as possible regarding service expectations and service semantic compatibilities. The four tenets are primarily designed to assist you in designing and developing your services. An Abstract SOA Reference Model While a well planned and executed SOA undertaking can help organizations realize greater responsiveness in a changing marketplace, not all service-oriented efforts have been successful. SOA projects may experience limited success when they are driven from the bottom up by developers unfamiliar with the strategic needs of the organization. Building SOA for the sake of SOA without reference to the business context is a project without organizing principles and guidance. The result is a chaotic implementation that has no business relevance. On the other hand, taking a top-down mega-approach to SOA requires such enormous time investments that by the time the project is complete, the solution no longer maps to business needs. (This of course is one of the problems SOA is supposed to solve!) By contrast, Microsoft advocates a ―middle out‖ approach which combines both top-down and bottom-up methodologies. In this approach, SOA efforts are driven by strategic vision and business need, and are met through incremental, iterative SOA projects that are designed to deliver on business goals one business need at a time. Microsoft has been using this technique to assist customers with their SOA initiatives since the .NET framework was first released in 1999. The concept of SOA can be viewed from several possible perspectives. While no single perspective or set of perspectives represents a definitive view of a SOA, from a holistic view these perspectives assist in understanding the underlying architectural requirements. Microsoft believes that there are three abstract capability layers exposed within a SOA: An illustration of these categories and their relationships to one another appears below:
Figure 6: An Abstract Reference Model for SOA Expose Expose focuses on how existing IT investments are exposed as a set of broad, standards-based services, enabling these investments to be available to a broader set of consumers. Existing investments are likely to be based upon a set of heterogeneous platforms and vendors. If these applications are unable to natively support Web services a set of application or protocol-specific set of adapters may be required. Service creation can be fine grained (a single service that maps on to a single business process), or coarse grained (multiple services come together to perform a related set of business functions). Expose is also concerned with how the services are implemented. The functionality of underlying IT resources can be made available natively if they already speak Web services, or can be made available as Web services though use of an adapter. A Service Implementation Architecture describes how services are developed, deployed and managed. Compose Once services are created, they can be combined into more complex services, applications or cross-functional business processes. Because services are exist independently of one another they can be combined (or ―composed‖) and reused with maximum flexibility. As business processes evolve, business rules and practices can be adjusted without constraint from the limitations of the underlying applications. Services compositions enable new cross-functional processes to emerge, allowing the enterprise to adopt new business processes, tune processes for greater efficiency, or improve service levels for customers and partners. A Service Integration
Architecture describes a set of capabilities for composing services and other components into larger constructs such as business processes. Composing services requires some sort of workflow or orchestration mechanism. Microsoft provides these capabilities via BizTalk Server 2006 (BTS) or Windows Workflow Foundation (WF). While BTS and WF may appear to serve similar needs, they are actually quite different. WF and BTS are complementary technologies designed to serve two very different needs: BTS is a licensed product designed to implement workflow (―orchestrations‖) across disparate applications and platforms. WF is a developer framework designed to expose workflow capabilities within your application. There are no fees or licensing restrictions associated with using or deploying WF. We will examine workflow, orchestration and the use of BizTalk/WF in Chapter Three (Workflow and Business Processes). Consume When a new application or business process has been created that functionality must be made available for access (consumption) by IT systems, other services or by end-users. Consumption focuses on delivering new applications that enable increased productivity and enhanced insight into business performance. Users may consume ―composed‖ services through a broad number of outlets including web portals, rich clients, Office business applications (OBA), and mobile devices. ―Composed‖ services can be used to rapidly roll out applications that result in new business capabilities or improved productivity. These application roll-outs can be used to measure the return on investment (ROI) in an SOA. A Service Oriented Application Architecture describes how ―composed services‖ are made available for consumption through as business processes, new services or new end-user applications. This concept is sometimes referred to as Composite Applications since it implies service consumption by end-user applications. Microsoft‘s Office Business Applications (OBAs) support the Composite Application notion of transactional systems while expanding the scope of user interaction via the familiar Office environment. We will examine consumption in greater detail in Chapter Five (User Experience). While the architectures described within Expose / Compose / Consume may be interdependent, they are designed to be loosely coupled. This enables services to be managed, versioned and configured independently of how they are exposed,
Recurring Architectural Capabilities As we saw earlier, the SOA architectural model is fractal. This means that a service can be used to Expose IT assets (such as a Line of Business system), be Composed into workflows or Business Processes (each of which may also be exposed as a service) and be Consumed by end users, systems or other services. SOA is a fractal, not layered model. While the Abstract SOA Reference Model provides a holistic view of several important SOA concepts, the Expose / Compose / Consume portions of the model should not be interpreted as layers (despite their apparent appearance in the model). Designing a SOA as a set of well-defined tiers (or layers) will constrain the value and flexibility of your services, resulting in dependencies across unrelated components. This is why the Expose / Compose / Consume portions of the model can be thought of as independent architectural initiatives referred to as a Service Implementation Architecture (Expose), a Service Integration Architecture (Compose) and an Application Architecture (Consume). While each of these architectures are designed to be independent of one another, they share a set of five common capabilities Figure 7: Recurring Architectural Capabilities Messaging and Services Messaging and Services focuses on how messaging is accomplished between senders and receivers. There are a broad array of options and patterns available – from pub/sub and asynchronous to message and service interaction patterns. Services provide an evolutionary approach to building distributed software that facilitates loosely coupled integration and resilience to change. The advent of the WS-* Web Services architecture has made service-oriented software development feasible by virtue of mainstream development tools support and broad industry interoperability. While most frequently implemented using industry standard Web services, service orientation is independent of technology and architectural patterns and can be used to connect with legacy systems as well. Messaging and Services are not a new approach to software design – many of the notions behind these concepts have been around for years. A
service is generally implemented as a coarse-grained, discoverable software entity that exists as a single instance and interacts with applications and other services through a loosely coupled (often asynchronous), message-based communication model. Messages tend to be based upon an agreed-upon set of semantics (such as an industry-standard Purchase Order) and serialized using an interoperable, extensible syntax (usually XML, although alternate approaches like JSON, RNC and ASN1 are sometimes used). Workflow and Process Workflow and Process are pervasive across multiple layers of an integration architecture – from formal orchestration of processes to flexible ad hoc systems and collaborative workflow across teams. Since business processes are dynamic and evolve with the organization, the workflow that models these processes must be equally adaptable. In addition, effective workflow involves not only process modeling, but also monitoring and analytics in order to respond to exceptions and optimize the workflow system over time. Data The lynchpin to success in many integration architectures is the ability to provide Data management. The need to deliver a shared view across disparate, often duplicate sources of data is more important than ever, as businesses strive to achieve a 360-degree view of organizational information. Entity aggregation, master data management, and the ability to make data useful via analytics and mining are crucial elements of integration architecture. User Experience Successful integration architectures depend upon both service delivery and the ability to consume services in a rich and meaningful way. Service consumption needs to be contextual, mapping to the natural workflow of employees, customers, and partners. To that end, an integrated User Experience spanning smart clients, rich clients, lightweight Web applications, and mobile devices enables service consumption by the broadest possible audience. Identity and Access To support integrated user experiences, customers require the ability to manage the identity lifecycle – providing integrated Single Sign-On (SSO), access management, directory services, and federated trust across heterogeneous systems. Today, many solutions are built using fragmented technologies for authentication and authorization. In the new application model, access decisions and entitlements need to be made at multiple layers and tiers, in which a federated Identity and Access across trust boundaries becomes a key requirement.
Management During the lifetime of a service, the service most probably changes in different perspectives as listed below. As a result, one service will probably have to be available in several versions. Difference in interface (e.g. extended interface, but same business object) Difference in semantics with same interface (business objects changed) Difference in QoS, e.g. slower but cheaper or high-available but more expensive Service management encompasses many capabilities, some of which are listed below: A comprehensive solution for change and configuration management, enabling organizations to provide relevant software and service updates to users quickly and cost- effectively. Reducing the complexity associated with managing the IT infrastructure environment with a focus on lowering the cost of operations. Centralized backup services capturing changed files to disk. Centralized backup should enable rapid, reliable recovery from disk while providing end-user recovery without IT intervention. Pre-deployment capacity planning coupled with best-practice guidance and hardware- specific knowledge to help information technology professional make low-risk architectural decisions. Data warehouse and reporting to help IT better support corporate decision making, improve the quality of service provided, and better administer resources through improved reporting capabilities and management data integration from a broad variety of resources. Supporting the Common Architectural Capabilities The five architectural capabilities discussed above are supported by the Microsoft SOA Platform. The remainder of this book discusses the common architectural capabilities in greater detail, starting with Messaging and Services in Chapter Two.
Figure 8: SOA Capabilities on the Microsoft Platform Common Architectural Capabilities and the Abstract SOA Model We can also think of these five common architectural capabilities as set of perspectives for viewing and understanding the Abstract SOA Model. The five architectural capabilities serves as a set of lenses to help us view and better understand the challenges associated with Exposing existing IT investments as services, Composing the services into business processes and Consuming these processes across the organization. Expose Service Enablement Expose focuses on how we design and expose our services. We will most likely start out by enabling our IT investments to be exposed as web services. As our organization matures we will start adding new services, most likely as proxies for other resources within the organization. One of the hardest parts of service implementation is deciding where to begin. The are a variety of choices here and there is no single recommendation that will work for everyone. Motion is a methodology that provides some guidance for identifying business capabilities that could be exposes as services. What are some best practices that one should follow when exposing IT investments as services? • Think big– but start small • Show value at every step along the way – not build it and they will come • Middle-out – not Top-down or bottom-up
• Be pragmatic • Vertical-slice – not build it and they will come • Risk-mitigation • Demonstrate value in rapid iterations – not waterfall • New approaches to development • Successful customers ‗snowball‘ The recurring architectural capabilities provide us with a set of considerations when exposing IT investments as services. Let‘s take a quick look at some of the considerations associated with each capability for service exposure (this is by no means a complete list): Messaging and Services Determining what to expose and how - Avoid falling into the granularity trap – focus on meeting your business requirements Service Operation Contracts Message and Data Contracts Configuration, behaviors and control SLAs Governance Versioning Workflow and Process Coordinator services for distributed, long-running processes Tracking services capable of logging specific events within a workflow Compensation services Data Entity services Entity Aggregation services: acts as a single point to access information that may exist in multiple systems. An Entity Aggregation service has the following responsibilities: Acts as a unified source of entities. Provides a holistic view of an entity. Provides a holistic view of the entity model—entities and their relationships with other entities Provides location transparency—consumers of the Entity Aggregation layer do not need to know who owns the information.
Enforces business rules that determine the segments of entities retrieved in a given context. Determines the system of record for each data element constituting an entity. Enriches the combined data model across systems—the whole being better than the sum of its parts. Entity factoring MDM focuses on exposing data across corporate or departmental boundaries. We‘ll discuss MDM in greater detail in Chapter Four. User Experience Specialized services for supporting user interfaces (caching resources, communications between UI and services, etc). Service wrappers provide coarse-grained interfaces for user app consumption, lightweight mash-ups, etc. Identity and Access Identity Management Impersonation and Delegation services Trusted Subsystem - A trusted subsystem model implies that services are trusted to perform specific tasks, such as processing customer orders. Authentication (Kerberos, SSL) Role-based access control (RBAC) Create/revoke trust relationships Services need to make authorization decisions, such as approving an order submission before performing the business transaction. The service must know the identity of the end user submitting the order. Need to flow the identity of the end user is an inherent property of the delegation model, it is not so for the trusted subsystem model and special efforts must be made to include this feature. To support the notion of trust as defined by the model, the services must at least be able to: 1. Authenticate / verify identity of upstream / downstream services 2. Decide if the service is a trusted subsystem for specific functions (including propagating identity claims) 3. Protect the integrity of the data being communicated between trusted subsystem and services.
Besides application data, application plumbing data, such as the identity claims of the original user, must also be protected so that no man-in-the-middle can modify the identity information that is in transit. Compose Service Composition Compose focuses on how we can combine or aggregate granular services into more complex processes. We will most likely start by using services that expose our existing IT investments. Service composition results in a new service instance that the rest of the organization can make use of. The composition provides capabilities such as correlated asynchronous service invocation, long running processes and other capabilities for orchestrating autonomous services. The recurring architectural capabilities provide us with a set of considerations when composing granular services into complex processes. Let‘s take a quick look at some of the considerations associated with each capability for service composition (this is by no means a complete list): Messaging and Services Service interaction patterns Exposing orchestrations as services Asynchronous service invocation patterns Workflow and Process Transactions High frequency of change Business Rules Service Orchestration Service Interaction Patterns (SIPs) Process Externalization Long Running Processes Auditing and analytics Data Tracking the state of a given workflow instance Data transformation (ETL) Reliable message processing and storage Replication
Synchronization Metadata repository and Management Instance reconciliation Schema reconciliation Document Replication Syndication/Aggregation User Experience Composite applications (OBAs) Human workflows (MOSS) Orchestrations initiate human workflows via SharePoint adapter Pageflows Identity and Access Impersonation and Delegation Provisioning Identity Repository synchronization Approval workflows Consume User Experience Consume focuses on how services and orchestrated processes (which may be exposed as services) are consumed by other services, applications and end-users. Any resource capable of interacting with services can be referred to as a ―consumer‖. Consumers may appear across the organization in several possible forms: Lightweight, browser-based applications Rich internet applications (RIA) are browser-based applications that can address and cache local and remote resources Configurable, portal-based user experiences Applications that are installed on the local machine (such as a custom Windows application) Corporate business applications with solution-specific extensions (such as Microsoft Office with context-aware activity panes) Applications designed for mobile devices Services may act as consumers of other services
Recall that the SOA model is fractal – services may be consumed by other services and service compositions may be exposed as new services. In the last couple of years a ―new breed‖ of consumers has emerged, enabling consumers to be aggregated and consumed by additional consumers. This ―new breed‖ of consumers is usually referred to as a ―mashup‖. A mashup is a set of services, websites or applications that combine content from multiple resources into a new integrated user experience. Content used by mashups is typically sourced from a third party (such as a service or website) via a public interface or API. Alternative methods of sourcing content for mashups include newsfeeds and JavaScript constructs (JSON). The recurring architectural capabilities provide us with a set of considerations for User Experience. Let‘s take a quick look at some of the considerations associated with each capability for User Experience (this is by no means a complete list): Messaging and Services Forms-based service consumption Web parts Service Registry – check in / check out / search AJAX, REST Workflow and Process Human workflows (MOSS) Event brokering (CAB) Page flows Data Entities (OBA Business Data Catalog) Single view of the customer problem JSON User Experience Composite applications (OBAs) Personalization, user profiles Portals Business Intelligence Reporting Content aggregation Declarative UX
Identity and Access Single Sign-On (password synchronization) User identification Role-based access (RBAC) Directory Services Password management Privacy (firewalls, encryption) Compliance Summary In this chapter we provided some useful analogies for understanding the fractal nature of SOA. Services are the fundamental building blocks of SOA, although services do not necessarily need to be web services. Ideally these services should follow the four service design tenets which describe a set of best practices for service scopes, dependencies, communications and policy- based configuration. While these tenets focus upon service design, it is important to realize that services alone are not necessarily solution architecture – Microsoft uses an abstract reference model to describe the various aspects of SOA. The abstract SOA reference model provides three fundamental concepts to help most organizations understand the role that services can play within their solution architectures: Expose focuses on how existing IT investments are exposed as a set of broad, standards-based services, enabling these investments to be available to a broader set of consumers. A Service Implementation Architecture describes how services are developed, deployed and managed. Compose focuses on combining services into applications or cross-functional business processes. A Service Integration Architecture describes a set of capabilities for composing services and other components into larger constructs such as business processes. Consume focuses on delivering new applications that enable increased productivity and enhanced insight into business performance. A Service Oriented Application Architecture describes how ―composed services‖ are made available for consumption through as business processes, new services or new end-user applications. Each aspect of the Expose / Compose / Consume abstract reference model encompasses a set of five recurring architectural capabilities: Messaging and Services, Workflow and Processes, Data, User Experience and Identity and Access. The five architectural capabilities serves as a
set of views to better understand the challenges associated with Exposing existing IT investments as services, Composing the services into business processes and Consuming these processes across the organization. In the Chapter Two we will enter into a more detailed analysis of both Service Orientation and the Messaging and Services architectural capability.
References: 1. ―Enabling ―Real World‖ SOA through the Microsoft Platform‖, A Microsoft White Paper, December 2006. Available at http://www.microsoft.com/biztalk/solutions/soa/whitepaper.mspx 2. Service Oriented Integration Pattern: http://msdn2.microsoft.com/en- us/library/ms978594.aspx 3. ―Business Component Factory‖, Peter Herzum and Oliver Sims, Wiley, 1999
Chapter 2: Messaging and Services “SOA is not something you buy, it's something you do.” – Jason Bloomberg Analyst Reader ROI Readers of this chapter will build upon the concepts introduced in Chapter One, specifically focusing on the Messaging and Services architectural capability. Figure 1: Recurring Architectural Capabilities The Messaging and Services architectural capability focuses on the concept of service orientation and how different types of services are used to implement SOA. This chapter also touches upon the role of the user – specifically how a user will interact with services within a SOA. Topics discussed in this chapter include: A maturity model for SOA A services lifecycle Sample SOA scenarios The role of the user within a SOA The concepts covered in this chapter are not necessarily new. SOA Maturity Models and Service Lifecycles have been published by a broad variety of vendors, consultants and industry analysts. Like SOA itself, there is no single Maturity Model or Service Lifecycle that everyone
agrees upon. Readers should review several of these efforts and draw from them the aspects that best fit your organizational needs. Acknowledgements This chapter consists of work from the following individuals: Mark Baciak (Service Lifecycle), Atanu Bannerjee (OBA), Shy Cohen (Service Taxonomy), William Oellermann (Enterprise SOA Maturity Model), and Brenton Webster (SOA Case Study).
Understanding Services A SOA Maturity Model (another one??) There is an abundance of SOA Maturity Models available from vendors, consulting firms, analysts and book authors. Most of these Maturity Models are either based upon or inspired by the Software Engineering Institute‘s (now retired) Capability Maturity Model (CMM). A recent search on the terms ―SOA Maturity Model‖ returned almost 10,000 relevant hits (including articles on what to look for in a Maturity Model). Given the intense interest and variety of maturity models available why introduce another? Unlike other Maturity Models, the one discussed here doesn‘t attempt to simply apply service orientation to the CMM. ESOMM (Enterprise Service Orientation Maturity Model) borrows CMM‘s notion of capability-driven maturity models and applies these principles to service-orientation paradigms, essentially building a road map from scratch. Unlike CMMI, the ESOMM Maturity Model doesn‘t focus on processes because the focus is on IT capabilities, not organizational readiness or adoption. While there are some conceptual similarities with CMM, ESOMM is a decidedly different application of the maturity model concept. The layers, perspectives, and capabilities defined in the ESOMM are designed as a road map to support services—not any specific service with any specific use, implementation, or application, but any service, or more specifically, any set of services. Developing an enterprise-wide SOA strategy is not a trivial undertaking and should not be treated as a short-term or one-time effort. SOA is an attempt to enable a higher level of agility for the entire organization to facilitate an expedient response to the needs of the business and customers. There are many aspects of SOA that will be much more critical in the near- instead of long-term, so it is important to align your group‘s efforts accordingly. To do so successfully, the development of a prioritized roadmap should be a high priority to plan for success. An ESOMM can provide a capability-based, technology-agnostic model to help identify an organization‘s current level of maturity, the short- and long-term objectives, and the opportunity areas for improvement. Figure 2 provides an overview of ESOMM. ESOMM defines 4 horizontal maturity layers, 3 vertical perspectives, and 27 capabilities necessary for supporting SOA.
Figure 2: An Enterprise Service Orientation Maturity Model (ESOMM) Most organizations will not naturally completely implement all of one layer‘s capabilities before moving up the model, but jumping too far ahead can be risky. This becomes very apparent as you recognize that some poor decisions concerning key building block capabilities can severely impact your ability to mature at higher layers. Capabilities are categorized into one of three perspectives: Implementation, Consumption, and Administration. Implementation capabilities target the development and deployment of Web services from the provider‘s perspective. Consumption capabilities are those that cater to the consumers of your services, making them easier to implement and therefore more widely and successfully adopted. Administration capabilities include those that facilitate the operational and governance aspects of Web services across the organization. While an organization can choose to focus on a single perspective, the overall maturity, and hence value, of your SOA depends on the appropriate level of attention to all three. ESOMM is a tool that can be used to: o simplify the complexity of a massively complex and distributed solution. o Identify and discuss an organization‘s adoption of service orientation. o understand the natural evolution of service adoption (such as skipping certain capabilities at lower levels comes with certain risk). o provide a cohesive and comprehensive service orientation plan for customers based on their objectives.
o align an organization‘s activities and priorities with a distinct level of value providing specific benefits through specific capabilities, In a SOA assessment, each capability is assessed independently as a level of strength, adequacy, or weakness. A level of strength demonstrates that the organization is well positioned to safely grow their use of Web services without fearing pitfalls in this area down the road. A level of adequacy signals that an organization has sufficient capabilities to meet today‘s needs, but is vulnerable to a growth in Web services that could cause problems in that area. A weakness represents a deficiency that is a problem for today‘s use of Web services and could be a serious detriment in the future. Any capabilities that are obviously not part of the organization‘s objectives and appear to have minimal impact in the near term for the organization are classified as not applicable. These areas may quickly become weaknesses if business or technical objectives were to change or accelerate. As in the CMM, individual capability levels drive an overall layer grade ranging from 1 to 5. A grade is assessed based on the average of the capability maturities within the layer. A rating of 5 represents a mastery of that level with little or no room for improvement. A 4 represents a solid foundation within an area that can be successfully built upon. A 3 is assigned when good efforts are established in that layer, but extending efforts to the next layer carries some risks. 2 means that only initial efforts have been made in the area and much more work is needed to extend to the next layer. A grade of 1 represents no effort to address the capabilities at that layer. Now that we've had a chance to briefly review the components of ESOMM, you are hopefully already thinking about how it can be applied to your organization. To be clear, applying this model should not be looked at as a one-time activity or short-term process. Instead, the model is best leveraged as a working plan that can be modified over time as the usage of services and your experience grows. Unfortunately, the downside of using the term maturity with a model is that people will immediately want to know what layer their organization is at to get a sense of their status or identity. As it happens, there is no appropriate way to answer the question, "what layer is my organization?" Instead of giving an overall grade based on one of the layers, we take the approach of giving each layer its own level of maturity, ranging from one to five, based on half- point increments.
ESOMM is intended to be leveraged as a road map, more so than as an SOA ―readiness‖ or assessment tool. While it is important to know where you are, getting an exact bearing is less important than identifying the capabilities you need to address to continue advancing the value of service enablement in your organization. As long as you are willing to ask yourself some hard questions in an objective manner across all the relevant groups, you should be able to get a good understanding for your current challenges. If you apply the strategy and objectives of your organization, you should be able to identify which capabilities you will need to address in the near, short, and long term. ESOMM is one of many possible maturity models that can be used to assess the capabilities of the enterprise in adopting and implementing SOA. It is terribly difficult to have a concise, constructive conversation about a service-enabled enterprise without some common agreement on capabilities and maturity – ESOMM is one of many maturity models that attempt to address this issue. Unlike other maturity models, however, ESOMM can also be leveraged as a prioritized roadmap to help enterprises identify the capabilities necessary for a successful implementation. The goal of ESOMM and other maturity models is to empower your organization with the tools and information needed for a successful SOA implementation. In a way, SOA does render the infrastructure more complex because new capabilities will need to be developed that did not exist before, such as registry and repository. In some sense, it can be compared to the construction of a highway network—broad, safe roads are more expensive to build, and the users need to upgrade their means of transportation to make the most of that infrastructure, but the cost per trip (in time and safety) is driven down. Until the network reaches a critical mass, drivers still need to be prepared to go ―off road‖ to reach their destination. Many enterprise applications were not developed with SOA in mind, and are either incapable of leveraging an SOA infrastructure, or will need to be upgraded to take advantage of it. However, with more/new applications being created consistently, there is a great opportunity to drive down new interoperability costs as the various technology vendors enable their products for SOA. The biggest challenge for SOA is convincing the application owners to invest more today to achieve those promised long term savings. While maturity models like ESOMM can help clarify the capabilities necessary for SOA, the types of services your organization will need usually remains unanswered. A simple service taxonomy can assist you in better understanding the breadth of services that will typically exist within a SOA.
A Service Taxonomy As we examine service types we notice two main types of services: those who are infrastructural in nature and provide common facilities that would not be considered part of the application, and those who are part of the application and provide the application‘s building blocks. Software applications utilize a variety of common facilities ranging from the low-level services offered by the Operating System such as the memory management and I/O handling, to the high- level runtime-environment-specific facilities such as the C Runtime Library (RTL), the Java Platform, or the .NET Framework. Solutions built using a SOA make use of common facilities as well, such as a service-authoring framework (for example, Windows Communication Foundation) and a set of Services that are part of the supporting distributed computing infrastructure. We will name this set of services Bus Services. Bus Services further divide into Communication Services which provide message transfer facilities such as message-routing and publish-subscribe mechanisms, and Utility Services which provide capabilities unrelated to message transfer such as service-discovery and federated security. The efficiency of software applications development is further increased through reuse of coarse grained, high-level building blocks. The RAD programming environments that sprang up in the Component Oriented era (such as Delphi or Visual Basic) provided the ability to quickly and easily compose the functionality and capabilities provided by existing building blocks with application specific code to create new applications. Examples of such components range from the more generic GUI constructs and database access abstractions, to more specific facilities such as charting or event-logging. Composite applications in a SOA also use building blocks of this nature in their composition model. We will name these building blocks Application Services. Application services further divide into Entity Services which expose and allow the manipulation of business entities, Capability Services and Activity Services which implement the functional building blocks of the application (sometimes referred to as components or modules), and Process Services which compose and orchestrate Capability and Activity Services to implement business processes.
The following diagram shows a possible composition of services in the abovementioned service categories. Utility Process Services Activity Capability Discover y Activity STS Activity Capability Capability … Entity Entity Entity Bus Services Routing Publish- Queuing … Subscribe Figure 3: Bus Services Bus Services are common facilities that do not add any explicit business value, but rather are a required infrastructure for the implementation of any business process in a SOA. Bus Services are typically purchased or centrally built components that serve multiple applications and are thus typically centrally managed. Communication Services Communication Services transport messages into, out of, and within the system without being concerned with the content of the messages. For example, a Bridge may move messages back and forth across a network barrier (i.e. bridging two otherwise-disconnected networks) or across a protocol barrier (e.g. moving queued messages between WebSphere MQ and MSMQ). Examples of Communication Services include relays, publish-subscribe systems, routers, queues, and gateways. Communication Services to not hold any application state, but in many cases they are configured to work in concert with the applications that use them. A particular application may need to instruct or configure a Communication Service on how to move the messages flowing inside that application such that inter-component communication is made possible in a loosely coupled architecture. For example, a content-based router may require the application to provide routing
instructions such that the router will know where to forward messages to. Another example may be a publish-subscribe server which will deliver messages to registered subscribers based on a filter that can be applied to the message‘s content. This filter will be set by the application. In both cases the Communication Service does not process the content of the message but rather (optionally) uses parts of it as instructed by the application in advance for determining where it should go. In addition to application-specific requirements, restrictions imposed by security, regulatory, or other sources of constraints may dictate that in order to use the facilities offered by a particular Communication Service users will need to possess certain permissions. These permissions can be set at the application scope (i.e. allowing an application to use the service regardless of the specific user who is using the application), at the user scope (i.e. allowing a specific user to use the service regardless of the application that the user is using), or at both scopes (i.e. allowing the specific user to access the service while running a specific application). For example, a publish- subscribe service may be configured to restrict access to specific topics by only allowing specific users to subscribe to them. Other application-level facilities that may be offered by Communication Services pertain to monitoring, diagnostics, and business activity monitoring (BAM). Communication Services may provide statistical information about the application such as an analysis of message traffic patterns (e.g. how many messages are flowing through a bridge per second), error rate reports (e.g. how many SOAP faults are being sent through a router per day), or business-level performance indicators (e.g. how many purchase orders are coming in through a partner‘s gateway). Although they may be specific to a particular application, these capabilities are not different than the configuration settings used to control message flow. This information is typically provided by a generic feature of the Communication Service, which oftentimes needs to be configured by the application. The statistical information being provided typically needs to be consumed by a specific part of the application that knows what to do with it (e.g. raise a security alert at the data center, or update a BAM-related chart on the CFO‘s computer screen). Utility Services Utility Services provide generic, application-agnostic services that deal with aspects other than transporting application messages. Like Communication Services, the functionality they offer is part of the base infrastructure of a SOA and is unrelated to any application-specific logic or business process. For example, a Discovery service may be used by components in a loosely coupled composite-application to discover other components of the application based on some specified criteria (e.g. a service being deployed into a pre-production environment may look for another service which implements a certain interface that the first component needs and that is also deployed in the pre-production environment). Examples of Utility Services include security
and identity services (e.g. an Identity Federation Service or a Security Token Service), discovery services (e.g. a UDDI server), and message transformation services. As in the case of Communication Services, Utility Services may too be instructed or configured by a particular application on how to perform an operation on their behalf. For example, a Message Transformation service may transform messages from one message schema to another message schema based on a transformation mapping that is provided by the application using the Message Transformation service. Although Utility Services do not hold any application state, the state of a Utility Service may be affected by system state changes. For example, a new user being added to the application may require an update to the credential settings in the Security Token Service. Unlike in the case of Communication Services, client services directly interact with the Utility Services who process and (if needed) respond to the messages that the clients send to them. Users of Utility Services may require a permission to be configured for them in order to use the service, be it at the application, user, or the application-user scope. For example, a Discovery service may only serve domain-authenticated users (i.e. users who have valid credentials issues by a Windows domain controller). Like Communication Services, Utility Services may provide application-level facilities for monitoring, diagnostics, BAM, etc. These may include statistical information about usage patterns (e.g. how many users from another organization authenticated using a federated identity), business-impacting error rates (e.g. how many message format transformations of purchase orders failed due to badly formatted incoming messages), etc. As with Communication Services, these facilities are typically generic features of the Utility Service and they need to be configured and consumed by the particular solution in which they are utilized. Application Services Application Services are services which take part in the implementation of a business process. They provide an explicit business value, and exist on a spectrum which starts with generic services that are used in any composite-application in the organization on one end, ends with specialized services that are part of a single composite-application on the other end, and has services that may be used by two or more applications in between. Entity Services Entity Services unlock and surface the business entities in the system. They can be thought of as the data-centric components ("nouns") of the business process: employee, customer, sales-order, etc. Examples of Entity Services include services like a Customers Service that manages the
customers‘ information, an Orders Service that tracks and manages the orders that customers placed, etc. Entity Services abstract data stores (e.g. SQL Server, Active Directory, etc.) and expose the information stored in one or more data stores in the system through a service interface. Therefore, it is fair to say that Entity Services manage the persistent state of the system. In some cases, the information being managed transcends a specific system and is used in several or even all the systems in the organization. It is very common for Entity Services to support a CRUD interface at the entity level, and add additional domain-specific operations needed to address the problem-domain and support the application‘s features and use-cases. An example for a domain-specific operation is a Customers service that exposes a method called FindCustomerByLocation which can locate a customer ID given the customer‘s address. The information that Entity Services manage typically exists for a time span that is longer than that of a single business process. The information that Entity Services expose is typically structured, as opposed to the relational or hierarchical data stores which are being fronted by the service. For example, a service may aggregate the information stored in several database tables or even several separate databases and project that information as a single customer entity. In some cases, typically for convenience reasons, Entity Service implementers choose to expose the underlying data as DataSets rather than strongly-schematized XML data. Even though DataSets are not entities in the strict sense, those services are still considered Entity Services for classification purposes. Users of Entity Services may require a permission to be configured for them in order to use the service, be it at the application, user, or the application-user scope. These permissions may apply restrictions on data access and/or changes at the ―row‖ (entity) or ―column‖ (entity element) level. An example for ―column‖ level restriction would be an HR application might have access to both the social security and home address elements of the employee entity while a check-printing service may only have access to the home address element. An example for ―row‖ level restriction would be an expense report application which lets managers see and approve expense reports for employees that report to them, but not for employees who do not report to them. Error compensation in Entity Services is mostly limited to seeking alternative data sources, if at all. For example, if an Entity Service fails to access a local database it may try to reach out to a remote copy of the database to obtain the information needed. To support system-state consistency, Entity Services typically support tightly-coupled distributed atomic transactions. Services that support distributed atomic transactions participate in transactions that are flowed to them by callers and subject any state changes in the underlying data store to the outcome of these distributed atomic transactions. To allow for a lower degree of state-change coupling, Entity
Services may provide support for the more loosely-coupled reservation pattern, either in addition to or instead of supporting distributed atomic transactions. Entity Services are often built in-house as a wrapper over an existing database. These services are typically implemented by writing code to map database records to entities and exposing them on a service interface, or by using a software factory to generate that mapping code and service interface. The Web Services Software Factory from Microsoft‘s Patterns & Practices group is an example of such a software factory. In some cases, the database (e.g. SQL Server) or data- centric application (e.g. SAP) will natively provide facilities that enable access to the data through a service interface, eliminating the need to generate and maintain a separate Entity Service. Entity Services are often used in more than one composite-application and thus they are typically centrally managed. Capability Services Capability Services implement the business-level capabilities of the organization, and represent the action-centric building blocks (or "atomic verbs") which make up the organization‘s business processes. A few examples of Capability Services include third-party interfacing services such as a Credit Card Processing service that can be used for communication with an external payment gateway in any composite-application where payments are made by credit card, a value-add building block like a Rating Service that can process and calculate user ratings for anything that can be rated in any application that utilizes ratings (e.g. usefulness of a help page, a book, a vendor, etc.), or a communication service like an Email Gateway Service that can be used in any composite-application that requires the sending of emails to customers or employees. Capability Services can be further divided by the type of service that they provide (e.g. third-party interfacing, value-add building block, or communication service), but this further distinction is out of scope for this discussion. Capability Services expose a service interface specific to the capability they represent. In some cases, an existing (legacy) or newly acquired business capability may not comply with the organization‘s way of exposing capabilities as services, or even may not expose a service interface at all. In these cases the capability is typically wrapped with a thin service layer that exposes the capability‘s API as a service interface that adheres to the organization‘s way of exposing capabilities. For example, some credit card processing service companies present an HTML-based API that requires the user to fill a web-based form. A capability like that would be wrapped by an in-house-created-and-managed-façade-service that will provide easy programmatic access to the capability. The façade service is opaque, and masks the actual nature of the capability that‘s behind it to the point where the underlying capability can be replaced without changing the service interface used to access it. Therefore, the façade service is
considered to be the Capability Service, and the underlying capability becomes merely an implementation detail of the façade service. Capability Services do not typically directly manage application state; to make state changes in the application they utilize Entity Services. If a Capability Service does manage state, that state is typically transient and lasts for a duration of time that is shorter than the time needed to complete the business process that this Capability Service partakes in. For example, a Capability Service that provides package shipping price quotes might record the fact that requests for quotes were sent to the shipping providers out until the responses come back, thereafter erasing that record. In addition, a Capability Service that is implemented as a workflow will manage the durable, transient execution state for all the currently running instances of that workflow. While most of the capabilities are ―stateless‖, there are obviously capabilities such as event logging that naturally manage and encapsulate state. Users of Capability Services may require a permission to be configured for them in order to use the service, be it at the application, user, or the application-user scope. Access to a Capability Service is typically granted at the application level. Per-user permissions are typically managed by the Process Services that make use of the Capability Services to simplify access management and prevent mid-process access failures. Error compensation in Capability Services is limited to the scope of meeting the capability‘s Service Level Expectation (SLE) and Service Level Agreements (SLA). For example, the Email Gateway Service may silently queue up an email notification for deferred delivery if there‘s a problem with the mail service, and send it at a later time, when email connectivity is restored. A Shipping Service which usually compares the rates and delivery times of 4 vendors (e.g. FedEx, UPS, DHL, and a local in-town currier service) may compensate for a vendor‘s unavailability by ignoring the failure and continuing with the comparison of the rates that it was able to secure as long as it received at least 2 quotes. These examples come to illustrate that failures may result in lower performance. This degradation can be expressed in terms of latency (as in the case of the Customer Emailing Service), the quality of the service (e.g. the Shipping Service would only be comparing the best of 2 quotes instead of 4), and many other aspects, and therefore needs to be described in the SLE and SLA for the service. Capability Services may support distributed atomic transactions and/or the reservation pattern. Most of the Capability Services do not manage resources whose state needs to be managed using atomic transactions, but a Capability Service may flow an atomic transaction that it is included in to the Entity Services that it uses. Capability Services are also used to implement a reservation pattern over Entity Services that do not support that pattern, and to a much lesser extent over other Capability Services that do not support that pattern.
Capability Services can be developed and managed in-house, purchased from a third party and managed in-house, or ―leased‖ from an external vendor and consumed as SaaS that is externally developed, maintained, and managed. When developed in-house, Capability Services may be implemented using imperative code or a declarative workflow. If implemented as a workflow, a Capability Service is typically modeled as a short-running (atomic, non-episodic) business-activity. Long running business-activities, where things may fail or require compensation typically fall into the Process Service category. A Capability Service is almost always used by multiple composite-applications, and is thus typically centrally managed. Activity Services Activity Services implement the business-level capabilities or some other action-centric business logic elements (―building blocks‖) that are unique to a particular application. The main difference between Activity Services and Capability Services is the scope in which they are used. While Capability Services are an organizational resource, Activity Services are used in a much smaller scope, such as a single composite-application or a single solution (comprising of several applications). Over the course of time and with enough reuse across the organization, an Activity Service may evolve into a Capability Service. Activity Services are typically created to facilitate the decomposition of a complicated process or to enable reuse of a particular unit-of-functionality in several places in a particular Process Service or even across different Process Services in the application. The forces driving for the creation of Activity Services can stem from a variety of sources, such as organizational forces, security requirements, regulatory requirements, etc. An example of an Activity Service create in a decomposition scenario is a Vacation Eligibility Confirmation Service that due to security requirements separates a particular part of a vacation authorization application‘s behavior such that that part could run behind the safety of the HR department‘s firewall and access the HR department‘s protected databases to validate vacation eligibility. An example of an Activity Service used for sharing functionality would be a Blacklist Service that provides information on a customer‘s blacklist status such that this information can be used by several Process Services within a solution. Like Capability Services, Activity Services expose a service interface specific to the capability they represent. It is possible for an Activity Services to wrap an existing unit of functionality, especially in transition cases where an existing system with existing implemented functionality is being updated to or included in a SOA-based solution. Like Capability Services, Activity Services do not typically directly manage application state, and if they do manage state that state is transient and exists for a period of time that is shorter than the
lifespan of the business process that the service partakes in. However, due to their slightly larger granularity and the cases where Activity Services are used to wrap an existing system, it is more likely than an Activity Services will manage and encapsulate application state. Users of Activity Services may require a permission to be configured for them in order to use the service, be it at the application, user, or the application-user scope. Like in the case of Capability Services, access to an Activity Service is typically granted at the application level and managed for each user by the Process Services that are using the Activity Service. Activity Services have the same characteristics for error compensation and transaction use as Capability Services. Activity Services are typically developed and managed in-house, and may be implemented as imperative code or a declarative workflow. Like in the case of a Capability Service, if implemented as a workflow an Activity Service is typically modeled as a short-running business-activity. Activity Services are typically used by a single application or solution and are therefore typically managed individually (for example, at a departmental level). If an Activity Service evolves into a Capability Service, the management of the service is typically transitions to a central management facility. Process Services Process Services tie together the data-centric and action-centric building blocks to implement the business processes of the organization. They compose the functionality offered by Activity Services, Capability Services, and Entity Services and tie them together with business logic that lives inside the Process Service to create the blueprint that defines the operation of the business. An example of a Process Service is a Purchase Order Processing service that receives a purchase order, verifies it, checks the Customer Blacklist Service to make sure that the customer is OK to work with, checks the customer‘s credit with the Credit Verification Service, adds the order to the order-list managed by the Orders (Entity) Service, reserves the goods from the Inventory (Entity) Service, secures the payment via the Payment Processing Service, confirms the reservation made with the Inventory (Entity) Service, schedules the shipment with the Shipping Service, notifies the customer of the successful completion of the order and the ETA of the goods via the Email Gateway Service, and finally marks the order as completed in the order- list. Process Services may be composed into the workflows of other Process Services but will not be re-categorized as Capability or Activity Services due to their long-running nature. Since Process Services implement the business processes of the organization, they are often fronted with a user interface that initiates, controls, and monitors the process. The service interface that these services expose is typically geared towards consumption by an end user
application, and provides the right level of granularity required to satisfy the use cases that the user facing front-end implements. Monitoring the business process will at times require a separate monitoring interface that exposes BAM information. For example, the Order Processing Service may report the number of pending, in-process, and completed orders, and some statistical information about them (median time spent processing and order, average order size, etc.). Process Services typically manage the application state related to a particular process for the duration of that process. For example, the Purchase Order Processing service will manage the state of the order until it completes. In addition, a Process Service will maintain and track the current step in the business process. For example, a Process Service implemented as a workflow will hold the execution state for all the currently running workflow instances. Users of Process Services may require a permission to be configured for them in order to use the service, be it at the application, user, or the application-user scope. Access to a Process Service is typically granted at the user level. Process Services very rarely support participating in a distributed atomic transaction since they provide support for long-running business activities (a.k.a. long-running transactions) where error compensation happens at the business logic level and compensation may involve human workflows. Process Services may utilize distributed atomic transactions when calling into the services they use. Process Services are typically developed and manages in-house since they capture the value- add essence of the organization, the ―secret sauce‖ that defines the way in which the organization does its business. Process Services are designed to enable process agility (i.e. to be easily updatable) and the process that they implement is typically episodic in nature (i.e. the execution comprises of short bursts of activity spaced by long waits for external activities to complete). Therefore, Process Services are best implemented as declarative workflows implemented using an integration server (such as BizTalk Server) or a workflow framework (such as Windows Workflow Foundation). Process Services are typically used by a single application and can therefore be managed individually (for example, at a departmental level). In some cases a reusable business process may become a commodity that can be offered or consumed as SaaS When designing business software, we should remind ourselves that the objective is delivering agile systems in support of the business; not service orientation (SO). Rather, SO is the approach by which we an enable business and technology agility, and is not an end in itself. This must particularly be borne in mind with references to Web services. Achieving the agility that so often accompanies Web services is not just a consequence of adopting Web service protocols in the deployment of systems, but also of following good design principles. In this article, we consider
several principles of good service architecture and design from the perspective of their impact on agility and adaptability. A Services Lifecycle Now that we have examined the types of services that may exist within a SOA, a more holistic look at services is needed. A Services Lifecycle can be used to understand the activities, processes and resources necessary for designing, building, deploying and ultimately retiring the services that comprise a SOA. A Service comes to life conceptually as a result of the rationalization of a business process and decomposition and mapping of that business process into the existing IT assets as wells the new IT assets to fill the gaps. The new IT assets once identified will be budgeted and planned for SDLC activities that result in deployable services (assuming that our goal is to create reusable IT assets). Following are various important activities that happen (not necessarily in this strict order) during the life time of a service from the service provider perspective: Figure 4: A Services Lifecycle Service Analysis Service Analysis is the rationalization of business and technical capabilities with the express notion of enabling them via services. Other aspects such as SLAs, localization / globalization, and basic service contracts will be established for future use in the life cycle.
Service Development Rationalization of contracts (XML Schemas) and designing new contracts will be one of the primary activities in this phase. Object libraries supporting the service implementation will be acquired or designed. Security policies, trust boundaries, authentication/authorization, data privacy, instrumentation, WSDL, etc. will be the outcome of this phase. Distributing WSDL or service consumer proxies will be strategized during this phase. Services will be developed using the selected IDE, Web services stack and the language of choice. Service Testing Services will be unit, smoke, functional and load tested to ensure that all the service consumer scenarios and SLA ranges are met. Service Provisioning Service metadata as identified in the ―Service Consumption‖ will be deployed into the directory. This will be associated with a deployment record into a repository that models deployment environment. Supported SLA policies will be an important metadata for successful operation of a service. Service gets a production endpoint in an appropriately designed production infrastructure. Support teams will be trained and appropriate processes for support among various roles (business versus IT) will be established. Access to service consoles and reports will be authorized to these roles. Service Operation This is the most important activity as the ROI will be realized through the operation of the services in production. The management infrastructure will do the following: Service Virtualization Service Metering (client usage metering and resource metering) Dynamic discovery of service endpoints Uptime and performance management Enforce security policies (authentication, authorization, data privacy, etc.) Enforce SLAs based on the provisioning relationship Generate business as well as technology alerts for a streamlined operation of the service Provide administrative interfaces for various roles Generate logs and audit trails for non-repudiation Dynamical provisioning (additional instances of the service as necessary) Monitor transactions and generate commit/rollback statistics
Integrate well with the systems management tools Service, contract and metadata versioning Enforce service decommissioning policies Monetization hooks Reporting Service Consumption This activity is equally applicable to service consumers and providers as providers may consume services as well. During this activity, services will be discovered to understand the following: Service security policies Supported SLA policies Service semantics (from the lifecycle collateral attached to the service definition) Service dependencies Service provisioning (will be requested by the consumer) Pre and post-conditions for service invocation Service development schematics (proxies, samples, etc.) Service descriptor artifacts Service impact analysis Other documentation (machine readable as well as for human consumption) During this activity, service consumers will be authorized to discover the service and its metadata. SLAs will be pruned to meet the desired level of availability based on the negotiated contract. Service Change Management Service like any IT application asset will go through several iterations during its lifetime. Service contracts will change, service security as well as SLA policies will change, the implementation will change, and the technology platform may change. Some of the above changes may be breaking changes. So, the management infrastructure has to be resilient for all the mutations by providing necessary deployment support across all the above changing dimensions. Service Decommission As a result of a change in the business strategy or as a result of better alternatives or as a result of waning consumer interest, a service may be decided for decommissioning. Management infrastructure should be able to enforce retirement policies by gracefully servicing the consumers until the last request.
SOA Scenarios There are a number of business and technical scenarios for which SOA delivers a clear benefit. This section lists several of the most commonly used scenarios for SOA (this is not a comprehensive list). Information Integration The Information Integration scenario is sometimes referred to as ―the single view of the customer problem‖. The complete description of a customer might be spread across a dozen business applications and databases. This information is rarely completely in sync, and aggregating this information for optimal customer (or partner or employee) interaction is poorly supported. Information integration services are an effective means for both presenting your application portfolio with a unified view of these key entities, and for ensuring the consistency of the information across all of your back-end systems. Information integration projects can run from the tactical to the broadly strategic; incrementally re-engineering information access and management across the enterprise. This scenario is frequently associated with the following industry acronyms (each of which are sometimes used interchangeably): MDM: Master Data Management is an approach for providing and maintaining a consistent view of the organization‘s core business entities (not just customers). EII: Enterprise Information Integration is broader than MDM, using data abstraction to address the challenges typically associated with data heterogeneity and context. CDI: Customer Data Integration is the combination of the technology, processes and services needed to create and maintain a complete view of the customer across multiple channels, business lines and enterprises. CDI is typically associated with CRM systems. Chapter Four discusses the Information Integration scenario in greater detail. Legacy Integration The Legacy Integration scenario focuses on the tactical use of services to preserve existing investments in business applications, while extending the functionality of the capabilities upon which they deliver. For example, a service might add support to comply with new regulations in front of an existing ERP package. Applications would be engineered to exchange messages with the service, which would extract the compliance-relevant data and then communicate the request to the ERP package. Process Governance Process Governance is a far broader than either Information or Legacy Integration. In a Process Governance scenario, "header" elements are used to communicate key business metadata; from the turnaround time on customer requests to the identity of the approvers for specific business
decisions. This metadata is captured by a utility service (as discussed previously), for real-time and/or aggregated analysis. "Service native" processes would include this information in SOAP headers, while non-native applications would need to be re-engineered to transmit the metadata as a message to the governance server. Consistent Access Consistent Access is a more technical and subtly different scenario than any of the scenarios previously discussed. This scenario enables a services layer to ensure consistent enforcement of a variety of operational requirements when a diverse set of applications needs to connect to a critical back-end resource. By mandating that all access be routed through a service facade, an organization might enforce consistent access authorization, cost distribution and load management. Resource Virtualization A Resource Virtualization scenario can be utilized to help enforce loose coupling between resources and consumers, effectively insulating consumers from the implementation details of the targeted resources. Typical examples of Resource Virtualization may include: Context-sensitive and content-sensitive routing of requests, such as sending a real-estate inquiry to the agent in the specified geography who specializes in farm properties. Routing of requests to partitioned information stores (without requiring the requestor to understand partitioning schemes). Load balancing requests across available resources; from customer service representatives to streaming video feeds. Process Externalization Process Externalization scenarios utilize Web services to help securely negotiate common processes such as payroll processing, employee expense reimbursement, and logistical support. Cell phone service providers and Internet portals frequently use Web services to aggregate content while customer-facing organizations may use services to build composite offers (such as travel packages that include airfare and rental cars). The key to successful process externalization on today's technology stack is to manage your own expectations; compromise your requirements to the limits of the existing technologies so that you don't spend your profits or savings on building infrastructure services that you will replace in a few years' time. Other Scenarios There are far too many SOA scenarios to document them all. The scenarios discussed above represent some of the more common scenarios in which we have seen SOA succeed. Another common scenario is human interaction. How can services within a SOA interact with end users?
SOA and the End User Figure 5: Comparing Systems and Users Enterprise Application Integration (EAI) typically deals with system-to-system integration, ignoring computer-human interactions. System-to-system interactions tend to be very structured in terms of both process and data – for example, an Order to Cash process will use well-defined processes (Process PO) and business documents (Purchase Order). System-to-system interactions rarely mirror the real world. Processes tend to follow what is traditionally termed the ―happy path,‖ since exceptions are poorly or rarely handled effectively. How people work in the real world is much different – processes are ad-hoc and may change frequently within a given time period. The data we work with is equally unstructured since it may take the form of Office documents, videos, sound files, and other formats. The intersection of these two worlds represents the intersection of system-to-system interactions and human workflows (we will discuss this topic in greater detail in Chapter Three). Applications that effectively support these human workflows can be difficult to design and develop. The vast majority of end users today use Microsoft Office for email and information work. Microsoft Office represents the results of many years of research and investments geared towards effectively supporting human workflows (in terms of both data and processes). Microsoft Office 2007 has evolved to become a first-class integration platform, providing a familiar user experience for service consumption, enabling: Models for many business concepts including business entities, business events and event-driven business rules, task assignment and fulfillment, modeled workflows, and many others. An application lifecycle model with customization and versioning, self-contained zero- touch deployment, and management through the whole lifecycle.
Tools support for creating models, composing applications and managing through the lifecycle, based upon Visual Studio Tools for Office (VSTO) and other platform tools. Microsoft Office 2007 supports simple event handling support for common line of business (LOB) events, including workflow synchronization events. Since most requirements for integration of LOB content with Office revolve around surfacing business entities such as Account, Invoice, Quote and Product, Office Business Entities are exposed in a way that creates unique value for the knowledge worker. Entity relationship models are hardly new. The design of business applications has traditionally been based on business entities along with the business rules and logic associated with the data. Office Business Entities (OBEs) have a few special aspects: OBEs can be turned into Office native content, maintaining the link to the LOB sources while being subject to all the rules and features of the Office application to which the content is native. For instance a list of parts in a product entity can be inserted as a table in Word, while maintaining the ability to refresh and drill into the data in the LOB application source. This is a kind of self-describing smart tag in that it does not have to be recognized, it comes with its special behavior already built into the content. OBEs enable business entities to be treated as first-class citizens of the Office world. OBEs will be used offline, attached to e-mails, created, correlated, shared and edited in collaborative environments like SharePoint and Groove with user control over the commitment of their data to LOB data sources. Most of the knowledge work in a business happens in applications like Excel, Word, and Outlook, before, after and aside from creating the data of record—for instance a quote to be created or updated must be worked on by many people before being committed to the system of record. Business work is like a 3-dimensional creative and collaborative world from which the transactional applications capture a 2-dimensional projection of committed data. With the Office- friendly behavior of OBEs, the full lifecycle of business data usage can maintain coherence without resorting to awkward and error-prone cut-and-paste transitions between the document world and the business application world. OBEs are coupled with reusable UI experiences that can be used in rapid application development (RAD) to quickly produce context-driven business solutions. UI parts can be associated with OBE views, which can be used in a drag-and-drop design experiences to surface LOB data within Office. Relationships between UI Parts can be navigated dynamically using links, creating a web-like experience around business entities. The client runtime also provides a declarative programming model that allows user experience to be driven by standard Office context events (such as item open) with the experience tailored by parameters such as role and locale.
The content of OBEs can be bound to the content of Office entities, without becoming a part of it. This is easiest to observe in Outlook items, where, for instance, a contact item in Outlook can be bound to a customer contact in a CRM system. Both the Outlook entity and the CRM entity exist independently, each with its own identity and behavior, but some of their properties (such as address) are conceptually shared by being bound to each other and synchronized automatically. Thus a hybrid Outlook/CRM entity is created conceptually with correlation and data synchronization rather than data sharing. This becomes visible in Outlook as extended data and user experience for such hybrid contacts; surfacing some CRM contact data and behavior as extensions of the Outlook contact UX. Hybrid entities create a deep but non-invasive association. The use of hybrid Office/LOB entities is most interesting for Outlook items today because Outlook items possess a firm identity which is needed for correlation with OBEs. As document sharing occurs in more controlled SharePoint/Groove environments as part of processes like document assembly, based for instance on Word templates such as ―contract‖ or ―RFP‖, more Office entities will gain stable identities and become available for two-way correlation with OBEs. LOB entities in many cases are fragmented into data silos and the data in these silos is often of questionable quality. OBEs can mask these problems while creating a rich user experience linked deeply to the real work context of the knowledge worker. What are Composite Applications? A composite application is a collection of software assets that have been assembled to provide a business capability. These assets are artifacts that can be deployed independently, enable composition, and leverage specific platform capabilities.
Figure 6: High-level representation of a composite application In the past, an enterprise's software assets were usually a set of independent business applications that were monolithic and poorly-integrated with each other. However, to get the business benefits of composition, an enterprise must treat its software assets in a more granular manner, and different tiers of architecture will require different kinds of assets such as presentation assets, application assets, and data assets. For example, a Web service might be an application asset, an OLAP cube might be a data asset, and a particular data-entry screen might be a presentation asset. An inventory of software assets by itself does not enable composite applications. This requires a platform with capabilities for composition—that is, a platform that provides the ability to deploy assets separately from each other, and in combination with each other. In other words, these assets must be components, and the platform must provide containers. Containers provided by the platform need to be of different types, which map to the different tiers in the architecture. Enterprise architectures are usually decomposed into three tiers: presentation, application (or business logic), and data. So the platform needs to provide containers for these. However the 3-tier architecture assumes structured business processes and data, where all requirements are made known during the process of designing and building the system. By their very nature, composite applications presume that composition of solutions can occur after assets have been built and deployed – and so need to explicitly account for people-to-people interactions between information workers that are essential to get any business process complete. Usually these interactions are not captured by structured processes, or traditional business applications, and therefore it is critical to add a fourth tier - the productivity tier – to account for these human interactions. This is shown in Figure 7.
Figure 7: The four tiers of a composite application Traditional discussions around the architecture of business applications tend to focus on the application tier as being the connection between people and data. Typically, however, the application tier contains structured business logic; and this holds for discussions around Service Oriented Architectures (SOAs), Enterprise Service Buses (ESBs), Service Component Architectures (SCAs), or most other architectural perspectives in the industry today – including first-generation discussions around composite applications. However, building a composite application requires a mindset that not only is the productivity tier a critical element of the stack, but also contains the most business value. To expand on the comparison between composite applications and SOA, both of them target flexibility and modularization. However, SOA provides flexibility at just one tier: the structured business logic in the middle tier. Composite applications target flexibility at all four tiers. That said, a composite application is a great way to surface information out of an SOA, and having line-of- business (LOB) applications exposed as services makes it easier to build support for cross- functional processes into a composite application. Therefore to design a composite application, a solutions architect must: Choose a composition stack –Pick one or more containers from each tier, and a set of components types that are deployable into those containers. Choose components – Define the repository of assets that must be built from this set of component types, based on business needs.
Specify the composite application – Define the ways in which those assets will be connected, to provide a particular cross-functional process. The platform should enable these connections to be loosely-coupled. Then after deployment, users will have the opportunity to personalize both assets and connections, as the composition stack should enable this through loose coupling and extensibility mechanisms. Figure 8: A Composite Application Architecture What does a Composite Application look like? A figurative representation of a composite application is shown in Figure 8, which shows a very abstract representation of an enterprise solution, deconstructed along the lines of Figure 7. At the top are information workers, who access business information and documents through portals that are role specific views into the enterprise. They create specific documents during the course of business activities, and these activities are part of larger business processes. These processes coordinate the activities of people and systems. The activities of systems are controlled through process specific business rules that invoke back end LOB applications and resources through service interfaces. The activities of people plug into the process through events that are raised when documents specific to the process are created, or modified. Then business rules are applied to the content of those documents, to extract information, transform it, and transfer it to the next stage of the process.
Figure 9: Deconstructing an enterprise application Today most line-of-business applications (LOB) are a collection of resources, hard-coded business processes, and inflexible user interfaces. However based on the previous section, it is clear that enterprise solutions need to be broken down into a collection of granular assets that can be assembled into composite applications. A high-level approach for doing this to any business process is listed below: 1. Decompose the solution for a business process into software assets corresponding to the elements shown in Table 1 below. 2. Package all assets corresponding to a given business process into a ―process pack‖ for redistribution and deployment. This would contain metadata and software components, and solution templates that combine them. The process pack would also contain service interface definitions that would enable connections to other IT systems. These connections would be enabled by implementing the service interfaces, for example. to connect to LOB applications and data. The goal is to be able to easily layer a standardized business process onto any heterogeneous IT landscape. 3. Deploy the process pack onto a platform that provides containers for the types of assets that the solution has been decomposed into. The platform should provide capabilities for rapid customization, personalization, reconfiguration, and assembly of assets. 4. Connect the assets within the process pack, to existing LOB systems, and other enterprise resources by implementing the services interfaces. These connections could be made using Web services technologies, other kinds of custom adapters, or potentially even Internet protocols like RSS.
Documents UI Screens Workflows Data connections Business activities Authorizations Business rules Reports Schemas Metrics Interfaces to connect to back end systems (Web service APIs) Table 1: List of application assets for composition Expected Benefits of Composition, and How to Achieve Them Deployment of enterprise applications should be tied to business benefits in the Triple-A sense (agility, adaptability, alignment). These benefits need to be demonstrated from two perspectives: The Solution Provider Perspective (or Development Perspective) – This is the perspective of the organization that builds an enterprise application. This might be an Independent Software Vendor (ISV), or a Systems Integrator (SI), or even an in-house IT department. The solution provider perspective is concerned primarily with benefits gained in activities relating to designing, implementing, and deploying enterprise applications. The Solution Consumer Perspective (or User Perspective) – This is the perspective of the organization that uses an enterprise application. Typically this is the business unit that commissioned the enterprise application. The solution consumer perspective is concerned primarily with benefits gained by the business after the solution has gone into production The benefits of composition that can be reasonably expected in each of these two perspectives are listed here, along with some high-level best practices to achieve these expected benefits. Conclusion In this chapter we examined the concept of SOA from the perspective of services: architectural and organizational maturity, service types, service lifecycles, scenarios and the role of users and composite applications within a SOA initiative. Service oriented architecture (SOA) is a design approach to organizing existing IT assets such that the heterogeneous array of distributed, complex systems and applications can be transformed into a network of integrated, simplified and highly flexible resources. A well-executed SOA project aligns IT resources more directly with business goals, helping organizations to build stronger connections with customers and suppliers, providing more accurate and more readily available business intelligence with which to make better decisions, and helping businesses
streamline business processes and information sharing for improved employee productivity. The net result is an increase in organizational agility. In an SOA, the concept of applications will still exist, especially when one considers an enterprise‘s IT investments. However, the concept of one vendor supplying a complete ―SOA solution‖ with a monolithic set of products is being replaced with a ―best-of-breed‖ approach, enabling customers to adopt a capabilities-based approach to implementing their architectural requirements. Organizations should remain focused on solving their business problems and avoid being distracted by integration trends and buzzwords. SOA should be a means for making the business more agile, not the end goal. Designing and implementing SOA should be an incremental process with rapid deployments and ROI realization. SOA should not be a top-down, multi-year ―boil-the- ocean‖ effort – these types of projects rarely succeed because they are unable to keep up with the shifting needs of the organization. Users will also undergo a transformation in the way they work with applications. Depending on the type of application, a user could either be exposed to specific tasks of a process, e.g. working in the context of a document workflow in Microsoft Office SharePoint Server, or, an application might encapsulate a business process internally and let a user start the process, but not interact with it during its execution. Chapter Three provides a more detailed discussion of the Workflow and Process recurring architectural capability.
SOA Case Study: Commonwealth Bank of Australia This case study describes how the Commonwealth Bank of Australia designed, developed, and implemented its CommSee application – a relationship banking solution, custom-built by the Commonwealth Bank of Australia using Microsoft® .NET technologies. This solution was developed as a Microsoft® Windows® Forms–based Smart Client that consumes standards- based .NET Web services. The Web services are responsible for orchestrating data from a variety of back-end data sources including mainframes, databases, and various backend legacy systems. At the time of this writing, CommSee has been successfully deployed to 30,000 users at more than 1,700 sites across Australia. After considering the business needs that CommSee was designed to address, this case study will examine the solution architecture, technical project details, and best practices employed in this major software development effort. CBA Architectural Overview The entire Case Study is available online at http://msdn2.microsoft.com/en- us/library/bb190159.aspx. .
See other SOA case studies at http://www.microsoft.com/casestudies/search.aspx?Keywords=SOA.
References: 1. ―Enabling the Service-Oriented Enterprise‖, Architecture Journal, April 2006. Available at http://msdn2.microsoft.com/en-us/library/bb245664.aspx 2. ―Ontology and Taxonomy of Services in a Service-Oriented Architecture‖, Architecture Journal, April 2007. Available at http://msdn2.microsoft.com/en-us/library/bb491121.aspx 3. ―Office Business Applications: Building Composite Applications Using the Microsoft Platform‖, December 2006. Available at http://msdn2.microsoft.com/en- us/library/bb220800.aspx 4. ―Service Oriented Infrastructure‖, Mark Baciak. Available at http://msdn2.microsoft.com/en-us/architecture/aa973773.aspx 5. ―Service Orientation and Its Role in Your Connected Systems Strategy‖. Available at http://msdn.microsoft.com/architecture/solutions_architecture/service_orientation/default. aspx?pull=/library/en-us/dnbda/html/srorientwp.asp
Chapter 3: Workflow and Process "The art of progress is to preserve order amid change and to preserve change amid order." - Alfred North Whitehead Mathematician, philosopher (1861-1947) Reader ROI Readers of this chapter will build upon the concepts introduced in previous chapters, specifically focusing on the Workflow and Process architectural capability. Figure 1: Recurring Architectural Capabilities The Workflow and Process architectural capability focuses on the concept of workflow and how workflow can be used to orchestrate and aggregate granular services into larger constructs that we call processes. Topics discussed in this chapter include: Explaining the concepts of workflow Clarifying the relationship of BPM, SOA, orchestration and choreography to workflow Describing the capabilities necessary for supporting workflow The benefits of adopting a model-based approach to workflow
Acknowledgements This chapter consists of work from Dave Green (concepts, semantics, value and capabilities, Windows Workflow Foundation) and John Evdemon (workflow concepts, terms and manifesto).
Understanding Workflow What is Workflow? The concept of workflow is not new. Workflow technologies first emerged in the mid-1970s with simple office automation prototypes at Xerox Parc and the University of Pennsylvania‘s Wharton School of Business. Interest in workflow and office automation began to wane in the early 1990s until the book Reengineering the Corporation reignited interest in workflow and business processes. The reengineering trend of the 1990s gave us several books and methodologies for process analysis – unfortunately the technologies such as CASE and their ilk were immature and required significant manual intervention, exposing projects and executive stakeholders to significant levels of risk. Even worse, other ―workflow products‖ were thinly disguised efforts to sell hardware such as scanners, printers, and other peripherals. Clearly the term ―workflow‖ was being abused to take advantage of confusion in the market So what is workflow? Workflow is fundamentally about the organization of work. It is a set of activities that coordinate people and / or software. Communicating this organization to humans and automated processes is the value-add that workflow provides to our solutions. Workflows are fractal. This means a workflow may consist of other workflows (each of which may consist of aggregated services). The workflow model encourages reuse and agility, leading to more flexible business processes. Workflow Terminology As stated above, workflow is about the organization of work. This is a very broad definition and may be interpreted in a number of ways. To avoid confusion we will identify several terms that are commonly associated with the concept of workflow: o Business Process Management (BPM): BPM is a business philosophy that views processes as a set of competitive assets to be managed. The processes that business seeks to manage are largely composed upon and executed by a SOA infrastructure. SOA and BPM have several objectives in common: o Both are focused on making the organization more agile. SOA focuses on providing a loosely coupled infrastructure for service enablement while BPM focuses on effectively designing and managing loosely coupled business processes (which are aggregations of granular services). o Both seek to make the business proactive instead of reactive. A proactive business is able to spot market patterns, trends and competitive threats before they can negatively impact the organization. Simply identifying a threat or opportunity is not enough – the organization must also be able to proactively
modify its existing processes to take advantage of the opportunity (or avoid the threat). Achieving this level of agility requires loosely-coupled business processes atop by an SOA infrastructure capable of supporting rapid reconfigurations to meet the needs of the organization. o Both are focused on process-centric instead of functional capabilities. Processes can no longer be constrained by a limited set functional capabilities – an SOA infrastructure is a fabric of configurable, composable services that can be assembled/disassembled or reconfigured to meet the shifting business needs of the organization. o Orchestration: Workflows and orchestrations are effectively the same thing since each is designed to compose and execute a series of tasks. Orchestrations require a ―conductor‖ that is always in charge of the execution. The conductor is typically manifested as an integration server (such as BizTalk Server) which monitors execution, raises events, creates execution logs and performs various other duties to ensure the process executes as expected. o Choreography: Choreography is a set of peer to peer relationships between individual participants. There is no ―conductor‖ in choreography. Participants interact with one another based upon a set of agreed-upon principles or contracts. For example, a Supplier might enter into an agreement with a Retailer to ship a specific set of goods and services within a given timeframe. This means the Supplier and Retailer must agree upon business events, business documents, service level agreements (SLAs) and penalties if the SLAs are not met. The Supplier may in turn enter into another agreement with a Manufacturer to supply the raw materials that the Supplier needs to meet the Retailer‘s demands. A choreography is usually implemented by ―splitting‖ it up into multiple orchestrations (one for each participant involved in the choreography). Discussions about workflow frequently mention one of the terms explained above. Orchestration is the only term that is comparable to the concept of workflow. Why Workflow? The problems which a workflow approach has been applied often display three characteristics: The key business value delivered is coordination, for instance, in organizing multiple contributions to the preparation of a quote or driving a document review. Each instance of the business process concerned is of long duration, measured often in days, weeks, or months, rather than minutes. The business process has human participants, who usually contribute most of the work product. However, only a small proportion of the business problems with these characteristics are solved using a workflow approach. Most commonly, the business process is not recorded as machine-
readable information at all. Rather, each of the humans participating in the business process interacts with business systems that are not aware of the semantics of the process as a whole, such as a customer information system, and with other human participants through content- neutral communications channels such as e-mail. Each human participant uses a mental model of their part in the overall business process to determine their behavior. Three key benefits that a workflow model can bring are insight, monitoring, and optimization. A set of related workflow models can be used to gain insight into the flow of work through an organization. For monitoring, knowing which individuals are contributing work to which business process is very useful when trying to understand costs and workloads. For optimization, having a model of the work being undertaken, and being able to use the model to interpret behavior, together make it possible to reason about how to optimize the business process. The concept of a model-driven approach is attractive it is not new – developers have been using UML and other modeling techniques for many years prior to the maturation of workflow development tools. Workflow provides either a sequential or state machine model that is used to develop, manage and run the workflow itself. Code-only approaches may take advantage of models to lesser success – for example we can parse C# code into an abstract CodeDOM model – but the usefulness of such a model is limited. A Workflow Model Given these compelling benefits, why haven't workflow models been used more widely? The most likely answer is that the cost of using them has been too high. These costs include product costs, that is, the direct cost of purchasing a workflow product; integration costs, where processes modeled as workflows need to be integrated as part of a larger business system; and standardization costs, where it is difficult for a large organization to standardize on a single workflow technology. Variations in workflow products also mean that skills and model portability are issues. Let's look at the possibility of addressing these blocking issues by building applications on a workflow platform that is low cost, ubiquitous, uniform, and easily integrated in applications. To be clear, the idea is not to replace workflow products. Rather, the hypothesis is that it is useful to factor out support for some core workflow concepts into a platform on which both workflow products and other applications can be built (see Figure 1, below). A workflow is a model, which means it is a machine-readable description of business behavior that is not code. The meaning and benefits of this concept in the context of the value of a workflow platform will be discussed later.
A workflow model describes an organization of work units. For instance, suppose that a document review process specifies that Joe writes the document and then Fred reviews it. Here, the work units are first writing and second reviewing the document, and the organization is that one task must follow the other. This concept is not a radical idea. Code that makes successive calls to two subroutines is a valid example of the concept. The interest lies rather in the forms that this organization takes. To test the workflow platform hypothesis, we will consider a range of real-world applications and explore the characteristics that a workflow platform should have if it is to prove useful. A document review process takes as an input parameter a set of [reviewer, role] pairs that describe which humans are involved in the workflow in which roles. Possible values for the role are required, optional, final approver, and owner. The review process then proceeds until all reviewers have performed their assigned roles and notifies the owner of the outcome. Here, the work items are the document reviews organized by the review process. There are three interesting characteristics to call out, namely, multiple points of interaction, human and automated activity, and the need to handle dynamic change. Workflow Contracts The workflow has multiple points of interaction, or contracts. First, there is a contract with a reviewer. This contract involves asking a reviewer to review a document, accepting the verdict and any review comments, and also telling a reviewer that his or her input is no longer required (if the review is canceled, or perhaps if enough reviewers have voted yes). The contract might also allow a reviewer to delegate a review. Then there is a second contract with the final approver, which is a specialization of the reviewer contract. Third, there is a contract with the owner of the review that allows the owner to cancel the review and be notified of the outcome of the review. Finally, there is a contract with the initiator of the review process, who instantiates the review and supplies the required parameters. It is typical of workflows that they connect multiple parties through a variety of contracts (see Figure 2). The document review workflow is essentially a coordinator, initiated through one contract that is coordinating a variety of participants through one or more additional contracts. The document review workflow drives human activity. However, it might also drive automated activities, such as storing versions of the document in a repository as the review progresses. From the point of view of the workflow, there is no essential difference. A workflow can be thought of as communicating, in general, with services through contracts. One special case of a service is another workflow. Another special case is a human. In many ways, a human is the original asynchronous service: one never knows when or if it is going to respond.
A characteristic of this type of workflow is that the participants will ask for changes to the workflow as it executes. For example, a reviewer might delegate a review task to a colleague or share the work involved in a review task with a subordinate. There are two ways of addressing this requirement. One is to build an understanding of all the possible changes into the workflow. Then, a delegation request becomes just another function of the contract between the workflow and the reviewer. The other possibility is to see change as something separate from the workflow, where change is implemented as an external function that changes the workflow model. In this approach, the result of delegation is a new workflow model identical to one in which the review task was assigned to the delegate from the beginning. Requesting an additional approval step would add a new approval task to the workflow model, which might well have contained no approval steps at all in its original form. The workflow no longer has to anticipate all possible modifications; at the most it will be concerned with restricting the areas of the model that are subject to change. Both approaches are useful. Building understanding into a workflow is simple to model and understand. Generalizing operations is more complex to model, but more powerful and agile. In an extreme but interesting case of the latter approach, the workflow begins execution with little or no content, and the required behavior is added dynamically by the participants in the workflow. Here, the available operations for modifying the workflow become a vocabulary that a user can use to construct the desired behavior as the workflow progresses. Problem-Resolution Collaboration To look at a specific example of a problem-resolution collaboration application, consider an inventory shortfall. An assembly line is making a gadget, and the computer indicated that there were enough widgets in stock for the purpose. However, when the stockroom manager went to fetch the widgets for delivery to the assembly line, a shortfall of 10 widgets was discovered. Collaboration among the stockroom manager, the customer's account manager, the supplies department, and the production manager is required to resolve the problem. Each role in the collaboration may take characteristic actions. The supplies department could order more widgets, perhaps using a different supplier or paying an existing supplier more money for faster delivery. The account manager could go to the customer and request deferred delivery or split the delivery into two parts and bear the extra shipping cost. The production manager could divert assembled gadgets from an order for another customer. The stockroom manager could search his or her physical stock in an attempt to find the missing widgets. Any given action might be performed multiple times. One obvious constraint is that the collaboration is not completed until the shortfall is resolved by some combination of the previous actions. There will often also be business constraints. For
instance, there might be a rule that says deferral of delivery to gold customers is never permitted. Also, the actions will affect each other. For instance, there might be a policy that states that total added cost from corrective action may not exceed 5 percent of original factory cost. Thus, placing an order for accelerated supplies at a higher price might prevent a shipment from being split. Figure 9: A monolithic workflow stack In this case the work items are the actions that the various participants can take as they seek to resolve the inventory shortfall. The organization, however, is not the same as that required in document review. The participants are not dictated to; instead, they choose which actions to perform and when to perform them. However, these choices are constrained by the organization of the workflow, which has two aspects: 1) The actions are focused on achieving a goal; in this case, resolving the inventory shortfall. A bounded collaboration space is created when the problem resolution starts, and is not closed until the goal has been reached. 2) The participants are not free to perform arbitrary actions. Instead, the available actions are determined by the role the participant is performing and the state of the collaboration. The set of actions available is determined by policies related to the goal and global policies such as the restriction on shorting gold customers. The actions available vary as the collaboration progresses. Figure 10: A contract diagram for the document review application The experience of the participant is no longer that of performing assigned tasks. Instead, a participant queries for the actions currently available to him or her, performs none or one of these actions, and then repeats the cycle.
The main new requirement here, therefore, is for a form of organization of work items that is essentially data state and goal driven. There is also a requirement to support a query/act-style of contract with a workflow participant. Scripted Operations Scripted operations are simply a set of operations that are composed using a script. An example might be a desktop tool that allows a user to define and execute a series of common tasks, such as copying files and annotating them. It would be unusual to consider using a typical workflow product for this purpose. However, it does fit the workflow platform pattern of a set of work units organized by a model. In this case the model is a sequence, perhaps with support for looping and conditional execution. Therefore, if a workflow platform were sufficiently low cost and ubiquitous, it would be possible to consider applying it to this sort of problem. Would doing so add any value? One feature of scripted operations that is not addressed by their typical implementations today is the question of data flow. It is common for the data required by one operation to be the output of some previous operation, but this information is not typically modeled in the script. Thus, a user assembling tasks using a desktop tool might not be told when creating a script that the prerequisite data for a task hasn't been supplied, and would only discover the error when running the script. A workflow model that can describe these data dependencies would add clear value for script authors. One approach is simply to include data flow constructs in the workflow model. It is highly arguable that the basic workflow model needs to include basic structural features such as sequences, conditions, and loops; but it is not clear that data flow is sufficiently universal to be represented by first-class elements of the model. An alternative approach is to layer support for data flow on top of an extensible, basic workflow. A workflow model that can be enriched with abstractions appropriate to a variety of problem domains fits well with the notion of a workflow platform. This approach avoids both the complexity created by including in the base model a large variety of semantic constructs specialized for different problems and also the limitations imposed by limiting the workflow model to a fixed set of constructs. Now let's look at a guided user application. One example is an interactive voice response (IVR) system, and another is a call center system guiding telephone operators through support or sales scripts. The essence of these applications is to guide users through the series of operations needed to achieve their goal. The organization of these operations is typically used to drive the presentation to the user, whether this is generated speech or a set of enabled and disabled command buttons on a form.
A characteristic of this type of application is that the workflow is the most frequently changed part of the application. Also, the business sponsors of the system are often heavily involved in specifying the changes, making it important to provide a way for IT staff and business personnel to communicate clearly and efficiently about the changes. A workflow model that expresses the core business purpose of the application, stripped of any irrelevant technical material, is an effective way to achieve this communication. These applications also require flexibility within the workflow structure. In an IVR application the user will typically be heavily constrained, moving through a hierarchically structured set of menus. However, there will also be escape commands—for example, "return to root menu" or "skip out of current subtree." A call center application will have more flexibility than an IVR application, changing the options offered to the user in response to the state of an order or in response to the input from a customer, such as skipping sales prompts if the customer starts to react negatively. This sort of application requires support for a mix of organizations of work items, combining sequences, loops, and conditions with jumps from one state to another, and also the kind of data- driven behavior seen in problem-resolution collaboration. Rule and Policy As discussed previously, one way in which the workflow approach can deliver value is by isolating the focus of change in an application. Often, this focus is on the way in which the work items are structured, but in some applications the focus of change is on expressions tied to a relatively slow-changing structure. An example of this focus is an insurance policy quotation system, where a set of frequently changing calculations is used to drive decision making in the quotation process. The requirement is for the workflow to model these expressions, which has two key benefits: First, the testing and deployment costs are much lower than those that would typically be incurred if the expressions were written as code, since the model provides a strong sandbox restricting the scope of possible changes. Second, the changes can be made by personnel who understand the business significance of the expressions but do not have the skills to understand the technical code in which expressions written as code would inevitably need to be embedded. The Model-View-Controller (MVC) pattern often is used to wire a UI to an underlying object model (see Figure 3). The model represents the behavior of the system, independent of any particular UI representation. The controller is a part of the UI layer that is used to map the events generated by the UI into the method invocations required to drive the model. The UI itself is thus not polluted by any assumptions about the underlying model.
The workflows considered so far, viewed from this standpoint, all fall into the category of Models in the MVC sense. However, the controller can also be seen as a workflow. The work items it organizes are the methods provided by Model objects. The controller also interacts with the UI and the model through well-defined contracts. A model of this kind is often termed a page flow. As with scripted operations, page flow would not today be implemented using a typical workflow product. There are two reasons to consider building a page flow using a workflow platform. First, a model readily can be represented visually, helping developers and analysts to express and communicate the required behavior. Second, if the page flow is frequently changing, then the abstraction of the page flow as a model improves agility. There are two main requirements if this problem is to be addressed using a workflow platform. The workflow runtime must be lightweight, since a page flow may be running within a small application on a desktop, and the contracts supported must include the event-based contract characteristic of UIs, as well as the object-method contracts exposed by the Model. Now let's look at a test record/replay application example. The intent of this final example is to test the limits of the applicability of the workflow platform hypothesis. Figure 11: An MVC application The application here is a tool for testing applications built as a set of services. The tool uses an interception mechanism to record all the interaction between services that occur during manual performance of a test case for the application. This recording can then be replayed. During replay, externally sourced messages are generated without manual intervention, and messages between the set of services that make up the application are checked for sequence and content against the original recording. The workflow is the test case, organizing the work units that are the participating services. The workflow is both active, in that it simulates the behavior of externally sourced messages, and passive, in that it monitors the interactions between services.
A unique feature of this application is that the workflow is written, not by a developer or a user, but by a program, as part of the act of recording a test case. Workflow model creation must be fully programmable. There are also requirements for extensibility and dynamic update. Extensibility is required because the structural semantics are rich. For instance, just because two messages arrived at a service one after the other in the recording, there is no necessary implication that this order needs to be preserved in a replay. If there is no causal dependency between the messages, then a replay that reverses the order of the messages is correct. So the semantics of sequence in the model used to record the test cases need to include a notion of causality, which is not likely to be a feature of the core workflow model of sequence. Dynamic update is required because human interaction with the model will occur during replay. Discrepancies discovered during replay between recorded and observed behavior are popped up to a tester. If the discrepancy is because a message includes a timestamp that varies from run to run, then the tester would update the model to mark the field "don't care." If the discrepancy occurs in a regression test because the software has changed, then the tester might approve the change and update the test to expect the new behavior in all subsequent runs. Workflow Platform Value A workflow platform does not, by definition, have the full set of features offered by today's typical workflow products. Rather, the workflow platform considered here focuses on supporting the concept of a workflow as a model of the organization of work items. We have seen that this idea of an organization of work items is indeed applicable across a broad range of applications, but the fact that a workflow platform can be used doesn't mean that it should be used. Figure 12: The document review implementation schematic Two questions must be asked: What additional value is derived from the workflow platform approach? And, is this approach practical? This value of the workflow platform approach must
come from the expression of the organization of work as a model, which we'll discuss later. Let's summarize the characteristics that a practical and effective workflow platform must display. To demonstrate how a model differs from code, this code snippet is a valid workflow by the definition used here, that is, an organization of work units. public void HandleLoanRequest (string customerID, Application app) { if (CheckCredit(customerId, app.Amount)) { MakeOffer (customerId, app); } } And, in a sense, it is a model. It is possible to parse this code and build a CodeDOM tree that represents it. However, the semantics of the resulting model are so general as to be opaque. It is possible to tell that the code contains function invocations, but it isn't too easy to distinguish a function that represents the invocation of a work item from a function that converts integers to strings. A workflow model explicitly distinguishes these ideas. Typically, a specialized model element is used to represent the invocation of a work item, and conversion functions cannot be expressed directly in the model at all. A workflow model, then, is one in which its graph is built from elements that are meaningful in the workflow domain. The semantic richness of such a model can be exploited in several ways. Visualization. Visual representation of the model—typically in graphical form—is useful for the developer, during both development and maintenance, and also for workflow users who want to know why they have been assigned a given task or the IT operations worker who wants to understand what a misbehaving application should be doing. Insight. The workflow model is amenable to programmatic access for a variety of purposes. An example is static analysis to determine dependencies and flow of work across a set of cooperating workflows or using the model to drive a simulation that predicts the workloads that will be generated by a new version of a process. Expressiveness. The specialization of the workflow model to the workflow domain means that characteristic problems can be expressed more quickly and compactly. It is a domain-specific language (DSL), specialized to support characteristic problems. Consider a document review process where three positive votes out of five reviews mean that the document is good, and any outstanding reviews can be canceled. This process is quite difficult to code, but a workflow model can supply out-of-the-box constructions that address such problems.
More Semantic Exploitation As we have seen in the scripted operations application discussion, extending the workflow model to further specialize the out-of-the-box model language is a very powerful technique for delivered additional value. An example is the creation of a language intended for end users, as in the document review conducted using an improvised definition of the review process that was discussed previously. Execution. The specialization of the model makes it possible to add run-time support for common problems. A good example is long-running state. Of the applications discussed here, management of long-running state is required for the document review process, problem- resolution collaboration, and guided user applications. The workflow platform runtime can solve such difficult problems once, using simple expressive model elements to control a common capability and freeing up the developer to focus on the business problem. Monitoring. The existence of a model makes it possible to produce an event stream with a meaningful semantic without any additional developer effort. Of the applications described here, this event stream is useful in the document review, problem-resolution collaboration, test record/replay, and guided user applications. The event stream can be used to monitor instances of workflows or build aggregate views of the state of a large number of workflow instances. The standardization of the event stream makes it much easier to build such aggregate views across workflows that were developed independently of each other. Another powerful idea is the presentation of errors using a business semantic. Often, a technical failure such as the nondelivery of a message leads to escalation to a technical expert because the significance of the failure is unclear without specialist investigation. If the error can be mapped to a workflow model—so that it is clear that the error concerns a noncritical change notification, for instance—then escalation can be restricted to cases where it is necessary. Composition. If an application is factored into work units, then these work units, with their well- understood interfaces, can be reused by other workflows. Workflows themselves also define work units that can also be used by other workflows. Customization. Suppose that an ISV ships a workflow, which is customized by a VAR, and then again by a customer. Reapplying these customizations when the ISV ships a new base version is a challenging maintenance problem. The use of a shared, well-understood model for the workflow makes the consequent three-way merges much more tractable. Customization and composition together enable ecosystems where definitions of work and flow become shared or traded artifacts. Manipulation. As we have seen in the discussions of the document review and test record/replay applications, there are often requirements to invent or modify workflows on the fly. This
modification cannot be done securely if changing code is required. Using a model makes possible dynamic manipulation that is both controllable and comprehensible. These benefits make a compelling list, and it demonstrates clearly that the description of an organization of work items as a model has a lot to offer. Platform Characteristics There must be support for basic structural concepts like sequences, conditions, and loops. However, there also needs to be support for data-driven approaches to deal with the less- structured organizations that appear in applications like problem-resolution collaboration and guided user. It is also important to allow new semantic elements to be added to create rich, specialized languages such as the data flow-aware composition in scripted operations. Adding new semantic elements might go so far as to require the redefinition of such fundamental ideas as sequence— for example, in the test record/replay application. The workflow must also be able to communicate in a rich variety of ways. Workflows respond to UI events, drive different types of services (human, programmatic, other workflows), and support queries over the current state of their contracts—for instance, when determining the actions available to an actor in a problem-resolution collaboration application. If the workflow platform is to be used in all the applications where it adds value, such as MVC, then it must be lightweight. Equally, it needs to address the scale and performance requirements implied by applications such as document review. In addition, the workflow model itself must be fully programmable, which includes model creation—such as in the test record/replay application—and dynamic model update to support unanticipated change, as in both the document review and test record/replay applications. Now let's look at the realization of these required characteristics in the Windows Workflow Foundation (WF). WF implements the idea of workflow as an organization of work items, abstracted away from the related ideas with which it has been coupled in traditional workflow products. The abstractions fall under three main categories: design and visualization, hosting, and semantics. Design and visualization. A workflow in WF is a tree of work items (called activities). This tree can be manipulated directly as an object model. A designer is provided, but its use is not mandated. It is possible to create new designers specialized to particular user communities or to particular organizations of work items. It is also possible to specialize the provided designer, which can be used not only within Visual Studio but from within an arbitrary hosting application.
Hosting. The WF runtime is sufficiently lightweight to be hosted in a client context such as a controller in a rich-client application shell. It is also performant enough to scale when embedded in a server host, such as the SharePoint Server delivered by Office 2007. The WF runtime's expectations of its host are abstracted as provider interfaces for services such as threading, transactions, persistence, and communications. Useful provider implementations are supplied out of the box, but they may be substituted as required. Semantics. Different problems respond to different model semantics. WF supports three main styles of workflow out of the box: flow, state machine, and data driven. Flow is optimal for applications where the workflow is in control such as the scripted operations example. State machine is best when the workflow is driven by external events, as in the MVC or guided user applications. A data-driven approach is suited to applications where actions depend on state, as in problem-resolution collaboration. These semantics can be extended by building custom activities to create a domain-specific vocabulary for use in any of these styles. However, since the structure of a workflow is itself expressed as a set of activities, the same approach can be used to define new styles, and entirely novel semantics, if required. A Common Workflow Runtime The focus of the WF runtime is to deliver facilities required by any workflow, and therefore avoid the need to re-implement them time and again in different applications, but without compromising the flexibility of the workflow abstraction. These common facilities fall into four main categories: activity scheduling, transactions and long-running state, exceptions and compensation, and communications. Let's look at each in more detail. Activity scheduling. The WF runtime defines an activity protocol that all work items implement. This protocol defines the basic activity life cycle (initialized, executing, and closed) and the additional states needed to handle exceptions (faulted, canceling, and compensating). This definition enables the WF runtime to provide work scheduling for all workflows. Transactions and long-running state. The WF runtime supports the execution of ACID transactions. These transactions are particularly useful for maintaining consistency across workflow state and external state such as application and message state. However, ACID transactions are not suitable for managing long-running state because of their resource and locking implications. The WF runtime implements a broader checkpoint-and-recovery mechanism to handle long-running state. From this point of view, ACID transactions become units of execution within a larger framework. The developer needs not do any work to get the benefit of WF's support for long-running state, as it is default behavior. However, if more detailed control is required, a set of simple model elements are supplied for the purpose.
Exceptions and compensation. The familiar idea of throw-try-catch exceptions is supported by the WF runtime and represented in the out-of-the-box workflow model. However, the WF runtime also supports a broader view of fault handling that includes the idea of compensation for successfully completed transactional units. Communications. As we have seen, workflows need to communicate in a variety of ways, which is reflected in the WF, that supports communication through .NET method, event interfaces, and Web service interfaces. Support for Windows Communication Framework will also be made available in the future. Thus, WF does indeed realize the workflow-platform approach proposed here. Figure 4 illustrates the high-level implementation schematic of the document review application and how all of the foregoing comes together. An implementation uses SharePoint as the workflow host. The WF runtime uses the default scheduling service provided out of the box with WF. However, the default persistence and communications services are replaced with implementations specialized for the SharePoint host. The persistence service stores long-running workflow state in the SharePoint database, and the communications service makes the rich-user interaction facilities of SharePoint available to the workflow. Both of these services are in fact delivered out of the box with Microsoft Office 2007. Three sorts of activities are used to define the document review workflow itself. First, out-of-the- box WF activities are used to provide structural elements such as If-Else and While. Second, activities provided as part of Office 2007 are used to access the user communication services of SharePoint. Third, custom activities are used to implement organization-specific semantics for forwarding and delegation in a standard and reusable way. The WF designer is used as a means to define the workflow and also provide diagrammatic representations of the state of a document review workflow instance to the workflow owner. Attacking the Problems In summary, the workflow platform supports an abstraction of the ideas that have made workflow products an attractive attack on business problems. It does not replace today's workflow products, however. Rather, it factors them into platform and superstructure. The workflow platform embodies two key ideas: a workflow is an organization of work units, and a workflow is a model, that is, a machine-readable description other than code. These ideas are valuable in a broad range of applications, both within and beyond the problem domain addressed by typical workflow products. Such a workflow platform is most useful if it is low cost and ubiquitous. The principal benefits delivered arise from the expression of an organization of work items as a model, which has several advantages over a representation in code:
Transparency. The business purposes of the system are clear, allowing users and IT staff to communicate effectively about the desired behavior and IT staff coming onto the project to get up to speed quickly. Isolation of change. The areas of the application most likely to change are expressed as workflow rather than code. By isolating the rapidly moving parts of the application, changes can be made more reliably. Agility. The bottom line of all these benefits is business agility. If business users can understand the system, developers can get up to speed quickly, and the risks associated with change are minimized. Then the system may be termed agile. A broadly useful workflow platform must have these characteristics: define a core workflow model as a standard that is extensible and fully programmable at design time and runtime, be able to communicate in a rich variety of ways, be lightweight and embeddable, and be able to scale and perform well in high-volume environments. WF is a product that displays all of these characteristics. As a component of .NET 3.0 and a part of the Windows platform, WF is also low cost and ubiquitous. A Workflow Manifesto The ―workflow manifesto‖ is a set of statements designed to help you consider the flexibility, value and benefits that workflow adds to your solutions architecture. The emergence of a flexible, extensible framework like WF enables the implementation of workflow-oriented solutions in ways that may have been impossible in the past. The ―workflow manifesto‖ was designed to help clarify some of the opportunities for applying and using workflow in our solution architectures: o Workflow is everywhere o Workflow is expressive o Workflow is fluid o Workflow is inclusive o Workflow is transparent While each of these statements provides a significantly different perspective of workflow, they all share two specific benefits in common: agility and abstraction. Before reviewing the manifesto statements in greater detail let‘s better understand the reason these two benefits are shared in such a broad fashion. Agility Agility is a key concept for both IT and business – a flexible and highly extensible solutions architecture is much more likely to meet the rapidly changing needs of the organization. Because they are model based, workflows are more suited for change than a pure-code approach. For
example, workflows typically enable developers to quickly and easily add new steps to a given workflow instance at runtime. Performing a similar function in a code-only approach requires developers to understand how to use a reflection API and manipulate the low-level internals of a given set of classes. This approach exposes the solution to risk and significantly limits the pool of maintenance developers qualified to work on it. Workflows also provide support for business rules that can be modified and re-deployed at run-time, impacting the behavior of one or more workflows and instances that may utilize the associated business rules. Note: While workflows are better suited for change than a pure code approach, workflows are not necessarily designed to replace all code. Workflows are nothing more than a set of specialized classes generated by the workflow modeling environment. Abstraction Abstraction is another key concept for IT and business, but for slightly different reasons. From an IT perspective, abstraction can be used to hide complexity, potentially reducing both development time and long-term maintenance costs. From a business perspective, the maintenance costs are attractive while the workflow model makes it easier for non-technical personnel to understand the objective of a given workflow without having to review the code used to implement the solution. The scope of the workflow model can also be established by a business analyst prior to handing it off to a developer for implementation. The concept of a model-driven approach is not new – developers have been using UML and other modeling techniques for many years prior to thSe maturation of workflow development tools. Workflow provides both sequential and state machine models that can be used to develop, manage and run the workflow itself. Developers usually prefer to work with the text (code) representation while a business analyst may well be overwhelmed by the detail of the model itself. Most workflow models also enable the model itself to be modified to meet the needs of the organization. For example, WF‘s presentation surface can be ―skinned‖ to change the look and feel of the modeling surface prior to embedding within your application. The Workflow Manifesto The remainder of this chapter examines the Workflow Manifesto statements in greater detail. Several examples of each topic are also included. Workflow is Everywhere Workflow doesn‘t just live in the datacenter on integration servers. Any sequence of steps or activities can be represented as a workflow. These steps may be handled by automated processes in a system-to-system (S2S) workflow, a human–to-human (H2H) workflow, or a fractal model that touches upon both (discussed below). Workflow can be used to orchestrate services across applications or within a single application, brokering user events or exposing workflow
capabilities to an end-user. Workflow may also be used within dedicated devices. For example, WF-based solutions can easily be deployed to x86-based XP-embedded devices such as a point- of-sale (POS) terminal or a bank ATM. There are several scenarios which may benefit from a workflow-oriented approach. The two scenarios we will examine are User Interface (UI) flows and creating and Web services. UI Flows A workflow model can serve as the controller within a Model-View-Controller (MVC) pattern. The MVC pattern provides a separation of objects into one of three categories — models for maintaining data, views for displaying all or a portion of the data, and controllers for handling events that may impact the model or views. There are three UI flow examples worth considering: o WinForms: A workflow can act as the event broker for common user events such as button clicks, data selection, and others. The state of a given workflow instance can also be used to configure the user interface. For example, Figure 5 shows how a workflow state is used to enable or disable buttons on a WinForm based on user interaction with a given workflow instance. (The code for this example is available in the WF Samples folder after downloading and installing the Windows SDK for .NET 3.0.) Figure 5: Workflow instances drive the WinForms user interface o Composite UI Application Block (CAB): CAB is an application block that provides a programming model for building smart client applications based upon the Composite Pattern. The Composite Pattern is applied to UI by combining multiple components to create complex user interfaces.
The individual UI components can be independently developed, tested, and deployed (conceptually similar to SharePoint‘s Web Parts model). The original implementation of CAB was developed before the advent of WF and required a significant amount of code for event brokering and management between the components that make up the interface (see Figure 6 for an overview of CAB). CAB is being re-architected to include WF for event brokering and UI component management (see the area highlighted in Figure 3). Figure 6: The Composite UI Application Block o Pageflow – A page flow is a directory of Web application files that work together to implement a UI experience. A server-side workflow implementation is used to control the order in which pages are served to the user, possibly changing the user experience on the fly based upon events raised by the current page. Navigation logic remains within the workflow, effectively isolating navigation from presentation. While there is no ―official‖ way to implement Pageflow using WF, the WF product team has made an example available at http://wf.netfx3.com/files/folders/sample_applications/entry10996.aspx. The example illustrates how one can implement a generic navigation framework with WF that is capable of supporting UI technologies like ASP.NET and WPF. Web services There are two approaches in which we can consider the relationship of workflows and Web services. A workflow can be used to orchestrate multiple Web services, potentially repurposing the return values from one Web service as input values to another. Using workflows to invoke Web services simplifies the process of working with asynchronous services since the invocation and correlation requirements can be handled by the workflow itself. We can also use workflows
to build Web services, publishing the workflow as a Web service. The model is fractal since we can use workflows to both create and consume (or orchestrate) Web services. Since most services were developed and designed to operate autonomously, bringing them together into a new service composition (workflow) requires that several challenges must be met. Figure 7 illustrates some of the most common challenges facing service orchestration. Figure 7: Capabilities for Service Composition Most integration servers (such as BizTalk Server) support the capabilities in Figure 7, relieving developers from having to write, test and maintain ―plumbing code‖ that provides little value to the organization. Figure 8 illustrates how a workflow can be both exposed as a service and orchestrate services. Figure 8: Workflow and Web services In .NET 3.0, WF includes three out of the box (OOB) activities for consuming or publishing simple ASMX Web services: o The InvokeWebService activity enables a workflow to consume a simple ASMX Web service. Adding an InvokeWebService activity to your workflow triggers the Add Web Reference wizard, enabling you to enter
the URI and proxy name of the service to be invoked. The MethodName dropdown property enables you to select which method on a given endpoint should be invoked. The InvokeWebService activity also includes optional parameter properties to support One-Way or Request- Response Message Exchange Patterns (MEPs). o A WebServiceReceive activity is typically added as the first element of a workflow to be published as a Web service. WebServiceReceive binds the activation of a workflow instance to a Web service invocation. The interface definition is automatically generated based upon the value of the WebServiceReceive activity‘s InterfaceName property. The InterfaceName property is a simple interface that defines the method names and signatures to be exposed by the Web service. The WebServiceReceive activity also includes optional parameter properties to support One-Way or Request-Response Message Exchange Patterns (MEPs). A workflow with a WebServiceReceive activity is published as an ASMX Web service by simply right-clicking on the project containing the workflow and selecting ―Publish Workflows as Services…‖ o A WebServiceResponse activity may be added to a workflow to be published as a Web service with a Request-Response MEP (there is no WebServiceResponse in a One-Way MEP). WebServiceResponse includes a ReceiveActitivityID property which is used to associate the response message with a previously defined WebServiceReceive activity. The NET 3.0 WF out-of-box activities do not provide support for Windows Communication Foundation (WCF). WCF and WF are more closely aligned In NET 3.5, enabling you to more easily expose workflows as WCF services and orchestrate WCF services. Using WF to consume or create Web services provides a far richer set of options than a ―pure-code‖ approach. WF enables developers to take advantage of various features such as service orchestration, asynchronous invocations, persisting/tracking staIn this section we illustrated how we can use workflows within user interfaces and how we can use workflows to create and consume Web services. Additional scenarios could include implementing a business logic layer within your applications, supporting long-running processes and exposing a set of configurable business rules that may be modified at runtime. Any sequence of steps or events can be represented as a workflow.
Workflow is Expressive Workflows can do virtually anything that a ―pure-code‖ approach can do, at a higher level of abstraction. This doesn‘t necessarily mean you should replace all of your code with workflows. What it does mean is that you can use a workflow-oriented approach to raise the level of abstraction in your solutions, potentially raising your level of productivity and reducing long term maintenance and ownership costs. Let‘s examine these assertions in greater detail. The model is the workflow Since the model is a direct representation of the actual workflow there is no loss of fidelity in the implementation of the model. This is a major change from older model-based technologies such as CASE and some UML-based development tools. These tools suffered from the fact that their modeling environments were far more expressive than their code generation capabilities. This meant that the models were, in fact, pictures because they were unable to generate the code necessary for implementing the solution. Moving between a workflow model and the underlying implementation code is a lossless process – the model and the code are different views of the same workflow. The model also makes communicating and maintaining the workflow much easier than a ―pure- code‖ approach. The model becomes part of the documentation of the workflow and can be shared with (or created by) a business analyst to illustrate the objective of the workflow. The model can also be used by maintenance programmers to understand and maintain the workflow without having to read through large sections of opaque, possibly undocumented code. Which Workflow Model to use when? Workflow helps us realize the fundamental difference between a picture and a model. A workflow typically includes some sort of model used to both develop and execute the workflow. With WF we have two models available – a sequential model and a state machine. Sequential models flow from top to bottom in a simple sequential fashion while a state machine model is used to trap various events that may arise during the execution of the workflow. If we review the state machine model in Figure 1 we see there are several states that a Purchase Order moves through while being processed (e.g. OrderOpened, OrderProcessed, OrderCompleted, and so on.). Each state may include one or more activities that fire based upon the context of the currently executing instance. WF supports two models for designing your workflows – Sequential and State Machine. The model you select should be determined by your solution requirements. o Sequential models are most effective when the workflow is in control, dictating both the activities and the order in which they are to be performed. Sequential models are most likely to be used in system-to-system (S2S) workflows. For example, an Order Validation process consists of a well-defined series of steps and operates on a highly structured set
of data (e.g. a Purchase Order). An Order Validation process may be best defined using a sequential workflow model. o State machine models are most effective when the workflow is reacting to events that are raised outside of the workflow itself. The workflow supplies a list of events that may occur while the work is being performed. The order in which these events initially occur is not controlled by the workflow – the workflow only needs to know what can be done next based upon the current state. State machine models are most likely to be used in human-to-human (H2H) workflows. For example, in a document routing and review process, a reviewer may optionally decide to make revisions to the document or route the document back to the original authors for a complete rewrite. Modeling these types of actions in a sequential model can be quite challenging. There is a third approach that some people call ―data driven.‖ A ―data driven‖ approach uses a set of business rules to react to data being processed by the workflow (business rules are created within WF using the Policy activity). A ―data driven‖ approach can be used with both sequential and state machine workflow models. Your solution may choose to use a combination of these workflow models. For example, a given state within a state machine may encapsulate a sequential workflow. The final choice is up to you and should ultimately be influenced by your solution requirements. Workflow is Fluid Because they are model-based, workflows can be modified, interrogated or otherwise manipulated at design time or run time. That is, both users and analysts participate in both the design and updates to the workflow. Items within the workflow can change frequently, such as the order of operations; method calls; and services, data, or applications to be integrated. For example in a mortgage solution, rates can change based on the market, the loan type, the lendee‘s history, or several other reasons. Changes may need to flow from design to deployment in a matter of minutes. This requires a different approach than classical compiled and distributed applications. Some workflow systems use a runtime repository of business applications and rules that require updating on a frequent basis. This approach enables workflow systems to rapidly respond to market and organizational conditions. Workflow flexibility can be maximized by judicious use of business rules and orchestrations. Business Rules One approach to fluidity is using business rules. Business rules govern conduct of business processes. Business processes tend to stay the same over time but the business rules used by these processes are highly volatile. Separating rules from application code enables users to invoke or change rules at run time, resulting in greater workflow flexibility. Business rules are best used for: o Discrete, point-in-time evaluations and calculations.
o Large no of permutations to encode in a control flow. o Fact-based inferencing where control flow cannot be predefined. Orchestrations Orchestrations are similar to business rules, but more formal. You would use them in place of business rules where formal workflows require: o Long-running semantics o Transactions/ Compensations o Messaging Orchestrations are also used when there is a recognized control-flow that needs to be rigorously managed for performance or scale and visibility and tracking of the state of the orchestration is critical. Business rules and orchestrations both enable workflows to be modified faster and better deal with changes in the organization. This flexibility lowers the costs typically associated with development, maintenance and ownership of a given process. Workflow is Inclusive Automation in a system-to-system (S2S) environment is fairly straightforward – processes tend to be fixed with a finite number of variations. The data processed within a S2S environment tends to be structured, representing well-known business messages such as a Purchase Order. Users, however, work in an ad-hoc manner and may use unstructured or semi-structured data. For example, a user may receive an email at home, convert it to a document, enter related data into a remote tracking system, and communicate these events by leaving someone a voice mail message. Computer systems are much more suited for structured, long-running asynchronous problems with limited flexibility. Workflow models can be used to describe with both human-to- human (H2H) and system-to-system (S2S) interactions. We can also use these models to describe system-to-human (S2H) and human-to-system (H2S) interactions. Translating such models into an executable representation can be challenging and usually results in a task- notification (S2H) and task-completion-response (H2S) interaction pattern. Workflow is Transparent Humans can be brought into process design right from the start by the abstraction and model- driven design workflow offers. User interfaces like SharePoint Designer let an experienced end user build or modify workflows quickly and easily. A workflow model makes the workflow easier to understand without having to review and maintain opaque blocks of code. Tools such as WFPad (http://www.schmidt6.com/blogfiles/WF/WFPad.exe ) can be used to display the workflow model in one windows and the workflow code in another. This enables experienced users and developers to view or maintain a workflow using either a model or the code. As stated we stated earlier, the workflow is the model. The workflow model offers:
o state management across service interactions o service aggregation and orchestration o transparent implementation of services o visibility into the state of the service o configurable runtime behavior of: tracking, persistence, transactions, threading, and so on. Workflow transparency is important because state persistence is a critical requirement for long running processes. Processes may last for days, weeks, months or even years. Processes may need to be started, held, and restarted by humans or runtime services – sometimes with different timeframes with no loss of fidelity. Understanding the Relationship between BizTalk Server and WF In this chapter we made numerous references to Windows Workflow Foundation (WF). When WF was first announced many people incorrectly assumed WF was a replacement for BizTalk Server (BTS). WF and BTS are complementary technologies designed to serve two very different needs: BTS is a licensed product designed to implement workflow (―orchestrations‖) across disparate applications. WF is a developer framework designed to expose workflow capabilities within your application. There are no fees or licensing restrictions associated with using or deploying WF. Use BizTalk Server if… You need to implement system-to-system (S2S) workflows across disparate applications or platforms. You need an enterprise-strength Business Process Management (BPM) suite that enables complex transformations, support for popular application/wire-level protocols and integration with line-of-business (LOB) systems like SAP, PeopleSoft, Oracle, and JD Edwards. You need to interface with a human-to-human (H2H) workflow. You need to take advantage of Business Activity Monitoring (BAM). You need to map authentication information between Windows and non-Windows systems (Single Sign-On (SSO)) You need to set up and manage trading partners in a B2B scenario.
You need a complete set of tools for managing the infrastructure and scalability of your solutions architecture. Use Windows Workflow Foundation if… You need to expose workflow capabilities to end users through your application Your application only needs a message or event broker. You need to build a Human to Human (H2H) workflow capable of interfacing with a System to System (S2S) workflow. (SharePoint 2007 includes several predefined Human Workflows built with WF). You only need workflow or simple business rules and are not interested in the other features that BizTalk Server provides.
Conclusion In this chapter we examined the concept of workflow and discussed other terms and concepts frequently associated with workflow. Workflow is the organization of work using activities that represent both people and software. Models represent this organization, providing a higher level of abstraction and flexibility to our solutions. This chapter also clarified the relationship of workflow-oriented capabilities on the Microsoft platform (Windows Workflow Foundation and BizTalk Server). Chapter Four provides a more detailed discussion of the Data recurring architectural capability.