Application Integration Criteria

What makes for good application integration?
If integration needs were always the same, there would only be one integration style. Yet like any complex technological effort, application integration involves a range of considerations and consequences that should be taken into account for any integration opportunity.

The first criterion is application integration itself. If you can develop a single, stand-alone application that doesn’t need to collaborate with any other applications, you can avoid the whole integration issue entirely. Realistically, though, even a simple enterprise has multiple applications, applications that need to work together to provide a unified experience for the enterprise’s employees, partners, and customers.

The other main decision criteria are:

Application coupling — Even integrated applications should minimize their dependencies on each other so that each can evolve without causing problems for the others. Tightly coupled applications make numerous assumptions about how the other applications work; when the applications change and break those assumptions, the integration breaks. The interface for integrating applications should be specific enough to implement useful functionality, but general enough to allow that implementation to change as needed.

Integration simplicity — When integrating an application into an enterprise, developers should strive to minimize changing the application and minimize the amount of integration code needed. Yet changes and new code will usually be necessary to provide good integration functionality, and the approaches with the least impact on the application may not provide the best integration into the enterprise.

Integration technology — Different integration techniques require varying amounts of specialized software and hardware. These special tools can be expensive, can lead to vendor lock-in, and increase the burden on developers to understand how to use the tools to integrate applications.

Data format — Integrated applications must agree on the format of the data they exchange, or must have an intermediate traslator to unify applications that insist on different data formats. A related issue is data format evolution and extensibility—how the format can change over time and how that will affect the applications.

Data timeliness — Integration should minimize the length of time between when one application decides to share some data and other applications have that data. Data should be exchanged frequently in small chunks, rather than waiting to exchange a large set of unrelated items. Applications should be informed as soon as shared data is ready for consumption. Latency in data sharing has to be factored into the integration design; the longer sharing can take, the more opportunity for shared data to become stale, and the more complex integration becomes.

Data or functionality — Integrated applications may not want to simply share data, they may wish to share functionality such that each application can invoke the functionality in the others. Invoking functionality remotely can be difficult to achieve, and even though it may seem the same as invoking local functionality, it works quite differently, with significant consequences for how well the integration works.

Asynchronicity —Computer processing is typically synchronous, such that a procedure waits while its subprocedure executes. It’s a given that the subprocedure is available when the procedure wants to invoke it. However, a procedure may not want to wait for the subprocedure to execute; it may want to invoke the subprocedure asynchronously, starting the subprocedure but then letting it execute in the background. This is especially true of integrated applications, where the remote application may not be running or the network may be unavailable—the source application may wish to simply make shared data available or log a request for a subprocedure call, but then go on to other work confident that the remote application will act sometime later.

EAI Best Practices

This Post provides some of the Important Industry Best Practices on EAI. Best Practices are building blocks of an EAI.Best Practices will guide for implementing better Integration Solution. They offer practical advice for the EAI practitioner.


Best Practice-1 Build decoupled and distributed Architecture
Plan a head before building any Integration Architecture.Architecture should be decoupled for flexibility purpose as well it’s distributed. Components should be service oriented as well as Event driven.

• Identify Synchronous/Asynchronous Operations
• use Appropriately in the EAI Framework

Best Practice-2 Identify Common Services and Start building
Identify commonly used Services like (Error-Handling, Event Tracking, Central Logging) in the Component oriented Architecture and plan for building these components for sharing across the Enterprise with in the EAI Domain. Business processes will be used these services as central back bone services in the EAI Hub.

Best Practice-3 Plan for Common Objects
Plan for Common Objects as Global Business Objects that can use across Applications in the Integration space for that Applications can communicate easily without enforcing any Application specific objects in the EAI Hub.

Best Practice-4 Create a common vocabulary
Planning requires the creation of common vocabularies (i.e. taxonomy) to ensure proper understanding and consideration of how to manage, change and use taxonomies. A common vocabulary also facilitates the collaboration and sharing of information across different business areas. Further, as an enterprise defines a common vocabulary, the enterprise should also develop related vocabulary governance.

With both, vocabulary mapping between business areas or communities of interest (for example, relating tracks to targets or installations to supply points or units to weapons systems) becomes much more manageable.

Best Practice-5 Design for connections, change, and control
The enterprise should build a topology of services that reflects the business processes, not the systems, thereby giving the enterprise the ability to make changes. Organization leaders should think in terms of how the enterprise business model and EAI will operate in the context of both the business as well as within the IT infrastructure.

Best Practice-6 Build Templates for Business Processes and Human Workflows.
Build Templates a head for Business Processes and Workflows that include proper logging and Error Handling and common services for that developers follow certain standards for developing any business process across Enterprise while Integration
These usage of these templates should be documented in Developer’s GUIDE for detail reference.

Best Practice-7 Build standards for Messaging and Business Processes
Build Standards for Messaging and Business Processes a head and include those in developer’s guide for reference.

Best Practice-8 Plan a head for Testing Integration Applications
Build Standards for Messaging and Business Processes a head and include those in developer’s guide for reference.

Best Practice-8 Plan a head for Testing Integration Applications
Plan a head for Testing Integration Applications in different Environments like Development, Test and Performance.
• Plan to load production data into Test Env.
• Plan for Initial Load
• Plan for loading any cross-reference data that applications are using in the Integration Env.
• Plan for Integration as well as System Tests on Test Env.

Best Practice-9 Build standards for Messaging and Business Processes
Build Standards for Messaging and Business Processes a head and include those in developer’s guide for reference.

Best Practice-10 Building with Deployment in Mind
It is vital to keep the end goal in mind--deployment. In a sense, the EAI isn't real to the rest of the company until a solution is deployed. Yet, often, key deployment issues are ignored in the design and construction phases of the project. Here are four ways to actively address deployment in the solution:
Instrument components for monitoring: Primary system components should be instrumented for monitoring. These include the ability to provide information on the availability of the system as well as other metrics such as CPU utilization, memory utilization or open resource connections. This information is vital for systems administrators to tune the system for optimal performance. It also affords the development team detailed knowledge of how the system performs in a production scenario.
Account for installations and updates. Any deployed solution must account not only for the initial installation but also for ongoing updates to system components. In some deployments, updates to integration rules and components such as adapters may prove to be a regular activity. The ability to install and update without the loss of continuous operations is essential once the system is in production.
Establish service level agreements. Service level agreements (SLAs) should be established and articulated well before deployment. In fact, they should be articulated as part of the requirements process. The SLA defines the system's requirements at run-time with regard to availability, resource consumption and system performance characteristics.
Build a deployment plan upfront. Building a deployment plan before a single line of code is written is a way to ensure that deployment issues surface earlier rather than later. A deployment plan should walk through how each phase of deployment will be enacted. It needs to articulate all the necessary activities required to support deployment and ongoing maintenance of the solution. It should include coverage of installation, documentation, technical training and the timing of the deployment.

Best Practice-11 Plan for incremental transformation and deployment.
EAI is an iterative process requiring incremental transformation and deployment plans. The key point is that Organizations should start with a simple first step, automating the data collection process.

Best Practice-12 Profile Performance Early and Often
Profiling system performance should be a practice conducted at key points throughout the development process. Measuring and analyzing system performance carries two primary benefits.
First, it is far less costly to correct performance issues and bottlenecks early in the development life cycle. This is particularly true if the performance issues are related to a design flaw. Addressing fundamental design issues during or after the initial deployment is very expensive. Second, discovering the root of a performance problem after the entire system has been constructed can be extremely challenging from a technical perspective. You are often limited by the scope of what can be observed, and this in turn is a result of the tracing and auditing facilities built into the system. It can also be influenced by many more variables, such as network traffic. Instead, by testing not only the individual components as they are constructed but also the baseline architecture, you can establish a more granular view of system performance. Discovering and resolving performance limitations early is simply easier and leads to better results at the end of the project.

Best Practice-13 Build a Traceable System
One overlooked aspect of EAI is the challenge of run-time debugging. Detecting problems in distributed integration architectures is extremely tricky and requires that the system be instrumented with tracing code. Tracing code allows you to track the progress of data and the execution of code segments as that data is processed. Often, in an effort to meet the project schedule, the time allotted to building in tracing code is eliminated in preference for functionality. Whether you are constructing your own EAI system from scratch or leveraging a product to implement an EAI solution, you need to ensure that the system is traceable. The reality is that no matter how much testing is conducted in the lab environment, some problems inevitably arise only in production scenarios. The introduction of "live" data coupled with production application usage often results in extending the system in ways that are not possible to duplicate in the test lab. When such problems are discovered at run-time--or sometimes even when production deployment is under way--it becomes difficult to fix them without being able to trace the flow and execution of the EAI solution.
Also bear in mind that levels of tractability need to be defined, since tracing can hurt overall system performance. It helps to set a low tracing option initially as you debug the system.

Best Practice-14 Reviewing and Addressing Secondary Scenarios
Secondary scenarios arise when the flow of data is not completed within the expected course of execution. For Example as shown below:

The process is loading AP/GL Data from PAL to Oracle Financials through EAI Hub.
However, what if the database in Oracle Financials is busy processing other operations and the Load AP/GL data times out? How does the system handle a failed retrieval or a returned error? This is an obvious but common secondary scenario. Another scenario may occur if, in the course of executing the rules in the Integration Process, if Integration server goes down. How does the system deal with an integration flow halted in mid process as a result of failure at one point? The number of secondary scenarios for a given system is seemingly infinite. Practically speaking, it is not possible to address every one of them. However, a regular practice of conducting reviews of possible secondary scenarios--documenting and subsequently addressing them--must be an essential practice. An EAI solution that addresses secondary scenarios will adhere to certain principles of behavior.
• Message preservation. The system should never lose data in the event of a system failure.
• Data integrity. In the event of a system failure, the system should always preserve the integrity of the message.
• Predefined exception handling. Failed operations must have a predefined course of execution.
• Graceful exits. Operations that fail because of unavailable resources or errors should always "exit gracefully" and return to a known state. Failure should never leave the data flow in an undetermined state whereby data is not reliable and the corrective course of action is unknown.

SOA vs EAI vs ESB

This post aims to bring clarity to terms EAI, ESB, SOA and provide a clear distinction.

SOA : Service Oriented Architecture
Service oriented architecture is approach to have software resources in an enterprise
available and discoverable on network as well defined services. Each service would achieve a predefined business objective

and perform discrete units of work. The services are independent and do not depend on the context or state of the other

services. They work within distributed systems architecture.
I read a definition which is as follows :
"SOA is a loosely-coupled architecture designed to meet the business needs of the organization using services."

and in another definition some one says that

A Service Oriented Architecture is a structural style to establish information needs of a organization based on services.

See Something confuses here.In both the definition it is said that SOA is a style - loosely-coupled (which is a characteristic) versus structural style (architectural style would be better). There is difference here in business needs versus information needs, which is not quite the same. Business need can be having insight in how business is running in total or being flexible (agile) in running business processes. Information need can also be insight in how business is run in total and having services to provide this.

Now read this definition
"SOA is an enterprise architecture style, not an application architecture style." by Anne Thomas Manes from Burton group.

SOA is an architecture style, definitions above are right. and the reality is SOA is an architecture la style, where services play an important role, hence they are the building blocks to create a SOA.

Earlier SOA used COM or ORB based on CORBA specifications and recent SOA stress on web services using standard description (WSDL), discovery (UDDI) and messaging (SOAP). Service oriented architecture may or may not use web services but yes web services provide a simple way towards service oriented architecture albeit with the age old security and reliability limitations.

EAI : Enterprise Application Integration
Enterprise application integration is a business need to make diverse applications in an enterprise including partner systems to communicate to each other to achieve a business objective in a seamless reliable fashion irrespective of platform and geographical location of these applications. It is a business need and business never dies it only evolves.

I have seen people saying that EAI is a thing of past now SOA is here, it is just like saying "Transportation is a thing of past now road is here"

EAI comprises of message acceptance, transformation, translation, routing, message
delivery and business process management. Usually messages transportation is asynchronous but for a business need it can be synchronous as well. There are two basic architectures to achieve this, bus and hub/spoke architecture. Both of these can be used to develop services and then it also becomes service orientated architecture.

• HUB/SPOKE
  Hub/Spoke architecture uses a centralized broker (Hub) and adapters (Spoke) which
  connect applications to Hub. Spoke connect to application and convert application data format to a format which Hub understands and vice versa. Hub on the other hand brokers all messages and takes care of content transformation/translation of the incoming message into a format the destination system understands and routing the message. Adapters take data from source application and publish messages to the message broker, which, in turn, does transformation/translation/routing and passes messages to subscribing adapter which sends it to destination application(s). Having a single Hub makes system with this architecture easy to manage but scalability takes a hit. At some point of time as number of messages increase, scalability gets dependent on hardware. Having a bigger box to scale application has never been an ideal solution so to overcome this limitation most vendors have incorporated the concept of federated hub and spoke architecture in which multiple hubs can be present, each hub would have local metadata and rules as well as global metadata. Changes to global rules and metadata are automatically propagated to other hubs. Federated hub spoke architecture alleviates scalability issue while central management of multiple hubs makes this architecture easy to manage and brings down support cost.
  
• BUS
  Bus architecture uses a central messaging backbone (bus) for message propagation.
  Applications would publish messages to bus using adapters. These messages would flow to subscribing applications using message bus. Subscribing applications will have adapters which would take message from bus and transform the message into a format required for the application. Key difference between hub/spoke and bus topology is that for the bus architecture, the integration engine that performs message transformation and routing is distributed in the application adapters and bus architecture requires an application adapter to run on the same platform as the original applications. Since adapters have integration engine and run on same platform on which source and target applications run, this scales much better and is
  complex to maintain compared to hub/spoke topology.
  

ESB : Enterprise Service Bus
Enterprise service bus is an infrastructure to facilitate SOA. It gives API which can be used to develop services and makes services interact with each other reliably. Technically ESB is a messaging backbone which does protocol conversion, message
format transformation, routing, accept and deliver messages from various services and application which are linked to ESB. Current EAI landscape is seeing many vendors who offer enterprise service bus and claim it to be a brand new concept. This brings a question on what exactly is the difference between ESB and the bus based implementations which have been there in market for quite a long time now. Actually there is not much difference between ESB and proprietary buses except for a few subtle ones. Main difference between ESB and proprietary bus implementation is of cost which is significantly low for ESB.

Reason for this cost difference is two fold, first proprietary bus offers lot of built in functionalities as a suit of product which need to be developed for ESB implementations based on business requirement, second most proprietary buses use
some proprietary formats to enhance the performance and that increases the cost. ESB on the other hand is usually standard based, so it is a tradeoff between performance and cost between proprietary bus and ESB. Main advantage of ESB is that it
costs much less then hub/spoke or bus based product suits and that it is standard based.
Conclusion
SOA brings cost effective, reusable and low lead time solutions to an organization but EAI and SOA are both going to coexist. Web services alone as SOA can not handle the complex, secure and SLA based applications of an enterprise currently and unless we see a technological break through it is going to remain that way.

Enterprise service bus would enable low cost integration and would be used by companies with limited IT resources and environments that involve a handful of systems and moderate transaction volumes. Packaged EAI solutions would have SOA as basic tenet and would continue to be used for large scale integration by companies having huge number of diverse system and high transaction volumes. Next generation EAI solutions would use more and more of SOA to provide reliable, secure, low cost and flexible solutions.

EAI - Business Case

PepsiCo is a world leader in convenient foods and beverages, with 2006 revenues of more than $35 billion and 168,000 employees.

The company consists of Frito-Lay North America, PepsiCo Beverages North America, PepsiCo International and Quaker Foods North America. PepsiCo brands are available in nearly 200 countries and territories and generate sales at the retail level of about $92 billion.

PepsiCo's QTG division consists of the Quaker, Tropicana and Gatorade businesses. QTG's powerhouse brands include Tropicana Pure Premium, Gatorade Thirst Quencher, Propel, Quaker Oatmeal, Rice-a-Roni, Cap'n Crunch and Life cereals.

Some of PepsiCo's brand names are more than 100-years-old, but the corporation is relatively young. PepsiCo was founded in 1965 through the merger of Pepsi-Cola and Frito-Lay. Tropicana was acquired in 1998 and PepsiCo merged with The Quaker Oats Company, including Gatorade, in 2001.

Pepsico OneUp
The PepsiCo One Up Project is to collaborate the whole business in SAP System from the legacy systems. Five Companies Pepsi Cola , Frito lays , Tropicana , Gatorade and Quaker have been merged together to form today’s PepsiCo. This PepsiCo OneUp Project is implementing an integrated technology to empower the real time data and tools access.

Need of EAI in Pesico
PepsiCo understands that, to survive the competition in today's fast paced economy and to reduce its time-to-market, it needs to put an integration solution in place as soon as possible. The integration model should integrate all the existing legacy systems irrespective of the platforms and operating systems that these systems are based upon. The model should take care of the various data formats that would pass through this integration scenario and apply the necessary transformation, translation, and data validation rules upon them. And last, but definitely not the least, this integration solution should be scalable enough such that any new system could be easily plugged into this solution in the future with minimal changes.

The integration model should provide:

• Integration between the various legacy systems
  
• Automated process for procurement of goods
  
• Data transformation and translation logic according to the prescribed rules
  
• Persistent storage mechanism for the data
  
• Validation of business data to a certain extent
  
• Connectivity to various drop zones and databases
  

See the following picture :


As-is scenario, these legacy systems are acting as independent data silos with absolutely no kind of data-sharing happening between them. If the North America SAP system wants to procure something from any of its vendors, it does so by using some kind of manual process at its end. The same scenario is replicated for all the other systems, too. And, this is leading to a lot of data duplication taking place. For example, if two different systems are using the same vendor (say, Vendor A) for procuring beverages, this vendor (Vendor A) information is getting stored at both the legacy systems. Also, this kind of scenario makes it impossible to leverage existing information across multiple systems.

Project Life Cycle
Design and Architecture

The EAI solution proposed would make use of the integration capabilities of the Seebeyond eGate software to seamlessly integrate the various individual legacy systems of the Pepsico company with the SAP-based purchasing order (PO) system.

TIBCO provides the core integration platform, comprehensive systems connectivity, guaranteed messaging, and robust transformation capabilities.




In this scenario, the SAP-based PO system acts as a data hub for all the legacy systems such that all the procurement orders at PepsiCo would be routed through this SAP based PO system. Also, TIBCO is acting as the integration hub between the SAP-based PO system on one end and the legacy systems at the PepsiCo company on the other, thereby enabling a bi-directional flow of data.

EAI - An Introduction

Enterprise Application Integration or EAI for short is integration technology. EAI is a buzzword that represents the task of integration of various applications so that they may share information and processes freely.



Today's business world is infinitely complex than it was a long time ago. Modern companies have a large number of applications that take care of running the business. Such diverse applications weren't a problem initially because they were meant to provide stand-alone, independent, and automated functions. The result of this diversity was a collection of stovepipe applications rather than a unified network of linked systems. But now, the companies are realizing the utmost need to integrate these independent data silos to leverage the information stored in them across the various vertical and horizontal domains as well as surmount the ever-increasing costs of building new applications.

So here Enterprise Application Integration comes into the picture.

EAI is a collection of processes, software and hardware tools, methodologies, and technologies. When implemented together, they have the aim of consolidating, connecting, and organizing all the businesses computer applications, data, and business processes (both legacy and new) into a seamlessly interfaced framework of system components that allow real-time exchange, management, and easy reformulation of the company's mission-critical information and knowledge. It is an unrestricted sharing of data throughout the networked applications or data sources in an enterprise.



When designing an Enterprise Application Integration (EAI) solution, it is important to recognize that there are different levels of integration, each with its own requirements and considerations. Successful implementation of consistent, scalable, reliable, incremental, cost-effective EAI solutions depends on the standards and methodologies that we define for these levels. It must be determined how we need to share information:

• Within an application
  
• Between applications within an enterprise
  
• Between enterprises
  
• Directly with customers
  

This architecture provides sufficient high-level consistency for interoperability and a certain degree of local freedom. For example, the architecture supports semantic diversity (different interpretations of the same data) and permits use of diverse technical tools and techniques. It also provides the basis for organizational responsibility and ownership of each layer. For example, business data persists at the application component layer and not in the middleware/EAI layer. Hence, this data is the responsibility of the application owners and not the middleware solution developers.