Understanding Compute Grid applications

Compute Grid is a component of WebSphere Extended Deployment Version 6.1 that provides a complete enterprise Java batch programming solution. It includes a concise, yet powerful, Plain Old Java Object (POJO) based programming model, simple packaging and deployment model, full-feature job control language, sophisticated job scheduler, robust execution environment, and comprehensive workload management and administrative tools. These all combine to make WebSphere Extended Deployment Compute Grid the most complete Java batch solution available today.

As the following image shows, a batch job is a series of program executed in the background under control of a job manager. Each program consumes zero, one or more input data sources and produces zero, one or more output data sinks. The z/OS operating system is an example of a mature batch job processing environment. WebSphere Extended Deployment Compute Grid generalizes this processing model and make it available in a WebSphere/Java 2 Enterprise Edition (J2EE ) environment.

A WebSphere Extended Deployment Compute Grid application is a POJO class that implements a batch interface and is invoked by the WebSphere Extended Deployment batch container. It has access to container services that expose batch data streams (BDS), which represent data sources and sinks. Typically, the purpose of such an application is to perform bulk operations on a large set of records. A job definition describes a series of steps, where each step invokes a batch POJO. The job definition describes both the order of step execution and the batch data streams that are inputs/outputs and are used by each step.

A Compute Grid job executes inside a WebSphere Application Server in a special container called the batch container, as shown in the following image:

xJCL file - job definition

The job definition is described by a special xJCL file, which is an XML-based job control language. An xJCL file is submitted to the WebSphere Extended Deployment job scheduler, which then dispatches the job represented by that xJCL file to a WebSphere Application Server that hosts the POJOs that comprise the target application. Inside the WebSphere Application Server, the batch container creates a managed thread, called an asynchronous bean, to process the job. The asynchronous bean processes the job by invoking the POJO identified by each job step, starting with the first step , then the second, as so on. The POJO invoked by each step has access to the input and output batch data streams described by the job definition (the xJCL file).

The POJO typically processes many input records and produces many output records. Since the POJO executes inside the WebSphere Application Server, it has full access to both the J2EE programming model, as well as to all WebSphere Application Server services, such as transaction and connection management. It can invoke other Web and J2EE services locally for efficiency.

External scheduler integration

Since an external scheduler does not know how to directly manage WebSphere Extended Deployment jobs, a proxy model is used. The proxy model uses a regular JCL job to submit and/or monitor the WebSphere Extended Deployment job. The JCL job step invokes a special program provided by WebSphere Extended Deployment, named WSGRID. The WSGRID application submits and monitors a specified WebSphere Extended Deployment job. WSGRID writes intermediary results of the job into the JCL job’s joblog. WSGRID does not return until the underlying job is complete, thereby providing a synchronous execution model. Since the external scheduler can manage JCL jobs, it can manage a JCL job that invokes WSGRID. Using this pattern, the external scheduler can indirectly manage a job. An optional plugin interface in the job scheduler enables a user to add code that updates the external scheduler operation plan to reflect the unique state of the underlying job, such as job started, step started, step ended, job ended. The WSGRID program is written with special recovery processing so that if the JCL job is cancelled, the underlying job is cancelled also, thus ensuring synchronized lifecycle of the two jobs.

 The following diagram shows the job control by an external workload scheduler for the z/OS platform environment. In this diagram, the Tivoli Workload Scheduler (TWS) is shown as an example workload scheduler.

 The job control for distributed platforms is similar, as shown in the following diagram, with the exception that the Job Entry Subsystem (JES) is not required for the distributed platforms.

Many customers already use TWS to manage batch workloads on the z/OS platform. While Java batch executed inside a WebSphere Application Server environment is attractive, a way to control the WebSphere Extended Deployment Compute Grid jobs through TWS is needed. Since TWS does not know how to directly manage WebSphere Extended Deployment jobs, a proxy model is used. The proxy model uses a regular JCL job to submit/monitor the WebSphere Extended Deployment job. The JCL job step invokes a special program provided by WebSphere Extended Deployment, called the WSGrid utility. See the .

The WSGrid application submits and monitors a specified WebSphere Extended Deployment job. WSGrid writes intermediary results of the WebSphere Extended Deployment job into the JCL job’s job log. WSGrid does not return until the underlying WebSphere Extended Deployment job is complete, thereby providing a synchronous execution model. Since TWS can manage JCL jobs, it can manage a JCL job that invokes WSGrid. Using this pattern, TWS can indirectly manage an WebSphere Extended Deployment job.

An optional plugin interface in the WebSphere Extended Deployment job scheduler enables a user to add code that updates the TWS operation plan to reflect the unique state of the underlying WebSphere Extended Deployment job, such as job started, step started, step ended, job ended. The WSGrid program is written with special recovery processing so that if the JCL job is cancelled, the underlying WebSphere Extended Deployment job is cancelled also, thus ensuring synchronized lifecycle of the two jobs.