Difference between revisions of "COSMOS Design 197867"

Revision as of 13:04, 22 August 2007

Data Broker for COSMOS DC framework

This design document addresses COSMOS Bugzilla enhancement request 197867.

Change History:

Name: Date: Revised Sections:

Jimmy Mohsin 7/25/2007 Initial version

Bill Muldoon 8/1/2007 Added content

Bill Muldoon 8/17/2007 Updated content

Jimmy Mohsin 8/17/2007 Please see my comments on the Talk page Talk for 197867

Hubert Leung 8/21/2007 Updated broker design as discussed in meeting on Aug 20: http://wiki.eclipse.org/COSMOSBrokerSpecification. Need to consolidate the new design with this deisgn doc.

Bill Muldoon 8/22/2007 Consolidated and reorganized the design

Overview

The Broker is the component in the COSMOS data collection framework that holds a registry of Data Managers (ie: Management Data Repositories (MDRs)). The registry stores the addresses of the Data Managers in the form of end point references, indexed according to the data classification and the data source types of the data. Each Data Manager provides the data classification and data source type information upon registration with the Broker and maintains the relevant registry data over its life cycle. The Broker provides interfaces for clients to make queries for the classifications and data source types available in the registry and for addresses of Data Managers based on classifications and/or data source types. The interfaces are provided in the form of WSDM capabilities and Java interfaces.

The “Broker” component is a logically centralized component in the COSMOS data collection framework. In practice, there can be multiple brokers, each responsible for a different group of management resources, for example, the distinction between data brokers and service brokers. However, the role and responsibilities of different kinds of brokers are the same.

In the case where there are multiple Brokers, there is a higher level lookup registry (the Management Domain) to find the address of the Brokers. The Management Domain can be considered a "Broker of Brokers".

Implementation Stages and Corporate Use Cases

The Broker will evolve over time to accommodate the needs of the corporate use cases as follows:

Version 1 will satisfy the initial CA use cases.

Version 2 will support the data source types requirements of the IBM use cases.

Version 3 will support the CMDBf use cases from CA and IBM.

Note that Compuware has no use cases which require the Broker at this time.

Terminologies/Acronyms

The terminologies/acronyms below are commonly used throughout this document. The list below defines each term:

Term Definition

 Data Broker	This is the component where all the web services that share data register themselves
 Data Store	This is a physical software artifact that stores data, e.g. Oracle, a File System, etc
 Data Manager	Largely corporate implementation component that implements the SOA API for data exchange. A Data Manager can be an MDR.
 Service Broker	This is the component where all the web services that share functional behavior register themselves
 Management Domain    A "Broker of Brokers"

Broker Use Cases

1. Broker initialization

Broker undergoes initialization: Loops through the rows in the registry, ping each Data Manager for their status. If a Data Manager is reachable, set its state to "online"; otherwise, set it to "offline".

Broker waits for requests

2. Data Manager (DM) initialization

DM contacts the Broker via well-known (pre-configured) EPR. DM may also get the Broker address from the Management Domain.

DM invokes the "registration" operation of the Broker, providing the following information:

• EPR for the DM
• name of the DM
• classification of the DM
• dialect of the DM
• optional data source type identifiers

Broker populates tables in registry database with the information about the DM

Broker changes the state to online and updates the timestamp

If the DM is already registered with the Broker, then the DM invokes the "ping" operation, which updates the timestamp in the Broker registry

3. Data Manager shutdown

Data Manager invokes the deregister operation of the Broker, providing the name and EPR as a parameter

Broker removes the registry entry for the Data Manager

4. Client queries Broker for a Data Manager

Client invokes the getDataManagers operation of the Broker, providing the classification and optional data source types

Broker searches its registry and returns the Data Managers that match the classification and data source types

Data Broker External Interfaces

Data Broker Capabilities

The Data Broker exposes this capability definition:

public interface DataBrokerCapability {
	
	/**
	 * Get the Data Managers
	 * 
	 * @param String classification
	 * @return IDataQueryResult containing an XML Element object
	 */
	@ManagedOperation
	public IDataQueryResult getDataManagers( String classification ) throws Exception;

	/**
	 * register the Data Manager
	 * 
	 * @param String endpointAddress
	 * @param String endpointResourceId
	 * @param String dataManagerName
	 * @param String classification
	 * @param String dialect
	 */
	@ManagedOperation
	public boolean registerDataManager( String endpointAddress, String endpointResourceId, 
                            String dataManagerName, String  classification, String dialect ) throws Exception

	/**
	 * deregister the Data Manager
	 * 
	 * @param String dataManagerName
	 */
	@ManagedOperation
	public boolean deregisterDataManager( String dataManagerName) throws Exception;

	/**
	 * ping the Data Broker
	 * 
	 * @param String dataManagerName
	 */
	@ManagedOperation
	public boolean pingDataBroker( String dataManagerName) throws Exception;

Data Broker WSDL

The DC runtime converts the capability into a WSDM Endpoint with operations that correspond to the methods. For example, the WSDL contains:

 <wsdl:operation name="getDataManagers">
   <wsdl:input name="getDataManagersRequest" wsa:Action="http://www.eclipse.org/xmlns/cosmos/1.0/DataBrokerCapability/getDataManagers"
               message="dyn:getDataManagersRequest"/>
   <wsdl:output name="getDataManagersResponse" wsa:Action="http://www.eclipse.org/xmlns/cosmos/1.0/DataBrokerCapability/getDataManagersResponse"
                message="dyn:getDataManagersResponse"/>
     <wsdl:fault name="ResourceUnknownFault" message="dyn:ResourceUnknownFault"/>
     <wsdl:fault name="ResourceUnavailableFault" message="dyn:ResourceUnavailableFault"/>
   </wsdl:operation>
 <wsdl:operation name="registerDataManager">
   <wsdl:input name="registerDataManagerRequest" wsa:Action="http://www.eclipse.org/xmlns/cosmos/1.0/DataBrokerCapability/registerDataManager"
               message="dyn:registerDataManagerRequest"/>
   <wsdl:output name="registerDataManagerResponse" wsa:Action="http://www.eclipse.org/xmlns/cosmos/1.0/DataBrokerCapability/registerDataManagerResponse" 
                message="dyn:registerDataManagerResponse"/>
     <wsdl:fault name="ResourceUnknownFault" message="dyn:ResourceUnknownFault"/>
     <wsdl:fault name="ResourceUnavailableFault" message="dyn:ResourceUnavailableFault"/>
   </wsdl:operation>
 <wsdl:operation name="deregisterDataManager">
   <wsdl:input name="deregisterDataManagerRequest" wsa:Action="http://www.eclipse.org/xmlns/cosmos/1.0/DataBrokerCapability/deregisterDataManager" 
               message="dyn:deregisterDataManagerRequest"/>
   <wsdl:output name="deregisterDataManagerResponse" wsa:Action="http://www.eclipse.org/xmlns/cosmos/1.0/DataBrokerCapability/deregisterDataManagerResponse" 
                message="dyn:deregisterDataManagerResponse"/>
     <wsdl:fault name="ResourceUnknownFault" message="dyn:ResourceUnknownFault"/>
     <wsdl:fault name="ResourceUnavailableFault" message="dyn:ResourceUnavailableFault"/>
   </wsdl:operation>
   <wsdl:operation name="pingDataBroker">
     <wsdl:input name="pingDataBrokerRequest" wsa:Action="http://www.eclipse.org/xmlns/cosmos/1.0/DataBrokerCapability/pingDataBroker" 
                 message="dyn:pingDataBrokerRequest"/>
     <wsdl:output name="pingDataBrokerResponse" wsa:Action="http://www.eclipse.org/xmlns/cosmos/1.0/DataBrokerCapability/pingDataBrokerResponse" 
                  message="dyn:pingDataBrokerResponse"/>
     <wsdl:fault name="ResourceUnknownFault" message="dyn:ResourceUnknownFault"/>
     <wsdl:fault name="ResourceUnavailableFault" message="dyn:ResourceUnavailableFault"/>
   </wsdl:operation>

Client application can invoke the Broker operation using the WSDM endpoint operations.

Data Broker Client API

Alternatively, client application can use the Broker Client API to invoke the Broker operations. The Broker Client API contains a Java class that provides a convenience mechanism to invoke the Broker WSDM endpoint operations.

Broker Client API class:

public class DataBrokerClient extends WsResourceClient{

	/** 
	 * get the Data Managers 
	 *
	 * @param String classification
	 */
	
	public Element getDataManagers( String classification ) throws Exception {
 
	
	/**
	 * register the Data Manager 
	 *
	 * @param String endpointAddress
	 * @param String endpointResourceId
	 * @param String dataManagerName
	 * @param String classification
	 * @param String dialect
	 */
	
	public Element registerDataManager( String endpointAddress, String endpointResourceId, 
                  String dataManagerName, String classification, String dialect ) throws Exception

	/**
	 * deregister the Data Manager 
	 *
	 * @param String name
	 */
	
	public Element deregisterDataManager( String name) throws Exception


	/**
	 * ping the Data Broker 
	 *
	 * @param String name
	 */
	
	public Element pingDataBroker( String name) throws Exception 

sample code for the Broker Client API:

			//
			// get the Data Broker client from its endpointReference
			//
			DataBrokerClient brokerClient = new DataBrokerClient( brokerEPR ); 
			
			//
			// get the Data Managers from the Data Broker for the "Performance" classification
			//
			Element result = brokerClient.getDataManagers( "Performance" );
					
			System.out.println(XmlUtils.toString(result));

Sample Output (containing the EPRs):

<?xml version="1.0" encoding="UTF-8"?>
<DataManagers xmlns="http://cosmos.eclipse.org/dc/databroker">
    <DataManager lastmsgtime="8/17/07 11:36 AM" name="NSM">
        <Classification>Performance</Classification>
        <Dialect>SDMX</Dialect>
        <EndpointReference xmlns="http://www.w3.org/2005/08/addressing">
            <Address>http://138.42.158.197:8080/cosmos/services/com.ca.mr2.cosmos.dc.nsmpdg.NSMDataManager</Address>
            <ReferenceParameters>
                <ResourceId>NSMIdentifierValue</ResourceId>
            </ReferenceParameters>
        </EndpointReference>
    </DataManager>
    <DataManager lastmsgtime="8/17/07 11:36 AM" name="eHealth">
        <Classification>Performance</Classification>
        <Dialect>SDMX</Dialect>
        <EndpointReference xmlns="http://www.w3.org/2005/08/addressing">
            <Address>http://138.42.158.197:8080/cosmos/services/com.ca.mr2.cosmos.dc.ehealth.eHealthDataManager</Address>
            <ReferenceParameters>
                <ResourceId>eHealthIdentifierValue</ResourceId>
            </ReferenceParameters>
        </EndpointReference>
    </DataManager>
</DataManagers>

Command extensions for the data broker.

Refer to org.eclipse.cosmos.dc.spec\src\org\eclipse\cosmos\dc\spec\ConsoleExtension.java

osgi> broker
Usage: broker query <classification>
       broker register EPAddress EPResourceID DataManagerName classification dialect
       broker deregister DataManagerName

Data Broker Implementation Details

The Data Broker is implemented as a COSMOS DC query assembly with the following components:

1. Query Assembly definition. Refer to org.eclipse.cosmos.dc.sample.configurations\META-INF\cosmos\DataBroker.xml

 <?xml version="1.0" encoding="UTF-8"?>
 <context xmlns="http://www.eclipse.org/xmlns/cosmos/1.0"
 	xmlns:cosmos="http://www.eclipse.org/xmlns/cosmos/1.0"
 	xmlns:sample="http://www.eclipse.org/xmlns/sample/1.0"
 	cosmos:name="DataBroker" cosmos:direction="out">
 	<query cosmos:factory="org.eclipse.cosmos.dc.sample.components.query.DataBroker" cosmos:optimizable="true">
 		<sample:binding/>
 	</query>    
 </context>

2. Implementation Class, which is referenced by the Query Assembly definition. The implementation class maintains the persistent store. Refer to org.eclipse.cosmos.dc.sample.components\src\org\eclipse\cosmos\dc\sample\components\query\DataBroker.java

3. Capability file, which defines the API and interface of the implementation class (see the next section). Refer to org.eclipse.cosmos.dc.spec\src\org\eclipse\cosmos\dc\spec\capabilitiy\DataBrokerCapability.java

4. The registry is internal to the Broker. It is implemented in the COSMOS DC runtime derby database. Refer to org.eclipse.cosmos.dc.local.registry\persistence_setup\RegistrySchema.sql

CREATE TABLE COSMOS.DATA_MANAGER ( 
       ENDPOINTADDRESS VARCHAR(255) NOT NULL , 
       ENDPOINTRESOURCEID VARCHAR(50) NOT NULL , 
       NAME VARCHAR(50) NOT NULL ,
       CLASSIFICATION VARCHAR(50) NOT NULL,
       DIALECT VARCHAR(50) NOT NULL,
       LASTMSGTIME TIMESTAMP NOT NULL );
     
ALTER TABLE COSMOS.DATA_MANAGER
  ADD CONSTRAINT COSMOS.DATA_MANAGER_PK Primary Key (
     ENDPOINTADDRESS, ENDPOINTRESOURCEID );
  
ALTER TABLE COSMOS.DATA_MANAGER
  ADD CONSTRAINT COSMOS.DATA_MANAGER_UNQ_NM Unique (
     NAME);

Each Data Manager entry in the store contains the EPR (address and resource id), the name, the dialect, the classification keyword(s) and the timestamp of the last message received. The timestamp is updated automatically when the Data Manager is registered and also by the pingDataManager operation.

The Persistent Store is accessed using an ibatis sql map. Refer to org.eclipse.cosmos.dc.sample.components\src\org\eclipse\cosmos\dc\sample\components\persistence\sql\DataManager.xml

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE sqlMap PUBLIC "-//ibatis.apache.org//DTD SQL Map 2.0//EN"
   "http://ibatis.apache.org/dtd/sql-map-2.dtd">
<sqlMap namespace="StatisticalDataset">
 <typeAlias alias="DataManager" type="org.eclipse.cosmos.dc.sample.persistence.impl.DataManagerImpl"/>
 <statement id="getDataManagers" resultClass="DataManager" parameterClass="String">
   select
     ENDPOINTADDRESS as endpointAddress,
     ENDPOINTRESOURCEID as endpointResourceId,
     NAME as name,
     CLASSIFICATION as classification,
     DIALECT as dialect,
     LASTMSGTIME as lastMsgTime
   FROM COSMOS.DATA_MANAGER
   WHERE CLASSIFICATION = #classification#
 </statement>
 <statement id="getAllDataManagers" resultClass="DataManager">
   select
     ENDPOINTADDRESS as endpointAddress,
     ENDPOINTRESOURCEID as endpointResourceId,
     NAME as name,
     CLASSIFICATION as classification,
     DIALECT as dialect,
     LASTMSGTIME as lastMsgTime
   FROM COSMOS.DATA_MANAGER
 </statement>
 <insert id="addDataManager" parameterClass="DataManager">
   INSERT INTO COSMOS.DATA_MANAGER (ENDPOINTADDRESS, ENDPOINTRESOURCEID, NAME, CLASSIFICATION, DIALECT, LASTMSGTIME) 
   VALUES(#endpointAddress#, #endpointResourceId#, #name#, #classification#, #dialect#, #lastMsgTime#)
 </insert>
 <delete id="deleteDataManager" parameterClass="String">
   DELETE from COSMOS.DATA_MANAGER 
       WHERE NAME = #name#
 </delete>
 <update id="pingDataManager" parameterClass="DataManager">
   UPDATE COSMOS.DATA_MANAGER
       SET LASTMSGTIME = #lastMsgTime# 
           WHERE NAME = #name#
 </update>
</sqlMap>

Version 2 of the Broker includes the Data Source Type information. The schema requires 2 additional tables to support the many to many relationship between the Data Managers and the Data Source Types as follows: