Difference between revisions of "COSMOS Design 197867"

Revision as of 13:13, 21 August 2007

Data Broker for COSMOS DC framework

Change History:

Name: Date: Revised Sections:

Jimmy Mohsin 7/25/2007 Initial version

Bill Muldoon 8/1/2007 Added content

Bill Muldoon 8/17/2007 Updated content

Jimmy Mohsin 8/17/2007 Please see my comments on the Talk page Talk for 197867

Hubert Leung 8/21/2007 Updated broker design as discussed in meeting on Aug 20: http://wiki.eclipse.org/COSMOSBrokerSpecification. Need to consolidate the new design with this deisgn doc.

Overview

The Data Broker is intended to be a simple component.

Workload Estimation (required)

 Rough workload estimate in person weeks: 
 Process	Sizing	Names of people doing the work
 Design	1	MR 2 development team
 Code	.5
 Test	1
 Documentation	.2
 Build and infrastructure	.5	Bill Muldoon
 Code review & other committer work (e.g. check-in, contribution tracking) 
 if this is to be contributed by someone who is not a committer in the component	0	
 N/A - will be done by committer
 Total	3.2

Terminologies/Acronyms (required)

The terminologies/acronyms below are commonly used throughout this document. The list below defines each term:

Term Definition

 Data Broker	This is the component where all the web services that share data register themselves
 Data Store	This is a physical software artifact that stores data, e.g. Oracle, a File System, etc
 Data Manager	Largely corporate implementation component that implements the SOA API for data exchange
 Service Broker	This is the component where all the web services that share functional behavior register themselves

• [mdw] Can we add these to the glossary?

Function:

• [Assembly] Retrieving records from the data set
• [Data Broker] Associating a data manager with a data set
• Storing and maintaining a list of data sources, as references to where we get the data from
• Introspect the keyset definition
• Provide a cross reference to services. How does this differ from the first one?
• Registration of new keysets & datasets
• Client to chose which

• Data source is typing information for a data set
• Data set is the actual information, key set is the metadata

• A Data Manager is: A query assembly for a specific type of data. This includes its ability to be exposed as a WSDM endpoint
• A Data Manager can read many data sets, as long as they are all the same shape.
• A Data Set is an observation of data
• A Data Manager can deal with multiple instances of the same kind of data sets.
• Most of the function of the data manager is done as of today.
• Need to add the ability to support multiple data sets
• We think the Data Broker and the Provider are the same

• [Bill] For comparison purposes, the current COSMOS Provider Registry schema:

Stuff to do

• Jimmy/Mark: Get a picture that we can all agree on
• Jimmy/Mark: Clearly define the components that are in the picture
• Jimmy/Mark: Distill & cleanup this list of stuff and move to data broker what belongs there
• Joel/Hubert/Martin: Identify what needs to be reconciled in the code, e.g. packages, et..
• Joel/Martin: Publish the Data Manager Capability (WSDL/RMD/Schema)
  • see org.eclipse.cosmos.dc.remote.wsdm (this is the pojo version)

Purpose (required)

This design document addresses COSMOS Bugzilla enhancement request 197867.

The Data Broker maintains a catalog of information about Data Managers according to a classification type.

The Data Managers can register and deregister themselves with the Data Broker.

The Client applications can query the Data Broker to locate the Data Managers.

Requirements (required)

The “Data Broker” does the following:

1. Processes registration and unregistration requests from Data Managers

2. Processes queries from the client to locate the Data Managers

Use Cases (required)

<Include a set of common use cases that your feature will introduce or impact>

Need to expand more details on these use cases, e.g. input/output/desired results

• Use case 1: Register a Data Manager
• Use case 2: Query for a Data Manager
  • (Joel to update)
• Use case 3: Deregister a Data Manager
• Use case 4: Data Broker Initialization

Need more definition on the following:

• Use case 4: CMDBf considerations i7

• Use case 5: Event handling i7

• Use case 6: TTL for the Broker itself i7

Graphical Layout (optional)

Not applicable.

Class Diagram and Implementation Details

The Data Broker is implemented as a COSMOS DC query assembly with the following components:

1. Query Assembly definition. Refer to org.eclipse.cosmos.dc.sample.configurations\META-INF\cosmos\DataBroker.xml

 <?xml version="1.0" encoding="UTF-8"?>
 <context xmlns="http://www.eclipse.org/xmlns/cosmos/1.0"
 	xmlns:cosmos="http://www.eclipse.org/xmlns/cosmos/1.0"
 	xmlns:sample="http://www.eclipse.org/xmlns/sample/1.0"
 	cosmos:name="DataBroker" cosmos:direction="out">
 	<query cosmos:factory="org.eclipse.cosmos.dc.sample.components.query.DataBroker" cosmos:optimizable="true">
 		<sample:binding/>
 	</query>    
 </context>

2. Implementation Class, which is referenced by the Query Assembly definition. The implementation class maintains the persistent store. Refer to org.eclipse.cosmos.dc.sample.components\src\org\eclipse\cosmos\dc\sample\components\query\DataBroker.java

3. Capability file, which defines the API and interface of the implementation class (see the next section). Refer to org.eclipse.cosmos.dc.spec\src\org\eclipse\cosmos\dc\spec\capabilitiy\DataBrokerCapability.java

4. The persistent store is implemented as a new table in the COSMOS DC runtime derby database. Refer to org.eclipse.cosmos.dc.local.registry\persistence_setup\RegistrySchema.sql

CREATE TABLE COSMOS.DATA_MANAGER ( 
       ENDPOINTADDRESS VARCHAR(255) NOT NULL , 
       ENDPOINTRESOURCEID VARCHAR(50) NOT NULL , 
       NAME VARCHAR(50) NOT NULL ,
       CLASSIFICATION VARCHAR(50) NOT NULL,
       DIALECT VARCHAR(50) NOT NULL,
       LASTMSGTIME TIMESTAMP NOT NULL );
     
ALTER TABLE COSMOS.DATA_MANAGER
  ADD CONSTRAINT COSMOS.DATA_MANAGER_PK Primary Key (
     ENDPOINTADDRESS, ENDPOINTRESOURCEID );
  
ALTER TABLE COSMOS.DATA_MANAGER
  ADD CONSTRAINT COSMOS.DATA_MANAGER_UNQ_NM Unique (
     NAME);

Each Data Manager entry in the store contains the EPR (address and resource id), the name, the dialect, the classification keyword(s) and the timestamp of the last message received. The timestamp is updated automatically when the Data Manager is registered and also by the pingDataManager operation.

The Persistent Store is accessed using an ibatis sql map. Refer to org.eclipse.cosmos.dc.sample.components\src\org\eclipse\cosmos\dc\sample\components\persistence\sql\DataManager.xml

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE sqlMap PUBLIC "-//ibatis.apache.org//DTD SQL Map 2.0//EN"
   "http://ibatis.apache.org/dtd/sql-map-2.dtd">
<sqlMap namespace="StatisticalDataset">
 <typeAlias alias="DataManager" type="org.eclipse.cosmos.dc.sample.persistence.impl.DataManagerImpl"/>
 <statement id="getDataManagers" resultClass="DataManager" parameterClass="String">
   select
     ENDPOINTADDRESS as endpointAddress,
     ENDPOINTRESOURCEID as endpointResourceId,
     NAME as name,
     CLASSIFICATION as classification,
     DIALECT as dialect,
     LASTMSGTIME as lastMsgTime
   FROM COSMOS.DATA_MANAGER
   WHERE CLASSIFICATION = #classification#
 </statement>
 <statement id="getAllDataManagers" resultClass="DataManager">
   select
     ENDPOINTADDRESS as endpointAddress,
     ENDPOINTRESOURCEID as endpointResourceId,
     NAME as name,
     CLASSIFICATION as classification,
     DIALECT as dialect,
     LASTMSGTIME as lastMsgTime
   FROM COSMOS.DATA_MANAGER
 </statement>
 <insert id="addDataManager" parameterClass="DataManager">
   INSERT INTO COSMOS.DATA_MANAGER (ENDPOINTADDRESS, ENDPOINTRESOURCEID, NAME, CLASSIFICATION, DIALECT, LASTMSGTIME) 
   VALUES(#endpointAddress#, #endpointResourceId#, #name#, #classification#, #dialect#, #lastMsgTime#)
 </insert>
 <delete id="deleteDataManager" parameterClass="String">
   DELETE from COSMOS.DATA_MANAGER 
       WHERE NAME = #name#
 </delete>
 <update id="pingDataManager" parameterClass="DataManager">
   UPDATE COSMOS.DATA_MANAGER
       SET LASTMSGTIME = #lastMsgTime# 
           WHERE NAME = #name#
 </update>
</sqlMap>

Class Diagrams (optional)

<This section is only required for features that are introducing APIs that will be leveraged by other components>

The Data Broker exposes this API through its capability definition:

public interface DataBrokerCapability {
	
	/**
	 * Get the Data Managers
	 * 
	 * @param String classification
	 * @return IDataQueryResult containing an XML Element object
	 */
	@ManagedOperation
	public IDataQueryResult getDataManagers( String classification ) throws Exception;

	/**
	 * register the Data Manager
	 * 
	 * @param String endpointAddress
	 * @param String endpointResourceId
	 * @param String dataManagerName
	 * @param String classification
	 * @param String dialect
	 */
	@ManagedOperation
	public boolean registerDataManager( String endpointAddress, String endpointResourceId, 
                            String dataManagerName, String  classification, String dialect ) throws Exception

	/**
	 * deregister the Data Manager
	 * 
	 * @param String dataManagerName
	 */
	@ManagedOperation
	public boolean deregisterDataManager( String dataManagerName) throws Exception;

	/**
	 * ping the Data Broker
	 * 
	 * @param String dataManagerName
	 */
	@ManagedOperation
	public boolean pingDataBroker( String dataManagerName) throws Exception;

Command extensions for the data broker.

Refer to org.eclipse.cosmos.dc.spec\src\org\eclipse\cosmos\dc\spec\ConsoleExtension.java

osgi> broker
Usage: broker query <classification>
       broker register EPAddress EPResourceID DataManagerName classification dialect
       broker deregister DataManagerName

Data Broker Client API.

the Client API for the Broker:

public class DataBrokerClient extends WsResourceClient{

	/** 
	 * get the Data Managers 
	 *
	 * @param String classification
	 */
	
	public Element getDataManagers( String classification ) throws Exception {
 
	
	/**
	 * register the Data Manager 
	 *
	 * @param String endpointAddress
	 * @param String endpointResourceId
	 * @param String dataManagerName
	 * @param String classification
	 * @param String dialect
	 */
	
	public Element registerDataManager( String endpointAddress, String endpointResourceId, 
                  String dataManagerName, String classification, String dialect ) throws Exception

	/**
	 * deregister the Data Manager 
	 *
	 * @param String name
	 */
	
	public Element deregisterDataManager( String name) throws Exception


	/**
	 * ping the Data Broker 
	 *
	 * @param String name
	 */
	
	public Element pingDataBroker( String name) throws Exception 

sample code for the Broker Client API:

			//
			// get the Data Broker client from its endpointReference
			//
			DataBrokerClient brokerClient = new DataBrokerClient( brokerEPR ); 
			
			//
			// get the Data Managers from the Data Broker for the "Performance" classification
			//
			Element result = brokerClient.getDataManagers( "Performance" );
					
			System.out.println(XmlUtils.toString(result));

Sample Output (containing the EPRs):

<?xml version="1.0" encoding="UTF-8"?>
<DataManagers xmlns="http://cosmos.eclipse.org/dc/databroker">
    <DataManager lastmsgtime="8/17/07 11:36 AM" name="NSM">
        <Classification>Performance</Classification>
        <Dialect>SDMX</Dialect>
        <EndpointReference xmlns="http://www.w3.org/2005/08/addressing">
            <Address>http://138.42.158.197:8080/cosmos/services/com.ca.mr2.cosmos.dc.nsmpdg.NSMDataManager</Address>
            <ReferenceParameters>
                <ResourceId>NSMIdentifierValue</ResourceId>
            </ReferenceParameters>
        </EndpointReference>
    </DataManager>
    <DataManager lastmsgtime="8/17/07 11:36 AM" name="eHealth">
        <Classification>Performance</Classification>
        <Dialect>SDMX</Dialect>
        <EndpointReference xmlns="http://www.w3.org/2005/08/addressing">
            <Address>http://138.42.158.197:8080/cosmos/services/com.ca.mr2.cosmos.dc.ehealth.eHealthDataManager</Address>
            <ReferenceParameters>
                <ResourceId>eHealthIdentifierValue</ResourceId>
            </ReferenceParameters>
        </EndpointReference>
    </DataManager>
</DataManagers>

Extension Points (optional)

<This section should only be included if new extension points are being introduced>

<Include any additional topics that will help in completing the implementation>

<Content related to the topic above>

Test Coverage (required)

<Include a description of the unit, functional, and system test cases that will be required to test the overall quality of the feature>

The following tests are performed by org.eclipse.cosmos.dc.tests\src\org\eclipse\cosmos\dc\tests\client\DataBrokerTests.java

1. Data Broker Client interface

2. Registration test

3. Query test

4. Deregistration test

Task Breakdown (required)

<Includes the individual tasks that need to be completed to meet the overall objective of the enhancement. A PERT chart is included to indicate the dependency of each task.>

1. Query Assembly

2. Interface class

3. Implementation class

4. Derby database and iBatis sql map

5. Command extensions

6. Test application

Open Issues/Questions

• How does the query work? Need better example of how to query for data
• Need example of how to map existing data to SDMX structure
• Have we agreed that SDMX is the default model we support?
  • What about other models, e.g. SQL?
  • Does the user NEED to understand SDMX to use this?
• How does the design support other things that SDMX
• How do we reconcile the CMDBf query structure?
• Using operational status / life cycle / TTL (time to live) for appropriate components - i7 item
• How to keep the index in the data broker up to date

File:Dcframework2.zip

Notes from Aug 15 Conversation

• We want a "generic" API on the data broker and hide the implementation details
• We all want the seperation of the source of data from the type of data
• The first query "dialect" for the data broker will be XPath
• The data broker will support WS-Resource Catalog
  • Note: The combination of WS-Resource Transfer and WS-Resource Catalog should give provide the *most basic use case*
• It is expected that the COSMOS Data Broker can be replaced with a more robust implementation
• The data broker (in i6) will not support CMBD Query APIs.
• We want to keep the broker as simple as possible.
• We need the list of capabilities that the data broker supports
  • Convert the Provider API into a WSDM Capability (this is the i6 target)
    • Joel will document the wsdm capability and the API
  • WS-RT
  • CMDBf Query
• Map the SDMX data flow et. to SML concepts. (valentina)