Stardust/Knowledge Base/Integration/Camel/Camel Based Monitoring and Auditing Transient Processes Using SEDA

Overview

In this article we are going to explore, how we can use Stardust Camel Applications,Triggers to monitoring and auditing the details of Transient process.

Use Case:

• There is one process which is marked as transient(having only non-interactive activities), so this process will not be persisted in audittrail database.
• For auditing purpose user wants minimum information to be captured.
• Monitoring of Transient process should be asynchronous and should not affect it's execution.

Why Transient Process ?

Before we jump to modeling above use case let's first discuss why do we have decided the process should not be saved in audiTrail and marked it as Transient ? see below process

• As per above process if you see there is one camel trigger which is polling to a directory for some files.
• It picks the file and passed to another camel application "Send File to JMS/FTP server" which does some processing and again sends to FTP or JMS server.
• Since this process is processing the files and files sizes may vary for e.g. 100 MB 200 MB or bigger. And we are storing file content into Process data and this data will persist to auditrail db.
• Storing the big files in process data will result un-necessary growing database size.
• To avoid above situation we decided not to persist this process in database and marked it as transient as below:

• So once the process is completed all the process data will be deleted.

Issue

As we discussed transient process will not be saved to auditrail database it means we will not be able to identify what all files are processed by which processes ? From an auditing purpose atleast following information should be captured:

• File name of the processed files.
• File processing started by transient process.
• File Processing completed by transient process.

Approach

To solve above issue and monitoring the execution of transient process we used camel SEDA component.Because it provides asynchronous SEDA(Staged Event Driven Architecture) behavior:

• Modified camel route in application "Send file to JMS/FTP server" as below:

• Above route will send events to SEDA endpoints before and after completing the file processing.
• Create consumer for SEDA START events

• Add process descriptor to this process

• Create consumer for SEDA END events

• Add below process descriptor

NOTE: If you notice the SEDA endpoints, we use "size"(size=500) parameter to prevent even if consumer stops working then it will not result into your server to go OUT OF MEMORY.

Execution

• To execute discussed process download the attached model from here
• Deploy the above downloaed model.
• Once the files will be processed by transient process for each processed file you could see below entries in process overview: