Notice: This Wiki is now read only and edits are no longer possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.
Difference between revisions of "SMILA/Project Concepts/Binary Storage"
Line 44: | Line 44: | ||
==== 2. Hierarchical structure ==== | ==== 2. Hierarchical structure ==== | ||
− | Through the hierarchical file system persistence, Binary Storage Service will manage by itself a configurable & hierarchicaly structure under the configured persistence storage path. | + | Through the hierarchical file system persistence, Binary Storage Service will manage by itself a configurable & hierarchicaly structure under the configured persistence storage root path (). |
+ | Following parameters are available for configuring the hierarchical structure: | ||
+ | * q - Maximum number of subfolders per folder | ||
+ | * r - Maximum number of persisted attachments per folder | ||
+ | |||
+ | The hiearchical (tree) structure is being created during the the storage of new data (the initial number of binary data which is going to be persisted is unknown for Binary Storage Service). | ||
+ | |||
+ | ===== Hierarchical structure nomenclature ===== | ||
+ | |||
+ | Fallowing picture outlines the hierarchic overview, file system nomenclature abd binary data distribution (persistence) inside the tree structure. | ||
+ | |||
[[Image:SMILA-BinaryStorage-Hierarchical.jpg]] | [[Image:SMILA-BinaryStorage-Hierarchical.jpg]] |
Revision as of 07:15, 16 October 2008
Contents
Overview
Design a service to easy store / access binary data documents.
Description
Client components will access the Binary Storage Service for persisting binary data (attachments) into the binary storage. The binary data shall be simply identified by a unique key / identifier as a String data type. No directely client component access to the persistence storage shall be available; the persistence storage will be only accessible through the Binary Storage Service API which provides the needed CRUD operations.
Backend mechanism of Binary Storage shall be completely transperent to the client, thus user shall have the oppurtinity to setup basic configuration of the service. Binary Storage shall be able to determine and use default/optimistic configuration in case no one is specified by the user.
Storage Mechanism Internal Structure.
Binary Storage will depend on the amount of data it needs to persist/manage. Because of this the persistence storage of service shall be able to deal with fallowing persistence structures/techniques, depending on service configuration:
- A. File System
- I. Local hard drive
- 1. Flat structure
- 2. Hierarchical structure
- II. Distributed file system (SFTP, FTP)
- I. Local hard drive
- B. Object DataBase
One of the persistence options will be used by the Binary Storage Service at the running time. Internally, the DAO & DAO Factory concept provides the appropriate/configured persistence option implementation to the Binary Storage Service independently. User can shall configure its appropiate persistence option that satisfyes his/her needs.
A. File System
Binary Storage Service saves the binary data directely in the file system.
I. Local hard drive
The service stores data in the local drive using a predefined persistence storage location - binary.storage.root.path. Under this root path Binary Storage will create its files system structures flat or hierarchical, depending on the configuration.
1. Flat structure
The file system - flat structure configuration shall be used in case of small amount of data, since all the attachements will be saved in the same path location. For huge amount of data the systems becomes very sllow, time responding increases significatelly.
This option shall only be used for debugging purpose, since it offers an easy way for locating a specific persisted attachment.
In case of no initial configuration is porvided by the user, the file system - flat structure option shall not be used as default.
2. Hierarchical structure
Through the hierarchical file system persistence, Binary Storage Service will manage by itself a configurable & hierarchicaly structure under the configured persistence storage root path (). Following parameters are available for configuring the hierarchical structure:
- q - Maximum number of subfolders per folder
- r - Maximum number of persisted attachments per folder
The hiearchical (tree) structure is being created during the the storage of new data (the initial number of binary data which is going to be persisted is unknown for Binary Storage Service).
Hierarchical structure nomenclature
Fallowing picture outlines the hierarchic overview, file system nomenclature abd binary data distribution (persistence) inside the tree structure.
II. Distributed file system (SFTP, FTP)
B. Object DataBase
Technical aspects for designing the Binary Storage Service
- The file system will be used as physical storage
- The Apache Commons Virtual File System (Commons VFS) framework is to be used for accessing the file system.
- For performance improvement (in case of ) the binary storage shall be hierarchically/tree organized; and not flat
- Binary storage shall internally manage its persistence hierarchy
- The binary service shall be designed as a single bundle / service. Main reason for doing this is that there is only one “client”, the Blackboard service
- Exception handling mechanism should treat all internal binary storage (logical and unexpected) errors and wrap the exceptions into a “binary storage exception” that makes sense for the Blackboard service
- Resources synchronization shall be done at the lowest possible level
- Binary Storage shall manage its configuration internally (highly couple classes are difficult to maintain and hard to understand in isolation – they tend to introduce internal dependencies). Decouple binary storage configuration from blackboard service
- The Binary Storage Service API shall stay as simple as possible
Current Binary Storage Service
Current implementation of Binary Storage is designed in five bundles, where the org.eclipse.eilf.binstorage(.impl) bundle acts as “service factory” for the Blackboard service, providing the files-service, configuration and file-content services.
Current Binstorage class diagram:
Current implementation is deprecated and it will be soon changed.