Spaces/Architecture

Proposal - Draft 1 - Jan 14, 2008, Henrik Lindberg

For latest revision of interfaces, checkout the org.eclipse.spaces.core project in spaces SVN.
Please add your comments in a box like this.

At a glance

Via a Space it is possible to Publish into a "file system" of some sort, get access to Source in a CVS or SVN repository, and access/hook up a Collaboration (such as issue tracking) with a collaboration technology. Via Management it is possible to add/remove/control services.

Spaces Architecture

At the highest abstraction level, the architecture consists of:

1. Applications (user interface, automation "headless builds" etc.)
2. API (the core spaces functionality)
3. SPI (Service Programming Interface)
4. Implementation for different service provider technologies

Applications

Examples:

• View into spaces added to a workspace
• Wizards - managing spaces in workspace
• Space properties view - services provided by space provider, etc.
• Signing up for an account at a service provider

It is important that headless mode is considered when structuring application plugins.

Here are the application specifications

API

Here is a UML diagram of the proposed API File:SpacesArchUML.png

Spaces Service Provider

Each provider of spaces creates a plugin for a "Spaces Service Provider" extension point. When a user wants to add a Space Resource to a project, the first selection is among the available providers (e.g. AOL (X-Drive), SourceForge, GenericSVNSpace, GenericCVSSpace - the two generic space providers are thought to be useful reference implementations provided by the spaces project based on SVN and CVS).

A space service provider plugin should be headless, but can be accompanied by a UI plugin when needed.

Common (headless) traits are:

• Name, the name of the service provider
• A description of the service
• An enumeration of available Space Services - to enable user to select a provider based on capabilities.
• A list of provider schemes supported by the plugin

And with UI traits

• Icon for display with the description (e.g. a space provider logo)
• Small Icon for use in association with a Space Resource
• A URL to read more about the provided service
• Wizards for

Space Service

A Space Service describes a service available for spaces from a Spaces Service Provider These services are grouped into:

• Account and Space Management Services
• Source Services
• Publishing Services
• Collaboration Services

Account and Space Management Services

Management services are about creation/maintenance of Accounts, and Space Resources associated with such an Account.

• Account Reference
  Mandatory. Adding/Modifying a reference to an account by passing account information. Different providers may have implementation specific ways of storing the account information (e.g. Eclipse key ring, some (probably encrypted) file in the local file system, etc.). The idea is that the spaces project should reuse these implementations and not use a separate "vault"/"key ring". Produces Account Reference information that can be stored in spaces project meta data that can be used to reconnect.

This is now implemented as a Space URI, and Spaces handles authorization information for spaces.

• Account creation / Sign up
  Optional. A full implementation allows an account to be created via the API, but a simple implementation may consist of just information and an URL to a website where a user can sign up. A full implementation should add the new account to the set of accounts (i.e. create an account reference).
• Space Reference
  Mandatory. Adding/Modifying a reference to a space for an Account. Produces Space Resource information that can be stored in spaces project meta data.

{{CommentBox|This is implemented completely in spaces - SpaceCatalog keeps a persistent record of added space URIs in the workspace meta data.

• Space Creation
  Optional. A full implementation allows creating a space (i.e. 'server side') and adding it to the set of referenced spaces.

Note that a technology such as a "remove file system" can easily provide for multiple spaces.

• Future - Additional management services:
  • Delete of space
  • Controlling access rights to space
  • Change password
  • More ?

Source Services

Services related to working with source in a versioned repository. Note that the spaces project does not define what projects and versions to check out from the source repository, the service simply creates the references to the space's source.

Providing a source service is optional, If provided the service should be one of the supported:

• CVS
  This service provides the hookup between a Space Resource and the Team/CVS implementation in CVS. Different Spaces Service Providers may have different ways of addressing the CVS. Once the connection has been made, the user uses the normal Team /CVS plugin to do operations on the CVS instances.
• SVN
  Same as CVS, but for SVN. There is now a team.svn at eclipse even if one piece needs to be downloaded from Polarion because of a LGPL license issue.

This is implemented by returning a URI to the source code repo associated with the space.

Publishing Services

Publishing services allows upload/publishing of bits by a publisher, and download/consumption by a user. Core publishing is provided by spaces.

Upload

Spaces uploads/publishes bits to a naming structure by using The Eclipse File System (EFS) API. EFS also defines the the SPI for EFS implementations. There are already EFS implementations that operate against different types of local and remote file systems, and an implementor of a new type of EFS can either choose to implement the EFS interfaces directly, or as a plugin to the Target Management Platform's Remote System Explorer (RSE).

• The EFS was picked as it seems pointless to implement yet another "file system API", and EFS is what Eclipse uses.
• A "file system API" was picked to have a clear separation of concerns (i.e. deal with "just files" as opposed to understanding various published structures).
• An EFS API towards remote file systems like AOL X-Drive is of general value and is not specific to spaces.

The spaces publishing API consists of high level operations on a Space Resource with methods that allows publishing for 'test' or 'production', and with options regarding handling of meta data - see more below.

• EFS Provider
  Mandatory. The Service Provider must provide an EFS implementation. This implementation may be a full implementation - it is then assumed that the published structure is preserved and made available verbatim via the consumption URLs. A limited implementation can also be provided that only supports individual files in the root folder. This is then indicated with a capability flag ("NestedFolders") which is used by publishing to determine the applicable "publish as" actions for such a space.

Spaces Publishing

Update description with information about IPublishingOperation and selection. As Eclipse Jobs are used, include link to more information Jobs API

Publishing is done by calling methods on a Space Resource. Spaces also provides a wizard to help with publishing (i.e. setting options, and how to handle meta data).

A user invokes the wizard on some produced artifact they want to publish.

The wizard does the following:

• The user selects a space (or takes alternate route via creating account and creating space first)
• If user published a plug-in, user enters options:
  • Should an OBR be created or an existing OBR be used, or perhaps no OBR?
  • Should an Eclipse feature be created or an existing feature be used, or perhaps no feature?
  • Should an Eclipse update site be created or an existing update site be used (both implies a feature), or no update site?
  • Should an Eclipse p2 profile be created or an existing p2 profile be used, or no p2 profile?
• If user published something that is not a plug-in:
  • What kind of meta-data should be provided (name, version, type)
  • Documentation
  • Short description
  • Or appoint known meta-data format (cspec, POM, etc.)
• A selection if the published material is for "test" or "production" (the implementation of the space publisher defines how this is mapped to the underlying technology).

A first implementation can have a small set of hardwired ways of publishing, but this should be an extension point in the future. The Wizard UI then gets slightly more complex as the different "publish as" may have their own options that needs to be specified by the user, etc.

Consumption

It is perfectly valid to only provide publishing using an EFS as this can be used to perform publishing to any type of remote system - perhaps a small device. The spaces project is however mostly focused on publishing to an area that is accessed using a download protocol.

• HTTP access
  Mandatory. Allows access to the published content via regular "read only" URLs. The API is very simple; all that is required is that the root URLs to 'test' and 'production' are available.
• HTTPS access
  Future.

Collaboration Services

From the proposal: Collaborating: in the simplest case, a menu item that opens a web page for a collaboration medium (e.g., a forum). More complex cases might involve menu items that start up the Mylar Bugzilla view, or Corona for a Spaces-mediated chat session. The exact definition of collaboration will be left to the adapter based on the back-end's collaboration services. The key user interface feature is that the path to the collaboration will be obviously connected with the project itself.

The key objective here is that the spaces project provides the mechanisms for hookup between the Space Resource and Collaborations provided via different Collaboration Technologies.

Obtaining a Collaboration

After having obtained a Space Resource an enumeration of available Collaborations is obtained from the resource. Providing an enumeration of available Collaborations is mandatory (but can be an empty list). A Collaboration describes:

• Name - to allow identification
• Type - enumerator e.g. project home, issue tracking, forums, chatroom, documentation, etc.
• A description - if the name and type is not enough to understand the purpose of the Collaboration
• Additional key/value properties that the Collaboration Technology makes use of for the hook up.

Type is similar to a mime string, and spaces project should define a default but expandable set. Examples:

• issues/bugzilla
• issues/mantis
• issues/track
• wiki
• chat/irq
• ...rest to be specified

With this information, it is possible to hook up services, or provide simple viewing of links and information.

Hooking up Collaboration with Collaboration Technology

Spaces provides a CollaborationTypeRegistry with associations between collaboration type-strings and Collaboration Technology. The reason for the separation between the type of collaboration and what technology to use is to allow different types of tools/viewers to access the collaboration without the space having to have this knowledge. As an example, a space can point to a bugzilla, but should not need to know that the user wants to use Mylyn to access this information.

The spaces project comes with a generic link implementation that simply displays some information about the collaboration, and this provided as a default in the registry. To integrate with something like Mylyn, a plugin is created that registers itself for the appropriate types (perhaps issues/*) and associates this with a CollaborationTechnology class that simply does the required setup for Mylyn for a provided issue tracking link.

(The CollaborationTypeRespository is probably not a good name, and the mechanism can perhaps be modeled after something similar already in Eclipse - like file type/editor associations).

Adding Collaborations

Adding collaborations to a space is an optional service. The space provider determines how this additional information is stored in the space.

Provisioning Collaboration

A future API is to allow provisioning of collaborations - i.e. to request that some collaboration is set up on the server side - this could be a request to set up a wiki, or a bugzilla, create a chatroom/channel, create a subforum etc.

Spaces Catalog

The spaces catalog is comparable to the list of repositories available for Team/CVS and SVN. Access to spaces is a cross cutting concern, the same settings apply to multiple projects, and may be required before projects are available (since a project can be checked out from a space). The data in this catalog is also comparable to Team Project Set.

• The catalog itself is part of the workspace meta data
• The API consists of CRUD operations of Space Resources in the catalog
• When adding a space, the user runs a wizard where a list of Spaces Service Providers are listed, the user selects the provider, and then if a new space should be created (if provided by the Spaces Service Provider), or if they want to use an existing space, properties are then edited as needed).
• When sharing of Space Resources is wanted, they are added to a template workspace that later is used when doing headless builds etc. (This is important as credentials are needed for secure access to both spaces, and services provided by the space provider but accessed via 'non spaces' common APIs - e.g CVS or SVN access).

Space Address

A Space Address is the data that makes it possible to connect to a space using a Space Provider. The address is a URI that points to a space.

A Space has the following traits:

• An address (URI)
• Credentials to access the space
• An enumeration of available Space Services
• IFileStore handles for a flat list of files, and for a full file system (one or both)
• URLs for http access to the roots of the flat list of files, and for the full file system (one or both)
• Methods for begin/end/fail of publishing called from publishing operations in the space's EFS (can be overridden by a space provider that needs to perform sync or batch operations on the backing store).

Events

To ease integration spaces should issue events to registered listeners.

• When an Account Reference is added, changed or removed
• When a Space Resource reference is added, changed or removed
• When a Collaboration is added, changed or removed.
• When something is published to a space

This can be done in a manner similar to how listeners are handled for Eclipse resources.

FAQ

How should a user publish source if they do not use a versioned repository?

In this case, a source bundle should be created and published just like any other bundle by using the Publishing API.

What is the relationship between spaces and the remote file system (X-Drive) implementation?

The implementation for AOL X-Drive is one way of implementing what the publishing layer requires (to have somewhere to store published bits). This plugin can naturally be used by others that want to support the same JSON based protocol, and can naturally be used to create other services than those provided by "spaces". The spaces project however views the world as consisting of "spaces" having EFS based "containers" for "published artifacts" and a "container" for "source" irrespective of the implementation technology.

Read more about the X-Drive API here.

How can building and publishing be automated?

This is a quite large domain, as it consists of getting the wanted versions of all required source, performing a build, and then publishing the resulting bits. Such building could be done manually in the IDE, automated in headless fashion, be driven by ANT tasks, handled by Buckminster builds, or something else. It should be the choice of the users, especially as they may migrate to using spaces after already having working build infrastructure.

(Note: it is the intention to provide automated build and publishing to a space within the Buckminster project).

What if a publishing container can only support a "list of files"?

It is possible that a space provider only supports publishing of single files (e.g. typically zip files). The provider may have them in some kind of database and may give users the opportunity to state some facts in addition to name, like version, a description, some category like "latest", "production"... How can this be handled using EFS?

In this case an EFS implementation is expected where it is only possible to write files in the top level folder. The implementation signals its capabilities via getCapabilities(). The publishing functionality in spaces will only present applicable publishing options (e.g. it is not possible to publish a regular update site, but it is possible to publish an "archived update site").

Where does things published under "test" and "production" end up in the remote file system?

This is controlled by the space provider - different URIs for the two roots - "test" and "production" are provided as EFS root entry points, and different URLs for consumption. This makes it possible to point to two completely separate areas.

How do I integrate with mapping or RSS feeds?

The event mechanism can be used to trigger actions when something has been published. This can be used to automate downstream builds or testing, publish information on a RSS feed, trigger mapping/indexing and other external services. Spaces may provide such extensions in the future. Have an idea or want to contribute something? Contact us.

Where can I read more about EFS?

• Eclipse Zone intro article