EclipseLink/Building/UpdatingDependencies

EclipseLink Dependency update process

The EclipseLink Product is made up of OSGi bundles. Those bundles are later merged and along with a new manifest file, they form the eclipselink.jar. As a result, all product compile dependencies must be available as OSGi bundles.

EclipseLink's OSGi bundles are built using "Tycho", a "manifest-first" OSGi compilation plugin for Maven. Tycho parses the project manifest to determine dependencies, then resolves them by searching p2 repositories. Then it compiles and afterwards packages the resulting OSGi bundle.

P2 is an Eclipse standard artifact storage and cataloging system designed to take advantage of OSGi standards. It is analogous to the maven repository system, but due to the nature of OSGi, it is slightly more versatile. Regardles, it is the standard for storing and retrieving Eclipse artifacts. It does, however, have one quirk similar to the Maven repository system. Stored artifacts must be named in the tokenized format (from OSGi manifest attributes):

  <Bundle-SymbolicName>_<Bundle-Version>.jar

or they will not be recognized as a "matching" bundle by the system (I think this is a lookup optimization).

Tycho automatically generates bundles to match this requirement. However, it should be kept in mind for any non-OSGi dependency that will be added to EclipseLink.

QA Tests, however, are not OSGi, as a result compile/runtime dependencies also need to be available via standard java compile mechanisms. Currently, that means they are checked into the git repo in a "plugin" directory >.

Since most product compile dependencies, are also test compile dependencies, this means that the checked in jars (that are both) will need to be bundles (have an OSGi manifest), and be appropriately named using the P2 naming conventions.

However, Tycho will not directly utilize on-disk dependencies. They must be available in a p2 repository, so for dependencies not available elsewhere, the build utilises a "compdep” repo (name is short for "compiler dependency").

This repository is pre-generated and stored in EclipseLink's "archive" area of the Eclipse download server (this is because it cannot be directly browsed externally, does not fall under the storage quota, but is still accessible external to Eclipse).

However, developer's will need to periodically update dependencies (or develop without relying on the network), and this would require some override mechanism. Therefore, the means to generate the p2 repository is built into the overall build system. By specifying "local.compdeps=true" (as a command- line variable or in the users local override properties file) the build will generate a local-p2 repository and use it as long as:

- it doesn't already exist (in which case it won't generate, but will use)
- the tools are available to generate ("eclipse.install.dir" defined to a
  valid Eclipse installation) 

When needing to generate a new global repo due to changing dependencies, this mechanism is also used, but the resulting repo is renamed before uploading to Eclipse. Another thing to note is that dependencies will vary depending upon the development branch (version) of EclipseLink, so there are actually multiple repositories stored online ("compdeps-update" for 2.4, and "compdeps25-update" and "compdeps26-update" for 2.5 and 2.6 respectively). Only The local compdep repo generates to "buildsystem/compdeps/p2repo" regardless which branch is currently checked out in git (this could be changed and would help developers avoid local compdeps clashes).

Below are the necessary steps for both developers and the build engineer to update or add a new dependency:

NOTE: it is usually easier to setup the override properties file than to constantly add variable definitions to the ant command-line, so the instructions below will describe that methodology, but it should be noted that adding a -D<variable>=<value> to the command-line will give the same result.

updating existing dependencies (new version):

1. Setup ${home}/build.properties file with:

    local.compdeps=true
    eclipse.install.dir=<location of eclipse installation>

2. Download and explode dependency jar.

3. update manifest to OSGi (can update old dependency's manifest to new version). The least it should contain:

    Bundle-SymbolicName: <some.bundle.name>
    Bundle-Name: <name (test)> 
    Bundle-Version: <version>
    Bundle-ManifestVersion: 2

however, exported packages may be necessary by having:

 Export-Package: <some.bundle.name.package>;version="<base bundle version>",

4. Re-package the dependency jar using the appropriate naming:

   <Bundle-SymbolicName>_<Bundle-Version>.jar
   ie: javax.validation_1.1.0.201406301033.jar (needs to match the values in
   the manifest)

5. Put the newly created dependency in the appropriate "plugins" dir.

6. If necessary, update the version criteria range for the specific dependency in

 <root>/buildsystem/compdeps/antbuild.xml:
    ie. "[0.0.1,1.0.0)" -> "[0.0.1,2.0.0)"

7. make sure there is no residual "p2repo" and/or "compdeps" directory in

  <root>/buildsystem/compdeps

8. run the build:

     eclipselink.runtime> ant -f antbuild.xml

or the compdeps build:

     eclipselink.runtime/buildsystem/compdeps> ant -f antbuild.xml

9. rename the resulting "p2repo" appropriately for the branch:

  2.4: compdeps-update
  2.5: compdeps25-update
  2.6: compdeps26-update
  Note: 2.7/3.0 may not have different dependencies to start so could continue
        to use compdeps26-update until such time as it needs to diverge.
        (to change update the "EL_CompDeps" repository setting in
        <root>/buildsystem/org.eclipse.persistence.parent/pom.xml)

10. ftp to build.eclipse.org/archive and remove the applicable "comdep???.old"

  dir, then rename the existing "compdep???" to "compdep???.old".
  (the .old backup is only needed in case a quick reversion is necessary. 
  once the new repo proves to be viable the backup is no longer needed)

11. ftp the newly created "compdep???" to build.eclipse.org/archive

12. test the build (non-local compdeps)

13. remove the old dependency jar from plugins.

14. merge any build changes

<done>

Developers could/should do steps 1-8 to test the new dependency and update any eclipselink code. The full process of generating the repo (5-14) needs to be completed by the build engineer before the transaction adopting it can be merged however.

The process to create a new dependency is the same, except for the addition of the following steps between 5 and 7:

6. Edit <root>/buildsystem/compdeps/antbuild.xml: 6a. bump "compdeps.version" to avoid metadata issues with tycho build. 6b. in the "Feature Dependency variables" section, add variables for

   <bundle>.prefix, and <bundle>.criterea

6c. in the "resolve-feature-dependencies" target add a "selectbundle" task for

   the dependency:
      <selectbundle
         basename="${<bundle>.prefix}"
         directory="${compdeps.2.common.plugins.dir}" <location of bundle>
         criterion="${<bundle>.criteria}" <version range to look for>
         property="<bundle>.version" <varable name to store discovered value>
         versiononly="true"
      />
   Also add an appropriate "say" message for the bundle.

6d. Add an appropriate filter task to the "create-deployable-features" target 6e. Add an appropriate incude to one of the copy tasks in "populate-p2-site":

   (The repos structure is assembled first in "create-deployable-features"
   and then the p2 metadata is created in "generate-p2")

6f. The feature.xml in:

    <root>/buildsystem/compdeps/org.eclipse.persistence.compdeps
   needs to have a filterable plugin section added for the new dependency:
   ie:
      <plugin
        id="<bundle.prefix>"
        download-size="0"
        install-size="0"
        version="@<BUNDLE>_VERSION@"
        unpack="false"/>

In this case the build changes as well as the new bundle need to be added to the git repository in 14.