Revision as of 13:38, 2 September 2010

{{#eclipseproject:technology.linux-distros}}

Linux Tools Release Engineering Procedures

Update sites

• Indigo nightly (what SVN trunk targets as of 2010-06-06): http://download.eclipse.org/technology/linuxtools/updates-nightly
• Helios nightly (what SVN "0.6.0" branches target): http://download.eclipse.org/technology/linuxtools/updates-nightly-helios
• Signed builds towards Helios train: http://download.eclipse.org/technology/linuxtools/update-helios
• Releases (latest is always http://download.eclipse.org/technology/linuxtools/update):
  • 0.1: http://download.eclipse.org/technology/linuxtools/update-0.1
  • 0.2: http://download.eclipse.org/technology/linuxtools/update-0.2 (shows as Access Forbidden in a browser due to no site.xml; using p2 Update Manager works fine)
  • 0.2.1: http://download.eclipse.org/technology/linuxtools/update-0.2.1 (shows as Access Forbidden in a browser due to no site.xml; using p2 Update Manager works fine)
  • 0.3: http://download.eclipse.org/technology/linuxtools/update-0.3
  • 0.4: http://download.eclipse.org/technology/linuxtools/update-0.4
  • 0.4.1: http://download.eclipse.org/technology/linuxtools/update-0.4.1
  • 0.5: http://download.eclipse.org/technology/linuxtools/update-0.5
  • 0.5.1: http://download.eclipse.org/technology/linuxtools/update-0.5.1
  • 0.6: http://download.eclipse.org/technology/linuxtools/update-0.6

Hudson builds

build.eclipse.org runs Hudson which runs builds of our project every 6 hours (0 */6 * * *) if something has changed in SVN.

For builds towards Helios service releases (SR1, SR2), ensure the code is on the 0.6.0 SVN branch of your component. For builds towards our future Indigo contributions, ensure the code is on the trunk of your component's SVN.

As of 2010-06-06 we have two active Hudson jobs:

https://build.eclipse.org/hudson/job/cbi-linuxtools-Hudson (built from 0.6.0 branches)

https://build.eclipse.org/hudson/job/cbi-linuxtools-Indigo (built from trunk)

Don't forget to commit fixes to *both* trunk *and* the 0.6.0 branch if they're targetted for both Helios service releases and Indigo contributions

Releases

We have nightly builds happening from trunk. The input source code for the build is taken from what is listed in the map file in org.eclipse.linuxtools.releng/maps. When it comes time for a release, we will:

1. Write Linux_Tools_Project/Releng#New_and_Noteworthy items as new noteworthy stuff is committed. Put this into new-{next release} in our website CVS location (:ext:youruserid@dev.eclipse.org:/cvsroot/org.eclipse).
2. Create branches for each sub-project (ex. 0.5.0); the branch should be the same name across all sub-projects
   1. for f in autotools changelog eclipse-build gcov gprof libhover lttng man oprofile profiling releng rpm rpmstubby systemtap valgrind; do svn cp svn+ssh://commiterId@dev.eclipse.org/svnroot/technology/org.eclipse.linuxtools/$f/trunk svn+ssh://commiterId@dev.eclipse.org/svnroot/technology/org.eclipse.linuxtools/$f/branches/0.5.0 -m "Branch for 0.5.0"; done
3. Change the map file in the branch of the releng directory' to build from the head of the branch
   1. cd org.eclipse.linuxtools.releng
   2. sed -i "s|trunk|branches/0.5.0|g" maps/linuxtools.map
   3. cd psfs
   4. for f in `ls`; do sed -i "s|trunk|branches/0.5.0|g" $f; done
4. Alter the Hudson job configuration (Helios or Indigo) to use the branch of the releng project
5. Continue to have automated builds with test runs from the branches
6. Continue to commit any work not targetted at the release to trunk
7. Commit changes specifically targetted towards the release only on the branch
8. Commit changes targetted at both the release and trunk on both the branch and trunk
9. Ensure all dependencies in build.properties are the correct versions against which we should be building/testing <--- this is important!
10. When all sub-project leads are satisfied that the branch is stable, we will run a signed build (Hudson options: type "S" and "-buildAlias <release number optionally followed by RC or something>" <-- the quotes are important!)
11. Submit IP log with tool
12. After IP log has been cleared, save a copy in the website CVS (the legal team usually emails a PDF or HTML snapshot to the IP log approval requestor)
13. Email technology-pmc@eclipse.org requesting permission to release (only required for a major (x.0) or minor ({*.x) release)
14. Email emo@eclipse.org requesting scheduling of a release review (only required for a major (x.0) or minor ({*.x) release)
15. If things are acceptable with the signed build, we will tag the SVN revisions that were used by the build with the release (ex. R0_5_0)
16. The Hudson job will then be switched back to use trunk of the releng project which takes input entirely from trunk
17. The Hudson job that will become the build should be locked to prevent automatic deletion (ex. build #263)
18. Ensure the archive of the entire build is placed into the downloads directory (~/downloads/technology/linuxtools) (Andrew Overholt and Alex Kurtakov are currently the only committers with access to this area)
    1. wget --no-check-certificate https://build.eclipse.org/hudson/job/cbi-linuxtools-Helios/237/artifact/*zip*/archive.zip
    2. unzip archive.zip
    3. mv archive/build/<build timestamp> .
    4. rmdir archive/build; rmdir archive
    5. chgrp -R technology.linux-distros <build timestamp>
19. Ensure the release's Linux_Tools_Project/Releng#New_and_Noteworthy is ready to go
20. Update linuxtools/{index.php,downloads.php,new} to reflect the release
21. Add a news item to the main wiki page
22. Copy our update site to a versioned copy to archive it (ex. update-0.5)
23. Verify the update site and all download links (including zip of update site) work
24. Add release to list of versions in bugzilla (in portal)
25. Add release.1 to list of target milestones in bugzilla (in portal)
26. Announce release on mailing list, newsgroup and blog(s)

Simultaneous Release Inclusion

As of Helios, Linux Tools is a part of the annual Eclipse simultaneous release. The simultaneous release aggregator takes content from the update sites that are listed in our b3aggrcon files (Helios, Indigo). Builds that we would like to promote to the simultaneous release must:

• be signed (re-run a nightly build with type 'I' when wishing to push a nightly to Indigo; a released build will already have been signed because it will be of type 'S' or 'R' so it can just be pushed to the aggregation update site and its versions/qualifiers updated in the b3aggrcon file)
• have their versions/qualifiers listed in our b3aggrcon file
• exist in the update site listed in the b3aggrcon file with the exact same versions/qualifiers

Note also that categories are set in the main simultaneous release file (Helios, Indigo) and the order of the features must match that in our contribution files (Helios, Indigo).

The builds must remain in the update site until we change the feature versions/qualifiers in the b3aggrcon file. This will prevent future aggregation runs from failing. More information can be found here:

• Helios/Contributing_to_Helios_Build <-- This one contains the CVS location for org.eclipse.helios.build where the linuxtools.b3aggrcon file lives and will need to be updated to new qualifiers [and versions] when we want to include a new release in Helios (for SR1 or SR2).
• Indigo/Contributing_to_Indigo_Build <-- This one contains the CVS location for org.eclipse.helios.build where the linuxtools.b3aggrcon file lives and will need to be updated to new qualifiers [and versions] when we want to include a new release in Indigo (for M2, M3, M4, etc.)
• Helios/Simultaneous_Release_Plan <-- Dates!
• Indigo/Simultaneous_Release_Plan <-- Dates!
• Eclipse_b3/aggregator/manual

The cross-project-issues-dev mailing list must be monitored by project leads and release engineers. This mailing list can help with any problems with the simultaneous release as well as with EPP packages. David Williams leads the simultaneous release creation and Markus Knauer coordinates the EPP packages including ours.

Builds and how they get places

New and Noteworthy

New and noteworthy (N&N) items should be written as soon as possible after the item has been committed. This will prevent a frantic scramble at the end of the release cycle. For example, for the 0.3.0 release, we have http://www.eclipse.org/linuxtools/new-0.3/ in CVS (:ext:youruserid@dev.eclipse.org:/cvsroot/org.eclipse and module: www/linuxtools). Please add new items under the various headings. Feel free to add new headings as necessary. The parenthesized numbers in the pseudo-index at the top of the file indicate the number of N&N items in that section. There is a template to be used for each release at new-template. Copy this for releases and be sure all recent releases are listed in the table and list at the top.

N&N images

Screenshots should be cropped to show only the pertinent region of the screen and so the N&N page doesn't appear too wide. Use the gimp to add drop shadows (pick the default values for radius and opacity). Save images as PNGs.

Accepting Patches

Please ensure that bugs which have patches contributed by non-committers have the "contributed" keyword set on them as well as the iplog flag set to +1. The flag should be set on the patch attachment itself and not the bug.

Adding a new component

Like most Eclipse projects, our build process is feature-driven. A top-level dummy feature, org.eclipse.linuxtools.all, acts as a container which lists all of our sub-project's top-level features. These then cascade down into their plugins and the whole thing gets wrapped up into our update site and zips. In order for a new component to be built, it needs to fit into the hierarchy somewhere. When adding a new top-level sub-project feature (sub-features of existing features should get added to their containing feature), follow these steps:

Code-level checklist

1. Create your feature and containing plugins
   1. Name the feature(s) org.eclipse.linuxtools.mycoolstuff and put it in SVN as org.eclipse.linuxtools.mycoolstuff-feature (replacing "mycoolstuff", obviously)
   2. Name the plugin(s) org.eclipse.linuxtools.mycoolstuff.core, org.eclipse.linuxtools.mycoolstuff.ui, etc.
   3. Ensure your packages are all in the org.eclipse.linuxtools namespace (and in the .mycoolstuff.core, .mycoolstuff.ui packages where appropriate)
2. Create your JUnit test plugins
   1. Use JUnit 3 and not JUnit 4
   2. Name your test plugin the same as your functional plugin but with a ".tests" tacked onto the end
   3. Create a test.xml for use in the automated builds (see an example here) *OR* add a property like the example in bug 276574 does.

SVN-level checklist

1. If this is a new sub-project, create a directory to contain it, like "oprofile" or "autotools"
2. Ensure that the standard directory structure of "branches", "tags", and "trunk" exists under your containing directory
3. Check your features and plugins into SVN under trunk
4. Create a tag called "initial-checkin" which is a snapshot of your initial work

Build-related checklist

1. Add any new top-level feature to org.eclipse.linuxtools.all
   1. If you are adding a new sub-feature, ensure it is in a containing feature which itself is a part of the org.eclipse.linuxtools.all tree
2. If your sub-project has dependencies outside the existing ones (BIRT, EMF, DTP, CDT, GEF), notify Andrew Overholt
   1. Hopefully this will have been caught in the CQ (legal review)
   2. Add dependency zips to build.properties
3. Ensure your BREEs are correct in your plugin MANIFEST.MF files
4. Ensure the version on your feature ends with ".qualifier" (without the quotation marks)
5. Add your test feature to the org.eclipse.linuxtools.test feature
   1. Alternatively, add to an existing test feature that is already in the org.eclipse.linuxtools.test feature hierarchy
6. Add new features to our update site's category.xml file
7. Add all new features, plugins, and fragments to our map file
8. Modify or create a new PSF file and put it here
9. To make Hudson do nice things like show which SVN commits when into which build, add your SVN location to the Hudson job configuration
   1. Ensure an existing committer has added your eclipse.org userid to the list of users and groups allowed to modify the job
      1. After logging in to Hudson, pick the job you want to modify and click "Configure"
      2. Under "Enable project-based security", add the new committer userid(s)
      3. Finally, check the Configure and Update checkboxes next to the new userid(s)
   2. Galileo builds (SVN trunk as of 2009-12-22):
      1. Go to https://build.eclipse.org/hudson/job/cbi-linuxtools-Galileo/configure
      2. For Repository URL, enter your new component's SVN trunk URL (ex. http://dev.eclipse.org/svnroot/technology/org.eclipse.linuxtools/valgrind/trunk)
      3. For Local module directory (optional), enter your new component's name followed by "-trunk" (ex. valgrind-trunk)
   3. Helios builds (SVN trunk or a Helios branch depending upon the plugin -- see linuxtools.map for details):
      1. Go to https://build.eclipse.org/hudson/job/cbi-linuxtools-Helios/configure
      2. Under Source Code Management, click the Add more locations... button
      3. For Repository URL, enter your new component's SVN trunk URL (ex. http://dev.eclipse.org/svnroot/technology/org.eclipse.linuxtools/valgrind/trunk)
      4. For Local module directory (optional), enter your new component's name followed by "-trunk" (ex. valgrind-trunk)
      5. If you have Helios-specific branches, enter them (ex. http://dev.eclipse.org/svnroot/technology/org.eclipse.linuxtools/valgrind/branches/Helios and valgrind-Helios)

Building locally

• check out releng/{trunk,branches/Galileo}/org.eclipse.linuxtools.releng using this PSF and Athena (the Eclipse common builder) using this PSF
  • File -> Import -> Team Project Set ... give it the location where you saved the above .psf

Compilation/build path errors in Athena's basebuilder can be safely ignored, as can any errors in the Linux Tools projects. Please note that you will need to install CVS and SVN plugins to complete the Team Project Set checkouts.

Within the build.properties file in org.eclipse.linuxtools.releng:

• ensure the dependencyURLs are correct
  • specifically the eclipse sdk tarball is architecture dependent, so you will need the version that corresponds to your architecture (for instance on x86, you'd simply remove "-ppc" from the filename and on x86_64 replace it with '-x86_64')
  • if you have any/all of these locally, you can create the directory "/tmp/build/downloads" and put the dependency zips or symlinks to them in this directory. This location can be changed with the downloadsDir variable.
• uncomment the JAVA60_HOME and JavaSE-1.6 variables and adjust them so they point to the correct locations on your system.
• set the variables JAVA50_HOME and J2SE-1.5 variables to either:
  • the values of JAVA60_HOME and JavaSE-1.6 variables respectively
  • or if you want to build using a separate 1.5 JDK you can specify the corresponding paths for it
• right-click on org.eclipse.linuxtools.releng/build.xml and run it as an ant build

Building using a local checkout

If you want to avoid checking everything out of SVN, you can build using a local source checkout. This can be useful if you have made local changes and want to verify that they will work.

• check out org.eclipse.linuxtools.releng and Athena as above
• if you do not already have them checked out, check out all of the Linux Tools projects using the Import Project Set command on this PSF.
• open the build.properties file in org.eclipse.linuxtools.releng
• uncomment localSourceCheckoutDir and set it to a directory on your filesystem
  • this will be used to create the necessary directory hierarchy to perform the build
• once build.properties is complete as above, run the script org.eclipse.linuxtools.releng/tools/localSourceSetup.sh from a terminal
  • this will create symbolic links to your local checkout in your workspace in the directory specified in localSourceCheckoutDir
• right-click on org.eclipse.linuxtools.releng/build.xml and run it as an ant build

If you have subversion-javahl installed you may encounter JVM crash when trying to fetch sources due to incompatible ABI. The easiest fix is to uninstall/remove your local javahl.

Adding a new test plugin to those that are run during the automated build

• Create test plugin(s) (ex. org.eclipse.linuxtools.mycoolfeature.ui.tests)
  • Add a test.xml like the other suites (this is used when the automated build is run)
  • Ensure test.xml and any resources are listed in build.properties
  • Check your plugin(s) into SVN
• If you are adding an entire test feature, add it to the included features in org.eclipse.linuxtools.test-feature
• If it's just a single test plugin (or more than one), add it/them to the included plugins in org.eclipse.linuxtools.test-feature
• Ensure the test plugin(s) do not get packed in the including feature.xml (ie. don't have "unpack=false" in the relevant plugin section of feature.xml)
• Add the test plugin(s) to the list of those to be run in testing.properties in org.eclipse.linuxtools.releng
• Add the test plugin(s)/feature(s) to the map file (be sure to update both the Galileo branch and trunk)
• Add the test plugin(s)/feature(s) to your sub-project's PSF (again, be sure to update both Galileo branch and trunk)
• Verify that your tests are built/run with a local build (see instructions on this page)

The next time a build happens, your test plugin(s) will be built and run. If you need a build pushed sooner than the next 6 hour mark when our scheduled builds happen, speak with Andrew Overholt.

eclipse-build tarballs

Tarballs for Galileo go here and Helios here. Older releases should be moved here. Don't forget to update md5sums.txt and sha1sums.txt!

• Tag (something like svn cp svn+ssh://commiterId@dev.eclipse.org/svnroot/technology/org.eclipse.linuxtools/eclipse-build/branches/0.4 svn+ssh://commiterId@dev.eclipse.org/svnroot/technology/org.eclipse.linuxtools/eclipse-build/tags/0.4 -m "Tag 0.4 final"; done if you want to use the command line)
• Generate with buildEclipseBuildSource.sh (ensure proper tag is used!): ./buildEclipseBuildSource.sh -eclipseBuildTag 0.4
• Upload: scp to downloads/technology/linuxtools/eclipse-build/whereyouwantittogo and chgrp technology.linux-distros; chmod 775 fileYouJustUploaded; md5sum fileYouJustUploaded >> md5sums.txt; sha1sum fileYouJustUploaded >> sha1sums.txt. For scp access to download.eclipse, a committer must be in special groups in the Eclipse Foundation systems. If you require access, please check with Andrew Overholt or Alex Kurtakov and then contact the Eclipse webmaster via a bug.