TPTP-Test-Automation-20071022

Attendees

• Present:
  • Paul
  • Alan
  • Joel
  • Jonathan
  • Joanna

Reference Platform

• Target: IBM Java 1.5 (latest SR) and XP/x86.
• Developers are expected to unit test on reference platform before check-in.
• Nightly builds invoke automated test execution on reference platform.
• Goal to use a second reference platform for better test coverage (target: Sun Java 5 and Linux/x86).
• Move to Java 6 once a complete test pass is completed to expose initial defects.

Common Test Infrastructure

• ASF integrated with the TPTP builds (invoked via ANT):
  • Invoke tests.
  • Interrogate results.
  • Generate test reports.

• TPTP test types:
  • AGR for functional UI testing.
  • TPTP JUnit and TPTP JUnit Plug-in test for unit testing.

• TPTP build infrastructure:
  • Extract test suites from CVS.
  • Provision reference platform (Agent Controller and Eclipse/TPTP).
  • Install and configure Agent Controller and Eclipse/TPTP on the reference platform.
  • Clean-up reference platform.
  • Post test reports (test and code coverage) on TPTP web site.

Deliverables

• Alan:
  • Reviewed the TPTP Testing Strategy.
  • Reviewed comments and questions with Paul.

• Jonathan:
  • Completed port of existing profiler automated tests to used the Test Project framework (JUnit tests).

• Joel:
  • Located a dedicate test machine for the target platform.
  • Reviewing ASF documentation and experimenting with sample code from Joe Toomey.

Delivery dates

• October 31:
  • Complete the testing process, extending the existing TPTP Testing Strategy.
  • Profiler test suite automation.

• November 14:
  • Proof of concept (PoC) for integration of ASF and TPTP builds for PMC approval and integrate with TPTP builds.
  • Presentation to TPTP committers.
  • Measure cost savings.

• November 28:
  • Each project automates their manual test suites and converge existing automated test suites over time to leverage the test infrastructure.
  • Measure cost savings.
  • Report cost savings to PMC.

Action Items

• Alan: Complete the editions to the TPTP Testing Strategy, resulting in our new test process.
• Jonathan/Kendric/Paul/Joe Toomey: Assist Joel with integrating the ASF with the TPTP builds.
• Joel: Integrate the ASF with the TPTP builds to check-out a 'Hello World' TPTP JUnit test suite from CVS, run the test suite on an Eclipse instance running on the target platform, and generate a BIRT report on the test execution using existing sample code, possibly reuse some function from the Test Dashboard.
• Paul: Determine labor cost (PW) for a) full test pass and b) smoke test pass from project leads.
• Paul: Encourage Project Leads to automate existing manual test suites.
• Paul: Book another meeting in two weeks (November 6, 2007).

Notes

• Kiryl Kazakevich (Intel) has expressed interest in joining this initiative, representing the native components (e.g. JVMTI profiler).

References

• ASF
• ASF Tutorial
• TPTP Testing Strategy
• Test Dashboard (see enhancement 112928)
• Test Dashboard Plug-in