TPTP-Test-Automation-20071203

Attendees

• Present:
  • Paul
  • Jonathan
  • Joel
  • Joanna
  • Alan

• Absent:
  • Kiryl - conflict

Deliverables

• Testing Process:
  • Alan integrated the review comments and Paul posted the second draft of the Testing Process (part 1) document for review.
    • Allan will integrate review comments for the second draft of the Testing Process (part 1) document by Monday, December 3.
  • Alan will add more detailed content to the first draft of the Testing Process (part 2) by Wednesday, November 28.
  • Please review the latest drafts for both Testing Process documents for Monday, December 3.

• Common Test Infrastructure:
  • Completed:
    • CVS check-out of all of the test plug-ins in the /test-results/<project>/* modules.
    • Configure and run the Agent Controller on the reference platform.
    • Paul created the root-level test suites for BVTs. The test infrastructure will execute the org.eclipse.hyades.tests/AllBVT<test type>Tests.testsuite test suites. We do not need to check-in the execution results to CVS due to disk space limitations and polluting of our test pass results. Each developer can rerun the automated tests to reproduce a failure.
    • Paul created the BVT deployment to the reference platform.
    • Automatically generating the Test Report based on the existing test execution results in CVS.
    • Paul modified the BIRT tabular test report to walk an execution result to generate a detailed test report from a root-level execution result.
  • Outstanding:
    • Check for Intel build to complete (every x hours) before launching the BVTs.
      • Run when Intel is complete.
    • Automatically posting the Test Report to /tptp/test/reports/<release>.
      • Will copy code
    • Cleanup of the reference platform.
      • Done when staring the BVT.
    • Documenting the test infrastructure for use in the second testing process document, which will be required by TPTP adopters or extenders.
      • Open a defect for i5.
    • Email notifications to component leads for failing test suites.
  • Issues:

1. Defect 200194: Targeted to i5.
2. Defect 190453: Targeted to i6.
3. Defect 210289: Targeted to i6.
4. Defect 210292: Targeted to i6.
5. Defect 210955: DTP has resolved the issue, available in the December 1st build. Paul has verified the patch but the symptoms still exist due to defect 211381. Both defects are now resolved and Paul will verify the fixes.
6. Defect 211476: Paul will fix during i4 TP1.

• Few minor issues: Executable bit on Linux not set properly and no other roadblocks. Broken LTA dependencies consumed most of Joel but still breaking IAC.

Open a defect for Joel and Alan. Part 2 document...need defect AG presentation

• Metrics:
  • The monitoring project lead has replied to a request to estimate the current testing costs.
    • Joanna and Guru are gathering their estimates.

Action Items

• All:
  • Review the latest drafts for both Testing Process documents for Monday, December 3.

• Alan:
  • Integrate review comments for the second draft of the Testing Process (part 1) document by Monday, December 3.
  • Add more detailed content to the first draft of the Testing Process (part 2) by Monday, December 3.

• Joel:
  • Outstanding issues for the PoC (integrate the ASF with the TPTP builds to check-out a 'Hello World' TPTP JUnit test suite from CVS, run the test suite on an Eclipse instance running on the target platform, and generate a BIRT report on the test execution using existing sample code).

• Paul:
  • Determine labor cost (PW) for a) full test pass and b) smoke test pass from project leads.
  • Encourage Project Leads to automate existing manual test suites.

Upcoming Meeting

• Since this is We will meet next on Monday, December 3 (9 AM ET).

Upcoming Discussion

• Test framework reuse:
  • See defect 171827.
  • Existing test frameworks include:
    • Test Server for the Agent Controller.
    • Test Server for the JVMTI Profiler.
  • Proposed test frameworks include:
    • Test Dashboard (see enhancement 112928 and Test Dashboard Plug-in).
    • Test automation service (see defect 87285)