Notice: This Wiki is now read only and edits are no longer possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.
Difference between revisions of "TPTP-Test-Automation-20070924"
(New page: == Attendees == *Present: ** Paul ** Alan ** Joel ** Jonathan ** Joanna == Goals == * Consolidation of our testing process, and infrastructure. * Consolidation of custom test framework...) |
(→References) |
||
| Line 63: | Line 63: | ||
== References == | == References == | ||
| − | * | + | * [http://help.eclipse.org/help33/topic/org.eclipse.hyades.test.doc.user/tasks/tlaunching-tests-from-scripts-and-applications.htm ASF] |
| − | * | + | * [http://www.eclipse.org/tptp/test/documents/tutorials/eclipseWorld2006/Achieving_Continuous_Integration.html ASF Tutorial] |
| − | * | + | * [http://www.eclipse.org/tptp/home/documents/process/TPTP_Testing_Strategy.html TPTP Testing Strategy] |
| − | * | + | * [https://bugs.eclipse.org/bugs/attachment.cgi?id=46136 Test Dashboard] (see enhancement [https://bugs.eclipse.org/bugs/show_bug.cgi?id=112928 112928]) |
| − | * | + | * [http://dev.eclipse.org/viewcvs/index.cgi/test-results/platform/org.eclipse.tptp.internal.test.dashboard/?root=TPTP_Project Test Dashboard Plug-in] |
Latest revision as of 14:17, 30 October 2007
Contents
Attendees
- Present:
- Paul
- Alan
- Joel
- Jonathan
- Joanna
Goals
- Consolidation of our testing process, and infrastructure.
- Consolidation of custom test frameworks (Test Dashboard, AC/Profiler test server, variants).
- Reuse of Test Project framework (ASF, reporting, etc.) and tools (AGR, JUnit, etc.).
- Automatic test execution and report generation with every build (BVT).
Benefits
- Specialization of testing skills (e.g. builds).
- Provide an infrastructure that makes it easy for committers to contribute automated test cases/suites.
- Manual labor savings by using automated tests, which are automatically executed with every build.
- Decrease time/cost to resolve defects since uncovered earlier in the code-test-build cycle.
- Decrease testing overlap by a) localizing test executing and reporting to one group and b) using code coverage statistics.
Reference Platform
- Target: IBM Java 5* and XP/x86.
- Developers are expected to unit test on reference platform before check-in.
- Nightly builds invoke automated test execution on reference platform.
- Goal to use a second reference platform for better test coverage (target: Sun Java 5 and Linux/x86).
- Move to Java 6 once a complete test pass is completed to expose initial defects.
Deliver dates
- October 17:
- Define the testing process, extending the existing TPTP Testing Strategy.
- Define the common test infrastructure and its constraints.
- Proof of concept (PoC) for integration of ASF and TPTP builds for PMC approval.
- Profiler test suite automation.
- Determine the machine for the target platform.
- October 31:
- Fully integrate PoC with TPTP builds.
- Resolve infrastructure limitations.
- Presentation to TPTP committers.
- Measure cost savings.
- November 14:
- Each project automates their manual test suites and converge existing automated test suites over time to leverage the test infrastructure.
- Measure cost savings.
- Report cost savings to PMC.
Action Items
- Alan: Review and revamp the existing TPTP Testing Strategy as the new test process.
- Jonathan/Kendric/Paul: Port existing profiler automated tests to used the Test Project framework.
- Joel: Locate a dedicate test machine for the target platform.
- Joel: Integrate the ASF with the TPTP builds to check-out a 'Hello World' TPTP JUnit test suite from CVS, run the test suite on an Eclipse instance running on the target platform, and generate a BIRT report on the test execution using existing sample code, possibly reuse some function from the Test Dashboard..
- Paul: Determine labor cost (PW) for a) full test pass and b) smoke test pass from project leads.
- Paul: Encourage Project Leads to automate existing manual test suites.
- Paul: Book another meeting in two weeks (October 15).
References
- ASF
- ASF Tutorial
- TPTP Testing Strategy
- Test Dashboard (see enhancement 112928)
- Test Dashboard Plug-in