Difference between revisions of "TPTP-Test-Tools-20071112"

Latest revision as of 08:31, 28 November 2007

Attendees

• Present:
  • Paul Slauenwhite
  • Jerome Bozier
  • DuWayne Morris
  • Kendric Wang

• Absent:
  • Joseph Toomey - conflict

4.4.1

• 4.4.1 will ship with the Europa Winter Maintenance Release (target release date: February 29, 2008).

• Detailed Schedule.

• Defect query.

• Download.

• Test reports.

• All defects will require project/MC approval.

• All fixes MUST be checked-in to both 4.4.1 (TIPTOP-4_1_1) and 4.5 (HEAD) streams.

• Defect(s):
  • 209263 Deleted data pool row back in data pool upon import: Blocking a consuming product. Jerome B. completed the fix and needs to create the manual tests to verify the fix. Patch and patched plug-in attached to the Gizela. Once the manual tests are completed, Jerome G. will fill in the stop-ship template and send it to the Test Project mailing list for project/MC approval.

• After a review of the AP report for 4.4.1, the following Apes were added without the @provisional tag (see the TIPTOP AP Contract) and Java Doc comments:
  • JNIWrapper.java
  • InternetExplorerAdapter#clientConnectNotification()
  • Duwayne will resolve these problems under defect 168597 in both 4.4.1 and HEAD.

• There will be a smoke test pass in the next few weeks (target date: TBD) aligned with the latest release of the IBM 1.5 JRE.

4.5

• 4.5 will ship with the Ganymede Simultaneous Release (target release date: June 30, 2008).

• Detailed Schedule.

• Defect query.

• Download.

• Test reports.

• Reminder: The iteration numbers have been changed to reflect the Ganymede milestone numbers. For example, I3 has been changed to I3, which is aligned with Ganymede M3. See the detailed schedule and defect 208587.

• Newsgroup posts requiring replies:
  • Duwayne:
    • asf query test result and generate report from ant
    • BIRT classpath issue when publishing report
    • ant execution on linux
    • Create a Deployment programmatically

• We have finished TP1 and started TP2 (target date: November 14, 2007).
  • TP2 will be a smoke test pass (see TPTP Testing Strategy).
  • The candidate driver (200711050917) for TP2 has been posted, including regression fixes and without the OpenSSL code (see defect 195644).
  • Testing responsibilities are based on the new component ownership (see below table).
  • All of the Test Project committers must test with the EMMA-instrumented driver (see the instructions for details):
    • Everyone started using the EMMA-instrumented driver but could not continue:
      • Kendric/Jerome: First time testing TPTP so reverted to the TPTP SDK to reduce confusion.
      • Duwayne: Automated tests (AGR/JUnit/JUnit Plug-in tests) hang when running with EMMA since the EMMA JAR is missing from the class path and the emma.rt.control=false system property is required for the session JVM. Paul is investigating.
      • Paul: Complete manual tests using the EMMA-instrumented driver.
    • Paul will produce a EMMA report based on the manual testing statistics.
  • Status:
    • Paul: Done.
    • Duwayne: Will finish Test.Execution.AutoGUIRunner and Test.UI.AutoGUIRunner by Wednesday.
    • Kendric: Done. Paul is investigating failure (exception) when running Test.JUnit.Facade and Test.JUnit.CodeSync.
    • Jerome B.: Will finish TestNavigator_Logical_auto and TestNavigator_Resource_auto by Wednesday.

• Duwayne has automated several manual tests for URL test with the AGR but has found several issues.
  • Duwayne will itemize these issues and discuss them on next week's call.

• Joe has fixed several key ASF defects in 4.4.1 and 4.5 that may assist the Test Automation initiative.
  • Joe will itemize these fixes and discuss them on next week's call.

• Please vote this week on the following component changes (see the mailing list):
  • Notes:
    • Currently, Joe, Mark, and Jerome have sent nomination emails.
      • Paul will remind Liz and Kent to send their component lead nomination emails.
      • Paul will remind the other committers to vote.
    • Current Component Leads maintain their committer status as developer committers for the component.
    • Paul has sent instructions to the current Component Leads for nominating the proposed Component Leads.
    • Paul has removed the unnecessary components under defect 207968.
    • TPTP Project Matrix updates will be handled under defect 207968.
    • Duwayne will need to be a committer on the Platform Project to commit ASF fixes.
    • Joe will nominate Duwayne using the MyFoundation Portal for committer elections.

• In 4.5, we will accommodate the decrease in resources by:
  • Isolating and moving Manual Test to Technology Preview (see defect 207777).
  • Smoke testing the Technology Previews (Manual Test, API Recorder, and AGR) per iteration and maintenance release.
  • Fixing only critical and blocking defects for the Technology Previews (Manual Test, API Recorder, and AGR).
  • Reduced user support (news group and mailing list posts) for the Technology Previews (Manual Test, API Recorder, and AGR).
  • Focus on GA function for TPTP/Test Project presentations (e.g. EclipseCon 2008).

• Enhancements:
  • The following AG-approved enhancements have been requested for PG/PMC approval:
    • 202695 Encrypted datapool variable support: Will be completed by a non-committer from IBM (Xin Ying Huang). Paul and Joe are working with Xin Ying to close on the design.
    • 162605 Test execution should support parallel execution of tests: Duwayne has reviewed the enhancement and Description Document and with limited assistance from Joe, will start the design after I3 TP1 is completed.
    • 166025 Move, Copy, Delete, Rename and Import/Export test assets.: See the meeting minutes for the design review.
    • 145146 Generic Recorder Framework improvements.: Replaced with enhancement 208110
    • 208110 Generic Recorder Wizard enhancement to filter the list of recorders: Paul has reviewed the enhancement with the consuming product request the feature. They require an extension to disable individual recorders. Paul will hold an AG review call this week.
  • We will start the design for 4.5 on Thursday.
    • The milestones for these enhancements have been scheduled on the weekly schedule.
    • We will start reviewing/discussing the design work on next week's call.

• Defects:
  • All of the defects have been triaged and 81 defects have been put in plan:
    • Defect query
  • Required functional defects (target: 4.5i4):
    • 187864 Skewed completion rates in BIRT test reports.: Paul is finishing the implementation.
  • Required by consuming product(s) (target: 4.5i4):
    • 107432 (Test) IPv6 support in URL Test HTTP Recorder (required by Platform enhancement 165409): Duwayne has two IPv6 machines (Vista) installed/configured. Testing with JVM configured to use IPv6 addresses, by default. There are no known issues. This is primarily a testing effort.
    • 139589 Allow programmatic context for initially selected project in Generic Recorder Wizard
    • 201412 Repeated launch through Automated Services Framework eventually fails
    • 201150 Text in Recorder Control should be copyable.
    • 145070 Test Log Viewer: Link to deployment is displayed when using temp deploy not named 'default'
    • 180262 variables are listed twice in the "EquivalenceClass" pane of Datapool
    • 205476 copy/paste of a simple folder does not work in Test Navigator view
    • 205477 Copy / Paste should not be available in the logical test navigator
    • 162407 Test assets' editors (locations, datapools etc) allow save with empty internal names
    • 180280 Unmovable rows in Datapool Editor
    • 163150 Usability - adding existing artifacts/locations to deployment should select by name (not filename)
    • 195203 Accessibility-Need shortcuts for all Test Navigator pulldowns
    • 200586 Context sensitive help requires refactoring and new content.
    • 201849 No context help for New Test Deployment - Add Location Association dialog
    • 201892 doc updates for section Creating a test deployment
    • 201844 Time Frame Historical Report view does not have tabs and is missing info
    • 201827 Generate Report: Test Navigator does not refresh to show new report
    • 201828 Report Icon is incorrect for Test Pass and Time Frame Historic
  • Required by test automation initiative (target: 4.5i4):
    • 208616 Can't launch junit plug-in test with remote deployment: Jerome B. is investigating.
    • 205093 API Recorder's Generate Test Suite dialog does not select a default selection.
    • 194770 Verdict Summary BIRT chart in Test Log viewer displays 0%.
    • 200194 ASF Test Execution Service races with AutoBuildJob -- can lead to unexpected test results.
    • 171083 AGR Recorder does not distinguish between two controls on the same panel if they have the same text and type
    • 135103 [Test Automation] All external APIs should have JUnit tests
    • 171827 [Test Automation] Merge the various existing/proposed TPTP test execution frameworks.
    • 204849 AGR test suites model disabled test invocations incorrectly.
    • 87285 [Test Automation] [Quality Assurance] Automated Test Execution Harness Tests:
    • 170654 [Test Automation] Automate manual test suites using the Automated GUI Recorder.
    • 170655 [Test Automation] Automate manual test suites using the Automated GUI Recorder.
    • 170656 [Test Automation] Automate manual test suites using the Automated GUI Recorder.
    • 170652 [Test Automation] Automate manual test suites using the Automated GUI Recorder.
    • 170651 [Test Automation] Automate manual test suites using the Automated GUI Recorder.: Work-around required until 171083 is resolved.
    • 170653 [Test Automation] Automate manual test suites using the Automated GUI Recorder.: Paul has provided some AGR test suites created from the Test Navigator manual test suites for review. Once reviewed, please check them into CVS (HEAD), remove the manual test suites that they cover from the behavior AllManualTests.testsuite (and AllManualSmokeTests.testsuite), and add the new AGR tests to the behavior of AllGUITests.testsuite (and AllGUISmokeTests.testsuite).
  • Required to improve usability (target: 4.5i4):
    • 123650 [Usability] HTTP tests do not automatically generate JUnit Java code like JUnit tests.: Duwayne is investigating.
    • 206355 [Usability] Switch to Test Perspective after test execution.: Paul is investigating.
    • 206356 [Usability] Switch to Test Perspective after creating a test asset.: Paul is investigating.
    • 182128 [Usability] Test invocation error message cannot be copied.
    • 151922 [Usability] Hyperlinks in Test Log view are rendered differently for the Test Execution event type.
  • Required for translation/TVT (target: 4.5i5):
    • 190453 OutOfMemoryError when generating TPTP test reports.
    • 190558 BIRT test report types generate incorrect verdicts in test suite window.
    • 194752 Confusing label in Test preferences.
    • 164835 Remove unused messages for BIRT test report genreators.
    • 203273 Incorrect german translation of the word "Pass"
    • 200969 Formatting messages make translation difficult.
    • 200725 Simplified Chinese translation when editing a datapool is incorrect
  • Required/strategic/easily fixed (target: TBD):
    • 199782 Irreproducible test results when running AGR test suite from Ant script
    • 189128 Vista-specific error dialog when recording using IAC/IE on XP.
    • 200923 URL test runner does not capture standard out and error.
    • 195683 Datapools are not portable.
    • 194754 Location does not contain port number.
    • 200237 Can not debug launched eclipse process via ASF without rebuilding TPTP jar
    • 202269 Details page extension does not show any properties.
    • 164699 JUnit/Plug-in tests do not allow plugin root as source folder.
    • 200201 Need new icon for testlog files
    • 199783 NullPointerException when requesting publication of AGR test results from Ant script
    • 201402 [Testing Tool] Invalid characters inserted in source folder textbox when using Browse button of "Generate..." menu item
    • 199421HTTP recording does not work with Internet Explorer 7 and Firefox 2.0

• The following 4.5 defects require a target for a specific iteration:
  • Defect query.

• The following 4.5 defects require a sizing:
  • Defect query.

• 144950 [Scalability] Address TPTP EMF model scalability problems: Although withdrawn as a 4.5 candidate enhancement, the outcome/status of the work done in 4.4 on this enhancement was not updated in the WIKI page. Joe will update the WIKI this week.

• Test project topics for upcoming AG calls include:
  • Enhancement 166025 (Move, Copy, Delete, Rename and Import/Export test assets): Jerome B./Jerome G.
  • Test automation initiative: Alan/Joel
  • EMMA: Paul

Roundtable

• Status updates and summary of defects, blocking issues, and architectural issues/concerns that we will focus on completing the upcoming week:
  • Paul:
    • Investigating issue with automated tests (AGR/JUnit/JUnit Plug-in tests) hanging when running with EMMA Issue running automated.
    • Finish reporting defect 187864.
    • Design and AG review for enhancement 208110.
    • 4.5 release planning and scheduling.
    • Start fixing defects.
  • Duwayne:
    • Finish 4.5 I3 TP2.
    • Finish IPv6 defect 107432.
    • Design enhancement 162605.
    • Update defect sizings.
    • Fix ASF defect 200194.
    • Note, Duwayne will be off Wed. Nov. 21 - Fri. Nov. 23 for Thanksgiving.
  • Jerome:
    • Finish 4.5 I3 TP2.
    • Design enhancement 166025.
    • Update defect sizings.
    • Create manual test(s) for datapool defect 209263.
  • Kendric:
    • Automating Test Log Viewer test cases (see defect 170652).
    • Automating API Recorder test cases (see defect 170656).
    • Review, select, and prioritize candidate Test Project defects:
      • Defect query

Action items

• Duwayne, Jerome, and Paul to discuss the design work for the 4.5 enhancements on next week's call.

• Duwayne will itemize and discuss several AGR issues on next week's call.

• Joe will itemize and discuss several key ASF fixes on next week's call.

• Joe will nominate Duwayne as a committer on the Platform project using the MyFoundation Portal for committer elections.

• Paul will remind Liz and Kent to send their component lead nomination emails.

• Paul will remind the other committers to vote.

• The following 4.5 defects require a target for a specific iteration:
  • Defect query.

• The following 4.5 defects require a sizing:
  • Defect query

Notes

• EclipseWorld 2007:
  • Joe and Scott Schneider presented a tutorial (Achieving Continuous Integration With Eclipse TPTP and Java), which was well attended and received. There were ~25 paid attendees, and nearly all of them worked through the hands-on portion successfully (implementing an ASF service that appends resources to an existing working set, creating a new one if necessary.)
  • The Probekit talk that Joe presented for Guru drew about 18 people, and was also very well received. They asked good questions, the demos all went well, and several people have contacted me asking for the probes used in the demos for their own use.
  • Joe attended Steve's Static Analysis talk, which was standing room only and was also well received.

• EclipseCon 2008:
  • Reminder to participate in EclipseCon 2008 by proposing a talk or tutorial for the Test and Performance category. This category covers topics related to test and performance tooling that leverage Eclipse. Test and Performance tooling is used throughout the software development and test life cycle, from early unit testing, profiling, and tuning to post-deployment monitoring, tracing, and log analysis. This category is focused on the implementation, extension, and use of test and performance tooling - both open source and commercial - targeting a broad spectrum of systems including embedded, stand-alone, distributed, and enterprise. There is particular interest in exploitation and user experience of the Test and Performance Tools Platform (TPTP) project. Presentation types include:
    • Tutorial: A two hour presentation, with coding and/or a hands on workshop.
    • Long Talk: A one hour presentation that goes into detail and/or demonstration on a topic.
    • Short Talk: A five to ten minute presentation that covers the highlights of a topic.
    • Posters: Posters are presented at the poster reception session.
    • Birds of a Feather (BoF): BoF meetings are informal one-and-a-half-hour gatherings of people who want to talk about a particular topic.
  • If you are interested in proposing a talk or tutorial, please:
    • Review the EclipseCon 2008 Submission Guidelines
    • Submit your talk or tutorial
  • Important dates include:
    • November 19, 2007: Submission deadline.
    • December 10, 2007: Tutorials, Long Talks, and Short Talks selected.
    • January 19, 2008: Submissions open for posters and BoFs.

• Paul and the Release Engineering Team are working on consolidating our testing process and automation infrastructure. For more information, see the WIKI page and meeting minutes.

Reminders

• Lead Committers:
  • Continually review copyright reports and ensure files will missing copyright or incorrect copyright year are corrected.
  • Add the Original Estimate (PH) when triaging a new defect.
  • Update the Hours Worked (PH) when completing a defect.
  • Test their components or review/track testing contributed by other committers/resources.
  • Monitor the TPTP news group and answer posts specific to their Test Project components.
  • Verify that all API changes (modifications, additions, and removals) do not breech the TPTP API Contract.
  • Use the MyFoundation Portal for committer elections. The Portal adds more automation to the elections and makes the individual project elections more transparent to the entire Eclipse community. If you have been experiencing problems or have other issues with the Portal election process, please contact the EMO (emo@eclipse.org).

• All Committers:
  • Monitor the TPTP news group and answer posts on topics other than the Test Project to support our user community.