Profiles Magazine: John, thank you for joining us.

John Vreeland: Thank you for this opportunity to share PTC’s philosophy and activities that relate to this very important topic. There are many software quality related activities currently underway at PTC. They are largely driven by corporate sponsored Customer Satisfaction initiatives and other direct involvement with customers. Both Product Development and Quality Assurance organizations are major contributors and responsible for built-in and tested-in product quality improvements.

PM: John, that being said, how does PTC define software quality?

JV: We believe that software quality is the measure of a product’s ability to efficiently and consistently perform the functions for which it was designed. There are many characteristics that together define and contribute to the level of quality delivered. Functionally complete, robust, reliable, scalable, fast, efficient, consistent and intuitive are a number of such quality characteristics. A quality product should also have an effective help system, training and technical support. Quality product design, implementation and validation steps all contribute to delivering a quality product.

PM: So what are PTC’s quality objectives?

JV: PTC’s quality objective is to increase customer satisfaction as it relates to Product Quality measures that we take on a quarterly basis. This is a major PTC corporate-wide initiative where everyone is measured and rewarded on contribution to goal. Specifically, product development is looking to meet well-documented release criteria for all product releases. This criteria is geared to address quality metrics that are reported upon throughout the lifecycle of each release.

PM: What specific metrics and process controls are used at PTC?

JV: PTC internally publishes product schedules and milestones for every release. Examples of milestones include; requirements freeze, projects committed, specification freeze, code freeze, pre-production, release-to-manufacturing (RTM) & first-customer-shipment (FCS)… PTC’s Product Line Review Board (PLRB), made up of senior management, reviews progress against the gating criteria for each milestone throughout the product development process. Published document and code review approvals are also significant quality control mechanisms. Additionally, electronic code submission and testing signoff serves as an effective means to control ownership, track accountability and enforce discipline. Final software quality metrics include but are not limited to:

• Number of open defects
• Severity of open defects
• Time-to-crash
• Time-to-defect
• Project tests
• Regression tests
• Data retrieval tests
• Performance tests
• Plot & bitmap tests
• Interoperability tests
• Internal production tests
• Customer pre-production feedback
• Documentation reviews
• Language tests
• License checks
• Installation checks

PM: To what level are regression tests used to evaluate release quality?

JV: Regression test automation is a critical part of our release qualification process. Thousands of tests have been written and are maintained to ensure that the system behaves consistently from release to release. As new capabilities are added to the system, quality assurance engineers execute against test plans that includes creating new and updating existing regression tests. Over time, our regression test suite has methodically grown, is constantly maintained and is cataloged by module and area of coverage.

The regression test suite is consistently used throughout the product life cycle (during development and throughout the maintenance period). In addition to playing a role in the final release qualification, regression tests are used by developers prior to every submission. Electronic submission form signoff requires that developers run regression tests as part of their pre-submission testing.

Before final release qualification and signoff, we perform and analyze regression tests on all platforms and languages. Issues identified by these tests are resolved prior to shipment. Although significant, regression testing is just one of many release criteria we require.

Recently, we have extended regression test tools and methods to enhance product interoperability qualification. Automation in all areas is significant as it allows more time for quality tests to be created and other manual testing tasks to be performed by our most skilled and experienced users.

PM: Does the QA organization have responsibilities for documentation quality?

JV: Yes, Quality Assurance is responsible along with Development and Product Definition teams to review and report issues against funded changes to documentation. Re-engineering documentation and training aids for Pro/ENGINEER Wildfire is one of PTC’s highest product priorities.

PM: What specific actions has PTC taken to address software quality concerns?

JV: PTC is committed to and has provided complete and ongoing support for improved product quality initiatives. We have established corporate goals, which include Product Quality, and have clearly communicated and extended them to all employees within PTC; not just management.

With resources and focused alignment to goals in place, we are taking literally dozens of tactical actions. Beginning with Pro/ENGINEER 2001, all quality procedures and tools have been reviewed, optimized and documented to new standards. Those accountable for each procedure are asked to present in bi-weekly Innovation Seminars. We hold these seminars to train or inform everyone in the organization. Update training is scheduled using a just-in-time approach that matches the upcoming product development phase activities.

Standardized procedures and instructions are documented, reviewed, approved and archived in PTC’s Windchill® document management system. They are easily accessible to all development sites for reference and must be used.

We have linked many of the tools and reports to PTC business systems (i.e. CRM and Project database) to ensure tracability and to leverage automation that ensures data accuracy and accessibility. Electronic tools have also been implemented to consolidate release status, display ownership and expedite qualification signoff by those responsible. Increased responsibility and accountability has had a profoundly positive effect.