5.3. Test Monitoring, Test Control and Test Completion

Test monitoring is concerned with gathering information about testing. This information is used to assess test progress and to measure whether the exit criteria or the test tasks associated with the exit criteria are satisfied, such as meeting the targets for coverage of product risks, requirements, or acceptance criteria.

Test control uses the information from test monitoring to provide, in a form of the control directives, guidance and the necessary corrective actions to achieve the most effective and efficient testing. Examples of control directives include:

• Reprioritizing tests when an identified risk becomes an issue
• Re-evaluating whether a test item meets entry criteria or exit criteria due to rework
• Adjusting the test schedule to address a delay in the delivery of the test environment
• Adding new resources when and where needed

Test completion collects data from completed test activities to consolidate experience, testware, and any other relevant information. Test completion activities occur at project milestone such as when a test level is completed, an agile iteration is finished, a test project is completed (or cancelled), a software system is released, or a maintenance release is completed.

5.3.1. Metrics used in Testing

Test metrics are gathered to show progress against the planned test schedule and budget, the current quality of the test object, and the effectiveness of the test activities with respect to the test objectives or an iteration goal. Test monitoring gathers a variety of metrics to support the test control and test completion.

Common test metrics include:

• Project progress metrics (e.g., task completion, resource usage, test effort)
• Test progress metrics (e.g., test case implementation progress, test environment preparation progress, number of test cases run/not run, passed/failed, test execution time)
• Product quality metrics (e.g., availability, response time, mean time to failure)
• Defect metrics (e.g., number and priorities of defects found/fixed, defect density, defect detection percentage)
• Risk metrics (e.g., residual risk level)
• Coverage metrics (e.g., requirements coverage, code coverage)
• Cost metrics (e.g., cost of testing, organizational cost of quality)

5.3.2. Purpose, Content and Audience for Test

Test reporting summarizes and communicates test information during and after testing. Test progress reports support the ongoing test control and must provide enough information to make modifications to the test schedule, resources, or test plan, when such changes are needed due to deviation from the plan or changed circumstances. Test completion reports summarize a specific test activity (e.g., test level, test cycle, iteration) and can give information for subsequent testing.

During test monitoring and test control, the test team generates test progress reports for stakeholders to keep them informed. Test progress reports are usually generated on a regular basis (e.g., daily, weekly, etc) and include:

• Testing period
• Test progress (e.g., ahead or behind schedule), including any notable deviations
• Impediments for testing, and their workarounds
• Test metrics (see section 5.3.1 for examples)
• New and changed risks within testing period
• Testing planned for the next period

A test completion report is prepared during test completion, when a project, test level, or test type is complete and when, ideally, its exit criteria have been met. This report uses test progress reports and other data. Typical test completion report include:

• Test Summary
• Testing and product quality evaluation based on the original test plan (i.e., test objectives and exit criteria)
• Deviations from the test plan (e.g., differences from the planned test schedule, duration, and effort)
• Testing impediments and workarounds
• Test metrics based on test progress reports
• Unmitigated risks, defects not fixed
• Lessons learned that are relevant to the testing

Different audiences require different information in the reports and influence the degree of formality and the frequency of test reporting. Test progress reporting to others in the same team is often frequent and informal, while test completion reporting follows a set template and occurs only once.

The IEEE 29119-3 standardincludes templates and examples for test progress reports (called test status reports) and test completion reports.

5.3.3. Communicating the Status of Testing

The best means of communicating test status varies, depending on test management concerns, organizational test strategies, regulatory standards, or, in the case of self-organizing teams (see section 1.5.2), on the team itself. The options include:

• Verbal communication with team members and other stakeholders
• Dashboards (e.g., CI/CD dashboards, task boards, and burn-down charts)
• Electronic communication channels (e.g., email, chat)
• Online documentation
• Formal test reports (see section 5.3.2)

One or more of these options can be used. More formal communication may be more appropriate for distributed teams where direct face-to-face communication is not always possible due to geographical distance or time differences. Typically, different stakeholders are interested in different types of information, so communication should be tailored accordingly.