1. Requirements

5.5.1 The project manager shall track and maintain software non-conformances (including defects in tools and appropriate ground software).  

1.1 Notes

NPR 7150.2, NASA Software Engineering Requirements, does not include any notes for this requirement.

1.2 History

1.3 Applicability Across Classes

Key:    - Applicable | - Not Applicable

2. Rationale

To make sure that all software non-conformances are addressed. Managing and tracking non-conformances, software problem reports, or software issues are critical steps to ensuring that software defects are flagged and handled properly.

3. Guidance

A Software non-conformance is any difference between plans, specifications, and software functionality that remains in the products after software delivery.  Even with a rigorous development process and thorough testing, some defects typically remain in the delivered products resulting in non-conformances.  

Maintaining records of non-conformances is essential to enable future software reuse or software upgrades.  Engineers who plan to reuse existing software products need thorough documentation of the root cause of anomalous behavior, documentation discrepancies, operational constraints, and operational workarounds.  Records of non-conformances can also be used to analyze the reliability and quality of delivered software products.  Non-conformance records are also useful in pinpointing specific software components or subsystems with excessive numbers of defects that remain in the delivered software.  

Software Change Tracking Tools can be used to document non-conformances, but the non-conformance data must be accessible to current and potential future users of software products.  The following is a suggested list of items that should be included in non-conformance tracking data:

• Date non-conformance was discovered
• How the non-conformance was discovered
• The severity of non-conformance (Refer to SWE-202 for a discussion of Severity Levels)
• Configuration information (i.e., Version information for the software, version information for any tools used, document revision number, etc…) 
• Contact information for responsible engineers
• Location of any supporting data 

Recording, tracking, and analyzing the root cause of non-conformances is a best practice.  (Refer to SWE-204 for a discussion of Root Cause Analysis.) 

Be cautious about closing overlapping software change tickets to ensure that the full scope of all associated software change tickets is addressed via retest. Be careful about missing any unique elements to the individual software change ticket.

Any aspects of control board decisions that are modified must be re-approved by the board (e.g., Impacted artifacts that need updating).

4. Small Projects

No additional guidance is available for small projects.

5. Resources

5.1 References

5.2 Tools

6. Lessons Learned

6.1 NASA Lessons Learned

No Lessons Learned have currently been identified for this requirement.

6.2 Other Lessons Learned

Software Change Tickets

• Any tests (formal or informal) that fail should be rerun and verified before software change tickets are closed, in the original environment, or as close to it as possible. Preferably this would be done with the original author of the software change ticket but with appropriate control board approval.
• Be cautious about closing overlapping software change tickets to ensure that the full scope of all associated software change tickets is addressed via retest. Be careful about missing any unique elements to the individual software change ticket.

7. Software Assurance

7.1 Tasking for Software Assurance

1. Confirm that all software non-conformances are recorded and tracked to resolution.

2. Confirm that accepted non-conformances include the rationale for the non-conformance.

7.2 Software Assurance Products

• List of non-conformances for which SA has verified the closure.

  
  

  Objective Evidence

  • Software defect or problem reporting data
  • Software configuration management data
  • Software assurance audit results on the change management process
  • Software milestone results
  • Software version description documents
  • Software control board data or presentations
   Definition of objective evidence

  Objective evidence is an unbiased, documented fact showing that an activity was confirmed or performed by the software assurance/safety person(s). The evidence for confirmation of the activity can take any number of different forms, depending on the activity in the task. Examples are:

  • Observations, findings, issues, risks found by the SA/safety person and may be expressed in an audit or checklist record, email, memo or entry into a tracking system (e.g. Risk Log).
  • Meeting minutes with attendance lists or SA meeting notes or assessments of the activities and recorded in the project repository.
  • Status report, email or memo containing statements that confirmation has been performed with date (a checklist of confirmations could be used to record when each confirmation has been done!).
  • Signatures on SA reviewed or witnessed products or activities, or
  • Status report, email or memo containing a short summary of information gained by performing the activity. Some examples of using a “short summary” as objective evidence of a confirmation are:
    • To confirm that: “IV&V Program Execution exists”, the summary might be: IV&V Plan is in draft state. It is expected to be complete by (some date).
    • To confirm that: “Traceability between software requirements and hazards with SW contributions exists”, the summary might be x% of the hazards with software contributions are traced to the requirements.
  • The specific products listed in the Introduction of 8.16 are also objective evidence as well as the examples listed above.

7.3 Metrics

• Trends of Cybersecurity Non-Conformances over time
• # of Non-Conformances (activities not being performed)
• # of Non-Conformances accepted by the project
• # of Non-Conformances (Open, Closed, Total)
• Trends of Non-Conformances over time
• # of Open vs. Closed Audit Non-Conformances over time
• Trends of # of Non-Conformances from audits over time (Include counts from process and standards audits and work product audits.)
• # of software work product Non-Conformances identified by life-cycle phase over time

7.4 Guidance

One of the key items in assuring that software quality is achieved in a system is the management of non-conformances that arise as the software system goes through the different levels of testing. Software assurance needs to keep a close watch on activities during those periods by either witnessing the tests or reviewing test reports to make sure all non-conformances (including errors, discrepancies, as well as other types of non-conformances) are being handled properly.

Activities that software assurance needs to perform associated with non-conformances:

• Confirm all discrepancies are recorded in the discrepancy reporting system (SWE-201, Task 1)
  
  • When discrepancies are discovered while the test is being witnessed, check to see that the discrepancies are noted on the test results. If the discrepancy is not also recorded in the discrepancy system during the test, look at the discrepancy system after the test to verify that the discrepancy has been recorded. If the test was not witnessed, examine the resulting test results and check that any discrepancies noted are recorded also in the discrepancy system.
• Confirm discrepancies recorded include defects in tools and ground systems (SWE-201)
  • The same process as above can be used to check those discrepancies in tools and ground systems being tested are recorded. If the discrepancies in the tools or ground systems are discovered during the testing of some other software, check to see that they are recorded in the test results where they were discovered and then verify that they have been recorded in the discrepancy system. 
• Confirm discrepancies recorded include discrepancies received from vendors of COTS, MOTS, GOTS, OSS, reused software, as well as discrepancies from this software found during testing of the project software (as per SWE-202)
  • Review the vendor websites of the COTS, MOTS, GOTS, OSS, reused being used to get a list of the discrepancies that have been documented. Check to see that these are also listed in the project discrepancy system. For GOTS or reused software, it may be necessary to contact the developers of this software to obtain a list of the known discrepancies. Also check to see if any discrepancies in the COTS, MOTS, GOTS, OSS, or reused software are discovered during the testing of project software, verify that those discrepancies have been added to the project discrepancy list and that the vendor/developers have been notified.
• Confirm that all recorded non-conformances have associated severity levels - correctly assigned (as per SWE-202, Task 2,3)
  • When reviewing the discrepancies documented in the project discrepancy system,  check that the appropriate severity level for each discrepancy has been recorded and that the severity level assigned meets the project-defined severity definition.
• Confirm that each non-conformance does not exist elsewhere in the system (SWE-201, Task 2)
  • Work with the project developer to identify areas where the code may be repeated or is similar, or where a similar error in logic or design may have caused a similar discrepancy and then examine those areas to determine if the same discrepancies exist (or work with the developer to assure that he/she has checked those areas.
• If a non-conformance is accepted for implementation of a correction, confirm that the rationale for the decision is recorded (SWE-201-Task 3)
  • Review the non-compliance records in the discrepancy system after the non-conformance has been approved for implementation and check to see that the decision rationale has been recorded.  
• Track each non-conformance to closure
  • Periodically, review the project discrepancy system to assure that all discrepancies are being addressed promptly. Discrepancies approved for implementation must be implemented and then tested, including regression testing. Other discrepancies should eventually have some final resolution like a work-around or a decision that the discrepancy doesn't need to be fixed.
• Review metrics on the non-conformances, including:
  • Number of non-conformances at each severity level ( SWE-202)
  • Total number of non-conformances/discrepancies (SWE-201)
  • Number of non-conformances/discrepancies closed (SWE-201)
  • Number of non-conformances/discrepancies open during this period (SWE-201)
  • Age of non-conformance/discrepancy (SWE-201)
  • The severity of each non-conformance/discrepancy (SWE-201)

• Be cautious about closing overlapping software change tickets to ensure that the full scope of all associated software change tickets is addressed via retest. Be careful about missing any unique elements to the individual software change ticket.

• Any aspects of control board decisions that are modified must be re-approved by the board (e.g., Impacted artifacts that need updating).