1. Requirements

5.1.9 The project manager shall participate in any joint NASA/developer audits. 

1.1 Notes

NPR 7150.2, NASA Software Engineering Requirements, does not include any notes for this requirement.

1.2 History

1.3 Applicability Across Classes

Key:    - Applicable | - Not Applicable

2. Rationale

Per IEEE Std 1028-2008, Software Reviews and Audits, "The purpose of a software audit is to provide an independent evaluation of conformance of software products and processes to applicable regulations, standards, guidelines, plans, specifications, and procedures." ²¹⁹  Audits are part of the supplier/provider monitoring activities performed by the acquirer ²²⁴, but may also be external audits, internal audits, or some other type of audit.  

Audits are conducted by audit teams and require the participation and cooperation of the personnel involved with the software being audited, both acquirer and provider personnel, including contractors, as appropriate for the particular audit being performed.

3. Guidance

This requirement is not intended to force joint audits, but when audits occur, the project needs to be made aware of and participate at some level in those audits, whether they are internal audits, contractor audits, external audits by an independent organization, or any other type of internal or external audit. Project participation can benefit the audit by providing domain knowledge, planning assistance, and technical expertise to the audit team.

This requirement was written to require projects to participate in audits that include any or all of the software portion of a project. The project's participation can take many forms, including, but not limited to, simply keeping abreast of the audit's progress as well as participating as an observer in the actual audit.

It is the responsibility of the project to make available appropriately prepared and qualified project personnel to participate or support audits as needed to fulfill the project's chosen level of involvement, including software assurance personnel described in the project's software assurance plan (see NASA-STD-8739.8 ²⁷⁸, Software Assurance, and Software Safety Standard,  for software assurance involvement in audits).

See topic 7.03 - Acquisition Guidance in this Handbook for additional guidance. Additionally, guidance related to joint audits may be found in the following related requirements in this Handbook:

4. Small Projects

For projects with limited personnel, consider limiting the audit participation in monitoring progress and reviewing the results as this would cause less interference and requires resources.

5. Resources

5.1 References

5.2 Tools

6. Lessons Learned

6.1 NASA Lessons Learned

The NASA Lesson Learned database contains the following lesson learned related to joint audits:

• Acquisition and Oversight of Contracted Software Development. Lesson Number 0921 ⁵²⁸: "The loss of Mars Climate Orbiter (MCO) was attributed to, among other causes, the lack of a controlled and effective process for acquisition of contractor-developed, mission-critical software.  NASA Centers should ... assure ... verification of the adequacy of the software design approach and overall contractor implementation throughout the software life cycle."  Audits are one way to assess the adequacy of contractor implementation throughout the software life cycle."

6.2 Other Lessons Learned

No other Lessons Learned have currently been identified for this requirement.

7. Software Assurance

7.1 Tasking for Software Assurance

1. Participate in or assess the results from any joint NASA/developer audits. Track any findings to closure.

7.2 Software Assurance Products

• Assessment of Joint NASA/developer Audit Results

  
  

  Objective Evidence

  • Software Problem reporting or defect tracking data
  • Software assurance audit results of the change management processes.
   Definition of objective evidence

  Objective evidence is an unbiased, documented fact showing that an activity was confirmed or performed by the software assurance/safety person(s). The evidence for confirmation of the activity can take any number of different forms, depending on the activity in the task. Examples are:

  • Observations, findings, issues, risks found by the SA/safety person and may be expressed in an audit or checklist record, email, memo or entry into a tracking system (e.g. Risk Log).
  • Meeting minutes with attendance lists or SA meeting notes or assessments of the activities and recorded in the project repository.
  • Status report, email or memo containing statements that confirmation has been performed with date (a checklist of confirmations could be used to record when each confirmation has been done!).
  • Signatures on SA reviewed or witnessed products or activities, or
  • Status report, email or memo containing a short summary of information gained by performing the activity. Some examples of using a “short summary” as objective evidence of a confirmation are:
    • To confirm that: “IV&V Program Execution exists”, the summary might be: IV&V Plan is in draft state. It is expected to be complete by (some date).
    • To confirm that: “Traceability between software requirements and hazards with SW contributions exists”, the summary might be x% of the hazards with software contributions are traced to the requirements.
  • The specific products listed in the Introduction of 8.16 are also objective evidence as well as the examples listed above.

7.3 Metrics

• None at this time

7.4 Guidance

Audits provide management with information about the project team, the project processes, and help identify best practices and areas of improvement.

Audits are useful to assess:

• Adequacy of project plans, processes, systems
• Compliance with those plans, processes, systems
• Effectiveness of those plans, processes, systems, and internal project controls on those processes
• Product fitness for use/compliance to specifications         
• Areas for improvement 

The results of audits allow project management to make adjustments and corrections to ensure high-quality products are being produced and delivered and that the team is functioning efficiently and effectively.  Trending audit results over time allows management to identify systemic issues and areas of risk while monitoring the effect of process and product improvements.

Software assurance personnel should either perform or participate in any audits that NASA or the project does jointly with a developer organization. NASA software assurance personnel will also be doing insight/oversight on any project providers and confirming that they are compliant with NPR 7150.2 to the extent specified in their contract. Providers should be performing audits against their procedures, plans, and activities and NASA software assurance should be participating, or at the very least, reviewing the results and tracking the findings to closure. Any findings from audits that software assurance performs or participates in should be tracked to closure.

Track Findings to closure

After the audit report is delivered, the audit team continues to have closure responsibilities such as those listed below.

• Review corrective action (CA) plans or responses from the project following an agreed-upon timeframe with the project.  Two weeks to 30 days is a reasonable timeframe depending on the project’s schedule and when they can reasonably implement the solutions, once approved.
  • Review CA plans for expected content:
    • Problem/Issue/Finding statement
    • Root Cause investigation plan, including “where else does this need to be applied” and a due date
    • Short term correction plan and a due date
    • Long term CA plan  (plan to avoid recurrence) and a due date
  • Assess any provided rationale for why corrective action is not needed, e.g.:
    • The project provided more evidence
    • Project clarified an audit team misunderstanding
  • Assess timeliness of CA plans, coverage of the associated Findings,  and recurrence prevention plan
  • Work any concerns with the project; this responsibility lies with the Lead Auditor and project manager
• Once the project has implemented the CA plans, review the results to assess how well those plans addressed the relevant audit Findings.  This includes a review of any revised documents, an assessment of revised process implementation, and perhaps a follow-up audit.
• When Findings are satisfactorily closed, the Lead Auditor notifies the project and audit team management in writing and captures the written notifications in the official audit records, including the rationale for closure of each Finding.

For information see the software assurance topic on software auditing.