1. Requirements

4.6.3 The project manager shall complete and deliver the software product to the customer with appropriate records, including as-built records, to support the operations and maintenance phase of the software’s life cycle. 

1.1 Notes

NPR 7150.2, NASA Software Engineering Requirements, does not include any notes for this requirement.

1.2 History

1.3 Applicability Across Classes

Key:    - Applicable | - Not Applicable

2. Rationale

The ultimate goal of software development is to provide a product to the customer. Documentation must accompany that delivery to ensure proper understanding, use, and maintenance of the delivered product.  

3. Guidance

Typical contents of a software delivery package for a completed software project include the source files, executables (the product), a user's manual, and a version description document that describes the delivered product and contains procedures for creating executable software and modifying the software.  Suggestions for documentation contents are defined in 7.18 - Documentation Guidance of this Handbook. Open-source software licenses are also considered part of the delivery package and are reviewed by the Center's Chief of Patent/Intellectual Property Counsel before being accepted. Other documentation considered for delivery as appropriate for the project and its software classification (SWE-020) includes:

1. 1. Summary and status of all accepted Change Requests to the baselined Software Requirements Specifications.
   2. Summary and status of all major software capability change since baselining of the Software Design Documents.
   3. Summary and status of all major software tests (including development, verification, and performance testing).
   4. Summary and status of all Problem Report written against the software.
   5. Summary and status of all software requirements deviations and waivers.
   6. Summary and status of all software user notes.
   7. Summary and status of all quality measures historically and for this software.
      Definition of openwork, if any.
   8. Software configuration records define the verified and validated software, including requirements verification data (e.g., requirements verification matrix).
   9. The final version of the software documentation, including the final Software Version Description document(s).
   10. Summary and status of any open software-related risks.

In addition to the items listed in this requirement, consider the following items for the software delivery package:

• Certifications. ²⁷⁸
• Software safety plan. ²⁷¹
• Hazard analyses. ²⁷¹
• Safety-critical software development audit reports. ²⁷¹
• Safety-related verification reports. ²⁷¹
• Installation instructions, including a description of the hardware environment. ²⁷⁶
• Operational constraints, including environmental limitations. ²⁷⁶
• Configuration files. ²⁷⁶
• Project documentation (e.g., software development/management plan (SDP/SMP), assurance plan, software requirements specification (SRS), design documents, configuration management plan (CMP), test plan). ²⁷⁶
• Testing performed and the results of those tests. ²⁷⁶
• Training manuals ⁰⁴¹, training agreements. ²²⁴
• Maintenance/service/support agreement, processes, procedures. ²²⁴
• Development environment (any specialized hardware and software needed to build the executable software during the maintenance phase).
• Hardware is needed to test the software if specialized hardware is needed (for maintenance).

Once the project team establishes the set of documentation to be delivered with the software, the team keeps documents up-to-date throughout the project life cycle to avoid any delivery delays. Near the end of the project life cycle, the team updates any inconsistent documentation to ensure it includes the information needed to maintain the software in the future.

Before software delivery, a check of the software is made to verify that it conforms as-built to its technical documentation.   When mismatches occur this close to software delivery, the project team updates the documentation to match the delivered software before delivery to the customer. It is important to perform an audit before delivery to ensure that "all delivered products are complete, contain the proper versions, and that all discrepancies, openwork, and deviations and waivers are properly documented and approved." ²⁷⁸

Both the delivered software and the delivered documentation are generated/pulled from the project's configuration management system (SWE-085) as a baseline to ensure the latest versions are delivered to the customer.

Delivery package contents from a contracted software provider need to be fully described in the contract to ensure the acquirer receives all critical information required to operate and maintain the software. Other contract considerations related to delivery include:

• Ownership and delivery of source code.
• Usage considerations for off-the-shelf (OTS) software.
• Delivery format, security. ²⁷³  and method for all deliverables.
• Acceptance criteria (SWE-034 - Acceptance Criteria).

See topic 7.03 - Acquisition Guidance in this Handbook for additional guidance regarding delivery for contracted software development.

NASA-specific software delivery information and resources are available in Software Processes Across NASA (SPAN), accessible to NASA users from the SPAN tab in this Handbook. 

Additionally, guidance related to the delivery of software products may be found in the following related requirements in this Handbook:

4. Small Projects

No additional guidance is available for small projects.

5. Resources

5.1 References

5.2 Tools

6. Lessons Learned

6.1 NASA Lessons Learned

A documented lesson from the NASA Lessons Learned database notes the following:

• International Space Station (ISS) Program/Command and Data Handling/Firmware Documentation (Firmware Documentation.) Lesson Number 1024 ⁵³⁴: "NASA should ensure that all firmware code...is properly documented and archived for future reference. Further, NASA should ensure that it retains the rights to such software. Direction to deliver copies of the documentation (requirement, design, test, etc.) of the firmware controller software prepared as part of their software development process is being given to each vendor."

6.2 Other Lessons Learned

No other Lessons Learned have currently been identified for this requirement.

7. Software Assurance

7.1 Tasking for Software Assurance

1. Confirm that the correct version of the products is provided, including as-built documentation and project records.

2. Perform audits on the configuration management processes to verify that all products are being delivered and are the correct versions.

7.2 Software Assurance Products

• Software Configuration Management Baseline and Process/Procedure Audit Report
• Confirmation that Task 1 has occurred.
• Configuration management audits with audit findings to verify delivery products.
• Configuration management process audits with findings, issues identified.

  
  

  Objective Evidence

  • Version description documents for the software builds.
  • Software assurance audit results.
  • Software configuration management data.
   Definition of objective evidence

  Objective evidence is an unbiased, documented fact showing that an activity was confirmed or performed by the software assurance/safety person(s). The evidence for confirmation of the activity can take any number of different forms, depending on the activity in the task. Examples are:

  • Observations, findings, issues, risks found by the SA/safety person and may be expressed in an audit or checklist record, email, memo or entry into a tracking system (e.g. Risk Log).
  • Meeting minutes with attendance lists or SA meeting notes or assessments of the activities and recorded in the project repository.
  • Status report, email or memo containing statements that confirmation has been performed with date (a checklist of confirmations could be used to record when each confirmation has been done!).
  • Signatures on SA reviewed or witnessed products or activities, or
  • Status report, email or memo containing a short summary of information gained by performing the activity. Some examples of using a “short summary” as objective evidence of a confirmation are:
    • To confirm that: “IV&V Program Execution exists”, the summary might be: IV&V Plan is in draft state. It is expected to be complete by (some date).
    • To confirm that: “Traceability between software requirements and hazards with SW contributions exists”, the summary might be x% of the hazards with software contributions are traced to the requirements.
  • The specific products listed in the Introduction of 8.16 are also objective evidence as well as the examples listed above.

7.3 Metrics

•  #  of Configuration Management Audits conducted by the project – Planned vs. Actual
• # of Non-Conformances per audit (including findings from process and compliance audits, process maturity)
• Trends of # of Non-Conformances from audits over time (Include counts from process and standards audits and work product audits.)
• # of Open vs. Closed Audit Non-Conformances over time
• # of Compliance Audits planned vs. # of Compliance Audits performed
• # of software process Non-Conformances by life-cycle phase over time
• # of Non-Conformances identified in release documentation (Open, Closed)
• # of software components (e.g. programs, modules, routines, functions, etc.) planned vs. # released in each build

7.4 Guidance

Software assurance will confirm that:

• all the software planned for the delivery has been completed and 
• all software has been tested and has successfully met the acceptance test criteria, verifying that all the capabilities as per the requirements are ready for operations and maintenance. 
• the software includes all the approved changes and the changes have been tested,
• all defects approved for implementation are implemented and have been successfully tested (Where similar software exists elsewhere in the system being delivered, confirm that developers have checked to see that the defect does not exist in those places)
• all planned documentation for the build is completed and included with the delivery (user manuals, as-built documentation, operations manuals, build procedures or scripts, regression test sets with expected results, maintenance handbooks, etc.)
• delivery includes a list of any changes or defects that have not been implemented and their status (deferred to maintenance, accepted with a workaround, accepted as is)
• all other products needed for operations and maintenance are included in the delivery

To confirm that all the software has been successfully tested and meets the acceptance criteria, software assurance should perform or participate in a functional configuration audit (FCA). This confirms that all requirements for the build have been tested and have successfully passed. Software assurance should confirm that any changes or defects implemented in the software have also been successfully tested, including regression testing. 

In some cases, the software system may be delivered with known defects or non-functional capabilities if those are documented in the delivery documentation and the customer has agreed to accept the system with those defects. If the system is being delivered with known defects, the software assurance personnel should highlight any risks that those defects cause in the system, particularly if there are risks to the safety, security, or reliability of the software system. If risks are severe enough in those areas, software assurance might consider recommending not delivering the system until those risks are addressed.

In order the confirm that all products are included in the delivery in the correct versions, including the as-built documentation and project records, a physical configuration audit (PCA) against the documentation defining the delivery items needs to be performed. Software assurance will either perform this audit or participate in the audit and will sign off that the delivery is complete (i.e., it contains all the items it is supposed to have, in their correct versions for the delivery). The delivery items will be defined in the project documentation defining the planned builds and deliveries. Recorded version numbers of the items for delivery will be in the project configuration management system recording the baseline version planned for the delivery. These versions are usually also listed in the delivery letter or version description document (see 5.16 - VDD - Version Description Document) which are part of the delivery package. Other items that might be necessary for the delivery package are mentioned in the software guidance section of the requirement.

These tasks can be accomplished by Software Assurance reviewing the results of the configuration audits. Every task that involves performing an audit should also clarify that all audit findings are promptly shared with the project will be addressed in the handbook guidance.