1. Requirements

2.2.1.3 If a project is selected for software Independent Verification and Validation (IV&V) by the NASA Chief, Safety and Mission Assurance, the NASA IV&V program shall develop an IV&V Project Execution Plan (IPEP).

1.1 Notes

Note: The selection of projects and capabilities by the Chief, Safety and Mission Assurance, for IV&V is based on recommendations from the NASA IV&V Board of Advisors. The recommendations provided by the NASA IV&V Board of Advisors are the result of a multi-step process. First, the NASA Software Inventory, maintained by the Office of the Chief Engineer, is used to generate a candidate list of projects. The candidate project criteria include:

          a. Whether the project contains safety-critical software.
          b. The project's software classification(s) per this NPR.
          c. The level of software effort on the project.
          d. The project's NASA Category (e.g., Category 1, 2, or 3, as defined in NPR 7120.5D, NASA Space Flight Program and Project Management Requirements).
          e. Whether the responsible Center recommends the project for IV&V.

Second, the IV&V program utilizes this list and associated Center-provided data, captured in the NASA Software Inventory, to initiate a portfolio based risk assessment (PBRA) for capabilities within the candidate projects. The PBRA process assesses the risk of capabilities which have software associated with them, based on factors including:

          a. Complexity of the software implementing the capability under assessment.
          b. Risk to safety and mission success associated with the capability under assessment.
          c. Consequences of failure of the capability under assessment.
          d. Time to criticality (i.e., the amount of time before the system or subsystem enters into a critical condition).

The results of these assessments are prioritized, by capability, in order to provide recommendations for IV&V support on specific capabilities associated with specific projects. These recommendations are provided to the NASA IV&V Board of Advisors, which in turn provides its recommendations to the Chief, Safety and Mission Assurance.

Note: The IV&V program determines and documents the services to be provided for projects selected for IV&V by the NASA Chief, Safety and Mission Assurance. IV&V support is funded and managed independent of the selected project. The IPEP is developed by the IV&V program and serves as the operational document that will be provided to the project receiving IV&V support. The IV&V program and the project will establish a mutual understanding of the NASA IV&V program's activities and IV&V project interfaces. Per the responsibilities defined in NPD 7120.4, NASA Engineering and Program/Project Management Policy ²⁵⁷, section 5.J.(5), projects ensure that software providers allow access to software and associated artifacts to enable implementation of IV&V. Additional information on the content of the IPEP is found in Chapter 5 [of NPR 7150.2]. Additional detail on the IV&V PBRA and IPEP may be found in the NASA IV&V Management System, located at: http://www.nasa.gov/centers/ivv/ims/home/index.html.

Note from 5.1.8 IV&V Project Execution Plan (IPEP):

The IPEP is divided into two major parts: The body of the document and the appendices. The body includes mission overview, IV&V goals and objectives, high-level description of the IV&V approach, associated milestones, activities and deliverables, resources, roles and responsibilities, and lines of communication to establish a formal mutual agreement, as necessary to execute the project. The appendices include information that will change over the course of the effort and include schedules, relevant IV&V PBRA results, IV&V Coverage Diagram Data, and an acronym list. Additional information on the IV&V PBRA and IPEP may be found in the NASA IV&V Management System, located at: http://www.nasa.gov/centers/ivv/ims/home/index.html.

1.2 Applicability Across Classes

This requirement applies to all classes and safety criticalities selected for IV&V except as noted in the following table:

2. Rationale

The IPEP provides a mechanism for the IV&V Program to coordinate with a project to identify and describe IV&V activities that apply. This helps IV&V fulfill its mission to the Agency.

The IV&V Project Execution Plan (IPEP) Template ⁰²⁸ states that its purpose is two-fold. "First, it is to communicate IV&V interactions, interfaces, roles and responsibilities, technical products, and reporting methods with the Project. Second, the IPEP serves as the operational document for the IV&V efforts. The IPEP is prepared and maintained by the IV&V Project Manager (PM). The IV&V PM coordinates the creation and maintenance of this document with affected individuals and organizations (within the NASA IV&V Program as well as with the Project)."

3. Guidance

Because the IV&V Program is responsible for developing the IPEP, this guidance will address coordination efforts needed from projects chosen for IV&V to support the development and execution of the IPEP. Note that both an IPEP and a separate Memorandum of Understanding (MOU) may be established for project's receiving IV&V services, but only if the project requests the MOU.

Note that the IPEP is reviewed when it is initially developed, not at any particular life cycle milestone review. The milestone review at which the IPEP may be reviewed depends on which milestone the mission is approaching when IV&V gets involved with the mission. Per the IV&V Program, IPEPs are reviewed every 6-months for applicability or when there are significant changes made on the mission (e.g., a milestone slips out several months) or there is a significant change to IV&V (e.g., an IV&V budget cut which forces IV&V to reduce their work on a mission).

Provider Responsibilities

The responsibilities below and the means for carrying them out are documented as part of the IPEP (or other appropriate documents, in the case of contracted software efforts).

Per NASA-STD-8739.8: ²⁷⁸

• "When IV&V has been selected for a project, the provider shall coordinate with IV&V personnel to share data and information."
• "When the IV&V function is required, the provider shall provide all required information to NASA IV&V Facility personnel. (This requirement includes specifying on the contracts and subcontracts, IV&V's access to system and software products and personnel.)"
• "The NASA IV&V Facility shall initially conduct a planning and scoping exercise to determine the specific software components to be analyzed and the tasks to be performed. The IV&V approach will be documented in an IV&V plan."
• "The IV&V team shall provide input to the appropriate software assurance personnel, as well as provide feedback to the project manager as agreed in the IV&V Plan." (This helps to ensure that the IV&V team and the project's software assurance team maintain an active and productive relationship. Specific details of these interactions and the information to be exchanged for each project may be documented in the IPEP.)

Per NASA-STD-8739.8, ²⁷¹the Center S&MA organizations are assumed to carry out the following:

• "Assure that if IV&V is required on a program, project, or facility, project risk and software criticality determinations are shared between the safety personnel and IV&V."
• "If this project is a candidate for IV&V, the Software Safety Plan shall address, either specifically or by reference to the IV&V MOA, the role of IV&V for the safety-critical software and detail how IV&V will work with the software safety program and personnel."

NASA-GB-8719.13, NASA Software Safety Guidebook, ²⁷⁶ states that for contractor-developed software, "At a minimum, management, software development, and software assurance will need to work with the IV&V or IA (Independent Assessment) personnel, providing information, answers, and understanding of the software system and the development process.

"Depending on the level of IV&V levied on the project, some independent tests may be performed. These may require some contractor resources to implement."

IV&V Responsibilities

The activities to be carried out by IV&V are coordinated with the appropriate stakeholders, including the project manager. The IV&V team captures the following types of information in the IV&V Plan:

• IV&V activities to be performed for the project.
• Milestones.
• IV&V schedule coordinated with project management reviews and technical reviews.
• Interface among IV&V, Mission Project office, and software developer.
• Data exchange requirements.
• IV&V access to appropriate artifact management systems, issue and risk tracking systems, etc.
• Need for a heritage review to identify relevant products and artifacts from previous projects.
• Heritage review results.
• Preliminary list of processes and products to be evaluated.
• IV&V access rights to proprietary and classified information; related process to obtain those rights.

The IV&V team captures detailed fiscal year activity information in the appendices as activities, risks, schedules, and required resources may be subject to change each year based on project life cycle progress and other factors.

The following actions, summarized from NASA-GB-8719.13 ²⁷⁶ and other sources may be performed (this is not an all-inclusive list):

• IV&V – IV&V analysts conduct the in-depth analyses and verifications.
• Attend project reviews – summarize IV&V findings for software products.
• Integrated addition to software quality assurance (QA) – not a replacement.
• May perform specific safety analyses – not a replacement for the safety role.
• May perform some software engineering functions – e.g., systems engineering trade studies, such as studies used to determine if software architectures can satisfy performance needs of the system.
• Verify and validate concept documentation (the description of a specific implementation solution).
• Validate requirements and testing.
• Verify software architectures and designs.
• Verify requirements and design.
• Perform independent system testing.
• Ensure consistency from requirements to testing.
• Verify and validate test documentation.
• Verify and validate implementation.
• Verify and validate operations and maintenance content.
• Capture and report findings – tracking them to closure.

Execution Issues

Issues inherent in performing IV&V include the following for which projects and IV&V coordinate to minimize the impact to project success while assuring safety:

• IV&V "effort needs to be allocated at the right points in the development of a product (e.g. a document), so that the product is mature enough to be analyzed, but not so mature that it cannot be changed." ¹⁸⁰
• "There is always a delay between the delivery of an interim product to the IV&V team, and the completion of analysis of that product. During this time, the development process continues." ¹⁸⁰
• "...the IV&V contractor has less access to the development team than is ideal." ¹⁸⁰
• "Documentation from the development team is usually made available to ... IV&V ... in draft form... The drawback is that documents may be revised while the IV&V team is analyzing them, making the results of the analysis irrelevant before it is finished." ¹⁸⁰
• "Often, problems found by IV&V have cost and schedule implications..." 2 (Issues found in development products are expected to be fixed, which takes time and resources. Therefore, it is important that IV&V can work in phase with the development project so that errors in the system can be resolved efficiently.)

4. Small Projects

The IV&V Program does not distinguish between "large" and "small" projects. All projects with mission- or safety-critical software and selected by the NASA Chief S&MA, based on recommendations from the IV&V IBA to have IV&V, are treated the same. The same process is used for developing an IPEP, regardless of project size, and IPEPs all have the same basic elements (objectives, schedules, interfaces, etc.). The actual content of each IPEP is, of course, different for each project.

5. Resources

5.1 Tools

Tools to aid in compliance with this SWE, if any, may be found in the Tools Library in the NASA Engineering Network (NEN).

NASA users find this in the Tools Library in the Software Processes Across NASA (SPAN) site of the Software Engineering Community in NEN.

The list is informational only and does not represent an “approved tool list”, nor does it represent an endorsement of any particular tool. The purpose is to provide examples of tools being used across the Agency and to help projects and centers decide what tools to consider.

6. Lessons Learned

The NASA Lessons Learned database contains the following lesson learned related to the IV&V plan:

Independent Verification and Validation of Embedded Software (Use of IV&V Procedures). Lesson Number 723: "The use of Independent Verification and Validation (IV&V) processes ensures that computer software is developed in accordance with original specifications, that the software performs the functions satisfactorily in the operational mission environment for which it was designed, and that it does not perform unintended functions. Identification and correction of errors early in the development cycle are less costly than identification and correction of errors in later phases, and the quality and reliability of software are significantly improved." ⁵¹⁸