1. Requirements

3.6.2 For projects reaching Key Decision Point A the program manager shall ensure that software IV&V is performed on the following categories of projects:

1. 1. Category 1 projects as defined in NPR 7120.5.
   2. Category 2 projects as defined in NPR 7120.5 that have Class A or Class B payload risk classification per NPR 8705.4.
   3. Projects selected explicitly by the NASA Chief of the Office of Safety and Mission Assurance to have software IV&V.

1.1 Notes

The NASA IV&V Board of Advisors supports the NASA Chief, Safety and Mission Assurance by recommending significant project needs for software IV&V beyond projects meeting the criteria in items a. and b. of SWE-141. Exceptions to the above requirement will be written by the project and responsible Center SMA organization, adjudicated by the NASA IV&V Board of Advisors, with the final decision by the NASA Chief, Safety and Mission Assurance. Additional projects, projects in other phases, or projects without a payload risk classification can be selected by the NASA Chief, SMA to be required to have software IV&V. It is NASA policy to use the NASA Independent Verification and Validation Facility as the sole provider of IV&V services when software created by or for NASA is selected for IV&V by the NASA Chief, Safety and Mission Assurance. IV&V support is funded and managed independently of the selected project.

1.2 History

1.3 Applicability Across Classes

Key:    - Applicable | - Not Applicable

2. Rationale

The rationale for independent validation and verification (IV&V) on a project is to reduce the risk of failures due to software and provide assurance that the software will operate as intended, not operate unexpectedly and respond appropriately to adverse conditions. Performing IV&V on projects yields greater confidence that the delivered software products are error-free and meet the customer’s needs.  IV&V across the project lifecycle increases the likelihood of uncovering high-risk errors early in the life cycle.

3. Guidance

Independent validation and verification (IV&V) is a part of Software Assurance playing a role in the NASA software risk mitigation strategy.  IV&V is an objective examination of safety and mission-critical software processes and products.  The key parameters for independence are technical independence, managerial independence, and financial independence.  The goal of IV&V is to determine if the right system has been built and that it has been built correctly.  From that perspective IV&V strives to answer the questions:  will the system’s software do what it is supposed to do, will the system’s software not do what it is not supposed to do and will the system’s software respond as expected under adverse conditions.

IV&V leads to higher quality products, reduced risk, greater insight, reduced cost, and knowledge transfer.  IV&V also facilitates the transfer of system and software engineering best practices. IV&V status reports provide another status indicator and performance report to decision-makers (program managers).

In 2012, the Aerospace Safety Advisory Panel recommended that “NASA should establish a standard identifying the level of criticality that requires software IV&V, i.e., at what risk level must IV&V be required and therefore either be resourced or if that is not possible, a formal waiver process be in place for an accountable individual to accept the associated risk and document it.”

The Agency concurred and directed the Office of the Chief Engineer, with assistance from the Office of Safety and Mission Assurance and the NASA IV&V Program, to develop a NASA Interim Directive (NID), now incorporated into NPR 7150.2, that would replace the existing multi-step process for determining which projects must-have software IV&V with a standard identifying the level of criticality that requires software IV&V. 

NPD 7120.4D, NASA Engineering, and Program/Project Management Policy reiterate the point made in the note associated with SWE-141 that, “It is NASA policy to use the NASA Independent Verification and Validation (IV&V) Facility as the sole provider of IV&V services when software created by or for NASA is selected for IV&V by the NASA Chief, Safety and Mission Assurance.”

The earlier IV&V is involved in a project, the better the provided service and value of the results will be given the additional time to understand the project, its risks, and safety-critical aspects.

Expanded criteria definitions

Category 1 projects are defined in NPR 7120.5E, NASA Space Flight Program and Project Management Requirements, as human space flight projects, projects with a life-cycle cost exceeding $1B, or projects with significant radioactive material.

Category 2 projects are defined in NPR 7120.5E as projects that have life-cycle costs greater than $250M and less than $1B or have life-cycle costs less than $250M with a high priority level based on “the importance of the activity to NASA, the extent of international participation (or a joint effort with other government agencies), the degree of uncertainty surrounding the application of new or untested technologies” and a Class A or Class B payload risk classification.

Class A payload risk classifications are defined in NPR 8705.4, Risk Classification for NASA Payloads, as payloads with high priority, very low (minimized) risk, very high national significance, very high to high complexity, greater than 5 year mission lifetime, high cost, critical launch constraints, no alternative or re-flight opportunities, and/or payloads where “all practical measures are taken to achieve a minimum risk to mission success. The highest assurance standards are used.” 

Class B payload risk classifications are defined in NPR 8705.4 as payloads with high priority, low risk, high national significance, high to medium complexity, 2- 5 year mission lifetime, high to medium cost, medium launch constraints, infeasible or difficult in-flight maintenance, few or no alternative or re-flight opportunities, and/or payloads where “stringent assurance standards [are applied] with only minor compromises in application to maintain a low risk to mission success.” 

Per NPR 8705.4, “The importance weighting assigned to each consideration is at the discretion of the responsible Mission Directorate.”

Additional guidance related to IV&V may be found in the following related requirement in this Handbook.

4. Small Projects

All projects are assessed against these criteria to determine whether they should receive IV&V services. 

5. Resources

5.1 References

5.2 Tools

6. Lessons Learned

6.1 NASA Lessons Learned

The NASA Lessons Learned database contains the following lessons learned related to IV&V:

• Independent Verification and Validation of Embedded Software (Use of IV&V Procedures). Lesson Number 723 ⁵¹⁸: "The use of Independent Verification and Validation (IV&V) processes ensures that computer software is developed following original specifications, that the software performs the functions satisfactorily in the operational mission environment for which it was designed, and that it does not perform unintended functions. Identification and correction of errors early in the development cycle are less costly than identification and correction of errors in later phases, and the quality and reliability of software are significantly improved."

• Does Software IV&V Provide Clear Benefits to NASA Projects?  Lesson Number 6656 ⁵⁸⁴: “Recent NASA spaceflight projects that have undergone IV&V of their mission software perceive the process as offering concrete benefits beyond those accrued through VV performed solely by project personnel. The specific benefits accrued to four recent JPL spaceflight projects from participation by the NASA IVV Center are discussed, and some recommendations for future IVV programs are proposed.”

6.2 Other Lessons Learned

• Software IV&V
  • IV&V approaches and resources must align with the program risk posture.
  • A closed-loop, auditable, corrective action toolset and process must be used to manage all IV&V identified issues and risks.

7. Software Assurance

7.1 Tasking for Software Assurance

1. Confirm that IV&V requirements (section 4.4) are complete on projects that are required to have IV&V.

7.2 Software Assurance Products

• Issues and risks if IV&V is not being performed as required by NPR 7150.2. 

  
  

  Objective Evidence

  • Evidence that the confirmation of IV&V involvement is occurring, if applicable.
   Definition of objective evidence

  Objective evidence is an unbiased, documented fact showing that an activity was confirmed or performed by the software assurance/safety person(s). The evidence for confirmation of the activity can take any number of different forms, depending on the activity in the task. Examples are:

  • Observations, findings, issues, risks found by the SA/safety person and may be expressed in an audit or checklist record, email, memo or entry into a tracking system (e.g. Risk Log).
  • Meeting minutes with attendance lists or SA meeting notes or assessments of the activities and recorded in the project repository.
  • Status report, email or memo containing statements that confirmation has been performed with date (a checklist of confirmations could be used to record when each confirmation has been done!).
  • Signatures on SA reviewed or witnessed products or activities, or
  • Status report, email or memo containing a short summary of information gained by performing the activity. Some examples of using a “short summary” as objective evidence of a confirmation are:
    • To confirm that: “IV&V Program Execution exists”, the summary might be: IV&V Plan is in draft state. It is expected to be complete by (some date).
    • To confirm that: “Traceability between software requirements and hazards with SW contributions exists”, the summary might be x% of the hazards with software contributions are traced to the requirements.
  • The specific products listed in the Introduction of 8.16 are also objective evidence as well as the examples listed above.

7.3 Metrics

• None identified at this time

7.4 Guidance

The requirements in the IV&V section (Section 4.4) of NASA-STD-8739.8 apply to all IV&V efforts performed on a software development project.  They also serve as the definition of what NASA considers to be IV&V.  IV&V as a risk mitigation activity, and as such, the application of IV&V analysis and the rigor of that analysis is driven by the IV&V Provider’s assessment of software risk.

IV&V is a technical discipline of SA, which employs rigorous analysis and testing methodologies identifying objective evidence and conclusions to provide an independent assessment of products and processes throughout the life cycle. The independent assessment of products and processes throughout the life-cycle demonstrates whether the software is fit for nominal operations (required functionality, safety, dependability, etc.), and off-nominal conditions (response to faults, responses to hazardous conditions, etc.). The goal of the IV&V effort is to provide assurance conclusions to the Project based on evidence found in software development artifacts and risks associated with the intended behaviors of the software.

The IV&V Provider performs two primary activities, often concurrently: verification and validation. Each of the activities provides a different perspective on the system/software. Verification is the process of evaluating a system and its software to provide objective evidence as to whether or not a product conforms to the build-to requirements and design specifications. Verification holds from the requirements through the design and code and into testing. Verification seeks to demonstrate that the products of a given development phase satisfy the conditions imposed at the start of or during that phase. Validation tasks, on the other hand, seek to develop objective evidence that shows that the content of the engineering artifact is the right content for the developed system/software. The content is accurate and correct if the objective evidence demonstrates that it satisfies the system requirements (e.g., user needs, stakeholder needs, etc.), that it fully describes the required capability/functionality needed, and that it solves the right problem.

Software assurance should assess the IV&V activities being planned and performed against the seven  IV&V requirements, listed below:

The IV&V Provider shall: (SASS-02)

1. Conduct an initial planning and risk assessment effort to determine the specific system/software behaviors (including the software components responsible for implementing the behaviors) to be analyzed, the IV&V tasks to be performed, and the rigor to be applied to the tasks.
2. Develop an IV&V Execution Plan documenting the activities, methods, level of rigor, environments, and criteria to be used in performing verification and validation of in-scope system/software behaviors (including responsible software components) determined by the planning and scoping effort.
3. Provide analysis results, risks, and assurance statements/data to all the responsible organizations’ Project management, engineering, and assurance personnel.
4. Participate in Project reviews of software activities by providing status and results of software IV&V activities including, but not limited to, upcoming analysis tasks, artifacts needed from the Project, the results of current or completed analysis, defects, and risks to stakeholders, customers, and development project personnel.
5. Participate in planned software peer reviews or software inspections and record peer review measurements guided by the planning and scoping risk analysis performed by the IV&V Provider as well as by the content of the items being reviewed or inspected.
6. Identify, analyze, plan, track, communicate, and record risks to the software and development project following NPR 8000.4.
7. Track, record, and communicate defects/issues and other results found during the execution of IV&V analysis during the software development effort to include issues and results found during the conducting of independent IV&V testing.

The IV&V Provider shall verify and validate that the concept documentation represents the delineation of a specific implementation solution to solve the Acquirer’s problem. (SASS-03)

The IV&V Provider shall verify and validate: (SASS-04)

1. That the project implements the requirements for the software listed in NPR 7150.2 (SWE-134) and risk-driven assessments determine the types of IV&V analyses.
2. That the in-scope SW requirements and system requirements are, at a minimum, correct, consistent, complete, accurate, readable, traceable, and testable.
3. The software usually provides the interface between the user and the system hardware as well as the interface between system hardware components and other systems. These interfaces are critical to the successful operation and use of the system.
4. That the mitigations for identified security risks are in the software requirements.

The IV&V Provider shall verify and validate: (SASS-05)

1. That the relationship between the in-scope system/software requirements and the associated architectural elements is traceable correct, consistent, and complete.
2. That the software architecture meets the user’s safety and mission-critical needs as defined in the requirements.
3. That the detailed design products are traceable, consistent, complete, accurate, and testable.
4. That the interfaces between the detailed design components and the hardware, users, operators, other software, and external systems are correct, consistent, complete, accurate, and testable.
5. That the relationship between the software requirements and the associated detailed design components is correct, consistent, and complete.

The IV&V Provider shall verify and validate: (SASS-06)

1. That the software code products are consistent, complete, accurate, readable, and testable.
2. That the software code meets the project software coding standard.
3. That the security risks in the software code are identified and mitigated as necessary.
4. Analysis to assess the source code for the presence of open-source software.
5. That software problem reports generated by the IV&V provider have been addressed entirely by the project.
6. That the project identifies and plans for the security risks in software systems and the security risk mitigations for these systems.
7. That the project assesses the software systems for possible security vulnerabilities and weaknesses.
8. That the project implements and tests the required software security risk mitigations to ensure that security objectives for software are satisfied.
9. The software code through the use of analysis tools (to include but not be limited to static, dynamic, and formal analysis tools) as determined by the IV&V risk analysis process.
10. That the relationship between the software design elements and the associated software units is correct, consistent, and complete.
11. That the relationship between software code components and corresponding requirements is correct, complete, and consistent.

The IV&V Provider shall: (SASS-07)

1. Verify and validate that in-scope test plans, design, cases, and procedures at all levels of testing (unit, integration, system, acceptance, etc.) are correct, complete, and consistent to allow for the verified implementation of software code products as well as system/software capabilities/behaviors.
2. Verify and validate that relationships, between test plans, designs, cases, and procedures and software code products and system/software capabilities/behaviors, are correct, complete, and consistent.
3. Verify that the test plans, designs, cases, and procedures contain objective acceptance criteria that support the verification of the associated requirements for both nominal and off-nominal conditions.
4. Validate that the test environment (including simulations) is complete, correct, and accurate concerning the intended testing objectives.
5. Verify that the software code test results meet the associated acceptance criteria to ensure that the software correctly implements the associated requirements.

The IV&V Provider shall assess the software maintenance plan concerning software elements to support the planning of IV&V activities during the maintenance phase. (SASS-08)

Software assurance considers using an audit approach to determine that the IV&V provider is performing the IV&V functions defined in these requirements.