1. Requirements

4.5.13 The project manager shall develop acceptance tests for loaded or uplinked data, rules, and code that affects software and software system behavior. 

1.1 Notes

These acceptance tests should validate and verify the data, rules, and code for nominal and off-nominal scenarios.

1.2 History

1.3 Applicability Across Classes

Key:    - Applicable | - Not Applicable

2. Rationale

Any uploaded or uplinked data, rules, and code can affect the behavior of the software and/or system. Special acceptance tests should be developed to validate and verify the uplinked or uploaded information for nominal and off-nominal scenarios.

3. Guidance

Acceptance Test (see SWE-034) is a system-level test, usually performed during the final integration of formally tested software with the intended flight or operations hardware before Project level acceptance. Additional information may be found in NPR 7123.1  ⁰⁴¹ and NASA-SP-2007-6105.  ²⁷³

Uploaded or uplinked data, rules, and code may affect the behavior of the software and/or system. Special acceptance tests should be developed to validate and verify the uplinked or uploaded information for nominal and off-nominal scenarios.

Acceptance activities for software development begin with planning and developing acceptance criteria during the Formulation phase of the project. These activities and acceptance criteria can be documented in the Software Development/Management Plan (see 5.08 - SDP-SMP - Software Development - Management Plan ) or a separate Software V&V Plan. Understanding the system grows and the requirements are better understood, the acceptance criteria can be reviewed to ensure they are up to date and consistent with the system being built. They typically conclude with the system acceptance review late in the Implementation phase of the project (see the entrance and exit criteria for the Systems Acceptance Review in Topic 7.09 - Entrance and Exit Criteria)

Acceptance Testing is the formal testing conducted to determine whether a software system satisfies its acceptance criteria, enabling the customer to determine whether or not to accept the system. Acceptance testing is designed to determine whether the software work product is fit for use. Acceptance testing of the software work product typically forms a major portion of the acceptance plan. Once developed, the team runs the acceptance testing, commonly called a test suite, against the supplied input data and conditions. The software testing personnel are typically but not always independent of the project team. Software assurance personnel observe the tests. The team compares the obtained test results with the expected results. If the results match or fall within a previously agreed-to band or tolerance, the test suite is said to pass, and the work product is acceptable. If not, the work product may either be rejected or accepted on conditions previously agreed to between the customer and the software development team.

Verification results that are used in software acceptance reviews are typically documented in software verification reports. Test results for software work products subjected to acceptance testing must be documented in a test report or an acceptance data package. (See 5.11 - STR - Software Test Report and 5.03 - Inspect - Software Inspection, Peer Reviews, Inspections for related information.)

Additional guidance related to acceptance testing may be found in the following related requirements in this handbook: 

4. Small Projects

No additional guidance is available for small projects.

5. Resources

5.1 References

5.2 Tools

6. Lessons Learned

6.1 NASA Lessons Learned

No Lessons Learned have currently been identified for this requirement.

6.2 Other Lessons Learned

Configurable Data Loads (CDL)

• Definition: CDLs contain updateable parameters that are loaded into flight software and can control safety-critical functions.
• Safety-critical data is a shared responsibility between Subsystem Responsible Engineers and Flight Software Team with oversight from Systems Engineering.
• Maintain traceability between data loads and software verification test procedures to support timely verification of late-breaking changes.
• Predefine verification/validation is needed for all CDLs.
• Pre-declare CDL values that are expected/allowed to change with associated nominal verification activities.
  • Changes outside this list need Engineering Control Board approval and must have a verification plan for every change.

Bottom Line: Safety-critical data must be treated with the same rigor as safety-critical software

• Configuration Management
• Verification and Validation

7. Software Assurance

7.1 Tasking for Software Assurance

1. Confirm that the project develops acceptance tests for loaded/uplinked data, rules, and code that affect software and software system behavior.

2. Confirm that the loaded/uplinked data, rules, scripts, or code that affects software and software system behavior is baselined on the software configuration system.

7.2 Software Assurance Products

• None at this time.

  
  

  Objective Evidence

  • Software test reports
  • Software test procedures
  • Software configuration data
   Definition of objective evidence

  Objective evidence is an unbiased, documented fact showing that an activity was confirmed or performed by the software assurance/safety person(s). The evidence for confirmation of the activity can take any number of different forms, depending on the activity in the task. Examples are:

  • Observations, findings, issues, risks found by the SA/safety person and may be expressed in an audit or checklist record, email, memo or entry into a tracking system (e.g. Risk Log).
  • Meeting minutes with attendance lists or SA meeting notes or assessments of the activities and recorded in the project repository.
  • Status report, email or memo containing statements that confirmation has been performed with date (a checklist of confirmations could be used to record when each confirmation has been done!).
  • Signatures on SA reviewed or witnessed products or activities, or
  • Status report, email or memo containing a short summary of information gained by performing the activity. Some examples of using a “short summary” as objective evidence of a confirmation are:
    • To confirm that: “IV&V Program Execution exists”, the summary might be: IV&V Plan is in draft state. It is expected to be complete by (some date).
    • To confirm that: “Traceability between software requirements and hazards with SW contributions exists”, the summary might be x% of the hazards with software contributions are traced to the requirements.
  • The specific products listed in the Introduction of 8.16 are also objective evidence as well as the examples listed above.

7.3 Metrics

• None identified at this time.

7.4 Guidance

When acceptance testing is being done, it is important to ensure that the entire software system will perform correctly in the intended environment for operations. Uploaded or uplinked data, rules, and code form an important part of the system that needs to contain correct data and be working properly for the system to work as intended. Since uploaded or uplinked data, rules, scripts, and code may affect the behavior of the software and/or system, special acceptance tests should be developed to validate and verify the uplinked or uploaded information, including the information for nominal and off-nominal scenarios. Software assurance needs to verify that tests have been developed and successfully run to test these items.

Since the loaded/uplinked data, rules, scripts, and code can be easily changed during operations, it is also important for software assurance to confirm that they are kept under configuration managed and only changed using the configuration management process.