bannerc
SWE-034 - Acceptance Criteria

1. Requirements

3.1.5 The project manager shall define and document the acceptance criteria for the software. 

1.1 Notes

NPR 7150.2, NASA Software Engineering Requirements, does not include any notes for this requirement.

1.2 History

SWE-034 - Last used in rev NPR 7150.2D

RevSWE Statement
A

2.5.3 The project shall define and document or record the acceptance criteria and conditions for the software.

Difference between A and B

No change

B

3.12.3 The project manager  shall define and document the acceptance criteria and conditions for the software.

Difference between B and C

Removed "conditions" from requirement.

C

3.1.5 The project manager shall define and document the acceptance criteria for the software. 

Difference between C and DNo change
D

3.1.5 The project manager shall define and document the acceptance criteria for the software. 



1.3 Applicability Across Classes

Class

     A      

     B      

     C      

     D      

     E      

     F      

Applicable?

   

   

   

   

   

   

Key:    - Applicable | - Not Applicable

2. Rationale

Software development teams can better organize their development tasks if they have well-defined plans for software activities. Customers and project personnel must include pre-determined criteria in these plans to evaluate the software work products. These criteria include clearly understanding what the customer wants the product to do and what constitutes an acceptable software product. The development team documents the agreed-to acceptance criteria and the planned activities in a software acceptance plan contained in the Software Development/Management Plan (see 5.08 - SDP-SMP - Software Development - Management Plan) or a separate Verification and Validation (V&V) Plan, as a formal means of guiding these activities.

3. Guidance

"Acceptance criteria" are defined as (1) those criteria that a system or component must satisfy to be accepted by a user, customer, or other authorized entity (ISO/IEC/IEEE 24765:2010 Systems and software engineering – Vocabulary 230), and, (2) those criteria, including performance requirements and essential conditions, which must be met before project deliverables are accepted (A Guide to the Project Management Body of Knowledge 302). The purpose of this requirement is to cause the software development team to work with the customer upfront to define the criteria to be used for software acceptance. The acceptance criteria and timing can help to do the following:

  • Define the software support manpower levels on a project.
  • Determine the "handover" for the software development organization to a software operation and maintenance organization.
  • Help define what a contractor is to complete before flight certification can be achieved.

The software acceptance criteria definition is a key element in the overall software planning process. The software acceptance criteria need to address both software and data. If the software work product is delivered in phases, each delivery may have its acceptance criteria.

Acceptance activities for software development begin with the planning and the development of acceptance criteria during the Formulation phase of the project. These activities and acceptance criteria can be documented in the Software Development/Management Plan (see 5.08 - SDP-SMP - Software Development - Management Plan ) or a separate Software V&V Plan. As an understanding of the system grows, and the requirements are better understood, the acceptance criteria can be reviewed to assure they are up to date and consistent with the system being built. They typically conclude with the system acceptance review late in the Implementation phase of the project (see the entrance and exit criteria for the Systems Acceptance Review in 7.09 - Entrance and Exit Criteria )

For software acquired via the acquisition process (see 7.03 - Acquisition Guidance ) the acceptance criteria development follows this guidance. The final criteria must be put into the contract statement of work.

The paragraphs below deal with the acceptance criteria creation and documentation. Acceptance testing, which generates the information used to assess the satisfaction of the acceptance criteria, is also discussed.

Establish Acceptance Criteria

The software lead engineer works with the customer and the software development team to develop the appropriate acceptance plans and activities to ensure that functional requirements are properly translated into system acceptance criteria. A well-defined acceptance plan will help the software development team and the software assurance team to identify the resources required to properly support the effort. This, in turn, contributes to efficient and cost-effective development and V&V activities, where many of the acceptance activities occur. The team defines acceptance criteria during the Formulation phase of the software development life cycle to assure sufficient time to prepare for and perform the acceptance activities.

The software development team and the customer work together to ensure that they do the following:

  • Identify interim and final products that will be part of acceptance activities.
  • Develop the acceptance criteria and activities schedule.
  • Plan how and by whom each acceptance activity will be performed.
  • Schedule adequate time for the customer to examine and review the product.
  • Prepare the acceptance plan.
  • Perform formal acceptance testing at scheduled times.
  • Plan the decision-making process that is based on the results of acceptance testing.

The software lead engineer works with the software team (including the software assurance personnel) to develop appropriate product reviews, tests, and any audits necessary. This work includes identifying:

  • The types of criteria to consider, such as customer expectations and requirements, technology limitations, environmental impact, safety, risks, total ownership, and life-cycle costs, and schedule impact.
  • The variations in criteria for computer software configuration items (CSCI), units, and or systems.
  • The acceptable range of the criteria.
  • The rank of each criterion by its importance.

Acceptance criteria may include:

  • Product V&V completed successfully.
  • V&V of individual products, integration of products into systems, and that system V&V has been performed or witnessed by the technical team.
  • The technical data package is current (as-built) and complete.
  • Transfer of certifications, warranties, or representations is complete.
  • Transfer of software products, licenses, data rights, intellectual property rights, etc., is complete.
  • If an acquisition: Technical documentation required in contract clauses is complete (e.g., new technology reports).
  • Correctness criteria (e.g., round-off, accuracy, precision).
  • Performance criteria (e.g., speed, time).
  • Throughput criteria.
  • System availability.
  • System reliability.
  • Input/output (I/O) performance.
  • Memory performance.
  • System communications.
  • Networking capability.
  • Software compatibility.
  • System maintenance (component level/systems level) plan availability.
  • Documentation availability.
  • Readiness for the software release. 373

Acceptance Testing

Acceptance Testing is the formal testing conducted to determine whether a software system satisfies its acceptance criteria, which enables the customer to determine whether or not to accept the system. Acceptance testing is designed to determine whether the software work product is fit for use. Acceptance testing of the software work product typically forms a major portion of the acceptance plan. Once developed, the team runs the acceptance testing, commonly called a test suite, against the supplied input data and conditions. The team typically uses an acceptance test procedure to direct the software testing personnel. The software testing personnel are typically but not always independent of the project team. Software assurance personnel observe the tests. The team compares the obtained test results with the expected results. If the results match or fall within a previously agreed-to band or tolerance, the test suite is said to pass, and the work product is acceptable. If not, the work product may either be rejected or accepted on conditions previously agreed to between the customer and the software development team.

The planning for acceptance testing may consider:

  • Acceptance period (is it open? is it constrained?).
  • Diagnostics tests (are they part of the software requirements specification?).
  • Functionality tests (what is the software work product designed to do?).
  • Use of the latest production version of code (are earlier versions acceptable?).
  • Differences/discrepancies (when are they acceptable? or the cause for an issue and corrective action activity?).
  • Availability and readiness of the test environment.
  • Availability of test and software assurance personnel to support the testing.

Acceptance testing activities include:

  • Alpha testing takes place at developers' sites and involves testing of the operational system by internal staff before it is released to external customers.
  • Beta testing takes place at customers' sites and involves testing by a group of customers who use the system at their locations and provide feedback before the system is released to other customers. This is often called "field testing."
  • System testing is invariably performed by the development team or preferably by someone independent of the developer
  • User Acceptance Testing (UAT) is a process to obtain confirmation that a system meets mutually agreed-upon requirements. In software development, UAT is one of the final stages of a project and often occurs before a customer accepts the new system

Acceptance tests may also be used as regression tests before a production release. This means that new acceptance tests must be created for each iteration of the software build.

All software products should have acceptance criteria defined, this includes items like software documents, software code, databases, software models, and software defect and change reports. The acceptance criteria can be documented in the software development or management plans or the contract documentation, an example of acceptance criteria in contract documentation could be the Data Requirements Documents (DRDs). Criteria for software acceptance testing are often documented in the software test plans. 

 After a software work product is designed, coded, and tested against its requirements, if any deviations from the requirements and acceptance criteria still exist, they will have to be negotiated with the customer to determine if they can be accepted, or if they must be fixed before the customer accepts the product. The customer must review and agree to the acceptance test plan.


The entrance criteria and the exit (success) criteria are developed and documented during the acceptance planning activities (see topic 7.09 - Entrance and Exit Criteria) for the System Acceptance Review.


Additional guidance related to acceptance testing may be found in the following related requirements in this handbook:

4. Small Projects

No additional guidance is available for small projects. The community of practice is encouraged to submit guidance candidates for this paragraph.

5. Resources

5.1 References


5.2 Tools

Tools to aid in compliance with this SWE, if any, may be found in the Tools Library in the NASA Engineering Network (NEN). 

NASA users find this in the Tools Library in the Software Processes Across NASA (SPAN) site of the Software Engineering Community in NEN. 

The list is informational only and does not represent an “approved tool list”, nor does it represent an endorsement of any particular tool.  The purpose is to provide examples of tools being used across the Agency and to help projects and centers decide what tools to consider.

6. Lessons Learned

6.1 NASA Lessons Learned

No Lessons Learned have currently been identified for this requirement.

6.2 Other Lessons Learned

No other Lessons Learned have currently been identified for this requirement.

7. Software Assurance

SWE-034 - Acceptance Criteria
3.1.5 The project manager shall define and document the acceptance criteria for the software. 

7.1 Tasking for Software Assurance

  1. Confirm software acceptance criteria are defined and assess the criteria based on guidance in the NASA Software Engineering Handbook, NASA-HDBK-2203.

7.2 Software Assurance Products

  • Software Assurance acceptance criteria.
  • Software Design Analysis.
  • Source Code Analysis.
  • Verification Activities Analysis
  • Assessment of Software Reviews results
  • Evidence that software acceptance criteria exist.
  • Assessment that acceptance criteria for Software Engineering are reasonable per Topic 7.18 - Documentation Guidance of this Handbook, including corrective actions.


    Objective Evidence

    • Definition of the software acceptance criteria for each software product

    Objective evidence is an unbiased, documented fact showing that an activity was confirmed or performed by the software assurance/safety person(s). The evidence for confirmation of the activity can take any number of different forms, depending on the activity in the task. Examples are:

    • Observations, findings, issues, risks found by the SA/safety person and may be expressed in an audit or checklist record, email, memo or entry into a tracking system (e.g. Risk Log).
    • Meeting minutes with attendance lists or SA meeting notes or assessments of the activities and recorded in the project repository.
    • Status report, email or memo containing statements that confirmation has been performed with date (a checklist of confirmations could be used to record when each confirmation has been done!).
    • Signatures on SA reviewed or witnessed products or activities, or
    • Status report, email or memo containing a short summary of information gained by performing the activity. Some examples of using a “short summary” as objective evidence of a confirmation are:
      • To confirm that: “IV&V Program Execution exists”, the summary might be: IV&V Plan is in draft state. It is expected to be complete by (some date).
      • To confirm that: “Traceability between software requirements and hazards with SW contributions exists”, the summary might be x% of the hazards with software contributions are traced to the requirements.
    • The specific products listed in the Introduction of 8.16 are also objective evidence as well as the examples listed above.

7.3 Metrics

  • # of Correct Actions (CAs) raised by SA vs. total #
    • CA Attributes (Type, Severity, # of days Open, Life-cycle Phase Found)
    • CA State (Open, In work, Closed)
    • Trends of CA closures over time

7.4 Guidance

Step 1 Determine that the project or engineering has documented acceptance criteria for the software products.

Step 2 Assess the acceptance criteria based on guidance in the NASA Software Engineering Handbook, NASA-HDBK-2203.

Step 3 Develop acceptance criteria for all software assurance and software safety products planned to be produced for the project.

Step 4 Assess both software engineering and software assurance acceptance criteria for reasonableness.

Guidance on Perform development of acceptance criteria for all software assurance products.

Software assurance teams can better organize their development tasks if they have well-defined plans for software assurance and software safety activities. Customers and project personnel must include pre-determined criteria in these plans to evaluate software assurance and software safety work products. These criteria include clearly understanding what the customer wants the product to do and what constitutes an acceptable software assurance and software safety product. The development team documents the agreed-to acceptance criteria and the planned activities in a software assurance and software safety plan.

The software assurance and software safety team and the customer work together to ensure that they do the following:

  • Identify interim and final products that will be part of acceptance activities.
  • Develop the acceptance criteria and activities schedule.
  • Plan how and by whom each acceptance activity will be performed.

Acceptance criteria may include:

  • The product was completed successfully.
  • The technical data package is current (as-built) and complete.
  • If an acquisition: Technical documentation required in contract clauses is complete (e.g., new technology reports).
  • Correctness criteria.
  • Risk criteria.
  • Reliability.
  • Communications process.
  • Documentation and data availability.
  • Readiness for software milestones.

If using an Agile or incremental development process, then code verification and verification of Sprint and daily tasks need to be assured within the Sprint time frame.  At the end of the Sprint, the code and any supporting products such as documentation and test suites needed to meet the definition of “Done” as determined by the Project are saved.  Daily regression testing needs to be updated with new features and functions. The definition of “Done” for each Sprint should be chosen so that the final software system will meet the acceptance criteria

In an Agile development environment where testing is a daily activity, and the test cases, test suites, grow in step with the implementation of the story/function points, periodic examination of the tests should be performed for required reliability and safety tests as well as performance. At the end of each sprint, a working code segment is to be completed and shown to work via testing and demonstration. When sprint teams are integrating working code segments within a sprint team or among multiple sprint teams, they need to show working code not only performs but also meets required reliability, quality, and safety factors. The Software Assurance members of the sprint teams need to assure the testing is thorough and that tests are captured and that both the code and tests are configuration managed. The final integrated build of the system must still meet the acceptance criteria.


  • No labels