1. Requirements

4.1.6 The project manager shall identify, initiate corrective actions, and track until closure inconsistencies among requirements, project plans, and software products. 

1.1 Notes

NPR 7150.2, NASA Software Engineering Requirements, does not include any notes for this requirement.

1.2 History

1.3 Applicability Across Classes

If Class D software is safety-critical, this requirement applies to the safety-critical aspects of the software.

Key:    - Applicable | - Not Applicable

2. Rationale

Requirements are the basis for a project. They identify the need to be addressed, the behavior of the system, and the constraints under which the problem is to be solved. They also specify the product to be delivered by a provider of software. Software project plans describe how the project will create a software product that fulfills those requirements. Any inconsistencies among these three - requirements, project plans, and software products – will most likely result in a product that does not meet its requirements. Corrective actions address not only new or changed requirements, but also failures and defects in the work products (requirements, project plans, and software products). Corrective actions need to be analyzed to determine the impact that the change will have on the work product, related work products, budget, and schedule. Corrective actions need to be tracked until closure to ensure decisions regarding the closure of those actions are followed through and to prevent the problems described in those corrective actions from propagating through the project or recurring.

3. Guidance

Typically, the corrective action process is described in a plan, such as the software development or software management plan (5.08 - SDP-SMP - Software Development - Management Plan) or the software quality assurance plan (Software Assurance Plan). Follow this documented process to capture and track closure inconsistencies among requirements, plans, and software products.

A recommended practice for this requirement is that all corrective actions be formally submitted descriptions of the desired modifications to work products. If corrective actions are not documented consistently, they are difficult to analyze and track. A database provides a flexible environment for storing and tracking corrective actions.

Identify inconsistencies among requirements, project plans, and software products

Suggested activities to identify inconsistencies and "ensure that project plans and work products remain aligned with requirements" ¹⁵⁷, include:

• "Review project plans, activities, and work products for consistency with requirements and changes made to them." ¹⁵⁷
• "Identify the source of the inconsistency." ¹⁵⁷
• "Identify any changes that should be made to plans and work products resulting from changes to the requirements baseline." ¹⁵⁷

When looking for inconsistencies, consider these "warning signs" or potential causes:

• Unstable requirements.
• Unclear, incomplete, inconsistent, non-cohesive requirements.
• Incomplete project plans or plans developed before requirements stabilize.
• Budgetary issues that result in changes to staff and/or the requirements.
• Inadequate communications among the development team, management, customers, contractors, other stakeholders.
• Inadequate change or configuration management procedures.
• Inexperienced staff.
• Personnel turnover.

Inconsistencies among plans, requirements, and the resulting software products need to be identified throughout the project life cycle.  One way to ensure this activity is not forgotten is to conduct periodic reviews of requirements against project plans and software products. 

Initiate corrective actions and track until closure

Before corrective action can be taken, consider performing analysis to identify and weigh options when the corrective action is not readily apparent. This analysis could be similar to that used for change requests and problem reports (see 5.01 - CR-PR - Software Change Request - Problem Report).

Sample corrective actions include:

• Split development into multiple releases, addressing unstable requirements later.
• Use more experienced or senior personnel.
• Stabilize project personnel, i.e., reduce turnover.
• Audit project processes and act on the findings.

To initiate corrective actions and track them to closure, consider the following for implementation on the project:

• A system or process for entering and tracking corrective actions (e.g., Problem Report and Corrective Action (PRACA), change request/problem reporting system or tool).
• A corrective action review process typically involves a review panel or board including engineers and assurance personnel (e.g., Configuration Control Board (CCB)).
• A corrective action implementation process.
• A time frame for resolution (i.e., number of days or weeks).
• An escalation procedure (e.g., report to Project Manager (PM)).
• A procedure to archive actions and conclusions for reference, input to future projects.

Additional guidance related to reporting and tracking corrective actions to closure may be found in the following related requirement in this handbook:

4. Small Projects

No additional guidance is available for small projects.

5. Resources

5.1 References

5.2 Tools

6. Lessons Learned

6.1 NASA Lessons Learned

The NASA Lessons Learned database contains the following lessons learned related to ensuring plans and products remain aligned with requirements:

• Risk Assessment in Software Development Projects (Uncertainty caused by changing requirements.) Lesson Number 1321 ⁵⁴⁹: "Even with expert and experienced programmers, each new software program presents new technical challenges. Changing methodology and requirements during the design phase of a software project adds uncertainty to the project."
• Deficiencies in Mission Critical Software Development for Mars Climate Orbiter (1999) (Inconsistency led to the loss of mission.) Lesson Number 0740 ⁵²¹: "The root cause of the MCO mission loss was an error in the "Sm_forces" program output files, which were delivered to the navigation team in English units (pounds-force seconds) instead of the specified metric units (Newton-seconds). Comply with preferred software review practices, identify software that is mission-critical (for which staff must participate in major design reviews, walkthroughs, and review of acceptance test results), train personnel in software walkthroughs, and verify consistent engineering units on all parameters.

6.2 Other Lessons Learned

No other Lessons Learned have currently been identified for this requirement.

7. Software Assurance

7.1 Tasking for Software Assurance

1. Monitor identified differences among requirements, project plans, and software products to confirm they are addressed. 

7.2 Software Assurance Products

• List of any inconsistencies identified among products, including risks and issues

  
  

  Objective Evidence

  • Document change management system results.
  • Software problem reporting or defect tracking system results.
   Definition of objective evidence

  Objective evidence is an unbiased, documented fact showing that an activity was confirmed or performed by the software assurance/safety person(s). The evidence for confirmation of the activity can take any number of different forms, depending on the activity in the task. Examples are:

  • Observations, findings, issues, risks found by the SA/safety person and may be expressed in an audit or checklist record, email, memo or entry into a tracking system (e.g. Risk Log).
  • Meeting minutes with attendance lists or SA meeting notes or assessments of the activities and recorded in the project repository.
  • Status report, email or memo containing statements that confirmation has been performed with date (a checklist of confirmations could be used to record when each confirmation has been done!).
  • Signatures on SA reviewed or witnessed products or activities, or
  • Status report, email or memo containing a short summary of information gained by performing the activity. Some examples of using a “short summary” as objective evidence of a confirmation are:
    • To confirm that: “IV&V Program Execution exists”, the summary might be: IV&V Plan is in draft state. It is expected to be complete by (some date).
    • To confirm that: “Traceability between software requirements and hazards with SW contributions exists”, the summary might be x% of the hazards with software contributions are traced to the requirements.
  • The specific products listed in the Introduction of 8.16 are also objective evidence as well as the examples listed above.

7.3 Metrics

• # of Corrective Actions (CAs) raised by SA vs. total #
• CA Attributes (Type, Severity, # of days Open, Life-cycle Phase Found)
• CA State (Open, In work, Closed)
• Trends of CA Open vs. Closures over time
• # of incorrect, missing, and incomplete requirements (i.e., # of requirements issues) vs. # of requirements issues resolved
• Trend the # of inconsistencies or corrective actions identified, and # closed.
• # of software work product Non-Conformances identified by life-cycle phase over time

7.4 Guidance

1. Analyze the different project planning documents and confirm that all inconsistencies are documented and addressed. Items that are often overlooked include some of the following:

• A set of requirements with estimated effort too great for the allotted time and effort planned in the schedule (Also, budget and schedule mismatch with requirements)
• Items planned for implementation or purchase too late for the "need" date for integration
• Development schedules not aligned with "need" dates
• Assurance activities planned that don't align with project phases or activities
• Requirements needing expertise not planned or coordination with another group or development that has not been planned

Any inconsistencies identified during SA review of project plans, requirements, and software products should be documented along with suggested corrective actions, submitted to the project management, and tracked to closure. As the project progresses, inconsistencies among different project products should be kept in mind, particularly during reviews and meetings, and continually identified and addressed as needed

2. Software assurance should analyze all proposed changes for impacts, looking closely at any impacts the change may have in any of the software related to safety or security. The analysis should also consider whether there will be any impacts on existing interfaces or the use of any COTS, GOTS, MOTS, or reused software in the system and whether the change will impact any future maintenance effort. Any identified risks should be brought up to discuss the approval/rejection of the change.

Examples:

• Ensure that the software requirements are met by the design and code.
• Ensure that the planned activities in the project software plans are followed and completed
• Ensure that the software products meet the requirements, conform to the software plans, and meet the acceptance criteria 
• Ensure that the software code meets the software requirements and does not contain features and capabilities that are not included in the requirements, make sure that all software features and capabilities are tested.

3. Confirm:

• That the project tracks the changes

Software assurance will check to see that any changes that are submitted are properly documented and tracked through all the states of resolution (investigation, acceptance/rejection, implementation, test, closure) in the project tracking system.

• That the changes are approved and documented before implementation

Software assurance should track the changes from their submission to their closure or rejection. Initially, SA should confirm that all changes follow the change management process that the project has established. Initially, the change will be documented and submitted to the authorizing CCB for consideration. The authorizing CCB (which will include a software assurance person) will evaluate any changes for impacts.

If the software is safety-critical, the responsible software assurance personnel will perform software safety change analysis to evaluate whether the proposed change could invoke a hazardous state, affect a control for a hazard, condition, or state, increase the likelihood or severity of a hazardous state, adversely affect safety-critical software, or change the safety criticality of an existing software element.  Keep in mind that changes to the hardware or the software can impact the overall system’s safety and while the focus is on software changes, the software also needs to be aware of changes to the hardware that may impact how software controls, monitors, and analyzes inputs from that hardware.  Hardware and software changes can alter the role of software from non-safety critical to safety-critical or change the severity from moderate to critical. 

Some other considerations for the evaluation of changes:

• • • Is the change an error correction or a new requirement?
    • Will the change fix the problem without major changes to other areas?
    • If major changes to other areas are needed, are they specified, and is this change really necessary?
    • If the change is a requirements change, has the new requirement been approved?
    • How much effort will be required to implement the change?
    • If there is an impact on safety or reliability, are there additional changes that need to be made in those areas? Note: If there is a conflict between safety and security, safety changes have priority.

When all the impacts are considered, the CCB votes on acceptance/rejection. Software assurance is a voting member of the CCB.  Software assurance verifies that the decision is recorded and is acceptable, defined as:

• When the resolution is to “accept as is”, verify that the impact of that resolution on quality, safety, reliability, and security is compatible with the Project’s risk posture and is compliant with NPR 7150.2 and other Center and Agency requirements for risk.
• When the resolution is a change to the SW, the change will sufficiently address the problem and will not impact quality, safety, reliability, security, and compliance with NPR 7150.2; the change will not introduce new or exacerbate other, discrepancies or problems.
• In either case, the presence of other instances of the same kind of discrepancy/problem has been sought out and, if detected, addressed accordingly.
• Verify that appropriate software severity levels are assigned and maintained.
• Assure any risk associated with the change is added to the Project/facility risk management system and is addressed, as needed, in safety, reliability, or other risk systems
• That the implementation of the changes is complete

Software assurance will check to see if the implementation of the approved changes has been coded as per the change request. Check to see that any associated documentation changes are submitted/approved and/made as needed (i.e., updates to requirements, design, test plans/procedures, etc.)

• That the project tests the changes

Software assurance will check to see that the project test any of the code that has changed and runs a set of regression tests to see that the change has not caused a problem anywhere else in the software system. If the software is safety-critical, a full set of regression tests should be run to ensure that there was no impact on the safety-critical functions.

4. Confirm software changes are done in the software control process

Software assurance will check that the software control process has been followed throughout the handling of the submitted change and that the status of the change is  recorded and confirmed as closed