5.5.2 The project manager shall define and implement clear software severity levels for all software non-conformances (including tools, COTS, GOTS, MOTS, OSS, reused software components, and applicable ground systems).
At a minimum, classes should include loss of life or loss of vehicle, mission success, visible to the user with operational workarounds, and an ‘other’ class that does not meet previous criteria.
Click here to view the history of this requirement:
1.3 Applicability Across Classes
Severity is defined as the degree of impact a defect has on the development or operation of the software being tested. A higher effect on the system functionality will lead to the assignment of higher severity to the bug. Severity indicates the seriousness of the defect on the software functionality. These software severity levels should be defined and implemented clearly.
Assigning software severity levels for software non-conformances allows engineers to focus their efforts on problems with the highest potential impact. Severity levels are different for each project based on the criticality of the system to NASA’s major programs and projects; potential risk to astronauts, employees, or the public; and magnitude of the potential loss of government investment. Other project unique characteristics can also affect severity levels of particular non-conformances (e.g., non-conformances that disrupt science data operations, causing missed science opportunities). Project Managers should enlist software engineers, systems engineers, subsystem engineers, and other project personnel to help define severity levels for a particular project based on the project's unique characteristics. This same set of personnel should also participate, as applicable, in reviewing assigned severity levels to non-conformances that are discovered.
Severity levels are particularly important for non-conformances discovered in COTS, GOTS, MOTS, OSS, and reused software components. Most COTS, OSS, and some GOTS, MOTS, and reused software components share lists of known defects and non-conformances. The intent of the requirement is for the user of the software to verify if a site or list of non-conformances or reported bugs is maintained by the developing organization and to review the list of known non-conformances or report bugs to see if the non-conformances or reported bugs could or do impact the software component. Most commercial products and open-source software have a site that shows a list of well know bugs or non-conformances. The requirement is to research the information and see if any of the known bugs impact the software component being used by the project. Non-conformance tracking data can be used to assess the quality and determine the latent risk to the system inherent in using COTS, GOTS, MOTS, OSS, and reused software components.
At a minimum, software non-conformance severity levels should be defined as Major (High) or Minor (Low). Additionally, levels such as Trivial or Editorial can be used for items that do not impact normal operations. More severe non-conformances can be classified as Critical or Unacceptable. As stated earlier, these severity levels are project-specific and should be defined and documented. But keeping the number of definitions to a minimum will simplify tracking and reporting metrics.
The following example of severity definitions is used by one organization at NASA:
P1 - Critical: A critical priority change request is considered to be imperative to the success of the project, and likewise, may have a detrimental impact on the project if not addressed promptly. This type of change request is mandatory and must be completed. The timeframe for estimating the effort and cost required to implement a critical change request should be one (1) week or less. Examples of critical change requests are legal mandates, functionality to meet core business process requirements, or data integrity concerning database content.
P2 - High: A high-priority change request is considered to be important to the success of the project. The timeframe for estimating the effort and cost required to implement a high-priority change request should be two (2) to four (4) weeks. Examples of high-priority change requests are issues and problems resulting from data integrity, legal mandates, and add-ons to improve data quality.
P3 - Medium: A medium priority change request has the potential to impact the successful completion of the project but is not an immediate help or hindrance. The timeframe for estimating the effort and cost required to implement a medium priority change request should be four (4) to six (6) weeks. Examples of medium-priority change requests are requests that improve workflow processes.
P4 - Low: Low priority change requests need to be addressed if the time and budget permit. Low priority changes requests are managed, as resources are available. The timeframe guideline for estimating the effort and cost required to implement a low-priority change request is more than six (6) weeks. Examples of low priority change requests are cosmetic changes or “fixes” that do not affect business functional requirements or deliverables.
4. Small Projects
No additional guidance is available for small projects.
Visible to editors only
Enter the necessary modifications to be made in the table below:
SWEREFs to be added
SWEREFS to be deleted
SWEREFs NOT called out in text but listed as germane: none
SWEREFs called out in text: none
Tools Table Statement
Tools Table Statement
6. Lessons Learned
6.1 NASA Lessons Learned
No Lessons Learned have currently been identified for this requirement.
6.2 Other Lessons Learned
No other Lessons Learned have currently been identified for this requirement.
7. Software Assurance
SWE-202 - Software Severity Levels
SWE-202 - Software Severity Levels
7.1 Tasking for Software Assurance
Confirm that all software non-conformances severity levels are defined.
Assess the application and accuracy of the defined severity levels to software non-conformances.
Confirm that the project assigns severity levels to non-conformances associated with tools, Commerical-Off-The-Shelf (COTS), Government-Off-The-Shelf (GOTS), Modified-Off-The-Shelf (MOTS), Open Source Software (OSS), reused software components, and applicable ground systems.
Maintain or have access to the number of software non-conformances at each severity level for each software configuration item.
7.2 Software Assurance Products
Assessment of Accuracy of Severity-Level Application to Non-conformances
SA assessment of severity levels assigned to software non-conformances, including risks and issues
Information for accessing the number of software non-conformances at each severity level for each software configuration item.
Software defect or problem reporting data
Software configuration management data
Software assurance audit results on the change management process
Software milestone results
Software version description documents
Software control board data or presentations
Definition of objective evidence
SITE:Definition of Objective Evidence
SITE:Definition of Objective Evidence
Total # of Non-Conformances over time (Open, Closed, # of days Open, and Severity of Open)
# of Non-Conformances in the current reporting period (Open, Closed, Severity)
# of Non-Conformances identified in embedded COTS, GOT, MOTS, OSS, or reused components in-ground or flight software vs. # of Non-Conformances successfully closed
# of Non-Conformances identified in source code products used (Open, Closed)
# of software Non-Conformances at each Severity level for each software configuration item (Open, Closed)
Software assurance confirms that the project has defined and documented severity levels for non-compliances before the start of any testing. Typically, NASA severity levels are from 1 to 5 or 1 to 4 with 1 being the most severe. Some example definitions are found in the software requirement guidance for SWE-202 (this requirement). The project definitions of the severity levels can be documented in several places. In many cases, they may be documented in either the test plans or in the test procedures. Often a project will build the definitions of the severity levels in the discrepancy tracking tool being used to track the errors found during testing. Severity levels may not be used officially for unit testing, but even there it is important to know which errors are the most critical and which should be fixed first. Severity levels should be used for all other levels of testing.
Once the project begins testing, software assurance needs to be monitoring the activity by either witnessing tests or reviewing test report documentation. Check to be sure that all the non-conformances that are found are documented in the project discrepancy report system along with the severity level of the non-conformance. Review the assigned severity level to ensure that severity levels are being assigned as per the project definitions of the categories of non-conformances.
Software assurance also needs to review the developer maintained lists of non-conformances found or reported in COTS, MOTS, GOTS, OSS, and other reused software. These non-conformances should have been assigned the appropriate severity levels that match the project definitions for non-conformances occurring in the project software. (In other words, don’t just assume the severity levels recorded in received lists of reused software will reflect the same severity level when in use in the project software).
Software assurance confirms that the project metrics are kept on the number of non-conformances found at each severity level for each configuration item andiron-conformances should be tracked to closure. Other useful information to have is the status of the non-conformances (i.e., found, fix identified, fix implemented, fix tested/closed).
SA Tasking for SWE-201 is related to this requirement and additional information that needs to be captured on the non-conformances is below: