bannerd

This topic is generic and applies to all SWEHB versions. All links to version specific SWEs and Topics have been removed. SWEs and Topics are in Green Bold Underlined text to make them easy to find. 

Refer to the appropriate Requirements or Topics buttons in the SWEHB version you are using to locate the SWE or Topic you need.

8.24 - Software Assurance Risk

1. Introduction and Chart

This chart summarizes SA Risks by Risk Phases

Legends

Phase / Area 

  • Prog - Programmatic 
  • Req - Requirements
  • Design - Design
  • Code
  • Test - Testing
  • Ops - Operations

Risk Levels 

Risk levels are color coded for the reviews in which they are discussed. 

Review Types

MCR = Mission Concept Review,SRR = System Requirements ReviewMDR = Mission Definition Review
SDR = System Definition ReviewPDR = Preliminary Design ReviewCDR = Critical Design Review
SIR = System Integration ReviewTRR = Test Readiness ReviewSAR = System Acceptance Review
ORR = Operational Readiness Review

All data is also available in an Excel Spreadsheet: Software Assurance Risks v3.xlsx

1.1 Development Risks

Click on a header to sort. 

Risk 

Phase / Area

M
C
R

S
R
R

M
D
R

S
D
R

P
D
R

C
D
R

S
I
R

T
R
R

S
A
R

O
R
R

More than 20% of the code has not been through a code peer review 

Code peer reviews are one of the best ways to identify errors and problems in the code when the reviews follow practices such as including people with different roles in the project participate, and spend adequate time reviewing the code in advance of the meetings. This technique should be used particularly on the mission critical code to reduce the error or problems with the code. This technique supplements the use of tools and testing for errors, and often allows the errors to be found earlier when they are cheaper to fix. Generally fewer errors and problems are found in critical code critical code when a higher percentage  has been peer reviewed. A substantial body of data and experience justifies the use of inspections on requirements. Finding and fixing requirements problems during requirements analysis is cheaper than doing so later in the life cycle and is substantially cheaper than finding and fixing the same defects after delivering the software.

Code









Incomplete implementation of static code analysis results or missing static analysis testing by development team. No or limited static analysis being used in development and testing of the software.

The static analysis for NASA software projects increases the quality and safety of code developed for NASA Missions. Using static analysis helps to ensure that code meets the coding standards/criteria established by the project team and finds common coding errors that need to be are eliminated before system integration and test . Studies show that the cost of catching errors dramatically increases from one phase of development to the next by roughly a factor of 10. Eliminating errors during implementation results in cost and time savings during integration and testing, which is a particularly important cost-saving factor for projects using high-fidelity testbeds.

Code









Software code is not in a configuration management system before the start of software testing. 

Software Configuration Management planning encompasses the practices and procedures for administering source code, producing software development builds, controlling change, and managing software configurations for all of the software products, tools, data, and components produced by the project.

Code









Software products are engineered under the practice of using processes to improve the quality of a software development effort and a methodical approach to software development results in fewer defects. Significant evidence and multiple findings from the software assurance audits that the software processes are not being followed by the software development team(s). The increased complexities and huge size of the software development projects make software development hard to control without formal practices and standard measurements.

The risk that the NASA projects are not being supported by software development organization(s) that have the necessary skills and processes in place to produce reliable products within cost and schedule estimates

Code









Not using a secure coding standard. Risk that the software has security vulnerabilities that could compromise software security.

Unsafe coding practices result in costly vulnerabilities in application software that leads to the theft of sensitive data.   Some secure practices include but not limited to the strict language adherence and use of automated tools to identify problems either at compile time or run time; language-specific guidance, domain-specific guidance, local standards for code organization and commenting, and standard file header format are often specified as well.

Code









Lack of coding standards or insufficient use of the coding standard for developing safety-critical software for use in critical, real-time operations; coding standards not enforced due to poor software development practices; and/or lack of code reviews or lack of static/dynamic code analysis tools to find code defects. A coding standard gives a uniform appearance to the codes written by different engineers. It improves readability, and maintainability of the code and it reduces complexity also. It helps in code reuse and helps to detect error easily. It promotes sound programming practices and increases efficiency of the programmers.

Without using a good coding standard. code will not have a uniform appearance when written by different engineers. Coding standards improves readability, and maintainability of the code and it reduces complexity. It helps in code reuse and helps to detect error easily. It promotes sound programming practices and increases efficiency of the programmers.

Code









Existence of any compiler warnings. Compiler warnings are messages produced by a compiler regarding program code fragments to be considered by the developer, as they may contain errors.

Warnings report other unusual conditions in your code that may indicate a problem, although compilation can (and does) proceed.

Code









Software common cause not addressed (if applicable). . Credible risk of common cause failures exists for fail-silent and erroneous-output cases. In human spaceflight systems, software common cause errors introduce a risk such that a single failure in flight software can result in a catastrophic loss-of-vehicle-control hazard. 

A common-mode primary Flight Software fault due to a coding/logic error, a processor resource overrun, a database error, or computer virus may result in Loss of Vehicle (LOV). The risk is especially high during dynamic flight phases when there is a short time-to-effect, or when faster than-human response times are required. Safe Mode coverage during on-orbit quiescent phases of flight, as well as backup flight software (BFS) coverage during, entry, descent and landing (EDL) operations, provide additional layers of protection against LOV in the event of a common-mode primary Flight Software fault.

Code









Software code/test coverage percentages for all identified safety-critical components is less than ( CDR 80%, SIR 90%. TRR 100%). 

Risk is incomplete testing, all Safety-critical software decisions must be tested to protect against loss of crew or vehicle.

Code




80%

90%

100%



# of Cybersecurity vulnerabilities and weaknesses identified by tool exceeds 5

Cybersecurity vulnerabilities and weaknesses identified in a system indicate that there are insecure areas in the code where critical information could be altered or where control functions  could be taken over or altered. Vulnerablies and weaknesses that are found need to be mitigated to avoid security problems. The more vulnerabilities and weaknesses that are identified, the higher the risk that security issues may cause problems with the system's operations.

Code









Use of different compliers for test code than for flight code

Risk associated with using a different compiler for test code is that the test code may not be the same as the flight code.  Using a different compiler may miss complier problems in the flight code.  Test code may be different than flight code.

Code









Coding standard violations identified

Coding standards help ensure safety, security, reliability, quality, maintainability, readability, and testability of the NASA code products. NASA programs and projects have multi-year life cycle times. Often the software personnel who develop the original software work products move on to other projects. These developers are then backfilled on the team by other developers for the remainder of the development, operations, maintenance, and disposal phases of the life cycle. This personnel turnover process may occur several times during the project's life cycle. The use of uniform software coding methods, standards, and/or criteria ensures uniform coding practices, reduces errors through safe language subsets, and improves code readability. Verification that these practices have been adhered to reduces the risk of software malfunction for the project during its operations and maintenance phases. There are five key benefits of using coding standards:

  1. Understanding, maintainability, and readability of the code;
  2. Consistent code quality — no matter who writes the code;
  3. Software security from the start;
  4. Reduced development costs;
  5. Testability of the code.

Coding standards help prevent or reduce unsafe coding practices such as defect-prone coding styles, security issues from specific coding sequences, and numerous coding errors, mistakes, and misunderstandings. 

Code









Any safety-critical components that exceed the Software cyclomatic complexity of 15

The risk is incomplete software testing and reduced reliability associated with safety-critical software code components. The intent is to minimize risk, minimize testing, and increase reliability associated with safety-critical software code components, thus reducing the chance of software failure during a hazardous event. A section of source code's cyclomatic complexity is the number of linearly independent paths within it—where "linearly independent" means that each path has at least one edge that is not in one of the other paths.

Code









software volatility metric is greater than (at TRR 10%. at SIR 20%, at CDR 40%)

Requirements volatility is one of the leading causes of the software development effort not completing on schedule and budget.  Software requirements volatility is one key factor in assessing the status of a software project.  The later in the project requirements changes occur, the more impact those changes can have on project completion on time and within budget. 

Code




40%

20%

10%



# of static code errors and warnings identified as “positives” vs. # of total errors and warnings identified by tool exceeds TBD

The software should have demonstrable correctness properties, supported by evidence from the use of appropriate static analysis techniques.

Code









Unit test results should follow test procedures and be repeatable.

Unit test procedures are to be repeatable so that future runs can confirm that any identified flaws have been corrected and for regression purposes to ensure that any new changes do not introduce new flaws in the software. Unit testing can be described as the confirmation that the unit performs the capability assigned to it, correctly interfaces with other units and data, and represents a faithful implementation of the unit design.

Code









Any % of the Safety/Mission Critical code that has not been through a peer review 

Code peer reviews are one of the best ways to identify errors and problems in the code when the reviews follow practices such as including people with different roles in the project participate, and spend adequate time reviewing the code in advance of the meetings. This technique should be used particularly on the mission critical code to reduce the error or problems with the code. This technique supplements the use of tools and testing for errors, and often allows the errors to be found earlier when they are cheaper to fix. Generally fewer errors and problems are found in critical code critical code when a higher percentage  has been peer reviewed. A substantial body of data and experience justifies the use of inspections on requirements. Finding and fixing requirements problems during requirements analysis is cheaper than doing so later in the life cycle and is substantially cheaper than finding and fixing the same defects after delivering the software.

Code









Missing hardware support at software peer reviews

Risk is incorrect software control of hardware components. Peer reviews are focused, in-depth technical reviews that support the evolving design and development of a product, including critical documentation or data packages. The participants in a peer review are the technical experts and key stakeholders for the scope of the review. The software peer reviews should be an independent evaluation by internal or external subject matter experts who do not have a vested interest in the work product under review. Projects can plan peer reviews, and focused reviews conducted on selected work products by the producer’s peers to identify defects and issues before that work product moves into a milestone review or approval cycle

Design









Flight software development tool(s) are not validated and accredited.

Risk is errors in the flight software development tools create errors and defects in the flight software product. Performing verification and validation (V&V) to accredit software models, simulations, and analysis tools is important to ensure the credibility of the results produced by those tools. Critical decisions may be made, at least in part, based on the results produced by models, simulations, and analysis tools. Reducing the risk associated with these decisions is one reason to use accredited tools that have been properly verified and validated. Performing V&V to approve Software models, simulations, analysis tools, and software tools is important to ensure the credibility of those tools’ results and how the tool directly affects the Software being developed. Critical decisions may be made, at least in part, based on the results produced by these tools. Reducing the risk associated with these decisions is one reason to assure that the approved tools have been properly verified and validated. Information regarding specific 

Design









"Less than 95% of the Data dictionary data definitions are complete. The attributes shall include but not be limited to:

  1. Derivation or origin of parameters so that they can be maintained and dependencies with other parameters are visible,
  2. Parameter type such as enumeration, alphanumeric, floating point, integer, etc.,
  3. Nominal value, precision, accuracy, and allowable range for numeric types,
  4. Physical units of measure and reference frames shall be specified and checked for consistency (automatically where possible)
  5. Other specific attributes for non-numeric types such as organization and format."

Risk is incomplete interface definitions and misunderstanding of the data use in the software algorithms.

Design









Incomplete or missing software design analysis activity or  Software Architecture Review Board (SARB) Analysis

The design addresses the software architectural design and software detailed design.  The objective of doing design analysis is to ensure that the design:  is a correct, accurate, and complete transformation of the software requirements that will meet the operational needs under nominal and off-nominal conditions, is safe, is secure with known weaknesses and vulnerabilities mitigated, introduces no unintended features, and does not result in unacceptable operational risk.

Design









A flawed system architecture, flawed software implementation, or misconfigured software could cause loss of vehicle control and loss of crew as a result of Inadequate Software Development Process or Inadequate Guidelines and Best Practices. System/Software architecture contains assumptions that can lead to unnecessary or unacceptable level of risk.

Software architecture deals with the fundamental organization of a system, as embodied in its components and their relationships to each other and the environment. Software architecture reviews are conducted to inspect quality attributes, principles of design, verifiability, and operability of a software system. The software architecture is reviewed to evaluate its effectiveness, efficiency, and robustness relative to the software requirements.  The software architecture review helps manage and/or reduce flight software complexity through improved software architecture, improved mission software reliability and cost savings. The software architecture underpins a system's software design and code; it represents the earliest design decisions, ones that are difficult and costly to change later. The transformation of the derived and allocated requirements into the software architecture results in the basis for all software development work. Software architecture: Formalizes precise subsystem decompositions; Defines and formalizes the dependencies among software work products within the integrated system; Serves as the basis for evaluating the impacts of proposed changes; Maintains rules for use by subsequent software engineers that assure a consistent software system as the work products evolve; and provides a stable structure for use by future groups through the documenting of the architecture. its views and patterns, and its rules.

Design









Inability of the software design to address or catch external acts that could bypasses or contravenes security policies, practices, or procedures, leading to unauthorized use of software applications, issuing of commands, and changes to the software configuration.

Missing capabilities to monitor key software observables (e.g. number of failed login attempts, performance changes, internal communication changes) to detect adversarial actions that threaten mission success.

Design









"Data is incorrect or incomplete due to:

  • Incorrect display legends
  • Incorrect data formats
  • Incorrect conversion algorithms
  • Incorrect data types used in computations and/or data conversions
  • Metadata check failures going undetected or unreported, overlapping the display of numbers, words, graphics
  • Data corrupted during transmission"

Any uploaded or uplinked data, rules, and code can affect the behavior of the software and/or system. Special acceptance tests should be developed to validate and verify the uplinked or uploaded information for nominal and off-nominal scenarios.

Design









Flight software shall be designed to detect and respond safely to corrupted commands, data, or loads, and memory faults allocated to the software, such as stuck bits or single event effects (SEE).

The risk is that the software will crash in conditions of corrupted commands, data, or loads, and memory faults allocated to the software. 

Design









Software Implementations are incomplete or missing per flow down from Software Req/Design. (>=30%/20%/10%/Any%)

Risk is incomplete software components and untested software capabilities. Requirements are the basis for a software product. They identify the needs to be addressed, the behavior of the system, the desired quality of the software, and the constraints under which the problem is to be solved. They specify the product the provider is to deliver to a customer.  Requirements serve as the basis for verification activities allowing the developing organization and the customer to judge the completeness of the product.

Design




30%

20%

10%

Any%


All expected/agreed changes of significant impact have not been reflected in the code

Risk is incomplete software components and untested software capabilities. All changes need to be addressed in the are the software products.  They specify the product the provider is to deliver to a customer.  

Design









Missing user guide for the code. A user guide is a key external user experience document to assist users in using a given product, service or application. 

Risk is misuse or misfunctioning of the software capabilities and functionality. Very few software projects result in a product that users understand and can use efficiently, completely, and successfully without some guidance. Software documentation "explains the capabilities of the software or provides operating instructions for using the software to obtain the desired results." Along with software and other documentation, the final delivered package for software projects, including contracted projects, includes operational instructions for the delivered software. These instructions are typically provided by a Software User Manual, which contains both user instructions and a description of the functions and features provided by the software.

Ops









Missing or incomplete software configuration management plan

Software Configuration Management planning encompasses the practices and procedures for administering source code, producing software development builds, controlling change, and managing software configurations for all of the software products, tools, data, and components produced by the project. 

Prog









IV&V findings are not being addressed by the project

Risk of not performing IV&V is potential software defects and a lack of rigorous analysis and testing methodologies to identify objective evidence and conclusions to provide an independent assessment of critical products and processes throughout the life cycle. The evaluation of products and processes throughout the life cycle demonstrates whether the software is fit for nominal operations (required functionality, safety, dependability, etc.) and off-nominal conditions (response to faults, responses to hazardous conditions, etc.).IV&V is a technical discipline of software assurance that employs rigorous analysis and testing methodologies to identify objective evidence and conclusions to provide an independent assessment of critical products and processes throughout the life cycle. The evaluation of products and processes throughout the life cycle demonstrates whether the software is fit for nominal operations (required functionality, safety, dependability, etc.), and off-nominal conditions (response to faults, responses to hazardous conditions, etc.). The goal of the IV&V effort is to contribute to the assurance conclusions to the project and stakeholders based on evidence found in software development artifacts and risks associated with the intended behaviors of the software.

Prog









Software process audit finding not being addressed

Risk is that the software development practices and processes are not being followed and that software defects may exist as result of not following or upstanding the software processes

Prog









Missing clear software acceptance criteria for all software products. Acceptance criteria (AC) are the conditions that a software product must meet to be accepted by a user, a customer, or other systems.

Risk is unclear guidance on milestone reviews and final software product capability, functionality, and testing.  Acceptance criteria is an important component. It clearly defines the scope, desired outcomes of, and testing criteria for pieces of functionality that the delivery team is working on. The process of creating and agreeing on acceptance criteria itself is also an invaluable communication opportunity between developers and product.

Prog









Missing software hazard conditions. Hazard Analysis should consider software’s ability, by design, to cause or control a given hazard. It is a best practice to include the software within the system hazard analysis. The general hazard analysis should consider software common-mode failures that can occur in instances of redundant flight computers running the same software. 

Risk is missing hazards associated with software and mechanizations' of software safety criticality. It is important to determine the safety criticality of each software component to identify the most critical software system components and to ensure that the software safety-critical requirements and processes are followed. Software safety analysis supplements the system hazard analysis by assessing the software performing critical functions serving as a hazard cause or control. A typical software safety analysis process identifies the must work and must not work functions in the hazard reports. The system hazard analysis and software safety analysis process should assess each function for compliance with the levied functional software requirements. The system hazard analysis and software safety analysis also assure the redundancy management performed by the software supports fault tolerance requirements. 

Prog









A cyber security risk that originates from within an organization. It typically occurs when a current or former employee, contractor, vendor or partner with legitimate user credentials misuses their access to the detriment of the organization's networks, systems and data. Missing build processes that take into account insider thread mitigation activities

Risk of a current or former employee, contractor, vendor or partner with legitimate user credentials misuses their access to the detriment of the organization's software, networks, systems or data.

Prog









Inability to access the software and software development status

Risk is that the software development does not have the necessary people in place, skills and processes in place to produce reliable products within cost and schedule estimates. 

Prog









Large number of critical software defects and operational workarounds in the flight release code

Risk is unusable software or high complexity in software operations

Prog









Hazard requirement flow down tracing delivered very late thereby not available for identifying when verification tests were safety critical test cases of hazard controls. 

Risk is incomplete hazard verification












Any Severity 1 or 2 IV&V findings are not being addressed by the project

Risk of not performing IV&V is potential software defects and a lack of rigorous analysis and testing methodologies to identify objective evidence and conclusions to provide an independent assessment of critical products and processes throughout the life cycle. The evaluation of products and processes throughout the life cycle demonstrates whether the software is fit for nominal operations (required functionality, safety, dependability, etc.) and off-nominal conditions (response to faults, responses to hazardous conditions, etc.).IV&V is a technical discipline of software assurance that employs rigorous analysis and testing methodologies to identify objective evidence and conclusions to provide an independent assessment of critical products and processes throughout the life cycle. The evaluation of products and processes throughout the life cycle demonstrates whether the software is fit for nominal operations (required functionality, safety, dependability, etc.), and off-nominal conditions (response to faults, responses to hazardous conditions, etc.). The goal of the IV&V effort is to contribute to the assurance conclusions to the project and stakeholders based on evidence found in software development artifacts and risks associated with the intended behaviors of the software.

Prog









Missing software requirement implementation for the requirements for NASA-STD-1006

Risk is that NASA missions may not be resilient to purposeful threats. Current trends such as technology proliferation, accessibility to space, globalization of space programs and industries, commercialization of space systems and services, and foreign knowledge about U.S. space systems increases the likelihood that the U.S. will experience a "Space Pearl Harbor." For example, in July 2000, the Xinhua news agency reported that China's military is developing methods and strategies for defeating the U.S. military in a high-tech and space-based future war. It noted, "For countries that could never win a war by using the method of tanks and planes, attacking the U.S. space system may be an irresistible and most tempting choice..."(2) These reports illustrate an unpleasant but little noticed view of the future. The ability to restrict or deny freedom of access to and operations in space is no longer limited to global military powers. The reality is that there are many extant capabilities to deny, disrupt or physically destroy space systems and the ground facilities that command and control them. Knowledge of U.S. space systems functions, locations and physical characteristics, as well as the means to conduct counterspace operations, is increasingly available on the international market. Nations or groups hostile to the U.S. possess or can acquire the means to disrupt or destroy U.S. space systems by attacking the satellites in space, their communications nodes on the ground and in space, or ground nodes that command the satellites.

Req









Missing software requirement for encryption on uplink and downlink

Command link incidents with civil space missions have demonstrated potential impacts to safe operations. Additionally, NASA end of mission (EOM) experiments found that spacecraft without encryption or authentication are particularly susceptible to these impacts.

Req









Space flight software development organizations work with the Project Protection Plan 362  to evaluate all software security risks identified in the Project Protection Plan.  The project performs a software cybersecurity assessment on the software components per the Agency security policies and the project requirements, including risks posed by the use of COTS, GOTS, MOTS, OSS, or reused software components.

The purpose of security risk management is to identify potential software security problems before they occur so that software security risk handling activities can be planned and invoked as needed across the life of the space flight software product or project. 

Req









High software requirements quality risk score (software requirements quality score of 3 or higher).  Less then 3 is a risk, fewer than 5% of the requirements have a  fewer than 0% of the requirements high risk or very high risk score

Risk is incomplete software components and untested software capabilities. Requirements are the basis for a software product. They identify the needs to be addressed, the behavior of the system, the desired quality of the software, and the constraints under which the problem is to be solved. They specify the product the provider is to deliver to a customer.  Requirements serve as the basis for verification activities allowing the developing organization and the customer to judge the completeness of the product.

Req









Missing approvals for tailored significant and risky requirements in the requirement matrices (NPR 7150.2 and NASA-STD-8739.8) by both Engineering and SMA TAs

Risk is that all technical authorities have not agree with the risk associated with any tailored NPR 7150.2 requirements. incomplete implementation of requirements. Software requirements tailoring is the process used to seek relief from NPR requirements consistent with program or project objectives, acceptable risk, and constraints. To accommodate the wide variety of software systems and subsystems, application of these requirements to specific software development efforts may be tailored where justified and approved. To effectively maintain control over the application of requirements in this directive and to ensure proposed tailoring from specific requirements are appropriately mitigated, NASA established TA governance. 

Req









Incomplete software requirements to code/design traceability, code or design functions without existing traceable software requirements. Software requirements not fully defined or incorrectly translated from requirement to design, caused by insufficient requirements or design reviews. Errors in software requirements include requirements that do not involve the necessary stakeholders in order to implement the desired functionality. Software requirements error resulting from invalid or insufficient data used in simulations and models developed for verification. Errors not detected and/or removed due to lack of design reviews or poor software development practices that do not include the necessary stakeholder participation on design review teams.

Requirements are the basis for a software product. They identify the needs to be addressed, the behavior of the system, the desired quality of the software, and the constraints under which the problem is to be solved. They specify the product the provider is to deliver to a customer.  Requirements serve as the basis for verification activities allowing the developing organization and the customer to judge the completeness of the product. Missing details in the software requirements, # of estimated SLOC to be developed by the project/ # of detailed software requirements is greater than 50 or the number of TBD/TBC/TBRs in the software requirements document exceeds 10.

Req









Software requirements are baselined late in the software development life cycle.

Limited or no control of the software development and test schedule, limited insight into the complete set of required software functionality, missing project schedule milestones

Req









Missing software requirements relating to the detection of adversarial actions

Inability to monitor key software observables (e.g. number of failed login attempts, performance changes, internal communication changes) to detect adversarial actions that threaten mission success. Monitoring of key software observables (e.g., number of failed login attempts, performance changes, internal communication changes) is needed to detect adversarial actions that threaten mission success. When an adversarial action occurs, it should be reported. Raw event data should be further analyzed to determine whether an anomalous event represents an attack and if so, the nature of the attack.

Req









Unclear or undefined mission and fault management requirements for software implementation. Test/diagnostic code shall be designed and incorporated into the software early, and be accessible through flight interfaces, so that problem resolution can be done rapidly and easily at element and flight system level in development and during flight operations.

Fault management requirements are the basis for a software product fault detection, isolation and recovery activities. They identify the fault needs to be addressed, the behavior of the system during a fault situation, the desired quality of the software, and the constraints under which the fault is to be solved. They specify the product the provider is to deliver to a customer.  Requirements serve as the basis for verification activities allowing the developing organization and the customer to judge the completeness of the product.

Req









Missing Sensor range checking

Risk is incorrect sensor reading and premature fault indications

Req









The OTS software component is not verified and validated to the same level required to accept a similar developed software component for its intended use

OTS Software testing is required to ensure that the OTS software meets the agreed requirements and design. The application works as expected. The application doesn’t contain serious bugs, and the software meets its intended use as per user expectations. Testing of embedded COTS, GOTS, MOTS, Open Source Software (OSS), or reused software component is required to be at the same level required to accept a custom-developed software component for its intended use.

Req









Missing data quality indicators. The semantics of data conveyed across public interfaces, whether inside an executable or as input to or output from an executable, shall be clearly specified and, if possible, verified in an automated way at build time. Data semantics may include range, precision, physical units of measure, and coordinate frames, as applicable. This principle applies primarily to public interfaces that cross system, subsystem, or component boundaries --- specifically, boundaries between software elements developed by different programmers. The importance of critical (formal) verification of these interfaces should be commensurate with the organizational separation between developers on each side of the interface.

The principle risk is miscommunication between engineering teams about the meaning of data conveyed across major public interfaces. It's not enough to specify semantics in an interface control document and rely on human inspection. This principle presumes that elements developed by an individual programmer, or within a cohesive team are more likely to be self-consistent, whereas external interfaces are less likely to be consistent. Data quality indicators are for operators or software to recognize unreliable data so that actions should not be taken as a response to displayed data.  With missing quality indicators, Operators or software may act on unreliable data and respond inappropriately thus affecting system reliability 

Req









Missing detailed software requirements for all COTS, GOTS, MOTS, OSS, or reused software components 

OTS Requirements are the basis for a software product. They identify the needs to be addressed, the behavior of the system, the desired quality of the software, and the constraints under which the problem is to be solved. They specify the product the provider is to deliver to a customer.  OTS Requirements serve as the basis for verification activities allowing the developing organization and the customer to judge the completeness of the product. 

Req









less than (CDR 75%, SIR 90%, TRR 100%) complete on bi directional traceability for all detailed software requirements. 

Bidirectional traceability matrices help ensure that all the required software requirements (only what is required is developed) are addressed in software design and tested in the software testing activities. Bidirectional traceability matrices also make it less likely that requirements are misinterpreted as they are refined.

Req




75%

90%

100%



Software Requirements are incomplete or missing per flow down from System Requirements. (>=30%/20%/10%/Any%)

Risk is incomplete software components and untested software capabilities. Requirements are the basis for a software product. They identify the needs to be addressed, the behavior of the system, the desired quality of the software, and the constraints under which the problem is to be solved. They specify the product the provider is to deliver to a customer.  Requirements serve as the basis for verification activities allowing the developing organization and the customer to judge the completeness of the product.

Req


30%20%10%

Any%




Software Designs are incomplete or missing per flow down from Software Requirements. (>=30%/20%/10%/Any%)

Risk is incomplete software components and untested software capabilities. Requirements are the basis for a software product. They identify the needs to be addressed, the behavior of the system, the desired quality of the software, and the constraints under which the problem is to be solved. They specify the product the provider is to deliver to a customer.  Requirements serve as the basis for verification activities allowing the developing organization and the customer to judge the completeness of the product.

Req



30%20%

10%

Any%



# of estimated SLOC to be developed by the project/ # of detailed software requirements is greater than 50

Risk is incomplete or missing software requirements, insufficent requirements can lead to software functionality not being tested.

Req









The % of TBD/TBC/TBRs in the software requirements document exceeds 1%

Risk is incomplete or missing software requirements, insufficent requirements can lead to software functionality not being tested.

Req









Software regression testing not identified

Regression testing calls for Identification of a set of tests that can show that critical functions still perform as expected. These regression tests should be rerun after any changes are made in the system to ensure that the critical functions still perform correctly following the changes made to the system.

Test









Missing or incomplete cybersecurity testing

Cybersecurity testing is done to ensure that the mitigations put into place to address any cybersecurity weaknesses or vulnerabilities work properly, and allow the system to operate as planned with no security breaches causing unexpected and potentially hazardous operational conditions.

Test









Use of uncertified Flight Software on flight vehicle during testing and launch preparation 

Risk is damage to the system and or test delays due to software bugs. Software testing is required to ensure that the software meets the agreed requirements and design. The application works as expected. The application doesn’t contain serious bugs, and the software meets its intended use as per user expectations.

Test









Incomplete software test plans

Ensuring the Software Test Plan follows a template and includes specified information ensures consistency of test plans across projects, ensures proper planning occurs, and prevents repeating problems of the past. Planning the software testing activity allows thorough deliberation of tasks, methods, environments, and related criteria before they are implemented. Planning also allows the project team to improve a current project, based on lessons learned from previous projects, including using more appropriate or efficient techniques and ensuring the performance of steps previously missed or not included. As with any task, having a plan in place ensures that all necessary and required tasks are performed. Development of that plan provides the opportunity for stakeholders to give input and assist with the documentation and tailoring of the planned testing activities to ensure the outcome will meet the expectations and goals of the project.

Test









Insufficient fidelity in the software test environment or lack of test avionics hardware for software testing

Validation is a process of evaluating work products to ensure that the right behaviors have been built into the work products. The right behaviors adequately describe what the system is supposed to do and what the system is supposed to do under adverse conditions. They may also describe what the system is not supposed to do. Validation is performed to assure that the specified software systems fulfill their intended use when placed on the targeted platform in the target environment (or simulated target environment). The methods used to accomplish validation on the actual target platform or in a high fidelity simulator may include aspects that were applied to previous software work products (requirements, designs, prototypes, etc.). The use of these methods provides continuity of results through the assembling system. The use of the high-fidelity or targeted system allows the software developers to check systems-level interfaces, memory performance and constraints, event timing, and other characteristics that can only be evaluated properly in the real system or near-system environment. Validation activities include preparation, performance, analysis of results, and identification of corrective action. Validation at the systems level ensures that the correct product has been built.

Test









Software test schedule does not have enough time to complete adequate software testing

Incomplete software testing. Software testing is required to ensure that the software meets the agreed requirements and design. The application works as expected. The application doesn’t contain serious bugs, and the software meets its intended use as per user expectations.

Test









Incomplete or inaccurate software test procedures, including missing or incomplete software test procedures and test scripts or large number of changes in the test procedures

Incomplete software testing. Software testing is required to ensure that the software meets the agreed requirements and design. The application works as expected. The application doesn’t contain serious bugs, and the software meets its intended use as per user expectations.

Test









Planned use of STB (Simulation Test Bed) instead of using flight hardware for flight certification

Incomplete software testing or software defects because software was tested to a simulation in place of the real hardware. Software testing is required to ensure that the software meets the agreed requirements and design. The application works as expected. The application doesn’t contain serious bugs, and the software meets its intended use as per user expectations.

Test









No independent software testing

Incomplete software testing. Independent software testing is a software testing approach that employs rigorous analysis and testing methodologies to identify objective evidence and conclusions to provide an independent assessment of critical products and processes throughout the life cycle. The evaluation of products and processes throughout the life cycle demonstrates whether the software is fit for nominal operations (required functionality, safety, dependability, etc.), and off-nominal conditions (response to faults, responses to hazardous conditions, etc.). The goal of the independent software testing effort is to contribute to the assurance conclusions to the project and stakeholders based on evidence found in software development artifacts and risks associated with the intended behaviors of the software

Test









Use of Engineering Flight Test (EFT) test results for REAL Flight vehicle flight certification

Incomplete software testing or software defects because software was tested using engineering hardware in place of the real flight hardware. Software testing is required to ensure that the software meets the agreed requirements and design. The application works as expected. The application doesn’t contain serious bugs, and the software meets its intended use as per user expectations.

Test









Missing or incomplete software test reports, including Selective recording of software verification tests for credit only when no failures and Undefined software test procedure outputs and criteria, including all I/O definitions

Incomplete software testing and missing software defects. Test results are the basis for confirming that the team has fulfilled the software requirements in the resulting software product. To make such decisions, test results must be reviewed and evaluated using a documented, repeatable process. The team can derive quality conclusions by capturing the actual test results, comparing them to expected results, analyzing those results against pre-established criteria, and documenting that analysis/evaluation process. Software testing is required to ensure that the software meets the agreed requirements and design. The application works as expected. The application doesn’t contain serious bugs, and the software meets its intended use as per user expectations.

Test









Use of software simulations for formal software testing that have not been certified 

Risk is errors in the flight software development tools create errors and defects in the flight software product. Performing verification and validation (V&V) to accredit software models, simulations, and analysis tools is important to ensure the credibility of the results produced by those tools. Critical decisions may be made, at least in part, based on the results produced by models, simulations, and analysis tools. Reducing the risk associated with these decisions is one reason to use accredited tools that have been properly verified and validated. Performing V&V to approve Software models, simulations, analysis tools, and software tools is important to ensure the credibility of those tools’ results and how the tool directly affects the Software being developed. Critical decisions may be made, at least in part, based on the results produced by these tools. Reducing the risk associated with these decisions is one reason to assure that the approved tools have been properly verified and validated. Information regarding specific 

Test









Missing or incomplete software test criteria

Risk is unclear guidance on milestone reviews and final software product capability, functionality, and testing.  Acceptance criteria is an important component. It clearly defines the scope, desired outcomes of, and testing criteria for pieces of functionality that the delivery team is working on. The process of creating and agreeing on acceptance criteria itself is also an invaluable communication opportunity between developers and product.

Test









Software test approach not defined for arrays, commands,  data, stored in contiguous memory locations, and Multi-Logic Testing approach

Risk is unclear guidance on software product capability, functionality, and testing.  Acceptance software test criteria is an important component. It clearly defines the scope, desired outcomes of, and testing criteria for pieces of functionality that the delivery team is working on. The process of creating and agreeing on acceptance criteria itself is also an invaluable communication opportunity between developers and product.

Test









Less than (TRR 80%, SRR 90%, ORR 100%) test recording procedure are not defined

Risk is unclear guidance on milestone reviews and final software product capability, functionality, and testing.  Acceptance criteria is an important component. It clearly defines the scope, desired outcomes of, and testing criteria for pieces of functionality that the delivery team is working on. The process of creating and agreeing on acceptance criteria itself is also an invaluable communication opportunity between developers and product.

Test






80%

90%

100%

Less than 100% flight software commands have been tested

Incomplete software testing or software defects because software commands were not tested. Software command testing is required to ensure that the software meets the agreed requirements and design. The application works as expected. The application doesn’t contain serious bugs, and the software meets its intended use as per user expectations.

Test









Less than 100% flight software data inputs have been tested including software data loads, data configuration loads, I-loads, flight software configuration files, missing or incomplete verification approaches for data-driven architectures 

Incomplete software testing or software defects because software data inputs were not tested. Software data input testing is required to ensure that the software meets the agreed requirements and design. The application works as expected. The application doesn’t contain serious bugs, and the software meets its intended use as per user expectations.

Test









Software update capabilities have not been tested

Incomplete software testing or software defects because software commands for updated and update capabilities were not tested. Software commands for updated and update capabilities testing is required to ensure that the software meets the agreed requirements and design. The application works as expected. The application doesn’t contain serious bugs, and the software meets its intended use as per user expectations.

Test









Missing or undefined software stress testing approach. Software stress Testing is a software testing activity that determines the robustness of software by testing beyond the limits of normal operation. Stress testing is particularly important for "mission critical" software, but is used for all types of software.

The risk is that the software will crash in conditions of insufficient computational resources (such as memory or disk space), unusually high concurrency, or denial of service attacks.

Test









Software Verifications are incomplete or missing per flow down from Software Req. (>=30% SIR/20% TRR/10% SAR/0% ORR)

Risk is incomplete software components and untested software capabilities. Requirements are the basis for a software product. They identify the needs to be addressed, the behavior of the system, the desired quality of the software, and the constraints under which the problem is to be solved. They specify the product the provider is to deliver to a customer.  Requirements serve as the basis for verification activities allowing the developing organization and the customer to judge the completeness of the product.

Test





30%20%10%0%

1.2 Programmatic Risks

Risk 

Phase / AreaM
C
R
S
R
R
M
D
R
S
D
R
P
D
R
C
D
R
S
I
R
T
R
R
S
A
R
O
R
R

Software acquisition or make strategy has significant software risks due to capability maturity experience or processes of software development organization

Risk is that the software development does not have the necessary skills and processes in place to produce reliable products within cost and schedule estimates. 

Prog









Unrealistic software development or test schedules 

Risk not enough time in the schedule to perform software development and testing activities. 

Prog









Missing IV&V support or limited IV&V support. Objectives of performing IV&V include: Facilitate early detection and correction of cost and schedule variances. Enhance management insight into process and product risk. Support project life cycle processes to ensure compliance with regulatory, performance, schedule, and budget requirements.

Risk of not performing IV&V is potential software defects and a lack of rigorous analysis and testing methodologies to identify objective evidence and conclusions to provide an independent assessment of critical products and processes throughout the life cycle. The evaluation of products and processes throughout the life cycle demonstrates whether the software is fit for nominal operations (required functionality, safety, dependability, etc.) and off-nominal conditions (response to faults, responses to hazardous conditions, etc.).IV&V is a technical discipline of software assurance that employs rigorous analysis and testing methodologies to identify objective evidence and conclusions to provide an independent assessment of critical products and processes throughout the life cycle. The evaluation of products and processes throughout the life cycle demonstrates whether the software is fit for nominal operations (required functionality, safety, dependability, etc.), and off-nominal conditions (response to faults, responses to hazardous conditions, etc.). The goal of the IV&V effort is to contribute to the assurance conclusions to the project and stakeholders based on evidence found in software development artifacts and risks associated with the intended behaviors of the software.

Prog









Lack of or delayed electronic access for IV&V to all software products, software data, software source code, and software metrics

NASA requires electronic access to software source code developed by suppliers with NASA funding to enable independent evaluations, checks, reviews, and testing.  Source code is also needed to be able to run static code analysis on the product to check for software errors, security issues, and coding standard compliance. The electronic availability of the software work products, and associated process information, facilitates post-delivery testing that is necessary for assessing as-built work product quality, and for the porting of products to the appropriate hosts. This access also accommodates the longer-term needs for performing maintenance, assessing operation or system errors, addressing hardware and software workarounds, and allowing for the potential reuse of the software on future NASA projects.

Prog









Delayed start to software assurance and or IV&V Activities

Risk of not performing software assurance and IV&V is potential software defects and a lack of rigorous analysis and testing methodologies to identify objective evidence and conclusions to provide an independent assessment of critical products and processes throughout the life cycle. The evaluation of products and processes throughout the life cycle demonstrates whether the software is fit for nominal operations (required functionality, safety, dependability, etc.) and off-nominal conditions (response to faults, responses to hazardous conditions, etc.). Software assurance and IV&V are technical disciplines of software assurance that employs rigorous analysis and testing methodologies to identify objective evidence and conclusions to provide an independent assessment of critical products and processes throughout the life cycle. The evaluation of products and processes throughout the life cycle demonstrates whether the software is fit for nominal operations (required functionality, safety, dependability, etc.), and off-nominal conditions (response to faults, responses to hazardous conditions, etc.). The goal of the software assurance and IV&V efforts is to contribute to the assurance conclusions to the project and stakeholders based on evidence found in software development artifacts and risks associated with the intended behaviors of the software.

Prog









Missing or incomplete software measurement program

NASA projects demonstrate the following three key reasons for software measurement activities: to understand and model software engineering processes and products, to aid in assessing the status of software projects, and to guide improvements in software engineering processes. What gets measured gets managed.

Prog









Incomplete flow down of NPR 7150.2 and NASA-STD-8739.8 requirements

Baseline set of requirements, software engineering practices and software assurance practices to reduce software engineering risks on NASA projects and programs. 

Prog









Incorrect software classifications

Risk is missing or incorrectly flowing down a baseline set of requirements, software engineering practices and software assurance practices to reduce software engineering risks on NASA projects and programs. 

Prog









Lack of verifiable or certifiable software development practices by the organizations developing the critical software components

Risk is that the software development does not have the necessary skills and processes in place to produce reliable products within cost and schedule estimates. 

Prog









Software assigned or assumed to be re-used is either not a good match for the proposed mission or it is not fully understood through missing or incomplete feasibility studies.

Risk is incompatible software, understanding the functionality of the software code, retesting the software in the new environment, maintenance issues, and legal issues

Prog









Operational scenarios are not defined enough to support the definition and allocation of system and software requirements

Risk is incomplete software components and untested software capabilities. Requirements are the basis for a software product. They identify the needs to be addressed, the behavior of the system, the desired quality of the software, and the constraints under which the problem is to be solved. They specify the product the provider is to deliver to a customer.  Requirements serve as the basis for verification activities allowing the developing organization and the customer to judge the completeness of the product.

Prog









1.3 Resource Risks

Risk 

Phase / AreaM
C
R
S
R
R
M
D
R
S
D
R
P
D
R
C
D
R
S
I
R
T
R
R
S
A
R
O
R
R

The project cost allocation or software assurance resources unavailable for the project’s Software Assurance support to meet the requirements

Missing insurance that the processes, procedures, and products used to produce and sustain the software conform to all specified requirements and standards that govern those processes, procedures, and products, Missing ability to determine the degree of software quality obtained by the software products. Inability to ensure that the software systems are safe and that the software safety-critical requirements are followed and that the software products are secure.  Missing employing rigorous analysis and testing methodologies to identify objective evidence and conclusions to provide an independent assessment of critical products and processes throughout the life cycle.

Prog









Insufficient software workforce or software skillsets

Risk is that the software development does not have the necessary skills and processes in place to produce reliable products within cost and schedule estimates. 

Prog









Software development activities are behind schedule due to underperforming resources, missing resources, medical related issues, or personnel priorities 

Risk is that the software development does not have the necessary people in place, skills and processes in place to produce reliable products within cost and schedule estimates. 

Prog









Software development activities are behind schedule due to high turnover of software engineering or software assurance personnel on the project

Risk is that the software development does not have the necessary people in place, skills and processes in place to produce reliable products within cost and schedule estimates. 

Prog









1.4 Technological Risks

Risk 

Phase / AreaM
C
R
S
R
R
M
D
R
S
D
R
P
D
R
C
D
R
S
I
R
T
R
R
S
A
R
O
R
R

Violations of Margin for CPU Utilization. CPU Utilization: This is a key metrics which measures the percentage of time CPU spends in handling a process. High CPU utilization by any task is red flagged to check any performance issues. Timely detection and planned response to oversubscriptions can preserve critical system capabilities.  There are several common methods for tolerating these situations, most of which relate to reducing demand from non-essential items, especially if they are the source of over subscription. Software development is complicated by inadequate or impractical timing allocations or margins associated with selected CPU. The software design should contain a robust response to situations where computer resources are oversubscribed. The action to be taken in such situations shall be specified as part of the requirements on the design.

Resource oversubscription is a severe fault condition that can lead to unpredictable behavior of the software system and render it inoperable. Examples of these situations include buffers overflowing, exceeding a rate group time boundary, and excessive inputs or interrupts. Software should be designed to provide easy and timely visibility into the use of computing resources during testing and operations.

Code









Poor Software programing language selection by the project, insufficient software analysis and development tools, and the software development organization is not trained in the use of the software language

Risk is that the software development does not have the necessary skills and processes in place to produce reliable products within cost and schedule estimates. 

Prog









Personnel supporting the project do not have adequate experience or training to perform the processes

Risk is that the software development does not have the necessary skills and processes in place to produce reliable products within cost and schedule estimates. The staffing plan should provide for filling key software roles early in the project or task life-cycle and provide for retention of a cadre of experienced development and test personnel through delivery and operations.

Prog









Risk associated with developing software at the same time as the hardware is being developed, risk of misunderstanding the software hardware interfaces and not having the hardware available to use in the software development

Risk is that the software development team does not have the necessary hardware interface data and experience in place to produce reliable products within cost and schedule estimates. Risk associated with developing software at the same time as the hardware interfaces are being developed or defined, risk of misunderstanding the software hardware interfaces and not having the hardware interfaces available to use in the software development.

Prog









Software development is complicated by inadequate or impractical data throughput allocations or margins associated requirements and bus selections

Resource oversubscription is a severe fault condition that can lead to unpredictable behavior of the software system and render it inoperable. Examples of these situations include buffers overflowing, exceeding a rate group time boundary, and excessive inputs or interrupts.

Prog









Software development is complicated by undefined software processes or misunderstood processes

Risk is that the software development does not have the necessary skills and processes in place to produce reliable products within cost and schedule estimates. 

Prog










See also 7.19 - Software Risk Management Checklists, SWE-086 - Continuous Risk Management 

1.1 Additional Guidance

Links to Additional Guidance materials for this subject have been compiled in the Related Links table. Click here to see the in the Resources tab.

2. Resources

2.1 References

[Click here to view master references table.]

No references have been currently identified for this Topic. If you wish to suggest a reference, please leave a comment below.



2.2 Tools


Tools to aid in compliance with this SWE, if any, may be found in the Tools Library in the NASA Engineering Network (NEN). 

NASA users find this in the Tools Library in the Software Processes Across NASA (SPAN) site of the Software Engineering Community in NEN. 

The list is informational only and does not represent an “approved tool list”, nor does it represent an endorsement of any particular tool.  The purpose is to provide examples of tools being used across the Agency and to help projects and centers decide what tools to consider.

2.3 Additional Guidance

This topic is generic and applies to all SWEHB versions. All links to version specific SWEs and Topics have been removed. SWEs and Topics are in Green Bold Underlined text to make them easy to find. 

Refer to the appropriate Requirements or Topics buttons in the SWEHB version you are using to locate the SWE or Topic you need.

Additional guidance related to this requirement may be found in the following materials in this Handbook:

Related Links

  • SWE-086 - Continuous Risk Management

  • 7.19 - Software Risk Management Checklists

2.4 Center Process Asset Libraries

SPAN - Software Processes Across NASA
SPAN contains links to Center managed Process Asset Libraries. Consult these Process Asset Libraries (PALs) for Center-specific guidance including processes, forms, checklists, training, and templates related to Software Development. See SPAN in the Software Engineering Community of NEN. Available to NASA only. https://nen.nasa.gov/web/software/wiki  197

See the following link(s) in SPAN for process assets from contributing Centers (NASA Only). 

SPAN Links

3. Lessons Learned

3.1 NASA Lessons Learned

No Lessons Learned have currently been identified for this requirement.

3.2 Other Lessons Learned

No other Lessons Learned have currently been identified for this requirement.

  • No labels