1. Introduction

This topic provides a definition with some examples of "objective evidence" and contains a listing of all the tasks in NPR-8739.8 where "objective evidence" is the product. The list of all the tasks having objective evidence as their product is formatted as a checklist. This can be used to record the type of evidence available from performing each task and the location where the evidence is stored.

2. Definition of Objective Evidence

Objective evidence is an unbiased, documented fact showing that an activity was confirmed or performed by the software assurance/safety person(s). The evidence for confirmation of the activity can take any number of different forms, depending on the activity in the task. Examples are:

Observations, findings, issues, risks found by the SA/safety person and may be expressed in an audit or checklist record, email, memo or entry into a tracking system (e.g. Risk Log).
Meeting minutes with attendance lists or SA meeting notes or assessments of the activities and recorded in the project repository.
Status report, email or memo containing statements that confirmation has been performed with date (a checklist of confirmations could be used to record when each confirmation has been done!).
Signatures on SA reviewed or witnessed products or activities, or
Status report, email or memo containing a short summary of information gained by performing the activity. Some examples of using a “short summary” as objective evidence of a confirmation are:
- To confirm that: “IV&V Program Execution exists”, the summary might be: IV&V Plan is in draft state. It is expected to be complete by (some date).
- To confirm that: “Traceability between software requirements and hazards with SW contributions exists”, the summary might be x% of the hazards with software contributions are traced to the requirements.
The specific products listed in the Introduction of 8.16 are also objective evidence as well as the examples listed above.

3. Confirmations

You may download a usable copy of this list to use a a checklist in your project. Tasks Needing Objective Evidence.docx

Tasks Needing Objective Evidence

Don’t forget that you may have tailored out some of these tasks and also one piece of objective evidence may apply to several tasks.

SWE #	Task	Evidence Type	Have Evidence (Y/N)/ Notes
033	1. Confirm that the options for software acquisition versus development have been evaluated.
033	2. Confirm the flow down of applicable software engineering, software assurance, and software safety requirements on all acquisition activities. (NPR 7150.2 and NASA-STD-8739.8).
013	1. Confirm that all plans are in place and have expected content for the life-cycle events, with proper tailoring for the classification of the software.
024	2. Confirm that closure of corrective actions associated with the performance of software activities against the software plans, including closure rationale.
034	1. Confirm software acceptance criteria are defined and assess the criteria based on guidance in the NASA Software Engineering Handbook, NASA-HDBK-2203.
036	1. Confirm the following are approved, implemented, and updated per requirements: a. Software processes, including software assurance, software safety, and IV&V processes. b. Software documentation plans, c. List of developed electronic products, deliverables, d. List of tasks required or needed for the project’s software development.
036	2. Confirm that any required government actions upon receipt of deliverables (e.g., approvals, reviews) are established and maintained.
037	1. Confirm that milestones for reviewing and auditing software developer progress are defined and documented.
039	1. Confirm that software developer(s) are periodically reporting status and providing insight to the project manager.
039	2. Monitor product integration.
039	8. Confirm that the project manager provides responses to software assurance and software safety submitted issues, findings, and risks and that the project manager tracks software assurance and software safety issues, findings, and risks to closure.
040	1. Confirm that software artifacts are available in electronic format to NASA.
042	1. Confirm that software developers are providing NASA with electronic access to the source code generated for the project in a modifiable form.
121	1. Confirm that any requirement tailoring in the Requirements Mapping Matrix has the required approvals.
125	1. Confirm that the project is maintaining a requirement mapping matrix (matrices) for all requirements in NPR 7150.2.
027	1. Confirm that the conditions listed in "a" through "f" are complete for any COTS, GOTS, MOTS, OSS, or reused software that is acquired or used. a. The requirements to be met by the software component are identified. b. The software component includes documentation to fulfill its intended purpose (e.g., usage instructions). c. Proprietary rights, usage rights, ownership, warranty, licensing rights, and transfer rights have been addressed. d. Future support for the software product is planned and adequate for project needs. e. The software component is verified and validated to the same level required to accept a similar developed software component for its intended use. f. The project has a plan to perform periodic assessments of vendor reported defects to ensure the defects do not impact the selected software components.
015	1. Confirm that the required number of software cost estimates are complete and include software assurance cost estimate(s) for the project, including a cost estimate associated with handling safety-critical software and safety-critical data.
174	1. Confirm that all the software planning parameters, including size and effort estimates, milestones, and characteristics, are submitted to a Center repository.
174	2. Confirm that all software assurance and software safety software estimates and planning parameters are submitted to an organizational repository.
018	1. Confirm the generation and distribution of periodic reports on software schedule activities, metrics, and status, including reports of software assurance and software safety schedule activities, metrics, and status.
018	2. Confirm closure of any project software schedule issues.
046	1. Confirm the project's schedules are updated.
017	1. Confirm that any project-specific software training has been planned, tracked, and completed for project personnel, including software assurance and software safety personnel.
176	1. Confirm that records of the software Requirements Mapping Matrix and each software classification are maintained and updated for the life of the project.
141	1. Confirm that IV&V requirements (section 4.4) are complete on projects that are required to have IV&V.
131	1. Confirm that the IV&V Program Execution Plan (IPEP) exists.
178	1. Confirm that IV&V has access to the software development artifacts, products, source code, and data required to perform the IV&V analysis efficiently and effectively.
179	1. Confirm that the project manager provides responses to IV&V submitted issues, findings, and risks and that the project manager tracks IV&V issues, findings, and risks to closure.
205	1. Confirm that the hazard reports or safety data packages contain all known software contributions or events where software; either by its action, inaction or incorrect action, lead to a hazard.
205	4. Confirm that the traceability between software requirements and hazards with software contributions exist.
023	1. Confirm that the identified safety-critical software components and data have implemented the safety-critical software assurance requirements listed in this standard.
134	3. Confirm 100% code test coverage is addressed for all identified software safety-critical software components or assure that software developers provide a risk assessment explaining why the test coverage is not possible for the safety-critical code component.
134	4. Confirm that all identified safety-critical software components have a cyclomatic complexity value of 15 or lower. If not, assure that software developers provide a risk assessment explaining why the cyclomatic complexity value needs to be higher than 15 and why the software component cannot be structured to be lower than 15.
134	5. Confirm that the values of the safety-critical loaded data, uplinked data, rules, and scripts that affect hazardous system behavior have been tested.
134	7. Participate in software reviews affecting safety-critical software products.
206	1. Confirm that NASA, engineering, project, software assurance, and IV&V have electronic access to the models, simulations, and associated data used as inputs for auto-generation of software.
032	1. Confirm that Class A and B software that is acquired, developed, and maintained by NASA is performed by an organization with a non-expired CMMI-DEV rating, as per the NPR 7150.2 requirement.
147	1. Confirm that the project has considered reusability for its software development activities.
148	1. Confirm that any project software contributed as a reuse candidate has the identified information in items “a” through “e.” a. Software Title. b. Software Description. c. The Civil Servant Software Technical Point of Contact for the software product. d. The language or languages used to develop the software. e. Any third party code contained therein, and the record of the requisite license or permission received from the third party permitting the Government’s use, if applicable.
156	1. Confirm the project has performed a software cybersecurity assessment on the software components per the Agency security policies and the project requirements, including risks posed by the use of COTS, GOTS, MOTS, OSS, or reused software components.
154	1. Confirm that cybersecurity risks, along with their mitigations, are identified and managed.
157	1. For software products with communications capabilities, confirm that the software requirements and software design documentation addresses unauthorized access.
158	1. Confirm that space or flight software systems have been assessed for potential cybersecurity vulnerabilities and weaknesses.
155	1. Confirm that space or flight software systems have been assessed for potential cybersecurity vulnerabilities and weaknesses.
159	1. Confirm that testing is complete for the cybersecurity mitigation.
052	1. Confirm that bi-directional traceability has been completed, recorded, and maintained.
052	2. Confirm that the software traceability includes traceability to any hazard that includes software.
050	1. Confirm that all software requirements are established, captured, and documented as part of the technical specification, including requirements for COTS, GOTS, MOTS, OSS, or reused software components.
053	1. Confirm the software requirements changes are documented, tracked, approved, and maintained throughout the project life-cycle.
054	1. Monitor identified differences among requirements, project plans, and software products to confirm they are addressed.
055	1. Confirm that the project software testing has shown that software will function as expected in the customer environment.
058	4. Confirm that the software design implements all of the required safety-critical functions and requirements.
060	1. Confirm that the software code implements the software designs.
060	2. Confirm that the code does not contain functionality not defined in the design or requirements.
135	1. Confirm the static analysis tool(s) are used with checkers to identify security and coding errors, and defects.
135	3. Confirm that the software code has been scanned for security defects and confirm the result.
062	1. Confirm that the project successfully executes the required unit tests, particularly those testing safety-critical functions.
062	2. Confirm that the project addresses or otherwise tracks to closure errors, defects, or problem reports found during unit test.
186	1. Confirm that the project maintains the procedures, scripts, results, and data needed to repeat the unit testing (e.g., as-run scripts, test procedures, results).
063	1. Confirm that the project creates a correct software version description for each software release.
063	2. For each software release, confirm that the software has been scanned for security defects and coding standard compliance and confirm the results.
136	1. Confirm that the software tool(s) needed to create and maintain software is validated and accredited.
065a	1. Confirm that software test plans have been established, contain correct content, and are maintained.
065a	2. Confirm that the software test plan addresses the verification of safety-critical software, specifically the off-nominal scenarios.
065b	1. Confirm that test procedures have been established and are updated when changes to tests or requirements occur.
065c	1. Confirm that the project creates and maintains the test reports throughout software integration and test.
065c	2. Confirm that the project records the test report data and that the data contains the as-run test data, the test results, and required approvals.
065c	3. Confirm that the project records all issues and discrepancies found during each test.
065c	4. Confirm that the project tracks to closure errors, defects, etc. found during testing.
066	1. Confirm test coverage of the requirements through the execution of the test procedures.
066	3. Ensure that any newly identified software contributions to hazards, events, or conditions found during testing are in the system safety data package.
187	1. Confirm that software items to be tested are under configuration management before the start of testing.
187	2. Confirm the project maintains the software items under configuration management through the completion of testing.
068	1. Confirm that test results are assessed and recorded.
068	2. Confirm that the project documents software non-conformances in a tracking system.
068	3. Confirm that test results are sufficient verification artifacts for the hazard reports.
070	1. Confirm the software models, simulations, and analysis tools used to achieve the qualification of flight software or flight equipment have been validated and accredited.
073	1. Confirm that the project validates the software components on the targeted platform or a high-fidelity simulation.
189	1. Confirm that code coverage measurements have been selected, performed, tracked, recorded, and communicated with each release.
190	1. Confirm that the project performs code coverage analysis using the results of the tests or by use of a code coverage tool.
191	1. Confirm that the project plans regression testing and that the regression testing is adequate and includes retesting of all safety-critical code components.
191	2. Confirm that the project performs the planned regression testing.
191	4. Confirm that the regression test procedures are updated to incorporate tests that validate the correction of critical anomalies.
192	1. Confirm that the project verifies the software requirements, which trace to a hazardous event, cause, or mitigation techniques, through testing.
193	1. Confirm that the project develops acceptance tests for loaded or uplinked data, rules, and code that affect software and software system behavior.
193	2. Confirm that the loaded or uplinked data, rules, scripts, or code that affects software and software system behavior is baselined in the software configuration system.
211	1. Confirm that the project is testing COTS, GOTS, MOTS, OSS, or reused software components to the same level as developed software for its intended use.
075	2. Confirm that the project implements software operations, software maintenance, and software retirement plans.
077	1. Confirm that the correct version of the products is provided, including as-built documentation and project records.
194	1. Confirm that the project has identified the software requirements to be met, the approved changes to be implemented, and defects to be resolved for each delivery.
194	2. Confirm that the project has met all software requirements identified for the delivery.
194	3. Confirm that approved changes have been implemented and tested.
194	4. Confirm that the approved changes to be implemented and the defects to be resolved have been resolved.
196	1. Confirm that the project has identified the records and software tools for archival.
196	2. Confirm the project has an archival location and the procedures for archiving and accessing products for software retirement or disposal.
196	3. Confirm that the project archives all software and records as planned.
080	2. Confirm: a. that the project tracks the changes, b. that the changes are approved and documented before implementation, c. that the implementation of changes is complete, and d. that the project tests the changes.
080	3. Confirm software changes follow the software change control process.
081	1. Confirm that the project has identified the configuration items and their versions to be controlled.
082	1. Confirm that software assurance has participation in software control activities.
083	1. Confirm that the project maintains records of the configuration status of the configuration items.
084	1. Confirm that the project manager performed software configuration audits to determine the correct version of the software configuration items and verify that the results of the audit conform to the records that define them.
085	1. Confirm that the project establishes procedures for storage, processing, distribution, release, and support of deliverable software products.
086	1. Confirm and assess that a risk management process includes recording, analyzing, planning, tracking, controlling, and communicating all of the software risks and mitigation plans.
087	1. Confirm that software peer reviews are performed and reported on for project activities.
087	2. Confirm that the project addresses the accepted software peer review findings.
087	4. Confirm that the source code satisfies the conditions in the NPR 7150.2 requirement SWE-134, "a" through "l," based upon the software functionality for the applicable safety-critical requirements at each code inspection/review.
087	5. For code peer reviews, confirm that all identified software safety-critical components have a cyclomatic complexity value of 15 or lower or develop a risk for the software safety-critical components that have a cyclomatic complexity value over 15.
088	1. Confirm that the project meets the NPR 7150.2 criteria in "a" through "d" for each software peer review.
088	2. Confirm that the project resolves the actions identified from the software peer reviews.
089	1. Confirm that the project records the software peer reviews or software inspection results measurements.
090	1. Confirm that a measurement program establishes, records, maintains, reports, and uses software assurance, management, and technical measures.
093	1. Confirm software measurement data analysis conforms to documented analysis procedures.
094	1. Confirm access to software measurement data, analysis, and status as requested to the following entities: - Sponsoring Mission Directorate - NASA Chief Engineer - Center Technical Authorities - Headquarters SMA
199	1. Confirm the project monitors and updates planned measurements to assure the software will meet or exceed performance and functionality requirements, including satisfying constraints.
199	2. Monitor and track any performance or functionality requirements that are not being met or are at risk of not being met.
200	1. Confirm that the project collects, tracks, and reports on the software volatility metrics.
201	1. Confirm that all software non-conformances are recorded and tracked to resolution.
201	2. Confirm that accepted non-conformances include the rationale for the non-conformance.
202	1. Confirm that all software non-conformances severity levels are defined.
202	3. Confirm that the project assigns severity levels to non-conformances associated with tools, COTS, GOTS, MOTS, OSS, reused software components, and applicable ground systems.
203	1. Confirm the evaluations of reported non-conformances for all COTS, GOTS, MOTS, OSS, or reused software components are occurring throughout the project life-cycle.
204	1. Perform or confirm that a root cause analysis has been completed on all identified high severity software nonconformances, the results are recorded, and that the results have been assessed for adequacy.
204	4. Perform or confirm tracking of corrective actions to closure on high severity software non-conformances.

4. Resources

4.1 References

Click here to view master references table.

No references have been currently identified for this Topic. If you wish to suggest a reference, please leave a comment below.

4.2 Tools

Tools to aid in compliance with this SWE, if any, may be found in the Tools Library in the NASA Engineering Network (NEN).

NASA users find this in the Tools Library in the Software Processes Across NASA (SPAN) site of the Software Engineering Community in NEN.

The list is informational only and does not represent an “approved tool list”, nor does it represent an endorsement of any particular tool. The purpose is to provide examples of tools being used across the Agency and to help projects and centers decide what tools to consider.

Content

Space Tools