1. Requirements

3.8.1 The project manager shall define the approach to the automatic generation of software source code including: 

a. Validation and verification of auto-generation tools.
b. Configuration management of the auto-generation tools and associated data.
c. Description of the limits and the allowable scope for the use of the auto-generated software.
d. Verification and validation of auto-generated source code using the same software standards and processes as hand-generated code.
e. Monitoring the actual use of auto-generated source code compared to the planned use.
f. Policies and procedures for making manual changes to auto-generated source code.
g. Configuration management of the input to the auto-generation tool, the output of the auto-generation tool, and modifications made to the output of the auto-generation tools.

1.1 Notes

NPR 7150.2, NASA Software Engineering Requirements, does not include any notes for this requirement.

1.2 History

1.3 Applicability Across Classes

Key:    - Applicable | - Not Applicable

2. Rationale

Defining the approach to be used for the automatic generation of software source code allows projects to review and verify their plans for the use and management of auto-generated software before implementation to ensure the development approach and the resulting software will meet the expectations and goals of the project without introducing unacceptable levels of risk.  

3. Guidance

3.1 Auto-generated software

Auto-generated software is the result of translating a model of the system behavior into different software languages through the use of a code generator. It is important to capture the approach used for the automatic generation of software because issues can exist with auto-generated software and projects need to be prepared to address them. 

When capturing their approach for the creation, maintenance, and use of auto-generated software, the project captures key elements including:

• Validation and verification of auto-generation tools - The correct operation of the functionality of the tools used to automatically generate code is to be confirmed before those tools are used.  Tools that do not function correctly only serve to reduce confidence in the quality and functionality of the resulting software source code.  It is important to validate and verify every version of the tools used on the project, especially if updates to the tools are made during the project life cycle (see SWE-136 - Software Tool Accreditation for recommended approaches).

• Configuration management of auto-generation tools and associated data - Typically, the software source code is configuration managed, but auto-generated software needs to have its input model and generation tools configuration managed ¹³³.  The code which is entirely generated is a disposable good - the important artifacts are the models.  Any reference models used for confirming the correct functionality of the tool are also recommended for configuration control as are initialization scripts, tool configuration scripts, and any other data needed to run the tool.

• Identification of the allowable scope for the use of auto-generated software - All of the software in the project is unlikely to be auto-generated software source code; therefore, the scope of the use of auto-generated code should be identified, documented (with rationale), and made available to the project team. Scoping could be established based on safety-criticality, complexity, the ability to segment the auto-generated code from manually-generated code, as well as cost-benefit analyses of auto-generated source code versus manually-developed source code.

• Verification and validation of auto-generated source code - Auto-generated software is required to be verified and validated to the level required to ensure its fitness for use in the intended application (See SWE-027 - Use of Commercial, Government, and Legacy Software for recommended approaches.).   “Since code generators are typically not qualified, there is no guarantee that their output is correct, and consequently auto-generated code still needs to be fully tested and certified.” ¹⁹³

• Monitoring the extent of use of auto-generated source code compared to the planned use – Capture the delta between the project’s planned and actual use of auto-generated code as one way to ensure the scope of the auto-generated code is maintained.  This information also allows monitoring of adherence to software development plans and improvement in future estimations of the use of auto-generated source code.

• Policies and procedures for making manual changes to auto-generated source code - Typically, auto-generated source code is not modified; it is important that the code can be regenerated (based on updates to the input model) without concerns of losing manual updates to the generated code.  However, if the project expects to manually modify the auto-generated source code at any stage, perhaps to allow integration with other project software, policies and procedures for how and when that code can be manually updated should be planned, documented, and implemented.  Additionally, procedures for tracking those changes and ensuring they are incorporated into the next iteration of the auto-generated source code need to be documented to reduce the risk of losing those changes.

• Configuration management of the input to the auto-generation tool, the output of the auto-generation tool, and modifications made to the output of auto-generation tools - The approach should include the project’s guidance for capturing the input (model), output (generated software source code), any modified intermediate output (e.g., the output from a compiler that is modified and then input to an assembler), and any modified auto-generated software (modified after generation).  Include guidance such as at what level of maturity the model is to be configuration controlled and whether the generated source code has and follows its configuration control guidelines or follows the configuration control guidelines for manually-generated code. Typically, source code that is 100 percent generated is not configuration managed because “derived artifacts are 100% redundant, i.e. they don't contain any non-reproducible information.” ¹³³

The approach to auto-generated software is typically captured in project documentation such as the Software Development Plan.  Information can also be captured in project documentation relevant to the topic such as configuration management or verification and validation.

For recommended practices, considerations, and additional guidance related to auto-generated software, see Topic 8.11 - Auto-Generated Code. See also SWE-206 - Auto-Generation Software Inputs, 

3.2 Additional Guidance

Additional guidance related to this requirement may be found in the following materials in this Handbook:

3.3 Center Process Asset Libraries

See the following link(s) in SPAN for process assets from contributing Centers (NASA Only). 

4. Small Projects

This requirement applies to all projects regardless of size.

5. Resources

5.1 References

5.2 Tools

6. Lessons Learned

6.1 NASA Lessons Learned

The NASA Lessons Learned database contains the following lessons learned related to auto-generated code:

• Computer Software/Configuration Control/Verification and Validation (V&V). Lesson Number 1023 ⁵³³: “The use of the Matrix X auto code generator for ISS software can lead to serious problems if the generated code and Matrix X itself are not subjected to effective configuration control or the products are not subjected to unit-level V&V. These problems can be exacerbated if the code generated by Matrix X is modified by hand."

6.2 Other Lessons Learned

No other Lessons Learned have currently been identified for this requirement.

7. Software Assurance

7.1 Tasking for Software Assurance

7.2 Software Assurance Products

• Software Engineering Plans Assessment
• SA assessment of auto-generated code approach, including any issues or risks identified with items "a" through "g".

  
  

  Objective Evidence

  None at this time.

   Definition of objective evidence

  Objective evidence is an unbiased, documented fact showing that an activity was confirmed or performed by the software assurance/safety person(s). The evidence for confirmation of the activity can take any number of different forms, depending on the activity in the task. Examples are:

  • Observations, findings, issues, risks found by the SA/safety person and may be expressed in an audit or checklist record, email, memo or entry into a tracking system (e.g. Risk Log).
  • Meeting minutes with attendance lists or SA meeting notes or assessments of the activities and recorded in the project repository.
  • Status report, email or memo containing statements that confirmation has been performed with date (a checklist of confirmations could be used to record when each confirmation has been done!).
  • Signatures on SA reviewed or witnessed products or activities, or
  • Status report, email or memo containing a short summary of information gained by performing the activity. Some examples of using a “short summary” as objective evidence of a confirmation are:
    • To confirm that: “IV&V Program Execution exists”, the summary might be: IV&V Plan is in draft state. It is expected to be complete by (some date).
    • To confirm that: “Traceability between software requirements and hazards with SW contributions exists”, the summary might be x% of the hazards with software contributions are traced to the requirements.
  • The specific products listed in the Introduction of 8.16 are also objective evidence as well as the examples listed above.

7.3 Metrics

• # of software work product Non-Conformances identified by life cycle phase over time.

See also Topic 8.18 - SA Suggested Metrics. 

7.4 Guidance

Auto-generated software is the result of translating a model of the system behavior into different software languages through the use of a code generator. It is important to capture the approach used for the automatic generation of software because issues can exist with auto-generated software and projects need to be prepared to address them.  “Users not only need to be sure that the code implements the model, but also that the code generator is correctly used and configured, that the target adaptations are correct, that the generated code meets high-level safety requirements, and that it is integrated with legacy code, and so on." ¹⁹³

When capturing their approach for the creation, maintenance, and use of auto-generated software, the project captures key elements including:

• Validation and verification of auto-generation tools - The correct operation of the functionality of the tools used to automatically generate code is to be confirmed before those tools are used.  Tools that do not function correctly only serve to reduce confidence in the quality and functionality of the resulting software source code.  It is important to validate and verify every version of the tools used on the project, especially if updates to the tools are made during the project life cycle (see SWE-136 - Software Tool Accreditation for recommended approaches).

• Configuration management of auto-generation tools and associated data - Typically, the software source code is configuration managed, but auto-generated software needs to have its input model and generation tools configuration managed ¹³³.  The code which is entirely generated is a disposable good - the important artifacts are the models.  Any reference models used for confirming the correct functionality of the tool are also recommended for configuration control as are initialization scripts, tool configuration scripts, and any other data needed to run the tool.

• Identification of the allowable scope for the use of auto-generated software - All of the software in the project is unlikely to be auto-generated software source code; therefore, the scope of the use of auto-generated code should be identified, documented (with rationale), and made available to the project team. Scoping could be established based on safety-criticality, complexity, the ability to segment the auto-generated code from manually-generated code, as well as cost-benefit analyses of auto-generated source code versus manually-developed source code.

• Verification and validation of auto-generated source code - Auto-generated software is required to be verified and validated to the level required to ensure its fitness for use in the intended application (See SWE-027 - Use of Commercial, Government, and Legacy Software for recommended approaches.).   “Since code generators are typically not qualified, there is no guarantee that their output is correct, and consequently auto-generated code still needs to be fully tested and certified.” ¹⁹³

• Monitoring the extent of use of auto-generated source code compared to the planned use – Capture the delta between the project’s planned and actual use of auto-generated code as one way to ensure the scope of the auto-generated code is maintained.  This information also allows monitoring of adherence to software development plans and improvement in future estimations of the use of auto-generated source code.

• Policies and procedures for making manual changes to auto-generated source code - Typically, auto-generated source code is not modified; it is important that the code can be regenerated (based on updates to the input model) without concerns of losing manual updates to the generated code.  However, if the project expects to manually modify the auto-generated source code at any stage, perhaps to allow integration with other project software, policies and procedures for how and when that code can be manually updated should be planned, documented, and implemented.  Additionally, procedures for tracking those changes and ensuring they are incorporated into the next iteration of the auto-generated source code need to be documented to reduce the risk of losing those changes.

• Configuration management of the input to the auto-generation tool, the output of the auto-generation tool, and modifications made to the output of auto-generation tools - The approach should include the project’s guidance for capturing the input (model), output (generated software source code), any modified intermediate output (e.g., the output from a compiler that is modified and then input to an assembler), and any modified auto-generated software (modified after generation).  Include guidance such as at what level of maturity the model is to be configuration controlled and whether the generated source code has and follows its configuration control guidelines or follows the configuration control guidelines for manually-generated code. Typically, source code that is 100 percent generated is not configuration managed because “derived artifacts are 100% redundant, i.e. they don't contain any non-reproducible information.” ¹³³

The approach to auto-generated software is typically captured in project documentation such as the Software Development Plan.  Information can also be captured in project documentation relevant to the topic such as configuration management or verification and validation.

For recommended practices, considerations, and additional guidance related to auto-generated software, see Topic 8.11 - Auto-Generated Code.

7.5 Additional Guidance

Additional guidance related to this requirement may be found in the following materials in this Handbook: