1. Requirements

3.2.1 To better estimate the cost of development, the project manager shall establish, document, and maintain:

1. 1. Two cost estimate models and associated cost parameters for all Class A and B software projects that have an estimated project cost of $2 million or more.
   2. One software cost estimate model and associated cost parameter(s) for all Class A and Class B software projects that have an estimated project cost of less than $2 million.
   3. One software cost estimate model and associated cost parameter(s) for all Class C and Class D software projects.
   4. One software cost estimate model and associated cost parameter(s) for all Class F software projects.

1.1 Notes

NPR 7150.2, NASA Software Engineering Requirements, does not include any notes for this requirement.

1.2 History

1.3 Applicability Across Classes

Key:    - Applicable | - Not Applicable

2. Rationale

Because estimates are now prepared much earlier in the acquisition life cycle, well before concrete technical information is available, they are subject to greater uncertainty than they have been in the past. Establishing two independent cost estimates for all software class A and B projects that have an estimated project cost of 2 million dollars reduces the uncertainty of the cost estimation and better validates the software cost estimation on large software projects. Credible software cost estimates are a key activity performed as part of a project and software life cycle management (NPR 7150.2, NASA Software Engineering Requirements, Section 3.2) with regular updates made throughout the entire life cycle. Unreliable cost estimates result in severe problems in planning and budgeting, which then result in staffing and budgeting decisions that could impair the effective use of resources.

3. Guidance

Determining the cost of a line of software code for embedded flight software on a space mission can be challenging, as it depends on various factors such as the complexity of the code, the development environment, the qualifications of the development team, and the specific requirements of the mission. However, I can provide some general considerations:

1. Development Costs: The cost of writing a line of code typically includes the salaries or wages of the software engineers involved in the development process. This can vary significantly depending on factors such as the location of the development team, their level of expertise, and the duration of the project.

2. Tooling and Infrastructure Costs: Developing embedded flight software often requires specialized tools and infrastructure, such as integrated development environments (IDEs), simulation software, hardware-in-the-loop (HIL) testing platforms, and version control systems. These tools and infrastructure come with associated costs for licensing, maintenance, and support.

3. Quality Assurance Costs: Ensuring the reliability and safety of embedded flight software involves extensive testing, verification, and validation activities. This includes costs for test equipment, test automation tools, test engineers, and any necessary certifications or compliance audits.

4. Documentation and Compliance Costs: Developing software for space missions often requires compliance with rigorous standards and regulations, such as NASA standards (e.g., NASA-STD-8739) and aerospace industry standards (e.g., DO-178C for avionics software). Meeting these standards involves documentation efforts and compliance activities, which contribute to the overall cost.

5. Risk Management Costs: Space missions involve inherent risks, and mitigating these risks through careful software development practices incurs additional costs. This may include conducting risk assessments, implementing risk mitigation strategies, and contingency planning for potential software failures.

6. Lifecycle Costs: The cost of a line of code extends beyond the initial development phase and includes maintenance, updates, and support throughout the software lifecycle. This includes costs for bug fixes, software patches, enhancements, and ongoing technical support.

Given the complexity and criticality of embedded flight software for space missions, the cost per line of code is likely to be higher compared to other software development projects. However, it's important to note that the focus should be on ensuring reliability, safety, and mission success rather than solely on minimizing the cost per line of code. Investing in quality software development practices and rigorous testing can ultimately save costs by reducing the risk of mission failures and costly rework.

The key parts of the requirement that need to be addressed as a collaborative effort among software project managers and Center project and cost offices are:

• Establish a software cost estimation process.  Preferably a project will use an industry-standard process or tool like Constructive Cost Model (COCOMO™) or Software Evaluation and Estimation of Resources - Software Estimating Model (SEER® SEM) tailored for the organization's performance and parameters.
• Document the process, parameters, and results of the software estimations.  Maintain records of the software estimates and data associated with uncertainty and assumptions for the given estimate.
• Document and maintain all associated software cost parameters.  The ability to reproduce the software estimate and understand the risk associated with the estimate is generally contained in the software cost model parameters.
• The cost estimate covers the entire life cycle given the starting point of the estimate. 
• Cost estimates are developed and maintained over the software life cycle including any time the schedule or resource allocations change or requirements change.
• Different software cost estimates are developed for the different classes of software on a project and different organizations developing software on a project.  Software cost parameters are maintained for each software cost estimate.

As software’s importance in missions has grown, the focus on its overall performance both technically and programmatically has also increased. As a result, the software has been blamed for launch slips, mission failures, and contributing to major project cost growth.  Estimation errors resulting in software cost growth rates of 50 percent to over 100 percent are typically identified as a contributing factor in post-development lessons learned.  Barry Boehm's seminal chart shown below captures one of the main reasons why estimation is difficult. The figure illustrates that estimation uncertainty is so large in the early stages of the life cycle, that estimates can be off by a factor of four.

Fortunately, 30 years of cost estimation and modeling research and practice have identified methods for addressing these known issues.  These are reflected in the process described in this requirement.

Software cost estimates are a key activity performed as part of software life cycle planning (NPR 7150.2, Section 3.1) with regular updates made throughout the entire life cycle.  This activity is performed as a part of software tasks of all sizes and all classes with the primary difference being the level of detail and amount of review of the estimates.  Two software cost estimates are required for software class A and B projects with an estimated project cost of 2 million dollars or more.  Two estimates are also recommended when the project scope is not well understood.

 It is highly recommended that model-based and analogy estimates be completed as verification or backup to the primary estimate, which is typically a bottom-up estimate.  Multiple estimates are especially important due to the extensive estimation uncertainty that exists in the early part of the life cycle.  Most models provide a method for expressing the estimated uncertainty that the project team can use as part of the basis of the estimate. 

The key elements of the process are the development of the following:

• Two estimates, one of which is a model-based estimate.
• A software size estimate and other key planning parameters.
• A risk list with mitigations (SWE-151 - Cost Estimate Conditions).
• A documented basis of estimate (BOE) that can be revised as plans are updated.

3.1 Scope of the Job

The project team should determine the scope of the software task, including what the task is and is not, and what it covers. When the project manager collaborates with Center project and cost offices to generate cost estimates, the results of scoping the work is a key input from the project. 

The goal is to identify clearly and in sufficient detail the entire set of software products and activities for producing a good cost estimate.  This includes understanding the intended concept of operations and operating environment to fully characterize the complexity of the work. The project team also identifies values for key planning parameters, including the systems schedule, risk posture, new technology assumptions, expected inheritance, and complexity.

Typically, the scope of a software development effort includes all activities associated with software management, software engineering, programming, and software test engineering over the software life cycle. Included in this effort are software requirements analysis, software design, software implementation, and software integration and testing.  See also SWE-016 - Software Schedule, SWE-033 - Acquisition vs. Development Assessment, SWE-046 - Supplier Software Schedule,

See also Topic 7.03 - Acquisition Guidance, 7.06 - Software Test Estimation and Testing Levels. 5.10 - STP - Software Test Plan, 

3.2 Bottom-Up Estimate

When developing the bottom-up cost estimate, whether at the project level or in conjunction with the appropriate Center offices, the first step is to develop the work breakdown structure (WBS).  A critical aspect of developing the WBS is ensuring that it is easily mapped to other key task breakdowns, including the functional or object decomposition and schedule elements.  It is important to be able to quickly modify the cost, schedule, and functional design as changes occur to keep them consistent.

A recommended sequence of steps for the bottom-up estimate is the following.

• Define a WBS.
• Estimate size and/or effort for each element.
• If available, map analogous size, effort, and/or costs into the current WBS.
• Adjust the baseline for your mission and document how and why it was adjusted as part of the bottom-up estimate.
• Lay the estimate out over time.
• Use an approved tool to get costs.
• Use reference data to assist in an effort estimate or to check the distribution of effort.

3.3 Estimate Software Size

Software size is a required input for the use of a cost model and can also be used as part of a bottom-up estimate.  Estimating software size is one of the most complicated estimation activities. Software size can be measured in various ways but the most common measure is to count lines of code (LOC).  The most used LOC metrics are physical lines (carriage returns excluding comments and blanks) and logical lines, which count logical delimiters such as terminal semi-colons in C along with preprocessor directives and terminal close-braces.  The accuracy of size estimate is greatly improved when anchored by actual code counts from similar software products or similar software components, which require a code counter such as Source Lines Counter (SLiC), found on the NASA Software PAL (See the Resources tab). It is also required to distinguish between new, reused and modified-reused code as the cost of developing code is very different for each type of code. 

A common estimation mistake is the overestimation of cost savings when reused and modified-reused software is included in a system.  Care should be taken not to overestimate the amount of reused code or underestimate the effort to modify that code for use in the system.  Incorporating reused code requires the developer to do more than just testing and may not necessarily result in significant cost savings.  An assessment of the code may be needed to determine how much code can be used in the new system.  Code modification is necessary and that effort is affected by several factors including how effectively the code can be understood (i.e., how well it is commented, how well it is structured, etc.), the amount of documentation that exists for the reused code, whether the code was designed for reuse, etc. 

For more information and a detailed description of how to handle the different types of code, consult Software Engineering Economics, by Boehm ¹³⁶, Software Cost Estimation with COCOMO II, by Boehm, et. al. ¹³⁵, and the Handbook for Software Cost Estimation by Lum, et. al., ³⁶⁶. Other methods that can be used to estimate the software size include function point and use case counting.

3.4 Model-Based Estimates

Model-based estimates are estimates made using parametric cost models. Parametric cost models have been in use at NASA for over 20 years.  The two most used parametric software cost models are COCOMO^TM II and SEER®-SEM.  The Software COCOMO™ Analysis Tool (SCAT) is a probabilistic version of COCOMO^TM II and can be found on the NASA Software PAL. (Refer to the Resources tab for additional information on these tools.) Cost models can be used as a primary estimate early in the life cycle, as a secondary backup estimate for validation, and to help you reason about the cost and schedule implications of software decisions you may need to make. 

Cost models require an estimate of the size of the system and a set of inputs describing the system and development environment; these items are referred to as cost drivers or effort multipliers.  

As an example, COCOMO^TM II has 17 effort multipliers. Each of the COCOMO^TM parameters is associated with up to six ratings – “very low,” “low,” “nominal,” “high,” “very high,” and “extra high”, which correspond to a real number based upon the factor and the degree to which the factor can influence productivity.  A rating equal to 1 does not increase nor decrease the schedule and effort (this rating is called “nominal”). A rating of less than 1 denotes a factor that can decrease the schedule and effort.  A rating greater than 1 denotes a factor that increases the schedule or effort.  For a detailed description of the COCOMO^TM model and the user guides for the commercial cost models, consult Software Engineering Economics, by Boehm ¹³⁶, Software Cost Estimation with COCOMO II, by Boehm, et. al. ¹³⁵, and the Handbook for Software Cost Estimation by Lum, et al.

3.5 Validation and Reconciliation

The purpose of this step is to review the validated estimates concerning the project-imposed budget and schedule and to resolve the differences. If an inconsistency arises, there is a tendency to incorrectly address the issue as only a problem of incorrect estimation. However, in most cases, the real solution is to reduce scope or functionality and then to reduce scope again, until the task fits the budget. This step is another area where the project can provide valuable input to Center costing offices since the project team has the best understanding of the work to be performed. 

See also, SWE-086 - Continuous Risk Management.

3.6 Review, Approve, and Maintain the Estimates

The purpose of this step is to review the software estimates and to obtain stakeholder approval.

The project team should conduct a peer review with the following objectives:

• Confirm the scope and Work Breakdown Structure (WBS) are complete and accurate.
• Verify the methods used for deriving the size, effort, schedule, and cost. • Ensure the assumptions and input data used to develop the estimates are captured in the cost models and are correct, • Ensure that the estimates are reasonable and accurate, given the input data.
• Formally confirm and record the approved software estimates and underlying assumptions for the project.

The stakeholders, which include the software manager, software estimators, line management, and project management, approve the software estimates after the review is complete and problems have been resolved. Remember that costs cannot be reduced without reducing functionality.

The approved software estimates and supporting information such as the BOE should be configuration-managed in a location accessible to stakeholders.  The estimates should be updated in response to changes in schedule, budget, technical scope, development methodology, personnel, or other factors that could alter the accuracy of the approved estimates.

If the project has a cost repository, that repository should be updated with data related to this cost estimate, including software size, effort, and schedule data. See also, SWE-142 - Software Cost Repositories.

For Additional Guidance on Peer Reviews see SWE-087 - Software Peer Reviews and Inspections for Requirements, Plans, Design, Code, and Test Procedures, SWE-088 - Software Peer Reviews and Inspections - Checklist Criteria and Tracking, Topic 7.10 - Peer Review and Inspections Including Checklists. 

3.7 Additional Guidance

Additional guidance related to this requirement may be found in the following materials in this Handbook:

3.8 Center Process Asset Libraries

NASA-specific cost estimation process information is available in Software Processes Across NASA (SPAN), accessible to NASA users from the SPAN tab in this Handbook. Also, see below for SPAN

See the following link(s) in SPAN for process assets from contributing Centers (NASA Only). 

4. Small Projects

The basic estimation techniques apply to small as well as large tasks but they are tailored or scaled to fit the size and scope of the task. One area of concern is with the cost models, as they tend to be calibrated to medium size and larger projects. It is recommended that a greater reliance be placed upon analogy-based cost estimates unless the model being used can be verified for use on small tasks in your local environment.

5. Resources

5.1 References

5.2 Tools

6. Lessons Learned

6.1 NASA Lessons Learned

The NASA Lessons Learned database contains the following lessons learned related to cost estimation:

• Know How Your Software Measurement Data Will Be Used. Lesson Number 1772 ⁵⁶⁷: When software measurement data used to support cost estimates is provided to NASA by a project without an understanding of how NASA will apply the data, discrepancies may produce erroneous cost estimates that disrupt the process of project assessment and approval. Major flight projects should verify how NASA plans to interpret such data and use it in their parametric cost estimating model and consider duplicating the NASA process using the same or a similar model before submission.

6.2 Other Lessons Learned

• Other Lessons Learned from experience (Hihn, Jairus M. JPL. 818.354.1248) and the JPL Cost Estimation Handbook include the following concepts:
  • Keep your history.
    
    • Keep the basis of estimates (BOE) and cost data to support future analogies.
    • Keep data on your estimation vs. actual performance.
    • For incremental deliveries, the best data you have is what it took to get the job done on the earlier deliveries.

• • When the budget is cut, descope the effort.

• • When in sales mode, there is tremendous pressure to be optimistic about the cost and capability to sell the job.

• • Budget 'bogies' get set very early in the life cycle. Sometimes based on casual conversations.
    • You will typically get held to this number!

• • Software estimation is fundamentally an uncertain business under the best of conditions.

• • Common causes of cost growth. Historically, there is a pattern of being overly optimistic in cost estimates by not taking sufficient account for the following:
    • Changes and increases in scope.
    • Concurrent hardware development.
    • Inability to scope software functionality due to inadequate project definition.
    • Software is used for risk mitigation but never planned for upfront.
    • Software is the system complexity sponge.
    • Test Bed and simulator development, availability, and maturity.
    • Optimistic software inheritance assumptions.
    • Is anything new or being done for the first time?
    • Developing Technologies.
    • Autonomy.
    • Precision landing.
    • Hazard avoidance.
    • Design.
    • Language.
    • Tools.
    • Development environment.
    • Processes.
    • Customer or sponsor-initiated changes.
      
      
      
• In addition to these Lessons Learned, Chapter 12 of the GAO Cost Estimating and Assessment Guide, Best Practices for Developing and Managing Capital Program Costs (GAO-09-3SP) ⁴¹⁹, discusses problems with underestimating software costs, including this excerpt from a case study, "The original estimate for the Space-Based Infrared System for nonrecurring engineering, based on experience in legacy sensor development and assumed software reuse, was significantly underestimated. Nonrecurring costs should have been two to three times higher, according to historical data and independent cost estimators. Program officials also planned on savings from simply rehosting existing legacy software, but those savings were not realized because all the software was eventually rewritten. It took 2 years longer than planned to complete the first increment of software."

7. Software Assurance

7.1 Tasking for Software Assurance

7.2 Software Assurance Products

• The cost estimate for the project’s software assurance and safety support.

  
  

  Objective Evidence

  • Evidence of confirmation that the correct number of cost estimates are complete, including any issues or risks, identified.
   Definition of objective evidence

  Objective evidence is an unbiased, documented fact showing that an activity was confirmed or performed by the software assurance/safety person(s). The evidence for confirmation of the activity can take any number of different forms, depending on the activity in the task. Examples are:

  • Observations, findings, issues, risks found by the SA/safety person and may be expressed in an audit or checklist record, email, memo or entry into a tracking system (e.g. Risk Log).
  • Meeting minutes with attendance lists or SA meeting notes or assessments of the activities and recorded in the project repository.
  • Status report, email or memo containing statements that confirmation has been performed with date (a checklist of confirmations could be used to record when each confirmation has been done!).
  • Signatures on SA reviewed or witnessed products or activities, or
  • Status report, email or memo containing a short summary of information gained by performing the activity. Some examples of using a “short summary” as objective evidence of a confirmation are:
    • To confirm that: “IV&V Program Execution exists”, the summary might be: IV&V Plan is in draft state. It is expected to be complete by (some date).
    • To confirm that: “Traceability between software requirements and hazards with SW contributions exists”, the summary might be x% of the hazards with software contributions are traced to the requirements.
  • The specific products listed in the Introduction of 8.16 are also objective evidence as well as the examples listed above.

7.3 Metrics

• Planned SA resource allocation versus actual SA resource allocation.
• Comparison of initial SA cost estimates vs. final cost (capturing assumptions and differences)

See also Topic 8.18 - SA Suggested Metrics. 

7.4 Guidance

Confirm that the required number of software cost estimates are complete.

• Two cost estimate models and associated cost parameters for all Class A and B software projects that have an estimated project cost of $2 million or more.
• One software cost estimate model and associated cost parameter(s) for all Class A and Class B software projects that have an estimated project cost of less than $2 million.
• One software cost estimate model and associated cost parameter(s) for all C and D software projects.
• One software cost estimate model and associated cost parameter(s) for all Class F software projects.

Confirm that the required number of software cost estimates are complete - Obtain the cost estimate(s) from the project or access them in their repository.  Confirm the project generated the correct number of estimates based on SWE-015. Inform the project if a cost estimate is missing and track to closure the resolution to this missing information.

Develop a software assurance cost estimate for the project, including cost estimation associated with handling safety-critical software - Determine if the SA and Software Safety costs are included in the project's software cost estimate. The general rule of thumb is that Software Assurance costs should be between 5 and 6% of the overall software estimated cost.  If the SA and Software Safety costs are not included, work with the project to update the cost estimate to include or reference the SA-generated cost estimate for the project.

Confirm the cost estimate satisfies the conditions in SWE-151 - Cost Estimate Conditions.

Covers the entire software life cycle.

It is based on selected project attributes (e.g., assessment of the size, functionality, complexity, criticality, reuse code, modified code, and risk of the software processes and products).

It is based on the cost implications of the technology to be used and the required maturation of that technology.

Incorporates risk and uncertainty, including cybersecurity.

Includes the cost of the required software assurance support.

Includes other direct costs.

Confirm the cost estimate satisfies the conditions in SWE-151 - Cost Estimate Conditions - Confirm that the project cost estimate(s) contains all of the required information per SWE-151 - Cost Estimate Conditions. Track to closure with the project any missing information.

7.5 Additional Guidance

Additional guidance related to this requirement may be found in the following materials in this Handbook: