1. Requirements

2.2.3 The project shall establish, document, and maintain at least one software cost estimate and associated cost parameter(s) that satisfies the following conditions:
     a. Covers the entire software life cycle.
     b. Is based on selected project attributes (e.g., assessment of the size, functionality, complexity, criticality, and risk of the software processes and products).
     c. Is based on the cost implications of the technology to be used and the required maturation of that technology.

1.1 Notes

In the event there is a decision to outsource, it is a best practice that both the acquirer (NASA) and the provider (contractor/sub) should develop software cost estimates and negotiate to arrive at an acceptable estimate.

1.2 Applicability Across Classes

Class D Non-Safety Critical and Class E Non-Safety Critical are labeled with "P (Center)." This means that a non-null set of local requirements or procedures describe cost estimation sufficiently to meet the intent of this requirement.

2. Rationale

Credible software cost estimates are a key activity performed as part of a project and software life cycle management (NPR 7150.2, NASA Software Engineering Requirements, Section 2.2) with regular updates made throughout the entire life cycle. Credible software cost estimates are required to meet NASA policies. The reliability of project cost estimates at every stage in the project development process is necessary for responsible fiscal management. Unreliable cost estimates result in severe problems in planning and budgeting which then result in staffing and budgeting decisions that could impair effective use of resources.

3. Guidance

The key parts of the requirement that need to be addressed are:

• Establish a software cost estimation process. Preferably a project will use an industry standard process or tool like Constructive Cost Model (COCOMO) or SEER SEM tailored for the organization performance and parameters.
• Document the process, parameters and results of the software estimations. Maintain records of the software estimates and data associated with uncertainly and assumptions for the given estimate.
• Document and maintain all associated software cost parameters. The ability to reproduce the software estimate and understand the risk associated with the estimate is generally contained in the software cost model parameters.
• The cost estimate covers the entire life cycle given the starting point of the estimate.
• Cost estimates are developed and maintained over the software life cycle including any time the schedule or resource allocations change or requirements change.
• Maintain all assumptions and data associated with the technology to be used on the software project, maintain good data and records associated with any cost saving identified from the use of software technology.
• Make sure that your software estimate maintains data showing cost required for maturation of any planned use of new technology. New technology can be defined as any new software development environments or tools or software development associated with a new or unproven technology.
• Different software cost estimates are developed for the different classes of software on a project and for different organizations developing software on a project. Software cost parameters are maintained for each software cost estimate.

As software's importance in missions has grown, the focus on its overall performance both technically and programmatically has also increased. As a result, software has been blamed for launch slips, mission failures, and contributing to major project cost growth. Estimation errors resulting in software cost growth rates of 50 percent to over a 100 percent are typically identified as a contributing factor in post development lessons learned. Barry Boehm's ¹³⁶ seminal chart shown below captures one of the main reasons why estimation is difficult. The figure illustrates that, estimation uncertainty is so large in the early stages of the life cycle, that estimates can be off by a factor of four.

Fortunately, 30 years of cost estimation and modeling research and practice have identified methods for addressing these known issues. These are reflected in the process described in this SWE.

Software cost estimates are a key activity performed as part of Software Life-Cycle Planning (NPR 7150.2, Section 2.2) with regular updates made throughout the entire life cycle. This activity is performed as a part of software tasks of all sizes and all classes with the primary difference being the level of detail and amount of review of the estimates. While only one estimate is required it is highly recommended that additional model-based and analogy estimates be completed as verification or backup to the primary estimate, which is typically a bottom-up estimate. Multiple estimates are especially important due to the extensive estimation uncertainty that exists in the early part of the life cycle. Most models provide a method for expressing the estimate uncertainty that the project team can use as part of the basis of estimate.

The key elements of the process are the development of the following:

• Two estimates, one of which is a model based estimate.
• A software size estimate and other key planning parameters.
• A risk list with mitigations.
• A documented basis of estimate (BOE) that can be revised as plans are updated.

Scope the job

The project team determines the scope of the software task, including what the task is and is not, and what it covers.

The goal is to identify clearly and in sufficient detail the entire set of software products and activities for producing a good cost estimate. This includes understanding the intended concept of operations and operating environment to fully characterize the complexity of the work. The project team also identifies values for key planning parameters, including the systems schedule, risk posture, new technology assumptions, expected inheritance, and complexity.

Typically, the scope of a software development effort includes all activities associated with software management, software engineering, programming, and software test engineering over the software life cycle. Included in this effort are software requirements analysis, software design, software implementation, and software integration and testing.

Bottom-Up Estimate (BOE)

When developing the bottom-up cost estimate, the first step is to develop the work breakdown structure (WBS). A critical aspect of developing the WBS is ensuring that it is easily mapped to other key task breakdowns, including the functional or object decomposition and schedule elements. It is important to be able to quickly modify the cost, schedule, and functional design as changes occur to keep them consistent.

A recommended sequence of steps for the bottom-up estimate is the following:

• Define a WBS.
• Estimate size and/or effort for each element.
• If available, map analogous size, effort, and/or costs into current WBS.
• Adjust baseline for your mission and document how and why it was adjusted as part of the BOE.
• Lay it out over time.
• Use approved tool to get costs.
• Use reference data to assist in effort estimate or to check distribution of effort.

Estimate Software Size

1. Software size is a required input for the use of a cost model and can also be used as part of a bottom-up estimate. Estimating software size is one of the most complicated estimation activities. Software size can be measured in various ways but the most common measure is to count lines of code (LOC). The most used LOC metrics are physical lines (carriage returns excluding comments and blanks) and logical lines which count logical delimiters, such as terminal semi-colons in C along with pre-processor directives and terminal close-braces. The accuracy of size estimate is greatly improved when anchored by actual code counts from similar software products or similar software components, which require a code counter, such as Source Lines Counter (SLiC), found on the NASA Software PAL (See the Resources tab ). It is also required to distinguish between new, reused and modified-reused code as the cost of developing a code is very different for each type of code. For more information and a detailed description of how handle the different types of code, consult Software Engineering Economics, by Boehm ¹³⁶, Software Cost Estimation with COCOMO II, by Boehm, et. al. ¹³⁵, and the Handbook for Software Cost Estimation by Lum, et. al. ³⁶⁶, also listed in the Resource tab of this SWE. Other methods that can be used to estimate the software size include function point and use case counting.

Model-Based Estimates

Model-based estimates are estimates made using parametric cost models. Parametric cost models have been in use at NASA for over 20 years. The two most used parametric software cost models are COCOMO II and SEER-SEM. SCAT is a probabilistic version of COCOMO II and can be found on the NASA Software PAL. (Refer to the Resources tab for additional information on these tools.) Cost models can be used as a primary estimate early in the life cycle, as a secondary backup estimate for validation, and to help you reason about the cost and schedule implications of software decisions you may need to make.

Cost models require an estimate of the size of the system and a set of inputs describing the system and development environment; these items are referred to as cost drivers or effort multipliers.

1. As an example, COCOMO II has 17 effort multipliers. Each of the COCOMO parameters is associated with up to six ratings – "very low," "low," "nominal," "high," "very high," and "extra high", which correspond to a real number based upon the factor and the degree to which the factor can influence productivity. A rating equal to 1 does not increase nor decrease the schedule and effort (this rating is called "nominal"). A rating less than 1 denotes a factor that can decrease the schedule and effort. A rating greater than 1 denotes a factor that increases the schedule or effort. For a detailed description of the COCOMO model and the user guides for the commercial cost models, consult Software Engineering Economics, by Boehm ¹³⁶, Software Cost Estimation with COCOMO II, by Boehm, et. al. ¹³⁵, and the Handbook for Software Cost Estimation by Lum, et. al. ³⁶⁶, also listed in the Resource tab of this SWE.

Risk

The purpose of this step is to identify common software risks, assess their impact on the cost estimate, and make revisions to the estimate based on these impacts.

Compile known major risks.

As you generate the software risk list, consider the following:

• What WBS elements are affected.
• When it would occur.
• Likelihood of occurrence.
• Impact.
• What it would cost to mitigate the impact.
• Stakeholder and user perspectives.

Risk can be estimated and analyzed in various ways:

• Risk Matrices (eg 5x5).
• Expected Risk (Likelihood * Impact).
• Monte Carlo techniques to produce distributions.

The project team may add the results of the expected risk impact or identified mitigations to the bottom up estimate. The cost models have Monte Carlo algorithms which produce a cost distribution and the identified risks are used to determine which model inputs will be entered with a wider range.

Validation and Reconciliation

The purpose of this step is to review the validated estimates with respect to the project-imposed budget and schedule, and to resolve the differences. If an inconsistency arises, there is a tendency to incorrectly address the issue as only a problem of incorrect estimation. However, in most cases, the real solution is to reduce scope or functionality, and then to reduce scope again, until the task fits the budget.

Review and Approve the Estimates

The purpose of this step is to review the software estimates and to obtain stakeholder approval.

The project team conducts a peer review with the following objectives:

• Confirm the scope and WBS are complete and accurate.
• Verify the methods used for deriving the size, effort, schedule, and cost. Signed work agreements may be necessary.
• Ensure the assumptions and input data used to develop the estimates are correct.
• Ensure that the estimates are reasonable and accurate, given the input data.
• Formally confirm and record the approved software estimates and underlying assumptions for the project.

The stakeholders, which include the software manager, software estimators, line management, and project management, approve the software estimates after the review is complete and problems have been resolved. Remember that costs cannot be reduced without reducing functionality.

Additional guidance related to software cost estimation may be found in the following related requirements in this Handbook:

4. Small Projects

The basic estimation techniques apply to small as well as large tasks but they are tailored or scaled to fit the size and scope of the task. One area of concern is with the cost models, as they tend to be calibrated to medium size and larger projects. It is recommended that that a greater reliance be placed upon analogy-based cost estimates unless the model being used can be verified for use on small tasks in your local environment.

5. Resources

5.1 Tools

Tools to aid in compliance with this SWE, if any, may be found in the Tools Library in the NASA Engineering Network (NEN).

NASA users find this in the Tools Library in the Software Processes Across NASA (SPAN) site of the Software Engineering Community in NEN.

The list is informational only and does not represent an “approved tool list”, nor does it represent an endorsement of any particular tool. The purpose is to provide examples of tools being used across the Agency and to help projects and centers decide what tools to consider.

6. Lessons Learned

The NASA Lessons Learned database contains the following lessons learned related to cost estimation:

• Know How Your Software Measurement Data Will Be Used. Lesson Number 1772: When software measurement data used to support cost estimates is provided to NASA by a project without an understanding of how NASA will apply the data, discrepancies may produce erroneous cost estimates that disrupt the process of project assessment and approval. Major flight projects should verify how NASA plans to interpret such data and use it in their parametric cost estimating model, and consider duplicating the NASA process using the same or a similar model prior to submission. ⁵⁶⁷

• Flight Software Engineering Lessons. Lesson Number 2218: The engineering of flight software is a major consideration in establishing JPL project total cost and schedule because every mission requires a significant amount of new software to implement new spacecraft functionality. Constraints to the development and testing of software concurrent to engineering the rest of the flight system has led to flight software errors, including the loss of some missions. The findings of several JPL studies and flight mishap investigations suggest a number of recommendations for mitigating software engineering risk ⁵⁷².

Other Lessons Learned from experience (Hihn, Jairus M. JPL. 818.354.1248) and the JPL Cost Estimation Handbook ³⁶⁶ include the following concepts:

• Keep your history.
  • Keep BOE (Basis of Estimate) and cost data to support future analogies.
  • Keep data on your estimation vs. actual performance.
  • For incremental deliveries the best data you have is what it took to get the job done on the earlier deliveries.

• When budget is cut, descope the effort.

• When in sales mode, there is tremendous pressure to be optimistic on cost and capability to sell the job.

• Budget 'bogies' get set very early in life cycle. Sometimes based on casual conversations.
  • You will typically get held to this number!

• Software estimation is fundamentally an uncertain business under the best of conditions.

• Common causes of cost growth. Historically, there is a pattern of being overly optimistic in cost estimates by not taking sufficient account for the following:
  • Changes and increases in scope.
    • Concurrent hardware development.
    • Inability to scope software functionality due to inadequate project definition.
    • Software is used for risk mitigation, but never planned for up front.
    • Software is the system complexity sponge.
    • Test Bed and simulator development, availability and maturity.
  • Optimistic software inheritance assumptions.
  • Anything new or being done for the first time.
  • Developing Technologies.
    • Autonomy.
    • Precision landing.
    • Hazard avoidance.
  • Design.
  • Language.
  • Tools.
  • Development environment.
  • Processes.
  • Customer or sponsor-initiated changes.

• Common costs left out of cost estimates.
  • Review preparation.
  • Documentation.
  • Fixing Anomalies and EGR (Engineering Change Requests)'s.
  • Testing.
  • Maintenance.
  • Basic management and coordination activities.
  • Technical leads do spend time doing management activities.
  • Mission Support Software Components.
  • Development and test environments.
  • Travel.
  • Training.

In addition to these Lessons Learned, Chapter 12 of the GAO Cost Estimating and Assessment Guide, Best Practices for Developing and Managing Capital Program Costs (GAO-09-3SP) ⁴¹⁹, discusses problems with underestimating software costs, include this excerpt from a case study, "The original estimate for the Space Based Infrared System for nonrecurring engineering, based on actual experience in legacy sensor development and assumed software reuse, was significantly underestimated. Nonrecurring costs should have been two to three times higher, according to historical data and independent cost estimators. Program officials also planned on savings from simply rehosting existing legacy software, but those savings were not realized because all the software was eventually rewritten. It took 2 years longer than planned to complete the first increment of software."