1. Purpose

The use of this topic is to provide the basics of assessing software engineering project processes and products for compliance with an established set of criteria.  The information is useful to anyone needing to provide an independent assessment of compliance with a set of defined processes, standards, requirements, etc.

1.1 Basic Principles

There are a few basic principles that apply to audits, regardless of audit type or purpose.  These principles ensure objectivity in the audit conduct and outcome as well as providing the basis for an efficient audit.

1.2 Basic Objectives

Audits provide management with information about the project team, the project processes and help identify best practices and areas of improvement.

Audits are useful to assess:

• Adequacy of project plans, processes, systems
• Compliance to those plans, processes, systems
• Effectiveness of those plans, processes, systems, and internal project controls on those processes
• Product fitness for use/compliance to specifications
• Areas for improvement

The results of audits allow project management to make adjustments and corrections to ensure high-quality products are being produced and delivered and that the team is functioning efficiently and effectively. Trending audit results over time allow management to identify systemic issues and areas of risk while monitoring the effect of process and product improvements.

2. Planning

The Lead Auditor manages the planning process with the audit team, once the team is identified (see Team Selection below).  This activity results in an audit plan or at least a summary of the planned audit activities that can be included in the audit notification.

If audits are not one-time or triggered by a change in governing documents, establish an overall audit plan that captures the following:

• Audit planning – Align audit schedules with Project activities for maximum effectiveness and efficiency; for example, a process planning audit is much more effective during the project planning phases, and a software configuration management audit is difficult if the primary configuration manager is unavailable.
• Audit schedule – Establish a schedule that ensures full and periodic coverage of governing documents within a specified timeframe for a single project or all projects within the auditor’s areas of responsibility. For example, establish a schedule to audit all areas of a project’s software development plan over the course of a year so that at the end of the year, all areas of the software development plan have been audited. Other triggers for audits include project milestones, changes in governing documents (e.g., a new version of an applicable standard is released and the project has a specified timeframe by which to comply), major staffing changes, identification of a major defect.
• Audit criteria/checklists – Use existing checklists or create a library of checklists to ensure full coverage of the governing documents and audit objectives for the audits on the schedule. Maintain checklists as the governing documents change; i.e., update the checklists with new revisions of their source documents. Audit checklists can take many forms, are captured in the audit records, and can be included by reference in the audit plan. See Checklist examples in the Resources tab.
• Audit methods – Determine and document the techniques to be used for the scheduled audits. Interviews, observing work in progress, remote record reviews, and onsite record reviews are all valid methods for an audit, and more than one method is typically used for each audit.
• Audit report generation and delivery – Work with the project to determine the list of audit report recipients and how the report will be delivered (email, posted to a shared repository, printed copy delivered to the project, etc.). Capture this information in the audit plan.
• Audit templates – Many outputs from an audit can be generated using templates to ensure completeness and consistency. Items that lend themselves to templates include the audit notification letter, in-brief, out-brief, and the audit report. Reference any templates to be used in the audit plan.
• Records to capture and retain – Identify in the audit plan the records to be captured as a record of the audit. Records include:
  • Audit notification letter/email
  • Audit plan
  • Completed audit checklists
  • Audit report
  • Audit notes, including those from the in-brief and out-brief
  • Audit Findings/Opportunities for Improvement (OFIs)/Observations
  • Any project evidence collected before, during or after (to close out open items) the audit
  • Corrective action plans and closure evidence to address audit Findings
• Records repository and retention guidance – Capture in the audit plan the location for the audit records and any applicable retention policies that apply.
• Follow-up activities and timeframes – Audits are not typically over once the onsite work is complete. Audit reports need to be delivered in a timely fashion. Findings need to be tracked to closure within a specified timeframe. Summary reports need to be delivered to auditor management. Capture these activities and their associated timeframes in the audit plan.

2.1 Audit Scope

For every audit, establish the scope of the audit to focus the audit team and to allow the project to understand the impact the audit will have on their work. Review results from any previous audits of the organization or project to be audited to help establish the scope and focus areas to be included. Key items to consider from past audits include:

• Major non-conformances addressed as a result of the last audit
• Trends of results across a series of audits
• Lingering Findings or Observations from past audits
• Areas that narrowly passed a previous audit
• Areas not assessed during the previous audit

Capture the audit scope in the audit plan, including:

• The audit purpose – the objective(s) of this particular audit; e.g., to assess a project’s compliance with their configuration management plan
• The audit criteria – the governing document(s) that will be used as the measuring stick for the audit; e.g., the project’s software development plan and related work instructions
• The project, processes, and products to be assessed – the specific project and project processes and artifacts to be audited
• The project personnel/roles to be audited – the list of personnel, typically by role, who will need to be available for the audit

The audit scope will also be conveyed to the project in the audit notification, so completeness and clarity are important to establish a clear understanding between the audit team and the project or organization being audited.

2.2 Audit Notification

It is important to set expectations for an audit. The audit team provides the project or organization to be audited with an audit notification letter (email) ahead of the audit. A best practice is to provide this notification 2 weeks to 30 days ahead of the audit, depending on the agreements with the project and program/project plans. The audit notification includes key pieces of information for both the audit team and the project to be audited. Recommended content includes:

• Audit purpose and scope – See Audit Scope in this topic for more information
• Schedule / Agenda – Based on past experience or using a known estimation method/tool and data for a similar scope and a similarly experienced audit team, generate an agenda to fit the audit timeframe, so both the audit team and project understand the audit timeline. The project can use this information to plan their resources needed to support the audit. Be prepared to adjust onsite if interviews run longer than planned, interviews lead down a new risk trail or access to records is interrupted or slowed, etc. Know that once the audit notification letter is received by the project, be prepared to iterate on the dates and agenda to ensure the project can fully support the audit. When establishing the agenda, include time for:
  • Audit team to prep and wrap-up daily
  • Audit team to prepare the out brief
  • Meals and breaks
  • Re-visits if findings/concerns are addressed while onsite
• Audit Team/Roles – List by name and role and whether attending in person (preferred) or remotely
• Auditees/Roles – List by name, if known, but this will likely be a list of the project roles needed to support the audit with enough detail to allow the project to identify the personnel needed to support the audit and the timeframes that personnel will be involved in the audit
• Governing Documents – List by name and version the documents serving as the basis of the audit criteria; e.g., software development plans, standards
• Onsite logistics needed from the project (see Audit Logistics in this topic) – Provide a list of items the audit team will need while onsite to efficiently conduct the audit. Consider the following items:
  • Webex / Teleconference, if needed
  • Location (e.g., conference room, audit team meeting room)
  • Projector and screen
  • Building and site maps
  • Access to the Internet, printer, etc.
  • Records/materials and/or repository access needed from the project (e.g., arrangements for accessing project artifacts behind firewalls)

2.3 Audit Logistics 

It is important to establish logistical items for the audit.  Consider:

• Travel – If the audit occurs at a location different than the location of the audit team, as soon as the audit timeframe is agreed upon with the project, any required travel should be planned.  This ensures the entire audit team is onsite and ready for the audit to occur.
• Repository for audit records – It is important to capture the evidence from the audit in a known repository accessible to the audit team and management.  These records serve as the official record of the audit as well as provide information for planning future audits. For example, identify a folder in a general audit repository to hold the items such as the audit plan, audit notification letter/email, completed audit checklists, audit notes, audit results, audit reports, audit results responses from the project, etc.)
• Method for tracking results of this audit to closure – If not captured in an overall audit plan, identify and document a method for working audit Findings to closure.  This could be as simple as a spreadsheet or an existing tool such as JIRA, Confluence, Sharepoint could be used.  Remember to consider confidentiality requirements when establishing the method for tracking audit results to closure (e.g., select a tool that has provisions for access permissions and restrictions).

2.4 Audit Communication

For an audit to be successful, communication between the audit team, the project, and management before, during, and after the audit is key.  Consider the following as key communication items:

•  Audit Authorization – Get the appropriate authority to authorize the audit in written form so a record of this authorization can be captured in the audit records.  This could be software assurance management, a contracting officer for audits of contracted development, or some other designated, pre-determined authority.  In some cases, the audit team may need to sign confidentiality agreements.
• Audit notification – Send the audit notification (see Audit Notification in this topic) to the project via email, in-person delivery, or another appropriate method and within the pre-determined timeline.
• Dates and logistics – Iterate with the project being audited on audit dates (see Audit Notification in this topic) and logistics (See Audit Logistics in this topic) until everyone is in agreement.  Iterations could be via email, in person, or by phone, to capture any decisions made for the audit records.

2.5 Audit Team Selection

When establishing an audit team, work with appropriate management to obtain the personnel and other resources needed to conduct the audit.  Keep in mind the goals of independence and objectivity while obtaining personnel with the appropriate skill sets and knowledge to efficiently conduct the audit and capture the notes and evidence needed to support the audit conclusions.

 Suggested considerations include:

• Expertise in the audit criteria / governing documents – Most of the audit team needs to be familiar with the audit criteria or governing documents used as the basis for the audit.  The audit team can do some familiarization as part of audit preparation, but familiarity with the relevant standards, process documents, development plans, etc. from other roles these personnel hold is even better because it gives them good, practical knowledge to use when interviews open new paths to explore during the audit.
• Experience or training in performing audits – The Lead Auditor, at a minimum, should be an experienced auditor.  This helps ensure a consistent process across audits and can be an asset for keeping the team on track during the audit, especially when a portion of the team is there for their technical background.  A mix of experienced auditors and newly trained auditors can be used to help newer auditors gain valuable experience.
• Involvement in the work being audited or any other potential conflict of interest – Audit team members must not be involved in any of the work being audited, or have any other ties to the work that would make their participation in the audit appear to be a conflict of interest. These connections to the work being audited can taint or invalidate the audit results (think about the concept of someone checking their own work) and reduce or even eliminate true objectivity and independence which are key concepts when performing an audit. 
• Skills for the role they will fulfill – The audit team has a few key roles that need to be fulfilled, and each role has its own set of skills needed for success.  These key roles include:
  • Lead Auditor – Person who leads the team, manages the overall audit work, interfaces with the project during planning, leads the onsite audit, and interfaces with the project to ensure audit results are tracked to closure.
  • Facilitator – Person who ensures the team remains on track and on schedule during the audit.  This person could have roles as a time-keeper or as the person who keeps order and prevents the team from straying too far off the main audit focus.
  • Documentarian/note-taker – Person who captures the interview notes, observation notes, attendance, evidence (actual or references), and other records during the audit to ensure all the information needed to back up the audit conclusions is collected and captured in the audit records.
  • Technical expert – Person(s) with experience in the focus of the audit.  Technical expertise could be configuration management, test/verification, risk management, etc.  Technical expertise could also be guidance, navigation, and control (GN&C), ground systems, flight operations, etc. to help the audit team identify areas of risk.
  • General auditor – Person(s) with audit experience or trainees who are not technical experts but may have some familiarity with the governing documents.

 When establishing an audit team, other decisions that need to be made include:

•  Team size/number of auditors – An audit team should not overwhelm the project being audited, so 10 auditors for a project team of 4 or 5 developers might be a mismatch.  On the other hand, a single auditor to fill the standard roles of Lead Auditor, Facilitator, Documentarian, and technical expert is inefficient and ineffective.  The person asking the questions cannot take notes and keep the discussion on track without slowing down the audit progress.  Team size is often based on the necessary technical or process expertise in addition to the standard roles, so a minimum team size might be 3 or 4 to conduct an audit efficiently.
• Lead auditor and other roles – Determine what roles are key for each particular audit and the number of technical experts needed and identify audit team members to fill those roles.   Be sure any necessary training, such as basic auditing for technical experts or review of the governing documents/audit criteria for the entire team, is completed well ahead of the audit.
• Auditor participation method – Identify who will audit in person and who will be remote, if applicable.  For a multi-day, offsite audit, it may be sufficient for a technical expert to participate by teleconference for a 2-hour audit session where their expertise is needed.  The Lead Auditor should be onsite for the audit as should the Documentarian, but where good teleconference facilities exist, it may be acceptable for the Documentarian to participate remotely.

2.6 Audit Checklists 

Having tools in place to support audits is key to consistent, well-planned, complete, and accurately documented audits.   One key tool is the audit checklist.                 

When establishing audit checklists or audit criteria, start with the governing documents. Using those governing documents, generate questions as the basis for collecting evidence by which compliance will be assessed/measured.  Potential sources, not an exhaustive list, for checklist questions include:

• NASA standards
• NASA Procedural Requirements (NPRs)
• An existing baseline checklist created from an entire governing document (tailor such checklists to focus on the specific audit area of interest)
• Requirements specifications  
• Project processes and procedures
• Software Development Plans (SDPs) / Software Management Plans (SMPs)
• Past audit results and corrective action plans, if available
• Known areas of concern (based on project performance, management reviews, project reviews, etc.) 

Some general concepts and questions to keep in mind when creating checklists include:

• Does the person performing the task have access to and understand the work procedures for their tasks?
• Are the procedures being used by the project up-to-date?
• Does the person perform the task have the right training, skills, experience?
• Is the work being performed efficiently and effectively?
• Does the person have the right tools, environment, infrastructure, etc. to do the task? 
• Are the work products verified, approved, controlled as required?
• Are defects reported and tracked to closure?
• Are the appropriate records/task evidence being captured?

Questions should have the following characteristics:

• Answers should be known and expected based on pre-audit work and the auditors’ knowledge of the project based on project documentation, familiarity with the team’s processes and work products, review of project artifacts/work products, etc. 
• Questions should focus on the work, not the individuals.
• Questions should be open-ended and context-free, for example, “What procedures do you follow when performing your work?” is a better audit question than “Do you follow work instruction ABC when you do your work?” 

A checklist or set of audit criteria often takes the form of a spreadsheet for ease of capturing notes, evidence references, and audit conclusions all in one place. A spreadsheet can include the following columns:

• Questions – often answerable by yes/no or in the form of “show us evidence that…”
• Pass / Fail slots per question
• Action slots per question – e.g., to capture follow up work by the project or audit team to close a possible Finding or provide additional evidence
• A place to capture evidence viewed to assess compliance per question
• A place to capture the rationale for the pass / fail results per question
• The priority of questions/topics (H/M/L – in case the audit team needs to reduce or adjust scope in real-time)

3.0 Conducting

With the planning complete and the audit tools in place, it is time to conduct the actual audit.  There are several steps to an audit, including pre-audit preparation, audit activities, and post-audit close-out work.  The following sections describe each step and offer guidance for success.

3.1 Audit Team Preparation

Before the audit team arrives for the audit, the following preparation is completed:

• Review governing process documentation – If not performed as part of checklist generation, obtain and review governing process documentation.  Process documentation may be Project, Contract, Agency, or Center-specific, and the audit team needs to be familiar with the specific governing documents for this audit.  Familiarity with the entire governing document, not just the area(s) under audit, is important as process areas can influence each other and audit discussions can flow into those related areas. Having the background knowledge of the governing documentation, including work-level process instructions, helps put discussions with project personnel into context as well as provides a basis for understanding personnel responses to questions.
• Audit plan and checklist review – It is important for the entire audit team to be familiar with the audit focus, the questions to be asked, the activities to be witnessed, the audit team member roles, etc. The audit team lead ensures this review is completed, and everyone understands their role in the audit and the purpose of each item in the checklist.
• Prior audit results review – If not performed as part of checklist generation, the audit team reviews the results of any relevant prior audits for this project.  Part of the audit could include follow-up on the root cause and corrective action for any prior Findings or systemic issues and a few spot checks on any specific areas of concern.
• Pre-audit assessment – Some elements of the audit can be completed before the audit begins.  If the audit team has or can obtain access to project repositories and records, the team carries out document reviews to assess checklist items that can be completed without involving the project.  This provides a basis for tailoring of checklist questions to focus on areas this pre-audit assessment identifies as potential compliance concerns.  For example, compliance with a process requirement indicating that all problem reports will include specific fields and signatures can be checked by sampling project problem reports. Any concerns found during this review can be used to tailor checklist questions to determine if the project is aware of the concerns and what is being done to address them.

3.2 In-brief

It is important for the audit team and the project to have a common understanding of the audit plans.  Initially, this was conveyed in the audit notification (see Audit Notification in this guidance), but on the first day of the audit, the ground rules should be repeated to ensure everyone is on board and has the same interpretation of the audit plan.  This is also the time for the audit team and project to ask and have answered any general questions related to the audit.                                        

The Lead Auditor typically runs this meeting on day 1 of the audit.  The in-brief can be short (15-30 minutes) and focused, but covers the following information: 

• Audit team introductions – Introduce the audit team, including any audit team members who are participating remotely.
• Project team introductions – While the entire project team may not attend the in-brief, key players such as the project manager and team leads should be invited to gain an understanding of the audit.  At the project’s discretion, other key auditees may choose to attend the in-brief.
• Audit scope review – Review the audit focus/scope, including any changes in the pre-delivered schedule and/or agenda.
• Audit method review, as needed – If the audit will include more than just personnel interviews, it is a good idea to review those audit activities during the in-brief.  Include a brief description of any planned document reviews, interviews, observations.   Remember to inform the group that auditing is a sampling technique, not an exhaustive review of all work products and interviews with every project team member.
• Project team participation – State expectations of the project team, if not covered elsewhere.  The goal should be to minimize the team’s time away from their project work while providing sufficient time to collect information for the audit.  Review the overall audit agenda and indicate when specific project roles are requested to participate.
• Closing expectations and activities – State for the group the expectations for the out brief (see Outbrief in this guidance) and post-audit activities/timeframes (see Follow-Up and Close-out in this guidance).  The out brief will occur on the final day of the audit and is typically attended by the same key personnel who attended the in-brief.  They will want to know any preliminary results as well as any timeframes for the final audit report and anticipated response by the project.
• Audit logistics – Both the Lead Auditor and project manager cover this material.  The Lead Auditor confirms any logistics such as Webex, teleconference, Internet / Wi-Fi access, room assignment(s), etc. previously provided by the project.  The project manager reviews building logistics (e.g., emergency evacuation instructions, restrooms, etc.), lunch possibilities, etc.

It is important to have a record of the in-brief, so the audit team captures the in-brief presentation(s), in-brief attendees with their roles, and notes from any discussion.  Typically, this information is captured in the official audit record by the audit team Documentarian / note-taker.

3.3 Interviews/process observations/meeting observations

Once the in-brief is concluded, the core audit work begins.  The team follows the agenda and checklist, adjusting as needed based on where the audit discussion takes the team.  The Lead Auditor or audit Facilitator ensures the audit team remains as close to schedule as possible while allowing sufficient time for an investigative and informative discussion.

For a full perspective on process compliance, interviews include project personnel at all project levels – managers, leads, engineers.  The Documentarian captures the names and roles of auditees in the official audit notes.  Consider confidentiality in choosing people who will be interviewed together.  The audit team should not interview workers in the same interview as their supervisors to help ensure that interviewees remain honest about what they actually do.

For each interview session, the Lead Auditor:

• Introduces the audit team and their roles
• States the interview purpose
• Informs the auditee(s) how the session will be conducted (e.g., notes will be taken, the focus is on the process, not the auditee, etc.) 

A few key concepts help keep the interview focused:

• Follow the prepared checklist
• Keep language-neutral – avoid leading the auditee answers or giving the impression that “all is well” or “all is bad”
• Auditee should do most of the talking
• Don’t provide feedback during interviews, observations, etc. – just collect evidence; feedback can be provided during out brief
• The auditee should not select record samples but could direct the auditor to a repository where the auditor selects the sample

These concepts help the audit team identify issues and concerns:

• View evidence: “if it’s not written down, it never happened.”
• Follow the evidence until satisfied or a dead end or a time constraint is reached – remember the need is to sample enough evidence to assess compliance       
• Look for trends in the interviews, collected evidence, etc.
• Look for process escapes

When interviews are not sufficient or are not appropriate to gather the required evidence, witness project activities to confirm employees are familiar with processes, procedures, policies, and their related roles and responsibilities. 

While the Documentarian captures the official audit record, the entire audit team takes their own notes to ensure the official audit record is complete and accurate.  Note that comparisons occur in the daily audit team meetings (see Daily audit team meeting in this guidance), and any discrepancies in evidence references can be cleared up with the project before the audit concludes.   The Documentarian captures the following evidence for the audit record:

• Objective criteria / Records assessed - Capture record identification and location (document name, document id, date, repository, URL, etc.) to uniquely identify the records assessed during the audit
• Who said what during interviews – Capture (for corroborating evidence) along with the interview notes, each auditee’s name and project role as well as the date/time of each interview
• Date/time/location/event - For observed meetings, processes, and witnessed activities, along with the notes identifying the process being assessed and what was observed/witnessed, capture the date, time, location, and event name or description.  Include audit team members participating as well as any project personnel and their roles.

3.4 Daily audit team meeting

At the start of each audit day or at the end of the day or both the audit team meets alone to plan and prepare.  

At the beginning of each day, the audit team reviews the plan for the day and discusses any adjustments needed to remain on track.  Any adjustments are communicated as soon as possible to the project manager to minimize the impact on the project team’s work. 

At the end of each day, the audit team reviews and compares notes to ensure the audit record is complete and accurate and makes any clear pass/fail decisions based on the evidence collected.  The team also comes to a consensus where possible on findings, observations, etc.   Collectively, the team:

• Looks for trends in the interviews, collected evidence, etc.
• Looks for multiple interviewees making the same statement
• Identifies if multiple auditors heard the same thing

This information is captured in the Documentarian’s audit record. 

The team discusses any open questions and determines if a follow-up is needed and where that might occur in the audit schedule.  If the team realizes they are off schedule, re-planning can occur to ensure the highest risk or most critical items are assessed during the audit. 

To ensure the out-brief presentation is sufficiently prepared, the audit team uses these daily meetings to collect content for the presentation.  This content is adjusted daily as appropriate until the presentation is finalized and becomes part of the audit record.

3.5 Out brief

The last official activity during the audit is an out brief presentation from the audit team.  The project will want results, but the audit team may need some time to process the collected evidence.   During the daily team meetings (see Daily audit team meeting in this guidance), the audit team should have been collecting content for this out-brief to provide the project with as much information as possible before the official audit report is generated.  Neither the audit team nor the project wants any surprises when that report is delivered, so the out brief is a preview of the report content. 

The out-brief presentation is not required to be lengthy, but includes the following information: 

• Summary of audit – Repeat a summary of the audit in-brief content (see In-brief section of this guidance), noting any deviations made during the audit.
• General take-aways – Capture and share any overall impressions, observations, etc. for the project, both good and bad.  This could include best practices identified by the audit team, project team cooperation and openness, concerns about missing or frequently changing project documentation, etc.  This is not a list of official findings and observations but is a general set of information for the project.
• Findings, Observations, OFIs – If available, provide any identified Findings, Observations, OFIs from this audit.  Full details may not yet be available, so sharing at a high level is acceptable, but make it clear to the project that the audit report will contain the official results. Even at a high level, these concerns should be clearly written because the project is likely to begin to push back even before the final report is delivered.  The audit team needs to be prepared for some discussion at this point in the out brief.
  Often with fairly large audits, the audit team holds preliminary findings meeting with the project to ensure that the audit team has understood what they heard. This may also give the auditees a chance to provide some evidence that was overlooked during the interviews.
• Next steps/actions – Include in the presentation the next steps and actions for this audit with due dates and assignments.  Typical items include:
  • Audit report generation and delivery – the audit team, typically delivered the official set of audit results to the project 2 weeks to 30 days after the audit ends
  • Timeframe for audit report response – the due date for the project to respond to the audit report with their feedback and/or plan to address any Findings; this could be 30 days after the project receives the official audit report
  • Additional evidence due from the project – if during the audit the project agreed to provide additional evidence (screenshots, copies of records not accessible to the audit team, etc.), list those items, the audit team point-of-contact (typically, the Lead Auditor) and relevant due dates 

As with the In-brief, it is important to have a record of the Out-brief, so the audit team captures the presentation, attendees with their roles, and notes from any discussion.  Typically, this information is captured in the official audit record by the audit team Documentarian / note-taker.

4. Follow-Up and Close-out

To conclude the audit, there are follow-up and close-out steps that occur.  The official results are generated in an audit report, Findings are tracked to closure, project trends are captured for potential future audits, and the audit process is reviewed and updated as necessary. 

4.1 Generate and Deliver Audit Report

Just as they were at the Out-brief, the project is anxious to receive the final audit results, so the first order of business for the audit team is to generate and deliver the audit report, which contains the official audit results to which the project will respond. 

The audit report summarizes the audit and provides the official results.  Typical audit report content includes: 

• Confidentiality Statement – Depending on the nature and ownership of the material assessed during the audit, any project contracts in place, etc. the audit report may be restricted to specific audiences or require a statement of confidentiality regarding the results it contains.  The Lead auditor makes this determination and ensures the audit report contains the necessary statements and access restrictions.
  If the audit team assessed several companies within a project, there might be company-confidential processes or information involved. Also, in the case of multiple companies, the audit team may not want to put company-specific audit results out publicly where they can be used for comparisons.
• Purpose, Scope, Schedule, and Governing Documents – These sections of the report reflect the audit plan and serve as the official record of what was audited, for what purpose, and the audit timeframe.  The audit notification (see Audit Notification in this guidance) is a good source for this information, but be sure the audit report content reflects any adjustments made during the audit.
• Personnel – List the auditors, auditees, other attendees as captured in the Documentarian record of the audit.  Depending on the audit report audience, it may be necessary only to capture key personnel and not every person who participated in the audit.  The Documentarian notes from the audit will include the full list of participants, so the audit report may include only key participants.
• Governing documents - See Audit Notification in this topic for guidance on this content.
• Assumptions, qualifications – Audits are sampling activities, so it is important to identify any assumptions and qualifications made when generating Findings and Observations.  Assumptions could include the number of samples taken given the full number of records available, the time available to conduct an interview or witness an activity, the availability of key personnel during the audit, etc.  These caveats impact the audit results, and so are important to list in the report.
• Overall Summary – Provide a statement or paragraph regarding the overall compliance of the project with the governing documents, or specific sections thereof, used for this audit.
• Results – List in clear statements, the Findings, Observations, Opportunities for Improvement (OFI) in that order with major Findings first.  List the relevant governing document requirement with each Finding, preferably the full text and reference to its location in the governing document.  Stick to the facts – what was heard, seen, collected, or not able to be seen, heard, or collected (i.e., no objective evidence could be found).  A good set of working definitions for audit results are listed below.
  • Finding (major) – major nonconformance or non-compliance with requirement or process or collection of minor non-conformances that indicate systemic issue (see also ISO 1021-1:2015E and AS9101^TMF); a major or total breakdown of a process; not meeting a requirement
  • Finding (minor) – minor nonconformance or partial non-compliance (see also ISO 1021-1:2015E and AS9101^TMF); isolated or single part of a requirement not being met; roll up minor Findings into a single Finding; e.g., not in compliance with configuration management (CM) control vs. a long list of minor CM Findings
  • Observation – positive and negative observations that are not non-conformances or a potential non-compliance outside the scope of the current audit; positive observations are observations that contribute to quality; negative observations are observations that detract from the quality and if not addressed could be non-compliances in the future
  • OFI – a recommendation that would improve compliance to a higher level of quality
• Actions, Next Steps, and Due Dates – List any actions and next steps relevant to delivery of the audit report with appropriate due dates.  Consider the following:
• The date for a formal review of the audit report with the project
• Dates Corrective Actions are due to the project for the findings
• Dates for any follow-up meetings planned, perhaps to review Corrective Action status

When planning to deliver the audit report, confirm the distribution with audit management and the project team.  Consider recipients from the following groups, depending on any access, confidentiality, and viewing restrictions associated with the report:

• Project distribution
• Audit team distribution
• Management distribution

4.2 Track Findings to closure

After the audit report is delivered, the audit team continues to have closure responsibilities such as those listed below.                           

• Review corrective action (CA) plans or responses from the project in accordance with an agreed-upon timeframe with the project.  Two weeks to 30 days is a reasonable timeframe depending on the project’s schedule and when they can reasonably implement the solutions, once approved.
  • Review CA plans for expected content:
    • Problem/Issue/Finding statement
    • Root Cause investigation plan, including “where else does this need to be applied” and a due date
    • Short term correction plan and a due date
    • Long term CA plan  (plan to avoid recurrence) and a due date
  • Assess any provided rationale for why corrective action is not needed, e.g.:
    • The project provided more evidence
    • Project clarified an audit team misunderstanding
  • Assess timeliness of CA plans, coverage of the associated Findings,  and recurrence prevention plan
  • Work any concerns with the project; this responsibility lies with the Lead Auditor and project manager
• Once the project has implemented the CA plans, review the results to assess how well those plans addressed the relevant audit Findings.  This includes a review of any revised documents, an assessment of revised process implementation, and perhaps a follow-up audit.
• When Findings are satisfactorily closed, the Lead Auditor notifies the project and audit team management in writing and captures the written notifications in the official audit records, including the rationale for closure of each Finding.

4.3 Track Observations and OFIs (no formal response needed)

While audit Findings are required to be addressed by the project, NASA does not always require Observations and OFIs to receive a formal response from the project given that these items are not non-compliances, but suggestions for improvement that could prevent non-compliances in the future.   If a project provides responses to Observations and OFIs, the audit team is responsible for reviewing those responses, and following-up as appropriate.  Follow-up could be simple feedback that the response is acceptable, or it could be to enter notations in the audit record that the Observation or OFI is a candidate for a future follow-up audit.  The audit team determines how to respond to the project for Observations and OFIs.

4.4 Track items for future audits

As noted in several places in this guidance, an audit can generate items to be assessed in future audits.  Capturing these items in an official audit repository ensures they are available for planning future audits and not lost over time. Consider the following:              

• Project trends identified during an audit might need to be re-assessed in a future audit.
• Corrective actions for Findings might be acceptable in the short-term, but those processes become potential candidates for future audits to ensure those corrective action plans remain in place. 
• Observations and OFIs provide candidate material for future audits to ensure those items are not trending negatively.  This includes areas that narrowly passed a previous audit
• Trends observed across a series of audits are good future audit material.
• Regular interaction with a project can identify possible audit material, including processes that seem to be followed loosely, work products that seem to have repeated issues, and process escapes. 

4.5 Identify and make adjustments to the audit process if needed

As with any process, continuous assessment and improvement of the audit process are necessary to focus and efficiently use available resources.   During each audit, the audit team learns lessons about what works with each project and each type of audit conducted.  Those lessons are captured and when the overall audit schedule permits, the team reviews those lessoned learned and makes appropriate adjustments to the audit process using all the appropriate checks and balances, peer reviews and inspections, and management approvals necessary to maintain an effective and efficient process.

5. Resources

5.1 References

• (SWEREF-050)
  STEP Level 2 NASA Software Auditor Skills course,
  SMA-SA-WBT-203 SATERN. System for Administration, Training, and Educational Resources for NASA (SATERN), accessible to NASA-users at https://satern.nasa.gov/. User needs account to access SATERN courses. This NASA-specific information and resource is available in at the System for Administration, Training, and Educational Resources for NASA (SATERN), accessible to NASA-users at https://satern.nasa.gov/.
• (SWEREF-218)
  IEEE Computer Society, "IEEE Standard for Software Quality Assurance Plans", IEEE 730-2002, 2002.
  IEEE 730-2002, 2002. NASA users can access IEEE standards via the NASA Technical Standards System located at https://standards.nasa.gov/. Once logged in, search to get to authorized copies of IEEE standards.
• (SWEREF-219)
  IEEE Standard for Software Reviews and Audits
  IEEE Std 1028, 2008. IEEE Computer Society, NASA users can access IEEE standards via the NASA Technical Standards System located at https://standards.nasa.gov/. Once logged in, search to get to authorized copies of IEEE standards.
• (SWEREF-291)
  ISO/IEC 17021-1:2015 Conformity assessment
  ISO/IEC 17021-1:2015 contains principles and requirements for the competence, consistency and impartiality of bodies providing audit and certification of all types of management systems. ISO 17021-1:2015(E) – Conformity assessment – Requirements for bodies providing audit and certification of management systems (accessible through the NASA Standards website (https://standards.nasa.gov) than searching the HIS Engineering Workbench)
• (SWEREF-340)
  CMMI Model Viewer
  CMMI online Model Viewer gives you complete access and control over the full complement of CMMI V2.0 content. CMMI Process Quality Assurance (PQA) information available in CMMI model. CMMI Institute account needed to access this material.
• (SWEREF-371)
  AS9101 Rev. F - Quality Management Systems - Audit Requirements for Aviation, Space, and Defense Organizations
  Aerospace Standard 9101 Rev F, Quality Management Systems

5.2 Tools and Examples

NOTE: Download these examples to get a usable copy. The thumbnails below only give a partial preview of what the example really looks like.

5.2.1 Sample audit notification letter

Click the link to download: Offsite Software Audit Notification Letter_Sample Template.docx

5.2.2 Sample checklist

Click the link to download: SSO_ISO 27001 Audit Checklist.xlsx

The thumbnail below only gives a partial preview of what the example really looks like. Download the spreadsheet to get a usable copy.

5.2.3 Sample In-brief

Click the link to download: Audit In-Brief Sample Template.pptx

5.2.4 Sample out-brief

Click the link to download: Audit Outbrief Sample Template.pptx

5.2.5 Sample audit report

Click the link to download: Audit Report_Sample Template.docx

5.2.6 Sample tracking sheet, if tool not used

Click the link to download: Audit Results Tracking Sheet_Sample.xlsx

The thumbnail below only gives a partial preview of what the example really looks like. Download the spreadsheet to get a usable copy.