Ortiz.james

Ortiz.james

• 1.
  Connect the Dots Dr. James Ortiz, Director Independent Program Assessment Office Seventh Annual NASA Program Management Challenge 2010 February 10, 2010 Used with Permission Page 1
• 2.
  Introduction • The IndependentProgram Assessment Office (IPAO) manages the independent review of the Agency's Programs and projects at life-cycle milestones to ensure the highest probability of mission success. •This presentation provides an overview of the Agency’s policy context for the Agency Independent Life Cycle Review process (The “Why”, the “What”, and the “How”). Page 2
• 3.
  Outline • Connecting thedots on the Agency policy context – The “Why” » NPD 1000.0; NPD 1000.5 – The “What” » NPR 7120.5D NID; NPR 7123.1 – The “How” » SRB Handbook » IPAO System Operating Procedures • Summary Page 3
• 4.
  Agency Policy Contextfor Independent Review Page 4
• 5.
  Policy Flow-down NPD 1000.0 NPD 1000.3 NPD 1000.5 NPD 7120.4 NPD 8700.1 NPD 8900.5A Mission Support Office NPR 7120.5D Support Organization NPR 7123.1A OSMA Directives OCHMO Directives and NID Directives 1 2 3 4 5 NASA -SP 2009 -10 -015 - HQ Center Engineering Mission Directorate NASA -SP -6105 SRB HANDBOOK Requirements Requirements SYSTEMS ENGINEERING HANDBOOK 7 6 IPAO 1000 IPAO MANAGEMENT PLAN LEGEND: DIRECT 1: ENGINEERING REQUIREMENTS 2: PROGRAM/PROJECT IPAO 2000 MANAGEMENT REQUIREMENTS STANDARD OPERATING INDIRECT 3. SAFETY AND MISSION PROCEDURE ASSURANCE REQUIREMENTS 4. HEALTH AND MEDICAL REQTS 5. MSO FUNCTIONAL REQTS 6. MD REQUIREMENTS 7. CENTER REQUIREMENTS Page 5
• 6.
  The “Why” Page 6
• 7.
  The Governance andStrategic Management Handbook (NPD 1000.0A) “NASA governance and strategic management provide the discipline and rigor to enable success of NASA’s Mission- …. Mission success delivers on our commitment to be good stewards of the resources entrusted to us by the taxpayer. To enable Agency-wide accomplishments, NASA’s governance framework is founded on the following tenets: • ….. NASA’s governance principles that provide this framework are: • Lean governance. • Clear roles, responsibility and decision making. • Strategic acquisition. • Checks and balances. …… There are many process-related checks and balances built into NASA’s way of doing business. They range from peer reviews conducted at the lowest level to oversight reviews conducted by the Agency’s Program Management Council. Three checks and balances of particular importance at the program or project level are: the independent life-cycle review process, the process for tailoring a specific prescribed requirement, and the Dissenting Opinion process.” Page 7
• 8.
  Why Independent Review The Governance and Strategic Management Handbook (NPD 1000.0A) The independent life-cycle review process provides a comprehensive review of programs and projects at each life-cycle milestone by competent, independent individuals. The purpose of these reviews is to provide: 1. Agency wants P/p to receive independent assurance that they are on- track 2. NASA senior management wants: – Independent validation at key decision points of the Program/project’s readiness to proceed into the next phase of its life-cycle – Externally-imposed impediments to Program/project success to be removed 3. Agency needs to give external stakeholders assurance we can deliver to our commitments 4. Significant additional benefit is that preparation for the review milestone allows for a holistic examination by the Program/project and the review team Page 8
• 9.
  Policy for NASAAcquisition NPD 1000.5 • NPD 1000.5A calling for the Joint Confidence Level (JCL) analysis became effective in January 2009 • Policy is directed to ensure appropriate level of funding is provided and maintained to increase probability of success of the Agency’s Programs and Projects • Requirements – (a) Programs are to be baselined or rebaselined and budgeted at a confidence level of 70% or the level approved by the decision authority of the responsible Agency- level management council. For a 70% confidence level, this is the point on the joint cost and schedule probability distribution where there is a 70% probability that the project will be completed at or lower than the estimated amount and at or before the projected schedule. The basis for a confidence level less than 70% is to be formally documented. – (b) Projects are to be baselined or rebaselined and budgeted at a confidence level consistent with the program’s confidence level. – The program's or project's proposed cost and schedule baseline are to be assessed by an independent review team. The program or project is to present and justify its resulting cost and schedule to the decision authority of the responsible Agency-level management council. The independent review team is to discuss with the decision authority its key concerns with the plans and baselines proposed by the program or 9 project. Page 9
• 10.
  The “What” Page 10
• 11.
  NPR 7120.5D NID Independent Review Requirements • Provides the basis for how NASA execute its space flight programs and projects. • Provided a standardized life cycle review process that is built around Key Decision Points (KDP). • Formalized the Technical Authority, Dissenting Opinion Resolution, and Waiver processes. • Defines the Independent review policy requirements  Standardizes the Standing Review Board concept across NASA  Combines technical (7123) and programmatic reviews into one Board the Standing Review Board (SRB)  One independent review board for each program or project  Conducts all 7120.5-required independent reviews for that program or project throughout its life cycle  Implemented by the IPAO for all Programs, Category 1 Projects and Category 2 projects with a life-cycle cost > 11 $250M Page 11
• 12.
  Project Life Cycle (NPR 7120.5) Page 12
• 13.
  Project Life Cycle (NPR 7120.5) Page 13
• 14.
  NPR 7123.1 SystemsEngineering Technical Review Requirements • NPR 7123.1A – Systems Engineering Processes and Requirements  Articulates and establishes the requirements on the implementing organization for performing, supporting, and evaluating systems engineering  This systems approach is applied to all elements of a system and all hierarchical levels of a system over the complete project life cycle • 5.2.1.3 A technical review is an evaluation of the project, or element thereof, by a knowledgeable group for the purposes of:  Assessing status of and progress toward accomplishing planned activities.  Validating technical tradeoffs explored and design solutions proposed.  Identifying technical weaknesses or marginal design and potential problems (risks) and recommending improvements and corrective actions.  Making judgments on activities’ readiness for follow-on events, incl additional future evaluation milestones to improve likelihood of successful outcome.  Making assessments and recommendations to project team, Center, and Agency management.  Providing historical record that can be referenced of decisions that were made during these formal reviews. • Entrance and Success criteria for each life-cycle milestone are contained in Appendix G. Page 14
• 15.
  The “How” Page 15
• 16.
  The SRB Handbook The How • The SRB is written to provide guidance to the NASA Program and project (P/p) community and the Standing Review Boards (SRBs) regarding expectations, processes, products, timelines, and working interfaces with Mission Directorates, P/p, Centers, other review organizations, and the Management Councils. • The SRB handbook was published by PA&E/IPAO with concurrence from OCE IR-16 Page 16
• 17.
  SRB Handbook Contents •The SRB Handbook consist of six sections – Section 1: Context for Independent Lifecycle Review (ILCR) Process. Objectives, intent, and governance of the Standing Review Boards (SRB). – Section 2: High level principles that govern the SRBs. Independence and issue resolution. Points to Annex with Agency policy for “Review Board Composition, Balance and Conflict of Interest”. – Section 3: Scope and expectations of ILCR. Provides guidelines (roadmaps) for application of review criteria in NPR 7120.5 as a function of the lifecycle stage of P/p, the types of mission (Robotics versus Human Spaceflight), and type of Program ( tightly coupled or loosely coupled). – Section 4: SRB Initiation process. Roles and responsibilities for Chair/RM nomination, team composition and staffing, and Terms of Reference (TOR). – Section 5: SRB Products. Technical and programmatic assessments, findings and recommendation, briefings and reports. – Section 6: Notional review timeline. Describes the review phases for a typical review. – Appendices: Agency Independence Policy, PIR guidance IR-17 Page 17
• 18.
  Example of Section3 (SRB engagement roadmap) NASA Life Cycle Formulation Implementation Phases Pre-Systems Acquisition Systems Acquisition Operations Pre-Phase A Phase A Phase B Phase C Phase D Phase E Reviews MCR 1 SRR SDR PDR CDR PRR 1 SIR SAR ORR FRR 2 LRR 2 PFAR 2 (PNAR) (NAR) Key Decision Points KDP A KDP B KDP C KDP D KDP E SRB Participation Case-by- Full Board Full Board Full Board Full Board Chair and Full Board Chair and Chair and Chair and Chair and Chair and case Member Member Member Member Member Member 3 3 3 3 3 3 Subset Subset Subset Subset Subset Subset Sample Assessments Preliminary Functional and Flowdown to Flowdown to Any Changes, Production Integration System Phase E, Flight Launch Flight Performance Functional Functional Flowdown to processes , Plan and meets Descope operations system and performance Baseline ; Elements; Elements, Functional certified Procedures acceptance plans are certified spacecraft/ reporting Requirements Requirements System Descope Elements, design criteria and to proceed payloads Traceability Requirements Plans Descope has been readiness Report; SEMP Document Plans verified and for launch validated Key Preliminary Mission Baseline: Design meets Design Previous Technical All waivers Hardware Launch Performance Technolgies, Approach, TRL, System Systems performance, documentati component, data and and system and report and TRL, MOEs, Technology Architecture, Engineeering TRL on, subsystem package, anomalies software spacecraft/ Anomaly MOPs Devlpt. TRL, TDMAP, Mgt Plan, Production and system Certification have been systems are payloads resolution 5 Maturity SEMP, SDP Design meets Plans tests have package closed; configured readiness Technical Assessment performance been verified Operational for flight for launch Plan (TDMAP), to support procedures Software integration and Devlpt. Plan contingency (SDP) planning ROM Cost ROM Preliminary: Baseline: Performance Production Performance Remaining Performance Performance Schedule Integrated BOEs IMS BOEs IMS against plans, Plans against liens or open against against Estimates Cost Schedule Cost/Budget Cost/Budget EVM, plans, items and plans, plans, Integrated cost and 6 Estimates UFE, CADRe UFE, CADRe UFE , LLIL EVM, plans for EVM, EVM, schedule WBS JCL , Detail UFE, closure UFE UFE ICE schedules, JCL ICE Preliminary: Preliminary: Preliminary: Baseline: Facilities Facilities Facilities Facilities Facilities Facilities Resources Facilities Facilities Facilities Facilities Workforce Workforce Workforce Workforce Workforce Workforce Workforce Workforce Workforce Workforce Preliminary Preliminary Preliminary: Baseline: Risk List Risk List Risk List Risk List Risk List Risk List Risks Risks Risk Mgt Plan Risk Mgt Plan Mitigation Mitigation Mitigation Mitigation Mitigation Mitigation Risk Risk list Risk list Mitigation Mitigation Preliminary: Preliminary: Preliminary: Baseline: Performance Production Performance Performance Phase E Performance Management Approach Plans Project Plans Project Plans against plans Engineering against plans against Mgt Plan against and Planning Plans plans Reporting Venues Project x x x x x x x x 4 Program x x x x x x x x 4 CMC x x x x x x x x DPMC x x x 4 x 4 x x x 4 x APMC Cat 1 Cat 1 Cat 1 Cat 1 Notes: 1. PRRs are only needed when multiple flight system copies are being developed; timing is discretionary. 2. LRRs, FRRs, PFARs are performed by the Mission Management Team (MMT); the SRB Chair (and/or Designee) attend as non-voting observers. 3. Chair determines which members should attend, including themselves. 4. The SRB chair and RM will make themselves available to support these interim briefings. 5. Compliance with NPR 7123.1 review entrance and success criteria will be assessed. 6. Compliance with NPR 1000.5. IR-18 Page 18
• 19.
  Example from Section4 (SRB Formulation ) SRB Initiation Schedule prepared monthly for the APMC BPR Cat 1 & Cat 3 & Programs Cat 2 Projects Cat 2 Projects >$250M <$250M MDAA CD CD Nominates Chair Nominates Chair Nominates Chair Reiterate selection steps with new Chair IPAO RM IPAO RM Center RM nomination Assignment Assignment assignment No MDAA/DA/TA/PA&E* Nominations vettedvetted all Approval Authorities (MDAA, DA, TA, PA&E) and cleared for OCI/PCI. Nominations with with all Approval Authorities (MDAA, DA, TA, PA&E) Approval/Concurrence ? Selection of the SRB Chair and RM Yes Reiterate as Chair/RM develop Chair/RM develop ToR necessary to SRB Membership List and review with DA and TA obtain Approvals No Nominations vetted with all Approval Authorities MDAA/DA/TA/PA&E* Approval/Concurrence ? and cleared for OCI/PCI. SRB Staffing and ToR Preparation Participating Organizations Yes Mission Directorate Host Center Proceed with SRB Reviews IPAO SRB * For Programs, Cat.1 and Cat 2 (LCC>$250M) Projects only Joint Effort IR-19 Page 19
• 20.
  Handbook Status • Initial version of Standing Review Board (SRB) Handbook (HB) developed October 2007 by Agency team (IPAO lead and representative from each Center). • SRB HB on hold pending review of Federal Advisory Committee Act (FACA) regulations and independence policy requirements. • SRB HB on hold for completion of NPR 7120.5D NID (~February 2009). • NID-aligned version of the SRB Handbook released for PPMB review on Sept 25, 2009 ( 2 week review). • Baseline version of SRB HB published ( December 2009) • Next revision to be published with NPR 7120.5E IR-20 Page 20
• 21.
  IPAO The NASA organization responsible for the agency independent life cycle review process • Enables independent review of programs and projects to ensure highest probability of mission success. • Ensures objectivity, quality, integrity and consistency of independent review process per NPRs 7120.5D and 7123.1A.  Works with SRB Chair and Convening Authorities  Reviews SRB products to ensure they meet Agency expectations • Provides Review Manager, cost, schedule and other programmatic analysis SRB members • Provides advice and recommendations to Agency on program and project policies  Book Manager for Agency’s SRB Handbook Page 21
• 22.
  IPAO Standard OperatingProcedure Detail processes • The Standard Operating Procedure (SOP) documents the Independent Program Assessment Office (IPAO) process for conducting Independent Life-Cycle Reviews (ILCRs). • The SOP is a guide for the IPAO personnel when conducting ILCRs of the Agency’s programs and projects. • The purpose of this document is to ensure ILCR quality and efficiency through the development of common definitions and processes. Page 22
• 23.
  How is itdone? (IPAO Process) I N P U T S  P/p Documentation  Agency Review  Agency/Directorate/Centers  Cost & Schedule  SRB Briefings to  Minutes/Decisions/Actions Schedule Documentation & Documentation Program/Project  Review Chair Requirements  Program/Project Nomination Briefings PLANNING PREPARING REVIEWING REPORTING CLOSING-OUT Continuous 2- Months 1-3 Months 2-3 Months 1- Month  Monitor Agency  Initiate Contact and Determine  Receive CADRe Inputs  Verbal Pre-Briefings to P/p  Review Closeout Baseline Program/ Review Budget  SRB Kick-Off Meetings  Write one-page summary  Customer Project Review  Identify and obtain approval of  Attend Reviews  Write the Report Feedback  Determine IPAO SRB Chair/RM/SRB members  Attend Site Visit  Prepare the Summary  Develop Lessons Review Budget  Determine funding mechanism  Develop ICE/ICA/ISA Briefings Learned  Review Assignments  Develop the TOR  Present the Briefings:  Process Review  Develop Schedule/Logistics - Program/ project and Improvement of the Review - CMC  Administrative  Develop the Cost Plan - MD PMC Close-out  Develop the Schedule Plan - APMC (if required)  Team Recognition P R O D U C T S  Individual Review Budget  ICE/ICA/ISA  One Page Summary  Review Summary  RM/CA/SA Assignments ToR, Cost Plan, Schedule Plan  SRB Findings  SRB Report  Contracts Close-out by Chair and Team Nomination  RFA  RRD Report COTR and Guidance Letters  Recommendations SRB Briefings  Review Records  Review Schedule  RRD Report  Contract Task Statement  Process improvements  R&R for SRB Page 23
• 24.
  SOP:Sample SRB Engagement Timeline SRB KO: PDR: 2/11/10 Site Visit: 5/11/10 - 5/21/10 KDP: 6/11/10 - 6/21/10 10/14/09 PDR KO:12/1/09 24 Page 24
• 25.
  Summary • The independent life cycle review process is an integral part of the Agency’s check and balances built into the NASA governance structure and complements the programmatic and technical lines of command and authority. • The independent lifecycle review process is encoded as part of NASA policy direction; its requirements are stipulated in policy requirements; and guidance to reviews team and implementing personnel is provided in handbooks and operating procedures. • Its processes are continuously assessed for improvement by IPAO and its stakeholders. • The independent lifecycle review process helps ensure the highest probability of success of the Agency’s program and projects. Page 25