PMO reporting – the importance of
KPIs and their analysis
An overview with Jon M. Quigley
Successful project management is all about delivering results for the business. But how can organizations most effectively measure those results?
In this edition of PPM Hub, we explore why KPIs matter in PMO reporting with Jon M. Quigley, a project management expert with over 30 years of experience.
Jon shared his insights on choosing the right KPIs, the challenges of gathering and using metrics, and ensuring that KPIs align with project goals. He also shared with us real-world examples and advice on PMO reporting.
Join us to learn from Jon’s extensive knowledge and experience in the field.
How do you determine which KPIs are most relevant and impactful for your projects?
Jon M. Quigley
The selection of KPIs is a practical process that depends on the scope of the work and the objectives. It’s about identifying what needs to happen and what cannot be tolerated. If you are in a particular industry, there may be some recurring theme of KPI needs. In general, it is good to be not so dogmatic, so look for the answer to these questions:
The selection of KPIs is a practical process that depends on the scope of the work and the objectives. It’s about identifying what needs to happen and what cannot be tolerated. If you are in a particular industry, there may be some recurring theme of KPI needs. In general, it is good to be not so dogmatic, so look for the answer to these questions:
- What matters to reach the project objectives?
- What must you know for the project and team to succeed?
- How would I know that I am on a successful path or, more importantly, an unsuccessful path?
Arriving at the appropriate KPI will require input from the sponsor and key stakeholders, project team members and key domains of expertise (e.g., manufacturing engineers, design engineers, and product test engineers). It will be necessary to trace what matters at the end of the project to what is believed will make that true from the start of the project – the things that need to be corrected to achieve a successful end state.
What challenges have you encountered when selecting or defining KPIs for PMO reporting?
Jon M. Quigley
Gathering and using metrics is challenging, particularly within corporate culture. If the organization does not prioritize the value of metrics, there is a risk of encountering the following failures at the organizational level:
Gathering and using metrics is challenging, particularly within corporate culture. If the organization does not prioritize the value of metrics, there is a risk of encountering the following failures at the organizational level:
- The metrics are distilled into green or red smiley faces that convey nothing about the situation. This is too simple a presentation.
- Political spinning of the measurements to avoid negative repercussions.
- An overconfidence / optimistic (biased) interpretation of the metrics perpetuated throughout the organization.
- A culture that does not value learning but prioritizes speed above all. Team members have too little time to learn how to gather, present, and interpret metrics effectively, and they and executives hold metrics with little value.
- Poorly articulated project scope in tangible objectives beyond the cost and perhaps the expected delivery date (schedule). This includes the inability to trace specific performance expectations to sponsors.
- There are too many KPIs, many of which have no bearing on the project’s outcome or habit rather than thinking things through.
- Poor system for generating and tracking, including limited access.
- Too much lag from the time of metrics gathering to availability for assessment, what is happening in real-time.
- In ability to adapt to what is learned from the metrics, time to dump some metrics for this project and pick up new ones for example.
Measuring is not so easy. If you have done carpentry work around the home, you will likely know some of the challenges. There is a saying, measure twice, cut once, for example, though I find this a little tricky; more like measure 4 times then cut. Besides determining what to measure, we must consider how to obtain those measurements. The variation in variables are listed below: what is sampled, how it is sampled, and in the case of multiple people taking the sample with a tool, the reproducibility of the measurement.
It is a little worse than that; cognitive biases will impact what is measured, how it’s measured, and how that measurement is interpreted. If you are interested in learning more about this, let us know. We have written and presented on these topics from a learning perspective and through tools like Total Quality Management, specifically for project managers.
How do you ensure that KPIs are aligned with project objectives and overall organizational goals?
Jon M. Quigley
Ideally, the project’s objective is tangible (measurable-metrics – ways). This includes project attributes such as cost, schedule, objectives, and product performance. Consider product cost, quality attributes, current consumption, and other similar variables to which the project output will be compared.
Ideally, the project’s objective is tangible (measurable-metrics – ways). This includes project attributes such as cost, schedule, objectives, and product performance. Consider product cost, quality attributes, current consumption, and other similar variables to which the project output will be compared.
In your opinion, what are the key benefits of using KPIs in PMO reporting?
Jon M. Quigley
The project manager’s job is to deliver the project outcome successfully, navigating the minefield of risks associated with the work. Understanding these potential difficulties will require measuring variables.
The project manager’s job is to deliver the project outcome successfully, navigating the minefield of risks associated with the work. Understanding these potential difficulties will require measuring variables.
This is not to suggest that all things can be measured and predicted or understood at the start of the project. Ideally, learning occurs as the project progresses, which happens by reviewing the metrics collected throughout the project and identifying potential outcomes. Some events emerge unexpectedly and are not noticed until they occur.
From my point of view, a PMO without a historical performance record is not much of a PMO. This historical record will consist, at least in part, of key metrics within the PMO, such as project and portfolio cost performance.
From my experience, PMOs set project management processes and artifacts and provide training. They also identify the project organization’s performance metrics, devise methods of acquiring those metrics from project execution, and track them over time. The examples below are not exhaustive and equally important, not prescriptive. What needs to be measured, monitored, and controlled depends on the organization and its domains. The organization must select the things that matter and avoid adding undue logistical challenges and complexity.
Project Planned Budget | Actual Project Spending |
Number of Change Requests (per project) | Average cost per change requests |
Project Estimated Completion Date | Actual Completion Date |
Predicted income per unit time (ex. Quarterly) connected to project milestones. | Number of Projects Closed |
Gate Review metrics | Number of Projects on Hold |
Number of Projects Terminated (closed but not completed) | Number of Project Managers |
Avg. Duration of Hold for Projects on Hold | Number of Projects assigned to project managers (avg, min, max) |
Actual income generated per project (maybe profit) | Expected profit generated per project per quarter |
The cool thing is that much of this metric gathering can be at least semi-automated, and the reports can use most modern tools, too. The Executive staff, PMO leaders, and project managers work together to determine what metrics matter and connect to the organizations’ strategic goals. The executive staff and the PMO officers will determine the appropriate objectives and goals, and the PMO officers and project managers will work to find metrics and methods of gathering to support those objectives.
Continuous improvement of the PMO will require a baseline to decide what needs to be improved. The PMO and team members will use the project data to update processes, artifacts, and training needs. The amalgamation of the organization’s metrics tells us something about its capability, which helps with strategic planning and future organizational development. I have worked with companies that used SharePoint entries for key project attributes, such as risk, cost, and schedule, and used Power BI to record and display.
Can you share an example of how KPI analysis influenced decision-making within your environment?
Jon M. Quigley
I can share a brief insight into how KPI analysis improved my perspective on one of my emission projects.
I can share a brief insight into how KPI analysis improved my perspective on one of my emission projects.
I was responsible for testing vehicle systems for an emissions project. There were more than 3000 test cases, and we had never done this level of testing for an entire new vehicle’s change of components and systems.
We could have tracked the testing budget expenditures solely, but that tells us nothing about the activity, only about the amount of money spent. This is important, but it does not tell us everything we need to know.
Based on the number of team members we had for the work and a guess as to the number of test cases that can be conducted per day, we derived the rate of accomplishment chart you see here. This was not what the project wanted, but it was perhaps possible. Of course, the sponsors wanted this to be accomplished in two days, so this was not well received.
However, this estimate of what can be done was proposed. We would track the rate of test case exercise- in roughly 6 weeks, all test cases should have been completed. That was the thinking.
Below you can see an actual rate of accomplishment of the test cases. This testing consisted of multiple loops, the progress of one loop compared to the planned. The graphic consists of the number of test cases executed per day (the sum of the execution by the individual team members).
This only tells us the rate of accomplishment and allows us to predict the date of the testing’s conclusion to some degree.
Better still, we have the test case progress and the defects found. The green bars are the failures found in testing; the purple bars are the failures that are severe enough to be mission critical and limit the system’s use beyond well-controlled environments, for example, no customer user as validation.
What guidance would you give to other project managers regarding using metrics effectively and how they serve the PMO?
Jon M. Quigley
I would separate this into two parts: metrics in the project and metrics in service of the PMO. There will likely be some overlap, and some short-term project metrics will also support the long-term PMO metrics. The story and metrics of the project for question 5 provide an example of the metrics for a specific project. The metrics in this example are specific to this project. It was associated with vehicle architecture and federal regulation compliance, and thus, the metrics might be different than the typical project without these entanglements.
I would separate this into two parts: metrics in the project and metrics in service of the PMO. There will likely be some overlap, and some short-term project metrics will also support the long-term PMO metrics. The story and metrics of the project for question 5 provide an example of the metrics for a specific project. The metrics in this example are specific to this project. It was associated with vehicle architecture and federal regulation compliance, and thus, the metrics might be different than the typical project without these entanglements.
The project metrics above show that the PMO will be concerned with aggregate performance issues (the organization’s portfolio). But that should not be the limit. The PMO is part of the gate reviews and preparing for those reviews.
Project managers are well advised to spend time with key team members and sponsors to understand the details of the project scope and identify the specific metrics that need tracking and monitoring. These metrics should be connected to specific processes for data gathering and the individuals responsible for gathering and presenting that information. Monitoring should be continuous and not confined to gate reviews. As the project progresses, vigilance for emerging events necessitating updates to project metric tracking is essential.
Jon M. Quigley PMP CTFL, a principal and founding member of Value Transformation
Jon is PMP and CTFL certified and has experience on many product development topics, including process, quality and cost improvement techniques.
His specialties include intellectual property, writing, software development, product quality, and training, making him an asset in both industry and academia.
Jon is PMP and CTFL certified and has experience on many product development topics, including process, quality and cost improvement techniques.
His specialties include intellectual property, writing, software development, product quality, and training, making him an asset in both industry and academia.