PMO reporting – the importance of KPIs and their analysis

Jon M. Quigley
Gathering and using metrics is challenging, particularly within corporate culture. If the organization does not prioritize the value of metrics, there is a risk of encountering the following failures at the organizational level:

1. The metrics are distilled into green or red smiley faces that convey nothing about the situation. This is too simple a presentation.
2. Political spinning of the measurements to avoid negative repercussions.
3. An overconfidence / optimistic (biased) interpretation of the metrics perpetuated throughout the organization.
4. A culture that does not value learning but prioritizes speed above all. Team members have too little time to learn how to gather, present, and interpret metrics effectively, and they and executives hold metrics with little value.
5. Poorly articulated project scope in tangible objectives beyond the cost and perhaps the expected delivery date (schedule). This includes the inability to trace specific performance expectations to sponsors.
6. There are too many KPIs, many of which have no bearing on the project’s outcome or habit rather than thinking things through.
7. Poor system for generating and tracking, including limited access.
8. Too much lag from the time of metrics gathering to availability for assessment, what is happening in real-time.
9. In ability to adapt to what is learned from the metrics, time to dump some metrics for this project and pick up new ones for example.

Measuring is not so easy. If you have done carpentry work around the home, you will likely know some of the challenges. There is a saying, measure twice, cut once, for example, though I find this a little tricky; more like measure 4 times then cut. Besides determining what to measure, we must consider how to obtain those measurements. The variation in variables are listed below: what is sampled, how it is sampled, and in the case of multiple people taking the sample with a tool, the reproducibility of the measurement.

It is a little worse than that; cognitive biases will impact what is measured, how it’s measured, and how that measurement is interpreted. If you are interested in learning more about this, let us know. We have written and presented on these topics from a learning perspective and through tools like Total Quality Management, specifically for project managers.

Jon M. Quigley
The project manager’s job is to deliver the project outcome successfully, navigating the minefield of risks associated with the work. Understanding these potential difficulties will require measuring variables.

This is not to suggest that all things can be measured and predicted or understood at the start of the project. Ideally, learning occurs as the project progresses, which happens by reviewing the metrics collected throughout the project and identifying potential outcomes. Some events emerge unexpectedly and are not noticed until they occur.

From my point of view, a PMO without a historical performance record is not much of a PMO. This historical record will consist, at least in part, of key metrics within the PMO, such as project and portfolio cost performance.

From my experience, PMOs set project management processes and artifacts and provide training. They also identify the project organization’s performance metrics, devise methods of acquiring those metrics from project execution, and track them over time. The examples below are not exhaustive and equally important, not prescriptive. What needs to be measured, monitored, and controlled depends on the organization and its domains. The organization must select the things that matter and avoid adding undue logistical challenges and complexity.

The cool thing is that much of this metric gathering can be at least semi-automated, and the reports can use most modern tools, too. The Executive staff, PMO leaders, and project managers work together to determine what metrics matter and connect to the organizations’ strategic goals. The executive staff and the PMO officers will determine the appropriate objectives and goals, and the PMO officers and project managers will work to find metrics and methods of gathering to support those objectives.

Continuous improvement of the PMO will require a baseline to decide what needs to be improved. The PMO and team members will use the project data to update processes, artifacts, and training needs. The amalgamation of the organization’s metrics tells us something about its capability, which helps with strategic planning and future organizational development. I have worked with companies that used SharePoint entries for key project attributes, such as risk, cost, and schedule, and used Power BI to record and display.

Jon M. Quigley
I can share a brief insight into how KPI analysis improved my perspective on one of my emission projects.

I was responsible for testing vehicle systems for an emissions project. There were more than 3000 test cases, and we had never done this level of testing for an entire new vehicle’s change of components and systems.

We could have tracked the testing budget expenditures solely, but that tells us nothing about the activity, only about the amount of money spent. This is important, but it does not tell us everything we need to know.

Based on the number of team members we had for the work and a guess as to the number of test cases that can be conducted per day, we derived the rate of accomplishment chart you see here. This was not what the project wanted, but it was perhaps possible. Of course, the sponsors wanted this to be accomplished in two days, so this was not well received.

However, this estimate of what can be done was proposed. We would track the rate of test case exercise- in roughly 6 weeks, all test cases should have been completed. That was the thinking.

Below you can see an actual rate of accomplishment of the test cases. This testing consisted of multiple loops, the progress of one loop compared to the planned. The graphic consists of the number of test cases executed per day (the sum of the execution by the individual team members).

This only tells us the rate of accomplishment and allows us to predict the date of the testing’s conclusion to some degree.

Better still, we have the test case progress and the defects found. The green bars are the failures found in testing; the purple bars are the failures that are severe enough to be mission critical and limit the system’s use beyond well-controlled environments, for example, no customer user as validation.

Jon M. Quigley
I would separate this into two parts: metrics in the project and metrics in service of the PMO. There will likely be some overlap, and some short-term project metrics will also support the long-term PMO metrics. The story and metrics of the project for question 5 provide an example of the metrics for a specific project. The metrics in this example are specific to this project. It was associated with vehicle architecture and federal regulation compliance, and thus, the metrics might be different than the typical project without these entanglements.

The project metrics above show that the PMO will be concerned with aggregate performance issues (the organization’s portfolio). But that should not be the limit. The PMO is part of the gate reviews and preparing for those reviews.

Project managers are well advised to spend time with key team members and sponsors to understand the details of the project scope and identify the specific metrics that need tracking and monitoring. These metrics should be connected to specific processes for data gathering and the individuals responsible for gathering and presenting that information. Monitoring should be continuous and not confined to gate reviews. As the project progresses, vigilance for emerging events necessitating updates to project metric tracking is essential.