An article about early detection of troubled projects in the March 2011 issue of PM Network caught my attention (“Raise The Red Flags”) – one of the many recommendations made was to employ multiple methods of determining whether a project was “on track” or not.
Delivery assurance can be done qualitatively or quantitatively – the key is that neither approach is perfectly suited to providing a holistic evaluation methodology and hence a hybrid approach should be considered.
Eight of the nine knowledge areas of PMI’s PMBOK Guide (Project Integration Management being excluded) are judged based on their quantitative vs. qualitative health fit.
Scope – for projects with very well defined scope, quantitative metrics such as number of work packages delivered to date vs number of work packages expected to be delivered to date can be a good measure of progress. But for conceptual or evolutionary projects, one can find scope increasing and decreasing over the lifetime of an iteration or phase. In such projects, subjectively judging “how much of what the customer wants have we delivered” might be a truer assessment of scope health.
Time & Cost – quantitative metrics are best for these knowledge areas and multiple options exist ranging from percent complete compared with expected percent complete to earned value management.
Quality – similar to Scope, for projects with well defined requirements, quantitative quality metrics can be used. For others, asking the customer “how comfortable do you feel that what we are developing and delivering to you is meeting your needs” may be just as accurate.
Human Resource – qualitative metrics rule best here. While you can attempt to evaluate team health using productivity metrics, soft skill methods are likely superior.
Communications – there is no objective way of assessing whether stakeholder expectations are being met, whether communication needs are being satisfied, and overall whether information being provided is achieiving expected goals. Projects with problems in this area will start to show issues in other knowledge areas, so qualitative evaluation based on inferred root causes might be the only option.
Risk – a combination of quantitative and qualitative metrics can be used – a risk score based on a bottom-up summary of a risk register balanced against a top down metric created by evaluating the impact of common risk factors is one approach. But one should normalize this data with a review of issue logs to see how many project issues were not proactively identified as risks. We might be identifying and responding to risks, but these might not be the risks being realized on our projects.
Procurement – quantitative metrics can be used to assess whether vendors are performing as expected/contracted. But qualitative methods are likely best for assessing vendor “relationship” status.
Zealots of objective metrics might argue against my position that both types of evaluation are needed, but most of us have worked on at least one project where the numbers indicated all was well, but the project was still judged a failure.
Easy in theory, difficult in practice 