Title: Software Project Management: Adding Stakeholder Metrics to Agile Projects

Gather from all the key stakeholders the top few (5 to 20) most critical goals that the project needs to deliver.

Projects need to learn to focus on all the stakeholders that arguably can affect the success or failure. The needs of all these stakeholders must be determined - by any useful methods - and converted into project requirements. By contrast, the typical agile model focuses on a user/customer 'in the next room'. Good enough if they were the only stakeholder, but disastrous for most real projects, where the critical stakeholders are more varied in type and number. Agile processes, due to this dangerously narrow requirements focus, risk outright failure, even if the one identified 'customer' gets all their needs fulfilled.

For each goal, define a scale of measurement and a 'final target' goal level. For example, Reliability: Scale: Mean Time Before Failure, Goal: >1 month.

Using Evo, a project is initially defined in terms of clearly stated, quantified, critical objectives. Agile methods do not have any such quantification concept. The problem is that vague targets with no quantification and lacking in deadlines do not count as true goals: they are not measurable, and not testable ideas.

Note that in Evo the requirements are not cast in concrete, even though they are extremely specific. During a project, the requirements can be changed and tuned based on practical experience, insights gained, external pressures, and feedback from each Evo step (See also point 4 under 'Project Policy').

Define approximately 4 budgets for your most limited resources (for example, time, people, money, and equipment).

Conventional methods do set financial and staffing budgets, but usually at too macro a level. They do not seem to directly, and in detail, manage the array of limited resources we have. Admittedly there are some such mechanisms in place in agile methods, such as the incremental weekly (or so) development cycle (which handles time). However, the Evo method sets an explicit numeric budget for any useful set of limited resources, effort, calendar time, financial spend, or memory space.

Write up these plans for the goals and budgets (Ensure this is kept to only one page).

All the key quantified performance targets and resource budgets should be presented simultaneously on a single overview page. Additional detail about them can, of course, be captured in additional notes, but not on this one 'focus' page.

Negotiate with the key stakeholders to formally agree these goals and budgets.

Once the requirements, derived from the project's understanding of the stakeholder needs, are clearly articulated, we need to go back to the real stakeholders and check that they agree with our 'clear' (but potentially incorrect or outdated) interpretation.

It is also certainly a wise precaution to check back later, during the project evolution, with the stakeholders, especially the specific stakeholders who will be impacted by the next Evo step:

Plan to deliver some benefit (that is, 'progress towards the goals') in weekly (or shorter) cycles (Evo steps).

A weekly delivery cycle is adopted by agile methods; this is good. However, the notion of measurement each cycle, on multiple performance and resource requirements, is absent.

Using Evo, the choice of the next Evo step is based on highest stakeholder value to cost ratios. It is not simply, "What shall we do next?" It is "What is most effective to do next - of highest value to the stakeholders with consideration of resources?"

The agile methods' notion of agreeing with a user about the function to be built during that weekly cycle is healthy, but the Evo method is focused on systematic, weekly, measured delivery towards long-range higher-level objectives, within numeric, multiple, resource constraints. This means that the Evo method is more clearly focused on the wider stakeholder-set values, and on total resource cost management.

The Evo method is not focused on simply writing code ('we are programmers, therefore we write code'). The Evo method is focused on delivering useful results to an organically whole system. We reuse, buy or exploit existing code just as happily as writing our own code. We build databases, train and motivate users, improve hardware, update telecommunications, create websites, improve the users' working environment, and/or improve motivation. So we become more like systems engineers ('any technology to deliver the results!'), than programmers ('what can we code for you today?').

Table 1 shows the use of an Impact Estimation table (Gilb 2004) to plan and track critical performance and cost characteristics of a system (Illustration courtesy of Kai Gilb). The pair of numbers in the three left hand columns (30, 5 etc.) show initial benchmarks (30, 99, 2500) and Goal levels (5, 200, 100,000). The '%' figures are the real scale impacts (like 20) converted to a % of the way from benchmark to the Goal levels (like 20% of the distance from benchmark to Goal).

Implement the project in Evo steps and report your progress after each Evo step.

Report to the project sponsors after each Evo step (weekly, or shorter) with your best available estimates or measures, for each performance goal and each resource budget.

All agile methods agree that the development needs to be done in short, frequent, delivery cycles. However, the Evo method specifically insists that the closed loop control of each cycle is done by:

The clear intention to react to the feedback from the metrics and to react to any changes in stakeholder requirements is a major feature of Evo. It helps ensure the project is kept 'on track' and it ensures relevance. It is only by the use of stakeholder metrics that Evo is allowed to have such control.

Figure 2 shows results being cumulated numerically step by step until the Goal level is reached. In a UK Radar system (Author experience), the system was delivered by gradually building database info about planes and ships, tuning recognition logic, and tuning the radar hardware.

Figure 2. Cumulation of results towards goal level

When all the goals are reached: 'Claim success and move on' (Gerstner 2002). Free remaining resources for more profitable ventures.

A major advantage of using numeric goal and budget levels, compared to 'a stream of yellow stickies from users' (a reference to agile method practice), is that it is quite clear when your goals are reached within your budgets. In fact, 'success' is formally well defined in advance by the set of the required numeric goal and budget levels.

Projects need to be evaluated on 'performance delivered' in relation to 'resources used'. This is a measure of project management 'efficiency'. When goals are reached, we need to avoid misusing resources to deliver more than is required. No additional effort should be expended to improve upon a goal, unless a new improved target level is set.

The project manager, and the project, will be judged exclusively on the relationship of progress towards achieving the goals versus the amounts of the budgets used.

The project team will do anything legal and ethical to deliver the goal levels within the budgets.

Projects need to be judged primarily on their ability to meet critical performance characteristics, in a timely and profitable way. This cannot be expected if the project team is paid 'by effort expended'.

The team will be paid and rewarded for benefits delivered in relation to cost.

Teams need to be paid according to their project efficiency, that is by the results they deliver with regard to the costs they incur. Even if this means that super efficient teams get terribly rich! And teams that fail go 'bankrupt.'

When only 13% of 1027 IT projects are 'successful' (Taylor 2001), we clearly need to find better mechanisms for rewarding success, and for not rewarding failure. I suggest that sharp numeric definition of success levels and consequent rewards for reaching them, is minimum appropriate behavior for any software project.

The team will find their own work process and their own design.

Agile methods believe we need to reduce unnecessarily cumbersome corporate mandated processes. I agree. They also believe in empowering the project team to find the processes, designs and methods that really work for them locally. I heartily agree!

However, I also strongly believe that numeric definition of goals and budgets, coupled with frequent estimation and measurement of progress, are much-needed additional mechanisms for enabling this empowerment. The price to pay for this, a few estimates and measures weekly, seems small compared to the benefits of superior control over project efficiency.

As experience dictates, the team will be free to suggest to the project 'sponsors' (one type of stakeholder) adjustments to 'more realistic levels' of the goals and budgets.

No project team should be 'stuck' with trying to satisfy unrealistic or conflicting stakeholder dreams within constrained resources.

Further, a project team can only be charged with delivering inside the 'state of the art' performance levels at inside the 'state of the art' costs. Exceeding 'state of the art' performance is likely to incur 'exponential' costs.