Title: How to Quantify Quality: Finding Scales of Measure

Try to re-use previously defined Scales and Meters.

Try to modify previously defined Scales and Meters.

If no existing Scale or Meter can be reused or modified, use common sense to develop innovative, homegrown quantification ideas.

Whatever Scale or Meter you start off with, you must be prepared to learn. Obtain and use early feedback, from colleagues and from field tests, to redefine and improve your Scales and Meters.

If we assume there is no 'off-the-shelf' definition that could be used, then you need to start describing the various aspects of the quality that are of interest.

There are always many distinct dimensions to qualities such as usability, maintainability, security, adaptability and their like [Gilb3]. (Suggestion: Try listing about 5 to 15 aspects of some selected quality that is critical to your project.)

For this example, let's select 'environmentally friendly' as the one of many aspects that we are interested in, and we shall work on this below.

Invent and specify a Tag: 'Environmentally Friendly' is sufficiently descriptive. Ideally, it could be shorter, but it is very descriptive left as it is. We indicate a formally defined concept by capitalizing the tag.

Note, we usually don't explicitly specify 'Tag: ' but this sometimes makes the tag identity clearer.

Check there is an Ambition statement, which briefly describes the level of requirement ambition. 'Ambition' is one of the defined Planguage parameters.

Ensure there is general agreement by all the involved parties with the Ambition definition. If not, ask for suggestions for modifications or additions to it. Here is a simple improvement to my initial Ambition statement. It actually introduces a 'constraint'.

Using the Ambition description, define an initial Scale that is somehow quantifiable (meaning - you can meaningfully attach a number to it). Consider what will be sensed by the stakeholders if the level of quality changes. What would be a visible effect if the quality improved? My initial, unfinished attempt, at finding a suitable Scale captured the ideas of change occurring, and of things getting better or worse:

However, I was not happy with it, so I made a second attempt. I refined the Scale by expanding it to include the ideas of specific things being effected in specific places over given times:

This felt better. In practice, I have added more [qualifiers] into the Scale, to indicate the variables that must be defined by specific things, places and time periods whenever the Scale is used.

Determine if the term needs to be defined with several different scales of measure, or whether one like this, with general parameters, will do. Has the Ambition been adequately captured? To determine what's best, you should list some of the possible sub-components of the term (that is, what can it be broken down into, in detail?). For example:

This example means: 'Thing' is defined as the set of things: Air, Water, Plant and Animal (which, since they are all four capitalized, are themselves defined elsewhere).

Or alternatively, instead of just the colon after the tag, '=' or the more explicit Planguage parameter, 'Consists Of' can be used to make this notation more immediately intelligible to novices in reading Planguage:

Then consider whether your defined Scale enables the performance levels for these sub-components to be expressed. You may have overlooked an opportunity, and may want to add one or more qualifiers to that Scale. For example, we could potentially add the scale qualifiers '…. under defined [Environmental Conditions] in defined [Countries]…' to make the scale definition even more explicit and more general.

Scale qualifiers (like …'defined [Place]'…) have the following advantages:

Start working on a Meter - a specification of how we intend to test or measure the performance of a real system with respect to the defined Scale. Remember, you should first check there is not a standard or company reference library Meter that you could use. Try to imagine a practical way to measure things along the Scale, or at least sketch one out. My example is only an initial rough sketch defined by a {set} of three rough measurement concepts. These at least suggest something about the quality and costs with such a measuring process.

Meter: {scientific data where available, opinion surveys, admitted intuitive guesses}.

The Meter must always explicitly address a particular Scale specification. It will help confirm your choice of Scale as it will provide evidence that practical measurements can feasibly be obtained on the given scale of measure.

Now try out the Scale specification by trying to use it to specify some useful levels on the Scale. Define some reference points from the past (Benchmarks) and some future requirements (Targets and Constraints). See Figure 3, at the bottom of the previous page, for an example.

If this seems unsatisfactory, then maybe I can find another, more specific, scale of measure? Maybe use a 'set' of different Scales to express the measured concept better? See examples below.

Here is an example of a single more-specific Scale:

Figure 4 shows an example of some other and more-specific set of Scales for the 'Environmentally Friendly' example. They are perhaps a complimentary set for expressing a complex Environmentally Friendly idea.

Many different scales of measure can be candidates to reflect changes in a single critical factor.

Environmentally Friendly is now defined as a 'Complex Attribute,' because it consists of a number of 'elementary' attributes: {Air, Water, Earth, Predators, Animals}. A different scale of measure now defines each of these elementary attributes. Using these Scales we can add corresponding Meters, benchmarks (like Past), constraints (like Fail), and target levels (like Goal), to describe exactly how Environmentally Friendly we want to be.

The Principle of 'Defining a Scale of Measure'

If you can't define a scale of measure, then the goal is out of control.

Specifying any critical variable starts with defining its units of measure.

The Principle of 'Quantification being Mandatory for Control'

If you can't quantify it, you can't control it.^[1]

If you cannot put numbers on your critical system variables, then you cannot expect to communicate about them, or to control them.

The Principle of 'Scales should control the Stakeholder Requirements'

Don't choose the easy Scale, choose the powerful Scale.

Select scales of measure that give you the most direct control over the critical stakeholder requirements. Chose the Scales that lead to useful results.

The Principle of 'Copycats Cumulate Wisdom'

Don't reinvent Scales anew each time - store the wisdom of other Scales for reuse.

Most scales of measure you will need, will be found somewhere in the literature, or can be adapted from existing literature.

The Cartesian Principle

Divide and conquer said René - put complexity at bay.

Most high-level performance attributes need decomposition into the list of sub-attributes that we are actually referring to. This makes it much easier to define complex concepts, like 'Usability', or 'Adaptability,' measurably.

The Principle of 'Quantification is not Measurement'

You don't have to measure in order to quantify!

There is an essential distinction between quantification and measurement. "I want to take a trip to the moon in nine picoseconds" is a clear requirement specification without measurement." The well-known problems of measuring systems accurately are no excuse for avoiding quantification. Quantification allows us to communicate about how good scalar attributes are or can be - before we have any need to measure them in the new systems.

The Principle of 'Meters Matter'

Measurement methods give real world feedback about our ideas.

A 'Meter' definition determines the quality and cost of measurement on a scale; it needs to be sufficient for control and for our purse.

The Principle of 'Horses for Courses'^[2]

Different measuring processes will be necessary for different points in time, different events, and different places.^[3]

The Principle of 'The Answer always being '42' '^[4]

Exact numbers are ambiguous unless the units of measure are well-defined and agreed.

Formally defined scales of measure avoid ambiguity. If you don't define scales of measure well, the requirement level might just as well be an arbitrary number.

The Principle of 'Being Sure About Results'

If you want to be sure of delivering the critical result - then quantify the requirement.

Critical requirements can hurt you if they go wrong - and you can always find a useful way to quantify the notion of 'going right' to help you avoid doing so.