Leyton Marsh Is Their No.1 Choice: But Do The Numbers Add Up?

Scoring the Sites 

It is important not to be taken in by the “scoring matrix” that the LVRPA has used to assess the relative merits of the four alternative sites for the new Ice Centre.  The fact that it is based on numbers lends it a spurious air of objectivity.  As it happens its whole methodology is seriously flawed.

Firstly, the numbers are all expressed as percentages.  But percentages of what — perfection?  For example, if a site scores 50% on car-parking, does that mean that it has only half the required number of car-parking spaces?  If that is so, then clearly all four sites must be rejected out-of-hand, since none of them scores more than 20% on any of the five criteria.  So what do the percentages represent?  Should we interpret them as purely relative numbers?  So (to use the example of car-parking again), if one site scores 10% for car-parking and another scores 20%, does that mean that the second site has twice as many parking spaces as the first?  But what if the first site has enough spaces?  It cannot then be seriously argued that the second site is twice as good (as regards car-parking), merely because it has a lot of superfluous spaces.

Even if the objection above can be overcome – if the scores shown for the four sites for one criterion truly represent the sites’ relative merits as regards that one criterion – we are still left with a problem when we come to combine the scores for different criteria.  Consider the following situation, with two sites and two criteria.

This shows that the LV Ice Centre is better than Eton Manor for physical characteristics, but worse for commercial/financial.  But how are we to obtain a combined score?  Remember that the numbers in the table only have a meaning when read horizontally – there is no meaningful comparison when read vertically.  For all we know, an acceptable score for Physical characteristics might be 100% while an acceptable score for commercial/financial might be 14%.  In that case the relative advantage of the LV Ice Centre in physical characteristics will be outweighed by its relative deficiency in commercial/financial.  Before we can begin to calculate a combined score, we need to know the theoretical ranges of the individual scores.

It may be case that the theoretical ranges of the individual scores are all the same.  But that itself is highly questionable, since it would imply that all the criteria are of equal importance.  Are they all of equal importance?  There seems no reason to assume so.  (And it could also be argued that there are some criteria missing altogether.)  If they are of unequal importance, then they need to be appropriately weighted before they can be added together.

But is adding the scores together the correct way to calculate an overall score in any case?  Suppose that all of the above problems have been addressed: we have two criteria (X and Y) that have the same ranges (let us say 0% to 20%), and they are of equal importance (so they do not need to be weighted).  Consider two sites (A and B); on criterion X Site B is 10 times better than Site A, on criterion Y Site A is 2 times better than Site B.  Which site is better overall?  Common sense would suggest Site B.  But that is not necessarily the answer that the LVRPA would give you.  Consider the following scores.

Note that on criterion X site B is 10 times better than site A, and on criterion Y site A is 2 times better than site B, as previously stated.  If you are the LVRPA, you would just add the scores for the two criteria together, getting 21% for Site A and 20% for Site B.  So Site A wins.  But is that the right answer?  It could well be argued that Site B should win, because its relative merit on criterion X outweighs its relative deficiency on criterion Y.  Indeed, it could be argued that overall scores should be calculated by multiplying the individual scores, not by adding them.

It does not appear that the LVRPA has taken any of these considerations into account in producing its “scoring matrix”.  That being so, its value is very dubious.