Operations Research: Multi Criteria Analysis (MCA)

MultiCriteria Analysis (MCA)

Background

Multi-criteria analysis (MCA) is an approach to evaluation and decision-making problems that has a special feature: one explicitly takes into account different assessment aspects and interests that do not all have to be equally important. Therefore, these aspects and interests do not all have an equal weight in a final assessment. We assume here that there are two or more options, otherwise there is little to choose and decide upon. The decision not to shave tomorrow morning also has two options: do or don’t. Each has its own (un)attractive consequences. In fact, most daily problems, small or large, in the private domain, business or in the policy sphere of government have that characteristic of multiplicity. Sometimes, however, there is one aspect or interest that predominates and therefore determines the final judgment: lowest price, fastest mode of transport, or shortest distance.

The realisation that even day-to-day decisions consist of considerations between different distinctive assessment aspects can already be read in a letter that the originally British publisher/printer/scientist/diplomat, who had moved to the United States (and there was also co-author of the American Declaration of Independence), Benjamin Franklin wrote to the British theologian/philosopher/scientist and political theorist Joseph Priestly in 1772. Priestly had a problem: he was offered a new job that was attractive, but his current job was attractive as well. So, what to do? A problem with only two alternatives, but according to Franklin a kind of problem that “are difficult chiefly because while we have them under consideration all the reasons pro and con are not present to the mind at the same time; but sometimes one set present themselves, and at other times another, the first being out of sight. Hence the various purposes or inclinations that alternately prevail, and the uncertainty that perplexes us.” [See reference]
Franklin advised him how to arrive at a decision: write on a sheet of paper on one half under “Pro” all the arguments for changing jobs and on the other half under “Con” the arguments for not doing so. Take the time for identifying these arguments. If the lists seem complete, then cross out the arguments against each other, looking at how many arguments in one half (one or more) outweigh an argument in the other half. The half that still has remaining arguments, determines your decision (hopefully they are not both completely eliminated!). He called this “Moral or Prudential Algebra“. Today we call that (a simple-heuristic form of) multi-criteria analysis.

From the 60s of the last century many methods and techniques and even complete theories have been developed to address multi-criteria problems. Attention was also paid to the type of information that is or should become available: quantitative (in the form of numbers from measurements or as a result of separate sub-studies) or qualitative (in the form of opinions of people or less firm study results), or, usually, a mix of both. Methods were also developed to express that information on a measurement scale suitable for each type of criterion.
The human factor plays an important role in the considerations, in particular in determining how important each assessment aspect is. This influences the weighting of the information associated with each aspect when conducting the required synthesis. Many scientists have their own vision on how that works or should work. This is why most of those theories differ from each other in how subjective information is processed, how uncertainty is to be treated, the significance of differences between assessments, and basically to begin with: how to elicit meaningful statements about appreciation and importance from human beings (experts, stakeholders).
The degree of mathematics of the methods and techniques also differs; seemingly very qualitative methods often hide complex mathematics in order to arrive at scientifically sound synthesis results. Over the last decades entire “schools” have emerged that hardly are able to unify – figuratively, as there are plenty of conferences, symposia and critical scientific publications.

MCA at TNO

In the 70s and 80s, the Operations Research department of TNO at Waalsdorp took the first steps in applying MCA. This started with a study of, in particular, the French ELECTRE-methods and the American Analytic Hierarchy Process (AHP)-method (“Saaty method”): completely different approaches in many respects. Through contacts with various universities and congress visits, the use of software developed for these methods (but at that time still rather primitive compared to those of today), and even the creation and chairing of a Dutch Working Group for MCA, TNO’s knowledge was expanded.
Meanwhile, in those years, there were already a few MCA-based studies performed for the Dutch Defence Material Process, for example, the replacement of non-track vehicles for the Royal Netherlands Army, a tender for Netherlands Armed Forces Integrated Network (NAFIN), the requirements document for Short Range Air Defence systems (SHORAD).

The potential of MCA was, however, increasingly recognised. The need for flexible software tooling that offered not just one method, but a range of methods, also increased. There was also the need to be able to conduct a sort of overarching analysis that could bring together the results of partial but related studies: a so-called “Top model”.
In the 90s, TNO started the development of its own software package “TOPSystem” (The Option Preferred System, or simply “TOPSYS”). Although this was once in a while a difficult development process of many years with improvements and expansions requiring a lot of money, large studies have been supported for more than 20 years (!).

Even without TOPSYS, MCA expertise has been frequently used in TNO studies. As a by-product of the MCA expertise at TNO Waalsdorp, many refereed or non-refereed publications about MCA developments at TNO have been published. MCA lectures were also given to the Defence academy institutions, the University of Nijmegen, and as part of internal colloquia to raise awareness of MCA.

MCA at TNO after the millennium

Examples of studies since 2000 mainly support procurement projects as part of the Dutch Defence Material Process: for example Stinger missiles, Future Ground Based Air Defence System (FGBADS), Medium-altitude long-endurance unmanned aerial vehicle (MALE-UAV), torpedo systems, replacement of F-16 fighter aircraft, integrated combat helmets, etc. Also more strategic-policy issues were supported with MCA: for example concepts for land operations in 2020, relative strategic importance of military functions and capabilities, prioritising defence research topics, assessing cooperation potential with other countries in defence research, prioritising concepts of non-lethal weapons, organisational alternatives to the position of the Chief of Defence Staff, etc.

In a brainstorming session, for example, post-its are used to identify aspects that can serve as assessment criteria
In a brainstorming session, for example, post-its are used to identify aspects that can serve as assessment criteria.

MCA knowledge was also introduced into civilian projects: for example, the selection of telecommunication providers for the government, and the national security impact assessment using scenarios that describe a possible threat to the nation (the “National Risk Assessment”). This assessment method was identified by the OECD as an international best practice together with the national risk assessment method of the United Kingdom in 2009.

For a (fictitious) problem situation in which plans must be assessed to improve the long-term profit position of a company, the assessment aspects are first determined and then arranged in an appropriate hierarchical structure.
For a (fictitious) problem situation in which plans (in red) must be assessed to improve the long-term profit position of a company, the assessment aspects (in blue) are first determined and then arranged in an appropriate hierarchical structure.
Simplified hierarchy of criteria for comparing plans for the improvement of the long-term (LT) profit-making position of a firm. The lower criteria determine the higher ones. Groups of criteria can be distinguished in this way. The criteria are weighted relative to each other (= weights) and alternative improvement plans are assessed on the bottom layer criteria (= scores). The information is combined to a single final judgement using one or more MCA calculation methods.
Simplified hierarchy of criteria for comparing plans for the improvement of the long-term (LT) profit-making position of a firm. The lower criteria determine the higher ones. Groups of criteria can be distinguished in this way. The criteria are weighted relative to each other (= weights) and alternative improvement plans are assessed on the bottom layer criteria (= scores). The information is combined to a single final judgement using one or more MCA calculation methods.

 

The so-called score card: assessments of the alternatives on the bottom layer of criteria
The so-called score card: assessments of the alternatives on the bottom layer of criteria.

Experience has shown that many clients have suspicions about the value of MCA: you get what you want out of it, it is all subjective, etc. Despite a core of truth in this, our message has always been that you can manage subjectivity in an objective way. The software in particular helps to visualise the often-complex assessment hierarchies of criteria. Calculations show the consequences of divergent opinions about certain information (for example the weights of criteria that indicate how important something should count in a final judgment) and whether somewhat differing assessment values on a specific criterion actually makes a difference (for example, whether a certain alternative still remains the most preferred one).

It has always proved important to show transparently and graphically how information is processed to a final result, that scientific principles are used to achieve such a result. Moreover, the consequences of disagreements for the final outcome can be investigated.

The final result per alternative on the "top criterion" Long Term Profit on a standardised scale. Comparable images can be made for each intermediate criterion (group of criteria)
The final result per alternative on the “top criterion” Long Term Profit on a standardised scale. Comparable images can be made for each intermediate criterion (group of criteria).

 

Example of sensitivity analysis concerning three different improvement plans
Example of sensitivity analysis concerning three different improvement plans.

 

Example of sensitivity analysis concerning three different improvement plans
Example of sensitivity analysis concerning three different improvement plans. The graph shows a change in the final value between plan 1 and plan 2 for a somewhat greater weight (0.46, black) for Productivity than the original (0.3, white), so be careful when making your conclusions.

 

Example of a comparison of the final judgments of individual (coloured) experts in relation to the group average (in blue). It concerned an analysis of arguments for intensifying (right) or not intensifying a specific (defence) research area
Example of a comparison of the final judgments of individual (coloured) experts in relation to the group average (in blue). It concerned an analysis of arguments for intensifying (right) or not intensifying (left)  a specific (defence) research area.

 

Impression of the criteria hierarchy to define the system effectiveness of aircraft candidates for the replacement of the F16s. These criteria have been used for their assessment. The final judgment together with a sensitivity analysis that yielded uncertainty limits about the final judgment was performed with the help of MCA.
Impression of the criteria hierarchy to define the system effectiveness of aircraft candidates for the replacement of the F16s. These criteria have been used for their assessment. The final judgement together with a sensitivity analysis that yielded uncertainty limits about the final judgement was performed with the help of MCA.

 

Example of a scorecard from the National Risk Assessment where scenarios of impairing national security are assessed on ten impact criteria.
Example of a scorecard from the National Risk Assessment where scenarios of impairing national security are assessed on ten impact criteria.

 

Another example from the National Risk Assessment, where the assessment of 13 scenarios on ten impact criteria is presented in the form of a graphic scorecard (spider web).
Another example from the National Risk Assessment, where the assessment of 13 scenarios on ten impact criteria is presented in the form of a graphic scorecard (spider web).

 

Example of a risk diagram from the National Risk Assessment (from the 2011 report, Analyst Network National Security, RIVM). The vertical axis shows the final score of the scenarios determined with the help of an MCA of their impact on national security according to a number of criteria (see previous pictures). The horizontal axis represents an estimate of their probability of occurrence.
Example of a risk diagram from the National Risk Assessment (from the 2011 report, Analyst Network National Security, RIVM). The vertical axis shows the final score of the scenarios determined with the help of an MCA of their impact on national security according to a number of criteria – see previous pictures (from limited to very serious). The horizontal axis represents an estimate of their probability of occurrence (very unlikely to very likely).

 

 

References

From Benjamin Franklin to Joseph Priestley, 19 September 1772”, Founders Online, National Archives, version of January 18, 2019, https://founders.archives.gov/documents/Franklin/01-19-02-0200.
[Original source: The Papers of Benjamin Franklin, vol. 19, January 1 through December 31, 1772, ed. William B. Willcox. New Haven and London: Yale University Press, New Haven and London, 1975, pp. 299–300.]  

The project National Risk Assessment/National Security Profile [in Dutch].

TNO brochure Multi Criteria Analyse [in Dutch].

Acknowledgement

This text was provided by Diederik Wijnmalen, for many years the driving force behind the TNO MCA research activities.