Operations Research: Multi Criteria Analysis (MCA)

Multi-Criteria Analysis (MCA)

Background

Multi-criteria analysis (MCA) is an approach to evaluation and decision-making problems that has a special feature: one explicitly takes into account different assessment aspects and interests that do not all have to be equally important. Therefore, these aspects and interests do not all have equal weight in a final assessment. We assume here that there are two or more options, otherwise, there is little to choose and decide upon. The decision not to shave tomorrow morning also has two options: do or don’t. Each has its own (un)attractive consequences. Most daily problems, small or large, in the private domain, business or in the policy sphere of government have that characteristic of multiplicity. Sometimes, however, there is one aspect or interest that predominates and therefore determines the final judgment: lowest price, fastest mode of transport, or shortest distance.

The realisation that even day-to-day decisions consist of considerations between different distinctive assessment aspects can already be read in a letter that the original British publisher/printer/scientist/diplomat, who had moved to the United States (and there was also co-author of the American Declaration of Independence), Benjamin Franklin wrote to the British theologian/philosopher/scientist and political theorist Joseph Priestly in 1772. Priestly had a problem: he was offered a new attractive job, but his current job was attractive as well. So, what to do? A problem with only two alternatives, but according to Franklin a kind of problem that “Is difficult chiefly because while we have them under consideration all the reasons pro and con are not present to the mind at the same time; but sometimes one set present themselves, and at other times another, the first being out of sight. Hence the various purposes or inclinations that alternately prevail, and the uncertainty that perplexes us.” [See reference]
Franklin advised him how to arrive at a decision: write on a sheet of paper on one half under “Pro” all the arguments for changing jobs and on the other half under “Con” the arguments for not doing so. Take the time to identify these arguments. If the lists seem complete, then cross out the arguments against each other, looking at how many arguments in one half (one or more) outweigh an argument in the other half. The half that still has remaining arguments, determines your decision (hopefully they are not both eliminated!). He called this “Moral or Prudential Algebra“. Today we call that (a simple-heuristic form of) multi-criteria analysis.

From the 60s of the last century, many methods and techniques and even complete theories have been developed to address multi-criteria problems. Attention was also paid to the type of information that is or should become available: quantitative (in the form of numbers from measurements or as a result of separate sub-studies) or qualitative (in the form of opinions of people or less firm study results), or, usually, a mix of both. Methods were also developed to express that information on a measurement scale suitable for each type of criterion.
The human factor plays an important role in the considerations, in particular in determining how important each assessment aspect is. This influences the weighting of the information associated with each aspect when conducting the required synthesis. Many scientists have their vision of how that works or should work. This is why most of those theories differ from each other in how subjective information is processed, how uncertainty is to be treated, the significance of differences between assessments, and basically to begin with how to elicit meaningful statements about appreciation and importance from human beings (experts, stakeholders).
The degree of mathematics of the methods and techniques also differs; seemingly very qualitative methods often hide complex mathematics to arrive at scientifically sound synthesis results. Over the last decades, entire “schools” have emerged that hardly can unify – figuratively, as there are plenty of conferences, symposia and critical scientific publications.

MCA at TNO

In the 70s and 80s, the Operations Research department of TNO at Waalsdorp took the first steps in applying MCA. This started with a study of, in particular, the French ELECTRE methods and the American Analytic Hierarchy Process (AHP)-method (“Saaty method”): completely different approaches in many respects. Through contacts with various universities and congress visits, the use of software developed for these methods (but at that time still rather primitive compared to those of today), and even the creation and chairing of a Dutch Working Group for MCA, TNO’s knowledge was expanded.
Meanwhile, in those years, there were already a few MCA-based studies performed for the Dutch Defence Material Process, for example, the replacement of non-track vehicles for the Royal Netherlands Army, a tender for the Netherlands Armed Forces Integrated Network (NAFIN), the requirements document for Short Range Air Defence systems (SHORAD).

The potential of MCA was, however, increasingly recognised. The need for flexible software tooling that offered not just one method, but a range of methods, also increased. There was also the need to be able to conduct a sort of overarching analysis that could bring together the results of partial but related studies: a so-called “Top model”.
In the 90s, TNO started the development of its software package “TOPSystem” (The Option Preferred System, or simply “TOPSYS”). Although this was once in a while a difficult development process of many years with improvements and expansions requiring a lot of money, large studies have been supported for more than 20 years (!).

Even without TOPSYS, MCA expertise has been frequently used in TNO studies. As a by-product of the MCA expertise at TNO Waalsdorp, many refereed or non-refereed publications about MCA developments at TNO have been published. MCA lectures were also given to the Defence Academy institutions, the University of Nijmegen, and as part of internal colloquia to raise awareness of MCA.

MCA at TNO after the millennium

Examples of studies since 2000 mainly support procurement projects as part of the Dutch Defence Material Process: for example Stinger missiles, Future Ground-Based Air Defence System (FGBADS), Medium-altitude long-endurance unmanned aerial vehicle (MALE-UAV), torpedo systems, replacement of F-16 fighter aircraft, integrated combat helmets, etc. Also, more strategic-policy issues were supported with MCA: for example concepts for land operations in 2020, the relative strategic importance of military functions and capabilities, prioritising defence research topics, assessing cooperation potential with other countries in defence research, prioritising concepts of non-lethal weapons, organisational alternatives to the position of the Chief of Defence Staff, etc.

In a brainstorming session, for example, post-its are used to identify aspects that can serve as assessment criteria.

MCA knowledge was also introduced into civilian projects: for example, the selection of telecommunication providers for the government, and the national security impact assessment using scenarios that describe a possible threat to the nation (the “National Risk Assessment”). This assessment method was identified by the OECD as an international best practice together with the national risk assessment method of the United Kingdom in 2009.

For a (fictitious) problem situation in which plans must be assessed to improve the long-term profit position of a company, the assessment aspects are first determined and then arranged in an appropriate hierarchical structure. — For a (fictitious) problem situation in which plans (in red) must be assessed to improve the long-term profit position of a company, the assessment aspects (in blue) are first determined and then arranged in an appropriate hierarchical structure.

Simplified hierarchy of criteria for comparing plans for the improvement of the long-term (LT) profit-making position of a firm. The lower criteria determine the higher ones. Groups of criteria can be distinguished in this way. The criteria are weighted relative to each other (= weights) and alternative improvement plans are assessed on the bottom layer criteria (= scores). The information is combined to a single final judgement using one or more MCA calculation methods. — Simplified hierarchy of criteria for comparing plans for the improvement of the long-term (LT) profit-making position of a firm. The lower criteria determine the higher ones. Groups of criteria can be distinguished in this way. The criteria are weighted relative to each other (= weights) and alternative improvement plans are assessed on the bottom layer criteria (= scores). The information is combined into a single final judgement using one or more MCA calculation methods.

The so-called score card: assessments of the alternatives on the bottom layer of criteria — The so-called scorecard: assessments of the alternatives on the bottom layer of criteria

Experience has shown that many clients have suspicions about the value of MCA: you get what you want out of it, it is all subjective, etc. Despite a core of truth in this, our message has always been that you can objectively manage subjectivity. The software, in particular, helps to visualise the often-complex assessment hierarchies of criteria. Calculations show the consequences of divergent opinions about certain information (for example the weights of criteria that indicate how important something should count in a final judgment) and whether somewhat differing assessment values on a specific criterion make a difference (for example, whether a certain alternative remains the most preferred one).

It has always proved important to show transparently and graphically how information is processed to a final result, that scientific principles are used to achieve such a result. Moreover, the consequences of disagreements for the outcome can be investigated.

The final result per alternative on the "top criterion" Long Term Profit on a standardised scale. Comparable images can be made for each intermediate criterion (group of criteria) — The final result per alternative on the “top criterion” Long long-term profit on a standardised scale. Comparable images can be made for each intermediate criterion (group of criteria).

Example of sensitivity analysis concerning three different improvement plans

Example of a comparison of the final judgments of individual (coloured) experts in relation to the group average (in blue). It concerned an analysis of arguments for intensifying (right) or not intensifying a specific (defence) research area — Example of a comparison of the final judgments of individual (coloured) experts concerning the group average (in blue). It concerned an analysis of arguments for intensifying (right) or not intensifying (left) a specific (defence) research area.

Impression of the criteria hierarchy to define the system effectiveness of aircraft candidates for the replacement of the F16s. These criteria have been used for their assessment. The final judgment together with a sensitivity analysis that yielded uncertainty limits about the final judgment was performed with the help of MCA. — Impression of the criteria hierarchy to define the system effectiveness of aircraft candidates for the replacement of the F16s. These criteria have been used for their assessment. The final judgement together with a sensitivity analysis that yielded uncertainty limits about the final judgement was performed with the help of MCA.

Example of a scorecard from the National Risk Assessment where scenarios of impairing national security are assessed on ten impact criteria. — Example of a scorecard from the National Risk Assessment where scenarios of impairing national security are assessed on ten impact criteria

Another example from the National Risk Assessment, where the assessment of 13 scenarios on ten impact criteria is presented in the form of a graphic scorecard (spider web). — Another example is from the National Risk Assessment, where the assessment of 13 scenarios on ten impact criteria is presented in the form of a graphic scorecard (spider web).

Example of a risk diagram from the National Risk Assessment (from the 2011 report, Analyst Network National Security, RIVM). The vertical axis shows the final score of the scenarios determined with the help of an MCA of their impact on national security according to a number of criteria (see previous pictures). The horizontal axis represents an estimate of their probability of occurrence. — Example of a risk diagram from the National Risk Assessment (from the 2011 report, Analyst Network National Security, RIVM). The vertical axis shows the final score of the scenarios determined with the help of an MCA of their impact on national security according to several criteria – see previous pictures (from limited to very serious). The horizontal axis represents an estimate of their probability of occurrence (very unlikely to very likely).

References

“From Benjamin Franklin to Joseph Priestley, 19 September 1772”, Founders Online, National Archives, version of January 18, 2019, https://founders.archives.gov/documents/Franklin/01-19-02-0200.
[Source: The Papers of Benjamin Franklin, vol. 19, January 1 through December 31, 1772, ed. William B. Willcox. New Haven and London: Yale University Press, New Haven and London, 1975, pp. 299–300.]
The project National Risk Assessment/National Security Profile [in Dutch].
TNO brochure Multi-Criteria Analyse [in Dutch].

Acknowledgement

This text was provided by Diederik Wijnmalen, for many years the driving force behind the TNO MCA research activities.