MultiCriteria Analysis (MCA)
Multi-criteria analysis (MCA) is an approach to evaluation and decision-making problems that has a special feature: one explicitly takes into account different assessment aspects and interests that do not all have to be equally important. Therefore, these aspects and interests do not all have an equal weight in a final assessment. We assume here that there are two or more options, otherwise there is little to choose and decide upon. The decision not to shave tomorrow morning also has two options: do or don’t. Each has its own (un)attractive consequences. In fact, most daily problems, small or large, in the private domain, business or in the policy sphere of government have that characteristic of multiplicity. Sometimes, however, there is one aspect or interest that predominates and therefore determines the final judgment: lowest price, fastest mode of transport, or shortest distance.
The realisation that even day-to-day decisions consist of considerations between different distinctive assessment aspects can already be read in a letter that the originally British publisher/printer/scientist/diplomat, who had moved to the United States (and there was also co-author of the American Declaration of Independence), Benjamin Franklin wrote to the British theologian/philosopher/scientist and political theorist Joseph Priestly in 1772. Priestly had a problem: he was offered a new job that was attractive, but his current job was attractive as well. So, what to do? A problem with only two alternatives, but according to Franklin a kind of problem that “are difficult chiefly because while we have them under consideration all the reasons pro and con are not present to the mind at the same time; but sometimes one set present themselves, and at other times another, the first being out of sight. Hence the various purposes or inclinations that alternately prevail, and the uncertainty that perplexes us.” [See reference]
Franklin advised him how to arrive at a decision: write on a sheet of paper on one half under “Pro” all the arguments for changing jobs and on the other half under “Con” the arguments for not doing so. Take the time for identifying these arguments. If the lists seem complete, then cross out the arguments against each other, looking at how many arguments in one half (one or more) outweigh an argument in the other half. The half that still has remaining arguments, determines your decision (hopefully they are not both completely eliminated!). He called this “Moral or Prudential Algebra“. Today we call that (a simple-heuristic form of) multi-criteria analysis.
From the 60s of the last century many methods and techniques and even complete theories have been developed to address multi-criteria problems. Attention was also paid to the type of information that is or should become available: quantitative (in the form of numbers from measurements or as a result of separate sub-studies) or qualitative (in the form of opinions of people or less firm study results), or, usually, a mix of both. Methods were also developed to express that information on a measurement scale suitable for each type of criterion.
The human factor plays an important role in the considerations, in particular in determining how important each assessment aspect is. This influences the weighting of the information associated with each aspect when conducting the required synthesis. Many scientists have their own vision on how that works or should work. This is why most of those theories differ from each other in how subjective information is processed, how uncertainty is to be treated, the significance of differences between assessments, and basically to begin with: how to elicit meaningful statements about appreciation and importance from human beings (experts, stakeholders).
The degree of mathematics of the methods and techniques also differs; seemingly very qualitative methods often hide complex mathematics in order to arrive at scientifically sound synthesis results. Over the last decades entire “schools” have emerged that hardly are able to unify – figuratively, as there are plenty of conferences, symposia and critical scientific publications.
MCA at TNO
In the 70s and 80s, the Operations Research department of TNO at Waalsdorp took the first steps in applying MCA. This started with a study of, in particular, the French ELECTRE-methods and the American Analytic Hierarchy Process (AHP)-method (“Saaty method”): completely different approaches in many respects. Through contacts with various universities and congress visits, the use of software developed for these methods (but at that time still rather primitive compared to those of today), and even the creation and chairing of a Dutch Working Group for MCA, TNO’s knowledge was expanded.
Meanwhile, in those years, there were already a few MCA-based studies performed for the Dutch Defence Material Process, for example, the replacement of non-track vehicles for the Royal Netherlands Army, a tender for Netherlands Armed Forces Integrated Network (NAFIN), the requirements document for Short Range Air Defence systems (SHORAD).
The potential of MCA was, however, increasingly recognised. The need for flexible software tooling that offered not just one method, but a range of methods, also increased. There was also the need to be able to conduct a sort of overarching analysis that could bring together the results of partial but related studies: a so-called “Top model”.
In the 90s, TNO started the development of its own software package “TOPSystem” (The Option Preferred System, or simply “TOPSYS”). Although this was once in a while a difficult development process of many years with improvements and expansions requiring a lot of money, large studies have been supported for more than 20 years (!).
Even without TOPSYS, MCA expertise has been frequently used in TNO studies. As a by-product of the MCA expertise at TNO Waalsdorp, many refereed or non-refereed publications about MCA developments at TNO have been published. MCA lectures were also given to the Defence academy institutions, the University of Nijmegen, and as part of internal colloquia to raise awareness of MCA.
MCA at TNO after the millennium
Examples of studies since 2000 mainly support procurement projects as part of the Dutch Defence Material Process: for example Stinger missiles, Future Ground Based Air Defence System (FGBADS), Medium-altitude long-endurance unmanned aerial vehicle (MALE-UAV), torpedo systems, replacement of F-16 fighter aircraft, integrated combat helmets, etc. Also more strategic-policy issues were supported with MCA: for example concepts for land operations in 2020, relative strategic importance of military functions and capabilities, prioritising defence research topics, assessing cooperation potential with other countries in defence research, prioritising concepts of non-lethal weapons, organisational alternatives to the position of the Chief of Defence Staff, etc.
MCA knowledge was also introduced into civilian projects: for example, the selection of telecommunication providers for the government, and the national security impact assessment using scenarios that describe a possible threat to the nation (the “National Risk Assessment”). This assessment method was identified by the OECD as an international best practice together with the national risk assessment method of the United Kingdom in 2009.
Experience has shown that many clients have suspicions about the value of MCA: you get what you want out of it, it is all subjective, etc. Despite a core of truth in this, our message has always been that you can manage subjectivity in an objective way. The software in particular helps to visualise the often-complex assessment hierarchies of criteria. Calculations show the consequences of divergent opinions about certain information (for example the weights of criteria that indicate how important something should count in a final judgment) and whether somewhat differing assessment values on a specific criterion actually makes a difference (for example, whether a certain alternative still remains the most preferred one).
It has always proved important to show transparently and graphically how information is processed to a final result, that scientific principles are used to achieve such a result. Moreover, the consequences of disagreements for the final outcome can be investigated.
“From Benjamin Franklin to Joseph Priestley, 19 September 1772”, Founders Online, National Archives, version of January 18, 2019, https://founders.archives.gov/documents/Franklin/01-19-02-0200.
[Original source: The Papers of Benjamin Franklin, vol. 19, January 1 through December 31, 1772, ed. William B. Willcox. New Haven and London: Yale University Press, New Haven and London, 1975, pp. 299–300.]
The project National Risk Assessment/National Security Profile [in Dutch].
TNO brochure Multi Criteria Analyse [in Dutch].
This text was provided by Diederik Wijnmalen, for many years the driving force behind the TNO MCA research activities.