Humans are uncomfortable with uncertainty. This innate discomfort with uncertainly drives us as a species to seek to understand the probability that events will occur. Those of us within the intelligence field are not exempt from this desire, though rather than seeking to understand the chances of winning the lottery, the events that we seek to understand in the intelligence field are often more negatively skewed. For example, an intelligence agency may wish to know the probability of an individual committing a terrorist attack or a policing authority may seek to know the chances of two rival gangs engaging in violent warfare.
It is the task of the intelligence professional to provide stakeholders with an indication of how likely it is that an event will occur i.e. an indication of probability. A task which seems almost impossible for certain scenarios, especially concerning human behaviour, but a task which is necessary and commonly requested in the field.
Let's take a hypothetical scenario, imagine you were asked to assess the probability of a 3 being thrown on a die, but you had no concept of what a die was. How would you approach this task? If you were to follow the intelligence cycle you would start collecting information around the subject that believed would help you to understand the question. You may have a website as a source that tells you that a die is a 6-sided object with numbers on it, but can you trust this website? You may have a photo of a die, but you can only see 3 sides with numbers on, how do you know with any confidence what is on the other 3 sides? Slowly after time, gathering more trustworthy sources and building a corroborated intelligence picture, you would conclude most dice have 6 sides, with numbers 1 to 6 on a side and each number has an equal chance of being thrown. At this point, you would feel pretty confident that you have a good evidence base to say the probability of a 3 being thrown is 1/6.
An intelligence task is arguably no different; you collect, collate and analyse information, building your intelligence picture. As more trustworthy sources provide intelligence that corroborates the same findings, you would have more confidence to indicate the probability of a given event unfolding.
However, unlike dice, the subject of intelligence questions are often shrouded in mystery, most of the time individuals are actively trying to guard information and most importantly there are often multiple moving parts that interact with each other. Thus, intelligence subjects are often more difficult to gain coverage over and at times require assessments of probability to be based on a less than desirable intelligence picture. This creates an uncomfortable position where you can have a high level of uncertainty on a subject, yet you are being tasked with passing an assessment on the subject matter.
There are two frameworks that can be implemented to help the intelligence professional to communicate probability or to demonstrate the level of uncertainty in intelligence products. The first framework is assigning a level of confidence and the other framework is a tool known as the probability yardstick.
Level of Confidence
For most intelligence subjects there is always going to be a level of uncertainty. This could be due to a number of reasons, such as; intelligence which is not corroborated, sources of intelligence are felt not to be credible or there are too many intelligence gaps to provide an assessment with any confidence.
By giving intelligence overall products or assessments a level of confidence, the intelligence professional can communicate with the stakeholder the nature of the intelligence base behind the product. This is achieved with a simple three-tiered system, as seen below.
High confidence is the top tier. This would indicate that the assessments made are based on high-quality intelligence from trustworthy sources and that information has been corroborated by multiple sources. This level of confidence does not determine 100% certainty but suggests that there is a good basis for believing the product is well informed.
Moderate/Medium Confidence is the middle tier. This tier could be used to represent an intelligence picture that has some limitations. This could be due to intelligence that has not corroborated by multiple sources, possibly because of some scepticism in the nature of the sources or because there may be alternative views to the assessment which cannot be ruled out. The intelligence picture may still be developing with the current intelligence felt to be comprehensive enough to satisfy a high confidence tier.
Low confidence is the lowest tier. Here the intelligence base is felt not to be sufficient or credible enough to make a good judgement on the subject matter at hand. This could reflect serious doubts around intelligence sources, a lack of corroboration from other sources or that there is not enough of an evidence base to rely upon. In this tier, uncertainty is at its highest.
A stakeholder may still wish to act upon an intelligence assessment with low confidence, but by clearly stating a level of confidence you have communicated the potential for the assessment to be undermined and the need to build the level of confidence around the subject matter.
A level of confidence is usually applied to an entire intelligence product. Within products, numerous assessments and judgements will be made that require a level of uncertainty to be defined. A tool which helps communicate uncertainty for these multiple assessments is the Probability Yardstick which is outlined below.
The Probability Yardstick
The probability yardstick was introduced by The Professional Head of Intelligence Assessment, influenced by defence intelligence practices. The tool provides a standardised language to communicate probability that numerous intelligence agencies and other departments have adopted.
Probabilistic Language
The most common way to communicate probability is through the use of probabilistic language. This is simply a collection of words that communicate a level of probability e.g. highly likely, a sure thing, 50% chance. These words allow you to communicate to others what you think the chances are of an event unfolding. One key issue with probabilistic language is the fact people assign different levels of probability to the same word. For example, if I was to say it is likely that I will go out tonight, this may mean to me there is a 70% chance I will go out, but to another, it could mean there is a 100% chance. As you can imagine, this level of disparity may have little consequences for our example, though, in a tactical or operational arena, this disparity could have serious implications.
Thus, the Probability Yardstick was introduced to reduce the risk of misinterpretation.
Source: Professional Head Of Intelligence Assessment (2018) Professional Development Framework.
This tool is used across intelligence agencies and other organisations. With all of these organisations all utilising a standardised tool, any assessment from any agency can be understood by others. This helps to encourage collaborative work between organisations whilst raising the level of understanding between agencies.
The probability yardstick is still subject to a level of interpretation and individuals may have a different view on what would be considered likely or a realistic probability, but by reducing the options of probabilistic words, the yardstick allows a more focused assessment which reduces the chances of a key stakeholder misunderstanding assessments.
Furthermore, as seen with the level of confidence, this tool allows users to have a method to understand when there is more or less uncertainty present. The user can know a "highly likely" assessment in a product is more likely to occur than an assessment which has been given a "probable" likelihood.
Overview
To conclude, the intelligence field requires intelligence professionals to make judgement calls on the probability of an event unfolding. This judgement is often very difficult given the level of uncertainty present with most subject matters. However, the tools explored above help the intelligence professional to communicate this level of uncertainty, ensuring that stakeholders are as well informed as possible.
Looking at the die example above, we could now say that we have high confidence in our evidence base concerning the die. With the chances of rolling a 3 being highly unlikely. A good intelligence professional would also consider other possible hypotheses such as "the die could be loaded?" or "is the die going to be thrown in the first place?", but the subject of competing hypothesis is a subject for another time.
Voltaire said, " Uncertainty is an uncomfortable position, but certainty is an absurd one". Thus, intelligence professionals are rarely comfortable, however, the ability to communicate the level of uncertainty should now be as easy as rolling the dice.
To learn more about how to deal with uncertainty, check out Dan's section on Communicating Uncertainty and Probability within our Introduction to Intelligence course.
Sources and further reading:
Analytic Confidence and Political Decision Making: Theoretical Principles and Experimental Evidence from National Security Professionals Jeffrey A. Friedman, Assistant Professor of Government, Dartmouth College & Visiting Fellow, Institute for Advanced Study in Toulouse Richard Zeckhauser, Frank P. Ramsey Professor of Political Economy, Harvard University Forthcoming in Political Psychology
National Intelligence Estimate on Iraq, 2007 - https://fas.org/irp/dni/iraq020207.pdf
College of Policing - https://www.app.college.police.uk/app-content/intelligence-management/analysis/delivering-effective-analysis/
Comments