Belief aggregation, also called risk aggregation, opinion aggregation or probabilistic opinion pooling, is a process in which different probability distributions, produced by different experts, are combined to yield a single probability distribution.

Background

Expert opinions are often uncertain. Rather than saying e.g. "it will rain tomorrow", a weather expert may say "it will rain with probability 70% and be sunny with probability 30%". Such a statement is called a belief. Different experts may have different beliefs; for example, a different weather expert may say "it will rain with probability 60% and be sunny with probability 40%". In other words, each expert has a subjeciive probability distribution over a given set of outcomes. A belief aggregation rule is a function that takes as input two or more probability distributions over the same set of outcomes, and returns a single probability distribution over the same space.

Applications

Documented applications of belief aggregation include: *

Prediction of volcanic activity Prediction of volcanic activity, and volcanic eruption forecasting, is an interdisciplinary monitoring and research effort to predict the time and severity of a volcano's eruption. Of particular importance is the prediction of hazardous eruptions ...

. * Predicting the likelihood of

abrupt climate change An abrupt climate change occurs when the climate system is forced to transition at a rate that is determined by the climate system energy-balance. The transition rate is more rapid than the rate of change of the external forcing, though it may ...

. * Predicting future

polar bear The polar bear (''Ursus maritimus'') is a large bear native to the Arctic and nearby areas. It is closely related to the brown bear, and the two species can Hybrid (biology), interbreed. The polar bear is the largest extant species of bear ...

population. During COVID-19, the European Academy of Neurology developed an ad-hoc three-round voting method to aggregate expert opinions and reach a consensus.

Common rules

Common belief aggregation rules include: * ''Linear aggregation'' (also called

average voting rule The average voting rule is a rule for group decision-making when the decision is a ''distribution'' (e.g. the allocation of a budget among different issues), and each of the voters reports his ideal distribution. This is a special case of budget-pr ...

) - selecting the weighted or unweighted

arithmetic mean In mathematics and statistics, the arithmetic mean ( ), arithmetic average, or just the ''mean'' or ''average'' is the sum of a collection of numbers divided by the count of numbers in the collection. The collection is often a set of results fr ...

of the experts' reports. * ''Geometric aggregation'' - selecting the weighted or unweighted

geometric mean In mathematics, the geometric mean is a mean or average which indicates a central tendency of a finite collection of positive real numbers by using the product of their values (as opposed to the arithmetic mean which uses their sum). The geometri ...

of the reports. * ''Multiplicative aggregation'' - selecting the product of probabilities. Dietrich and List present axiomatic characterizations of each class of rules. They argue that that linear aggregation can be justified “procedurally” but not “epistemically”, while the other two rules can be justified epistemically. Geometric aggregation is justified when the experts' beliefs are based on the same information, and multiplicative aggregation is justified when the experts' beliefs are based on private information.

Properties of belief aggregation rules

A belief aggregation rule should arguably satisfy some desirable properties, or axioms: * Zero preservation means that, if all experts agree that an event has zero probability, then the same should hold in the aggregated distribution. An equivalent axiom is consensus preservation or certainty preservation, which means that, if all experts agree that an event has probability 1, then the same should hold in the aggregated distribution. This is a basic axiom that is satisfied by linear, geometric and multiplicative aggregation, as well as many others. * Plausibility preservation means that, if all experts agree that an event has a positive probability, then the same should hold in the aggregated distribution. This axiom is satisfied by linear aggregation. * Proportionality means that, if each expert assigns probability 1 to a single outcome, then the aggregated distribution is the average (or the weighted average) of the expert beliefs. This axiom is satisfied by linear aggregation. * Diversity is stronger than proportionality. It means that the support of the aggregated distribution contains the supports of all experts' beliefs. In other words, if any event has a positive probability to at least one expert, that it has a positive probability to society. This axiom is satisfied by linear aggregation.

Truthful aggregation rules with money

Most literature on belief aggregation assumes that the experts report their beliefs honestly, as their main goal is to help the decision-maker get to the truth. In practice, experts may have strategic incentives. For example, the

FDA The United States Food and Drug Administration (FDA or US FDA) is a federal agency of the Department of Health and Human Services. The FDA is responsible for protecting and promoting public health through the control and supervision of food ...

uses advisory committees, and there have been controversies due to conflicts of interests within these committees. Therefore, a

truthful mechanism In mechanism design, a strategyproof (SP) mechanism is a game form in which each player has a weakly- dominant strategy, so that no player can gain by "spying" over the other players to know what they are going to play. When the players have privat ...

for belief aggregation could be useful. In some settings, it is possible to pay the experts a certain sum of money, depending both on their expressed belief and on the realized outcome. Careful design of the payment function (often called a "

scoring rule In decision theory, a scoring rule provides evaluation metrics for probabilistic forecasting, probabilistic predictions or forecasts. While "regular" loss functions (such as mean squared error) assign a goodness-of-fit score to a predicted value ...

") can lead to a truthful mechanism. Various truthful scoring rules exist.

Truthful aggregation rules without money

In some settings, monetary transfers are not possible. For example, the realized outcome may happen in the far future, or a wrong decision may be catastrophic. To develop truthful mechanisms, one must make assumptions about the experts' preferences over the set of accepted probability-distributions. If the space of possible preferences is too rich, then strong impossibility results imply that the only truthful mechanism is the

dictatorship mechanism In social choice theory, a dictatorship mechanism is a degenerate voting rule or mechanism where the result depends on one person's. A serial dictatorship is similar, but also designates a series of "backup dictators", who break ties in the origina ...

(see

Gibbard–Satterthwaite theorem The Gibbard–Satterthwaite theorem is a theorem in social choice theory. It was first conjectured by the philosopher Michael Dummett and the mathematician Robin Farquharson in 1961 and then proved independently by the philosopher Allan Gibbard in ...

Single-peaked preferences

A useful domain restriction is that the experts have

single-peaked preferences Single-peaked preferences are a class of preference relations. A group has single-peaked preferences over a set of outcomes if the outcomes can be ordered along a line such that: # Each agent has a "best outcome" in the set, and # For each agent, ...

. An aggregation rule is called one-dimensional strategyproof (1D-SP) if whenever all experts have single-peaked preferences, and submit their peaks to the aggregation rule, no expert can impose a strictly better aggregated distribution by reporting a false peak. An equivalent property is called uncompromisingness: it says that, if the belief of expert ''i'' is smaller than the aggregate distribution, and i changes his report, then the aggregate distribution will be weakly larger; and vice-versa. Moulin proved a characterization of all 1D-SP rules, as well as the following two characterizations: * A rule is

anonymous Anonymous may refer to: * Anonymity, the state of an individual's identity, or personally identifiable information, being publicly unknown ** Anonymous work, a work of art or literature that has an unnamed or unknown creator or author * Anonym ...

and 1D-SP for all single-peaked preferences iff it is equivalent to a

median voting rule The median voting rule or median mechanism is a rule for group decision-making along a one-dimensional domain. Each person votes by writing down his/her ideal value, and the rule selects a single value which is (in the basic mechanism) the ''medi ...

with at most ''n''+1 "phantoms". * A rule is

, 1D-SP and Pareto-efficient for all single-peaked preferences iff it is equivalent to a

with at most ''n''-1 phantoms. Jennings, Laraki, Puppe and Varloot present new characterizations of strategyproof mechanisms with single-peaked preferences.

Single-peaked preferences of the pdf

A further restriction of the single-peaked domain is that agents have single-peaked preferences with L1 metric on the

probability density function In probability theory, a probability density function (PDF), density function, or density of an absolutely continuous random variable, is a Function (mathematics), function whose value at any given sample (or point) in the sample space (the s ...

. That is: for each agent ''i'', there is an "ideal" probability distribution ''p_i'', and his utility from a selected probability distribution ''p''* is minus the L1 distance between ''p_i'' and ''p*''. An aggregation rule is called L1-metric-strategyproof (L1-metric-SP) if whenever all experts have single-peaked preferences with L1 metric, and submit their peaks to the aggregation rule, no expert can impose a strictly better aggregated distribution by reporting a false peak. Several L1-metric-SP aggregation rules were suggested in the context of

budget-proposal aggregation Budget-proposal aggregation (BPA) is a problem in social choice theory. A group has to decide on how to distribute its budget among several issues. Each group-member has a different idea about what the ideal budget-distribution should be. The probl ...

: * Goel, Krishnaswamy and Sakshuwong proved the existence of a Pareto optimal aggregation rule that is L1-metric-SP; * Freeman, Pennock, Peters and Vaughan presented a rule called ''moving phantoms'', which is L1-metric-SP and satisfies a fairness property (but it is not Pareto-optimal). They also presented a family of L1-metric-SP rules based on the median rule. However, such preferences may not be a good fit for belief aggregation, as they are ''neutral'' - they do not distinguish between different outcomes. For example, suppose there are three outcomes, and the expert's belief ''p_i'' assigns 100% to outcome 1. Then, the L1 metric between ''p_i'' and "100% outcome 2" is 2, and the L1 metric between ''p_i'' and "100% outcome 3" is 2 too. The same is true for any neutral metric. This makes sense when 1,2,3 are budget items. However, if these outcomes describe the potential strength of an earthquake in the Richter scale, then the distance between ''p_i'' to "100% outcome 2" should be much smaller than the distance to "100% outcome 3".

Single-peaked preferences on the cdf

Varloot and Laraki study a different preference domain, in which the outcomes are

linearly ordered In mathematics, a total order or linear order is a partial order in which any two elements are comparable. That is, a total order is a binary relation \leq on some set X, which satisfies the following for all a, b and c in X: # a \leq a ( re ...

, and the preferences are single-peaked in the space of ''cumulative distribution function (cdf)''. That is: each agent ''i'' has an ideal cumulative distribution function ''c_i'', and his utility depends negatively on the distance between ''c_i'' and the accepted distribution ''c*.'' They define a new concept called level-strategyproofness (Level-SP), which is relevant when society's decision is based on the question of whether the probability of some event is above or below a given threshold. Level-SP provably implies strategyproofness for a rich class of cdf-single-peaked preferences. They characterize two new aggregation rules: * The ''order-cumulative rules'' are the only aggregation rules that satisfy Level-SP, anonymity, certainty-preservation and plasubility-preservation. A special case of this family is the ''middlemost'' ''cumulative'', which is an order-cumulative based on the

median The median of a set of numbers is the value separating the higher half from the lower half of a Sample (statistics), data sample, a statistical population, population, or a probability distribution. For a data set, it may be thought of as the “ ...

. ** However, these rules are not ''diverse'', for example: if three experts report "99% outcome 1" and one expert reports "99% outcome 2", then every order-cumulative rule will choose either "99% outcome 1" of "99% outcome 2"; however, an outcome such as "75% outcome 1 and 25% outcome 2" is more reasonable. * The ''proportional-cumulative rule'' is the only aggregation rule that satisfies Level-SP and proportionality. It also handles profiles with dominations (where the cdf of each agent ''i'' is either entirely above or entirely below the cdf of any other agent ''j'') in a natural way. However, it violates plausibility-preservation. Other results include: * There is no aggregation rule that satisfies diversity, Level-SP and unanimity. * When there are at least 4 outcomes, the only rules that satisfy Level-SP, L1-metric-SP and certainty-preservation are

dictatorships A dictatorship is an autocratic form of government which is characterized by a leader, or a group of leaders, who hold governmental powers with few to no limitations. Politics in a dictatorship are controlled by a dictator, and they are facili ...

(there are rules that satisfy Level-SP and L1-metric-SP, but not certainty-preservation; with 3 outcomes, every level-SP rule is also L1-metric-SP). * Most results can be extended to assign different weights to different experts (representing their level of expertise). * A new voting method: majority judgement with uncertainty (MJU). It is a variant of

majority judgement Majority judgment (MJ) is a single-winner voting system proposed in 2010 by Michel Balinski and Rida Laraki. It is a kind of highest median rule, a cardinal voting system that elects the candidate with the highest median rating. Voting proce ...

which allows voters to express uncertainty about the qualities of each candidate.

Software

ANDURIL is a MATLAB toolbox for belief aggregation.

References

{{reflist Statistical forecasting