Spillover effects in experiments
   HOME

TheInfoList



OR:

In
experiment An experiment is a procedure carried out to support or refute a hypothesis, or determine the efficacy or likelihood of something previously untried. Experiments provide insight into Causality, cause-and-effect by demonstrating what outcome oc ...
s, a spillover is an indirect effect on a subject not directly treated by the experiment. These effects are useful for
policy analysis Policy analysis is a technique used in the public administration sub-field of political science to enable civil servants, nonprofit organizations, and others to examine and evaluate the available options to implement the goals of laws and elected ...
but complicate the statistical analysis of experiments. Analysis of spillover effects involves relaxing the non-interference assumption, or SUTVA (Stable Unit Treatment Value Assumption). This assumption requires that subject ''i'''s revelation of its potential outcomes depends only on that subject ''i'''s own treatment status, and is unaffected by another subject ''j'''s treatment status. In ordinary settings where the researcher seeks to estimate the
average treatment effect The average treatment effect (ATE) is a measure used to compare treatments (or interventions) in randomized experiments, evaluation of policy interventions, and medical trials. The ATE measures the difference in mean (average) outcomes between units ...
(\widehat), violation of the non-interference assumption means that traditional estimators for the ATE, such as difference-in-means, may be biased. However, there are many real-world instances where a unit's revelation of potential outcomes depend on another unit's treatment assignment, and analyzing these effects may be just as important as analyzing the direct effect of treatment. One solution to this problem is to redefine the causal
estimand An estimand is a quantity that is to be estimated in a statistical analysis. The term is used to more clearly distinguish the target of inference from the method used to obtain an approximation of this target (i.e., the estimator) and the specific v ...
of interest by redefining a subject's potential outcomes in terms of one's own treatment status and related subjects' treatment status. The researcher can then analyze various estimands of interest separately. One important assumption here is that this process captures all patterns of spillovers, and that there are no unmodeled spillovers remaining (ex. spillovers occur within a two-person household but not beyond). Once the potential outcomes are redefined, the rest of the statistical analysis involves modeling the probabilities of being exposed to treatment given some schedule of treatment assignment, and using inverse probability weighting (IPW) to produce unbiased (or asymptotically unbiased) estimates of the estimand of interest.


Examples of spillover effects

Spillover effects can occur in a variety of different ways. Common applications include the analysis of social network spillovers and geographic spillovers. Examples include the following: *
Communication Communication (from la, communicare, meaning "to share" or "to be in relation with") is usually defined as the transmission of information. The term may also refer to the message communicated through such transmissions or the field of inqui ...
: An intervention that conveys information about a
technology Technology is the application of knowledge to reach practical goals in a specifiable and Reproducibility, reproducible way. The word ''technology'' may also mean the product of such an endeavor. The use of technology is widely prevalent in me ...
or product can influence the take-up decisions of others in their
network Network, networking and networked may refer to: Science and technology * Network theory, the study of graphs as a representation of relations between discrete objects * Network science, an academic field that studies complex networks Mathematics ...
if it diffuses beyond the initial user. *
Competition Competition is a rivalry where two or more parties strive for a common goal which cannot be shared: where one's gain is the other's loss (an example of which is a zero-sum game). Competition can arise between entities such as organisms, indiv ...
: Job placement assistance for young job seekers may influence the job market prospects of individuals who did not receive the training but are competing for the same jobs. * Contagion: Receiving deworming drugs can decrease other's likelihood of contracting the disease. *
Deterrence Deterrence may refer to: * Deterrence theory, a theory of war, especially regarding nuclear weapons * Deterrence (penology), a theory of justice * Deterrence (psychology) Deterrence in relation to criminal offending is the idea or theory that t ...
: Information about government audits in specific municipalities can spread to nearby municipalities. * Displacement: A hotspot policing intervention that increases policing presence on a given street can lead to the displacement of crime onto nearby untreated streets. * Reallocation of resources: A hotspot policing intervention that increases policing presence on a given street can decrease police presence on nearby streets. * Social comparison: A program that randomizes individuals to receive a voucher to move to a new neighborhood can additionally influence the control group's beliefs about their housing conditions. In such examples, treatment in a randomized-control trial can have a direct effect on those who receive the intervention and also a spillover effect on those who were not directly treated.


Statistical issues

Estimating spillover effects in
experiment An experiment is a procedure carried out to support or refute a hypothesis, or determine the efficacy or likelihood of something previously untried. Experiments provide insight into Causality, cause-and-effect by demonstrating what outcome oc ...
s introduces three statistical issues that researchers must take into account.


Relaxing the non-interference assumption

One key assumption for
unbiased Bias is a disproportionate weight ''in favor of'' or ''against'' an idea or thing, usually in a way that is closed-minded, prejudicial, or unfair. Biases can be innate or learned. People may develop biases for or against an individual, a group, ...
inference is the non-interference assumption, which posits that an individual's potential outcomes are only revealed by their own treatment assignment and not the treatment assignment of others. This assumption has also been called the Individualistic Treatment Response or the stable unit treatment value assumption. Non-interference is violated when subjects can communicate with each other about their treatments, decisions, or experiences, thereby influencing each other's potential outcomes. If the non-interference assumption does not hold, units no longer have just two potential outcomes (treated and control), but a variety of other potential outcomes that depend on other units’ treatment assignments, which complicates the
estimation Estimation (or estimating) is the process of finding an estimate or approximation, which is a value that is usable for some purpose even if input data may be incomplete, uncertain, or unstable. The value is nonetheless usable because it is de ...
of the
average treatment effect The average treatment effect (ATE) is a measure used to compare treatments (or interventions) in randomized experiments, evaluation of policy interventions, and medical trials. The ATE measures the difference in mean (average) outcomes between units ...
. Estimating spillover effects requires relaxing the non-interference assumption. This is because a unit's outcomes depend not only on its treatment assignment but also on the treatment assignment of its neighbors. The researcher must posit a set of potential outcomes that limit the type of interference. As an example, consider an
experiment An experiment is a procedure carried out to support or refute a hypothesis, or determine the efficacy or likelihood of something previously untried. Experiments provide insight into Causality, cause-and-effect by demonstrating what outcome oc ...
that sends out political information to undergraduate students to increase their political participation. If the
study population Clinical trials are prospective biomedical or behavioral research studies on human participants designed to answer specific questions about biomedical or behavioral interventions, including new treatments (such as novel vaccines, drugs, dietar ...
consists of all students living with a roommate in a college dormitory, one can imagine four sets of potential outcomes, depending on whether the student or their partner received the information (assume no spillover outside of each two-person room): * ''Y''0,0 refers to an individual's potential outcomes when they are not treated (0) and neither was their roommate (0). * ''Y''0,1 refers to an individual's potential outcome when they are not treated (0) but their roommate was treated (1). * ''Y''1,0 refers to an individual's potential outcome when they are treated (1) but their roommate was not treated (0). * ''Y''1,1 refers to an individual's potential outcome when they are treated (1) and their roommate was treated (1). Now an individual's outcomes are influenced by both whether they received the treatment and whether their roommate received the treatment. We can estimate one type of spillover effect by looking at how one's outcomes change depending on whether their roommate received the treatment or not, given the individual did not receive treatment directly. This would be captured by the difference Y0,1- Y0,0. Similarly, we can measure how ones’ outcomes change depending on their roommate's treatment status, when the individual themselves are treated. This amounts to taking the difference Y1,1- Y1,0. While researchers typically embrace
experiment An experiment is a procedure carried out to support or refute a hypothesis, or determine the efficacy or likelihood of something previously untried. Experiments provide insight into Causality, cause-and-effect by demonstrating what outcome oc ...
s because they require less demanding assumptions, spillovers can be “unlimited in extent and impossible to specify in form.” The researcher must make specific assumptions about which types of spillovers are operative. One can relax the non-interference assumption in various ways depending on how spillovers are thought to occur in a given setting. One way to model spillover effects is a
binary Binary may refer to: Science and technology Mathematics * Binary number, a representation of numbers using only two digits (0 and 1) * Binary function, a function that takes two arguments * Binary operation, a mathematical operation that ta ...
indicator for whether an immediate neighbor was also treated, as in the example above. One can also posit spillover effects that depend on the number of immediate neighbors that were also treated, also known as k-level effects.


Exposure mappings

The next step after redefining the causal estimand of interest is to characterize the probability of spillover exposure for each subject in the analysis, given some vector of treatment assignment. Aronow and Samii (2017) present a method for obtaining a matrix of exposure probabilities for each unit in the analysis. First, define a diagonal matrix with a vector of treatment assignment probabilities \mathbf = \operatorname \left( p_ , p_ , \dots , p_ \right). Second, define an indicator matrix \mathbf of whether the unit is exposed to spillover or not. This is done by using an
adjacency matrix In graph theory and computer science, an adjacency matrix is a square matrix used to represent a finite graph. The elements of the matrix indicate whether pairs of vertices are adjacent or not in the graph. In the special case of a finite simp ...
as shown on the right, where information regarding a network can be transformed into an indicator matrix. This resulting indicator matrix will contain values of d_k, the realized values of a random binary variable D_i = f \left( \mathbf , \theta_i \right), indicating whether that unit has been exposed to spillover or not. Third, obtain the sandwich product \mathbf _k \mathbf \mathbf _k^, an ''N'' × ''N'' matrix which contains two elements: the individual probability of exposure \pi _ \left( d _ \right)on the diagonal, and the joint exposure probabilities \pi _ \left( d _ \right)on the off diagonals: : \mathbf _k \mathbf \mathbf _k^\prime = \left \begin & \pi_ (d_k) & \cdots & \pi_ (d_k) \\ \pi_(d_k) & \pi_2(d_k) & \cdots & \pi_(d_k) \\ \vdots & \vdots & \ddots & \\ \pi_(d_k) & \pi_(d_k) & & \pi_N ( d_k) \end \right/math>In a similar fashion, the joint probability of exposure of ''i'' being in exposure condition d_k and ''j'' being in a different exposure condition d_lcan be obtained by calculating \mathbf _ \mathbf \mathbf _ ^ : \mathbf _ \mathbf \mathbf _ ^ = \left \begin & & & \\ & & & \\ & & & \\ \pi_ (d_k, d_l ) & \pi_ (d_k, d_l) & & 0 \end \right/math>Notice that the diagonals on the second matrix are 0 because a subject cannot be simultaneously exposed to two different exposure conditions at once, in the same way that a subject cannot reveal two different potential outcomes at once. The obtained exposure probabilities \pithen can be used for inverse probability weighting (IPW, described below), in an estimator such as the
Horvitz–Thompson estimator In statistics, the Horvitz–Thompson estimator, named after Daniel G. Horvitz and Donovan J. Thompson, is a method for estimating the total and mean of a pseudo-population in a stratified sample. Inverse probability weighting is applied to ac ...
. One important caveat is that this procedure excludes all units whose probability of exposure is zero (ex. a unit that is not connected to any other units), since these numbers end up in the denominator of the IPW regression.


Need for inverse probability weights

Estimating spillover effects requires additional care: although treatment is directly assigned, spillover status is indirectly assigned and can lead to differential probabilities of spillover assignment for units. For example, a subject with 10 friend connections is more likely to be indirectly exposed to treatment as opposed to a subject with just one friend connection. Not accounting for varying probabilities of spillover exposure can
bias Bias is a disproportionate weight ''in favor of'' or ''against'' an idea or thing, usually in a way that is closed-minded, prejudicial, or unfair. Biases can be innate or learned. People may develop biases for or against an individual, a group ...
estimates of the average spillover effect. Figure 1 displays an example where units have varying probabilities of being assigned to the spillover condition. Subfigure A displays a
network Network, networking and networked may refer to: Science and technology * Network theory, the study of graphs as a representation of relations between discrete objects * Network science, an academic field that studies complex networks Mathematics ...
of 25 nodes where the units in green are eligible to receive treatment. Spillovers are defined as sharing at least one edge with a treated unit. For example, if node 16 is treated, nodes 11, 17, and 21 would be classified as spillover units. Suppose three of these six green units are selected randomly to be treated, so that \binom=20 different sets of treatment assignments are possible. In this case, subfigure B displays each node's probability of being assigned to the spillover condition. Node 3 is assigned to spillover in 95% of the randomizations because it shares edges with three units that are treated. This node will only be a control node in 5% of randomizations: that is, when the three treated nodes are 14, 16, and 18. Meanwhile, node 15 is assigned to spillover only 50% of the time—if node 14 is not directly treated, node 15 will not be assigned to spillover.


Using inverse probability weights

When analyzing
experiment An experiment is a procedure carried out to support or refute a hypothesis, or determine the efficacy or likelihood of something previously untried. Experiments provide insight into Causality, cause-and-effect by demonstrating what outcome oc ...
s with varying probabilities of assignment, special precautions should be taken. These differences in assignment probabilities may be neutralized by inverse-probability-weighted (IPW) regression, where each observation is weighted by the inverse of its likelihood of being assigned to the treatment condition observed using the Horvitz-Thompson estimator. This approach addresses the
bias Bias is a disproportionate weight ''in favor of'' or ''against'' an idea or thing, usually in a way that is closed-minded, prejudicial, or unfair. Biases can be innate or learned. People may develop biases for or against an individual, a group ...
that might arise if potential outcomes were systematically related to assignment probabilities. The downside of this
estimator In statistics, an estimator is a rule for calculating an estimate of a given quantity based on observed data: thus the rule (the estimator), the quantity of interest (the estimand) and its result (the estimate) are distinguished. For example, the ...
is that it may be fraught with sampling variability if some observations are accorded a high amount of weight (i.e. a unit with a low probability of being spillover is assigned to the spillover condition by chance).


Using randomization inference for hypothesis testing

In some settings, estimating the variability of a spillover effect creates additional difficulty. When the research study has a fixed unit of clustering, such as a school or household, researchers can use traditional
standard error The standard error (SE) of a statistic (usually an estimate of a parameter) is the standard deviation of its sampling distribution or an estimate of that standard deviation. If the statistic is the sample mean, it is called the standard error o ...
adjustment tools like cluster-robust standard errors, which allow for
correlations In statistics, correlation or dependence is any statistical relationship, whether causal or not, between two random variables or bivariate data. Although in the broadest sense, "correlation" may indicate any type of association, in statistics ...
in error terms within clusters but not across them. In other settings, however, there is no fixed unit of clustering. In order to conduct
hypothesis testing A statistical hypothesis test is a method of statistical inference used to decide whether the data at hand sufficiently support a particular hypothesis. Hypothesis testing allows us to make probabilistic statements about population parameters. ...
in these settings, the use of randomization inference is recommended. This technique allows one to generate p-values and confidence intervals even when spillovers do not adhere to a fixed unit of clustering but nearby units tend to be assigned to similar spillover conditions, as in the case of
fuzzy clustering Fuzzy clustering (also referred to as soft clustering or soft ''k''-means) is a form of clustering in which each data point can belong to more than one cluster. Clustering or cluster analysis involves assigning data points to clusters such that ...
.


See also

*
Social multiplier effect The social multiplier effect is a term used in economics, economic geography, sociology, public health and other academic disciplines to describe certain social externalities. It is based on the principle that high levels of one attribute amongst o ...


References

{{Reflist Design of experiments Survey methodology Asymptotic analysis Statistical inference Causal inference Systems analysis