In the
statistical
Statistics (from German: '' Statistik'', "description of a state, a country") is the discipline that concerns the collection, organization, analysis, interpretation, and presentation of data. In applying statistics to a scientific, industr ...
analysis of
time series
In mathematics, a time series is a series of data points indexed (or listed or graphed) in time order. Most commonly, a time series is a sequence taken at successive equally spaced points in time. Thus it is a sequence of discrete-time data. Ex ...
, autoregressive–moving-average (ARMA) models provide a parsimonious description of a
(weakly) stationary stochastic process in terms of two polynomials, one for the
autoregression (AR) and the second for the
moving average
In statistics, a moving average (rolling average or running average) is a calculation to analyze data points by creating a series of averages of different subsets of the full data set. It is also called a moving mean (MM) or rolling mean and is ...
(MA). The general ARMA model was described in the 1951 thesis of
Peter Whittle, ''Hypothesis testing in time series analysis'', and it was popularized in the 1970 book by
George E. P. Box
George Edward Pelham Box (18 October 1919 – 28 March 2013) was a British statistician, who worked in the areas of quality control, time-series analysis, design of experiments, and Bayesian inference. He has been called "one of the gre ...
and
Gwilym Jenkins.
Given a time series of data
, the ARMA model is a tool for understanding and, perhaps, predicting future values in this series. The AR part involves regressing the variable on its own lagged (i.e., past) values. The MA part involves modeling the
error term In mathematics and statistics, an error term is an additive type of error
An error (from the Latin ''error'', meaning "wandering") is an action which is inaccurate or incorrect. In some usages, an error is synonymous with a mistake. The etymol ...
as a
linear combination of error terms occurring contemporaneously and at various times in the past. The model is usually referred to as the ARMA(''p'',''q'') model where ''p'' is the order of the AR part and ''q'' is the order of the MA part (as defined below).
ARMA models can be estimated by using the
Box–Jenkins method.
Autoregressive model
The notation AR(''p'') refers to the autoregressive model of order ''p''. The AR(''p'') model is written as
:
where
are
parameter
A parameter (), generally, is any characteristic that can help in defining or classifying a particular system (meaning an event, project, object, situation, etc.). That is, a parameter is an element of a system that is useful, or critical, when ...
s and the random variable
is
white noise
In signal processing, white noise is a random signal having equal intensity at different frequencies, giving it a constant power spectral density. The term is used, with this or similar meanings, in many scientific and technical disciplines ...
, usually
independent and identically distributed
In probability theory and statistics, a collection of random variables is independent and identically distributed if each random variable has the same probability distribution as the others and all are mutually independent. This property is usual ...
(i.i.d.)
normal random variables.
In order for the model to remain
stationary, the roots of its
characteristic polynomial
In linear algebra, the characteristic polynomial of a square matrix is a polynomial which is invariant under matrix similarity and has the eigenvalues as roots. It has the determinant and the trace of the matrix among its coefficients. The c ...
must lie outside of the unit circle. For example, processes in the AR(1) model with
are not stationary because the root of
lies within the unit circle.
Moving-average model
The notation MA(''q'') refers to the moving average model of order ''q'':
:
where the
are the parameters of the model,
is the expectation of
(often assumed to equal 0), and the
,
,... are again i.i.d. white noise error terms that are commonly normal random variables.
ARMA model
The notation ARMA(''p'', ''q'') refers to the model with ''p'' autoregressive terms and ''q'' moving-average terms. This model contains the AR(''p'') and MA(''q'') models,
:
The general ARMA model was described in the 1951 thesis of
Peter Whittle, who used mathematical analysis (
Laurent series
In mathematics, the Laurent series of a complex function f(z) is a representation of that function as a power series which includes terms of negative degree. It may be used to express complex functions in cases where a Taylor series expansion c ...
and
Fourier analysis
In mathematics, Fourier analysis () is the study of the way general functions may be represented or approximated by sums of simpler trigonometric functions. Fourier analysis grew from the study of Fourier series, and is named after Joseph ...
) and statistical inference. ARMA models were popularized by a 1970 book by
George E. P. Box
George Edward Pelham Box (18 October 1919 – 28 March 2013) was a British statistician, who worked in the areas of quality control, time-series analysis, design of experiments, and Bayesian inference. He has been called "one of the gre ...
and Jenkins, who expounded an iterative (
Box–Jenkins) method for choosing and estimating them. This method was useful for low-order polynomials (of degree three or less).
The ARMA model is essentially an
infinite impulse response
Infinite impulse response (IIR) is a property applying to many linear time-invariant systems that are distinguished by having an impulse response h(t) which does not become exactly zero past a certain point, but continues indefinitely. This is in ...
filter applied to white noise, with some additional interpretation placed on it.
Specification in terms of lag operator
In some texts the models will be specified in terms of the
lag operator ''L''.
In these terms then the AR(''p'') model is given by
:
where
represents the polynomial
:
The MA(''q'') model is given by
:
where
represents the polynomial
:
Finally, the combined ARMA(''p'', ''q'') model is given by
:
or more concisely,
:
or
:
Alternative notation
Some authors, including
Box,
Jenkins & Reinsel use a different convention for the autoregression coefficients. This allows all the polynomials involving the lag operator to appear in a similar form throughout. Thus the ARMA model would be written as
:
Moreover, starting summations from
and setting
and
, then we get an even more elegant formulation:
Fitting models
Choosing p and q
Finding appropriate values of ''p'' and ''q'' in the ARMA(''p'',''q'') model can be facilitated by plotting the
partial autocorrelation function
In time series analysis, the partial autocorrelation function (PACF) gives the partial correlation of a stationary time series with its own lagged values, regressed the values of the time series at all shorter lags. It contrasts with the autocorre ...
s for an estimate of ''p'', and likewise using the
autocorrelation functions for an estimate of ''q''. Extended autocorrelation functions (EACF) can be used to simultaneously determine p and q. Further information can be gleaned by considering the same functions for the residuals of a model fitted with an initial selection of ''p'' and ''q''.
Brockwell & Davis recommend using
Akaike information criterion (AIC) for finding ''p'' and ''q''. Another possible choice for order determining is the
BIC criterion.
Estimating coefficients
ARMA models in general can be, after choosing ''p'' and ''q'', fitted by
least squares
The method of least squares is a standard approach in regression analysis to approximate the solution of overdetermined systems (sets of equations in which there are more equations than unknowns) by minimizing the sum of the squares of the re ...
regression to find the values of the parameters which minimize the error term. It is generally considered good practice to find the smallest values of ''p'' and ''q'' which provide an acceptable fit to the data. For a pure AR model the
Yule-Walker equations may be used to provide a fit.
Unlike other methods of regression (i.e. OLS, 2SLS, etc.) often employed in econometric analysis, ARMA model outputs are used primarily for the cases of forecasting time-series data. Their coefficients are then as such only utilized for prediction. Other areas of econometrics look at the causal inference, time-series forecasting using ARMA is not. The coefficients should then only be seen as useful for predictive modelling.
Implementations in statistics packages
* In
R, the ''arima'' function (in standard package ''stats'') is documented i
ARIMA Modelling of Time Series Extension packages contain related and extended functionality, e.g., the ''tseries'' package includes an ''arma'' function, documented i
th
''fracdiff'' packagecontains ''fracdiff()'' for fractionally integrated ARMA processes; and th
includes ''auto.arima'' for selecting a parsimonious set of ''p,q''. The CRAN task view o
contains links to most of these.
*
Mathematica
Wolfram Mathematica is a software system with built-in libraries for several areas of technical computing that allow machine learning, statistics, symbolic computation, data manipulation, network analysis, time series analysis, NLP, optimiza ...
has a complete library of time series functions including ARMA.
*
MATLAB
MATLAB (an abbreviation of "MATrix LABoratory") is a proprietary multi-paradigm programming language and numeric computing environment developed by MathWorks. MATLAB allows matrix manipulations, plotting of functions and data, implementat ...
includes functions such a
''arma''an
to estimate AR, ARX (autoregressive exogenous), and ARMAX models. See
an
for more information.
*
Julia has some community driven packages that implement fitting with an ARMA model such a
''arma.jl''
*
Statsmodels Python module includes many models and functions for time series analysis, including ARMA. Formerly part of the
scikit-learn library, it is now stand-alone and integrates well with
PandasSee here for more details
*
PyFlux has a Python-based implementation of ARIMAX models, including Bayesian ARIMAX models.
*
IMSL Numerical Libraries
IMSL (International Mathematics and Statistics Library) is a commercial collection of software libraries of numerical analysis functionality that are implemented in the computer programming languages C, Java, C#.NET, and Fortran. A Python int ...
are libraries of numerical analysis functionality including ARMA and ARIMA procedures implemented in standard programming languages like C, Java, C# .NET, and Fortran.
*
gretl
gretl is an open-source statistical package, mainly for econometrics. The name is an acronym for ''G''nu ''R''egression, ''E''conometrics and ''T''ime-series ''L''ibrary.
It has both a graphical user interface (GUI) and a command-line inter ...
can also estimate ARMA model
see here where it's mentioned
*
GNU Octave
GNU Octave is a high-level programming language primarily intended for scientific computing and numerical computation. Octave helps in solving linear and nonlinear problems numerically, and for performing other numerical experiments using a lan ...
can estimate AR models using functions from the extra packag
octave-forge
*
Stata
Stata (, , alternatively , occasionally stylized as STATA) is a general-purpose statistical software package developed by StataCorp for data manipulation, visualization, statistics, and automated reporting. It is used by researchers in many fie ...
includes the function ''arima'' which can estimate ARMA and
ARIMA
Arima, officially The Royal Chartered Borough of Arima is the easternmost and second largest in area of the three boroughs of Trinidad and Tobago. It is geographically adjacent to Sangre Grande and Arouca at the south central foothills of ...
models
See here for more details
*
SuanShu is a Java library of numerical methods, including comprehensive statistics packages, in which univariate/multivariate ARMA, ARIMA, ARMAX, etc. models are implemented in an object-oriented approach. These implementations are documented i
"SuanShu, a Java numerical and statistical library"
*
SAS has an econometric package, ETS, that estimates ARIMA models
See here for more details
Spectrum
The
spectral density
The power spectrum S_(f) of a time series x(t) describes the distribution of power into frequency components composing that signal. According to Fourier analysis, any physical signal can be decomposed into a number of discrete frequencies ...
of an ARMA process is
where
is the
variance
In probability theory and statistics, variance is the expectation of the squared deviation of a random variable from its population mean or sample mean. Variance is a measure of dispersion, meaning it is a measure of how far a set of numbe ...
of the white noise,
is the characteristic polynomial of the moving average part of the ARMA model, and
is the characteristic polynomial of the autoregressive part of the ARMA model.
Applications
ARMA is appropriate when a system is a function of a series of unobserved shocks (the MA or moving average part) as well as its own behavior. For example, stock prices may be shocked by fundamental information as well as exhibiting technical trending and
mean-reversion effects due to market participants.
Generalizations
The dependence of
on past values and the error terms ε
t is assumed to be linear unless specified otherwise. If the dependence is nonlinear, the model is specifically called a ''nonlinear moving average'' (NMA), ''nonlinear autoregressive'' (NAR), or ''nonlinear autoregressive–moving-average'' (NARMA) model.
Autoregressive–moving-average models can be generalized in other ways. See also
autoregressive conditional heteroskedasticity (ARCH) models and
autoregressive integrated moving average (ARIMA) models. If multiple time series are to be fitted then a vector ARIMA (or VARIMA) model may be fitted. If the time-series in question exhibits long memory then fractional ARIMA (FARIMA, sometimes called ARFIMA) modelling may be appropriate: see
Autoregressive fractionally integrated moving average. If the data is thought to contain seasonal effects, it may be modeled by a SARIMA (seasonal ARIMA) or a periodic ARMA model.
Another generalization is the ''multiscale autoregressive'' (MAR) model. A MAR model is indexed by the nodes of a tree, whereas a standard (discrete time) autoregressive model is indexed by integers.
Note that the ARMA model is a univariate model. Extensions for the multivariate case are the
vector autoregression
Vector autoregression (VAR) is a statistical model used to capture the relationship between multiple quantities as they change over time. VAR is a type of stochastic process model. VAR models generalize the single-variable (univariate) autoregres ...
(VAR) and Vector Autoregression Moving-Average (VARMA).
Autoregressive–moving-average model with exogenous inputs model (ARMAX model)
The notation ARMAX(''p'', ''q'', ''b'') refers to the model with ''p'' autoregressive terms, ''q'' moving average terms and ''b'' exogenous inputs terms. This model contains the AR(''p'') and MA(''q'') models and a linear combination of the last ''b'' terms of a known and external time series
. It is given by:
:
where
are the ''parameters'' of the exogenous input
.
Some nonlinear variants of models with exogenous variables have been defined: see for example
Nonlinear autoregressive exogenous model.
Statistical packages implement the ARMAX model through the use of "exogenous" (that is, independent,) variables. Care must be taken when interpreting the output of those packages, because the estimated parameters usually (for example, in
R[ARIMA Modelling of Time Series](_blank)
R documentation and
gretl
gretl is an open-source statistical package, mainly for econometrics. The name is an acronym for ''G''nu ''R''egression, ''E''conometrics and ''T''ime-series ''L''ibrary.
It has both a graphical user interface (GUI) and a command-line inter ...
) refer to the regression:
:
where
incorporates all exogenous (or independent) variables:
:
See also
*
Autoregressive integrated moving average (ARIMA)
*
Exponential smoothing
*
Linear predictive coding
Linear predictive coding (LPC) is a method used mostly in audio signal processing and speech processing for representing the spectral envelope of a digital signal of speech in compressed form, using the information of a linear predictive mod ...
*
Predictive analytics
Predictive analytics encompasses a variety of statistical techniques from data mining, predictive modeling, and machine learning that analyze current and historical facts to make predictions about future or otherwise unknown events.
In busine ...
*
Infinite impulse response
Infinite impulse response (IIR) is a property applying to many linear time-invariant systems that are distinguished by having an impulse response h(t) which does not become exactly zero past a certain point, but continues indefinitely. This is in ...
*
Finite impulse response
In signal processing, a finite impulse response (FIR) filter is a filter whose impulse response (or response to any finite length input) is of ''finite'' duration, because it settles to zero in finite time. This is in contrast to infinite impulse ...
References
Further reading
*
*
* .
{{DEFAULTSORT:Autoregressive-Moving-Average Model
Autocorrelation