Autoregressive Moving-average Model

In the statistical analysis of time series, autoregressive–moving-average (ARMA) models are a way to describe a (weakly) stationary stochastic process using autoregression (AR) and a moving average (MA), each with a polynomial. They are a tool for understanding a series and predicting future values. AR involves regressing the variable on its own lagged (i.e., past) values. MA involves modeling the error as a linear combination of error terms occurring contemporaneously and at various times in the past. The model is usually denoted ARMA(''p'', ''q''), where ''p'' is the order of AR and ''q'' is the order of MA.

The general ARMA model was described in the 1951 thesis of Peter Whittle, ''Hypothesis testing in time series analysis'', and it was popularized in the 1970 book by George E. P. Box and Gwilym Jenkins.

ARMA models can be estimated by using the Box–Jenkins method.

Mathematical formulation

Autoregressive model

The notation AR(''p'') refers to the autoregressive model of order ''p''. The AR(''p'') model is written as

:$X_t = \sum_^p \varphi_i X_+ \varepsilon_t$

where $\varphi_1, \ldots, \varphi_p$ are parameters and the random variable $\varepsilon_t$ is white noise, usually independent and identically distributed (i.i.d.) normal random variables.

In order for the model to remain stationary, the roots of its characteristic polynomial must lie outside the unit circle. For example, processes in the AR(1) model with $, \varphi_1, \ge 1$ are not stationary because the root of $1 - \varphi_1B = 0$ lies within the unit circle.

The augmented Dickey–Fuller test can assesses the stability of an intrinsic mode function and trend components. For stationary time series, the ARMA models can be used, while for non-stationary series, Long short-term memory models can be used to derive abstract features. The final value is obtained by reconstructing the predicted outcomes of each time series.

Moving average model

The notation MA(''q'') refers to the moving average model of order ''q'':

:$X_t = \mu + \varepsilon_t + \sum_^q \theta_i \varepsilon_\,$

where the $\theta_1,...,\theta_q$ are the parameters of the model, $\mu$ is the expectation of $X_t$ (often assumed to equal 0), and $\varepsilon_1$, ..., $\varepsilon_t$ are i.i.d. white noise error terms that are commonly normal random variables.

ARMA model

The notation ARMA(''p'', ''q'') refers to the model with ''p'' autoregressive terms and ''q'' moving-average terms. This model contains the AR(''p'') and MA(''q'') models,

:$X_t = \varepsilon_t + \sum_^p \varphi_i X_ + \sum_^q \theta_i \varepsilon_.\,$

In terms of lag operator

In some texts, the models is specified using the lag operator ''L''. In these terms, the AR(''p'') model is given by

:$\varepsilon_t = \left(1 - \sum_^p \varphi_i L^i\right) X_t = \varphi (L) X_t\,$

where $\varphi$ represents the polynomial

:$\varphi (L) = 1 - \sum_^p \varphi_i L^i.\,$

The MA(''q'') model is given by

:$X_t - \mu = \left(1 + \sum_^q \theta_i L^i\right) \varepsilon_t = \theta (L) \varepsilon_t , \,$

where $\theta$ represents the polynomial

:$\theta(L)= 1 + \sum_^q \theta_i L^i .\,$

Finally, the combined ARMA(''p'', ''q'') model is given by

:$\left(1 - \sum_^p \varphi_i L^i\right) X_t = \left(1 + \sum_^q \theta_i L^i\right) \varepsilon_t \, ,$

or more concisely,

:$\varphi(L) X_t = \theta(L) \varepsilon_t \,$

or

:$\fracX_t = \varepsilon_t \, .$
This is the form used in Box, Jenkins & Reinsel.

Moreover, starting summations from $i=0$ and setting $\phi_0 = -1$ and $\theta_0 = 1$, then we get an even more elegant formulation: $-\sum_^p \phi_i L^i \; X_t = \sum_^q \theta_i L^i \; \varepsilon_t \, .$

Spectrum

The spectral density of an ARMA process is$S(f) = \frac \left\vert \frac \right\vert^2$where $\sigma^2$ is the variance of the white noise, $\theta$ is the characteristic polynomial of the moving average part of the ARMA model, and $\phi$ is the characteristic polynomial of the autoregressive part of the ARMA model.

Fitting models

Choosing ''p'' and ''q''

An appropriate value of ''p'' in the ARMA(''p'', ''q'') model can be found by plotting the partial autocorrelation functions. Similarly, ''q'' can be estimated by using the autocorrelation functions. Both ''p'' and ''q'' can be determined simultaneously using extended autocorrelation functions (EACF). Further information can be gleaned by considering the same functions for the residuals of a model fitted with an initial selection of ''p'' and ''q''.

Brockwell & Davis recommend using Akaike information criterion (AIC) for finding ''p'' and ''q''. Another option is the Bayesian information criterion (BIC).

Estimating coefficients

After choosing ''p'' and ''q,'' ARMA models can be fitted by least squares regression to find the values of the parameters which minimize the error term. It is good practice to find the smallest values of ''p'' and ''q'' which provide an acceptable fit to the data. For a pure AR model, the Yule-Walker equations may be used to provide a fit.

ARMA outputs are used primarily to forecast (predict), and not to infer causation as in other areas of econometrics and regression methods such as OLS and 2SLS.

Software implementations

* In R, standard package stats has function arima, documented i
ARIMA Modelling of Time Series
Packag

has an improved script called sarima for fitting ARMA models (seasonal and nonseasonal) and sarima.sim to simulate data from these models. Extension packages contain related and extended functionality: package tseries includes the function arma(), documented i

packa
fracdiff
contains fracdiff() for fractionally integrated ARMA processes; and packag

includes auto.arima for selecting a parsimonious set of ''p, q''. The CRAN task view o

contains links to most of these.
* Mathematica has a complete library of time series functions including ARMA.
* MATLAB includes functions such a
arma
an

to estimate autoregressive, exogenous autoregressive and ARMAX models. Se

an

for details.
* Julia has community-driven packages that implement fitting with an ARMA model such a
arma.jl

* Python has the statsmodelsS
package which includes many models and functions for time series analysis, including ARMA. Formerly part of the scikit-learn library, it is now stand-alone and integrates well with Pandas.
* PyFlux has a Python-based implementation of ARIMAX models, including Bayesian ARIMAX models.
* IMSL Numerical Libraries are libraries of numerical analysis functionality including ARMA and ARIMA procedures implemented in standard programming languages like C, Java, C# .NET, and Fortran.
* gretl can estimate ARMA models, as mentione
here
* GNU Octave extra packag
octave-forge
supports AR models.
* Stata includes the function arima. for ARMA and ARIMA models.
* SuanShu is a Java library of numerical methods that implements univariate/multivariate ARMA, ARIMA, ARMAX, etc models, documented i
"SuanShu, a Java numerical and statistical library"

* SAS has an econometric package, ETS, that estimates ARIMA models
See details

History and interpretations

The general ARMA model was described in the 1951 thesis of Peter Whittle, who used mathematical analysis (Laurent series and Fourier analysis) and statistical inference. ARMA models were popularized by a 1970 book by George E. P. Box and Jenkins, who expounded an iterative ( Box–Jenkins) method for choosing and estimating them. This method was useful for low-order polynomials (of degree three or less).:

ARMA is essentially an infinite impulse response filter applied to white noise, with some additional interpretation placed on it.

In digital signal processing, ARMA is represented as a digital filter with white noise at the input and the ARMA process at the output.

Applications

ARMA is appropriate when a system is a function of a series of unobserved shocks (the MA or moving average part) as well as its own behavior. For example, stock prices may be shocked by fundamental information as well as exhibiting technical trending and mean-reversion effects due to market participants.

Generalizations

There are various generalizations of ARMA. Nonlinear AR (NAR), nonlinear MA (NMA) and nonlinear ARMA (NARMA) model nonlinear dependence on past values and error terms. Vector AR (VAR) and vector ARMA (VARMA) model multivariate time series. Autoregressive integrated moving average (ARIMA) models non-stationary time series (that is, whose mean changes over time). Autoregressive conditional heteroskedasticity (ARCH) models time series where the variance changes. Seasonal ARIMA (SARIMA or periodic ARMA) models periodic variation. Autoregressive fractionally integrated moving average (ARFIMA, or Fractional ARIMA, FARIMA) model time-series that exhibits long memory. Multiscale AR (MAR) is indexed by the nodes of a tree instead of integers.

Autoregressive–moving-average model with exogenous inputs (ARMAX)

The notation ARMAX(''p'', ''q'', ''b'') refers to a model with ''p'' autoregressive terms, ''q'' moving average terms and ''b'' exogenous inputs terms. The last term is a linear combination of the last ''b'' terms of a known and external time series $d_t$. It is given by:

:$X_t = \varepsilon_t + \sum_^p \varphi_i X_ + \sum_^q \theta_i \varepsilon_ + \sum_^b \eta_i d_.\,$
where $\eta_1, \ldots, \eta_b$ are the ''parameters'' of the exogenous input $d_t$.

Some nonlinear variants of models with exogenous variables have been defined: see for example Nonlinear autoregressive exogenous model.

Statistical packages implement the ARMAX model through the use of "exogenous" (that is, independent) variables. Care must be taken when interpreting the output of those packages, because the estimated parameters usually (for example, in RARIMA Modelling of Time Series
R documentation and gretl) refer to the regression:
: $X_t - m_t = \varepsilon_t + \sum_^p \varphi_i (X_ - m_) + \sum_^q \theta_i \varepsilon_.\,$
where $m_t$ incorporates all exogenous (or independent) variables:
: $m_t = c + \sum_^b \eta_i d_.\,$

See also

* Autoregressive integrated moving average (ARIMA)
* Exponential smoothing
* Linear predictive coding
* Predictive analytics
* Infinite impulse response
* Finite impulse response

References

Further reading

*
*
* .

Shumway, R.H. and Stoffer, D.S. (2017). ''Time Series Analysis and Its Applications with R Examples''. Springer. DOI: 10.1007/978-3-319-52452-8

{{DEFAULTSORT:Autoregressive-Moving-Average Model
Autocorrelation