data mining Data mining is the process of extracting and finding patterns in massive data sets involving methods at the intersection of machine learning, statistics, and database systems. Data mining is an interdisciplinary subfield of computer science and ...

, cluster-weighted modeling (CWM) is an algorithm-based approach to non-linear prediction of outputs ( dependent variables) from inputs (

independent variables A variable is considered dependent if it depends on (or is hypothesized to depend on) an independent variable. Dependent variables are studied under the supposition or demand that they depend, by some law or rule (e.g., by a mathematical function ...

) based on

density estimation In statistics, probability density estimation or simply density estimation is the construction of an estimate, based on observed data, of an unobservable underlying probability density function. The unobservable density function is thought o ...

using a set of models (clusters) that are each notionally appropriate in a sub-region of the input space. The overall approach works in jointly input-output space and an initial version was proposed by Neil Gershenfeld.

Basic form of model

The procedure for cluster-weighted modeling of an input-output problem can be outlined as follows. In order to construct predicted values for an output variable ''y'' from an input variable ''x'', the modeling and calibration procedure arrives at a

joint probability density function In probability theory, a probability density function (PDF), density function, or density of an absolutely continuous random variable, is a function whose value at any given sample (or point) in the sample space (the set of possible values ...

, ''p''(''y'',''x''). Here the "variables" might be uni-variate, multivariate or time-series. For convenience, any model parameters are not indicated in the notation here and several different treatments of these are possible, including setting them to fixed values as a step in the calibration or treating them using a

Bayesian analysis Thomas Bayes ( ; c. 1701 – 1761) was an English statistician, philosopher, and Presbyterian Presbyterianism is a historically Reformed Protestant tradition named for its form of church government by representative assemblies of elde ...

. The required predicted values are obtained by constructing the conditional probability density ''p''(''y'', ''x'') from which the prediction using the

conditional expected value In probability theory, the conditional expectation, conditional expected value, or conditional mean of a random variable is its expected value evaluated with respect to the conditional probability distribution. If the random variable can take on ...

can be obtained, with the

conditional variance In probability theory and statistics, a conditional variance is the variance of a random variable given the value(s) of one or more other variables. Particularly in econometrics, the conditional variance is also known as the scedastic function or s ...

providing an indication of uncertainty. The important step of the modeling is that ''p''(''y'', ''x'') is assumed to take the following form, as a

mixture model In statistics, a mixture model is a probabilistic model for representing the presence of subpopulations within an overall population, without requiring that an observed data set should identify the sub-population to which an individual observati ...

: :

p(y,x)=\sum_^n w_jp_j(y,x),

where ''n'' is the number of clusters and are weights that sum to one. The functions ''p_j''(''y'',''x'') are joint probability density functions that relate to each of the ''n'' clusters. These functions are modeled using a decomposition into a conditional and a marginal density: :

p_j(y,x)=p_j(y, x)p_j(x),

where: :*''p_j''(''y'', ''x'') is a model for predicting ''y'' given ''x'', and given that the input-output pair should be associated with cluster ''j'' on the basis of the value of ''x''. This model might be a regression model in the simplest cases. :*''p_j''(''x'') is formally a density for values of ''x'', given that the input-output pair should be associated with cluster ''j''. The relative sizes of these functions between the clusters determines whether a particular value of ''x'' is associated with any given cluster-center. This density might be a

Gaussian function In mathematics, a Gaussian function, often simply referred to as a Gaussian, is a function (mathematics), function of the base form f(x) = \exp (-x^2) and with parametric extension f(x) = a \exp\left( -\frac \right) for arbitrary real number, rea ...

centered at a parameter representing the cluster-center. In the same way as for regression analysis, it will be important to consider preliminary

data transformation In computing, data transformation is the process of converting data from one format or structure into another format or structure. It is a fundamental aspect of most data integrationCIO.com. Agile Comes to Data Integration. Retrieved from: https ...

s as part of the overall modeling strategy if the core components of the model are to be simple regression models for the cluster-wise condition densities, and

normal distribution In probability theory and statistics, a normal distribution or Gaussian distribution is a type of continuous probability distribution for a real-valued random variable. The general form of its probability density function is f(x) = \frac ...

s for the cluster-weighting densities ''p_j''(''x'').

General versions

The basic CWM algorithm gives a single output cluster for each input cluster. However, CWM can be extended to multiple clusters which are still associated with the same input cluster. Each cluster in CWM is localized to a Gaussian input region, and this contains its own trainable local model. It is recognized as a versatile inference algorithm which provides simplicity, generality, and flexibility; even when a feedforward layered network might be preferred, it is sometimes used as a "second opinion" on the nature of the training problem. The original form proposed by Gershenfeld describes two innovations: * Enabling CWM to work with continuous streams of data * Addressing the problem of local minima encountered by the CWM parameter adjustment process CWM can be used to classify media in printer applications, using at least two parameters to generate an output that has a joint dependency on the input parameters.

References

{{Reflist Multivariate statistics Cluster analysis algorithms Estimation of densities