Hadamard Product (matrices)

In mathematics, the Hadamard product (also known as the element-wise product, entrywise product or Schur product) is a binary operation that takes two matrices of the same dimensions and produces another matrix of the same dimension as the operands, where each element is the product of elements of the original two matrices. It is to be distinguished from the more common matrix product. It is attributed to, and named after, either French mathematician Jacques Hadamard or German Russian mathematician Issai Schur.

The Hadamard product is associative and distributive. Unlike the matrix product, it is also commutative.

Definition

For two matrices and of the same dimension , the Hadamard product $A \circ B$ (or $A \odot B$) is a matrix of the same dimension as the operands, with elements given by
:$(A \circ B)_ = (A \odot B)_ = (A)_ (B)_.$

For matrices of different dimensions ( and , where or ), the Hadamard product is undefined.

Example

For example, the Hadamard product for a 3 × 4 matrix with a 3 × 4 matrix is
:$\begin a_ & a_ & a_ & a_\\ a_ & a_ & a_ & a_\\ a_ & a_ & a_ & a_ \end \circ \begin b_ & b_ & b_ & b_\\ b_ & b_ & b_ & b_\\ b_ & b_ & b_ & b_ \end = \begin a_\, b_ & a_\, b_ & a_\, b_ & a_\, b_\\ a_\, b_ & a_\, b_ & a_\, b_ & a_\, b_\\ a_\, b_ & a_\, b_ & a_\, b_ & a_\, b_ \end.$

Properties

* The Hadamard product is commutative (when working with a commutative ring), associative and distributive over addition. That is, if ''A'', ''B'', and ''C'' are matrices of the same size, and ''k'' is a scalar: $\begin A \circ B &= B \circ A, \\ A \circ (B \circ C) &= (A \circ B) \circ C, \\ A \circ (B + C) &= A \circ B + A \circ C, \\ \left(kA\right) \circ B &= A \circ \left(kB\right) = k\left(A \circ B\right), \\ A \circ 0 &= 0 \circ A = 0. \end$
* The identity matrix under Hadamard multiplication of two matrices is an matrix where all elements are equal to 1. This is different from the identity matrix under regular matrix multiplication, where only the elements of the main diagonal are equal to 1. Furthermore, a matrix has an inverse under Hadamard multiplication if and only if none of the elements are equal to zero.
* For vectors and , and corresponding diagonal matrices and with these vectors as their main diagonals, the following identity holds: $\mathbf^* (A \circ B)\mathbf = \operatorname\left(_\mathbf^* A _\mathbf ^\mathsf\right),$ where denotes the conjugate transpose of . In particular, using vectors of ones, this shows that the sum of all elements in the Hadamard product is the trace of where superscript T denotes the matrix transpose. A related result for square and , is that the row-sums of their Hadamard product are the diagonal elements of : $\sum_i (A \circ B)_ = \left(B^\mathsf A\right)_ = \left(AB^\mathsf\right)_.$ Similarly, $\left(\mathbf\mathbf^*\right) \circ A = _\mathbf A _\mathbf^*$ Furthermore a Hadamard matrix-vector product can be expressed as: $(A \circ B) \mathbf = \operatorname\left( A D_\mathbf B^\mathsf \right)$ where $\operatorname(M)$ is the vector formed from the diagonals of matrix .
*The Hadamard product is a principal submatrix of the Kronecker product.
* The Hadamard product satisfies the rank inequality $\operatorname(A \circ B) \leq \operatorname(A) \operatorname(B)$
* If and are positive-definite matrices, then the following inequality involving the Hadamard product holds: $\prod_^n \lambda_i(A \circ B) \ge \prod_^n \lambda_i(A B),\quad k=1,\ldots,n,$ where is the th largest eigenvalue of .
* If and are diagonal matrices, then $\begin D (A \circ B) E &= (D A E) \circ B = (D A) \circ (B E) \\ &= (AE) \circ (D B) = A \circ (D B E). \end$
* The Hadamard product of two vectors $\mathbf a$ and $\mathbf b$ is the same as matrix multiplication of one vector by the corresponding diagonal matrix of the other vector: $\mathbf a \circ \mathbf b = D _ \mathbf b = D _ \mathbf a$
* The vector to diagonal matrix $\operatorname$ operator may be expressed using the Hadamard product as: $\operatorname(\mathbf) = (\mathbf \mathbf^T) \circ I$ where $\mathbf$ is a constant vector with elements $1$ and $I$ is the identity matrix.

The mixed-product property

$(A \otimes B) \circ (C \otimes D) = (A \circ C) \otimes (B \circ D) ,$ where $\otimes$ is Kronecker product, assuming $A$ has the same dimensions of $C$ and $B$ with $D$.

$(A \bull B) \circ (C \bull D) = (A \circ C) \bull (B \circ D) ,$ where $\bull$ denotes face-splitting product.

$(A \bull B)(C \ast D) = (A C) \circ (B D),$ where $\ast$ is column-wise Khatri–Rao product.

Schur product theorem

The Hadamard product of two positive-semidefinite matrices is positive-semidefinite. This is known as the Schur product theorem, after Russian mathematician Issai Schur. For two positive-semidefinite matrices and , it is also known that the determinant of their Hadamard product is greater than or equal to the product of their respective determinants:
$\det( \circ ) \ge \det() \det().$

In programming languages

Hadamard multiplication is built into certain programming languages under various names. In MATLAB, GNU Octave, GAUSS and HP Prime, it is known as ''array multiplication'', or in Julia ''broadcast multiplication'', with the symbol .*. In Fortran, R, APL, J and Wolfram Language (Mathematica), it is done through simple multiplication operator *, whereas the matrix product is done through the function matmul, %*%, +.×, +/ .* and the . operators, respectively.
In Python with the NumPy numerical library, multiplication of objects as a*b produces the Hadamard product, and multiplication as a@b produces the matrix product. With the SymPy symbolic library, multiplication of objects as both a*b and a@b will produce the matrix product, the Hadamard product can be obtained with a.multiply_elementwise(b).
In C++, the Eigen library provides a cwiseProduct member function for the class (a.cwiseProduct(b)), while the Armadillo library uses the operator % to make compact expressions (a % b; a * b is a matrix product). R packag
matrixcalc
introduces function hadamard.prod() for Hadamard Product of numeric matrices or vectors.

Applications

The Hadamard product appears in lossy compression algorithms such as JPEG. The decoding step involves an entry-for-entry product, in other words the Hadamard product.

In image processing, the Hadamard operator can be used for enhancing, suppressing or masking image regions. One matrix represents the original image, the other acts as weight or masking matrix.

It is used in the machine learning literature, for example to describe the architecture of recurrent neural networks as GRUs or LSTMs.

It is also used to study statistical properties of random vectors and matrices.

Analogous operations

Other Hadamard operations are also seen in the mathematical literature, namely the ' and ' (which are in effect the same thing because of fractional indices), defined for a matrix such that:

For
$\begin &= ^ \\ B_ &= ^2 \end$

and for
$\begin &= ^ \\ B_ &= ^\frac12 \end$

The ' reads:
$\begin &= ^ \\ B_ &= ^ \end$

A ' is defined as:

$\begin &= \oslash \\ C_ &= \frac \end$

The penetrating face product

According to the definition of V. Slyusar the penetrating face product of the ''p''×''g'' matrix $$ and ''n''-dimensional matrix $$ (''n'' > 1) with ''p''×''g'' blocks ( = _n) is a matrix of size $$ of the form:

circ =
\left begin
\circ _1 & \circ _2 & \cdots & \circ _n
\end\right

Example

If
=
\begin
1 & 2 & 3 \\
4 & 5 & 6 \\
7 & 8 & 9
\end,\quad
=
\left begin
_1 & _2 & _3
\end\right=
\left begin
1 & 4 & 7 & 2 & 8 & 14 & 3 & 12 & 21 \\
8 & 20 & 5 & 10 & 25 & 40 & 12 & 30 & 6 \\
2 & 8 & 3 & 2 & 4 & 2 & 7 & 3 & 9
\end\right

then

circ =
\left begin
1 & 8 & 21 & 2 & 16 & 42 & 3 & 24 & 63 \\
32 & 100 & 30 & 40 & 125 & 240 & 48 & 150 & 36 \\
14 & 64 & 27 & 14 & 32 & 18 & 49 & 24 & 81
\end\right

Main properties

: circ = circ;
: \bull = circ\left( \otimes \mathbf ^\textsf\right),

where $\bull$ denotes the face-splitting product of matrices,
:\mathbf \bull = \mathbf circ, where $\mathbf$ is a vector.

Applications

The penetrating face product is used in the tensor-matrix theory of digital antenna arrays. This operation can also be used in artificial neural network models, specifically convolutional layers.

See also

* Frobenius inner product
* Pointwise product
* Kronecker product
* Khatri–Rao product

References

{{Linear algebra

Matrix theory