The mean reciprocal rank is a
statistic
A statistic (singular) or sample statistic is any quantity computed from values in a sample which is considered for a statistical purpose. Statistical purposes include estimating a population parameter, describing a sample, or evaluating a hy ...
measure for evaluating any process that produces a list of possible responses to a sample of queries, ordered by probability of correctness. The reciprocal rank of a query response is the
multiplicative inverse
In mathematics, a multiplicative inverse or reciprocal for a number ''x'', denoted by 1/''x'' or ''x''−1, is a number which when multiplied by ''x'' yields the multiplicative identity, 1. The multiplicative inverse of a fraction ''a''/''b ...
of the rank of the first correct answer: 1 for first place, for second place, for third place and so on. The mean reciprocal rank is the average of the reciprocal ranks of results for a sample of queries Q:
:
where
refers to the rank position of the ''first'' relevant document for the ''i''-th query.
The reciprocal value of the mean reciprocal rank corresponds to the
harmonic mean
In mathematics, the harmonic mean is one of several kinds of average, and in particular, one of the Pythagorean means. It is sometimes appropriate for situations when the average rate is desired.
The harmonic mean can be expressed as the recipr ...
of the ranks.
Example
For example, suppose we have the following three sample queries for a system that tries to translate English words to their plurals. In each case, the system makes three guesses, with the first one being the one it thinks is most likely correct:
Given those three samples, we could calculate the mean reciprocal rank as (1/3 + 1/2 + 1)/3 = 11/18 or about 0.61.
If none of the proposed results are correct, reciprocal rank is 0.
Note that only the rank of the first relevant answer is considered, possible further relevant answers are ignored. If users are interested also in further relevant items,
mean average precision is a potential alternative metric.
See also
*
Information retrieval
*
Question answering
References
{{Machine learning evaluation metrics
Summary statistics
Information retrieval evaluation