Statistically improbable phrase

A statistically improbable phrase (SIP) is a phrase or set of words that occurs more frequently in a document (or collection of documents) than in some larger corpus. Amazon.com uses this concept in determining keywords for a given book or chapter, since keywords of a book or chapter are likely to appear disproportionately within that section. Christian Rudder has also used this concept with data from online dating profiles and Twitter posts to determine the phrases most characteristic of a given race or gender in his book ''Dataclysm''. SIPs with a linguistic density of two or three words, adjective, adjective, noun or adverb, adverb, verb, will signal the author's attitude, premise or conclusions to the reader or express an important idea.

Another use of SIPs is as a detection tool for plagiarism. (Almost) unique combinations of words can be searched for online, and if they have appeared in a published text, the search will identify where. This method only checks those texts that have been published and that have been digitized online. garden style, praising irregularity in design.

For example, a submission by, say, a student that contained the phrase "garden style, praising irregularity in design", might be searched for using Google.com and will yield the original Wikipedia article about Sir William Temple, English political figure and essayist.

Example

In a document about computers, the most common word is likely to be the word "the", but since "the" is the most commonly used word in the English language, it is probable that any given document will have the word "the" used very frequently. However, a phrase like "explicit Boolean algorithm" might occur in the document at a much higher rate than its average rate in the English language. Hence, it is a phrase unlikely to occur in any given document, but ''did'' occur in the document given. "Explicit Boolean algorithm" would be a statistically improbable phrase.

Statistically improbable phrases of Darwin's ''On the Origin of Species'' could be: ''temperate productions, genera descended, transitional gradations, unknown progenitor, fossiliferous formations, our domestic breeds, modified offspring, doubtful forms, closely allied forms, profitable variations, enormously remote, transitional grades, very distinct species'' and ''mongrel offspring''.Sociologically Improbable Phrases
Crooked Timber April 2005

See also

* Collocation – Any series of words that co-occur more often than would be expected by chance
* Googlewhack – A pair of words occurring on a single webpage, as indexed by Google
* tf-idf – A statistic used in information retrieval and text mining

References

{{comp-ling-stub
Amazon (company)
Bookselling
Information retrieval systems
Computational linguistics