The Google Ngram Viewer or Google Books Ngram Viewer is an online search engine that charts the frequencies of any set of search strings using a yearly count of

n-gram In the fields of computational linguistics and probability, an ''n''-gram (sometimes also called Q-gram) is a contiguous sequence of ''n'' items from a given sample of text or speech. The items can be phonemes, syllables, letters, words or ...

s found in printed sources published between 1500 and 2019 in

Google Google LLC () is an American multinational technology company focusing on search engine technology, online advertising, cloud computing, computer software, quantum computing, e-commerce, artificial intelligence, and consumer electronic ...

text corpora In linguistics, a corpus (plural ''corpora'') or text corpus is a language resource consisting of a large and structured set of texts (nowadays usually electronically stored and processed). In corpus linguistics, they are used to do statistical ...

in English, Chinese (simplified), French, German, Hebrew, Italian, Russian, or Spanish. There are also some specialized English corpora, such as American English, British English, and English Fiction. The program can search for a word or a phrase, including misspellings or gibberish. The n-grams are matched with the text within the selected corpus, optionally using

case-sensitive In computers, case sensitivity defines whether uppercase and lowercase letters are treated as distinct (case-sensitive) or equivalent (case-insensitive). For instance, when users interested in learning about dogs search an e-book, "dog" and "Dog" a ...

spelling (which compares the exact use of uppercase letters), and, if found in 40 or more books, are then displayed as a graph. The Google Ngram Viewer supports searches for

parts of speech In grammar, a part of speech or part-of-speech (abbreviated as POS or PoS, also known as word class or grammatical category) is a category of words (or, more generally, of lexical items) that have similar grammatical properties. Words that are ass ...

and wildcards. It is routinely used in research.

History

The program was developed by Jon Orwant and Will Brockman and released in mid-December 2010. It was inspired by a prototype called "Bookworm" created by Jean-Baptiste Michel and Erez Aiden from Harvard's Cultural Observatory, Yuan Shen from MIT, and

Steven Pinker Steven Arthur Pinker (born September 18, 1954) is a Canadian-American cognitive psychologist, psycholinguist, popular science author, and public intellectual. He is an advocate of evolutionary psychology and the computational theory of mind. P ...

. The Ngram Viewer was initially based on the 2009 edition of the Google Books Ngram Corpus. , the program supports 2009, 2012, and 2019 corpora.

Operation and restrictions

Commas delimit user-entered search terms, indicating each separate word or phrase to find. The Ngram Viewer returns a plotted

line chart A line chart or line graph or curve chart is a type of chart which displays information as a series of data points called 'markers' connected by straight line segments. It is a basic type of chart common in many fields. It is similar to a ...

within seconds of the user pressing the

Enter key On computer keyboards, the enter key and return key are two closely related keys with overlapping and distinct functions dependent on operating system and application. Functions The return key has its origins in two typewriter functions: ca ...

or the "Search" button on the screen. As an adjustment for more books having been published during some years, the data are normalized, as a relative level, by the number of books published in each year. Due to limitations on the size of the Ngram database, only matches found in at least 40 books are indexed in the database; otherwise, the database could not have stored all possible combinations. Typically, search terms cannot end with punctuation, although a separate

full stop The full stop (Commonwealth English), period ( North American English), or full point , is a punctuation mark. It is used for several purposes, most often to mark the end of a declarative sentence (as distinguished from a question or exclamatio ...

(a period) can be searched. Also, an ending

question mark The question mark (also known as interrogation point, query, or eroteme in journalism) is a punctuation mark that indicates an interrogative clause or phrase in many languages. History In the fifth century, Syriac Bible manuscripts used q ...

(as in "Why?") will cause a second search for the question mark separately. Omitting the periods in abbreviations will allow a form of matching, such as using "R M S" to search for "R.M.S." versus "RMS".

Corpora

The

corpora Corpus is Latin for "body". It may refer to: Linguistics * Text corpus, in linguistics, a large and structured set of texts * Speech corpus, in linguistics, a large set of speech audio files * Corpus linguistics, a branch of linguistics Music * ...

used for the search are composed of total_counts, 1-grams, 2-grams, 3-grams, 4-grams, and 5-grams files for each language. The file format of each of the files is tab-separated data. Each line has the following format: * total_counts file *: year TAB match_count TAB page_count TAB volume_count NEWLINE * Version 1 ngram file (generated in July 2009) *: ngram TAB year TAB match_count TAB page_count TAB volume_count NEWLINE * Version 2 ngram file (generated in July 2012) *: ngram TAB year TAB match_count TAB volume_count NEWLINE The Google Ngram Viewer uses match_count to plot the graph. As an example, a word "Wikipedia" from the Version 2 file of the English 1-grams is stored as follows: The graph plotted by the Google Ngram Viewer using the above data is here:

Limitations

The data set has been criticized for its reliance upon inaccurate OCR, an overabundance of scientific literature, and for including large numbers of incorrectly dated and categorized texts. Because of these errors, and because it is uncontrolled for bias (such as the increasing amount of scientific literature, which causes other terms to appear to decline in popularity), it is risky to use this corpus to study language or test theories. Since the data set does not include

metadata Metadata is " data that provides information about other data", but not the content of the data, such as the text of a message or the image itself. There are many distinct types of metadata, including: * Descriptive metadata – the descriptive ...

, it may not reflect general linguistic or cultural change and can only hint at such an effect. Guidelines for doing research with data from Google Ngram have been proposed that address many of the issues discussed above.

OCR issues

Optical character recognition, or OCR, is not always reliable, and some characters may not be scanned correctly. In particular, systemic errors like the confusion of "s" and "f" in pre-19th century texts (due to the use of the

long s The long s , also known as the medial s or initial s, is an archaic form of the lowercase letter . It replaced the single ''s'', or one or both of the letters ''s'' in a 'double ''s sequence (e.g., "ſinfulneſs" for "sinfulness" and "poſ ...

which was similar in appearance to "f") can cause systemic bias. Although Google Ngram Viewer claims that the results are reliable from 1800 onwards, poor OCR and insufficient data mean that frequencies given for languages such as Chinese may only be accurate from 1970 onward, with earlier parts of the corpus showing no results at all for common terms, and data for some years containing more than 50% noise.When n-grams go bad
digitalsinology.org.

References

Bibliography

External links

* {{Google Inc. Ngram Viewer Ngram Viewer 2010 software Natural language processing Computational linguistics Language modeling Speech recognition Corpus linguistics Probabilistic models