Letter frequency is the number of times

letters Letter, letters, or literature may refer to: Characters typeface * Letter (alphabet), a character representing one or more of the sounds used in speech or none in the case of a silent letter; any of the symbols of an alphabet * Letterform, the g ...

of the

alphabet An alphabet is a standard set of letter (alphabet), letters written to represent particular sounds in a spoken language. Specifically, letters largely correspond to phonemes as the smallest sound segments that can distinguish one word from a ...

appear on average in

written language A written language is the representation of a language by means of writing. This involves the use of visual symbols, known as graphemes, to represent linguistic units such as phonemes, syllables, morphemes, or words. However, written language is ...

. Letter

frequency analysis In cryptanalysis, frequency analysis (also known as counting letters) is the study of the frequency of letters or groups of letters in a ciphertext. The method is used as an aid to breaking classical ciphers. Frequency analysis is based on th ...

dates back to the

Arab Arabs (, , ; , , ) are an ethnic group mainly inhabiting the Arab world in West Asia and North Africa. A significant Arab diaspora is present in various parts of the world. Arabs have been in the Fertile Crescent for thousands of years ...

mathematician

Al-Kindi Abū Yūsuf Yaʻqūb ibn ʼIsḥāq aṣ-Ṣabbāḥ al-Kindī (; ; ; ) was an Arab Muslim polymath active as a philosopher, mathematician, physician, and music theorist Music theory is the study of theoretical frameworks for understandin ...

(c. AD 801–873), who formally developed the method to break

ciphers In cryptography, a cipher (or cypher) is an algorithm for performing encryption or decryption—a series of well-defined steps that can be followed as a procedure. An alternative, less common term is ''encipherment''. To encipher or encode i ...

. Letter frequency analysis gained importance in

Europe Europe is a continent located entirely in the Northern Hemisphere and mostly in the Eastern Hemisphere. It is bordered by the Arctic Ocean to the north, the Atlantic Ocean to the west, the Mediterranean Sea to the south, and Asia to the east ...

with the development of

movable type Movable type (US English; moveable type in British English) is the system and technology of printing and typography that uses movable Sort (typesetting), components to reproduce the elements of a document (usually individual alphanumeric charac ...

in AD 1450, wherein one must estimate the amount of type required for each

letterform A letterform, letter-form or letter form is a term used especially in typography, palaeography, calligraphy and epigraphy to mean a letter (alphabet), letter's shape. A letterform is a type of glyph, which is a specific, concrete way of writing a ...

. Linguists use letter frequency analysis as a rudimentary technique for

language identification In natural language processing, language identification or language guessing is the problem of determining which natural language given content is in. Computational approaches to this problem view it as a special case of text categorization, sol ...

, where it is particularly effective as an indication of whether an unknown

writing system A writing system comprises a set of symbols, called a ''script'', as well as the rules by which the script represents a particular language. The earliest writing appeared during the late 4th millennium BC. Throughout history, each independen ...

is alphabetic,

syllabic A syllable is a basic unit of organization within a sequence of speech sounds, such as within a word, typically defined by linguists as a ''nucleus'' (most often a vowel) with optional sounds before or after that nucleus (''margins'', which are ...

, or

ideographic An ideogram or ideograph (from Greek 'idea' + 'to write') is a symbol that is used within a given writing system to represent an idea or concept in a given language. (Ideograms are contrasted with phonograms, which indicate sounds of speech ...

. The use of letter frequencies and

plays a fundamental role in

cryptograms A cryptogram is a type of puzzle that consists of a short piece of encryption, encrypted text. Generally the cipher used to encrypt the text is simple enough that the cryptogram can be solved by hand. Substitution ciphers where each letter is ...

and several word puzzle games, including hangman, ''

Scrabble ''Scrabble'' is a word game in which two to four players score points by placing tiles, each bearing a single letter, onto a Board game, game board divided into a 15×15 grid of squares. The tiles must form words that, in crossword fashion, re ...

'', ''

Wordle ''Wordle'' is a web-based word game created and developed by the Welsh software engineer Josh Wardle. In the game, players have six attempts to guess a five-letter word, receiving feedback through colored tiles that indicate correct letters a ...

'' and the television game show '' Wheel of Fortune''. One of the earliest descriptions in classical literature of applying the knowledge of English letter frequency to solving a cryptogram is found in

Edgar Allan Poe Edgar Allan Poe (; January 19, 1809 – October 7, 1849) was an American writer, poet, editor, and literary critic who is best known for his poetry and short stories, particularly his tales involving mystery and the macabre. He is widely re ...

's famous story "

The Gold-Bug "The Gold-Bug" is a short story by American writer Edgar Allan Poe published in 1843. The plot follows William Legrand, who becomes fixated on an unusual gold-colored bug he has discovered. His servant Jupiter fears that Legrand is going insan ...

", where the method is successfully applied to decipher a message giving the location of a treasure hidden by

Captain Kidd William Kidd (c. 1645 – 23 May 1701), also known as Captain William Kidd or simply Captain Kidd, was a Scottish-American privateer. Conflicting accounts exist regarding his early life, but he was likely born in Dundee and later settled in Ne ...

. Herbert S. Zim, in his classic introductory cryptography text ''Codes and Secret Writing'', gives the English letter frequency sequence as "ETAON RISHD LFCMU GYPWB VKJXZQ", the most common letter pairs as "TH HE AN RE ER IN ON AT ND ST ES EN OF TE ED OR TI HI AS TO", and the most common doubled letters as "LL EE SS OO TT FF RR NN PP CC". Different ways of counting can produce somewhat different orders. Letter frequencies also have a strong effect on the design of some

keyboard layout A keyboard layout is any specific physical, visual, or functional arrangement of the keys, legends, or key-meaning associations (respectively) of a computer keyboard, mobile phone, or other computer-controlled typographic keyboard. Standard keybo ...

s. The most frequent letters are placed on the

home row Touch typing (also called blind typing, or touch keyboarding) is a style of typing. Although the phrase refers to typing without using the sense of sight to find the keys—specifically, a touch typist will know their location on the keyboard thr ...

of the

Blickensderfer typewriter The Blickensderfer typewriter was invented by George Canfield Blickensderfer (1850–1917) and patented on April 12, 1892. Blickensderfer was a nephew of John Celivergos Zachos, the inventor of the stenotype. Two models, Model 1 and Model 5 ...

, the

Dvorak keyboard layout Dvorak () is a keyboard layout for English patented in 1936 by August Dvorak and his brother-in-law, William Dealey, as a faster and more ergonomic alternative to the QWERTY layout (the ''de facto'' standard keyboard layout). Dvorak proponen ...

Colemak Colemak is a keyboard layout A keyboard layout is any specific physical, visual, or functional arrangement of the keys, legends, or key-meaning associations (respectively) of a computer keyboard, mobile phone, or other computer-controlled typog ...

and other optimized layouts.

Background

The frequency of letters in text has been studied for use in

cryptanalysis Cryptanalysis (from the Greek ''kryptós'', "hidden", and ''analýein'', "to analyze") refers to the process of analyzing information systems in order to understand hidden aspects of the systems. Cryptanalysis is used to breach cryptographic se ...

, and

in particular, dating back to the Arab mathematician

al-Kindi Abū Yūsuf Yaʻqūb ibn ʼIsḥāq aṣ-Ṣabbāḥ al-Kindī (; ; ; ) was an Arab Muslim polymath active as a philosopher, mathematician, physician, and music theorist Music theory is the study of theoretical frameworks for understandin ...

(c. AD 801–873 ), who formally developed the method (the ciphers breakable by this technique go back at least to the

Caesar cipher In cryptography, a Caesar cipher, also known as Caesar's cipher, the shift cipher, Caesar's code, or Caesar shift, is one of the simplest and most widely known encryption techniques. It is a type of substitution cipher in which each letter in t ...

used by

Julius Caesar Gaius Julius Caesar (12 or 13 July 100 BC – 15 March 44 BC) was a Roman general and statesman. A member of the First Triumvirate, Caesar led the Roman armies in the Gallic Wars before defeating his political rival Pompey in Caesar's civil wa ...

, so this method could have been explored in classical times). Letter frequency analysis gained additional importance in Europe with the development of movable type in AD 1450, wherein one must estimate the amount of type required for each letterform, as evidenced by the variations in letter compartment size in typographer's type cases. No exact letter frequency distribution underlies a given language, since all writers write slightly differently. However, most languages have a characteristic distribution which is strongly apparent in longer texts. Even language changes as extreme as from

Old English Old English ( or , or ), or Anglo-Saxon, is the earliest recorded form of the English language, spoken in England and southern and eastern Scotland in the Early Middle Ages. It developed from the languages brought to Great Britain by Anglo-S ...

to modern English (regarded as mutually unintelligible) show strong trends in related letter frequencies: over a small sample of Biblical passages, from most frequent to least frequent, of Old English compares to of modern English, with the most extreme differences concerning letterforms not shared.

Linotype machine The Linotype machine ( ) is a "line casting" machine used in printing which is manufactured and sold by the former Mergenthaler Linotype Company and related It was a hot metal typesetting system that cast lines of metal type for one-time use. Li ...

s for the English language assumed the letter order, from most to least common, to be based on the experience and custom of manual compositors. The equivalent for the French language was . Arranging the alphabet in Morse into groups of letters that require equal amounts of time to transmit, and then sorting these groups in increasing order, yields .

American Morse code American Morse Code — also known as Railroad Morse — is the latter-day name for the original version of the Morse Code, developed in the mid-1840s by Samuel Morse and Alfred Vail for their electric telegraph. The "American" qualifier was add ...

was developed in the 1830s by

Alfred Vail Alfred Lewis Vail (September 25, 1807 – January 18, 1859) was an American machinist and inventor. Along with Samuel Morse, Vail was central in developing and commercializing American electrical telegraphy between 1837 and 1844. Vail and Morse ...

, based on English-language letter frequencies, to encode the most frequent letters with the shortest symbols. Some efficiency was lost in the reformed version now used: the International Morse Code. Letter frequency was used by other telegraph systems, such as the

Murray Code The Baudot code () is an early character encoding for telegraphy invented by Émile Baudot in the 1870s. It was the predecessor to the International Telegraph Alphabet No. 2 (ITA2), the most common teleprinter code in use before ASCII. Each char ...

. Similar ideas are used in modern data-compression techniques such as

Huffman coding In computer science and information theory, a Huffman code is a particular type of optimal prefix code that is commonly used for lossless data compression. The process of finding or using such a code is Huffman coding, an algorithm developed by ...

. Letter frequencies, like

word frequencies A word list is a list of words in a lexicon, generally sorted by frequency of occurrence (either by graded levels, or as a ranked list). A word list is compiled by lexical frequency analysis within a given text corpus, and is used in corpus li ...

, tend to vary, both by writer and by subject. For instance, occurs with greater frequency in fiction, as most fiction is written in past tense and thus most verbs will end in the inflectional suffix -ed / -d. One cannot write an essay about x-rays without using frequently. Different authors have habits which can be reflected in their use of letters.

Hemingway Ernest Miller Hemingway ( ; July 21, 1899 – July 2, 1961) was an American novelist, short-story writer and journalist. Known for an economical, understated style that influenced later 20th-century writers, he has been romanticized f ...

's writing style, for example, is visibly different from

Faulkner William Cuthbert Faulkner (; September 25, 1897 – July 6, 1962) was an American writer. He is best known for his novels and short stories set in the fictional Yoknapatawpha County, Mississippi, a stand-in for Lafayette County where he s ...

's. Letter,

bigram A bigram or digram is a sequence of two adjacent elements from a string of tokens, which are typically letters, syllables, or words. A bigram is an ''n''-gram for ''n''=2. The frequency distribution of every bigram in a string is commonly used f ...

trigram Trigrams are a special case of the ''n''-gram, where ''n'' is 3. They are often used in natural language processing for performing statistical analysis of texts and in cryptography for control and use of ciphers and codes. See results of analysi ...

, word frequencies, word length, and sentence length can be calculated for specific authors and used to prove or disprove authorship of texts, even for authors whose styles are not so divergent. Accurate average letter frequencies can only be gleaned by analyzing a large amount of representative text. With the availability of modern computing and collections of large

text corpora In linguistics and natural language processing, a corpus (: corpora) or text corpus is a dataset, consisting of natively digital and older, digitalized, language resources, either annotated or unannotated. Annotated, they have been used in cor ...

, such calculations are easily made. Examples can be drawn from a variety of sources (press reporting, religious texts, scientific texts and general fiction) and there are differences especially for general fiction with the position of and , with becoming more common. Different dialects of a language will also affect a letter's frequency. For example, an author in the United States would produce something in which is more common than an author in the United Kingdom writing on the same topic: words like "analyze", "apologize", and "recognize" contain the letter in American English, whereas the same words are spelled "analyse", "apologise", and "recognise" in British English. This would highly affect the frequency of the letter , as it is rarely used by British writers in the English language. The "top twelve" letters constitute about 80% of the total usage. The "top eight" letters constitute about 65% of the total usage. Letter frequency as a function of rank can be fitted well by several rank functions, with the two-parameter Cocho/Beta rank function being the best. Another rank function with no adjustable free parameter also fits the letter frequency distribution reasonably well (the same function has been used to fit the amino acid frequency in protein sequences.) A spy using the

VIC cipher The VIC cipher was a pencil and paper cipher used by the Soviet Union, Soviet spy Reino Häyhänen, codenamed "VICTOR". If the cipher were to be given a modern technical name, it would be known as a "straddling bipartite monoalphabetic substitut ...

or some other cipher based on a

straddling checkerboard A straddling checkerboard is a device for converting an alphanumeric plaintext into digits whilst simultaneously achieving fractionation (a simple form of information diffusion) and data compression relative to other schemes using digits. It als ...

typically uses a mnemonic such as "a sin to err" (dropping the second "r") or "at one sir" to remember the top eight characters.

Relative frequencies of letters in the English language

There are three ways to count letter frequency that result in very different charts for common letters. The first method, used in the chart below, is to count letter frequency in

lemmas Lemma (from Ancient Greek ''premise'', ''assumption'', from Greek ''I take'', ''I get'') may refer to: Language and linguistics * Lemma (morphology), the canonical, dictionary or citation form of a word * Lemma (psycholinguistics), a mental a ...

of a dictionary. The lemma is the word in its canonical form. The second method is to include all word variants when counting, such as "abstracts", "abstracted" and "abstracting" and not just the lemma of "abstract". This second method results in letters like appearing much more frequently, such as when counting letters from lists of the most used English words on the Internet. is especially common in inflected words (non-lemma forms) because it is added to form plurals and third person singular present tense verbs. A final method is to count letters based on their frequency of use in actual texts, resulting in certain letter combinations like becoming more common due to the frequent use of common words like "the", "then", "both", "this", etc. Absolute usage frequency measures like this are used when creating keyboard layouts or letter frequencies in old fashioned printing presses. An analysis of entries in the Concise Oxford dictionary, ignoring frequency of word use, gives an order of "EARIOTNSLCUDPMHGBFYWKVXZJQ". The letter-frequency table above is taken from Pavel Mička's website, which cites Robert Lewand's ''Cryptological Mathematics''. According to Lewand, arranged from most to least common in appearance, the letters are: etaoinshrdlcumwfgypbvkjxqz. Lewand's ordering differs slightly from others, such as Cornell University Math Explorer's Project, which produced a table after measuring 40,000 words. In English, the space character occurs almost twice as frequently as the top letter () and the non-alphabetic characters (digits, punctuation, etc.) collectively occupy the fourth position (having already included the space) between and .

Relative frequencies of the first letters of a word in English language

The frequency of the first letters of words or names is helpful in pre-assigning space in physical files and indexes. Given 26

filing cabinet A filing cabinet (or sometimes file cabinet in American English) is a piece of office furniture for storing paper documents in file folders. In the most simple context, it is an enclosure for drawer (furniture), drawers in which articles are sto ...

drawers, rather than a 1:1 assignment of one drawer to one letter of the alphabet, it is often useful to use a more equal-frequency-letter code by assigning several low-frequency letters to the same drawer (often one drawer is labeled VWXYZ), and to split up the most-frequent initial letters () into several drawers (often 6 drawers Aa-An, Ao-Az, Ca-Cj, Ck-Cz, Sa-Si, Sj-Sz). The same system is used in some multi-volume works such as some

encyclopedia An encyclopedia is a reference work or compendium providing summaries of knowledge, either general or special, in a particular field or discipline. Encyclopedias are divided into article (publishing), articles or entries that are arranged Alp ...

s. Cutter numbers, another mapping of names to a more equal-frequency code, are used in some libraries. Both the overall letter distribution and the word-initial letter distribution approximately match the

Zipf distribution Zipf's law (; ) is an empirical law stating that when a list of measured values is sorted in decreasing order, the value of the -th entry is often approximately inversely proportional to . The best known instance of Zipf's law applies to the ...

and even more closely match the Yule distribution. Often the frequency distribution of the first digit in each datum is significantly different from the overall frequency of all the digits in a set of numeric data, an observation known as Benford's law. An analysis by

Peter Norvig Peter Norvig (born 14 December 1956) is an American computer scientist and Distinguished Education Fellow at the Stanford Institute for Human-Centered AI. He previously served as a director of research and search quality at Google. Norvig is th ...

on words that appear 100,000 times or more in Google Books data transcribed using

optical character recognition Optical character recognition or optical character reader (OCR) is the electronics, electronic or machine, mechanical conversion of images of typed, handwritten or printed text into machine-encoded text, whether from a scanned document, a photo ...

(OCR) determined the frequency of first letters of English words, among other things.

Relative frequencies of letters in other languages

*See İ and

dotless I I, or ı, called dotless i, is a letter used in the Latin-script alphabets of Azerbaijani, Crimean Tatar, Gagauz, Kazakh, Tatar and Turkish. It commonly represents the close back unrounded vowel , except in Kazakh where it represents the ...

. The figure below illustrates the frequency distributions of the 26 most common Latin letters across some languages. All of these languages use a similar 25+ character alphabet. Based on these tables, the ' etaoin shrdlu' equivalent for each language is as follows: *French: ''; (Indo-European: Romance; traditionally, 'esartinulop' is used, in part for its ease of pronunciationPerec, Georges; ''Alphabets''; Éditions Galilée, 1976) *Spanish: ''; (Indo-European: Romance) *Portuguese: '' (Indo-European: Romance) *Italian: ''; (Indo-European: Romance) *German: ''; (Indo-European: Germanic) *Swedish: ''; (Indo-European: Germanic) *Turkish: ''; (Turkic: Oghuz) *Dutch: ''; (Indo-European: Germanic) *Polish: ''; (Indo-European: Slavic) *Danish: ''; (Indo-European: Germanic) *Icelandic: ''; (Indo-European: Germanic) *Finnish: ''; (Uralic: Finnic) *Czech: ''; (Indo-European: Slavic) *Hungarian: ''; (Uralic: Ugric) *Welsh: '; (Indo-European: Celtic)

Explanatory notes

References

External links

Letter Frequencies
* * * *
Letter frequency
��simia.net

Useful tables

Useful tables for single letter, digram, trigram, tetragram, and pentagram frequencies based on 20,000 words that take into account word-length and letter-position combinations for words 3 to 7 letters in length: * * * * {{cite journal , last1=Mayzner , first1=M.S. , last2=Tresselt , first2=M.E. , last3=Wolin , first3=B.R. , title=Tables of pentagram frequency counts for various word-length and letter-position combinations , journal=Psychonomic Monograph Supplements , volume=1 , issue=5 , pages=144–190 , year=1965 Cryptography Frequency distribution Grammatology Phonology Quantitative linguistics