linguistics Linguistics is the scientific study of language. The areas of linguistic analysis are syntax (rules governing the structure of sentences), semantics (meaning), Morphology (linguistics), morphology (structure of words), phonetics (speech sounds ...

and

natural language processing Natural language processing (NLP) is a subfield of computer science and especially artificial intelligence. It is primarily concerned with providing computers with the ability to process data encoded in natural language and is thus closely related ...

, a corpus (: corpora) or text corpus is a dataset, consisting of natively digital and older, digitalized, language resources, either annotated or unannotated. Annotated, they have been used in

corpus linguistics Corpus linguistics is an empirical method for the study of language by way of a text corpus (plural ''corpora''). Corpora are balanced, often stratified collections of authentic, "real world", text of speech or writing that aim to represent a giv ...

for statistical

hypothesis testing A statistical hypothesis test is a method of statistical inference used to decide whether the data provide sufficient evidence to reject a particular hypothesis. A statistical hypothesis test typically involves a calculation of a test statistic. T ...

, checking occurrences or validating linguistic rules within a specific language territory.

Overview

A corpus may contain texts in a single language (''monolingual corpus'') or text data in multiple languages (''multilingual corpus''). In order to make the corpora more useful for doing linguistic research, they are often subjected to a process known as

annotation An annotation is extra information associated with a particular point in a document or other piece of information. It can be a note that includes a comment or explanation. Annotations are sometimes presented Marginalia, in the margin of book page ...

. An example of annotating a corpus is

part-of-speech tagging In corpus linguistics, part-of-speech tagging (POS tagging, PoS tagging, or POST), also called grammatical tagging, is the process of marking up a word in a text ( corpus) as corresponding to a particular part of speech, based on both its defini ...

, or ''POS-tagging'', in which information about each word's part of speech (verb, noun, adjective, etc.) is added to the corpus in the form of ''tags''. Another example is indicating the lemma (base) form of each word. When the language of the corpus is not a working language of the researchers who use it,

interlinear gloss In linguistics and pedagogy, an interlinear gloss is a gloss (series of brief explanations, such as definitions or pronunciations) placed between lines, such as between a line of original text and its translation into another language. When gloss ...

ing is used to make the annotation bilingual. Some corpora have further ''structured'' levels of analysis applied. In particular, smaller corpora may be fully parsed. Such corpora are usually called

Treebank In linguistics, a treebank is a parsed text corpus that annotates syntactic or semantic sentence structure. The construction of parsed corpora in the early 1990s revolutionized computational linguistics, which benefitted from large-scale empi ...

s or Parsed Corpora. The difficulty of ensuring that the entire corpus is completely and consistently annotated means that these corpora are usually smaller, containing around one to three million words. Other levels of linguistic structured analysis are possible, including annotations for

morphology Morphology, from the Greek and meaning "study of shape", may refer to: Disciplines *Morphology (archaeology), study of the shapes or forms of artifacts *Morphology (astronomy), study of the shape of astronomical objects such as nebulae, galaxies, ...

semantics Semantics is the study of linguistic Meaning (philosophy), meaning. It examines what meaning is, how words get their meaning, and how the meaning of a complex expression depends on its parts. Part of this process involves the distinction betwee ...

and

pragmatics In linguistics and the philosophy of language, pragmatics is the study of how Context (linguistics), context contributes to meaning. The field of study evaluates how human language is utilized in social interactions, as well as the relationship ...

Applications

Corpora are the main knowledge base in

. Other notable areas of application include: *

Language technology Language technology, often called human language technology (HLT), studies methods of how computer programs or electronic devices can analyze, produce, modify or respond to human texts and speech. Working with language technology often requires broa ...

computational linguistics Computational linguistics is an interdisciplinary field concerned with the computational modelling of natural language, as well as the study of appropriate computational approaches to linguistic questions. In general, computational linguistics ...

** The analysis and processing of various types of corpora are also the subject of much work in

speech recognition Speech recognition is an interdisciplinary subfield of computer science and computational linguistics that develops methodologies and technologies that enable the recognition and translation of spoken language into text by computers. It is also ...

and

machine translation Machine translation is use of computational techniques to translate text or speech from one language to another, including the contextual, idiomatic and pragmatic nuances of both languages. Early approaches were mostly rule-based or statisti ...

, where they are often used to create

hidden Markov model A hidden Markov model (HMM) is a Markov model in which the observations are dependent on a latent (or ''hidden'') Markov process (referred to as X). An HMM requires that there be an observable process Y whose outcomes depend on the outcomes of X ...

s for part of speech tagging and other purposes. Corpora and

frequency list A word list is a list of words in a lexicon, generally sorted by frequency of occurrence (either by graded levels, or as a ranked list). A word list is compiled by lexical frequency analysis within a given text corpus, and is used in corpus li ...

s derived from them are useful for

language teaching Language education refers to the processes and practices of teaching a second or foreign language. Its study reflects interdisciplinary approaches, usually including some applied linguistics. There are four main learning categories for lan ...

. Corpora can be considered as a type of

foreign language writing aid A foreign language writing aid is a computer program or any other instrument that assists a non-native language user (also referred to as a foreign language learner) in writing decently in their target language. Assistive operations can be classifi ...

as the contextualised grammatical knowledge acquired by non-native language users through exposure to authentic texts in corpora allows learners to grasp the manner of sentence formation in the target language, enabling effective writing.Yoon, H., & Hirvela, A. (2004)
ESL Student Attitudes toward Corpus Use in L2 Writing
''Journal of Second Language Writing, 13''(4), 257–283. Retrieved 21 March 2012. *

Machine translation Machine translation is use of computational techniques to translate text or speech from one language to another, including the contextual, idiomatic and pragmatic nuances of both languages. Early approaches were mostly rule-based or statisti ...

** Multilingual corpora that have been specially formatted for side-by-side comparison are called ''aligned parallel corpora''. There are two main types of

parallel corpora A parallel text is a text placed alongside its translation or translations. Parallel text alignment is the identification of the corresponding sentences in both halves of the parallel text. The Loeb Classical Library and the Clay Sanskrit Libra ...

which contain texts in two languages. In a ''translation corpus'', the texts in one language are translations of texts in the other language. In a ''comparable corpus'', the texts are of the same kind and cover the same content, but they are not translations of each other. To exploit a parallel text, some kind of text alignment identifying equivalent text segments (phrases or sentences) is a prerequisite for analysis.

algorithms for translating between two languages are often trained using parallel fragments comprising a first-language corpus and a second-language corpus, which is an element-for-element translation of the first-language corpus. * Philologies ** Text corpora are also used in the study of

historical document Historical documents are original documents that contain important historical information about a person, place, or event and can thus serve as primary sources as important ingredients of the historical methodology. Significant historical docume ...

s, for example in attempts to

decipher DECIPHER is a web-based resource and database of genomic variation data from analysis of patient DNA. It documents submicroscopic Chromosome abnormality, chromosome abnormalities (Deletion (genetics), microdeletions and Gene duplication, duplic ...

ancient scripts, or in

Biblical scholarship Biblical studies is the academic application of a set of diverse disciplines to the study of the Bible, with ''Bible'' referring to the books of the canonical Hebrew Bible in mainstream Jewish usage and the Christian Bible including the can ...

. Some archaeological corpora can be of such short duration that they provide a snapshot in time. One of the shortest corpora in time may be the 15–30 year Amarna letters texts (

1350 BC Events and trends * c. 1352 BC – Amenhotep III (Eighteenth Dynasty of Egypt) dies and is succeeded as Pharaoh by Amenhotep IV. * 1350 BC – Yin becomes the new capital of Shang dynasty The Shang dynasty (), also known as ...

). The ''corpus'' of an ancient city, (for example the "

Kültepe Kültepe ( Turkish: ), also known under its ancient name Kaneš (Kanesh, sometimes also Kaniš/Kanish) or Neša (Nesha), is an archaeological site in Kayseri Province, Turkey. It was already a major settlement at the beginning of the 3rd mille ...

Texts" of Turkey), may go through a series of corpora, determined by their find site dates.

Some notable text corpora

References

External links

ACL SIGLEX Resource Links: Text Corpora

Developing Linguistic Corpora: a Guide to Good Practice

Free samples (not free), web-based corpora (45-425 million words each): American (COCA, COHA, TIME), British (BNC), Spanish, Portuguese

Intercorp
Building synchronous parallel corpora of the languages taught at the Faculty of Arts of Charles University.
Sketch Engine: Open corpora with free access

TS Corpus – A Turkish Corpus freely available for academic research.

Turkish National Corpus – A general-purpose corpus for contemporary Turkish

Corpus of Political Speeches
Free access to political speeches by American and Chinese politicians, developed by Hong Kong Baptist University Library
Russian National Corpus
{{Natural Language Processing Discourse analysis Corpus linguistics Computational linguistics Works based on multiple works Test items lt:Tekstynas

Overview

Applications

Some notable text corpora

See also

References

External links