Music information retrieval (MIR) is the interdisciplinary science of retrieving

information Information is an Abstraction, abstract concept that refers to something which has the power Communication, to inform. At the most fundamental level, it pertains to the Interpretation (philosophy), interpretation (perhaps Interpretation (log ...

from

music Music is the arrangement of sound to create some combination of Musical form, form, harmony, melody, rhythm, or otherwise Musical expression, expressive content. Music is generally agreed to be a cultural universal that is present in all hum ...

. Those involved in MIR may have a background in academic

musicology Musicology is the academic, research-based study of music, as opposed to musical composition or performance. Musicology research combines and intersects with many fields, including psychology, sociology, acoustics, neurology, natural sciences, ...

psychoacoustics Psychoacoustics is the branch of psychophysics involving the scientific study of the perception of sound by the human auditory system. It is the branch of science studying the psychological responses associated with sound including noise, speech, ...

psychology Psychology is the scientific study of mind and behavior. Its subject matter includes the behavior of humans and nonhumans, both consciousness, conscious and Unconscious mind, unconscious phenomena, and mental processes such as thoughts, feel ...

signal processing Signal processing is an electrical engineering subfield that focuses on analyzing, modifying and synthesizing ''signals'', such as audio signal processing, sound, image processing, images, Scalar potential, potential fields, Seismic tomograph ...

informatics Informatics is the study of computational systems. According to the Association for Computing Machinery, ACM Europe Council and Informatics Europe, informatics is synonymous with computer science and computing as a profession, in which the centra ...

machine learning Machine learning (ML) is a field of study in artificial intelligence concerned with the development and study of Computational statistics, statistical algorithms that can learn from data and generalise to unseen data, and thus perform Task ( ...

, optical music recognition,

computational intelligence In computer science, computational intelligence (CI) refers to concepts, paradigms, algorithms and implementations of systems that are designed to show " intelligent" behavior in complex and changing environments. These systems are aimed at m ...

, or some combination of these.

Applications

Music information retrieval is being used by businesses and academics to categorize, manipulate and even create music.

Music classification

One of the classical MIR research topics is genre classification, which is categorizing music items into one of the pre-defined genres such as classical,

jazz Jazz is a music genre that originated in the African-American communities of New Orleans, Louisiana, in the late 19th and early 20th centuries. Its roots are in blues, ragtime, European harmony, African rhythmic rituals, spirituals, h ...

, rock, etc. Mood classification, artist classification, instrument identification, and music tagging are also popular topics.

Recommender systems

Several

recommender systems A recommender system (RecSys), or a recommendation system (sometimes replacing ''system'' with terms such as ''platform'', ''engine'', or ''algorithm'') and sometimes only called "the algorithm" or "algorithm", is a subclass of information fil ...

for music already exist, but surprisingly few are based upon MIR techniques, instead of making use of similarity between users or laborious data compilation.

Pandora In Greek mythology, Pandora was the first human woman created by Hephaestus on the instructions of Zeus. As Hesiod related it, each god cooperated by giving her unique gifts. Her other name—inscribed against her figure on a white-ground '' ky ...

, for example, uses experts to tag the music with particular qualities such as "female singer" or "strong bassline". Many other systems find users whose listening history is similar and suggests unheard music to the users from their respective collections. MIR techniques for similarity in music are now beginning to form part of such systems.

Music source separation and instrument recognition

Music source separation is about separating original signals from a mixture

audio signal An audio signal is a representation of sound, typically using either a changing level of electrical voltage for analog signals or a series of binary numbers for Digital signal (signal processing), digital signals. Audio signals have frequencies i ...

. Instrument recognition is about identifying the instruments involved in music. Various MIR systems have been developed that can separate music into its component tracks without access to the master copy. In this way, for example, karaoke tracks can be created from normal music tracks, though the process is not yet perfect owing to vocals occupying some of the same

frequency Frequency is the number of occurrences of a repeating event per unit of time. Frequency is an important parameter used in science and engineering to specify the rate of oscillatory and vibratory phenomena, such as mechanical vibrations, audio ...

space as the other instruments.

Automatic music transcription

Automatic music transcription is the process of converting an audio recording into symbolic notation, such as a score or a MIDI file. This process involves several audio analysis tasks, which may include multi-pitch detection,

onset detection Onset refers to the beginning of a musical note or other sound. It is related to (but different from) the concept of a transient: all musical notes have an onset, but do not necessarily include an initial transient. Onset detection In signal pro ...

, duration estimation, instrument identification, and the extraction of

harmonic In physics, acoustics, and telecommunications, a harmonic is a sinusoidal wave with a frequency that is a positive integer multiple of the ''fundamental frequency'' of a periodic signal. The fundamental frequency is also called the ''1st har ...

, rhythmic or melodic information. This task becomes more difficult with greater numbers of instruments and a greater polyphony level.

Music generation

The automatic generation of music is a goal held by many MIR researchers. Attempts have been made with limited success in terms of human appreciation of the results.

Methods used

Data source

Scores give a clear and logical description of music from which to work, but access to

sheet music Sheet music is a handwritten or printed form of musical notation that uses musical symbols to indicate the pitches, rhythms, or chords of a song or instrumental musical piece. Like its analogs – printed Book, books or Pamphlet, pamphlets ...

, whether digital or otherwise, is often impractical.

MIDI Musical Instrument Digital Interface (; MIDI) is an American-Japanese technical standard that describes a communication protocol, digital interface, and electrical connectors that connect a wide variety of electronic musical instruments, ...

music has also been used for similar reasons, but some data is lost in the conversion to MIDI from any other format, unless the music was written with the MIDI standards in mind, which is rare. Digital audio formats such as WAV, mp3, and

ogg Ogg is a digital multimedia container format designed to provide for efficient streaming and manipulation of digital multimedia. It is maintained by the Xiph.Org Foundation and is free and open, unrestricted by software patents. Its name is ...

are used when the audio itself is part of the analysis. Lossy formats such as mp3 and ogg work well with the human ear but may be missing crucial data for study. Additionally some encodings create artifacts which could be misleading to any automatic analyser. Despite this the ubiquity of the mp3 has meant much research in the field involves these as the source material. Increasingly,

metadata Metadata (or metainformation) is "data that provides information about other data", but not the content of the data itself, such as the text of a message or the image itself. There are many distinct types of metadata, including: * Descriptive ...

mined from the web is incorporated in MIR for a more rounded understanding of the music within its cultural context, and this recently consists of analysis of social tags for music.

Feature representation

Analysis can often require some summarising, and for music (as with many other forms of data) this is achieved by

feature extraction Feature may refer to: Computing * Feature recognition, could be a hole, pocket, or notch * Feature (computer vision), could be an edge, corner or blob * Feature (machine learning), in statistics: individual measurable properties of the phenome ...

, especially when the audio content itself is analysed and

is to be applied. The purpose is to reduce the sheer quantity of data down to a manageable set of values so that learning can be performed within a reasonable time-frame. One common feature extracted is the Mel-Frequency Cepstral Coefficient (MFCC) which is a measure of the

timbre In music, timbre (), also known as tone color or tone quality (from psychoacoustics), is the perceived sound of a musical note, sound or tone. Timbre distinguishes sounds according to their source, such as choir voices and musical instrument ...

of a piece of music. Other features may be employed to represent the key, chords,

harmonies In music, harmony is the concept of combining different sounds in order to create new, distinct musical ideas. Theories of harmony seek to describe or explain the effects created by distinct pitches or tones coinciding with one another; harm ...

melody A melody (), also tune, voice, or line, is a linear succession of musical tones that the listener perceives as a single entity. In its most literal sense, a melody is a combination of Pitch (music), pitch and rhythm, while more figurativel ...

, main pitch,

beats per minute Beat, beats, or beating may refer to: Common uses * Assault, inflicting physical harm or unwanted physical contact * Battery (crime), a criminal offense involving unlawful physical contact * Battery (tort), a civil wrong in common law of in ...

or rhythm in the piece. There are a number of available audio feature extraction toolsDavid Moffat, David Ronan, and Joshua D Reiss. "An Evaluation of Audio Feature Extraction Toolboxes". In Proceedings of the International Conference on Digital Audio Effects (DAFx), 2016.
Available here

Statistics and machine learning

*Computational methods for classification, clustering, and modelling — musical feature extraction for mono- and

polyphonic Polyphony ( ) is a type of musical texture consisting of two or more simultaneous lines of independent melody, as opposed to a musical texture with just one voice ( monophony) or a texture with one dominant melodic voice accompanied by chords ...

music, similarity and

pattern matching In computer science, pattern matching is the act of checking a given sequence of tokens for the presence of the constituents of some pattern. In contrast to pattern recognition, the match usually must be exact: "either it will or will not be a ...

, retrieval *

Formal methods In computer science, formal methods are mathematics, mathematically rigorous techniques for the formal specification, specification, development, Program analysis, analysis, and formal verification, verification of software and computer hardware, ...

and

databases In computing, a database is an organized collection of data or a type of data store based on the use of a database management system (DBMS), the software that interacts with end users, applications, and the database itself to capture and ana ...

— applications of automated

music identification An acoustic fingerprint is a condensed digital summary, a digital fingerprint, deterministically generated from an audio signal, that can be used to identify an audio sample or quickly locate similar items in a music database. Practical u ...

and recognition, such as score following, automatic accompaniment, routing and filtering for music and music queries, query languages, standards and other metadata or protocols for music information handling and retrieval, multi-agent systems, distributed search) *Software for music information retrieval —

Semantic Web The Semantic Web, sometimes known as Web 3.0, is an extension of the World Wide Web through standards set by the World Wide Web Consortium (W3C). The goal of the Semantic Web is to make Internet data machine-readable. To enable the encoding o ...

and musical digital objects, intelligent agents,

collaborative software Collaborative software or groupware is application software designed to help people working on a common task to attain their goals. One of the earliest definitions of groupware is "intentional group processes plus software to support them." Regar ...

, web-based search and semantic retrieval, query by humming /

Search by sound Search by sound is the retrieval of information based on audio input. There are a handful of applications, specifically for mobile devices that utilize search by sound. Shazam, Soundhound, Axwave, ACRCloud and others have seen considerable su ...

acoustic fingerprinting An acoustic fingerprint is a condensed digital summary, a Fingerprint (computing), digital fingerprint, Deterministic algorithm, deterministically generated from an audio signal, that can be used to identify an audio sample or quickly locate s ...

* Music analysis and knowledge representation —

automatic summarization Automatic summarization is the process of shortening a set of data computationally, to create a subset (a summary) that represents the most important or relevant information within the original content. Artificial intelligence algorithms are comm ...

, citing, excerpting, downgrading, transformation, formal models of music, digital scores and representations, music indexing and

Other issues

* Human-computer interaction and interfaces — multi-modal interfaces,

user interface In the industrial design field of human–computer interaction, a user interface (UI) is the space where interactions between humans and machines occur. The goal of this interaction is to allow effective operation and control of the machine fro ...

s and

usability Usability can be described as the capacity of a system to provide a condition for its users to perform the tasks safely, effectively, and efficiently while enjoying the experience. In software engineering, usability is the degree to which a softw ...

mobile applications A mobile application or app is a computer program or software application designed to run on a mobile device such as a phone, tablet, or watch. Mobile applications often stand in contrast to desktop applications which are designed to run on d ...

, user behavior * Music perception, cognition, affect, and emotions — music similarity metrics, syntactical parameters, semantic parameters, musical forms, structures, styles and music annotation methodologies * Music archives, libraries, and digital collections — music digital libraries, public access to musical archives, benchmarks and research databases *

Intellectual property Intellectual property (IP) is a category of property that includes intangible creations of the human intellect. There are many types of intellectual property, and some countries recognize more than others. The best-known types are patents, co ...

rights and music — national and international

copyright A copyright is a type of intellectual property that gives its owner the exclusive legal right to copy, distribute, adapt, display, and perform a creative work, usually for a limited time. The creative work may be in a literary, artistic, ...

issues,

digital rights management Digital rights management (DRM) is the management of legal access to digital content. Various tools or technological protection measures, such as access control technologies, can restrict the use of proprietary hardware and copyrighted works. DRM ...

, identification and traceability * Sociology and Economy of music — music industry and use of MIR in the production, distribution, consumption chain, user profiling, validation, user needs and expectations, evaluation of music IR systems, building test collections, experimental design and metrics