{{Unreferenced, date=June 2019, bot=noref (GreenC bot) Multimedia search enables information

search Searching or search may refer to: Computing technology * Search algorithm, including keyword search ** :Search algorithms * Search and optimization for problem solving in artificial intelligence * Search engine technology, software for findi ...

using queries in multiple data types including text and other

multimedia Multimedia is a form of communication that uses a combination of different content forms such as text, audio, images, animations, or video into a single interactive presentation, in contrast to tradi ...

formats. Multimedia search can be implemented through

multimodal search Multimodal search is a type of search that uses different methods to get relevant results. They can use any kind of search, search by keyword, search by concept, search by example, etc. Introduction A multimodal search engine is designed to i ...

interfaces, i.e., interfaces that allow to submit search queries not only as textual requests, but also through other media. We can distinguish two methodologies in multimedia search: *Metadata search: the search is made on the layers of metadata. *

Query by example Query by Example (QBE) is a database query language for relational databases. It was devised by Moshé M. Zloof at IBM Research during the mid-1970s, in parallel to the development of SQL. It is the first graphical query language, using visual t ...

: The interaction consists in submitting a piece of information (e.g., a video, an image, or a piece of audio) at the purpose of finding similar multimedia items.

Metadata search

Search is made using the layers in metadata which contain information of the content of a multimedia file. Metadata search is easier, faster and effective because instead of working with complex material, such as an audio, a video or an image, it searches using text. There are three processes which should be done in this method: * Summarization of media content (

feature extraction In machine learning, pattern recognition, and image processing, feature extraction starts from an initial set of measured data and builds derived values ( features) intended to be informative and non-redundant, facilitating the subsequent learning ...

). The result of feature extraction is a description. * Filtering of media descriptions (for example, elimination of Redundancy) * Categorization of media descriptions into classes.

Query by example

query by example Query by Example (QBE) is a database query language for relational databases. It was devised by Moshé M. Zloof at IBM Research during the mid-1970s, in parallel to the development of SQL. It is the first graphical query language, using visual t ...

, the element used to search is a

content (image, audio, video). In other words, the query is a media. Often, it's used audiovisual indexing. It will be necessary to choose the criteria we are going to use for creating metadata. The process of search can be divided in three parts: *Generate descriptors for the media which we are going to use as query and the descriptors for the media in our

database In computing, a database is an organized collection of data stored and accessed electronically. Small databases can be stored on a file system, while large databases are hosted on computer clusters or cloud storage. The design of databases spa ...

. *Compare descriptors of the query and our database’s media. *List the media sorted by maximum coincidence.

Multimedia search engine

There are two big search families, in function of the content: *

Visual search engine A visual search engine is a search engine designed to search for information on the World Wide Web through the input of an image or a search engine with a visual display of the search results. Information may consist of web pages, locations, oth ...

Audio search engine An audio search engine is a web-based search engine which crawls the web for audio content. The information can consist of web pages, images, audio files, or another type of document. Various techniques exist for research on these engines. Typ ...

Visual search engine

Inside this family we can distinguish two topics:

image search An image retrieval system is a computer system used for browsing, searching and retrieving images from a large database of digital images. Most traditional and common methods of image retrieval utilize some method of adding metadata such as captio ...

and

video search A video search engine is a web-based search engine which crawls the web for video content. Some video search engines parse externally hosted content while others allow content to be uploaded and hosted on their own servers. Some engines also allow ...

Image search An image retrieval system is a computer system used for browsing, searching and retrieving images from a large database of digital images. Most traditional and common methods of image retrieval utilize some method of adding metadata such as captio ...

: Although usually it's used simple metadata search, increasingly is being used indexing methods for making the results of users queries more accurate using

. For example,

QR codes A QR code (an initialism for quick response code) is a type of matrix barcode (or two-dimensional barcode) invented in 1994 by the Japanese company Denso Wave. A barcode is a machine-readable optical label that can contain information about th ...

. *

Video search A video search engine is a web-based search engine which crawls the web for video content. Some video search engines parse externally hosted content while others allow content to be uploaded and hosted on their own servers. Some engines also allow ...

: Videos can be searched for simple metadata or by complex metadata generated by indexing. The audio contained in the videos is usually scanned by audio search engines.

Audio search engine

There are different methods of audio searching: *

Voice search Voice search, also called voice-enabled, allows the user to use a voice command to search the Internet, a website, or an app. In a broader definition, voice search includes open-domain keyword query on any information on the Internet, for example ...

engine: Allows the user to search using speech instead of text. It uses algorithms of

speech recognition Speech recognition is an interdisciplinary subfield of computer science and computational linguistics that develops methodologies and technologies that enable the recognition and translation of spoken language into text by computers with the ma ...

. An example of this technology is

Google Voice Search Google Voice Search or Search by Voice is a Google product that allows users to use Google Search by speaking on a mobile phone or computer, i.e. have the device search for data upon entering information on what to search into the device by spe ...

. *Music search engine: Although most of applications which searches music works on simple metadata (artist, name of track, album…) . There are some programs of

music recognition Music information retrieval (MIR) is the interdisciplinary science of retrieving information from music. MIR is a small but growing field of research with many real-world applications. Those involved in MIR may have a background in academic musico ...

, for example

Shazam Shazam () may refer to: Comic book franchise * Captain Marvel (DC Comics), also known as Shazam, a superhero character published by Fawcett Comics and DC Comics ** Shazam (wizard), a character from the ''Shazam!/Captain Marvel'' comics, who give ...

SoundHound SoundHound Inc. is an audio and speech recognition company founded in 2005. It develops speech recognition, natural language understanding, sound recognition and search technologies. Its featured products include Houndify, a Voice AI developer p ...

Metadata search

Query by example

Multimedia search engine

Visual search engine

Audio search engine

See also