Azure Data Lake is a scalable

data storage Data storage is the recording (storing) of information (data) in a storage medium. Handwriting, phonographic recording, magnetic tape, and optical discs are all examples of storage media. Biological molecules such as RNA and DNA are cons ...

and

analytics Analytics is the systematic computational analysis of data or statistics. It is used for the discovery, interpretation, and communication of meaningful patterns in data. It also entails applying data patterns toward effective decision-making. It ...

service. The service is hosted in

Azure Azure may refer to: Colour * Azure (color), a hue of blue ** Azure (heraldry) ** Shades of azure, shades and variations Arts and media * ''Azure'' (Art Farmer and Fritz Pauer album), 1987 * Azure (Gary Peacock and Marilyn Crispell album), 2013 ...

, Microsoft's public

cloud In meteorology, a cloud is an aerosol consisting of a visible mass of miniature liquid droplets, frozen crystals, or other particles suspended in the atmosphere of a planetary body or similar space. Water or various other chemicals may ...

History

Azure Data Lake service was released on November 16, 2016. It is based on COSMOS, which is used to store and process data for applications such as Azure, AdCenter, Bing,

MSN MSN (meaning Microsoft Network) is a web portal and related collection of Internet services and apps for Windows and mobile devices, provided by Microsoft and launched on August 24, 1995, alongside the release of Windows 95. The Microsoft Net ...

Skype Skype () is a proprietary telecommunications application operated by Skype Technologies, a division of Microsoft, best known for VoIP-based videotelephony, videoconferencing and voice calls. It also has instant messaging, file transfer, ...

and Windows Live. COSMOS features a SQL-like query engine called SCOPE upon which U-SQL was built.

Azure Data Lake Store

Users can store structured, semi-structured or

unstructured data Unstructured data (or unstructured information) is information that either does not have a pre-defined data model or is not organized in a pre-defined manner. Unstructured information is typically text-heavy, but may contain data such as dates, n ...

produced from applications including

social networks A social network is a social structure made up of a set of social actors (such as individuals or organizations), sets of dyadic ties, and other social interactions between actors. The social network perspective provides a set of methods for ...

, relational data, sensors, videos, web apps, mobile or desktop devices. A single Azure Data Lake Store account can store trillions of files where a single file can be greater than a petabyte in size.

Azure Data Lake Analytics

Azure Data Lake Analytics is a parallel on-demand job service. The parallel processing system is based on Microsoft Dryad. Dryad can represent arbitrary

Directed Acyclic Graph In mathematics, particularly graph theory, and computer science, a directed acyclic graph (DAG) is a directed graph with no directed cycles. That is, it consists of vertices and edges (also called ''arcs''), with each edge directed from one v ...

s (DAGs) of computation. Data Lake Analytics provides a distributed infrastructure that can dynamically allocate or de-allocate resources so customers pay for only the services they use. Azure Data Lake Analytics uses Apache YARN, the part of Apache Hadoop which governs resource management across clusters. Microsoft Azure Data Lake Store supports any application that uses the Hadoop Distributed File System (HDFS) interface.

U-SQL

Using Data Lake Analytics, users can develop and run parallel data transformation and processing programs in U-SQL, a query language that combines SQL with C#. U-SQL was designed as an evolution of the declarative SQL language with native extensibility through the user code written in C#. U-SQL uses C# data types and the C# expression language.

References

External links