jq is a
very high-level lexically scoped
In computer programming, the scope of a name binding (an association of a name to an entity, such as a variable) is the part of a program where the name binding is valid; that is, where the name can be used to refer to the entity. In other part ...
functional programming language
In computer science, functional programming is a programming paradigm where programs are constructed by applying and composing functions. It is a declarative programming paradigm in which function definitions are trees of expressions that ...
in which every
JSON
JSON (JavaScript Object Notation, pronounced ; also ) is an open standard file format and data interchange format that uses human-readable text to store and transmit data objects consisting of attribute–value pairs and arrays (or other s ...
value is a constant. jq supports
backtracking
Backtracking is a class of algorithms for finding solutions to some computational problems, notably constraint satisfaction problems, that incrementally builds candidates to the solutions, and abandons a candidate ("backtracks") as soon as it de ...
and managing indefinitely long
streams
A stream is a continuous body of surface water flowing within the bed and banks of a channel. Depending on its location or certain characteristics, a stream may be referred to by a variety of local or regional names. Long large streams ...
of JSON data. It is related to the
Icon
An icon () is a religious work of art, most commonly a painting, in the cultures of the Eastern Orthodox, Oriental Orthodox, and Catholic churches. They are not simply artworks; "an icon is a sacred image used in religious devotion". The most c ...
and
Haskell
Haskell () is a general-purpose, statically-typed, purely functional programming language with type inference and lazy evaluation. Designed for teaching, research and industrial applications, Haskell has pioneered a number of programming lan ...
programming languages. The language supports a namespace-based module system and has some support for closures. In particular, functions and functional expressions can be used as parameters of other functions.
The original implementation of jq was in
C; gojq is a
"pure Go" implementation. There is also a
Rust
Rust is an iron oxide, a usually reddish-brown oxide formed by the reaction of iron and oxygen in the catalytic presence of water or air moisture. Rust consists of hydrous iron(III) oxides (Fe2O3·nH2O) and iron(III) oxide-hydroxide (FeO(OH), ...
implementation of a large subset of jq named jaq,
[
]
for which a
denotational semantics has been specified.
[
]
gojq should run on any platform where Go is supported, and likewise for jaq and Rust.
History
jq was created by Stephen Dolan, and released in October 2012.
It was famously described as being "like
sed for JSON data".
Support for
regular expressions
A regular expression (shortened as regex or regexp; sometimes referred to as rational expression) is a sequence of characters that specifies a search pattern in text. Usually such patterns are used by string-searching algorithms for "find" o ...
was added in jq version 1.5.
A "wrapper" program for jq name
yqadds support for
YAML
YAML ( and ) (''see '') is a human-readable data-serialization language. It is commonly used for configuration files and in applications where data is being stored or transmitted. YAML targets many of the same communications applications as Ext ...
,
XML
Extensible Markup Language (XML) is a markup language and file format for storing, transmitting, and reconstructing arbitrary data. It defines a set of rules for encoding documents in a format that is both human-readable and machine-readable. ...
and
TOML
TOML is a file format for configuration files. It is intended to be easy to read and write due to obvious semantics which aim to be "minimal", and is designed to map unambiguously to a dictionary. Its specification is open-source, and receives c ...
. It was first released in 2017.
The Go implementation, gojq, was initially released in 2019.
gojq notably extends jq to include support for
YAML
YAML ( and ) (''see '') is a human-readable data-serialization language. It is commonly used for configuration files and in applications where data is being stored or transmitted. YAML targets many of the same communications applications as Ext ...
.
The first version of jaq to include extensive support for
regular expression
A regular expression (shortened as regex or regexp; sometimes referred to as rational expression) is a sequence of characters that specifies a search pattern in text. Usually such patterns are used by string-searching algorithms for "find" ...
s
was released in March 2023. This version (0.10) also includes
a fast JSON parser but still lacks support for
nested function
In computer programming, a nested function (or nested procedure or subroutine) is a function which is defined within another function, the ''enclosing function''. Due to simple recursive scope rules, a nested function is itself invisible outside ...
s,
recursively defined functions,
and does not include a "streaming parser" for processing
very large JSON documents with minimal memory requirements.
Usage
Command-line usage
jq is typically used at the command line and can be used with other command-line utilities, such as
curl
cURL (pronounced like "curl", UK: , US: ) is a computer software project providing a library (libcurl) and command-line tool (curl) for transferring data using various network protocols. The name stands for "Client URL".
History
cURL was fir ...
. Here is an example showing how the output of a command can be piped to a jq filter to determine the category names associated with this
Wikipedia
Wikipedia is a multilingual free online encyclopedia written and maintained by a community of volunteers, known as Wikipedians, through open collaboration and using a wiki-based editing system. Wikipedia is the largest and most-read ref ...
page:
$ curl 'https://en.wikipedia.org/w/api.php?action=parse&page=jq_(programming_language)&format=json' , jq '.parse.categories[]["*"]'
The output produced by this pipeline consists of a stream of JSON strings, the first few of which are:
"Articles_with_short_description"
"Short_description_matches_Wikidata"
"Dynamically_typed_programming_languages"
"Functional_languages"
"Programming_languages"
The command above uses the
MediaWiki
MediaWiki is a Free and open-source software, free and open-source wiki software. It is used on Wikipedia and almost all other Wikimedia movement, Wikimedia Website, websites, including Wiktionary, Wikimedia Commons and Wikidata; these sit ...
API for this page to produce a JSON response.
The
pipe
Pipe(s), PIPE(S) or piping may refer to:
Objects
* Pipe (fluid conveyance), a hollow cylinder following certain dimension rules
** Piping, the use of pipes in industry
* Smoking pipe
** Tobacco pipe
* Half-pipe and quarter pipe, semi-circul ...
allows the output of to be accessed by jq, a standard
Unix shell
A Unix shell is a command-line interpreter or shell that provides a command line user interface for Unix-like operating systems. The shell is both an interactive command language and a scripting language, and is used by the operating system t ...
mechanism.
The jq filter shown is an abbreviation for the jq pipeline:
. parse", . categories", .[] , .["*"]
This corresponds to the nested JSON structure produced by the call to . Notice that the jq pipeline is constructed in the same manner using the character as the Unix-style pipeline.
Embedded usage
Both the C and the Go implementations provide libraries so that jq functionality can be embedded in other applications and programming environments.
For example, gojq has been integrated with
SQLite
SQLite (, ) is a database engine written in the C programming language. It is not a standalone app; rather, it is a library that software developers embed in their apps. As such, it belongs to the family of embedded databases. It is the mo ...
so that a function is available in
SQL statements.
This function is marked a
"deterministic"and
can therefore be used in
"CREATE INDEX" commands.
Modes of operation
jq by default acts as a "stream editor" for JSON inputs, much
like the
sed utility can be thought of as a "stream editor" for lines of text.
However jq has several other modes of operation:
# it can treat its input from one or more sources as lines of text;
# it can gather a stream of inputs from a specified source into a JSON array;
# it can parse its JSON inputs using a so-called "streaming parser" that produces a stream of
ath, valuearrays for all "leaf" paths.
The "streaming parser" is particularly useful when one of more of the
JSON inputs is too large to fit into memory, since its memory requirements
are typically quite small. For example, for an arbitrarily large array of JSON objects,
the peak memory requirement is not much more than required to handle the largest
top-level object.
These modes of operation can, within certain limitations, be combined.
Syntax and semantics
Types
Every JSON value is itself a value in jq, which accordingly has the types shown in the table below.
The gojq implementation distinguishes between integers and non-integer numbers, and supports unbounded-precision integer arithmetic.
is a value, just like any other JSON scalar; it is not a pointer or a "null-pointer".
(corresponding to
NaN
Nan or NAN may refer to:
Places China
* Nan County, Yiyang, Hunan, China
* Nan Commandery, historical commandery in Hubei, China
Thailand
* Nan Province
** Nan, Thailand, the administrative capital of Nan Province
* Nan River
People Given na ...
) and (see
IEEE 754
The IEEE Standard for Floating-Point Arithmetic (IEEE 754) is a technical standard for floating-point arithmetic established in 1985 by the Institute of Electrical and Electronics Engineers (IEEE). The standard addressed many problems found i ...
) are the only two jq scalars that are not also JSON values.
Forms
There are special syntactic forms for function creation, conditionals, stream reduction, and the module system.
Filters
Here is an example which shows how to define a named, parameterized filter for formatting an integer in any base
from 2 to 36 inclusive.
The implementation illustrates
tacit (or point-free) programming:
# Use gojq for infinite precision integer arithmetic
def tobase($b):
def digit: "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ" :.+1
def mod: . % $b;
def div: ((. - mod) / $b);
def digits: recurse( select(. >= $b) , div) , mod ;
select(2 <= $b and $b <= 36)
, digit, reverse , add;
The next example demonstrates the use of
generators in the classic "SEND MORE MONEY"
verbal arithmetic
Verbal arithmetic, also known as alphametics, cryptarithmetic, cryptarithm or word addition, is a type of mathematical game consisting of a mathematical equation among unknown numbers, whose digits are represented by letters of the alphabet. T ...
game:
def send_more_money:
def choose(m;n;used): ( ange(m;n+1)- used)[];
def num(a;b;c;d): 1000*a + 100*b + 10*c + d;
def num(a;b;c;d;e): 10*num(a;b;c;d) + e;
first(
1 as $m
, 0 as $o
, choose(8;9;[]) as $s
, choose(2;9;[$s]) as $e
, choose(2;9; s,$e as $n
, choose(2;9; s,$e,$n as $d
, choose(2;9; s,$e,$n,$d as $r
, choose(2;9; s,$e,$n,$d,$r as $y
, select(num($s;$e;$n;$d) + num($m;$o;$r;$e) num($m;$o;$n;$e;$y))
, s,$e,$n,$d,$m,$o,$r,$e,$m,$o,$n,$e,$y);
Parsing expression grammars
There is a very close relationship between jq and the
parsing expression grammar
In computer science, a parsing expression grammar (PEG) is a type of analytic formal grammar, i.e. it describes a formal language in terms of a set of rules for recognizing strings in the language. The formalism was introduced by Bryan Ford in ...
(PEG) formalism.
The relationship stems from the equivalence of the seven basic PEG operations and the jq constructs shown in the following table.
Notes
References
Bibliography
*
*
*
Others
External links
jq homepagegojq- the Pure Go implementation
jq FAQAwesome jq- curated listing of jq-related resources
The jq Programming Language pageon the
Rosetta Code
Rosetta Code is a wiki-based programming website with implementations of common algorithms and solutions to various programming problems in many different programming languages. It is named for the Rosetta Stone, which has the same text inscribe ...
comparative programming tasks project site
{{DEFAULTSORT:jq (Programming Language)
Dynamically typed programming languages
Functional languages
Programming languages
Programming languages created in 2012
Query_languages
2012 software