Heritrix is a

web crawler Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web and that is typically operated by search engines for the purpose of Web indexing (''web spider ...

designed for

web archiving Web archiving is the process of collecting, preserving, and providing access to material from the World Wide Web. The aim is to ensure that information is preserved in an archival format for research and the public. Web archivists typically ...

. It was written by the

Internet Archive The Internet Archive is an American 501(c)(3) organization, non-profit organization founded in 1996 by Brewster Kahle that runs a digital library website, archive.org. It provides free access to collections of digitized media including web ...

. It is available under a

free software license A free-software license is a notice that grants the recipient of a piece of software extensive rights to modify and redistribute that software. These actions are usually prohibited by copyright law, but the rights-holder (usually the author) ...

and written in

Java Java is one of the Greater Sunda Islands in Indonesia. It is bordered by the Indian Ocean to the south and the Java Sea (a part of Pacific Ocean) to the north. With a population of 156.9 million people (including Madura) in mid 2024, proje ...

. The main interface is accessible using a

web browser A web browser, often shortened to browser, is an application for accessing websites. When a user requests a web page from a particular website, the browser retrieves its files from a web server and then displays the page on the user's scr ...

, and there is a

command-line tool A console application or command-line program is a computer program (applications or utilities) designed to be used via a text-only user interface. A console application can be used with a computer terminal, a system console, or a terminal emu ...

that can optionally be used to initiate crawls. Heritrix was developed jointly by the Internet Archive and the Nordic national libraries on specifications written in early 2003. The first official release was in January 2004, and it has been continually improved by employees of the Internet Archive and other interested parties. For many years Heritrix was not the main crawler used to crawl content for the Internet Archive's web collection. The largest contributor to the collection, as of 2011, is

Alexa Internet Alexa Internet, Inc. was a web traffic analysis company based in San Francisco, California. It was founded as an independent company by Brewster Kahle and Bruce Gilliat in 1996. Alexa provided web traffic data, global rankings, and other info ...

. Alexa crawls the web for its own purposes, using a crawler named ''ia_archiver''. Alexa then donates the material to the Internet Archive. The Internet Archive itself did some of its own crawling using Heritrix, but only on a smaller scale. Starting in 2008, the Internet Archive began performance improvements to do its own wide scale crawling, and now does collect most of its content.

Projects using Heritrix

A number of organizations and national libraries are using Heritrix, among them: *

Austrian National Library The Austrian National Library (, ) is the largest library in Austria, with more than 12 million items in its various collections. The library is located in the Hofburg#Neue Burg, Neue Burg Wing of the Hofburg in Innere Stadt, center of Vienna. Sin ...

, Web Archiving *

Bibliotheca Alexandrina The Bibliotheca Alexandrina (Latin, 'Library of Alexandria'; , ) (BA) is a major library and cultural center on the shore of the Mediterranean Sea in Alexandria, Egypt. It is a commemoration of the Library of Alexandria, once one of the larg ...

's Internet Archive *

Bibliothèque nationale de France The (; BnF) is the national library of France, located in Paris on two main sites, ''Richelieu'' and ''François-Mitterrand''. It is the national repository of all that is published in France. Some of its extensive collections, including bo ...

British Library The British Library is the national library of the United Kingdom. Based in London, it is one of the largest libraries in the world, with an estimated collection of between 170 and 200 million items from multiple countries. As a legal deposit li ...

* California Digital Library's Web Archiving Service *

CiteSeerX CiteSeerX (formerly called CiteSeer) is a public search engine and digital library for scientific and academic papers, primarily in the fields of computer and information science. CiteSeer's goal is to improve the dissemination and access of a ...

* Documenting Internet2 *

Internet Memory Foundation The Internet Memory Foundation (formerly the European Archive Foundation) was a non-profit foundation whose purpose was archiving content of the World Wide Web. It hosted projects and research that included the preservation and protection of d ...

Library and Archives Canada Library and Archives Canada (LAC; ) is the federal institution tasked with acquiring, preserving, and providing accessibility to the documentary heritage of Canada. The national archive and library is the 16th largest library in the world. T ...

Library of Congress The Library of Congress (LOC) is a research library in Washington, D.C., serving as the library and research service for the United States Congress and the ''de facto'' national library of the United States. It also administers Copyright law o ...

National and University Library of Iceland ( Icelandic: ; English: ''The National and University Library of Iceland'') is the national library of Iceland which also functions as the university library of the University of Iceland. The library was established on 1 December 1994 in Reykjav ...

National Library of Finland The National Library of Finland (, ) is the foremost research library in Finland. Administratively the library is part of the University of Helsinki. From 1919 to 1 August 2006, it was known as the Helsinki University Library (). The Nationa ...

National Library of New Zealand The National Library of New Zealand () is charged with the obligation to "enrich the cultural and economic life of New Zealand and its interchanges with other nations" (National Library of New Zealand (Te Puna Mātauranga) Act 2003). Under the ...

Royal Library of the Netherlands The KB National Library of the Netherlands (legal Dutch name: Koninklijke Bibliotheek or KB ; ''Royal Library'') is the national library of the Netherlands, based in The Hague, founded in 1798. The KB collects everything that is published in ...

(Koninklijke Bibliotheek) * Netarkivet.dk *

National Library of Israel The National Library of Israel (NLI; ; ), formerly Jewish National and University Library (JNUL; ), is the library dedicated to collecting the cultural treasures of Israel and of Judaism, Jewish Cultural heritage, heritage. The library holds more ...

Arc files

Older versions of Heritrix by default stored the web resources it crawls in an Arc file. This file format is wholly unrelated to

ARC (file format) ARC is a Lossless compression, lossless data compression and file archiver, archival file format, format by System Enhancement Associates (SEA). The file format and the program were both called ARC. The format is known as the subject of controver ...

. This format has been used by the Internet Archive since 1996 to store its web archives. More recently it saves by default in the WARC file format, which is similar to ARC but more precisely specified and more flexible. Heritrix can also be configured to store files in a directory format similar to the

Wget GNU Wget (or just Wget, formerly Geturl, also written as its package name, wget) is a computer program that retrieves content from web servers. It is part of the GNU Project. Its name derives from "World Wide Web" and " ''get''", a HTTP reque ...

crawler that uses the URL to name the directory and filename of each resource. An Arc file stores multiple archived resources in a single file in order to avoid managing a large number of small files. The file consists of a sequence of URL records, each with a header containing metadata about how the resource was requested followed by the

HTTP header HTTP header fields are a list of strings sent and received by both the client program and server on every HTTP request and response. These headers are usually invisible to the end-user and are only processed or logged by the server and client ...

and the response. Example: filedesc://IA-2006062.arc 0.0.0.0 20060622190110 text/plain 76 1 1 InternetArchive URL IP-address Archive-date Content-type Archive-length http://foo.edu:80/hello.html 127.10.100.2 19961104142103 text/html 187 HTTP/1.1 200 OK Date: Thu, 22 Jun 2006 19:01:15 GMT Server: Apache Last-Modified: Sat, 10 Jun 2006 22:33:11 GMT Content-Length: 30 Content-Type: text/html Hello World!!!

Tools for processing Arc files

Heritrix includes a command-line tool called arcreader which can be used to extract the contents of an Arc file. The following command lists all the URLs and metadata stored in the given Arc file (i
CDX
format): arcreader IA-2006062.arc The following command extracts hello.html from the above example assuming the record starts at offset 140: arcreader -o 140 -f dump IA-2006062.arc Other tools:
Arc processing tools

WERA (Web ARchive Access)

Command-line tools

Heritrix comes with several command-line tools: * htmlextractor – displays the links Heritrix would extract for a given URL * hoppath.pl – recreates the hop path (path of links) to the specified URL from a completed crawl * manifest_bundle.pl – bundles up all resources referenced by a crawl manifest file into an uncompressed or compressed tar ball * cmdline-jmxclient – enables command-line control of Heritrix * arcreader – extracts contents of ARC files (see above) Further tools are available as part of the Internet Archive's warctools project.

References

External links