Singularity is a free and

open-source Open source is source code that is made freely available for possible modification and redistribution. Products include permission to use the source code, design documents, or content of the product. The open-source model is a decentralized sof ...

computer program that performs

operating-system-level virtualization OS-level virtualization is an operating system (OS) paradigm in which the kernel allows the existence of multiple isolated user space instances, called ''containers'' (LXC, Solaris containers, Docker, Podman), ''zones'' (Solaris containers), '' ...

also known as containerization. One of the main uses of Singularity is to bring containers and

reproducibility Reproducibility, also known as replicability and repeatability, is a major principle underpinning the scientific method. For the findings of a study to be reproducible means that results obtained by an experiment or an observational study or in a ...

to scientific computing and the

high-performance computing High-performance computing (HPC) uses supercomputers and computer clusters to solve advanced computation problems. Overview HPC integrates systems administration (including network and security knowledge) and parallel programming into a multi ...

(HPC) world. The need for reproducibility requires the ability to use containers to move applications from system to system. Using Singularity containers, developers can work in reproducible environments of their choosing and design, and these complete environments can easily be copied and executed on other platforms. In 2021 the Singularity open source project split into two projects called Apptainer and SingularityCE. Singularity container workflow

History

Singularity began as an open-source project in 2015, when a team of researchers at

Lawrence Berkeley National Laboratory Lawrence Berkeley National Laboratory (LBNL), commonly referred to as the Berkeley Lab, is a United States national laboratory that is owned by, and conducts scientific research on behalf of, the United States Department of Energy. Located in ...

, led by Gregory Kurtzer, developed the initial version written in the C programming language and released it under the

BSD license BSD licenses are a family of permissive free software licenses, imposing minimal restrictions on the use and distribution of covered software. This is in contrast to copyleft licenses, which have share-alike requirements. The original BSD li ...

. By the end of 2016, many developers from different research facilities joined forces with the team at Lawrence Berkeley National Laboratory to further the development of Singularity. Singularity quickly attracted the attention of computing-heavy scientific institutions worldwide: * Stanford University Research Computing Center deployed Singularity on their XStream and Sherlock clusters *

National Institutes of Health The National Institutes of Health, commonly referred to as NIH (with each letter pronounced individually), is the primary agency of the United States government The federal government of the United States (U.S. federal government or U ...

installed Singularity on Biowulf, their 95,000+ core/30 PB Linux cluster * Various sites of the Open Science Grid Consortium including

Fermilab Fermi National Accelerator Laboratory (Fermilab), located just outside Batavia, Illinois, near Chicago, is a United States Department of Energy United States Department of Energy National Labs, national laboratory specializing in high-energy parti ...

started adopting Singularity; by April 2017, Singularity was deployed on 60% of the Open Science Grid network. For two years in a row, in 2016 and 2017, Singularity was recognized by HPCwire editors as "One of five new technologies to watch". In 2017 Singularity also won the first place for the category "Best HPC Programming Tool or Technology". based on the data entered on a voluntary basis in a public registry, Singularity user base was estimated to be greater than 25,000 installations and included users at academic institutions such as

Ohio State University The Ohio State University, commonly called Ohio State or OSU, is a public land-grant research university in Columbus, Ohio. A member of the University System of Ohio, it has been ranked by major institutional rankings among the best pu ...

and

Michigan State University Michigan State University (Michigan State, MSU) is a public university, public Land-grant university, land-grant research university in East Lansing, Michigan. It was founded in 1855 as the Agricultural College of the State of Michigan, the fi ...

, as well as top HPC centers like

Texas Advanced Computing Center The Texas Advanced Computing Center (TACC) at the University of Texas at Austin, United States, is an advanced computing research center that provides comprehensive advanced computing resources and support services to researchers in Texas and acr ...

San Diego Supercomputer Center The San Diego Supercomputer Center (SDSC) is an organized research unit of the University of California, San Diego (UCSD). SDSC is located at the UCSD campus' Eleanor Roosevelt College east end, immediately north the Hopkins Parking Structure ...

, and

Oak Ridge National Laboratory Oak Ridge National Laboratory (ORNL) is a U.S. multiprogram science and technology national laboratory sponsored by the U.S. Department of Energy (DOE) and administered, managed, and operated by UT–Battelle as a federally funded research an ...

. In February 2018 the Sylabs company, founded by the Singularity author, was announced to provide commercial support for Singularity. In October of that year Sylabs released version 3.0.0 which was a rewrite in the Go programming language. In May 2020 Gregory Kurtzer left Sylabs but retained leadership of the Singularity open source project. In May 2021 Sylabs made a

fork In cutlery or kitchenware, a fork (from la, furca ' pitchfork') is a utensil, now usually made of metal, whose long handle terminates in a head that branches into several narrow and often slightly curved tines with which one can spear foods ...

of the project and called it SingularityCE. In November 2021 the Singularity open source project joined the

Linux Foundation The Linux Foundation (LF) is a non-profit technology consortium founded in 2000 as a merger between Open Source Development Labs and the Free Standards Group to standardize Linux, support its growth, and promote its commercial adoption. Addi ...

and was renamed to Apptainer.

Features

Singularity is able to support natively high-performance interconnects, such as

InfiniBand InfiniBand (IB) is a computer networking communications standard used in high-performance computing that features very high throughput and very low latency. It is used for data interconnect both among and within computers. InfiniBand is also us ...

and Intel Omni-Path Architecture (OPA). Similar to the support for InfiniBand and Intel OPA devices, Singularity can support any

PCIe PCI Express (Peripheral Component Interconnect Express), officially abbreviated as PCIe or PCI-e, is a high-speed serial computer expansion bus standard, designed to replace the older PCI, PCI-X and AGP bus standards. It is the common m ...

-attached device within the compute node, such as graphic accelerators. Singularity also has native support for Open MPI library by utilizing a hybrid MPI container approach where OpenMPI exists both inside and outside the container. These features make Singularity increasingly useful in areas such as

machine learning Machine learning (ML) is a field of inquiry devoted to understanding and building methods that 'learn', that is, methods that leverage data to improve performance on some set of tasks. It is seen as a part of artificial intelligence. Machine ...

, deep learning and most data-intensive workloads where the applications benefit from the high bandwidth and low latency characteristics of these technologies.

Integration

HPC systems traditionally already have resource management and job scheduling systems in place, so the container runtime environments must be integrated into the existing system resource manager. Using other enterprise container solutions like Docker in HPC systems would require modifications to the software. Docker containers can be automatically converted to stand-alone singularity files which can then be submitted to HPC resource managers. Singularity seamlessly integrates with many resource managers including: *

HTCondor HTCondor is an open-source high-throughput computing software framework for coarse-grained distributed parallelization of computationally intensive tasks. It can be used to manage workload on a dedicated cluster of computers, or to farm out wor ...

* Oracle Grid Engine (SGE) * SLURM (Simple Linux Utility for Resource Management) *

TORQUE In physics and mechanics, torque is the rotational equivalent of linear force. It is also referred to as the moment of force (also abbreviated to moment). It represents the capability of a force to produce change in the rotational motion of t ...

(Terascale Open-source Resource and QUEue Manager) *

PBS Pro Portable Batch System (or simply PBS) is the name of computer software that performs job scheduling. Its primary task is to allocate computational tasks, i.e., batch jobs, among the available computing resources. It is often used in conjunction ...

(PBS Professional)
HashiCorp Nomad
(A simple and flexible workload orchestrator)

References

External links

* {{Linux containers Free software programmed in Go Linux containerization Operating system technology Operating system security Software using the Apache license Software using the BSD license Virtualization software Virtualization software for Linux

History

Features

Integration

See also

References

Further reading

External links