HOME

TheInfoList



OR:

In
computer science Computer science is the study of computation, information, and automation. Computer science spans Theoretical computer science, theoretical disciplines (such as algorithms, theory of computation, and information theory) to Applied science, ...
, a sparse file is a type of
computer file A computer file is a System resource, resource for recording Data (computing), data on a Computer data storage, computer storage device, primarily identified by its filename. Just as words can be written on paper, so too can data be written to a ...
that attempts to use file system space more efficiently when the file itself is partially empty. This is achieved by writing brief information (
metadata Metadata (or metainformation) is "data that provides information about other data", but not the content of the data itself, such as the text of a message or the image itself. There are many distinct types of metadata, including: * Descriptive ...
) ''representing'' the empty blocks to the data storage media instead of the actual "empty" space which makes up the block, thus consuming less storage space. The full block is written to the media as the actual size only when the block contains "real" (non-empty) data. Most commonly, sparse files are created when blocks of the file are never written to. This is typical for random-access files like databases. Some operating systems or utilities go further by "sparsifying" files when writing or copying them: if a block contains only null bytes, it is not written to storage but rather marked as empty. When reading sparse files, the file system transparently converts metadata representing empty blocks into "real" blocks filled with null bytes at runtime. The application is unaware of this conversion. Most modern file systems support sparse files, including most
Unix Unix (, ; trademarked as UNIX) is a family of multitasking, multi-user computer operating systems that derive from the original AT&T Unix, whose development started in 1969 at the Bell Labs research center by Ken Thompson, Dennis Ritchie, a ...
variants and
NTFS NT File System (NTFS) (commonly called ''New Technology File System'') is a proprietary journaling file system developed by Microsoft in the 1990s. It was developed to overcome scalability, security and other limitations with File Allocation Tabl ...
. Apple's
HFS+ HFS Plus or HFS+ (also known as Mac OS Extended or HFS Extended) is a journaling file system developed by Apple Inc. It replaced the Hierarchical File System (HFS) as the primary file system of Apple computers with the 1998 release of Mac OS 8. ...
does not provide support for sparse files, but in OS X, the
virtual file system A virtual file system (VFS) or virtual filesystem switch is an abstract layer on top of a more concrete file system. The purpose of a VFS is to allow client applications to access different types of concrete file systems in a uniform way. A VFS ...
layer supports storing them in any supported file system, including HFS+.
Apple File System Apple File System (APFS) is a Proprietary software, proprietary file system developed and deployed by Apple Inc. for macOS macOS Sierra, Sierra (10.12.4) and later, iOS iOS 10 , 10.3, tvOS 10.2, watchOS 3.2, and all versions of iPadOS. It aim ...
(APFS) also supports them. Sparse files are commonly used for disk images, database snapshots, log files and in scientific applications.


Advantages

The advantage of sparse files is that storage space is only allocated when actually needed: Storage capacity is conserved, and large files can occasionally be created even if insufficient free space for the original file is available on the storage media. This also reduces the time of the first write as the system does not have to allocate blocks for the "skipped" space. If the initial allocation requires writing all zeros to the space, it also keeps the system from having to write over the "skipped" space twice. For example, a
virtual machine In computing, a virtual machine (VM) is the virtualization or emulator, emulation of a computer system. Virtual machines are based on computer architectures and provide the functionality of a physical computer. Their implementations may involve ...
image with max size of 100 GB that has 2 GB of files actually written would require the full 100 GB when backed by pre-allocated storage, yet only 2 GB on a sparse file. If the file system supports hole punching and the guest operating system issues TRIM commands, deleting files on the guest will accordingly reduce the space needed.


Disadvantages

Disadvantages are that sparse files may become fragmented; file system free space reports may be misleading; filling up file systems containing sparse files can have unexpected effects (such as disk-full or quota-exceeded errors when merely overwriting an existing portion of a file that happened to have been sparse); and copying a sparse file with a program that does not explicitly support them may copy the entire, uncompressed size of the file, including the zero sections which are not allocated on the storage media—losing the benefits of the sparse property in the file. Sparse files are also not fully supported by all backup software or applications. However, the VFS implementation sidesteps the prior two disadvantages. Loading executables on 32 bit Windows (exe or dll) which are sparse takes a much longer time since the file cannot be memory mapped in the limited 4 GB address space, and are not cached as there is no codepath for caching 32 bit sparse executables (Windows on 64 bit architectures can map sparse executables). On NTFS sparse files (or rather their non-zero areas) cannot be compressed. NTFS implements sparseness as a special kind of compression so a file may be either sparse or compressed.


Sparse files in Unix

Sparse files are typically handled transparently to the user. But the differences between a normal file and sparse file become apparent in some situations.


Creation

The
Unix Unix (, ; trademarked as UNIX) is a family of multitasking, multi-user computer operating systems that derive from the original AT&T Unix, whose development started in 1969 at the Bell Labs research center by Ken Thompson, Dennis Ritchie, a ...
command dd of=sparse-file bs=5M seek=1 count=0 will create a file of five
mebibyte The byte is a units of information, unit of digital information that most commonly consists of eight bits. Historically, the byte was the number of bits used to encode a single character (computing), character of text in a computer and for this ...
s in size, but with no data stored on the media (only
metadata Metadata (or metainformation) is "data that provides information about other data", but not the content of the data itself, such as the text of a message or the image itself. There are many distinct types of metadata, including: * Descriptive ...
). (
GNU GNU ( ) is an extensive collection of free software (394 packages ), which can be used as an operating system or can be used in parts with other operating systems. The use of the completed GNU tools led to the family of operating systems popu ...
dd has this behavior because it calls ftruncate to set the file size; other implementations may merely create an empty file.) Similarly the truncate command may be used, if available: truncate -s 5M On
Linux Linux ( ) is a family of open source Unix-like operating systems based on the Linux kernel, an kernel (operating system), operating system kernel first released on September 17, 1991, by Linus Torvalds. Linux is typically package manager, pac ...
, an existing file can be converted to sparse by: fallocate -d There is no portable system call to punch holes; Linux provides fallocate(FALLOC_FL_PUNCH_HOLE), and Solaris provides fcntl(F_FREESP).


Detection

The -s option of the ls command shows the occupied space in blocks. ls -ls sparse-file Alternatively, the du command prints the occupied space, while ls prints the apparent size. In some non-standard versions of du, the option prints the occupied space in bytes instead of blocks, so that it can be compared to the ls output: du --block-size=1 sparse-file ls -l sparse-file Note the above du usage has the abbreviated option syntax format "du -B 1 sf", itself equivalent to the shortest version "du -b sf" as stated in the du manual: is equivalent to . Also, the tool filefrag from e2fsprogs package can be used to show block allocation details of the file. filefrag -v sparse-file


Copying

Normally, the GNU version of cp is good at detecting whether a file is sparse, so cp sparse-file new-file creates new-file, which will be sparse. GNU also has a --sparse option, which is especially useful if a file containing long zero blocks is saved in a non-sparse way (i.e. the zero blocks have been written to the storage media in full). Storage space can be conserved by doing: cp --sparse=always file1 file1_sparsed Some cp implementations, like
FreeBSD FreeBSD is a free-software Unix-like operating system descended from the Berkeley Software Distribution (BSD). The first version was released in 1993 developed from 386BSD, one of the first fully functional and free Unix clones on affordable ...
's cp, do not support the --sparse option and will always expand sparse files. A partially viable alternative on those systems is to use
rsync rsync (remote sync) is a utility for transferring and synchronizing files between a computer and a storage drive and across networked computers by comparing the modification times and sizes of files. It is commonly found on Unix-like opera ...
with its own --sparse option instead of cp. Unfortunately, older versions of rsync do not support --sparse combined with --inplace. Via standard input, sparse file copying is achieved as follows: cp --sparse=always /dev/fd/0 new-sparse-file < somefile


See also

*
Comparison of file systems The following tables compare general and technical information for a number of file systems. General information Metadata All widely used file systems record a last modified time stamp (also known as "mtime"). It is not included i ...


References


External links

*
Creating sparse files in Windows Server using fsutil



View the Size of the Sparse File of a Database Snapshot

SEEK_HOLE or FIEMAP: Detecting holes in sparse files

virtsync is a commercial solution to rsync's --sparse and --inplace issue.


* ttps://www.vanheusden.com/Linux/phantom.php Phantom – a program to convert files to sparse files to reduce storage consumption
ArchLinux Wiki: Sparse file
{{DEFAULTSORT:Sparse file Computer files