Programming with Big Data in R (pbdR) is a series of
R packages and an environment for
statistical computing
Computational statistics, or statistical computing, is the bond between statistics and computer science. It means statistical methods that are enabled by using computational methods. It is the area of computational science (or scientific computi ...
with
big data by using high-performance statistical computation. The pbdR uses the same programming language as R with
S3/S4 classes and methods which is used among
statistician
A statistician is a person who works with theoretical or applied statistics. The profession exists in both the private and public sectors.
It is common to combine statistical knowledge with expertise in other subjects, and statisticians may wor ...
s and
data miners for developing
statistical software
Statistical software are specialized computer programs for analysis in statistics and econometrics.
Open-source
* ADaMSoft – a generalized statistical software with data mining algorithms and methods for data management
* ADMB – a softwa ...
. The significant difference between pbdR and R code is that pbdR mainly focuses on
distributed memory
In computer science, distributed memory refers to a multiprocessor computer system in which each processor has its own private memory. Computational tasks can only operate on local data, and if remote data are required, the computational task m ...
systems, where data are distributed across several processors and analyzed in a
batch mode
Computerized batch processing is a method of running software programs called jobs in batches automatically. While users are required to submit the jobs, no other interaction by the user is required to process the batch. Batches may automatically ...
, while communications between processors are based on
MPI
MPI or Mpi may refer to:
Science and technology Biology and medicine
* Magnetic particle imaging, an emerging non-invasive tomographic technique
* Myocardial perfusion imaging, a nuclear medicine procedure that illustrates the function of the hear ...
that is easily used in large
high-performance computing (HPC) systems. R system mainly focuses on single
multi-core
A multi-core processor is a microprocessor on a single integrated circuit with two or more separate processing units, called cores, each of which reads and executes program instructions. The instructions are ordinary CPU instructions (such ...
machines for data analysis via an interactive mode such as
GUI interface.
Two main implementations in
R using
MPI
MPI or Mpi may refer to:
Science and technology Biology and medicine
* Magnetic particle imaging, an emerging non-invasive tomographic technique
* Myocardial perfusion imaging, a nuclear medicine procedure that illustrates the function of the hear ...
are Rmpi
and pbdMPI of pbdR.
* The pbdR built on pbdMPI uses
SPMD parallelism where every processor is considered as worker and owns parts of data. The
SPMD parallelism introduced in mid 1980 is particularly efficient in homogeneous computing environments for large data, for example, performing
singular value decomposition
In linear algebra, the singular value decomposition (SVD) is a factorization of a real or complex matrix. It generalizes the eigendecomposition of a square normal matrix with an orthonormal eigenbasis to any \ m \times n\ matrix. It is r ...
on a large matrix, or performing
clustering analysis on high-dimensional large data. On the other hand, there is no restriction to use
manager/workers parallelism in
SPMD parallelism environment.
* The Rmpi
[ uses manager/workers parallelism where one main processor (manager) serves as the control of all other processors (workers). The manager/workers parallelism introduced around early 2000 is particularly efficient for large tasks in small clusters, for example, ]bootstrap method
Bootstrapping is any test or metric that uses random sampling with replacement (e.g. mimicking the sampling process), and falls under the broader class of resampling methods. Bootstrapping assigns measures of accuracy (bias, variance, confidenc ...
and Monte Carlo simulation
Monte Carlo methods, or Monte Carlo experiments, are a broad class of computational algorithms that rely on repeated random sampling to obtain numerical results. The underlying concept is to use randomness to solve problems that might be determ ...
in applied statistics since i.i.d. assumption is commonly used in most statistical analysis
Statistical inference is the process of using data analysis to infer properties of an underlying distribution of probability.Upton, G., Cook, I. (2008) ''Oxford Dictionary of Statistics'', OUP. . Inferential statistical analysis infers propertie ...
. In particular, task pull parallelism has better performance for Rmpi in heterogeneous computing environments.
The idea of SPMD parallelism is to let every processor do the same amount of work, but on different parts of a large data set. For example, a modern GPU is a large collection of slower co-processors that can simply apply the same computation on different parts of relatively smaller data, but the SPMD parallelism ends up with an efficient way to obtain final solutions (i.e. time to solution is shorter).
Package design
Programming with pbdR requires usage of various packages developed by pbdR core team. Packages developed are the following.
Among these packages, pbdMPI provides wrapper functions to MPI
MPI or Mpi may refer to:
Science and technology Biology and medicine
* Magnetic particle imaging, an emerging non-invasive tomographic technique
* Myocardial perfusion imaging, a nuclear medicine procedure that illustrates the function of the hear ...
library, and it also produces a shared library
In computer science, a library is a collection of non-volatile memory, non-volatile resources used by computer programs, often for software development. These may include configuration data, documentation, help data, message templates, Code r ...
and a configuration file for MPI environments. All other packages rely on this configuration for installation and library loading that avoids difficulty of library linking and compiling. All other packages can directly use MPI functions easily.
* pbdMPI --- an efficient interface to MPI either OpenMPI or MPICH2
MPICH, formerly known as MPICH2, is a freely available, portable implementation of MPI, a standard for message-passing for distributed-memory applications used in parallel computing. MPICH is Free and open source software with some public domain c ...
with a focus on Single Program/Multiple Data ( SPMD) parallel programming style
* pbdSLAP --- bundles scalable dense linear algebra libraries in double precision for R, based on ScaLAPACK The ScaLAPACK (or Scalable LAPACK) library includes a subset of LAPACK routines redesigned for distributed memory MIMD parallel computers. It is currently written in a Single-Program-Multiple-Data style using explicit message passing for interproce ...
version 2.0.2 which includes several scalable linear algebra packages (namely BLACS, PBLAS, and ScaLAPACK The ScaLAPACK (or Scalable LAPACK) library includes a subset of LAPACK routines redesigned for distributed memory MIMD parallel computers. It is currently written in a Single-Program-Multiple-Data style using explicit message passing for interproce ...
).
* pbdNCDF4 --- interface to Parallel Unidata NetCDF
NetCDF (Network Common Data Form) is a set of software libraries and self-describing, machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data. The project homepage is hosted by the Unidata ...
4 format data files
* pbdBASE --- low-level ScaLAPACK The ScaLAPACK (or Scalable LAPACK) library includes a subset of LAPACK routines redesigned for distributed memory MIMD parallel computers. It is currently written in a Single-Program-Multiple-Data style using explicit message passing for interproce ...
codes and wrappers
* pbdDMAT --- distributed matrix classes and computational methods, with a focus on linear algebra and statistics
* pbdDEMO --- set of package demonstrations and examples, and this unifying vignette
* pmclust --- parallel model-based clustering using pbdR
* pbdPROF --- profiling package for MPI codes and visualization of parsed stats
* pbdZMQ --- interface to ØMQ
* remoter --- R client with remote R servers
* pbdCS --- pbdR client with remote pbdR servers
* pbdRPC --- remote procedure call
* kazaam --- very tall and skinny distributed matrices
* pbdML --- machine learning toolbox
Among those packages, the pbdDEMO package is a collection of 20+ package demos which offer example uses of the various pbdR packages, and contains a vignette that offers detailed explanations for the demos and provides some mathematical or statistical insight.
Examples
Example 1
Hello World! Save the following code in a file called "demo.r"
### Initial MPI
library(pbdMPI, quiet = TRUE)
init()
comm.cat("Hello World!\n")
### Finish
finalize()
and use the command
mpiexec -np 2 Rscript demo.r
to execute the code where Rscript is one of command line executable program.
Example 2
The following example modified from pbdMPI illustrates the basic syntax of the language of pbdR.
Since pbdR is designed in SPMD, all the R scripts are stored in files and executed from the command line via mpiexec, mpirun, etc. Save the following code in a file called "demo.r"
### Initial MPI
library(pbdMPI, quiet = TRUE)
init()
.comm.size <- comm.size()
.comm.rank <- comm.rank()
### Set a vector x on all processors with different values
N <- 5
x <- (1:N) + N * .comm.rank
### All reduce x using summation operation
y <- allreduce(as.integer(x), op = "sum")
comm.print(y)
y <- allreduce(as.double(x), op = "sum")
comm.print(y)
### Finish
finalize()
and use the command
mpiexec -np 4 Rscript demo.r
to execute the code where Rscript is one of command line executable program.
Example 3
The following example modified from pbdDEMO illustrates the basic ddmatrix computation of pbdR which performs singular value decomposition
In linear algebra, the singular value decomposition (SVD) is a factorization of a real or complex matrix. It generalizes the eigendecomposition of a square normal matrix with an orthonormal eigenbasis to any \ m \times n\ matrix. It is r ...
on a given matrix.
Save the following code in a file called "demo.r"
# Initialize process grid
library(pbdDMAT, quiet=T)
if(comm.size() != 2)
comm.stop("Exactly 2 processors are required for this demo.")
init.grid()
# Setup for the remainder
comm.set.seed(diff=TRUE)
M <- N <- 16
BL <- 2 # blocking --- passing single value BL assumes BLxBL blocking
dA <- ddmatrix("rnorm", nrow=M, ncol=N, mean=100, sd=10)
# LA SVD
svd1 <- La.svd(dA)
comm.print(svd1$d)
# Finish
finalize()
and use the command
mpiexec -np 2 Rscript demo.r
to execute the code where Rscript is one of command line executable program.
Further reading
*
*
*
*
*
This article was read 22,584 times in 2012 since it posted on October 16, 2012 and ranked number 3
*
*
*
References
External links
*
{{DEFAULTSORT:PbdR
Cross-platform free software
Data mining and machine learning software
Data-centric programming languages
Free statistical software
Functional languages
Numerical analysis software for Linux
Numerical analysis software for macOS
Numerical analysis software for Windows
Parallel computing