TheInfoList

In
computer science Computer science deals with the theoretical foundations of information, algorithms and the architectures of its computation as well as practical techniques for their application. Computer science is the study of computation, automation, a ...
, the analysis of algorithms is the process of finding the computational complexity of algorithms – the amount of time, storage, or other resources needed to . Usually, this involves determining a
function Function or functionality may refer to: Computing * Function key A function key is a key on a computer A computer is a machine that can be programmed to carry out sequences of arithmetic or logical operations automatically. Modern comp ...
that relates the length of an algorithm's input to the number of steps it takes (its
time complexity In computer science Computer science deals with the theoretical foundations of information, algorithms and the architectures of its computation as well as practical techniques for their application. Computer science is the study of com ...
) or the number of storage locations it uses (its
space complexityThe space complexity of an algorithm or a computer program is the amount of memory space required to solve an instance of the computational problem as a function of characteristics of the input. It is the memory required by an algorithm until it exec ...
). An algorithm is said to be efficient when this function's values are small, or grow slowly compared to a growth in the size of the input. Different inputs of the same length may cause the algorithm to have different behavior, so
best, worst and average case In computer science Computer science deals with the theoretical foundations of information, algorithms and the architectures of its computation as well as practical techniques for their application. Computer science is the study of , , ...
descriptions might all be of practical interest. When not otherwise specified, the function describing the performance of an algorithm is usually an
upper bound In mathematics, particularly in order theory Order theory is a branch of mathematics Mathematics (from Ancient Greek, Greek: ) includes the study of such topics as quantity (number theory), mathematical structure, structure (algebra), s ...
, determined from the worst case inputs to the algorithm. The term "analysis of algorithms" was coined by
Donald Knuth Donald Ervin Knuth ( ; born January 10, 1938) is an American computer scientist A computer scientist is a person A person (plural people or persons) is a being that has certain capacities or attributes such as reason, morality, consciousnes ...
. Algorithm analysis is an important part of a broader
computational complexity theory Computational complexity theory focuses on classifying computational problems according to their resource usage, and relating these classes to each other. A computational problem is a task solved by a computer. A computation problem is solvable by ...
, which provides theoretical estimates for the resources needed by any algorithm which solves a given
computational problem In theoretical computer science Image:Maquina.png, An artistic representation of a Turing machine. Turing machines are used to model general computing devices. Theoretical computer science (TCS) is a subset of general computer science and mathema ...
. These estimates provide an insight into reasonable directions of search for efficient algorithms. In theoretical analysis of algorithms it is common to estimate their complexity in the asymptotic sense, i.e., to estimate the complexity function for arbitrarily large input.
Big O notation Big O notation is a mathematical notation that describes the limiting behavior of a function Function or functionality may refer to: Computing * Function key A function key is a key on a computer A computer is a machine that can be p ...
, Big-omega notation and
Big-theta notation Big O notation is a mathematical notation that describes the limiting behavior of a function when the argument In logic Logic (from Ancient Greek, Greek: grc, wikt:λογική, λογική, label=none, lit=possessed of reason, intel ...
are used to this end. For instance,
binary search In computer science, binary search, also known as half-interval search, logarithmic search, or binary chop, is a search algorithm that finds the position of a target value within a sorted array. Binary search compares the target value to the m ...

is said to run in a number of steps proportional to the logarithm of the length of the sorted list being searched, or in O(log(n)), colloquially "in
logarithmic time In computer science, the time complexity is the computational complexity that describes the amount of computer time it takes to run an algorithm. Time complexity is commonly estimated by counting the number of elementary operations performed by th ...
". Usually
asymptotic 250px, A curve intersecting an asymptote infinitely many times. In analytic geometry In classical mathematics, analytic geometry, also known as coordinate geometry or Cartesian geometry, is the study of geometry Geometry (from the grc ...
estimates are used because different
implementation Implementation is the realization of an application, or execution of a plan A plan is typically any diagram or list of steps with details of timing and resources, used to achieve an Goal, objective to do something. It is commonly understood as ...

s of the same algorithm may differ in efficiency. However the efficiencies of any two "reasonable" implementations of a given algorithm are related by a constant multiplicative factor called a ''hidden constant''. Exact (not asymptotic) measures of efficiency can sometimes be computed but they usually require certain assumptions concerning the particular implementation of the algorithm, called
model of computation A model is an informative representation of an object, person or system. The term originally denoted the Plan_(drawing), plans of a building in late 16th-century English, and derived via French and Italian ultimately from Latin ''modulus'', a meas ...
. A model of computation may be defined in terms of an abstract computer, e.g.,
Turing machine A Turing machine is a mathematical model of computation Mathematics (from Greek: ) includes the study of such topics as numbers (arithmetic and number theory), formulas and related structures (algebra), shapes and spaces in which they are ...

, and/or by postulating that certain operations are executed in unit time. For example, if the sorted list to which we apply binary search has ''n'' elements, and we can guarantee that each lookup of an element in the list can be done in unit time, then at most log2 ''n'' + 1 time units are needed to return an answer.

# Cost models

Time efficiency estimates depend on what we define to be a step. For the analysis to correspond usefully to the actual execution time, the time required to perform a step must be guaranteed to be bounded above by a constant. One must be careful here; for instance, some analyses count an addition of two numbers as one step. This assumption may not be warranted in certain contexts. For example, if the numbers involved in a computation may be arbitrarily large, the time required by a single addition can no longer be assumed to be constant. Two cost models are generally used:, section 1.3 * the uniform cost model, also called uniform-cost measurement (and similar variations), assigns a constant cost to every machine operation, regardless of the size of the numbers involved * the logarithmic cost model, also called logarithmic-cost measurement (and similar variations), assigns a cost to every machine operation proportional to the number of bits involved The latter is more cumbersome to use, so it's only employed when necessary, for example in the analysis of
arbitrary-precision arithmetic In computer science, arbitrary-precision arithmetic, also called bignum arithmetic, multiple-precision arithmetic, or sometimes infinite-precision arithmetic, indicates that calculations are performed on numbers whose numerical digit, digits of pre ...
algorithms, like those used in
cryptography Cryptography, or cryptology (from grc, , translit=kryptós "hidden, secret"; and ''graphein'', "to write", or ''-logia ''-logy'' is a suffix in the English language, used with words originally adapted from Ancient Greek ending in (''- ...

. A key point which is often overlooked is that published lower bounds for problems are often given for a model of computation that is more restricted than the set of operations that you could use in practice and therefore there are algorithms that are faster than what would naively be thought possible.

# Run-time analysis

Run-time analysis is a theoretical classification that estimates and anticipates the increase in ''
running time In computer science, the time complexity is the computational complexity that describes the amount of computer time it takes to run an algorithm. Time complexity is commonly estimated by counting the number of elementary operations performed by th ...
'' (or run-time) of an
algorithm In and , an algorithm () is a finite sequence of , computer-implementable instructions, typically to solve a class of problems or to perform a computation. Algorithms are always and are used as specifications for performing s, , , and other ...

as its '''' (usually denoted as ''n'') increases. Run-time efficiency is a topic of great interest in
computer science Computer science deals with the theoretical foundations of information, algorithms and the architectures of its computation as well as practical techniques for their application. Computer science is the study of computation, automation, a ...
: A program can take seconds, hours, or even years to finish executing, depending on which algorithm it implements. While software profiling techniques can be used to measure an algorithm's run-time in practice, they cannot provide timing data for all infinitely many possible inputs; the latter can only be achieved by the theoretical methods of run-time analysis.

## Shortcomings of empirical metrics

Since algorithms are
platform-independent In computing Computing is any goal-oriented activity requiring, benefiting from, or creating computing machinery. It includes the study and experimentation of algorithmic processes and development of both computer hardware , hardware and soft ...
(i.e. a given algorithm can be implemented in an arbitrary
programming language A programming language is a formal language In logic, mathematics, computer science, and linguistics, a formal language consists of string (computer science), words whose symbol (formal), letters are taken from an alphabet (computer science) ...

on an arbitrary
computer A computer is a machine that can be programmed to Execution (computing), carry out sequences of arithmetic or logical operations automatically. Modern computers can perform generic sets of operations known as Computer program, programs. These ...

running an arbitrary
operating system An operating system (OS) is system software System software is software designed to provide a platform for other software. Examples of system software include operating systems (OS) like macOS, Linux, Android (operating system), Android and Mi ...

), there are additional significant drawbacks to using an
empirical Empirical evidence for a proposition In logic and linguistics, a proposition is the meaning of a declarative sentence (linguistics), sentence. In philosophy, "Meaning (philosophy), meaning" is understood to be a non-linguistic entity which is s ...
approach to gauge the comparative performance of a given set of algorithms. Take as an example a program that looks up a specific entry in a sorted
list A ''list'' is any set of items. List or lists may also refer to: People * List (surname)List or Liste is a European surname. Notable people with the surname include: List * Friedrich List (1789–1846), German economist * Garrett List (194 ...
of size ''n''. Suppose this program were implemented on Computer A, a state-of-the-art machine, using a
linear search In computer science Computer science deals with the theoretical foundations of information, algorithms and the architectures of its computation as well as practical techniques for their application. Computer science is the study of Alg ...
algorithm, and on Computer B, a much slower machine, using a
binary search algorithm In computer science Computer science deals with the theoretical foundations of information, algorithms and the architectures of its computation as well as practical techniques for their application. Computer science is the study of Al ...
. Benchmark testing on the two computers running their respective programs might look something like the following: Based on these metrics, it would be easy to jump to the conclusion that ''Computer A'' is running an algorithm that is far superior in efficiency to that of ''Computer B''. However, if the size of the input-list is increased to a sufficient number, that conclusion is dramatically demonstrated to be in error: Computer A, running the linear search program, exhibits a
linear Linearity is the property of a mathematical relationship (''function Function or functionality may refer to: Computing * Function key A function key is a key on a computer A computer is a machine that can be programmed to carry out se ...

growth rate. The program's run-time is directly proportional to its input size. Doubling the input size doubles the run time, quadrupling the input size quadruples the run-time, and so forth. On the other hand, Computer B, running the binary search program, exhibits a
logarithm In mathematics Mathematics (from Greek: ) includes the study of such topics as numbers ( and ), formulas and related structures (), shapes and spaces in which they are contained (), and quantities and their changes ( and ). There is no ...

ic growth rate. Quadrupling the input size only increases the run time by a
constant Constant or The Constant may refer to: Mathematics * Constant (mathematics) In mathematics, the word constant can have multiple meanings. As an adjective, it refers to non-variance (i.e. unchanging with respect to some other Value (mathematics ...
amount (in this example, 50,000 ns). Even though Computer A is ostensibly a faster machine, Computer B will inevitably surpass Computer A in run-time because it's running an algorithm with a much slower growth rate.

## Orders of growth

Informally, an algorithm can be said to exhibit a growth rate on the order of a
mathematical function In mathematics Mathematics (from Greek: ) includes the study of such topics as numbers ( and ), formulas and related structures (), shapes and spaces in which they are contained (), and quantities and their changes ( and ). There is no ge ...
if beyond a certain input size ''n'', the function times a positive constant provides an upper bound or limit for the run-time of that algorithm. In other words, for a given input size ''n'' greater than some ''n''0 and a constant ''c'', the running time of that algorithm will never be larger than . This concept is frequently expressed using Big O notation. For example, since the run-time of
insertion sort Insertion sort is a simple sorting algorithm In computer science, a sorting algorithm is an algorithm that puts elements of a List (computing), list in a certain Total order, order. The most frequently used orders are numerical order and lexicogr ...

grows quadratically as its input size increases, insertion sort can be said to be of order ''O''(''n''2). Big O notation is a convenient way to express the
worst-case scenario A worst-case scenario is a concept in risk management wherein the planner, in planning for potential disasters, considers the most severe possible outcome that can reasonably be projected to occur in a given situation. Conceiving of worst-case scen ...
for a given algorithm, although it can also be used to express the average-case — for example, the worst-case scenario for
quicksort Quicksort is an in-place sorting algorithm In computer science Computer science deals with the theoretical foundations of information, algorithms and the architectures of its computation as well as practical techniques for their applica ...

is ''O''(''n''2), but the average-case run-time is .

## Empirical orders of growth

Assuming the execution time follows power rule, ', the coefficient ''a'' can be found by taking empirical measurements of run time $\$ at some problem-size points $\$, and calculating $t_2/t_1 = \left(n_2/n_1\right)^a$ so that $a = \log\left(t_2/t_1\right) / \log\left(n_2/n_1\right)$. In other words, this measures the slope of the empirical line on the
log–log plot In science and engineering, a log–log graph or log–log plot is a two-dimensional graph of numerical data that uses logarithmic scales on both the horizontal and vertical axes. Monomials – relationships of the form y=ax^k – appear as straigh ...
of execution time vs. problem size, at some size point. If the order of growth indeed follows the power rule (and so the line on log–log plot is indeed a straight line), the empirical value of ''a'' will stay constant at different ranges, and if not, it will change (and the line is a curved line) - but still could serve for comparison of any two given algorithms as to their ''empirical local orders of growth'' behaviour. Applied to the above table: It is clearly seen that the first algorithm exhibits a linear order of growth indeed following the power rule. The empirical values for the second one are diminishing rapidly, suggesting it follows another rule of growth and in any case has much lower local orders of growth (and improving further still), empirically, than the first one.

## Evaluating run-time complexity

The run-time complexity for the worst-case scenario of a given algorithm can sometimes be evaluated by examining the structure of the algorithm and making some simplifying assumptions. Consider the following
pseudocode In computer science, pseudocode is a plain language description of the steps in an algorithm or another system. Pseudocode often uses structural conventions of a normal programming language, but is intended for human reading rather than machine re ...
: 1 ''get a positive integer n from input'' 2 if n > 10 3 print "This might take a while..." 4 for i = 1 to n 5 for j = 1 to i 6 print i * j 7 print "Done!" A given computer will take a discrete amount of time to execute each of the
instructions Instruction or instructions may refer to: Computing * Instruction, one operation of a processor within a computer architecture instruction set * Computer program, a collection of instructions Music * Instruction (band), a 2002 rock band from New Y ...
involved with carrying out this algorithm. The specific amount of time to carry out a given instruction will vary depending on which instruction is being executed and which computer is executing it, but on a conventional computer, this amount will be
deterministic Determinism is the philosophical Philosophy (from , ) is the study of general and fundamental questions, such as those about existence Existence is the ability of an entity to interact with physical or mental reality Reality is the ...
. Say that the actions carried out in step 1 are considered to consume time ''T''1, step 2 uses time ''T''2, and so forth. In the algorithm above, steps 1, 2 and 7 will only be run once. For a worst-case evaluation, it should be assumed that step 3 will be run as well. Thus the total amount of time to run steps 1-3 and step 7 is: :$T_1 + T_2 + T_3 + T_7. \,$ The loops in steps 4, 5 and 6 are trickier to evaluate. The outer loop test in step 4 will execute ( ''n'' + 1 ) times (note that an extra step is required to terminate the for loop, hence n + 1 and not n executions), which will consume ''T''4( ''n'' + 1 ) time. The inner loop, on the other hand, is governed by the value of j, which iterates from 1 to ''i''. On the first pass through the outer loop, j iterates from 1 to 1: The inner loop makes one pass, so running the inner loop body (step 6) consumes ''T''6 time, and the inner loop test (step 5) consumes 2''T''5 time. During the next pass through the outer loop, j iterates from 1 to 2: the inner loop makes two passes, so running the inner loop body (step 6) consumes 2''T''6 time, and the inner loop test (step 5) consumes 3''T''5 time. Altogether, the total time required to run the inner loop body can be expressed as an
arithmetic progression An Arithmetic progression (AP) or arithmetic sequence is a sequence In mathematics Mathematics (from Greek: ) includes the study of such topics as numbers (arithmetic and number theory), formulas and related structures (algebra), shapes an ...

: :$T_6 + 2T_6 + 3T_6 + \cdots + \left(n-1\right) T_6 + n T_6$ which can be
factored In mathematics Mathematics (from Ancient Greek, Greek: ) includes the study of such topics as quantity (number theory), mathematical structure, structure (algebra), space (geometry), and calculus, change (mathematical analysis, analysis). I ...

as : The total time required to run the outer loop test can be evaluated similarly: :$\begin & 2T_5 + 3T_5 + 4T_5 + \cdots + \left(n-1\right) T_5 + n T_5 + \left(n + 1\right) T_5\\ =\ &T_5 + 2T_5 + 3T_5 + 4T_5 + \cdots + \left(n-1\right)T_5 + nT_5 + \left(n+1\right)T_5 - T_5 \end$ which can be factored as : Therefore, the total running time for this algorithm is: : which reduces to : As a rule-of-thumb, one can assume that the highest-order term in any given function dominates its rate of growth and thus defines its run-time order. In this example, n2 is the highest-order term, so one can conclude that f(n) = O(n2). Formally this can be proven as follows: A more
elegant Image:Pythagorean proof (1).svg, 200px, An example of "beauty in method"—a simple and elegant proof of the Pythagorean theorem. Elegance is beauty that shows unusual effectiveness and simplicity. Elegance is frequently used as a standard of Tas ...

approach to analyzing this algorithm would be to declare that 'T''1..''T''7are all equal to one unit of time, in a system of units chosen so that one unit is greater than or equal to the actual times for these steps. This would mean that the algorithm's running time breaks down as follows:This approach, unlike the above approach, neglects the constant time consumed by the loop tests which terminate their respective loops, but it is trivial to prove that such omission does not affect the final result

## Growth rate analysis of other resources

The methodology of run-time analysis can also be utilized for predicting other growth rates, such as consumption of memory space. As an example, consider the following pseudocode which manages and reallocates memory usage by a program based on the size of a
file File or filing may refer to: Mechanical tools and processes * File (tool) A file is a tool used to remove fine amounts of material from a workpiece. It is common in woodworking, metalworking, and other similar trade and hobby tasks. Most are ...
which that program manages: while ''file is still open:'' let n = ''size of file'' for ''every 100,000
kilobyte The kilobyte is a multiple of the unit byte The byte is a unit of digital information that most commonly consists of eight bit The bit is a basic unit of information in computing Computing is any goal-oriented activity requiring, bene ...
s of increase in file size'' ''double the amount of memory reserved'' In this instance, as the file size n increases, memory will be consumed at an
exponential growth Exponential growth is a process that increases quantity over time. It occurs when the instantaneous rate of change (that is, the derivative In mathematics, the derivative of a function of a real variable measures the sensitivity to change ...

rate, which is order O(2n). This is an extremely rapid and most likely unmanageable growth rate for consumption of memory
resources Resource refers to all the materials available in our environment which help us to satisfy our needs and wants. Resources can broadly be classified upon their availability — they are classified into renewable A renewable resource, also know ...
.

# Relevance

Algorithm analysis is important in practice because the accidental or unintentional use of an inefficient algorithm can significantly impact system performance. In time-sensitive applications, an algorithm taking too long to run can render its results outdated or useless. An inefficient algorithm can also end up requiring an uneconomical amount of computing power or storage in order to run, again rendering it practically useless.

# Constant factors

Analysis of algorithms typically focuses on the asymptotic performance, particularly at the elementary level, but in practical applications constant factors are important, and real-world data is in practice always limited in size. The limit is typically the size of addressable memory, so on 32-bit machines 232 = 4 GiB (greater if segmented memory is used) and on 64-bit machines 264 = 16 EiB. Thus given a limited size, an order of growth (time or space) can be replaced by a constant factor, and in this sense all practical algorithms are O(1) for a large enough constant, or for small enough data. This interpretation is primarily useful for functions that grow extremely slowly: (binary)
iterated logarithm In computer science, the iterated logarithm of n, written  n (usually read "log star"), is the number of times the logarithm function must be iteration, iteratively applied before the result is less than or equal to 1. The simplest formal def ...

(log*) is less than 5 for all practical data (265536 bits); (binary) log-log (log log ''n'') is less than 6 for virtually all practical data (264 bits); and binary log (log ''n'') is less than 64 for virtually all practical data (264 bits). An algorithm with non-constant complexity may nonetheless be more efficient than an algorithm with constant complexity on practical data if the overhead of the constant time algorithm results in a larger constant factor, e.g., one may have $K > k \log \log n$ so long as $K/k > 6$ and $n < 2^ = 2^$. For large data linear or quadratic factors cannot be ignored, but for small data an asymptotically inefficient algorithm may be more efficient. This is particularly used in
hybrid algorithm {{Unreferenced, date=May 2014 A hybrid algorithm is an algorithm In and , an algorithm () is a finite sequence of , computer-implementable instructions, typically to solve a class of problems or to perform a computation. Algorithms are always ...

s, like
Timsort Timsort is a hybrid stable sorting algorithm, derived from merge sort and insertion sort, designed to perform well on many kinds of real-world data. It was implemented by Tim Peters (software engineer), Tim Peters in 2002 for use in the Python (pr ...
, which use an asymptotically efficient algorithm (here
merge sort In computer science Computer science deals with the theoretical foundations of information, algorithms and the architectures of its computation as well as practical techniques for their application. Computer science is the study of , ...
, with time complexity $n \log n$), but switch to an asymptotically inefficient algorithm (here
insertion sort Insertion sort is a simple sorting algorithm In computer science, a sorting algorithm is an algorithm that puts elements of a List (computing), list in a certain Total order, order. The most frequently used orders are numerical order and lexicogr ...

, with time complexity $n^2$) for small data, as the simpler algorithm is faster on small data.

*
Amortized analysis In computer science Computer science deals with the theoretical foundations of information, algorithms and the architectures of its computation as well as practical techniques for their application. Computer science is the study of , , a ...

*
Analysis of parallel algorithmsOverview In computer science, the analysis of parallel algorithms is the process of finding the computational complexity of algorithms executed in parallel – the amount of time, storage, or other resources needed to execute them. In many respect ...
*
Asymptotic computational complexityIn computational complexity theory Computational complexity theory focuses on classifying computational problems according to their resource usage, and relating these classes to each other. A computational problem is a task solved by a computer. A ...
*
Best, worst and average case In computer science Computer science deals with the theoretical foundations of information, algorithms and the architectures of its computation as well as practical techniques for their application. Computer science is the study of , , ...
*
Big O notation Big O notation is a mathematical notation that describes the limiting behavior of a function Function or functionality may refer to: Computing * Function key A function key is a key on a computer A computer is a machine that can be p ...
*
Computational complexity theory Computational complexity theory focuses on classifying computational problems according to their resource usage, and relating these classes to each other. A computational problem is a task solved by a computer. A computation problem is solvable by ...
*
Master theorem (analysis of algorithms) In the analysis of algorithms Graphs of functions commonly used in the analysis of algorithms, showing the number of operations ''N'' versus input size ''n'' for each function In computer science Computer science deals with the theoretical ...
*
NP-Complete In computational complexity theory Computational complexity theory focuses on classifying computational problem In theoretical computer science An artistic representation of a Turing machine. Turing machines are used to model general computi ...
*
Numerical analysis Numerical analysis is the study of algorithms that use numerical approximation (as opposed to symbolic computation, symbolic manipulations) for the problems of mathematical analysis (as distinguished from discrete mathematics). Numerical analysis ...
*
Polynomial time In computer science, the time complexity is the computational complexity that describes the amount of computer time it takes to run an algorithm. Time complexity is commonly estimated by counting the number of elementary operations performed by th ...
*
Program optimization In computer science, program optimization, code optimization, or software optimization is the process of modifying a software system to make some aspect of it work more algorithmic efficiency, efficiently or use fewer resources. In general, a com ...
*
Profiling (computer programming) In software engineering Software engineering is the systematic application of engineering approaches to the software development, development of software. A software engineer is a person who applies the principles of software engineering to des ...
*
Scalability Scalability is the property of a system to handle a growing amount of work by adding resources to the system. In an economic An economy (; ) is an area of the production Production may be: Economics and business * Production (economics ...
*
Smoothed analysis In theoretical computer science, smoothed analysis is a way of measuring the Analysis of algorithms, complexity of an algorithm. Since its introduction in 2001, smoothed analysis has been used as a basis for considerable research, for problems rang ...
*
Termination analysis In computer science Computer science deals with the theoretical foundations of information, algorithms and the architectures of its computation as well as practical techniques for their application. Computer science is the study of Algorit ...
— the subproblem of checking whether a program will terminate at all *
Time complexity In computer science Computer science deals with the theoretical foundations of information, algorithms and the architectures of its computation as well as practical techniques for their application. Computer science is the study of com ...
— includes table of orders of growth for common algorithms *
Information-based complexity Information-based complexity (IBC) studies optimal algorithms In mathematics Mathematics (from Greek: ) includes the study of such topics as numbers ( and ), formulas and related structures (), shapes and spaces in which they are conta ...

* * * * * *