Tracing in software engineering refers to the process of capturing and recording information about the execution of a software program. This information is typically used by

programmer A programmer, computer programmer or coder is an author of computer source code someone with skill in computer programming. The professional titles Software development, ''software developer'' and Software engineering, ''software engineer' ...

s for

debugging In engineering, debugging is the process of finding the Root cause analysis, root cause, workarounds, and possible fixes for bug (engineering), bugs. For software, debugging tactics can involve interactive debugging, control flow analysis, Logf ...

purposes, and additionally, depending on the type and detail of information contained in a trace log, by experienced

system administrator An IT administrator, system administrator, sysadmin, or admin is a person who is responsible for the upkeep, configuration, and reliable operation of computer systems, especially multi-user computers, such as Server (computing), servers. The ...

s or technical-support personnel and by software monitoring tools to diagnose common problems with software. Tracing is a cross-cutting concern. There is not always a clear distinction between ''tracing'' and other forms of ''

logging Logging is the process of cutting, processing, and moving trees to a location for transport. It may include skidder, skidding, on-site processing, and loading of trees or trunk (botany), logs onto logging truck, trucksfunctional requirement of a program (therefore excluding logging of data from an external source, such as

data acquisition Data acquisition is the process of sampling signals that measure real-world physical conditions and converting the resulting samples into digital numeric values that can be manipulated by a computer. Data acquisition systems, abbreviated by the ...

in a

high-energy physics Particle physics or high-energy physics is the study of fundamental particles and forces that constitute matter and radiation. The field also studies combinations of elementary particles up to the scale of protons and neutrons, while the stu ...

experiment, and write-ahead logging). Logs that record program usage (such as a server log) or operating-system events primarily of interest to a system administrator (see for example '' Event Viewer'') fall into a terminological gray area. Tracing is primarily used for anomaly detection, fault analysis, debugging or diagnostic purposes in distributed software systems, such as microservices or serverless functions.

Software tracing

Software tracing is a tool for developers to gather information for debugging. This information is used both during development cycles and post-release. Unlike event logging, software tracing usually does not have the concept of a "class" of event or an "event code". Other reasons why event-logging solutions based on event codes are inappropriate for software tracing include: * Because software tracing is low-level, there are often many more types of messages that would need to be defined, many of which would only be used at one place in the code. The event-code paradigm introduces significant development overhead for these "one-shot" messages. * The types of messages that are logged are often less stable through the development cycle than for event logging. * Because the tracing output is intended to be consumed by the developer, the messages don't need to be localized. Keeping tracing messages separate from other resources that need to be localized (such as event messages) is therefore important. * There are messages that should never be seen. * Tracing messages should be kept in the code, because they can add to the readability of the code. This is not always possible or feasible with event-logging solutions.

Tools

OpenTelemetry is a CNCF open source project that provides comprehensive support for distributed tracing. Some vendors including Datadog, New Relic, Splunk also offer tracing SaaS services.

Google Google LLC (, ) is an American multinational corporation and technology company focusing on online advertising, search engine technology, cloud computing, computer software, quantum computing, e-commerce, consumer electronics, and artificial ...

and Meta have developed their own tracing frameworks namely Dapper and Canopy.

Application-specific tracing

* Tracing with GNU Debugger's trace command * Linux C/C++ application tracing with cwrap * Linux application tracing with UST – part of the same project as

LTTng LTTng (Linux Trace Toolkit: next generation) is a system software package for correlated tracing of the Linux kernel, applications and libraries. The project was originated by Mathieu Desnoyers with an initial release in 2005. Its predecessor is ...

* Windows software trace preprocessor (aka WPP) * Instruction set simulation

System-specific tracing

In operating systems, tracing can be used in situations (such as

booting In computing, booting is the process of starting a computer as initiated via Computer hardware, hardware such as a physical button on the computer or by a software command. After it is switched on, a computer's central processing unit (CPU) h ...

) where some of the technologies used to provide event logging may not be available. Linux offers system-level and user-level tracing capabilities with

kernel marker Kernel markers were a static kernel instrumentation support mechanism for Linux kernel source code, allowing special tools such as LTTng Immediate Values - Documentation] The initial motivation to create this static instrumentation infrastructure ...

s and

. ftrace also supports Linux kernel tracing. syslog is another tool in various operating systems for logging and tracing system messages.

FreeBSD FreeBSD is a free-software Unix-like operating system descended from the Berkeley Software Distribution (BSD). The first version was released in 1993 developed from 386BSD, one of the first fully functional and free Unix clones on affordable ...

and SmartOS employ DTrace for tracing in both user space and kernel space. In

embedded software Embedded software is computer software, written to control machines or devices that are not typically thought of as computers, commonly known as embedded systems. It is typically specialized for the particular hardware that it runs on and has tim ...

, tracing also requires special techniques for efficient instrumentation and logging and low CPU overhead.

Techniques

Trace generation and collection

Trace generation of method calls can be done with source code instrumentation, runtime information collection, or under debugger control. Tracing macros, aspect-oriented programming and related

instrumentation Instrumentation is a collective term for measuring instruments, used for indicating, measuring, and recording physical quantities. It is also a field of study about the art and science about making measurement instruments, involving the related ...

techniques can be employed. Libraries used in source code send data to an agent or directly to the collection component.

Trace analysis

To model execution trees, ISVis converts a rooted tree into a directed acyclic graph while Jinsight utilizes the call frame principle to gather and represent cumulative information about traces. The primary visualization method is the swimlane view, which is exemplified by tools like Jaeger and often includes annotations and key-value attributes. Despite its widespread use, this design lacks rigorous justification and users frequently face challenges like missing features and confusing navigation. Alternatives to swimlane views exist, like Jaeger’s service dependency view or SkyWalking’s List, Tree, and Table views. Aggregate visualizations are also used for analyzing large volumes of traces, with systems like Canopy offering queryable metrics and Jaeger providing trace comparison features.

Event logging

Event logging provides system administrators with information useful for diagnostics and

auditing An audit is an "independent examination of financial information of any entity, whether profit oriented or not, irrespective of its size or legal form when such an examination is conducted with a view to express an opinion thereon." Auditing al ...

. The different classes of events that will be logged, as well as what details will appear in the event messages, are often considered early in the development cycle. Many event logging technologies allow or even require each class of event to be assigned a unique "code", which is used by the event logging software or a separate viewer (e.g., Event Viewer) to format and output a human-readable message. This facilitates localization and allows system administrators to more easily obtain information on problems that occur. Because event logging is used to log high-level information (often failure information), performance of the logging implementation is often less important. A special concern, preventing duplicate events from being recorded "too often" is taken care of through event throttling. Difficulties in making a clear distinction between event logging and software tracing arise from the fact that some of the same technologies are used for both, and further because many of the criteria that distinguish between the two are continuous rather than discrete. The following table lists some important, but by no means precise or universal, distinctions that are used by developers to select technologies for each purpose, and that guide the separate development of new technologies in each area:

Challenges and limitations

Enabling or disabling tracing during runtime often necessitates the inclusion of extra data in the binary. This can lead to performance degradation, even when tracing is not active. If tracing is enabled or disabled at compile-time, collecting trace data from a client's system hinges on their willingness and capability to install a version of the software specifically enabled for tracing, and subsequently replicate the issue. Tracing in software typically demands high standards of

robustness Robustness is the property of being strong and healthy in constitution. When it is transposed into a system A system is a group of interacting or interrelated elements that act according to a set of rules to form a unified whole. A system, ...

, not only in the accuracy and reliability of the trace output but also in ensuring that the process being traced remains uninterrupted. Given its low-level nature, tracing can generate a large volume of messages. To mitigate performance issues, it's often necessary to have the option to deactivate software tracing, either at the time of compilation or during run-time.

Security and privacy

proprietary software Proprietary software is computer software, software that grants its creator, publisher, or other rightsholder or rightsholder partner a legal monopoly by modern copyright and intellectual property law to exclude the recipient from freely sharing t ...

, tracing data may include sensitive information about the product's

source code In computing, source code, or simply code or source, is a plain text computer program written in a programming language. A programmer writes the human readable source code to control the behavior of a computer. Since a computer, at base, only ...

References

{{Reflist Debugging