Log management (LM) comprises an approach to dealing with large volumes of
computer-generated
log messages (also known as
audit records,
audit trail
An audit trail (also called audit log) is a security-relevant chronological record, set of records, and/or destination and source of records that provide documentary evidence of the sequence of activities that have affected at any time a specific ...
s,
event-logs, etc.).
Log management generally covers:
* Log collection
* Centralized log aggregation
* Long-term log storage and retention
*
Log rotation
*
Log analysis (in real-time and in bulk after storage)
* Log search and reporting.
Overview
The primary drivers for log management implementations are concerns about
security" \n\n\nsecurity.txt is a proposed standard for websites' security information that is meant to allow security researchers to easily report security vulnerabilities. The standard prescribes a text file called \"security.txt\" in the well known locat ...
, system and network operations (such as
system or
network administration
Network management is the process of administering and managing computer networks. Services provided by this discipline include fault analysis, performance management, provisioning of networks and maintaining quality of service. Network manage ...
) and regulatory compliance. Logs are generated by nearly every computing device, and can often be directed to different locations both on a local
file system
In computing, file system or filesystem (often abbreviated to fs) is a method and data structure that the operating system uses to control how data is stored and retrieved. Without a file system, data placed in a storage medium would be one lar ...
or remote system.
Effectively analyzing large volumes of diverse logs can pose many challenges, such as:
* Volume: log data can reach hundreds of gigabytes of data per day for a large
organization
An organization or organisation (Commonwealth English; see spelling differences), is an entity—such as a company, an institution, or an association—comprising one or more people and having a particular purpose.
The word is derived fro ...
. Simply collecting, centralizing and storing data at this volume can be challenging.
* Normalization: logs are produced in multiple formats. The process of
normalization is designed to provide a common output for analysis from diverse sources.
* Velocity: The speed at which logs are produced from devices can make collection and aggregation difficult
* Veracity: Log events may not be accurate. This is especially problematic for systems that perform detection, such as
intrusion detection systems.
Users and potential users of log management may purchase complete commercial tools or build their own log-management and intelligence tools, assembling the functionality from various
open-source
Open source is source code that is made freely available for possible modification and redistribution. Products include permission to use the source code, design documents, or content of the product. The open-source model is a decentralized sof ...
components, or acquire (sub-)systems from commercial vendors. Log management is a complicated process and organizations often make mistakes while approaching it.
Logging can produce technical information usable for the maintenance of applications or websites. It can serve:
* to define whether a reported bug is actually a bug
* to help analyze, reproduce and solve bugs
* to help test new features in a development stage
Terminology
Suggestions were made to change the definition of logging. This change would keep matters both purer and more easily maintainable:
* Logging would then be defined as all instantly discardable data on the technical process of an application or website, as it represents and processes data and user input.
* Auditing, then, would involve data that is not immediately discardable. In other words: data that is assembled in the auditing process, is stored persistently, is protected by authorization schemes and is, always, connected to some end-user functional requirement.
Deployment life-cycle
One view of assessing the maturity of an organization in terms of the deployment of log-management tools might use successive levels such as:
# in the initial stages, organizations use different log-analyzers for analyzing the logs in the devices on the security perimeter. They aim to identify the patterns of attack on the perimeter infrastructure of the organization.
# with the increased use of integrated computing, organizations mandate logs to identify the access and usage of confidential data within the security perimeter.
# at the next level of maturity, the log analyzer can track and monitor the performance and availability of systems at the level of the
enterprise
Enterprise (or the archaic spelling Enterprize) may refer to:
Business and economics
Brands and enterprises
* Enterprise GP Holdings, an energy holding company
* Enterprise plc, a UK civil engineering and maintenance company
* Enterprise ...
— especially of those information assets whose availability organizations regard as vital.
# organizations integrate the logs of various
business applications into an enterprise log manager for a better
value proposition
In marketing, a company’s value proposition is the full mix of benefits or economic value which it promises to deliver to the current and future customers (i.e., a market segment) who will buy their products and/or services. It is part of a c ...
.
# organizations merge the physical-access monitoring and the logical-access monitoring into a single view.
See also
*
Audit trail
An audit trail (also called audit log) is a security-relevant chronological record, set of records, and/or destination and source of records that provide documentary evidence of the sequence of activities that have affected at any time a specific ...
*
Common Base Event
*
Common Log Format
*
DARPA
The Defense Advanced Research Projects Agency (DARPA) is a research and development agency of the United States Department of Defense responsible for the development of emerging technologies for use by the military.
Originally known as the Ad ...
PRODIGAL ''The Prodigal'' is a 1955 epic biblical film.
The Prodigal, Prodigal or Prodigals may also refer to:
Arts, entertainment and media Film and television
* ''The Prodigal'' (1931 film), an early talkie film
* ''The Prodigal'' (1983 film), directe ...
and
Anomaly Detection at Multiple Scales (ADAMS) projects.
*
Data logging
A data logger (also datalogger or data recorder) is an electronic device that records data over time or about location either with a built-in instrument or sensor or via external instruments and sensors. Increasingly, but not entirely, they a ...
*
Log analysis
*
Log monitor
*
Log management knowledge base
*
Security information and event management
Security information and event management (SIEM) is a field within the field of computer security, where software products and services combine security information management (SIM) and security event management (SEM). They provide real-time a ...
*
Server log
In computing, logging is the act of keeping a log of events that occur in a computer system, such as problems, errors or just information on current operations. These events may occur in the operating system or in other software. A message or ...
*
Syslog
In computing, syslog is a standard for message logging. It allows separation of the software that generates messages, the system that stores them, and the software that reports and analyzes them. Each message is labeled with a facility code, i ...
*
Web counter
A web counter or hit counter is a publicly displayed running tally of the number of visits a webpage has received.
Web counters are usually displayed as an inline digital image or in plain text. Image rendering of digits may use a variety of ...
*
Web log analysis software
References
* Chris MacKinnon: "LMI In The Enterprise". ''Processor'' November 18, 2005, Vol.27 Issue 46, page 33. Online at http://www.processor.com/editorial/article.asp?article=articles%2Fp2746%2F09p46%2F09p46.asp, retrieved 2007-09-10
* MITRE: Common Event Expression (CEE) Proposed Log Standard. Online at http://cee.mitre.org, retrieved 2010-03-03
* NIST 800-92: Guide to Security Log Management. Online at http://csrc.nist.gov/publications/nistpubs/800-92/SP800-92.pdf, retrieved 2010-03-03
External links
InfoWorld review and comparison of commercial Log Management products
{{DEFAULTSORT:Log Management And Intelligence
Network management
Computer systems
*