Logging Cut Down To Size ...

Logging: Cut Down To Size

Fred DeFrance
November 27, 2001

What is Logging, Exactly?
Briefly, logging is any procedure by which an operating system or application records events as they happen and preserves those records for later perusal. System logs are usually human-readable text files containing a timestamp and other information specific to the message or subsystem.

A logging facility is a central location for error reports, system alerts, and diagnostic messages. It is also used to create and maintain the files that are the repository for all event logging reports and data. An event is any type of detectable change that occurs in a system. Because a computer system experiences literally hundreds or thousands of events each second, it is important to distinguish which events require the immediate attention of the system administrators, which events warrant a mention with an entry written to a log, and which events can be safely ignored.
Any type of event that may be detected by a process running on a computer system is fair game to be reported and logged. This fact is reflected in the myriad of names for event logs you'll find in different systems - system logs, user logs, usage logs, program logs, activity logs, alarm logs, billing logs, security auditing logs, network traffic reports, and error report listings. The sum total of a computer system's operation may be expressed in the persistent (i.e., semi-permanent) medium of the event logs.

Logging is usually a native component of only larger operating systems and environments. Most single-user operating systems, such as MS-DOS and MacOS, Windows98 (and earlier) don't support any native form of event logging. Error messages are simply displayed on the screen and then scroll off, or vanish with the click of an OK button.

It's difficult to say when logging first became a staple procedure in computing, (syslog has been around since the inception of UNIX), but it hails from the discipline of programming. Even when you write a relatively simple program, it's useful to have diagnostic information on hand. For example:
- Whether the program faulted and if so, when and why.
- The program's UID and PID.
- Who has used the program, and when did they use it?
- Does the program perform tasks in the way you want it to?
Logging in *nix, NT and macOS-X operating systems is pervasive and occurs at the system, application, security and even protocol levels. And, though there are exceptions (third party software, for example), most logging services output log information to standard or even shared log files.
What's the use?
Event logging really is a necessity for multi-user systems, or at least for systems that may have multiple user accounts. When more than one person is using a single machine, a record of "who did what when" must be kept in the event that problems with the system occur (and fingers must be pointed). In such cases, the system's logs are the primary means for discovering if the security of a system or network has been breached, and for determining exactly what has happened.

To many IT departments, system logging is a necessary evil and often ignored. (It's a pain to set up well and a pain to maintain. If the first item is done right then maintenance can be minimal). In many companies today, very little attention is dedicated to the information that systems log or potentially log. The log info is not easy to use. It's difficult to look through, the information is usually(hopefully) routine and non-eventful. Humans are not very good at processing a large amount of monotonous detail. In many cases, IT staff will purposely set the systems to report minimal information. This practice is a double-edged sword. Just because you gag your computers so they won't complain doesn't mean the problems have gone away! Instead of hearing the warning signs of disaster before or during the crisis, your IT staff is only left with picking up the smoldering remains of the corpse after the fact. While this practice reduces the resources and costs initially, it's all paid back several fold after one or two catastrophes occur.

The bottom line, system logs are one of your best resources available to maintain, tune, alert, and troubleshoot problems on your systems. It's an established fact that properly logging system and network resources will save you time and money. If configured correctly, they can provide you with a wealth of information that many IT departments overlook. They can give you real-time health checkups, system-level hardware/software status, application-level information for maintenance and tuning, real-time security measures rarely utilized for over-all network protection. By properly harnessing the potential resources available here, organizations can save money converting their IT management style from reactionary to one that's proactive.

What's the logging standard?
There are two common logging utilities in use today which are the most popular in IT infrastructures. One is the mechanism used by Microsoft (event logging), and the other which has been adopted by all manufacturers of UNIX, Linux and recent macOS-X operating systems (syslog).

Microsoft Event Logging
This utility combines two components: the event logs (contained in a MS-type DB format), and the event viewer (the API which accesses and manages the event logs). Event logs contain the records of reported events. The logs are broken down into 3 main categories:
- System - Device driver load failures, I/O port or IRQ conflicts, system configuration not responding or misconfigured, low-level system or network data transport errors.
- Security - User log-on/off authentication, changes to system security and user privileges, attempts to access files/directories/printers and other system objects which are under audit control.
- Application - application-related events (for instance, some applications generate informational messages that appear as entries in the Application Event Log; application errors such as failing to load a DLL can also appear in the application log).

UNIX and Linux Syslog
Syslog, the general-purpose logging mechanism in UNIX and Linux, consists of the following:

syslog() - an application program interface (API) referenced by several standard system utilities and available to anyone writing software in the C programming language.

logger - a system command used to add single-line entries to the system log.

/etc/syslog.conf - the configuration file used to control the logging and routing of system log events.

syslogd - the system daemon used to receive and route system log events from syslog() calls and logger commands.

For *nix host logging, each message is a single line of text with an associated facility and severity. The facility can be thought of as a category that depends upon the program from which the message originates. The developer of a program decides which facility a program will utilize. In some cases, it may be configurable by the end user. Severities are hierarchical and range from emerg being the most important down to debug as the least significant.

Dilemma of having more than one standard.
As long as proprietary operating systems prevail, we'll be faced with supporting multiple standards. This is true for logging across our networks. The remedy in the past has come from the *nix camps with various public domain solutions developed by very talented people. This approach has been geared towards UNIX-people coping with non-UNIX systems. Because of this, unless you're a very talented person as well, with lots of development time on your hands, you won't be very successful at making all your networked devices share logging information. Most organizations leave their logs on the machine that generated them. This is quite common and contributes to the "not watching the logs" syndrome.

Another issue in the past has been that there's too many logging projects working in parallel trying to solve the same problem. Because of this, nobody has been able to establish a "best solution" and focused on the task of how to easily implement it. Corporations now realizing this are forced to either hire a talent-base capable of developing a site-specific design or commercial products which are just now hitting the market.

There has been many documents all over the internet revealing bits and pieces of how to incorporate homogeneous logging in a heterogeneous network; but, outside of eBuzzSaw(1) nobody is providing a turn-key solution from "front to back".

How can you sanely manage your system logs?
Experts from SANS Institute (System Administration and Network Security) and other organizations agree the only way to effectively and proactively manage system logging is via a centralized repository. (Generally system logs can only be viewed on a machine by machine basis). Without something like eBuzzSaw(1) this can be tedious and is one of the many reasons most organizations don't review their logs. In today's distributed computing architecture - consisting of numerous dedicated servers handling applications such as web servers, DNS, mail-relays, data bases, and ftp sites - it’s not only beneficial but necessary to create a centralized logging system. Having a central log server is also helpful for keeping records of failed attempts, who is spamming your servers, and a place for your firewalls to keep their logs. These logs are also extremely important in finding out the health and integrity of the systems and network you manage.

While experts are still debating and developing better methods for sending log information to a central repository, very little effort is focused on best ways to store, archive and access huge amounts of log data. Most of the technology designed today still houses log data in ascii text flat files. Because of this, many system administrators are still resorting to regular expression matching via basic system commands like grep to parse the sea of accumulated data. Very few implementations today are incorporating data base technology - which is a real tragedy, considering this technology is available to us. What better/faster way is there to parse huge chunks of data? Still an even scarier issue is knowing what patterns a system admin should even be looking for! Since each vendor writes unique device drivers for their hardware, many system errors may be unique only to their platform! Only years of experience and time well spent pouring over hundreds of megabytes of raw system logs will provide a staff member the knowledge he needs to quickly search for clues to troubleshoot and monitor each computer.

A brief word on local system logs: Many new versions of Linux are released with a log rotation apparatus to assist in a proper local log management. Public domain versions are available for all the popular flavors of UNIX also. While it will take some intimate knowledge and experience from trial and error to tune these utilities to your specific systems it's just good practice. The benefit to implementation of a centralized log server (CLS) is that it minimizes the level of time dedicated to initiate and follow-up maintenance of local system log mechanisms.
What Do Logs Have To Do With Security?
Attackers are well aware that system logs can give them away. Hence, every cracker keeps an up-to-date cache of sweepers and cleaners (programs that circumvent default logging systems). Log cleaners are concrete examples of how legitimate programming techniques can be used to circumvent system security. (In other words, much like scanners, log access utilities are tools that can be used equally effectively by well-intentioned and not-so-well intentioned users.) The attacker writes code that opens utmp, and, using something like strncpy (string copy), replaces the current line with user-specified data (or simply uses strncpy to replace the current line with whitespace or nothing) ie the cracker will rewrite or erase your logs in order to cover their break-in of your system or network.

To hedge your bets against crackers tampering with your log entries, you should use at least one third-party or proprietary logging tool. This approach offers two powerful advantages. First, few crackers will know (or bother to verify) that you are running special logging tools. Second, such tools can derive their logs independently, without using operating system logs as a starting index (a back-door syslogd is publicly available.) If you later compare this information to default system logs and find a discrepancy, you'll instantly know that an intrusion has taken place.

Also consider insulating your logs from tampering. For example, write them to a write-once media or remote log server such as eBuzzSaw(1). This will guarantee that you'll have one set of reliable logs and reliability is everything.

Many people today are implementing Intrusion Detection Systems which is a good thing and a valuable complement to any firewall / perimeter security. The problem is that many people are relying on these intrusion detection systems, instead of, not in addition to, effectively and proactively managing logging and log files. Remember, layers of security are the most effective methods to date to stop intrusions. Never underestimate the importance of keeping detailed logs. Not only are logs essential when you're investigating a network intrusion, they're also a requisite for bringing charges against an attacker.

Technical References

Anonymous, Maximum Linux Security. SAMS Press, 2000.

Gregory, Peter H., Solaris Security. Sun Microsystems Press/Prentice Hall, 2000.

Morton, Matt, Logging and critical logs files:...SANS Institute, Dec.9, 2000.

Murray, James D., Windows NT Event Logging, O'Reilly & Associates, Inc., Sept. 1998.

(1) eBuzzsaw, Product of Sentry Plus, http://eBuzzSaw.com.