Basically logs generated by server daemons such as Apache or applications, are streams and not files, though the application/daemon have a configuration parameter for logging into a file.
Why logs are considered as streams? Because logs do not have beginning or end, they are just ongoing.
How Linux handles logs(streams)?
In Linux, stdout and stderr, are two default output streams, which are automatically available to all programs. These streams can be turned into files using a redirect operator.
An application which uses stdout for logging can be made to log to a file as follows
$ application >> /var/log/applogfile
Syslog:
So when log messages from an application can be directed to go to files(using redirection operator) or user terminals or run them through other programs (with a pipe - to email, pager, or just a log file analyzer), here comes a question of how to do distributed logging or logging to a centralized place from multiple hosts. syslog protocol comes to the aid here. Now we have modern logging protocols such as Scribe and Splunk for example.
As per RFC-3164:
In its most simplistic terms, the syslog protocol provides a transport to allow a machine to send event notification messages across IP networks to event message collectors - also known as syslog servers. Since each process, application and operating system was written somewhat independently, there is little uniformity to the content of syslog messages. For this reason, no assumption is made upon the formatting or contents of the messages. The protocol is simply designed to transport these event messages. In all cases, there is one device that originates the message. The syslog process on that machine may send the message to a collector. No acknowledgement of the receipt is made.
syslog protocol helps to send logs from many components to a single location. Programs which wish to use syslog protocol for logging, should have syslog awareness implemented in them. However, a program which used stdout for logging, can use syslog without needing to implement any syslog awareness into the program, by piping to the standard logger command.
$ myapplication | logger
However we can split the log stream to a local file as well as to syslog as follows
$ myapplication | tee /var/log/myapplication.log | logger
When set up to use a syslog server, devices will send their log messages over the network wire to the syslog server rather than recording them in a local file or displaying them.
Syslog is three things:
1) /dev/log, a UNIX-domain socket. Applications can connect to it and send it "messages".
2) UDP port 514, which is another service upon which applications can send messages. More importantly, this allows messages to be transferred from one host to another.
3) A program — sysklogd, syslog-ng, rsyslog, or one of several other variants of syslog — that listens on these sockets and ports, reads the messages, and decides where the messages should be sent, usually one or more files.
In my system, rsyslog program is being used and it uses a Unix domain socket
[root@dhcppc0 ~]# netstat -anp | grep rsyslog
unix 22 [ ] DGRAM 10559 1331/rsyslogd /dev/log
In my system, rsyslog program is being used and it uses a Unix domain socket
[root@dhcppc0 ~]# netstat -anp | grep rsyslog
unix 22 [ ] DGRAM 10559 1331/rsyslogd /dev/log
rsyslog is an enhanced, multi-threaded syslog.
From here onwards, I shall be using rsyslog and syslog interchangeably as
What gets logged by rsyslogd daemon and where it goes is controlled by /etc/rsyslog.conf. A modern system uses rsyslog to centralize logging.
Here's how it works: A developer uses the rsyslog API function (or uses the logger program in shell scripts) to send log messages to syslogd. The information passed to rsyslogd includes the source of the log message (called a facility) and the priority of the log message.
rsyslogd then matches the facility and priority against selectors (combinations of facilities and priorities) in its configuration file. For the selector(s) that match the messages is sent to the corresponding destination(s).
Log files contain sensitive information! You must protect these files by setting permisions carefully!
Log messages don't only have to go to files, you can direct them to user terminals, run them through other programs (with a pipe, to email, pager, or just a log file analyzer), or send them to another host running syslogd.
Following severity levels that can assigned to messages
0 - Emergency (emerg)
1 - Alerts (alert)
2 - Critical (crit)
3 - Errors (err)
4 - Warnings (warn)
5 - Notification (notice)
6 - Information (info)
7 - Debug (debug)
The messages can be categorized as follows. These things are also called as "facilities"
auth The authorization system. Ex.: login, su, ftpd, rshd
authpriv User access messages use this
cron Used by the cron facility
daemon Other daemon programs without a facility of their own
ftp Used by ftp applications
kern Kernel messages
lpr The line printer spooling system
mail Used by mail applications
mark Used by syslogd to produce timestamps in log files
news Used by news applications
security Same as auth. Should not be used anymore.
syslog messages from the syslog process itself
user Messages generated by random user processes. Default.
uucp UUCP messages
local0 – local7 Reserved for local use.
* For all
In /etc/rsyslog.conf, we can specify
1) Different severity levels can be specified for different facilities.
2) Also, where the message goes(files, user, pipes)
The entry format is as follows
facility.severity log-file-name
Sample entries in /etc/rsyslog.conf file would look like as follows
1) Do NOT redirect facilities mail, authentication and cron and mail to /var/log/messages, look for the keyword none
*.info;mail.none;authpriv.none;cron.none /var/log/messages
2) The authpriv file has restricted access.
authpriv.* /var/log/secure
3) Log all the mail messages in one place.
mail.* -/var/log/maillog
4) Log cron stuff
cron.* /var/log/cron
5) To redirect all incoming messages from all facilities and with all severities to /var/log/syslog
*.* -/var/log/syslog
6) To filter out messages with severity critical and save to file /var/log/critical
*.crit -/var/log/critical
What's that dash in front of the filenames in /etc/rsyslog.conf file?
This is to avoid rsyslogd or syslog daemon becoming a bottleneck for the performance of the system. rsyslogd/syslogd daemon uses fsync() to flush very file write. This is done to reduce the chances of log information getting lost before being written to the disk in case of a crash. By prefixing the names of the log files with a dash/hyphen, we specify that log file writes should not be flushed to disk immediately.
How to configure rsyslog for centralized logging?
Central Log Host
1) In the central log host, first setup "rsyslogd" to accept remote messages. This shall be done by uncommenting the following lines under "Modules" section in "/etc/rsyslog.conf" file in central log host as follows
# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514
Opening up UDP port 514 for receiving messages from remote servers.
UDP/TCP capability is not enabled by default for rsyslog. Hence we need to load the modules imudp(for UDP) and imtcp(for TCP) to enable UDP or TCP support.
2) Restart syslogd in central log host
service rsyslog restart
Verify if rsyslog is listening on UDP port 514
[root@dhcppc0 etc]# netstat -tulnp | grep 514
udp 0 0 0.0.0.0:514 0.0.0.0:* 3826/rsyslogd
udp 0 0 :::514 :::* 3826/rsyslogd
Now the central machine shall start accepting log messages from other machines on UDP port 514
Remote Host
Now the client machines need to be configured to send it's log messages to central log host listening on udp port 514.
1) In client machines, edit the file /etc/rsyslog.conf as follows
user.* @central_host_ip:514
2) Then restart the syslogd service in client machines
service rsyslog restart
3) Test the working by running the logger command in another machine
logger -i -t <yourname> "This is test from client" E.g. logger -i -t root "This is a test from client"
In the central log host, check /var/log/messages for an entry This is a test from client
