Setting up syslog-ng

[dos2unix]

Setting Up a Syslog-ng Server on Linux​

Introduction​

Syslog-ng is a powerful and flexible logging daemon that can be used to collect and manage log data from various sources. This guide will walk you through setting up a syslog-ng server on both RPM-based and APT-based Linux systems. We will also cover how to configure a client to send logs to the syslog-ng server.

Prerequisites​

• Two Linux computers (one as the server and one as the client)
• Root or sudo access on both machines

Setting Up the Syslog-ng Server​

On RPM-Based Systems (e.g., CentOS, Fedora)​

1. Install syslog-ng:

    sudo yum install syslog-ng

2. Configure syslog-ng: Edit the syslog-ng configuration file, usually located at /etc/syslog-ng/syslog-ng.conf. Add the following configuration to receive logs from clients:

    source s_network { tcp(ip(0.0.0.0) port(514)); udp(ip(0.0.0.0) port(514)); };

    destination d_messages { file("/var/log/messages"); };

    destination d_apache_access { file("/var/log/apache/http-access.log"); };

    destination d_apache_error { file("/var/log/apache/http-error.log"); };

    log { source(s_network); destination(d_messages); };

    log { source(s_network); filter(f_apache_access); destination(d_apache_access); };

    log { source(s_network); filter(f_apache_error); destination(d_apache_error); };

3. Open firewall ports: Ensure that your firewall allows traffic on port 514 for both TCP and UDP:

    sudo firewall-cmd --add-port=514/tcp --permanent

    sudo firewall-cmd --add-port=514/udp --permanent

    sudo firewall-cmd --reload

4. Enable and start syslog-ng:

    sudo systemctl enable syslog-ng

    sudo systemctl start syslog-ng

On APT-Based Systems (e.g., Ubuntu, Debian)​

1. Install syslog-ng:

    sudo apt-get update

    sudo apt-get install syslog-ng

2. Configure syslog-ng: Edit the syslog-ng configuration file, usually located at /etc/syslog-ng/syslog-ng.conf. Add the following configuration to receive logs from clients:

    source s_network { tcp(ip(0.0.0.0) port(514)); udp(ip(0.0.0.0) port(514)); };

    destination d_messages { file("/var/log/messages"); };

    destination d_apache_access { file("/var/log/apache/http-access.log"); };

    destination d_apache_error { file("/var/log/apache/http-error.log"); };

    log { source(s_network); destination(d_messages); };

    log { source(s_network); filter(f_apache_access); destination(d_apache_access); };

    log { source(s_network); filter(f_apache_error); destination(d_apache_error); };

3. Open firewall ports: Ensure that your firewall allows traffic on port 514 for both TCP and UDP:

    sudo ufw allow 514/tcp

    sudo ufw allow 514/udp

    sudo ufw reload

4. Enable and start syslog-ng:

    sudo systemctl enable syslog-ng

    sudo systemctl start syslog-ng

Setting Up the Syslog-ng Client​

On RPM-Based Systems​

1. Install syslog-ng:

    sudo yum install syslog-ng

2. Configure syslog-ng: Edit the syslog-ng configuration file on the client, usually located at /etc/syslog-ng/syslog-ng.conf. Add the following configuration to send logs to the server:

    destination d_syslog_server { tcp("syslog-server-ip" port(514)); };

    log { source(s_src); filter(f_messages); destination(d_syslog_server); };

    log { source(s_src); filter(f_apache_access); destination(d_syslog_server); };

    log { source(s_src); filter(f_apache_error); destination(d_syslog_server); };

3. Enable and start syslog-ng:

    sudo systemctl enable syslog-ng

    sudo systemctl start syslog-ng

On APT-Based Systems​

1. Install syslog-ng:

    sudo apt-get update

    sudo apt-get install syslog-ng

2. Configure syslog-ng: Edit the syslog-ng configuration file on the client, usually located at /etc/syslog-ng/syslog-ng.conf. Add the following configuration to send logs to the server:

    destination d_syslog_server { tcp("syslog-server-ip" port(514)); };

    log { source(s_src); filter(f_messages); destination(d_syslog_server); };

    log { source(s_src); filter(f_apache_access); destination(d_syslog_server); };

    log { source(s_src); filter(f_apache_error); destination(d_syslog_server); };

3. Enable and start syslog-ng:

    sudo systemctl enable syslog-ng

    sudo systemctl start syslog-ng

Conclusion​

By following these steps, you can set up a syslog-ng server on both RPM-based and APT-based Linux systems and configure clients to send logs to the server. This setup will help you centralize your log management and make it easier to monitor and analyze log data from multiple sources.

 

[dos2unix]

Understanding Filters in syslog-ng​

How Filters Work in syslog-ng​

Filters in syslog-ng are expressions used to select or filter log messages based on certain criteria, ensuring that the right messages reach the right destinations.

Defining Filters​

Filters are defined in the configuration file and can be used to match log messages based on various attributes such as severity level, facility, hostname, program name, or even the content of the log message using regular expressions.

Example filter definitions:

 filter f_messages { level(info..emerg) and not (facility(mail)); }; filter f_apache_access { program("httpd") and match("access"); }; filter f_apache_error { program("httpd") and match("error"); };

Using Filters​

Once filters are defined, they can be used in log statements to direct specific log messages to particular destinations.

Example log statements using filters:

 log { source(s_src); filter(f_messages); destination(d_syslog_server); };

log { source(s_src); filter(f_apache_access); destination(d_syslog_server); };

log { source(s_src); filter(f_apache_error); destination(d_syslog_server); };

Common Filters for System Logs and Application Logs​

In syslog-ng, filters are essential for directing specific log messages to the appropriate destinations. Here are some common filters used for system logs and application logs:

Level​

Filters log messages based on their severity level.

 filter f_info { level(info); }; filter f_warning { level(warning); }; filter f_error { level(err); };

Facility​

Filters log messages based on the facility, which represents the source of the log message.

 filter f_auth { facility(auth); }; filter f_mail { facility(mail); }; filter f_daemon { facility(daemon); };

Host​

Filters log messages based on the hostname of the machine that generated the log.

 filter f_host1 { host("hostname1"); }; filter f_host2 { host("hostname2"); };

Program​

Filters log messages based on the name of the program that generated the log.

 filter f_sshd { program("sshd"); }; filter f_httpd { program("httpd"); }; filter f_cron { program("cron"); };

Match​

Filters log messages based on a regular expression match within the message content.

 filter f_critical_errors { match("CRITICAL"); }; filter f_access_logs { match("access"); }; filter f_error_logs { match("error"); };

Netmask​

Filters log messages based on the sender's IP address or subnet.

 filter f_local_network { netmask("192.168.1.0/24"); }; filter f_remote_network { netmask("10.0.0.0/8"); };

Example Usage​

Here’s an example of how you might use these filters in a syslog-ng configuration:

 source s_network { tcp(ip(0.0.0.0) port(514)); udp(ip(0.0.0.0) port(514)); };

destination d_auth_logs { file("/var/log/auth.log"); };

destination d_httpd_access { file("/var/log/apache/http-access.log"); };

destination d_httpd_error { file("/var/log/apache/http-error.log"); };

log { source(s_network); filter(f_auth); destination(d_auth_logs); };

log { source(s_network); filter(f_httpd); filter(f_access_logs); destination(d_httpd_access); };

log { source(s_network); filter(f_httpd); filter(f_error_logs); destination(d_httpd_error); };

Summary​

• Level: Filters by severity (e.g., info, warning, error).
• Facility: Filters by the source of the log (e.g., auth, mail, daemon).
• Host: Filters by hostname.
• Program: Filters by the program name (e.g., sshd, httpd, cron).
• Match: Filters by matching content within the log message using regular expressions.
• Netmask: Filters by sender IP address or subnet.

These filters help you manage and organize your log data more effectively, ensuring that relevant logs are directed to the appropriate destinations.

 

[dos2unix]

Adding Custom Application Logs to syslog-ng​

PostgreSQL Logs​

1. Configure PostgreSQL to use syslog: Edit the PostgreSQL configuration file, usually located at /var/lib/pgsql/data/postgresql.conf or /etc/postgresql/12/main/postgresql.conf (depending on your distribution). Set the following parameters:

    logging_collector = on log_destination = 'syslog' syslog_facility = 'LOCAL0' syslog_ident = 'postgres'

2. Create a filter for PostgreSQL logs: Add the following filter to your syslog-ng configuration file:

    filter f_postgres { program("postgres"); };

3. Add a log statement for PostgreSQL logs:

    destination d_postgres { file("/var/log/postgresql.log"); };
   log { source(s_src); filter(f_postgres); destination(d_postgres); };

Nginx Logs​

1. Configure Nginx to use syslog: Edit the Nginx configuration file, usually located at /etc/nginx/nginx.conf. Add the following lines to the http block:

    log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"';
   access_log syslog:server=unix:/dev/log main; error_log syslog:server=unix:/dev/log error;

2. Create filters for Nginx logs: Add the following filters to your syslog-ng configuration file:

    filter f_nginx_access { program("nginx") and match("access"); }; filter f_nginx_error { program("nginx") and match("error"); };

3. Add log statements for Nginx logs:

    destination d_nginx_access { file("/var/log/nginx/access.log"); };
   destination d_nginx_error { file("/var/log/nginx/error.log"); };
   log { source(s_src); filter(f_nginx_access); destination(d_nginx_access); };
   log { source(s_src); filter(f_nginx_error); destination(d_nginx_error); };

Bash History Logs​

1. Configure bash to log history to syslog: Add the following line to the user's .bashrc or /etc/bash.bashrc file:

    export PROMPT_COMMAND='history -a >(logger -t bash -p local1.info)'

2. Create a filter for bash history logs: Add the following filter to your syslog-ng configuration file:

    filter f_bash_history { program("bash"); };

3. Add a log statement for bash history logs:

    destination d_bash_history { file("/var/log/bash_history.log"); };
   log { source(s_src); filter(f_bash_history); destination(d_bash_history); };

Summary​

• PostgreSQL: Configure PostgreSQL to use syslog, create a filter for PostgreSQL logs, and add a log statement.
• Nginx: Configure Nginx to use syslog, create filters for Nginx access and error logs, and add log statements.
• Bash History: Configure bash to log history to syslog, create a filter for bash history logs, and add a log statement.