NGINX logging to syslog

Recently I've been tasked with creating a central syslog server. These are very useful when one maintain couple of boxes (or couple hundred and more) as it can provide a single point of checking out on what's up with the machines. If it's combined properly with metrics it serves as a super-boosting way of maintaining the overview of the entire infrastructure.

When it comes to NGINX, it defaults to storing log files in plain text. It's a sane default and I don't see a good reason to ship it in any other fashion. However, sometimes the needs change. It was the case for me -- I'm using rsyslog^[1] for all of the OS logs and it felt natural to me to have NGINX invited to join the party. As rsyslog client is pushing all of its logs further to the centralized server part already, I wanted to have NGINX logs included in the stream.

There are at least two ways of achieving this goal. First one is to instruct rsyslog to read the NGINX log files from /var/log/nginx and to push them forward whenever there's something new. This would require using imfile module from rsyslog and depend on inotify which is very fast and low on resources.^[2] I didn't go this route.

Second way is to instruct NGINX to push its log files directly to syslog. This approach has the benefit of streaming the logs directly towards /dev/log and have them picked up by rsyslog client to send them further to the central syslog server. Or store them locally. Or both. I still wanted to keep logs locally for a quick lookup. This gives me exactly the flexibility that I need while having essentially no delay.

Logging to syslog

NGINX documentation has section dedicated to this topic.^[3] That being said, it still took me a moment or two to actually get it to work. There's also rsyslog's part that needs handling.

Both error_log and access_log can be specified globally or on per vhost basis. Because I was only playing around, I specified the following two globally:

access_log syslog:server=unix:/dev/log,tag=nginx,nohostname,severity=info combined;
error_log  syslog:server=unix:/dev/log,tag=nginx,nohostname,severity=error;

This will catch all error and access logs from all of the vhosts configured in NGINX and stream them to syslog via UNIX socket. This step switches off logging to local files entirely. Logs from NGINX at this point land in /var/log/syslog or /var/log/messages (depending on the configuration of rsyslog).

Of course instead of using local UNIX socket to pass the logs to syslog, one can specify UDP socket instead and send them remotely straight-away. The problem with this approach is that it doesn't introduce any kind of data obfuscation, not even simplest one. This means that there's possibility that one would be sending access logs in plain text over the network -- it would be trivial to intercept this data, that potentially contains sensitive information. UDP is also an unreliable protocol, so if data that you are sending is important to you, you'd likely want to use at least TCP or RELP preferably.

server=unix:/dev/log specifies UNIX socket to which NGINX should stream its logs to. tag=nginx is actually default, but I always tend to be explicit when it comes to tagging the logs.^[4] nohostname is there cause when using UNIX socket we are going through syslog itself that will add both hostname and correct timestamp -- not having this is helpful in the event of sending logs to a remote destination. severity=info/severity=error specifies severity to which each kinds of logs should be assigned to. It made sense to me to have access logs streaming with information severity and to align error log with error severity.

Intercepting with syslog

The above setup streams all of the NGINX logs to syslog. That's great, but I don't really want to end up with HTTP server specific log files in my /var/log/syslog. For this reason there's one small "tweak" necessary on the rsyslog configuration side. For local storing of the logs produced by NGINX, I decided to go with the same location as the one that comes by default.

if $syslogfacility-text == 'local7' and $syslogseverity-text == 'info' and $programname == 'nginx' then /var/log/nginx/access.log & stop
if $syslogfacility-text == 'local7' and $syslogseverity-text == 'error' and $programname == 'nginx' then /var/log/nginx/error.log & stop

These two lines can safely be dropped to /etc/rsyslog.d/10-nginx.conf and once rsyslog daemon is reloaded, NGINX's logs are going to land yet again in plain-text files under /var/log/nginx. $syslogfacility-text defines which facility should be filtered,^[5] then $syslogseverity-text defines whether it's access or error logs and then just re-assuring that it's actual NGINX's logs only, $programname has to be set. & stop is a special sign that tells rsyslog to stop processing given logs here -- so once access or error logs are landing in the text files, they can no longer be further processed in the stream (as they are no longer in it).

Log rotation

There's one last thing. NGINX on Debian and Ubuntu comes with its own logrotate configuration. The problem is that it defaults to the www-data user,^[6] so whenever log rotation will kick in, it will change the default user from syslog. Configuration lies in /etc/logrotate.d/nginx and line create 0640 www-data adm can easily be changed to create 0640 syslog adm.

Closing notes

I'm happy with the result. I kept local files for quick check-ups whenever needed, but still have the flexibility to push them forward to the central syslog server. Also having these logs in the stream opens up further opportunities for filtering them and potentially pushing to different kinds of output.^[7]

While I still prefer straight-forward syntax of syslog-ng, I wanted to avoid as much as possible differing from whatever serves as default choice in the given distro I'm working with -- this includes rsyslog being the default syslog server.