Using regex on Syslog-ng to save to specific file locations
I had the problem of taking an external syslog feed (through UDP or TCP) that came from multiple devices and then separating them into folders that identified the feeds by customers / device type / host. The one thing I had on my side was that the hostname of the devices were broken up into three parts to help me get this information.
In the old Syslog-ng (1.6.x), there was not a way to regex out this information and then use it in the destination section. Each time we had a new combination, it required writing additional lines with additional regexs. Started to really feel it on performance. With the new branch of syslog-ng (2.0.x), this feature is now available.
You can use up to 256 different $NNN ($1 … $256) macros, but you can only use one regexp expression. This will be done using a filter.
Here’s a full example of what I had done then. What I wanted to do is put the line in a specific file based on part of the hostname. For example:
Hostname: subdomain.domain.tld
Save the log files for that device at:
/logs/$TLD/$DOMAIN/$SUBDOMAIN/$R_YEAR-$R_MONTH-$R_DAY.log
WHERE $TLD is the tld of the hostname, $DOMAIN is the domain from the hostname, and $SUBDOMAIN is the subdomain from the hostname.
I would like to be able to regex this information out of the hostname to use in the destination.
To accomplish this, I was able to use this configuration:
filter f_filter { host(”^([0-9a-zA-Z\-]+)\.([0-9a-zA-Z\-]+)\.([0-9a-zA-Z\-]+)$”); };
destination f_logs { file(”/logs/$3/$2/$1/$YEAR-$MONTH-$DAY.log”); };
log {
source(external);
filter(f_filter);
destination(f_logs);
flags(final);
};
