Fluent - Data Collector
Fluentd is an open-source data collector for a unified logging layer. Fluentd allows you to unify data collection and consumption for better use and understanding of data.
Docu
Install Ruby
pkg_add ruby-3.1.2
Set Symlinks
ln -sf /usr/local/bin/ruby31 /usr/local/bin/ruby
ln -sf /usr/local/bin/bundle31 /usr/local/bin/bundle
ln -sf /usr/local/bin/bundler31 /usr/local/bin/bundler
ln -sf /usr/local/bin/erb31 /usr/local/bin/erb
ln -sf /usr/local/bin/gem31 /usr/local/bin/gem
ln -sf /usr/local/bin/irb31 /usr/local/bin/irb
ln -sf /usr/local/bin/rdoc31 /usr/local/bin/racc
ln -sf /usr/local/bin/rake31 /usr/local/bin/rake
ln -sf /usr/local/bin/rdoc31 /usr/local/bin/rbs
ln -sf /usr/local/bin/rdoc31 /usr/local/bin/rdbg
ln -sf /usr/local/bin/rdoc31 /usr/local/bin/rdoc
ln -sf /usr/local/bin/ri31 /usr/local/bin/ri
ln -sf /usr/local/bin/typeprof31 /usr/local/bin/typeprof
Install Fluentd
gem install fluentd
more Symlinks
ln -sf /usr/local/bin/fluentd31 /usr/local/bin/fluentd
Create Config
fluentd --setup /etc/fluent
Edit config
cat << 'EOF ' > /etc/fluent/fluent.conf
# fluent.conf
# Nginx Log
<source>
@type tail
format nginx
path /var/log/nginx/blog.stoege.net.log
pos_file /var/log/nginx-pos/blog.stoege.net.log
time_key fluentd_time
time_format %d/%b/%Y:%H:%M:%S %z
tag nginx.access
</source>
# Output
<match nginx.access>
@type file
path /var/log/fluent/blog.stoege.net
</match>
EOF
Run
fluentd -c /etc/fluent/fluent.conf
Sample Data
tail -f /var/log/fluent/blog.stoege.net/buffer.b5e6e8e8ee14db5e3a401bb38f14a6de5.log
2022-08-23T15:54:58+02:00 nginx.access {"remote":"114.119.135.215","host":"-","user":"-","time":"23/Aug/2022:15:54:58 +0200","method":"GET","path":"/tags/monitoring/","code":"200","size":"33684","referer":"https://blog.stoege.net/tags/vpn","agent":"Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)","http_x_forwarded_for":"-"}
2022-08-23T15:55:42+02:00 nginx.access {"remote":"147.182.168.107","host":"-","user":"-","time":"23/Aug/2022:15:55:41 +0200","method":"GET","path":"/index.xml","code":"304","size":"0","referer":"-","agent":"NewsBlur Feed Fetcher - 2 subscribers - https://www.newsblur.com/site/8313934/blog-stoege-net (\\x22Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.1 Safari/605.1.15\\x22)","http_x_forwarded_for":"-"}
2022-08-23T15:55:42+02:00 nginx.access {"remote":"147.182.168.107","host":"-","user":"-","time":"23/Aug/2022:15:55:42 +0200","method":"GET","path":"/","code":"200","size":"56891","referer":"-","agent":"NewsBlur Page Fetcher - 2 subscribers - https://www.newsblur.com/site/8313934/blog-stoege-net (\\x22Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.1 Safari/605.1.15\\x22)","http_x_forwarded_for":"-"}
Any Comments ?
sha256: fb02e126f5e34e4be8630c5190e9af68d2c36e91e5c4d54a11f6b70a3be6563b