Fluent Bit is a Fast and Lightweight Logs and Metrics Processor and Forwarder for Linux, OSX, Windows and BSD family operating systems. It has been made with a strong focus on performance to allow the collection of events from different sources without complexity.

In this article, we’ll get started with fluent-bit, part 1 with installing it on macOS and configuring it to tail an Apache log file.

Prerequisite:

1. Homebrew

Install fluent-bit:

• Install the fluent-bit on macOS using homebrew:

brew install fluent-bit

• Once its installed, the binary will be in location : /usr/local/bin/fluent-bit
• You can test if the binary is working correctly or not by running : fluent-bit -V which should display Fluent Bit v1.9.3 or similar version.

Install flog:

• Download the binary from GitHub :

wget https://github.com/mingrammer/flog/releases/download/v0.4.3/flog_0.4.3_darwin_amd64.tar.gz

• Extract the binary using tar :

tar -xzvf flog_0.4.3_darwin_amd64.tar.gz

• Run the flog to dump a fake apache logs into a file :

./flog > /tmp/flog.log

Configure fluent-bit :

• Now that the log file is ready and fluent-bit installed, we’ll create a file called fluent-bit.conf and append the below files :

[SERVICE]
    # by default 'info' is set, that means it includes 'error' and 'warning'.
    log_level    info
    flush           1

[INPUT]
    Name              tail
    path              /tmp/flog.log
    Read_from_Head    true

[OUTPUT]
    Name        stdout
    Match       *

• Let’s take a look at the config file to understand what its doing :
• SERVICE: This is fluent-bit’s config section
  1. flush : this means on what time interval (seconds) it should push the logs to output. Here its 1 i.e every second.
• INPUT: This defines what to collect and where to collect it from.
  1. Name: the name of the input plugin. The tail input plugin allows to monitor one or several text files. Check more on input plugin here.
  2. path: specifies path of file we want to read and its name.
  3. Read_from_Head: specifies to read the file from starting. This is necessary of the file you’re reading has already contents in it and you want to grab those too. Else anything else added to the file after fluent-bit starts will only be taken into consideration.
• OUTPUT: defines where to send the logs it read in INPUT section.
  1. Name: the name of the output plugin. In this case we’re not sending the logs to any ElasticSearch or Loki etc. We’re just routing the input to stdout. You can read more about stdout plugin here.
  2. match : Using tags, we have the ability to send certain section of logs only to outputs. Since we’re not mentioning any tags in previous section, we’re saying send everything to stdout with * .

Action time

• Let’s start the fluent-bit with the config-file we created :

fluent-bit -c fluent-bit.conf

• Your output should now be similar to :

Fluent Bit v1.9.3
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2022/05/28 21:21:10] [ info] [fluent bit] version=1.9.3, commit=, pid=41837
[2022/05/28 21:21:10] [ info] [storage] version=1.2.0, type=memory-only, sync=normal, checksum=disabled, max_chunks_up=128
[2022/05/28 21:21:10] [ info] [cmetrics] version=0.3.1
[2022/05/28 21:21:10] [ info] [output:stdout:stdout.0] worker #0 started
[2022/05/28 21:21:10] [ info] [sp] stream processor started
[1993] tail.0: [1653753034.293609000, {"log"=>"33.28.73.181 - gaylord8047 [28/May/2022:20:51:41 +0530] "GET /robust/matrix HTTP/1.0" 416 21018"}]
[1994] tail.0: [1653753034.293609000, {"log"=>"80.225.195.38 - kulas8585 [28/May/2022:20:51:41 +0530] "PUT /distributed/b2b HTTP/2.0" 200 16980"}]
[1995] tail.0: [1653753034.293609000, {"log"=>"144.165.247.63 - - [28/May/2022:20:51:41 +0530] "HEAD /clicks-and-mortar/markets/dot-com HTTP/1.0" 500 15231"}]
[1996] tail.0: [1653753034.293610000, {"log"=>"80.18.209.21 - - [28/May/2022:20:51:41 +0530] "HEAD /magnetic HTTP/1.0" 404 12309"}]

• There you go, fluent-bit working smooth as butter.
• Keep in mind, we haven’t done any filtering or parsing on the log files. We’ll do that in the upcoming part of the series.