How to use Firelens with AWS for Fluent Bit plugin

So AWS Firelens is a tool that allows you to configure container logging and metrics through a task definition, using Fluent Bit. So before we can really understand what a Firelens is, and appreciate what it does, we first have to understand what Fluent Bit is.

So what is Fluent Bit? Fluent Bit is an open source log processor and forwarder that primarily is used in containerized environments to collect data, such as logs and metrics and send it to multiple destinations, such as CloudWatch, or Datadog or any other metric or logging tool. Some things to note about Fluent Bit is that it is open source and multi platform, which means that it can work on almost any platform and send data to almost any destination. Another thing to note is one of its most powerful features is that it’s able to enrich logs with fine grained filters, which allows you to use logging exactly how you want to use it, you know, easier to use.

So one thing to note, that there are two things – there’s Fluent Bit and Fluentd. You may have heard of Fluentd before. Fluent Bit is a newer iteration of Fluentd. They’re made by the same people and do almost the same thing. But the key differences are that Fluent Bit is much smaller and has no dependencies, which makes it the recommended tool. So in this demo, I’ll be focusing on Fluent Bit.

So to understand what Fluent Bit does, let’s look at a configuration file, and we can see how we can use it. So every configuration file has four sections. The first section is the service section, which is where we define the global Parameters and Properties like networking and logging level and things like that. And the next three sections are where we define our plugins. And there are three types of plugins. The first, which is the second section in the configuration file, is the input plugin. That’s where we determine what type of logs or metrics we want. The plugins depend on what kind of operating system we’re working on, or what kind of application you’re using. The example over here we see we’re using the CPU plugin. So we’re collecting CPU metrics. And there are a lot of different types of plugins we can use that work on all different types of environments and applications. The next plugin is the filter plugin. That is where we determine what kind of plugin we want, or what type of filter we want. The most basic is, of course, the standard out plugin. But there are different filters for different applications and different operating systems and for whatever your needs may be. And the last section, which is the one that I’ll be focused on most in this demo, is the output section. And that is where we define the plugin that will determine the destination where we want to ingest our data into. And there are a lot of partners that have their own plugins such as Datadog and others, so almost any plugin, almost any logging or metric service has their own plugin.

But what if that’s not enough? Because what if your basic needs exceed the basic capabilities of what can be done in a task definition, but you still want to be able to use the logging configuration inside a task definition. Some examples of things that you cannot do inside a task definition is log to two separate destinations, which we’ll talk about in a little bit. The other thing you can’t do is there are all those filter plugins we’re talking about before, most of those do not work in a task definition. So if you want to use many of the filter plugins, you have to define them in a Fluent Bit configuration file, or really any other features that come with Fluent Bit, for instance, the input plugins, or those, those global parameters that we talked about, those cannot be done through a task definition. So the solution is to build a custom Fluent Bit configuration for Firelens. And that brings us to the use case. The use case we had was a client wanting to send logs to both Datadog, and keep a history of logs in CloudWatch. So in order to do that, we have to build a custom Fluent Bit configuration file, and then tell Firelens which configuration file to use.

And that brings us to the demo. So for this demo, we’re going to build a custom configuration file, call that custom Fluent Bit. And here, we’re not going to set any global parameters, or input or filter plugins, we’re going to go straight to the output plugin. And the first one we want is a CloudWatch plugin. So this is going to send logs to CloudWatch. I’m just gonna copy and paste the rest of these values, you don’t have to type them all out. It just defines, you know, what the log group name is, the log prefix and tells it to create a new group if that one does not already exist. And then we’re going to create a second output plugin here. And this one, we’re going to use the Datadog plugin, and this is going to be sent to our Datadog school. And that’s the rest of that one. And we’re going to copy, put this file inside of an S3 bucket. I already have one here, so I’m gonna skip that step. But this is the same file custom v.com. And now we’re going to create a task definition. Create a task definition over here, and then we’ll just copy over into the UI. So here, I’m gonna copy over. So this is a basic template for a task definition and you see here it’s missing both the containers. So first, we are going to add the application container. And for that, we’re just going to use a random logger container. This is just a test container that will send random logs so we can see what kind of logs we get and where they go. And we can see here that for the log configuration, we’re specifying we want it to use Firelens as the log driver. And you notice here, we don’t really configure the Firelens log driver inside the task definition, because that we did already in this configuration file. And then when we specify the Firelens container, here, we this is the AWS managed Fluent Bit container. So we don’t need to make our own image, we can use the image AWS provides for us, and we’ll specify we want to use Fluent Bit. And then here is where we specify that file that we just created. The custom Fluent Bit configuration file, and here we’re using the S3 file type. The other file type you can use is just a plain file and have it somewhere locally. The one thing to note is that here I am using an EC2 type for this task definition. If you want to use a Fargate image as a Fargate container, you cannot use an S3. You can’t keep your configuration file in an S3 bucket. You have to use the file type. Just one thing to note if you’re using Fargate and then here we go. Let’s fire up a new task definition. And let’s just go straight to the Configure JSON and we’re just gonna grab this whole thing. And there we go. Okay, so we can see it’s using a Firelens, and it’s using the Fluent Bit type. There are two containers here we have the random logger – that’s the application. And then this is the log, the log container for Firelens. And let’s call it something different, we’ll just call it Firelens. Alright. Looks good. All right, and now let’s fire it up. I made a cluster special specification. And looks good. Let’s go. Alright, let’s wait for that to start up. Alright, so let’s check Datadog and see if we can see some logs here. And it’s gonna take a moment for it to start up. There we go. And here we can see these are the logs coming from that application. These are just, as the name implies, random logs. So we can see that it is sending the logs to Datadog. And we can check inside the CloudWatch log group. And here are our logs inside CloudWatch. So you can see here that it was able to send logs to two separate locations, something that is not normally available to be done with just an AWS log container. And that is a use case that we use for our clients. And something that hopefully will come in handy in the future, as well. So if anyone has any questions, if not all wrap this up here.