Bottlerocket for EKS Security

I’m going to focus this presentation on the host hardening side for the PCI compliance auditing. So, PCI, the PCI audit requirements, emphasizes the need to disable defaults, default configuration default passwords, and requires that all the server is up to date. It’s protected from vulnerabilities. And it requires that you install all the security patches available for the software you’re running.

On AWS, I think we all know about the Shared Responsibility Model for EKS. This means that AWS is responsible for the security of the EKS Control Plane, for example. But for the notes, AWS has these EKS optimized AMIs which are the ones that are run on the EKS nodes, while customers are still responsible for maintaining all the configurations and functions at the operating system level. Right. So here’s when it starts to get a little complicated, right to manage all these configurations in the context of PCI audit. So for example, for this EKS optimized AMI, AWS is responsible for creating new AMIs when there is a new patch for the software. So it is always constantly releasing new versions of the AMI. However, these default AMIs, although they’re specifically for EKS, still come with software that we need to configure to properly pass them the auditing.

AWS launched this new operating system called Bottlerocket. It is a free and open source Linux based operating system that is meant for hosting containers. And it’s very interesting because very simply it can follow the least privilege principle in the sense that it has only the software required to properly run containers that’s like their mindset. They don’t even have a shell in the host. There is no SSH server pre-installed and they have a lot of security features out of the box like SELinux enabled enforcing mode, things like that. So yeah, this operating system focuses on security maintainability. And yeah, the fact that Bottlerocket contains only this minimal software to run SDKs node, streamlines the process of maintaining secure patch environments.

There are some interesting facts about how this Bottlerocket operating system works. The first one is that all the software runs as containers. They have a control container and an ending container that are used to interact with the, like they have some API’s that are used to interact with the operating system, that’s how it’s designed. As I mentioned, there is no SSH server and no shell in the host. So they have this control container, which runs on an SSM agent, to allow you to use a session manager for example to kind of get access to the host file system. Or the most interesting fact that I found here was this Bottlerocket updater operator, Kubernetes operator, that helps us with automatically updating the OS to the latest version. So going back to that Shared Responsibility Model, for example, AWS is responsible for releasing new versions of AMIs. But it’s still the responsibility of the customer to update the nodes to the latest versions. So something like this updater is very handy, because then we can rely on the updater to perform those updates on those patches for us.

Just to show you how it looks in AWS. This is the EKS console and here you have the managed node groups. For example, whenever there is a new AMI version, you can update it here. But that’s not suitable for example, when you have like multiple clusters per customer. Then this Bottlerocket updater comes into action. So we decided to start using it. We tested it and it works very well. It performs the updates of the nodes in waves, and it performs all the necessary actions in Kubernetes environment like first it drains all the pods to different nodes then ? the node, then performs the update of the operating system in place and then reschedules ports again to that node. So one more tool to add to our setup that can help us with streamlining the management of Kubernetes cluster. What this meant for the customer was a more secure EKS environment, specifically talking about the nodes. Since we have to, we don’t have to care about configuring the nodes. You can find more information on documentation here in the Bottlerocket GitHub repo.