AWS CodeBuild with GitHub

Let’s discuss the problems. The main problem is that while creating an AWS CodeBuild project with a source as GitHub, which is a single GitHub repository, it will automatically create a GitHub webhook for your repository. If you want to use even change, for example, if you commit or push directly to your repository, it will automatically trigger your CodeBuild project and build whatever is in the build stages. That is defined by AWS itself, it will automatically manage and create those webhooks. You don’t have to do anything. Up until here, everything is fine. But the problem comes if you want to create more than 20 CodeBuild projects because each CodeBuild project, it will create a single webhook. GitHub repository has a hard limit of 20 webhooks per repository. As I said earlier, my client and I went forward to discuss this with GitHub support if there is any chance to increase this limit, from 20 to 50 or 60. But this was not supported. And they suggested using an alternative approach. So the 21st CodeBuild project was failing because it does not have any more space to manage and handle the GitHub webhooks. We tried one alternative to use a custom webhook. But the thing is that it’s one-directional. If we create a custom webhook, I don’t think we can use it or mention it in the AWS console or AWS ecosystem. And that’s going to be a complex thing to handle. We further explored what the best alternative solution is.

The solution that I came up with for this limitation for my client is to use an AWS CodeStar connection, a connection that is directly associated with AWS CodeSuite code. It means it has all the CodeSuite stuff like CodeBuild, CodeDeploy, CodePipeline, etc. When this connection is created, a single integrated GitHub application will be created and automatically configured in your GitHub repository settings. I’ll be showing how this application looks in the demo. That’s one step of the solution to using AWS CodeStar connection. It will prompt you to use your GitHub credentials and to point AWS to your concerned GitHub repository, and it will establish that connection for you. Then you can use or consume that connection within your AWS CodeSuite. Initially, this was the limitation of CodeBuild. But to overcome that, we have to use AWS CodePipeline with CodeBuild. Even if we just want to build something with CodeBuild, we have to introduce the CodeBuild project into our CodePipeline. Using the CodePipeline with CodeBuild, the CodeStar connection will act as a GitHub source. In CodePipeline, the source would be the GitHub directory, whereas, in the previous limitation for CodeBuild, the direct source was GitHub, which was making these webhook problems. So it will be replaced by a CodeStar connection using CodePipeline.

The first step is to set the pipeline source as GitHub. It supports two versions. Recently, GitHub and AWS introduced a new setup version, version two, which supports this CodeStar connection. For the CodeBuild project, the CodeBuild source will not be GitHub, but its source will be directly AWS CodePipeline because CodePipeline will be the one who will provide the source in GitHub to CodeBuild as an artifact. This solution will not create any webhook. It will just use a single point of source for the image changes. Point of source, which will be directly associated with our GitHub repository using CodeStar connection.

To explain this solution more explicitly, this is a basic flow diagram for the solution. This is our GitHub repository and this is our AWS CodeStar connection. Assuming that this is already created, and the repository is already in place. While creating the connection, we will define the repository name and its branch. CodeStar connection will configure a GitHub application in our GitHub repository. This will be automatically generated. We have AWS CodePipeline and CodeBuild in place. Once any user commits or pushes to the GitHub repository, it will generate an event, whether a commit or push, and it will trigger the CodePipeline for us. Now, you see the GitHub repository is this, but the source will be from the CodeStar connection, which is basically this one – index zero, index one, and index two. This source will be from the GitHub repository in the form of a CodeStar connection, and two CodePipeline. This source, everything in the Git repository will be provided to CodeBuild as a source in the form of artifacts, and it will run all the build stages that you have written in your CodeBuild project.

This is the landing page of AWS developer tools, which consists of CodeBuild, CodeDeploy, CodePipeline, CodeArtifact, and CodeCommit. In the settings, you can see there is an option, connections, click on this. Here, you can create one connection, which is actually a CodeStar connection. It will simply ask you for a source provider, Bitbucket, GitHub, or GitHub Enterprise Server. For our demo, we’ll use GitHub. Here we just provide a connection name and hit connect to GitHub. It will open up a dialog box, which will automatically redirect you to the GitHub landing page, asking for your credentials and repository name, and branch. We don’t have to do it yet, because one connection is already created – this one. For this connection, we have a unique Amazon resource name – this one. The main configuration is handled in your GitHub repository. This connection was created. Let’s go to our GitHub repository, go to Settings, and scroll down, you can see the integrations for GitHub applications. Here, you can see that AWS CodeStar automatically created an AWS connector application from GitHub. Click Configure, you can see what kind of permission AWS has granted us to use this repository. We selected one demo repository. I haven’t selected any branch because in my case, I was handling the branch name in the AWS CodeSuite itself. So that’s the alternative solution for CodeStar.

Now, let’s go back to the repository. Let me show you the webhook that we were previously using. Go to the settings and go to webhooks. This was the webhook that AWS CodeBuild was automatically created for us. If we just go ahead and remove this, it will disconnect event change and trigger stops. Let’s go back to settings, go to developer tools. I’ll show you I have created two different build projects. This one was the legacy CodeBuild project, Yasir-build-0, which has a source provided directly as GitHub. This CodeBuild automatically created that automated webhook for us, which we want to avoid. For that, we have created another CodeBuild Yasir-build-1 and here you can see the difference. The source provider is AWS CodePipeline. The reason is that this CodeBuild project is being used in stage four of AWS CodePipeline. These two CodeBuild projects are using the same repository, using the same GitHub as a source but in a different fashion.

Let’s go to the CodePipeline. Here, you can see the pipeline yasir-test. If we just open it, you can see the source is GitHub version two, and the build is AWS CodeBuild, which I just showed you earlier in the build console. That’s why it is trying the source provider as CodePipeline. If we just try to edit it and edit the source stage. You see the action provider is GitHub version two. Here, we have just provided the error for the CodeStar connection. It will just prompt you that your GitHub connection is ready to use. Here we have defined the demo repo, actually, we haven’t defined it, the CodeStar connection automatically filled it for us. But I use the branch name here because I was not handling the branch name in the CodeStar connection because it’s convenient for us to change the branch name whenever we want. Start the pipeline on source code change. By checking this option, it will automatically trigger the CodePipeline for us when there is an event change, or when a user or someone will try to push something and it will automatically trigger the CodePipeline for us. So other just basic options. Let’s try to trigger this. I’ll be committing a test commit here. Okay, the commit is done. If I just hit refresh, you can see that the pipeline has been triggered, and it’s in progress. It has checked out the GitHub repository and it has triggered the CodeBuild. Let’s double-check to see if AWS created an extra webhook for us. No, this was the old one, but it has not created the webhook for us. Using this AWS CodeStar connection with an integrated GitHub application, you can create many AWS CodeBuild projects with CodePipeline without any limitation of using webhooks. Of course, it will be using a webhook in the back end, but it’s not managed by us. It’s strictly managed by AWS. It has created and scoped the legacy webhooks, which is this one. Once these webhooks reach the number 20, it will complain “please remove any stale webbooks from the GitHub console and try again.” But using this email, it will make your life easier for introducing new CodeBuild projects for different branches or different applications.

And you see it has successfully built a build project so nothing was blocking it from triggering it. That was a quick demo for this solution that I presented to one of my clients so I thought to share it with you guys.