Rajdeep Pal
Setting up CI/CD Pipeline: Docker images with ECR, Lambda & SNS

Setting up CI/CD Pipeline: Docker images with ECR, Lambda & SNS

Rajdeep Pal's photo
Rajdeep Pal
·

4 min read

This blog will help you create a scenario where a development team needs to securely store Docker images in AWS ECR and manage their source code in AWS CodeCommit. Setting up a CI/CD pipeline that automatically builds Docker images from the CodeCommit repository and pushes them to ECR. Additionally, configuring email notifications to trigger image pushes to ECR, informing relevant team members via Amazon SNS.

Necessary Policies Required:

  1. AWSCodeCommitFullAccess

  2. AWSCodeBuildFullAccess

  3. AWSCodePipeline_FullAccess

  4. AmazonEC2ContainerRegistryFullAccess

  5. AWSLambda_FullAccess

  6. AmazonS3FullAccess

  7. CloudWatchEventsFullAccess

  8. AmazonSNSFullAccess

AWS Elastic Container Registry

Create a Repository:

  • Click on the "Create repository" button.

  • Choose a unique and descriptive name for your repository

  • Under "Visibility settings," ensure "Private" is selected. This creates a private repository accessible only within your AWS account.

AWS CodeCommit:

Create an IAM user to use AWS CodeCommit as it should not be used using root access. Attach required policies to the user. Create a repo using AWS CodeCommit. Clone the repo to the local and add the files to the created repo using Git commands.

AWS CodeBuild:

Requirements to build and push the image - buildspec.yml file & DockerHub repo. Configure the phases in the buildspec.yml file :

  • Specify the dependencies needed to be installed (if any).

  • Specify the commands the build the image.

  • Specify the post-build commands to push the image to ECR.

  • Remember to attach the required policies to the service role created for CodeBuild to grant ECR private repo access.

For example:

version: 0.2

phases:
  pre_build:
    commands:
      - echo LOGGING INTO AMAZON ECR....
      - aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin <AWS-Account-Number>.dkr.ecr.us-east-1.amazonaws.com
  build:
    commands:
      - echo BUILDING THE APPLICATION....
      - docker-compose build
      - docker tag my-nginx:latest <AWS-Account-Number>.dkr.ecr.us-east-1.amazonaws.com/nginx-private-repo:latest
  post_build:
    commands:
      - echo PUSHING THE IMAGE TO ECR....
      - docker push <AWS-Account-Number>.dkr.ecr.us-east-1.amazonaws.com/nginx-private-repo:latest
artifacts:
  files:
     - appspec.yml
     - before_install.sh
     - after_install.sh
     - start.sh

AWS CodePipeline:

Configure the pipeline to fetch code from AWS CodeCommit and build using AWS CodeBuild and push the image to AWS ECR private. During the configuration of the pipeline things to keep in mind:

  • You can select the pipeline type as "V1" and the execution mode as "Superseded.

  • On the next page, select "AWS CodeCommit" as the source provider and select the repo and branch accordingly.

  • During the build stage, you can select the source provider as AWS CodeBuild. Select your region and the CodeBuild Project name.

  • You can skip the deployment stage if you want. To deploy you need CodeDeploy access and S3 access too, as the artifacts are stored in S3 after the build.

NOTE: The CodeBuild and CodePipeline should have permission to log into your private ECR repo, so attach the required policies to their roles too.

AWS Simple Notification Service

Configure SNS Topic:

  • Navigate to the SNS service.

  • Click "Create topic."

  • Give your topic a name and create it.

AWS Lambda:

Configure a Lambda function using Python and provide a script that sends an email to all the subscribers who are subscribed to an SNS Topic. Note that the service role created for lambda should have the policies attached that allow its access to SNS and EventBridge.

For example:

import json
import boto3

def lambda_handler(event, context):

    # Configure SNS client
    sns_client = boto3.client('sns')

    # Replace 'YOUR_SNS_TOPIC_ARN' with your actual SNS topic ARN
    SNS_TOPIC_ARN = 'arn:aws:sns:us-east-1:<AWS-Account-Number>:codestar-notifications-pipeline'

    # Prepare message
    message = f"Image pushed successfully to ECR repository nginx-private-repo"

    # Publish message to SNS topic
    sns_client.publish(
        TopicArn=SNS_TOPIC_ARN,
        Message=message
    )

    print(f"Image update notification sent for nginx-private-repo")

    return {
        'statusCode': 200,
        'body': json.dumps('Notification sent!')
    }

Integrating Amazon ECR and SNS

Set up a private ECR Repo. Whenever the pipeline launches, it should build and push the image to the specified repo. As soon as the image is pushed, the EventBridge rule triggers which was created using the following settings:

  1. Name your Rule.

  2. Select the Event source as "AWS events or EventBridge partner events", Event source as "AWS Services", AWS Service as "ECR", and Event Type as "ECR Image Action" and also you can add Event Specs according to your requirements.

  3. Configure Target by selecting your Lambda function as your target.

  4. Lastly, you can add Tags and hit "Create".

Now, once the pipeline is started, an Image is pushed to ECR, and EventBridge receives a heads-up, which in turn triggers the Lambda which sends an email using the SNS Topic to all the subscribers.