Delete old AMI’s by filtering with tags using boto3 and Lambda

Hello,

When you are building custom AMI’s in AWS account you will need to manage them by deleting the old AMI’s and keep only few latest images. For this you can use the below python code in Lambda function. I took the below code as reference from here and modified it to delete the AMI’s by filtering the images which has only specified tags.

Filtering the images with tags is important as different teams/projects will be having their images and it avoids accidental deletion of the wrong images.

Note: Before executing this code make sure your AMI’s are tagged.

    Code explanantion:

* First import libraries datetime, boto3 and time.
* Next get the ec2 connection session using boto3.
* Assign a variable older_days and pass the value as days (all images which are older than specified days from the present date will be filtered)

* Invoke the main function lambda_handler and then
* Invoke the function get_ami_list by passing older_days as a parameter

* Function get_ami_list uses ec2 descirbe_images to get all the images details which has specified ownerid as the owner
* Next it will invoke the function get_delete_date, calculates and finds out the date which is 5 days past from the present date
* Next the images will be filtered according to the specified tag value and if the image is older then 5 days.
* Then images are further filtered if older than 5 days and deregistered by invoking function delete_ami

from datetime import datetime, timedelta, timezone
import boto3
import time

client.ec2 = boto3.client('ec2', region_name='us-east-1')

#Here all images which are older than 5 days from the present date will be filtered
older_days = 5

def lambda_handler(event, context):
    get_ami_list(older_days)

def get_ami_list(older_days):
    amiNames = client.ec2.describe_images(Owners=['123456789123'])
    print(amiNames)
    today_date = datetime.now().strftime('%d-%m-%Y')
    print("Today's date is " + today_date)
    deldate1 = get_delete_date(older_days)
    print("AMI images which are older than " + str(deldate1) + " will be deregistered")
    for image in amiNames['Images']:
        taginfo = image['Tags']
        for tagName in taginfo:
			#Filter only the images having tag value as Proj1AMI
            if (tagName['Value'] == 'Proj1AMI'):
                ami_creation = image['CreationDate']
                imageID = image['ImageId']
                print("=================================================")
                print("Image id is " + imageID)
                print("Creation date for above image is " + ami_creation)
                if (str(ami_creation) < str(get_delete_date(older_days))):
                    print("This AMI is older than " + str(older_days) + " days")
                    delete_ami(imageID)

def get_delete_date(older_days):
	delete_time = datetime.now(tz=timezone.utc) - timedelta(days=older_days)
	return delete_time;

def delete_ami(imageID):
	print("Deregistering Image ID: " + imageID)
	client.ec2.deregister_image(ImageId=imageID)
Advertisements

Update SSM parameter store on another AWS account using AssumeRole

Hi,

In this post we are going to update the SSM parameter store in 2nd AWS account with the details from 1st AWS account. For this we will create a AWS Lambda function with python code. The python code will assume the role from another account and uses the temporarily generated STS credentials to connect and update the SSM parameter on the 2nd AWS account.

Create a Lambda function by selecting Python 2.7 and add the below code into it


#!/usr/bin/python

import boto3
import time

ssmparam = boto3.client('ssm')

account_id = '112211221122'
account_role = 'AssumeRole-SSM'
region_Name = 'us-east-1'

AmiId = 'ami-119c8dc1172b9c8e'

def lambda_handler(event, context):
    print("Assuming role for account: " + account_id)
    credentials = assume_role(account_id,account_role)

	#Call the function to update the SSM parameter with value
    updateSSM_otherAccount(credentials,region_Name,account_id)

def assume_role(account_id, account_role):
    sts_client = boto3.client('sts')
    role_arn = 'arn:aws:iam::' + account_id + ':role/' + account_role
    print (role_arn)

    '''Call the assume_role method of the STSConnection object and pass the role
    ARN and a role session name'''

    assuming_role = True
    assumedRoleObject = sts_client.assume_role(RoleArn=role_arn,RoleSessionName="NewAccountRole")
    print (assumedRoleObject['Credentials'])

    '''From the response that contains the assumed role, get the temporary
    credentials that can be used to make subsequent API calls'''
    return assumedRoleObject['Credentials']
    
def updateSSM_otherAccount(creds, region_Name, account_id):
    client1 = boto3.client('ssm',region_name=region_Name,aws_access_key_id=creds['AccessKeyId'],aws_secret_access_key=creds['SecretAccessKey'],aws_session_token=creds['SessionToken'])

    ssmparam_update = client1.put_parameter(Name='DevAMI',
            Description='the latest ami id of Dev env',
            Value=AmiId, Type='String', Overwrite=True)

Steps to configure AssumeRole

Note: Make sure to modify the account id’s in the below json policy

1. Add the inline policy to role attached to the lambda in 1st AWS account (556655665566)

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "123",
            "Effect": "Allow",
            "Action": [
                "sts:AssumeRole"
            ],
            "Resource": [
                "arn:aws:iam::112211221122:role/AssumeRole-SSM"
            ]
        }
    ]
}

2. Create a role in 2nd AWS account (AssumeRole-SSM) (112211221122), edit the trust relationship and add below policy.
Attach EC2 full permissions to this role so that we will get access to SSM parameter store

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "sts:AssumeRole"
            ],
            "Resource": "arn:aws:iam::556655665566:role/lambda-ec2-role"
        }
    ]
}

Delete file in sub-directory of S3 using Python

Hi All,
We use boto3 libraries to connect to S3 and do actions on bucket for objects to  upload, download, copy, delete. But let’s say if you want to download a specific object which is under a sub directory in the bucket then it becomes difficult to its less known on how to do this.

Below are few python script examples on using prefix of the subdirectory with boto and boto3 libraries

Example 1: Copy a file/object which is residing in a subdiretory of Bucket1 to Bucket2

import boto
conn = boto.connect_s3()

srcBucket = conn.get_bucket('testProjBucket-1') #Source Bucket name
dstBucket = conn.get_bucket('testProjBucket-2') #Destination Bucket name
fileName='test.txt'

dstBucket.copy_key('Dir2/subDir2/'+fileName,srcBucket,'Dir1/subDir1/'+fileName)

Example 2: Downloads the test.txt from bucket ‘testProjBucket-1’ to the local system path /home/ec2-user/mydownloads/
Here the downloaded file name will be as hai.txt

import boto3
s3 = boto3.resource('s3')

fileName="test.txt"
prefix1=('Dir1/subDir1/'+fileName)

s3.meta.client.download_file('testProjBucket-1', prefix1, '/home/ec2-user/mydownloads/hai.txt')

Example 3: Delete a specific object from a specific sub-directory inside a bucket (Using boto libraries)

import boto
conn = boto.connect_s3(region_name='', aws_access_key_id = '', aws_secret_access_key = '')

fileName = "test.py"

srcBucket = conn.get_bucket('testProjBucket-1')
srcBucket.delete_key('Dir1/subDir1/'+fileName)

Example 4: Delete a specific object from a specific sub-directory inside a bucket (Using boto3 libraries)

import boto3
client = boto3.client('s3', region_name='us-east-1', aws_access_key_id = '', aws_secret_access_key = '')
fileName="test.txt"

prefix1=('Dir1/subDir1/'+fileName)

response = client.delete_object(
Bucket='testProjBucket-1',
Key=prefix1
)

Note: There is no move command for object in boto3 library. We can only use copy command. But we can use move in the aws-cli

Fetch the Elastic Beanstalk environment details using python script

Hi there!!

Its been quite sometime and have been busy working on multiple technologies. Recently my lead asked me to create a python script to fetch the minimum and maximum instances count of all the Elastic beanstalk environments. It was great to work on this requirement. Below is the python script

def get_details():
	row1=['Application Name','Environment Name','Min Count','Max count']
	with open('EB-instances-count',"a") as csvDataFile:
		writer = csv.writer(csvDataFile)
		writer.writerow(row1)
	try:
		eb = boto3.client('elasticbeanstalk',"us-east-1")
		NameInfo=eb.describe_environments()
		for names in NameInfo['Environments']:
			app_name=(names['ApplicationName'])
			env_name=(names['EnvironmentName'])  
			response = eb.describe_configuration_settings(
				EnvironmentName=env_name,
				ApplicationName=app_name
			 )
			minCount=response['ConfigurationSettings'][0]['OptionSettings'][4]
			maxCount=response['ConfigurationSettings'][0]['OptionSettings'][3]
			minVal=minCount['Value']
			maxVal=maxCount['Value']
			print "Gathering count for Environment: " + env_name
			fields=[app_name,env_name,minVal,maxVal]
			with open('EB-instances-count.csv',"a") as csvDataFile:
				writer = csv.writer(csvDataFile)
				writer.writerow(fields)
	except ClientError as e:
		if e.response['Error']['Code'] == "InvalidParameterValue":
			print env_name + " Environment not found, so skipping it"
		pass
	
if __name__ == '__main__':
	get_details()

In this I am getting the application names and environment names of Elastic beanstalk in a region and parsing through them and fetching the min and max instances count. Also after fetching the counts I am writing them to .csv (spreadsheet) file. We can run this script at any time to know the present count of instances being used. This script can be further updated/modified to fetch different information of the environments in Elastic beanstalk.

The interesting part would be to filter the required information from the response. And other thing is lets say if the environment is deleted it will take sometime to disappear from the console and we might see error as we can still get the environment name but not its settings as its already deleted right.
So in this case we have to capture the particular error the exception part and ignore it.

Note: Be careful about the indentation 🙂

Let me know for any questions. Thanks

EBS snapshots deletion by filtering tags

Hey guys!!

Having daily backups of your data is the most important thing in IT industry. EBS snapshots are used to backup Amazon EBS volume with data. Taking regular backup of the volumes decreases the risk of disaster incase of failures. For more detail refer to this post here

Here we taking EBS snapshots for Production environment daily and its not required to have many snapshots as the cost will increase. So in such cases we will be deleting the snapshot after 10 days from the backup date, so that we will endup having 10 snapshots at any given point of time.

The below python script will uses the boto3 library to connect to AWS and fetch the details of services. When a EBS snapshot is created for a EC2 instance, there will be a tag created for snapshot with instanceId details and DateToDelete key with value of future 10th day date.

We will be using two arrays to filter the snapshot tags with key ebsSnaphots_clean:true and instance tags with Environment:Prod
Next we will use for loop to parse through all the ec2 instance details which have tag value and key as Environment:Prod

Similarly we will parse through the EBS snapshots with ebsSnaphots_clean:true and Deletion_date having today’s date.
Next we will fetch the tags and compare the snapshot instanceID with the respective EC2 instanceID of production environment and if they match then that respective snapshot will be deleted.

import boto3
import datetime
import dateutil
from dateutil import parser
from boto3 import ec2

ec = boto3.client('ec2')

def lambda_handler(event, context):
    Deletion_date = datetime.date.today().strftime('%Y-%m-%d')
    firstFilter = [
        {'Name': 'tag-key', 'Values': ['DateToDelete']},
        {'Name': 'tag-value', 'Values': [Deletion_date]},
		{'Name': 'tag-key', 'Values': ['ebsSnaphots_clean']},
		{'Name': 'tag-value', 'Values': ['true']},
    ]

    secondFilter = [
        {'Name': 'tag-key', 'Values': ['Environment']},
		{'Name': 'tag-value', 'Values': ['Sandbox']},
    ]	

    snapshot_details = ec.describe_snapshots(Filters=firstFilter)
    ec2_details = ec.describe_instances(Filters=secondFilter)
	
    for myinst in ec2_details['Reservations']:
        for instID in myinst['Instances']:
            print "The instanceID is %s" % instID['InstanceId']
            Instance_ID = instID['InstanceId']
            for snap in snapshot_details['Snapshots']:
                print "Checking Snapshot %s" % snap['Snapshot_Id']
                for tag in snap['Tags']:
                    if tag['Key'] == 'snap_InstanceID':
                        match_instance = tag['Value']
                        if Instance_ID == match_instance:
                            print "Checking Snapshot %s" % snap['Snapshot_Id']
                            print "the instanceID " +Instance_ID+ " matches with the Snapshot assigned instanceID tag " +match_instance+ " for snapshot %s" % snap['Snapshot_Id']
                            print "Deleting snapshot %s" % snap['Snapshot_Id']
                            ec.delete_snapshot(SnapshotId=snap['Snapshot_Id'])
                        else:
                            print "The instance " +Instance_ID+" is of different environment and do not match with snapshot "+match_instance
                    else:
                        print "no matches"

Note: Please check and take care of indentation

Thanks!!