Lambda function to monitor EC2 snapshots

A snapshot is a point-in-time copy of data. The best thing about snapshot over a normal backup is it is an effortlessness to rollback.

Prerequisites

  • Ec2 instances
  • Snapshots with tags “eg – backup”
  • Creation of IAM role and policy that will allow Lamda to interact with EC2.

Check the below policy for reference
{
“Version”: “2012-10-17”,
“Statement”: [
{
“Effect”: “Allow”,
“Action”: [
“logs:*”
],
“Resource”: “arn:aws:logs:*:*:*”
},
{
“Effect”: “Allow”,
“Action”: “ec2:Describe*”,
“Resource”: “*”
}
]
}

Steps to create IAM role and to attach policy is as below 

  • Go to Services, IAM, Create a new Role
  • Select the option Lambda and not any policy
  • Click Next and Create a Role
  • Enter the role name (Eg:ebs-lambda-worker)
  • Select the new role, and click attach policies
  • Click the option Create Policy
  • Select the option JSON and insert the content of the above snippet
  • Click the button “Review Policy button”
  • Provide a name for the policy and click the “button create policy”
  • Now select the policy that you have created and click the “button policy actions” and select the option to attach

Steps to create a Lambda function

  • Go to Services, Lambda, and click Create a Lambda Function
  • Write a name for it
  • Select Python 2.7 as a Runtime option
  • Select the previously created IAM role
  • Click Create Function
  • Paste the code below in the inline editor

import boto3
from botocore.exceptions import ClientError
from datetime import datetime,timedelta

def lambda_handler(event, context):
filters = [{‘Name’: ‘tag-key’, ‘Values’: [‘backup’]}]
#define retention period(in days)
retention_days = 10
now = datetime.now()
#create EC2 client
ec2 = boto3.client(‘ec2’)
#list of regions
regions = ec2.describe_regions().get(‘Regions’,[] )
numberofsnapshot = 0
old = 0
# search in regions for instances
for region in regions:
print “Checking region %s ” % region[‘RegionName’]
reg=region[‘RegionName’]
ec2 = boto3.client(‘ec2’, region_name=reg)
result = ec2.describe_volumes( Filters=[{‘Name’: ‘status’, ‘Values’: [‘in-use’]}])
for volume in result[‘Volumes’]:#get the volume ID of the instance
result1 = ec2.describe_snapshots(Filters=filters)
for snapshot in result1[‘Snapshots’]:#get the snapshot details and store it in to the variable snapshot
print “Checking snapshot %s which was created on %s” % (snapshot[‘SnapshotId’],snapshot[‘StartTime’])
numberofsnapshot = numberofsnapshot + 1
time = snapshot[‘StartTime’].replace(tzinfo=None)
if (now – time) > timedelta(retention_days):#check if the timedelta is greater than retention days
old = old + 1
print “for volume %s found %s snapshots number of snapshots older than the retention date is %s “% (volume[‘VolumeId’],numberofsnapshot,old)

 

  • Make sure that timeout value is more than 1 minute
  • Please be noted that this lambda function will check for the snapshots which are having a tag “backup”. Hence if you are using any other scripts to generate snapshots please do make sure that the snapshot is having the same tag
  • Now click on the Test to execute the function
  • Once the execution is completed you will be able to see the result just below the lambda function

 

Conclusion

Lambda functions currently support the following languages: Node.js, Java, C#, and Python

AWS offers the run-time platform for Lamda to execute the “serverless” code.  Hence it is extremely profitable in terms of server space and cost.

Cloud Migration using CloudEndure

With highly automated lift and shift solution CloudEndure continually replicates your source machines into a staging area in your AWS account without any downtime or impacting the performance.

Benefits of CloudEndure

  • Replication of many machines in parallel as part of large-scale migration projects
  • Non-disruptive replication and testing so business operations continue as usual.
  • Support for any source infrastructure and all applications running on supported operating systems, including databases and other write-intensive workloads
  • Highly automated orchestration minimizes project length and IT skills needed.
  • Physical machines, including both on-premises and co-location data centers
  • Virtual machines from any hypervisor, including VMware, Microsoft Hyper-V, and others
  • Replication is also supported between Regions or Availability Zones in AWS

Migration Steps

Steps need to be performed on CloudEndure portal

  1. Login to CloudEndure portal at https://console.cloudendure.com/ and create a new Migration project by clicking on the “+” button in the upper left side > Give project name > Click “Create Project”

2. Go to Setup & Info > AWS Credentials

    • Create IAM using in the destination AWS account with the Permissions mentioned in this page
    • Paste the AWS access key ID and Secret Key ID in this page and click SAVE

3. Go to Replication Settings

a. Migration source:- Select “Other infrastructure” if the source server is outside AWS

b. Migration Target:- Select the destination AWS region

c. Replication Servers:- Choose the following details for replication servers

        • Instance type for replication and converter servers
        • Disk type, available options are Fast SSD disk, ordinary disks.
        • The subnet where replication servers will be launched
        • Security Groups to apply to the Replication Servers
        • Option to enable VPN connection to the source server
        • Enable/Disable disk encryption
        • Setup Tags for replication servers (Name Tag is reserved and can’t be used)
        • Enable/Disable Network Bandwidth Throttling

d. Click “Save”

4. Go to “Machines” tab in LHS

5. Copy the URL to download CloudEndure agent and the command to install the agent.

Steps need to be performed in Source Server

Windows

  1. Connect to Source server using RDP
  2. Open any web-browser and paste the URL to download CloudEndure agent
  3. Save the installation file and open CMD prompt in the same location and execute the command to install the agent
  4. The agent will check the number of disks attached and the total GB of data to be migrated and install the CloudEndure Service
  5. Exit from RDP connection

Linux

  1. Connect to Source server using SSH
  2. Copy the line that includes the download the installer to the source server terminal and execute the command in “run the installer” section to install the agent.
  3. The agent will check the number of disks attached and the total GB of data to be migrated and install the CloudEndure Service
  4. Exit from SSH session

Steps need to be performed on CloudEndure portal

  1. Once the CloudEndure agent installation is completed in the source machine, the source machine will be shown in CloudEndure portal’s Machines section.
  2. Then replication server is created in the AWS account and starts the initial data replication.

3. The ETA for initial data replication depends on the amount of the data to be transferred and the bandwidth

Setup Target Server configuration (CloudEndure Portal)

  1. Click on “Machines” > Select the source server > BLUEPRINT

2. Select the following details:

    • Instance Type:
    • Launch Type: On-demand/Dedicated instance/Dedicated Host
    • Subnet
    • Security Group
    • Private IP
    • Elastic IP
    • Public IP
    • Placement Group
    • IAM Role
    • Use Existing instance ID
    • Initial target instance state: – Started/Stopped
    • tags
    • Disks: – Choose Disk type & Disk IOPS

3. Save BLUEPRINT

Launch Target Server

Once the Initial data sync is completed, we can perform launch action. There are 2 types of Launch modes

    • Test Mode: To test and verify that data is migrated successfully.
    • CutOver Mode:
    1. Before you start the Cutover, open the User Console > Machines page. There, verify that each Source machine you want to cutover has the following status indications:
        • DATA REPLICATION PROGRESS – Continuous Data Replication
        • ETA | LAG – n/a | none
        • STATUS – Target machine can be launched
        • MIGRATION LIFE CYCLE – Ready for Testing/ Tested/ Cutover.

2. Click Launch target Machine and select Cutover

3. Click Continue

4. Check the Cutover progress on the Job Progress window

5. Once the job is finished, get the new server details from “Machines” > “Target”

  1. Once the cutover/migration is finished remove the source server from the Migration project.

2.  Click Continue