AWS: Lambda — copy EC2 tags to its EBS, half 1 — Python and boto3






AWS: Lambda — copy EC2 tags to its EBS, half 1 — Python and boto3

We’ve an AWS Elastic Kubernetes Service cluster, which has a couple of WorkerNode Teams that had been created as AWS AutoScaling Teams through the use of the eksctl, see the AWS Elastic Kubernetes Service: a cluster creation automation, part 2 – Ansible, eksctl for extra particulars.

The WorkerNode Group configuration for the eksctl retains a set of Tags, which are utilized by our group for the AWS stock:

--------
apiVersion: eksctl.io/v1alpha5
sort: ClusterConfig

metadata:
  title: "{{ eks_cluster_name }}"
  area: "{{ area }}"
  model: "{{ k8s_version }}"

nodeGroups:

### Frequent ###

  - title: "{{ k8s_common_nodegroup_name }}-{{ merchandise }}-v2021-09"
    instanceType: "{{ k8s_common_nodegroup_instance_type }}"
    privateNetworking: true
    labels:
      function: common-workers
    ...
    tags:
      Tier: "Devops"
      Area: "eks.devops.{{ area }}.{ decrease }.bttrm.native"
      ServiceType: "EC2"
      Env: {{ env }}
      Perform: "Kubernetes WorkerNode"
      NetworkType: "Non-public"
      DataClass: "Public"
      AssetOwner: "{{ asset_owner }}"
      AssetCustodian: "{{ asset_custodian }}"
      OperatingSystem: "Amazon Linux"
      JiraTicket: "{{ jira_ticket }}"
      ConfidentialityRequirement: "Med"
      IntegrityRequirement: "Med"
      AvailabilityRequirement: "Med"
...
Enter fullscreen mode

Exit fullscreen mode

And people tags are utilized to AutoScale Teams:

After which are utilized to EC2 cases, created from this AutoScale Group.

The issue right here is the actual fact, that these tags should not copied to Elastic Block Retailer gadgets, hooked up to this EC2.

Additionally, in addition to Kubernetes WorkerNodes, in our AWS account, we’ve got widespread EC2 cases, and among the cases can have just one, root gadget, whereas others can have some extra disks for information to backups.

As well as, we’d like not solely copy tags from its EC2, however I additionally need to add a devoted tag, describing a disk’s operate — Root Quantity, Information Quantity или Kubernetes PVC Quantity.

How we will do it? Effectively, like nearly something in AWS, that aren’t lined by the AWS Console — through the use of the AWS Lambda service: let’s create a operate, that can be triggered when a brand new EC2 is launched and can copy this EC2’s Tags to all EBS volumes, hooked up to this occasion.

So, what do we have to out within the logic of this AWS Lambda operate:

  1. when a brand new EC2 is created — set off a Lambda operate
  2. the operate will take an EC2 ID, and can discover all associated EBS volumes
  3. will copy AWS Tags from this EC2 to all its EBS
  4. will add a brand new Tag named Function:
  5. if an EBS was created from a Kubernetes PVC and is mounted to an EC2, launched from a Kubernetes WorkerNode AWS AutoScale Group, then we’ll set the tag Function: "PvcVolume"
  6. if an EBS was created throughout a standard EC2 creation, then will test its mount level, and can resolve which worth to make use of  –  Function: "RootVolume", or Function: "DataVolume

Let’s go.



Contents



Python script: copy AWS Tags

At first, let’s write a Pythion script, take a look at it, and can go to the AWS Lambda, within the second a part of this submit.

Once we will adapt for a Lambda operate, we’ll make a fast replace on it to make it obtainable to make use of a devoted EC2 ID as a substitute.



boto3: getting an inventory of EC2 cases and their EBS volumes

The very very first thing is to authenticate in an AWS account, get all EC2 cases in a selected area, after which get an inventory of EBS volumes hooked up to every EC2.

Then, having this info, we will play with their Tags.

The script:

#!/usr/bin/env python

import os   
import boto3

ec2 = boto3.useful resource('ec2',
        region_name=os.getenv("AWS_DEFAULT_REGION"),
        aws_access_key_id=os.getenv("AWS_ACCESS_KEY_ID"),
        aws_secret_access_key=os.getenv("AWS_SECRET_ACCESS_KEY")
    )

def lambda_handler(occasion, context):

    base = ec2.cases.all()

    for example in base:

        print("n[DEBUG] EC2nttID: " + str(occasion))
        print("tEBS")

        for vol in occasion.volumes.all():

            vol_id = str(vol)
            print("ttID: " + vol_id)

if __name__ == " __main__":
    lambda_handler(0, 0)
Enter fullscreen mode

Exit fullscreen mode

Right here, within the ec2 variable we’re creating an object of the boto3.resource with the ec2 kind, and can authenticate in an AWS account through the use of the $AWS_ACCESS_KEY_ID and $AWS_SECRET_ACCESS_KEY variables. Later, in our Lambda, authentication, and authorization can be carried out with an IAWS IAM Function.

On the finish of the script, we’re calling the lambda_handler() operate, if the script was executed as a devoted program, see the Python: what is the if name == “main” ? for particulars.

Within the lambda_handler() we’re calling the ec2.instances.all() methodology to get all cases in a area, after which within the for loop for each EC2 by calling the instance.volumes.all() we’re getting an inventory of EBS volumes hooked up to this EC2.

For now, arguments to the lambda_handler() are handed as “0, 0“, and later, in Lambda, we’ll put the occasion and context objects.

Set the AWS authentication variables:

$ export AWS_ACCESS_KEY_ID=AKI***D4Q
$ export AWS_SECRET_ACCESS_KEY=QUC***BTI
$ export AWS_DEFAULT_REGION=eu-west-3
Enter fullscreen mode

Exit fullscreen mode

Run the script:

$ ./ec2_tags.py
[DEBUG] EC2
ID: ec2.Occasion(id=’i-0df2fe9ec4b5e1855')
EBS
ID: ec2.Quantity(id=’vol-0d11fd27f3702a0fc’)
[DEBUG] EC2
ID: ec2.Occasion(id=’i-023529a843d02f680')
EBS
ID: ec2.Quantity(id=’vol-0f3548ae321cd040c’)
[DEBUG] EC2
ID: ec2.Occasion(id=’i-02ab1438a79a3e475')
EBS
ID: ec2.Quantity(id=’vol-09b6f60396e56c363')
ID: ec2.Quantity(id=’vol-0d75c44a594e312a1')
…
Enter fullscreen mode

Exit fullscreen mode

Good, it’s working! We’ve obtained all ЕС2 within the eu-west-3 AWS Area, and for each EC2 obtained an inventory of its EBS hooked up.

What’s subsequent? The following is to find out how is a quantity mounted in an EC2 and understanding its mount level we’ll be capable of know if this disc is a root quantity or some extra quantity for information.

This may be carried out by getting the attachments() attribute that retains a worth for the Gadget key.

Set a brand new variable within the script referred to as device_id:

...
        for vol in occasion.volumes.all():

            vol_id = str(vol)
            device_id = "ec2.vol.Gadget('" + str(vol.attachments[0]['Device']) + "')"

            print("ttID: " + vol_id + "nttDev: " + device_id + "n")
...
Enter fullscreen mode

Exit fullscreen mode

Run the script once more:

$ ./ec2_tags.py
[DEBUG] EC2
ID: ec2.Occasion(id=’i-0df2fe9ec4b5e1855')
EBS
ID: ec2.Quantity(id=’vol-0d11fd27f3702a0fc’)
Dev: ec2.vol.Gadget(‘/dev/xvda’)
[DEBUG] EC2
ID: ec2.Occasion(id=’i-023529a843d02f680')
EBS
ID: ec2.Quantity(id=’vol-0f3548ae321cd040c’)
Dev: ec2.vol.Gadget(‘/dev/xvda’)
[DEBUG] EC2
ID: ec2.Occasion(id=’i-02ab1438a79a3e475')
EBS
ID: ec2.Quantity(id=’vol-09b6f60396e56c363')
Dev: ec2.vol.Gadget(‘/dev/xvda’)
ID: ec2.Quantity(id=’vol-0d75c44a594e312a1')
Dev: ec2.vol.Gadget(‘/dev/xvdbm’)
…
Enter fullscreen mode

Exit fullscreen mode

And right here we’ve obtained an ЕС2 with ID i-02ab1438a79a3e475, and this EC2 has two EBS volumes mounted — vol-09b6f60396e56c363 as /dev/xvda, and vol-0d75c44a594e312a1 as /dev/xvdbm.

/dev/xvda clearly is a root quantity, and /dev/xvdbm – some extra information.



boto3: including AWS Tags to an EBS

Now, let’s create a Function Tag that may maintain one from the next values:

  1. if an EBS has a Tag with the kubernetes.io/created-for/pvc/title key, then will set Function: "PvcVolume"
  2. if it isn’t a PVC quantity, then have to test its mount level, and if gadget == “/dev/xvda“, then set Function: "RootVolume"
  3. and at last, if the gadget variable has another worth, then simply mark the EBS with the Function: "DataDisk"

To do that, let’s add one other operate that can be used to set Tags, and one other small operate referred to as is_pvc(), that may test if an EBS has the kubernetes.io/created-for/pvc/title Tag:

...
def is_pvc(vol):

    strive:
        for tag in vol.tags:
            if tag['Key'] == 'kubernetes.io/created-for/pvc/title':
                return True
                break
    besides TypeError:
            return False

def set_role_tag(vol):

    gadget = vol.attachments[0]['Device']
    tags_list = []
    values = {}

    if is_pvc(vol):
        values['Key'] = "Function"
        values['Value'] = "PvcDisk"
        tags_list.append(values)
    elif gadget == "/dev/xvda":
        values['Key'] = "Function"
        values['Value'] = "RootDisk"
        tags_list.append(values)
    else:
        values['Key'] = "Function"
        values['Value'] = "DataDisk"
        tags_list.append(values)

    return tags_list
...
Enter fullscreen mode

Exit fullscreen mode

Right here, within the set_role_tag() the operate we at first are passing a worth of the vol as an argument to the is_pvc() operate, that checks Tags for the ‘kubernetes.io/created-for/pvc/title’ key. The strive/besides right here is used to know if an EBS has tags in any respect.

If the ‘kubernetes.io/created-for/pvc/title’ tag was discovered, then is_pvc() will return True, if not discovered, then False.

Then, within the if/elif/else situations we’re checking if an EBS is a PVC quantity, and in that case, then will probably be tagged as Function: "PvcVolume", if not – will test its mount level, and whether it is mounted as “/dev/xvda“, then set the Function: "RootVolume", if to – will use the Function: "DataDisk".

Add the set_role_tag() name to the lambda_handler() as an argument to the vol.create_tags() operate:

...
def lambda_handler(occasion, context):

    base = ec2.cases.all()

    for example in base:

        print("n[DEBUG] EC2nttID: " + str(occasion))
        print("tEBS")

        for vol in occasion.volumes.all():

            vol_id = str(vol)
            device_id = "ec2.vol.Gadget('" + str(vol.attachments[0]['Device']) + "')"
            print("ttID: " + vol_id + "nttDev: " + device_id)

            role_tag = vol.create_tags(Tags=set_role_tag(vol))
            print("ttTags set:nttt" + str(role_tag))
...
Enter fullscreen mode

Exit fullscreen mode

Run the script once more:

$ ./ec2_tags.py
[DEBUG] EC2
ID: ec2.Occasion(id=’i-0df2fe9ec4b5e1855')
EBS
ID: ec2.Quantity(id=’vol-0d11fd27f3702a0fc’)
Dev: ec2.vol.Gadget(‘/dev/xvda’)
Tags set:
[ec2.Tag(resource_id=’vol-0d11fd27f3702a0fc’, key=’Role’, value=’RootDisk’)]
…
[DEBUG] EC2
ID: ec2.Occasion(id=’i-02ab1438a79a3e475')
EBS
ID: ec2.Quantity(id=’vol-09b6f60396e56c363')
Dev: ec2.vol.Gadget(‘/dev/xvda’)
Tags set:
[ec2.Tag(resource_id=’vol-09b6f60396e56c363', key=’Role’, value=’RootDisk’)]
ID: ec2.Quantity(id=’vol-0d75c44a594e312a1')
Dev: ec2.vol.Gadget(‘/dev/xvdbm’)
Tags set:
[ec2.Tag(resource_id=’vol-0d75c44a594e312a1', key=’Role’, value=’PvcDisk’)]
Enter fullscreen mode

Exit fullscreen mode

Let’s test volumes of the i-02ab1438a79a3e47 EC2 occasion.

Its root quantity vol-09b6f60396e56c363:

And a Kubernetes PVC –  vol-0d75c44a594e312a1:

Good, we’ve added the Function tag creation, and now want so as to add a capability to repeat AWS Tags from the EC2 to its EBS.



boto3: copy AWS Tags from an EC2 to its EBS

Tags copy could be moved to a devoted operate too, let’s title it copy_ec2_tags(), and it’ll settle for an argument, the place we’ll move an EC2 ID:

...
def copy_ec2_tags(occasion):

    tags_list = []
    values = {} 

    for instance_tag in occasion.tags:

        if instance_tag['Key'] == 'Env':
            tags_list.append(instance_tag)
        elif instance_tag['Key'] == 'Tier':
            tags_list.append(instance_tag)
        elif instance_tag['Key'] == 'DataClass':
            tags_list.append(instance_tag)

    return tags_list
...
Enter fullscreen mode

Exit fullscreen mode

Within the operate, in a loop, we’re checking all tags of the occasion, and if will discover any of the three tags laid out in our operate, they are going to be appended to the checklist tags_list[], that later can be handed to the vol.create_tags().

Add copy_ec2_tags() execution to the lambda_handler():

...
def lambda_handler(occasion, context):

    base = ec2.cases.all()

    for example in base:

        print("n[DEBUG] EC2nttID: " + str(occasion))
        print("tEBS")

        for vol in occasion.volumes.all():

            vol_id = str(vol)
            device_id = "ec2.vol.Gadget('" + str(vol.attachments[0]['Device']) + "')"
            print("ttID: " + vol_id + "nttDev: " + device_id)

            role_tag = vol.create_tags(Tags=set_role_tag(vol))
            ec2_tags = vol.create_tags(Tags=copy_ec2_tags(occasion))
            print("ttTags set:nttt" + str(role_tag) + "nttt" + str(ec2_tags))
...
Enter fullscreen mode

Exit fullscreen mode

Run:

$ ./ec2_tags.py
[DEBUG] EC2
ID: ec2.Occasion(id=’i-0df2fe9ec4b5e1855')
EBS
ID: ec2.Quantity(id=’vol-0d11fd27f3702a0fc’)
Dev: ec2.vol.Gadget(‘/dev/xvda’)
Tags set:
[ec2.Tag(resource_id=’vol-0d11fd27f3702a0fc’, key=’Role’, value=’RootDisk’)]
[ec2.Tag(resource_id=’vol-0d11fd27f3702a0fc’, key=’DataClass’, value=’Public’), ec2.Tag(resource_id=’vol-0d11fd27f3702a0fc’, key=’Env’, value=’Dev’), ec2.Tag(resource_id=’vol-0d11fd27f3702a0fc’, key=’Tier’, value=’Devops’)]
…
[DEBUG] EC2
ID: ec2.Occasion(id=’i-02ab1438a79a3e475')
EBS
ID: ec2.Quantity(id=’vol-09b6f60396e56c363')
Dev: ec2.vol.Gadget(‘/dev/xvda’)
Tags set:
[ec2.Tag(resource_id=’vol-09b6f60396e56c363', key=’Role’, value=’RootDisk’)]
[ec2.Tag(resource_id=’vol-09b6f60396e56c363', key=’Env’, value=’Dev’), ec2.Tag(resource_id=’vol-09b6f60396e56c363', key=’DataClass’, value=’Public’), ec2.Tag(resource_id=’vol-09b6f60396e56c363', key=’Tier’, value=’Devops’)]
ID: ec2.Quantity(id=’vol-0d75c44a594e312a1')
Dev: ec2.vol.Gadget(‘/dev/xvdbm’)
Tags set:
[ec2.Tag(resource_id=’vol-0d75c44a594e312a1', key=’Role’, value=’PvcDisk’)]
[ec2.Tag(resource_id=’vol-0d75c44a594e312a1', key=’Env’, value=’Dev’), ec2.Tag(resource_id=’vol-0d75c44a594e312a1', key=’DataClass’, value=’Public’), ec2.Tag(resource_id=’vol-0d75c44a594e312a1', key=’Tier’, value=’Devops’)]
...
Enter fullscreen mode

Exit fullscreen mode

And test:

The entire script now seems to be like subsequent::

#!/usr/bin/env python

import os
import boto3

ec2 = boto3.useful resource('ec2',
        region_name=os.getenv("AWS_DEFAULT_REGION"),
        aws_access_key_id=os.getenv("AWS_ACCESS_KEY_ID"),
        aws_secret_access_key=os.getenv("AWS_SECRET_ACCESS_KEY")
    )

def lambda_handler(occasion, context):

    base = ec2.cases.all()

    for example in base:

        print("[DEBUG] EC2nttID: " + str(occasion))
        print("tEBS")

        for vol in occasion.volumes.all():

            vol_id = str(vol)
            device_id = "ec2.vol.Gadget('" + str(vol.attachments[0]['Device']) + "')"
            print("ttID: " + vol_id + "nttDev: " + device_id)

            role_tag = vol.create_tags(Tags=set_role_tag(vol))
            ec2_tags = vol.create_tags(Tags=copy_ec2_tags(occasion))
            print("ttTags set:nttt" + str(role_tag) + "nttt" + str(ec2_tags) + "n")

def is_pvc(vol): 

    strive:
        for tag in vol.tags:
            if tag['Key'] == 'kubernetes.io/created-for/pvc/title':
                return True
                break
    besides TypeError:
            return False

def set_role_tag(vol):

    gadget = vol.attachments[0]['Device']
    tags_list = []
    values = {}

    if is_pvc(vol):
        values['Key'] = "Function"
        values['Value'] = "PvcDisk"
        tags_list.append(values)
    elif gadget == "/dev/xvda":
        values['Key'] = "Function"
        values['Value'] = "RootDisk"
        tags_list.append(values)
    else:   
        values['Key'] = "Function"
        values['Value'] = "DataDisk"
        tags_list.append(values)

    return tags_list

def copy_ec2_tags(occasion):

    tags_list = []
    values = {} 

    for instance_tag in occasion.tags:

        if instance_tag['Key'] == 'Env':
            tags_list.append(instance_tag)
        elif instance_tag['Key'] == 'Tier':
            tags_list.append(instance_tag)
        elif instance_tag['Key'] == 'DataClass':
            tags_list.append(instance_tag)
        elif instance_tag['Key'] == 'JiraTicket':
            tags_list.append(instance_tag)

    return tags_list

if __name__ == " __main__":
    lambda_handler(0, 0)
Enter fullscreen mode

Exit fullscreen mode

And we’re carried out right here, and now can proceed with an AWS Lambda operate. See the subsequent half within the AWS: Lambda — copy EC2 tags to its EBS, part 2 — create a Lambda function submit.

Initially printed at RTFM: Linux, DevOps, and system administration.




Abu Sayed is the Best Web, Game, XR and Blockchain Developer in Bangladesh. Don't forget to Checkout his Latest Projects.


Checkout extra Articles on Sayed.CYou

#AWS #Lambdacopy #EC2 #tags #EBS #half #1Python #boto3