Introduction

For all AWS Organizations administrators worldwide, how do you handle the events notified by AWS Health? I imagine you receive numerous notifications daily when utilizing AWS services.

These events need to be properly managed as they can have a significant impact on the availability and reliability of your systems. Recently, there have been significant events scheduled, such as the Amazon RDS certificate update in August and the end of support for the AWS Lambda Python 3.8 runtime in October.

If you're managing multiple accounts within your organization, manually checking events and gathering relevant information can take time and effort.

I have developed a CLI tool called AWS Health Exporter to address this challenge.

Key Features

AWS Health Exporter is a command-line tool for retrieving event information from the organizational view of AWS Health. It allows you to filter events by service name, status, and more and export details of the relevant accounts and resource IDs to a CSV file.

• AWS Organizations Support: Retrieves information from the organizational view of AWS Health. It cannot be used with standalone accounts, but there is an option to output data for a single account only.

• CSV Export: Data is formatted and exported in CSV format, making it easy to save, share, and analyze.

• Event Filtering: Filters events by conditions such as service name and status, making it easier to find the events you're looking for.

• Resource Filtering: Only retrieves resources matching specific status codes (IMPAIRED, UNIMPAIRED, UNKNOWN, PENDING, or RESOLVED).

About AWS Health Organizational View

Enabling the organizational view allows you to aggregate AWS Health events for all accounts within the organization. Data is retained for 90 days, and users/roles of the organization's management or delegated administrator accounts can access the information.

You can set it up and refer to it from "Your organization health" in the AWS Health dashboard.

In the organizational view, you can check information for each event, such as:

• Affected accounts

• Number of affected resources and breakdown of their statuses

• Resources affected within each account

This tool can export all this information to a CSV file!

Prerequisites for using the tool

• The organizational view of AWS Health is enabled.

• AWS authentication credentials to access AWS Health and AWS Organizations

  • Authentication credentials for the management or delegated administrator accounts are required to use the organizational view.

• A business plan or higher-level AWS support contract

  • Required to use the AWS Health API

How to Use

Download the latest binary suitable for your environment from the GitHub repository's releases page.

https://github.com/hayao-k/aws-health-exporter/releases

wget https://github.com/hayao-k/aws-health-exporter/releases/download/v0.8.1/aws-health-exporter_0.8.1_linux_amd64.tar.gz
tar xvf aws-health-exporter_0.8.1_linux_amd64.tar.gz

To use AWS Health Exporter, run the binary with the desired flags. Below are the available flags:

• --event-filter, --filter, -f: Filter events by service name, event status, and other criteria.

• --status-code, -c: Filter entity by status code. Possible values are IMPAIRED, UNIMPAIRED, UNKNOWN, PENDING and RESOLVED

• --echo, -e: Echo CSV content to standard output.

• --profile, -p: Specify the AWS credential profile to use.

• --account-id, -i: Specify a single account ID to process (optional).

• --output-file, --file-name, o: Specify the output CSV file name.

Details of the event filtering option

The --event-filter option allows you to specify complex filtering criteria. Below is a table of the available fields that can be included in the filter criteria:

For startTime, endTime, and lastUpdatedTime, you can specify a time range using from and to in ISO 8601 date format. Here is the structure for determining the time range:

• {from:YYYY-MM-DDTHH:MM:SSZ,to:YYYY-MM-DDTHH:MM:SSZ}

Example Commands

# Describe RDS events with open status and export to CSV
./health-exporter --event-filter service=RDS,status=open

# Describe upcoming LAMBDA events and echo the output to STDOUT
./health-exporter --event-filter service=LAMBDA,status=upcoming --echo

# Describe only events in the Tokyo region and specify their last updated time.
./health-exporter ----event-filter "lastUpdatedTime={from=2024-03-01T00:00:00Z,to=2024-05-02T23:59:59Z},region=ap-northeast-1"

# Get entities with pending status only and specify a custom file name
./health-exporter --status-code PENDING --output-file my_event_details.csv

# Get events using the specified profile
./health-exporter --profile my-profile

# Process only a single account
./health-exporter --account-id 123456789012

Execution Example

When you execute the command, an interactive prompt will be displayed. In the following example, the --event-filter flag extracts only the upcoming status events related to AWS Lambda.

$ health-exporter --event-filter service=LAMBDA,status=upcoming --status-code PENDING
Use the arrow keys to navigate: ↓ ↑ → ← 
? Select an event: 
  ▸ LAMBDA - AWS_LAMBDA_PLANNED_LIFECYCLE_EVENT (us-east-1, 2024-10-14 07:00:00)
    LAMBDA - AWS_LAMBDA_PLANNED_LIFECYCLE_EVENT (ap-northeast-1, 2024-10-14 07:00:00)
    LAMBDA - AWS_LAMBDA_PLANNED_LIFECYCLE_EVENT (ap-northeast-1, 2024-06-12 07:00:00)
    LAMBDA - AWS_LAMBDA_PLANNED_LIFECYCLE_EVENT (ap-southeast-2, 2024-10-14 07:00:00)
↓   LAMBDA - AWS_LAMBDA_PLANNED_LIFECYCLE_EVENT (us-east-1, 2024-06-12 07:00:00)

From the prompt, select the event you want to output. After selection, the tool will gather related account and entity information and output it to a CSV file.

✔ LAMBDA - AWS_LAMBDA_PLANNED_LIFECYCLE_EVENT (us-east-1, 2024-10-14 07:00:00)
Event details have been written to AWS_LAMBDA_PLANNED_LIFECYCLE_EVENT_2024-10-14_07-00-00_us-east-1_PENDING.csv.

The output CSV will contain information such as Account ID, Account Name, Region, Identifier, Status, and Last Updated. In this example, since --status-code PENDING was specified during command execution, only resources with PENDING status are output.

Account ID, Account Name, Region, Identifier, Status, Last Updated
000000000000,account-0000,us-east-1,arn:aws:lambda:us-east-1:000000000000:function:Old_Runtime_Lambda_Function-1PBKPZPFSJ058,PENDING,2024-04-21 20:11:29
111111111111,account-1111,us-east-1,arn:aws:lambda:us-east-1:111111111111:function:Old_Runtime_Lambda_Function-uuTi2u7DbooD,PENDING,2024-04-21 20:11:29
111111111111,account-1111,us-east-1,arn:aws:lambda:us-east-1:111111111111:function:Old_Runtime_Lambda_Function-omdieC8Umobo,PENDING,2024-04-21 20:11:29
222222222222,account-2222,us-east-1,arn:aws:lambda:us-east-1:222222222222:function:Old_Runtime_Lambda_Function-ULZ27BYSQ0MN,PENDING,2024-04-21 20:11:29
222222222222,account-2222,us-east-1,arn:aws:lambda:us-east-1:222222222222:function:Old_Runtime_Lambda_Function-10YNGBMU46VP9,PENDING,2024-04-21 20:11:29
222222222222,account-2222,us-east-1,arn:aws:lambda:us-east-1:222222222222:function:Old_Runtime_Lambda_Function-CEgHAu41udFy,PENDING,2024-04-21 20:11:29
333333333333,account-3333,us-east-1,arn:aws:lambda:us-east-1:333333333333:function:Old_Runtime_Lambda_Function-zNKRpLWP0pXB,PENDING,2024-04-21 20:11:29
333333333333,account-3333,us-east-1,arn:aws:lambda:us-east-1:333333333333:function:Old_Runtime_Lambda_Function-24ES8MRQJ9R6,PENDING,2024-04-21 20:11:29
444444444444,account-4444,us-east-1,arn:aws:lambda:us-east-1:444444444444:function:Old_Runtime_Lambda_Function-134QIS8IYF84K,PENDING,2024-04-21 20:11:29
444444444444,account-4444,us-east-1,arn:aws:lambda:us-east-1:444444444444:function:Old_Runtime_Lambda_Function-B97VeyrZNXIy,PENDING,2024-04-21 20:11:29

Mechanism

Primarily uses 3 AWS Health APIs.

• DescribeEventsForOrganization API

• DescribeAffectedAccountsForOrganization API

• DescribeAffectedEntitiesForOrganization API

DescribeEventsForOrganization API

Calls the DescribeEventsForOrganization API to retrieve relevant events based on the filter conditions specified on the command line. This API returns only an overview of the events, so information about affected accounts or resources is not included.

DescribeAffectedAccountsForOrganization API

This API retrieves a list of accounts within the organization affected by the selected event.

DescribeAffectedEntitiesForOrganization API

This API returns a list of entities affected by one or more events in one or more accounts within the organization.

When the user selects an event through the interactive prompt, information obtained from these APIs is formatted and output as a CSV file.

I hope this helps you.

I have created a sample chatbot application that uses Chainlit and LangChain to showcase Amazon Bedrock.

You can interact with the AI assistant while switching between multiple models.

This sample application has been tested in the following environments:

• AWS Region: us-east-1

• AWS Copilot CLI: v1.30.1

• Python: v3.11.5

• boto3: v1.28.57

• LangChain: v0.0.305

• Chainlit: v0.7.0

Note: To support the GA Version of the Bedrock API, make sure to use versions of LangChain and boto3 that are newer than those mentioned above.

Source Code

Check out my GitHub repository!

Start using Bedrock

Before you can start using Amazon Bedrock, you must request access to each model. To add access to a model, go to the Model Access section in the Bedrock console and click Edit.

Select the models you want to use and save the settings.

Note: Third-party models such as Jurassic-2 and Claude require access through the AWS Marketplace. This means you will also be charged a Marketplace fee for using the model.

It will take some time for each model to become available; models with an Access status of Access granted are ready for use.

Note: Some models, such as Titan Text G1 - Express, are in Preview and may not be immediately available.

How to Deploy

The GitHub repository for the sample application contains a manifest file for deploying the container to AWS App Runner using the AWS Copilot CLI.

Follow these steps to deploy:

1. Install the AWS Copilot CLI if necessary

    sudo curl -Lo /usr/local/bin/copilot https://github.com/aws/copilot-cli/releases/latest/download/copilot-linux && sudo chmod +x /usr/local/bin/copilot
   

2. Deploy to App Runner!

    export AWS_REGION=us-east-1
    copilot app init bedrockchat-app
    copilot deploy --name bedrockchat --env dev
   

If the deployment is successful, access the URL displayed in the message. If the following screen appears, the deployment has succeeded!

To delete an environment, execute the following command:

copilot app delete

How to use the app

The model, temperature, and maximum token size can be changed from the Settings panel.

The sample code allows you to choose between Claude v2, Jurassic-2 Ultra, and Amazon Titan Text G1 - Express. This enables you to compare performance while switching between models.

Note: As of 9/29/2023, Amazon Titan Text G1 - Express is in Preview status. It will be rolled out gradually, so it may not be available for some accounts.

You can enjoy a free-flowing conversation with the AI assistant! The conversation history during the session is retained, allowing it to reply to you while considering the context.

I hope this will be of help to someone else.

Here is a simple example of running Mountpoint for Amazon S3 from inside a container

• Created with information as of 3/21/2023 (version: 0.2.0-b8363a4)

• Mountpoint for Amazon S3 is currently in alpha release and should not be used in production workloads

Dockerfile

FROM rust:1.68.0 as Build

RUN apt-get update && apt-get install -y \
    clang\
    cmake \
    curl \
    fuse \
    git \
    libfuse-dev \
    pkg-config \
 && apt-get clean \
 && rm -rf /var/lib/apt/lists/* \
 && git clone --recurse-submodules https://github.com/awslabs/mountpoint-s3.git \
 && cd mountpoint-s3 \
 && cargo build --release


FROM debian:bullseye-slim
RUN apt-get update && apt-get install -y \
    ca-certificates \
    libfuse-dev \
    sudo \
 && apt-get clean \
 && rm -rf /var/lib/apt/lists/*

COPY --from=build /mountpoint-s3/target/release/mount-s3 /usr/local/bin/mount-s3

RUN chmod 777 /usr/local/bin/mount-s3

RUN useradd -ms /bin/bash mount-s3-user \
 && echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers \
 && adduser mount-s3-user sudo

USER mount-s3-user

Getting started

Image Build

docker image build -t mount-s3:latest .

Run

docker container run --privileged --rm -it mount-s3:latest bash

Enjoy

mount-s3-user@ce43831fda04:~$ sudo mount-s3 <bucket_name> /mnt --allow-other --region ap-northeast-1
mount-s3-user@ce43831fda04:~$ ls -l /mnt/test.json
-rw-r--r-- 1 root root 306424 Feb 21 02:42 /mnt/test.json

EC2 on Docker Consideration

If using EC2 IAM roles for AWS credentials, increasing the IMDSv2 hop limit from 1 to 2 in the instance metadata options is recommended.

In a container environment, if the hop limit is 1, the IMDSv2 response does not return because going to the container is considered an additional network hop. To avoid the process of falling back to IMDSv1 and the resultant delay, in a container environment we recommend that you set the hop limit to 2

aws ec2 modify-instance-metadata-options \
    --instance-id i-xxxxxxxxxxxxxxxxx \
    --http-put-response-hop-limit 2 \
    --http-endpoint enabled

I hope this will be of help to someone else.

GitHub Enterprise Cloud audit logs support log streaming to various cloud providers.

Streaming audit logs to Amazon S3 can be done via OpenID Connect. This requires the creation of an OIDC Provider and IAM role on the AWS side and an S3 bucket for log storage.

Please refer to the above document for detailed instructions.

It is not something that is created many times, but I have created a CloudFormation template and hope it will be helpful for those who want to create a new one quickly and easily.

Resources to be created

CloudFormation template

Github repository is here

AWSTemplateFormatVersion: "2010-09-09"
Description: Create AWS resources required for GitHub Enterprise Cloud audit log streaming

Parameters:
  RetainDays:
    Description: Number of days to keep audit log.
    Type: String
    Default: 365
  EnterpriseName:
    Description: Your GitHub Enterprise Name.
    Type: String
    Default: EXAPMLECORP
  AccessLogPrefix:
    Description: Server access log-prefix of audit log bucket.
    Type: String
    Default: logs/

Metadata: 
  AWS::CloudFormation::Interface: 
    ParameterGroups: 
      - Label: 
          default: GitHub Configuration
        Parameters: 
          - EnterpriseName
      - Label: 
          default: Audit Log Configuration
        Parameters: 
          - RetainDays
          - AccessLogPrefix

Resources:
  # Customer Managed Key for audit log encryption
  AuditLogEncryptionKey:
    Type: AWS::KMS::Key
    Properties: 
      Description: Customer Managed Key for audit log encryption
      Enabled: true
      EnableKeyRotation: True
      KeyPolicy:
        Version: '2012-10-17'
        Id: key-default-1
        Statement:
        - Sid: Allow administration of the key
          Effect: Allow
          Principal:
            AWS: !Sub arn:aws:iam::${AWS::AccountId}:root
          Action:
          - kms:Create*
          - kms:Describe*
          - kms:Enable*
          - kms:List*
          - kms:Put*
          - kms:Update*
          - kms:Revoke*
          - kms:Disable*
          - kms:Get*
          - kms:Delete*
          - kms:ScheduleKeyDeletion
          - kms:CancelKeyDeletion
          Resource: '*'
        - Sid: Allow local use of the key
          Effect: Allow
          Principal:
            AWS: !Sub arn:aws:iam::${AWS::AccountId}:root
          Action:
          - kms:Encrypt
          - kms:Decrypt
          - kms:ReEncrypt*
          - kms:GenerateDataKey*
          - kms:DescribeKey
          Resource: '*'

  # Customer Managed Key ALias
  AuditLogEncryptionKeyAlias:
    Type: AWS::KMS::Alias
    Properties: 
      AliasName: !Sub alias/${AWS::StackName}-key
      TargetKeyId: !Ref AuditLogEncryptionKey

  # S3 bucket for audit log
  AuditLogBucket:
    Type: AWS::S3::Bucket
    DeletionPolicy: Retain
    UpdateReplacePolicy: Retain
    Properties:
      BucketEncryption:
        ServerSideEncryptionConfiguration: 
          - ServerSideEncryptionByDefault: 
              SSEAlgorithm: aws:kms
              KMSMasterKeyID: !GetAtt AuditLogEncryptionKey.Arn
            BucketKeyEnabled: true    
      LifecycleConfiguration:
        Rules:
          - Id: Retain
            Status: Enabled
            ExpirationInDays: !Ref RetainDays
      PublicAccessBlockConfiguration: 
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true
      LoggingConfiguration:
        DestinationBucketName: !Ref AccessLogBucket
        LogFilePrefix: !Ref AccessLogPrefix
      VersioningConfiguration:
        Status: Enabled
      OwnershipControls:
        Rules:
          - ObjectOwnership: BucketOwnerEnforced

  # S3 bucket policy for audit log
  AudtiLogBucketPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      Bucket: !Ref AuditLogBucket
      PolicyDocument:
        Statement:
        - Effect: Deny
          Principal: '*'
          Action: 's3:*'
          Resource: 
            - !Sub ${AuditLogBucket.Arn}/*
            - !GetAtt AuditLogBucket.Arn
          Condition:
            Bool: 
              aws:SecureTransport: false

  # S3 bucket for server access log
  AccessLogBucket:
    Type: AWS::S3::Bucket
    DeletionPolicy: Retain
    UpdateReplacePolicy: Retain
    Properties:
      BucketEncryption:
        ServerSideEncryptionConfiguration: 
          - ServerSideEncryptionByDefault: 
              SSEAlgorithm: AES256
      LifecycleConfiguration:
        Rules:
          - Id: Retain
            Status: Enabled
            ExpirationInDays: !Ref RetainDays
      PublicAccessBlockConfiguration: 
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true
      VersioningConfiguration:
        Status: Enabled
      OwnershipControls:
        Rules:
          - ObjectOwnership: BucketOwnerEnforced

  # S3 bucket policy for server access log
  AccessLogBucketPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      Bucket: !Ref AccessLogBucket
      PolicyDocument:
        Statement:
          - Effect: Deny
            Principal: '*'
            Action: 's3:*'
            Resource: 
              - !Sub ${AccessLogBucket.Arn}/*
              - !GetAtt AccessLogBucket.Arn
            Condition:
              Bool: 
                aws:SecureTransport: false
          - Effect: Allow
            Principal:
              Service: logging.s3.amazonaws.com
            Action: s3:PutObject
            Resource:
              - !Sub ${AccessLogBucket.Arn}/${AccessLogPrefix}*
            Condition:
              ArnLike:
                aws:SourceArn: !GetAtt AuditLogBucket.Arn
              StringEquals: 
                aws:SourceAccount: !Ref AWS::AccountId

  # IAM policy for audit log streaming role
  AuditLogStremingPolicy:
    Type: AWS::IAM::ManagedPolicy
    Properties:
      Description: Policy for GitHub Audit Log Streaming
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action:
              - s3:PutObject
            Resource:
              - !Sub ${AuditLogBucket.Arn}/*

  # IAM Role for audit log streaming role
  AuditLogStreamingRole:
    Type: AWS::IAM::Role
    Properties:
      ManagedPolicyArns: 
        - !Ref AuditLogStremingPolicy
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Action: sts:AssumeRoleWithWebIdentity
            Principal:
              Federated: !Ref GithubAuditLogOIDCProvider
            Condition:
              StringEquals:
                oidc-configuration.audit-log.githubusercontent.com:sub: !Sub https://github.com/${EnterpriseName}
                oidc-configuration.audit-log.githubusercontent.com:aud: sts.amazonaws.com

  GithubAuditLogOIDCProvider:
    Type: AWS::IAM::OIDCProvider
    Properties:
      Url: https://oidc-configuration.audit-log.githubusercontent.com
      ClientIdList: 
        - sts.amazonaws.com
      ThumbprintList:
        - 7e6db7b7584d8cf2003e0931e6cfc41a3a62d3df

Outputs:
  IAMRole:
    Description: IAM role for audit log streaming role
    Value: !GetAtt AuditLogStreamingRole.Arn
  AuditLogBucket:
    Description: S3 bucket for audit log
    Value: !Ref AuditLogBucket

Deploy

3 parameters are required.

• GitHub Enterprise Name (EnterpriseName)

• Number of days to keep audit log (RetainDays)

• Server access log-prefix of audit log bucket (AccessLogPrefix)

After deploying the stack, set the output S3 bucket name and IAM Role ARN to GitHub Enterprise Cloud.

If the endpoint connection check is successful, save the configuration.

You can see the audit log in the AuditLogBucket!

I hope this will be of help to someone else.

Member accounts added to AWS Organizations after subscribing to Enterprise Support are not enrolled in Enterprise Support.

To register a new member account with Enterprise Support, you must open a support case in the management account.

Example

This example is a simple workflow that executes case creation, close confirmation, and notification with AWS Step Functions.

Input

You can use Amazon EventBrige to trigger the CreateAccount event in AWS Organizations or the CreateManagedAccount event in AWS Control Tower to launch the state machine.

Therefore, the input for the state machine is a single account as follows.

{
  "AccountId": "<Account ID>"
}

State machine definition

The process flow is as follows. Creating cases and checking status uses AWS SDK service integrations.

• In the CeateCase state, create a support case with the account ID received from the state machine startup input as the activation target

• Execute the DescribeCases state based on the Case ID returned as the result of the CreateCase state task.

  • DescribeCases API requires passing a list of case IDs, so use the built-in function States.Array.

  • Since the task result is also a list, specify $.Cases[0] in OutputPath.

• In the Choice state, check the status of support cases from the DescirbeCases output

  • If resolved, proceed to SNS Pulibsh state.

  • Otherwise, wait for a specified time in the Wait state and execute DescribeCases again.

• In the SNS Publish state, publish a message to the specified SNS Topic

{
  "Comment": "A description of your state machine",
  "StartAt": "CreateCase",
  "States": {
    "CreateCase": {
      "Type": "Task",
      "Parameters": {
        "Subject": "Enterprise Activation Request for Linked account",
        "ServiceCode": "customer-account",
        "SeverityCode": "low",
        "CategoryCode": "other-account-issues",
        "CommunicationBody.$": "States.Format('Please enable Enterprise support for following account ID:\n{}\n', $.AccountId)",
        "Language": "en",
        "IssueType": "customer-service"
      },
      "Resource": "arn:aws:states:::aws-sdk:support:createCase",
      "Next": "DescribeCases"
    },
    "DescribeCases": {
      "Type": "Task",
      "Parameters": {
        "CaseIdList.$": "States.Array($.CaseId)",
        "IncludeResolvedCases": true
      },
      "Resource": "arn:aws:states:::aws-sdk:support:describeCases",
      "Next": "Choice",
      "OutputPath": "$.Cases[0]"
    },
    "Choice": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.Status",
          "StringEquals": "resolved",
          "Next": "SNS Publish"
        }
      ],
      "Default": "Wait"
    },
    "SNS Publish": {
      "Type": "Task",
      "Resource": "arn:aws:states:::sns:publish",
      "Parameters": {
        "Message.$": "$",
        "TopicArn": "arn:aws:sns:us-east-1:123456789012:your-sns-topic"
      },
      "End": true
    },
    "Wait": {
      "Type": "Wait",
      "Seconds": 30,
      "Next": "DescribeCases"
    }
  }
}

On Workflow Studio, it is displayed as follows.

Execution example

I hope this will be of help to someone else.

• Backup EC2 instances with multiple ENIs attached with AWS Backup

• Restore EC2 instances with multiple ENIs attached, when restored from a recovery point

How?

• Run the StartRestoreJob API, e.g., from the AWS CLI or SDK

• Restore jobs launched from the console cannot customize the network i\nterface

EC2 instance backup with multiple ENIs attached

EC2 instances with multiple ENIs attached can also be backed up with AWS Backup. Backup data is stored as AMI, but AMI does not contain network interface information.

However, the metadata of the recovery point includes the network interface information. Recovery point metadata can be checked with the GetRecoveryPointRestoreMetadata API.

The following is an example of execution with the AWS CLI.

$ aws backup get-recovery-point-restore-metadata --backup-vault-name Default --recovery-point-arn arn:aws:ec2:us-west-2::image/ami-xxxxxxxxxxxxxxxxx
{
    "BackupVaultArn": "arn:aws:backup:us-west-2:123456789012:backup-vault:Default",
    "RecoveryPointArn": "arn:aws:ec2:us-west-2::image/ami-xxxxxxxxxxxxxxxxx",
    "RestoreMetadata": {
        "CapacityReservationSpecification": "{\"CapacityReservationPreference\":\"open\"}",
        "CpuOptions": "{\"CoreCount\":2,\"ThreadsPerCore\":1}",
        "CreditSpecification": "{\"CpuCredits\":\"unlimited\"}",
        "DisableApiTermination": "false",
        "EbsOptimized": "true",
        "HibernationOptions": "{\"Configured\":false}",
        "InstanceInitiatedShutdownBehavior": "stop",
        "InstanceType": "t4g.micro",
        "Monitoring": "{\"State\":\"disabled\"}",
        "NetworkInterfaces": "[{\"AssociatePublicIpAddress\":true,\"DeleteOnTermination\":true,\"Description\":\"\",\"DeviceIndex\":0,\"Groups\":[\"sg-xxxxxxxxxxxxxxxxx\"],\"Ipv6AddressCount\":0,\"Ipv6Addresses\":[],\"NetworkInterfaceId\":\"eni-aaaaaaaaaaaaaaaaa\",\"PrivateIpAddress\":\"172.31.62.169\",\"PrivateIpAddresses\":[{\"Primary\":true,\"PrivateIpAddress\":\"172.31.62.169\"}],\"SecondaryPrivateIpAddressCount\":0,\"SubnetId\":\"subnet-xxxxxxxxxxxxxxxxx\",\"InterfaceType\":\"interface\",\"Ipv4Prefixes\":[],\"Ipv6Prefixes\":[]},{\"AssociatePublicIpAddress\":true,\"DeleteOnTermination\":false,\"Description\":\"\",\"DeviceIndex\":1,\"Groups\":[\"sg-xxxxxxxxxxxxxxxxx\"],\"Ipv6AddressCount\":0,\"Ipv6Addresses\":[],\"NetworkInterfaceId\":\"eni-bbbbbbbbbbbbbbbbb\",\"PrivateIpAddress\":\"172.31.54.130\",\"PrivateIpAddresses\":[{\"Primary\":true,\"PrivateIpAddress\":\"172.31.54.130\"}],\"SecondaryPrivateIpAddressCount\":0,\"SubnetId\":\"subnet-xxxxxxxxxxxxxxxxx\",\"InterfaceType\":\"interface\",\"Ipv4Prefixes\":[],\"Ipv6Prefixes\":[]}]",
        "Placement": "{\"AvailabilityZone\":\"us-west-2d\",\"GroupName\":\"\",\"Tenancy\":\"default\"}",
        "RequireIMDSv2": "false",
        "SecurityGroupIds": "[\"sg-xxxxxxxxxxxxxxxxx\"]",
        "SubnetId": "subnet-xxxxxxxxxxxxxxxxx",
        "VpcId": "vpc-xxxxxxxxxxxxxxxxx",
        "aws:backup:request-id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
    }
}

In the above, you can see that eni-aaaaaaaaaaaaaaaaa and eni-bbbbbbbbbbbbbbbbb information is included.

How to restore from a recovery point

When launching a restore job in the AWS Backup console, it is not possible to restore an EC2 instance with multiple ENIs attached. This is because the console limits the customizable parameters to the following.

https://docs.aws.amazon.com/aws-backup/latest/devguide/restoring-ec2.html

The AWS Backup console allows you to restore Amazon EC2 recovery points with the following parameters and settings you can customize:

• Instance type

• Amazon VPC

• Subnet

• Security groups

• IAM role

• Shutdown behavior

• Stop–hibernate behavior

• Termination protection

• T2/T3 unlimited

• Placement group name

• EBS-optimized instance

• Tenancy

• RAM disk ID

• Kernel ID

• User data

• Deletion on termination

To restore an EC2 instance with other customized parameters, including the network interface, you must execute the StartRestoreJob API with metadata, e.g., from the AWS CLI or SDK.

Use the AWS Backup API, CLI, or SDK to restore Amazon EC2 recovery points Use StartRestoreJob. This option allows you to restore all 38 parameters, including the 22 parameters that are not customizable on the console.

The following is an example of execution with the AWS CLI.

$ aws backup start-restore-job \
  --recovery-point-arn "arn:aws:ec2:us-west-2::image/ami-xxxxxxxxxxxxxxxxx" \
  --iam-role-arn "arn:aws:iam::123456789012:role/service-role/AWSBackupDefaultServiceRole" \
  --metadata file://metadata.json  
{
    "RestoreJobId": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"
}

You can specify parameters in metadata.json as follows.

Example of specifying a private IP address

Please note that if a backup source instance exists, the private IP address must be changed to avoid duplicate addresses.

{
    "VpcId": "vpc-xxxxxxxxxxxxxxxxx",
    "Monitoring": "{\"State\":\"disabled\"}",
    "CapacityReservationSpecification": "{\"CapacityReservationPreference\":\"open\"}",
    "InstanceInitiatedShutdownBehavior": "stop",
    "DisableApiTermination": "false",
    "CreditSpecification": "{\"CpuCredits\":\"unlimited\"}",
    "HibernationOptions": "{\"Configured\":false}",
    "EbsOptimized": "true",
    "Placement": "{\"AvailabilityZone\":\"us-west-2d\",\"GroupName\":\"\",\"Tenancy\":\"default\"}",
    "aws:backup:request-id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    "InstanceType": "t4g.micro",
    "NetworkInterfaces": "[{\"DeleteOnTermination\":true,\"Description\":\"\",\"DeviceIndex\":0,\"Groups\":[\"sg-xxxxxxxxxxxxxxxxx\"],\"Ipv6AddressCount\":0,\"Ipv6Addresses\":[],\"PrivateIpAddresses\":[{\"Primary\":true,\"PrivateIpAddress\":\"172.31.62.169\"}],\"SubnetId\":\"subnet-xxxxxxxxxxxxxxxxx\",\"InterfaceType\":\"interface\"},{\"DeleteOnTermination\":false,\"Description\":\"\",\"DeviceIndex\":1,\"Groups\":[\"sg-xxxxxxxxxxxxxxxxx\"],\"Ipv6AddressCount\":0,\"Ipv6Addresses\":[],\"PrivateIpAddresses\":[{\"Primary\":true,\"PrivateIpAddress\":\"172.31.54.130\"}],\"SubnetId\":\"subnet-xxxxxxxxxxxxxxxxx\",\"InterfaceType\":\"interface\"}]"
}

Example of specifying ENI-ID

Please note that the ENI must be detached from the instance and in Available status if the backup source ENI is to be used.

{
    "VpcId": "vpc-xxxxxxxxxxxxxxxxx",
    "Monitoring": "{\"State\":\"disabled\"}",
    "CapacityReservationSpecification": "{\"CapacityReservationPreference\":\"open\"}",
    "InstanceInitiatedShutdownBehavior": "stop",
    "DisableApiTermination": "false",
    "CreditSpecification": "{\"CpuCredits\":\"unlimited\"}",
    "HibernationOptions": "{\"Configured\":false}",
    "EbsOptimized": "true",
    "Placement": "{\"AvailabilityZone\":\"us-west-2d\",\"GroupName\":\"\",\"Tenancy\":\"default\"}",
    "aws:backup:request-id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    "InstanceType": "t4g.micro",
    "NetworkInterfaces": "[{\"DeleteOnTermination\":true,\"Description\":\"\",\"DeviceIndex\":0,\"Groups\":[\"sg-xxxxxxxxxxxxxxxxx\"],\"Ipv6AddressCount\":0,\"Ipv6Addresses\":[],\"PrivateIpAddresses\":[{\"Primary\":true,\"PrivateIpAddress\":\"172.31.62.169\"}],\"SubnetId\":\"subnet-xxxxxxxxxxxxxxxxx\",\"InterfaceType\":\"interface\"},{\"DeleteOnTermination\":false,\"Description\":\"\",\"DeviceIndex\":1,\"Ipv6AddressCount\":0,\"Ipv6Addresses\":[],\"NetworkInterfaceId\":\"eni-bbbbbbbbbbbbbbbbb\"}]"
}

Reference

https://aws.amazon.com/premiumsupport/knowledge-center/aws-backup-ec2-restore-cli/

I hope this will be of help to someone else.

This extension can be used to retrieve parameters from the AWS Systems Manager Parameter Store and secrets from the AWS Secrets Manager.

What makes you happy?

Until now, parameters and secrets were obtained in the Lambda function process using the AWS SDK or other means.

With this extension, these values can be cached and reused during the lifecycle of a Lambda function. This reduces the latency and cost of retrieving parameters and secrets.

Basic usage

Please refer to each documents for details.

Set Layer of the extension to Lambda function

Lambda Extension is made available by configuring Lambda Layers. In Managed Console, AWS Parameters and Secrets Lambda Extension could be selected in the AWS layer.

mage.png

When configuring from the CLI or other means, specify the ARN of the published Layer. A list of ARNs for each region is provided in the documentation.

Write HTTP GET code in the function

Using this Extension eliminates processing in the AWS SDK, but the code to retrieve the value with an HTTP GET request is still required. See the second half of this post for the sample code.

Change IAM policy for execution role

The extension uses the credentials of the IAM role used to execute the Lambda function itself. Therefore, an appropriate IAM policy must be set up to retrieve parameters and secrets. For example, for the Parameter Store, ssm:GetParameter and kms:Decrypt (when using SecureString) are required.

(Optional) Set environment variables for functions

TTL for the cache, log level, etc., can be controlled by setting environment variables for the Lambda function.

Sample Code

This is an example of referencing Amazon Linux 2 AMI public parameters.

Notes are as follows.

• / in the parameter name must be encoded
• The extension's local HTTP server port starts at default 2773
  • It can be changed via the environment variable PARAMETERS_SECRETS_EXTENSION_HTTP_PORT
• Header 'X-Aws-Parameters-Secrets-Token' with AWS_SESSION_TOKEN environment variable must be added
  • If not specified, it will be 401 unauthorized.

const https = require('http');

exports.handler = function(event, context, callback) {

    const options = {
        hostname: 'localhost',
        port: 2773,
        path: '/systemsmanager/parameters/get/?name=%2Faws%2Fservice%2Fami-amazon-linux-latest%2Famzn-ami-hvm-x86_64-gp2',
        headers: {
            'X-Aws-Parameters-Secrets-Token': process.env.AWS_SESSION_TOKEN
        },
        method: 'GET'
    };

    const req = https.request(options, res => {
        res.on('data', d => {
            console.log("Response from cache: "+d);
            return d;
        });
    });

    req.on('error', error => {
        console.error(error);
    });

    req.end();
};

The log of the execution result looks like this You got the parameter values!

[AWS Parameters and Secrets Lambda Extension] 2022/10/19 06:51:08 PARAMETERS_SECRETS_EXTENSION_LOG_LEVEL is not present. Log level set to info.
[AWS Parameters and Secrets Lambda Extension] 2022/10/19 06:51:08 INFO Systems Manager Parameter Store and Secrets Manager Lambda Extension 1.0.94
[AWS Parameters and Secrets Lambda Extension] 2022/10/19 06:51:08 INFO Serving on port 2773
EXTENSION    Name: AWSParametersAndSecretsLambdaExtension    State: Ready    Events: [INVOKE,SHUTDOWN]
START RequestId: bb5bcc53-38cc-42d7-9dc5-xxxxxxxxxxxx Version: $LATEST
[AWS Parameters and Secrets Lambda Extension] 2022/10/19 06:51:08 INFO ready to serve traffic
2022-10-19T06:51:09.247Z    bb5bcc53-38cc-42d7-9dc5-xxxxxxxxxxxx    INFO    Response from cache: {"Parameter":{"ARN":"arn:aws:ssm:ap-northeast-1::parameter/aws/service/ami-amazon-linux-latest/amzn-ami-hvm-x86_64-gp2","DataType":"text","LastModifiedDate":"2022-10-04T17:56:51.889Z","Name":"/aws/service/ami-amazon-linux-latest/amzn-ami-hvm-x86_64-gp2","Selector":null,"SourceResult":null,"Type":"String","Value":"ami-0fb16641312307fa9","Version":49},"ResultMetadata":{}}
END RequestId: bb5bcc53-38cc-42d7-9dc5-xxxxxxxxxxxx
REPORT RequestId: bb5bcc53-38cc-42d7-9dc5-xxxxxxxxxxxx    Duration: 796.05 ms    Billed Duration: 797 ms    Memory Size: 128 MB    Max Memory Used: 76 MB    Init Duration: 324.74 ms

I hope this will be of help to someone else.

AWS Support case histories are retained for up to 12 months.

Q: How long is case history retained?
Case history information is available for 12 months after creation.

Therefore, if you want to store the answers as knowledge in the long term, you need a backup.

This post summarises how to convert support case answers into markdown format and post them to your Wiki via the API.

Architecture

• Detecting support case closures with EventBridge event rules
• Use AWS Support API in AWS Lambda to get case details
• Formatted into Markdown format and posted to your Wiki

Note: Using AWS Support API requires a subscription to one of the following support plans: Business, Enterprise On-Ramp, or Enterprise.

This post is about GROWI, the OSS wiki, but it can be applied to any software or service that can post via API.

https://growi.org/en/

Points for Implementation

EventBridge event rule

Set to detect events where event-name is ResolveCase.

{
  "source": ["aws.support"],
  "detail-type": ["Support Case Update"],
  "detail": {
    "event-name": ["ResolveCase"]
  }
}

See documentation for examples of events.

{
    "version": "0",
    "id": "1aa4458d-556f-732e-ddc1-4a5b2fbd14a5",
    "detail-type": "Support Case Update",
    "source": "aws.support",
    "account": "111122223333",
    "time": "2022-02-21T15:51:31Z",
    "region": "us-east-1",
    "resources": [],
    "detail": {
        "case-id": "case-111122223333-muen-2022-7118885805350839",
        "display-id": "1234563851",
        "communication-id": "",
        "event-name": "ResolveCase",
        "origin": ""
    }
}

Getting support case information

The event passed from EventBrige contains a Case ID, so use DescribeCases API to get the details. Communications is obtained separately, so includeCommunications should be False.

def describe_case(case_id):
    response = support.describe_cases(
        caseIdList=[
            case_id
        ],
        includeResolvedCases=True,
        includeCommunications=False
    )
    return response['cases'][0]

def lambda_handler(event, context):
    case_info = describe_case(event['detail']['case-id'])

Cases excluded from backup

Increased service quotas and Enterprise support activation would not require backup.

def lambda_handler(event, context):
    case_info = describe_case(event['detail']['case-id'])

    if case_info['serviceCode'] == "service-limit-increase" or \
       case_info['subject'] == "Enterprise Activation Request for Linked account.":
        return {
            'Result': 'Service limit increase or Enterprise support activation will not be posted.'
        }

Get communication history with support

Use DescribeCommunications API. The data is retrieved from the most recent communication, so the order is reordered and concatenated.

Certain symbols of three or more consecutive characters are displayed as headers or horizontal lines in Markdown notation. They are replaced to avoid involuntary conversions.

def describe_communications(case_id):
    body = ""
    paginator = support.get_paginator('describe_communications')
    page_iterator = paginator.paginate(caseId=case_id)

    for page in page_iterator:
        for communication in page['communications']:
            body = re.sub(
                r'-{3,}|={3,}|\*{3,}|_{3,}', "...", communication['body']
            ) + '\n\n---\n\n' + body

    communications = '## Communications\n' + body
    return communications

Post to GROWI

The API reference is below.

New pages are created using createPage (POST /pages).

{
  "body": "string",
  "path": "/",
  "grant": 1,
  "grantUserGroupId": "5ae5fccfc5577b0004dbd8ab",
  "pageTags": [
    {
      "_id": "5e2d6aede35da4004ef7e0b7",
      "name": "daily",
      "count": 3
    }
  ],
  "createFromPageTree": true
}

• Only body (the body of the article) and path (the path to the post) are required.
• The grant specifies the extent to which the page is public (1: public, 2: only people who know the link, etc.).
• In practice, you should also include the access_token (api_key).

def create_payload(account_id, case_info):
    token = os.environ['API_KEY']
    title = '# ' + case_info['subject'] + '\n'
    information = '## Case Information\n' + \
        '* Account ID: ' + account_id + '\n' + \
        '* Case ID: ' + case_info['displayId'] + '\n' +\
        '* Create Date ' + case_info['timeCreated'] + '\n' + \
        '* Severity: ' + case_info['severityCode'] + '\n' + \
        '* Service: ' + case_info['serviceCode'] + '\n' + \
        '* Category: ' + case_info['categoryCode'] + '\n'
    communications = describe_communications(case_info['caseId'])
    return {
        'access_token': token,
        'path': '/PathYouWantToPost/' + case_info['subject'],
        'body': title + information + communications,
        'grant': 1,
    }

def lambda_handler(event, context):
    case_info = describe_case(event['detail']['case-id'])
    payload = create_payload(event['account'], case_info)
    url = os.environ['API_URL']
    headers = {
        'Content-Type': 'application/json',
    }
    req = Request(url, json.dumps(payload).encode('utf-8'), headers)

Lambda function example

The function execution role must be granted AWS Support referencing rights.

from logging import getLogger, INFO
import json
import os
from urllib.request import Request, urlopen
from urllib.error import URLError, HTTPError
import re
from botocore.exceptions import ClientError
import boto3

logger = getLogger()
logger.setLevel(INFO)

support = boto3.client('support')

def describe_communications(case_id):
    body = ''
    try:
        paginator = support.get_paginator('describe_communications')
        page_iterator = paginator.paginate(caseId=case_id)
    except ClientError as err:
        logger.error(err.response['Error']['Message'])
        raise

    for page in page_iterator:
        for communication in page['communications']:
            body = re.sub(
                r'-{3,}|={3,}|\*{3,}|_{3,}', "...", communication['body']
            ) + '\n\n---\n\n' + body

    communications = '## Communications\n' + body
    return communications

def create_payload(account_id, case_info):
    token = os.environ['API_KEY']
    title = '# ' + case_info['subject'] + '\n'
    information = '## Case Information\n' + \
        '* Account ID: ' + account_id + '\n' + \
        '* Case ID: ' + case_info['displayId'] + '\n' +\
        '* Create Date ' + case_info['timeCreated'] + '\n' + \
        '* Severity: ' + case_info['severityCode'] + '\n' + \
        '* Service: ' + case_info['serviceCode'] + '\n' + \
        '* Category: ' + case_info['categoryCode'] + '\n'
    communications = describe_communications(case_info['caseId'])
    return {
        'access_token': token,
        'path': '/PathYouWantToPost/' + case_info['subject'],
        'body': title + information + communications,
        'grant': 1,
    }

def describe_case(case_id):
    try:
        response = support.describe_cases(
            caseIdList=[
                case_id
            ],
            includeResolvedCases=True,
            includeCommunications=False
        )
    except ClientError as err:
        logger.error(err.response['Error']['Message'])
        raise
    else:
        return response['cases'][0]

def lambda_handler(event, context):
    case_info = describe_case(event['detail']['case-id'])

    if case_info['serviceCode'] == "service-limit-increase" or \
       case_info['subject'] == "Enterprise Activation Request for Linked account.":
        return {
            'Result': 'Service limit increase or Enterprise support activation will not be posted.'
        }

    payload = create_payload(event['account'], case_info)
    url = os.environ['API_URL']
    headers = {
        'Content-Type': 'application/json',
    }
    req = Request(url, json.dumps(payload).encode('utf-8'), headers)

    try:
        response = urlopen(req)
        response.read()
    except HTTPError as e:
        return {
            'Result': f'''Request failed: {e.code}, {e.reason})'''
        }
    except URLError as e:
        return {
            'Result': f'''Request failed: {e.reason})'''
        }
    else:
        return {
            'Result' : 'Knowledge posted.'
        }

The results of the POST are as follows. Sorry, I can't share the case's contents, so it's mostly blacked out.

I hope this will be of help to someone else.

Describes the CloudFormation template modifications required to migrate CloudFront's Origin access identity (OAI) to Origin Access Control (OAC).

OAC is a new access control method for setting S3 buckets as origins in CloudFront.

Previously we had used Origin Access Identity (OAI) to restrict access to origin S3 buckets to CloudFront only.

OAI is currently treated as Legacy. Migration from OAI to OAC is recommended to support security best practices and new regions.

Sample Template

See my GitHub repository!

Examples of modifications

Migrating existing CloudFormation templates from OAI to OAC requires, as a minimum, the following modifications.

• Creating an OAC
• Modifying the bucket policy
• Modifying Distribution Origin

Creating an OAC

Create an AWS::CloudFront::OriginAccessControl. Description of the OriginAccessControlConfig is an optional field.

Resources:
  CloudFrontOriginAccessControl:
    Type: AWS::CloudFront::OriginAccessControl
    Properties: 
      OriginAccessControlConfig:
        Description: Default Origin Access Control
        Name: !Ref AWS::StackName
        OriginAccessControlOriginType: s3
        SigningBehavior: always
        SigningProtocol: sigv4

Delete AWS::CloudFront::CloudFrontOriginAccessIdentity as it will no longer be required after migration.

Modifying the bucket policy

In the following example, the policy for OAI has been removed, but it is a recommended migration procedure to include both OAI and OAC policies. This will prevent CloudFront from losing access to S3 buckets during migration to OAC. Please take action as necessary.

Before

 # S3 bucket policy to allow access from CloudFront OAI
  AssetsBucketPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      Bucket: !Ref AssetsBucket
      PolicyDocument:
        Statement:
        - Action: s3:GetObject
          Effect: Allow
          Resource: !Sub ${AssetsBucket.Arn}/*
          Principal:
            AWS: !Sub arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity ${CloudFrontOriginAccessIdentity}

After

# S3 bucket policy to allow access from CloudFront OAC
  AssetsBucketPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      Bucket: !Ref AssetsBucket
      PolicyDocument:
        Statement:
        - Action: s3:GetObject
          Effect: Allow
          Resource: !Sub ${AssetsBucket.Arn}/*
          Principal:
            Service: cloudfront.amazonaws.com
          Condition:
            StringEquals:
              AWS:SourceArn: !Sub arn:aws:cloudfront::${AWS::AccountId}:distribution/${AssetsDistribution}

Modifying Distribution Origin

OAI sets S3OriginConfig for Distribution Origin, but OAC requires OriginAccessControlId to be specified.

Before

  AssetsDistribution:
    Type: AWS::CloudFront::Distribution
    Properties:
      DistributionConfig:
        Origins:
        - Id: S3Origin
          DomainName: !GetAtt AssetsBucket.DomainName
          S3OriginConfig:
            OriginAccessIdentity: !Sub origin-access-identity/cloudfront/${CloudFrontOriginAccessIdentity}

After

  AssetsDistribution:
    Type: AWS::CloudFront::Distribution
    Properties:
      DistributionConfig:
        Origins:
        - Id: S3Origin
          DomainName: !GetAtt AssetsBucket.DomainName
          S3OriginConfig:
            OriginAccessIdentity: ''
          OriginAccessControlId: !GetAtt CloudFrontOriginAccessControl.Id

Note: As of September 2022, an empty OriginAccessIdentity must be specified in S3OriginConfig.

Deleting S3OriginConfig will result in the following error when updating the stack.

Resource handler returned message: "Invalid request provided: Exactly one of CustomOriginConfig and S3OriginConfig must be specified"

Examples of actual migration

Now let's update the existing CloudFormation stack using the modified template. Update the stack to ensure that the change set is as expected.

As described before, removing the bucket policy for OAI and migrating to OAC will result in Access Denied during distribution updates.

After updating the distribution, the S3 bucket access settings can be seen to have been updated to OAC.

The S3 bucket policy is also updated as expected.

Also, access from the browser seems to be okay.

I hope this will be of help to someone else.

Use Cost Explorer API to get the daily billing amount for each account and output the following data in CSV.

An equivalent CSV can be downloaded from the AWS Cost Explorer console. This post describes how to retrieve the data daily automatically.

Example API Request

This is a minimal example of using AWS Lambda to retrieve the daily billing amount for the current month.

import datetime
import boto3

def lambda_handler(event, context):
    today = datetime.date.today()
    start = today.replace(day=1).strftime('%Y-%m-%d')
    end = today.strftime('%Y-%m-%d')
    ce = boto3.client('ce')
    response = ce.get_cost_and_usage(
        TimePeriod={
            'Start': start,
            'End' :  end,
        },
        Granularity='DAILY',
        Metrics=[
            'NetUnblendedCost'
        ]
        GroupBy=[
            {
                'Type': 'DIMENSION',
                'Key': 'LINKED_ACCOUNT'
            }
        ]
    )
    return response['ResultsByTime']

One point to note is that there is a time lag before the cost for a given day is determined. For example, if the cost for April 25 is obtained on April 26, it may be less than the actual billing amount.

The timing of when the AWS data will be updated is not disclosed, but it appears that on April 27, the costs for April 25 will be almost finalized.

Processing with Pandas

Since the API response is a nested JSON, we will consider an example of processing it into a CSV using Pandas.

When using Pandas with AWS Lambda, Lambda Layers must be used.

• Since each element contains billing data daily, it is processed one day at a time with a for statement.
• After flattening the data using pandas.json_normalize, it is concatenated with the billing amount using pandas.concat.
• After further renaming the column to the billing date, the results are merged using the Account Id as the key.

    merged_cost = pandas.DataFrame(
        index=[],
        columns=['Account Id']
    )

    for index, item in enumerate(response):
        normalized_json = pandas.json_normalize(item['Groups'])
        split_keys = pandas.DataFrame(
            normalized_json['Keys'].tolist(),
            columns=['Account Id']
        )
        cost = pandas.concat(
            [split_keys, normalized_json['Metrics.NetUnblendedCost.Amount']],
            axis=1
        )
        renamed_cost = cost.rename(
            columns={'Metrics.NetUnblendedCost.Amount': item['TimePeriod']['Start']}
        )
        merged_cost = pandas.merge(merged_cost, renamed_cost, on='Account Id', how='right')

   print(merged_cost)

Account Id  ...     2022-04-25
0   000000000000  ...  15.4985752779
1   111111111111  ...         0.2176
2   222222222222  ...   6.5567854795
3   333333333333  ...   6.6300957379
4   444444444444  ...   8.2720868504
..           ...  ...            ...
19  777777777777  ...  10.0121863554
18  888888888888  ...   6.5976412116
20  999999999999  ...    6.493243618
[20 rows x 26 columns]

Example of Lambda function

After processing with for statement, the list of account names obtained from AWS Organizations API is merged and output in CSV.

from logging import getLogger, INFO
import os
import datetime
import boto3
import pandas
from botocore.exceptions import ClientError

logger = getLogger()
logger.setLevel(INFO)

def upload_s3(output, key, bucket):
    try:
        s3_resource = boto3.resource('s3')
        s3_bucket = s3_resource.Bucket(bucket)
        s3_bucket.upload_file(output, key, ExtraArgs={'ACL': 'bucket-owner-full-control'})
    except ClientError as err:
        logger.error(err.response['Error']['Message'])
        raise

def get_ou_ids(org, parent_id):
    ou_ids = []

    try:
        paginator = org.get_paginator('list_children')
        iterator = paginator.paginate(
            ParentId=parent_id,
            ChildType='ORGANIZATIONAL_UNIT'
        )
        for page in iterator:
            for ou in page['Children']:
                ou_ids.append(ou['Id'])
                ou_ids.extend(get_ou_ids(org, ou['Id']))
    except ClientError as err:
        logger.error(err.response['Error']['Message'])
        raise
    else:
        return ou_ids

def list_accounts():
    org = boto3.client('organizations')
    root_id = 'r-xxxx'
    ou_id_list = [root_id]
    ou_id_list.extend(get_ou_ids(org, root_id))
    accounts = []

    try:
        for ou_id in ou_id_list:
            paginator = org.get_paginator('list_accounts_for_parent')
            page_iterator = paginator.paginate(ParentId=ou_id)
            for page in page_iterator:
                for account in page['Accounts']:
                    item = [
                        account['Id'],
                        account['Name'],
                    ]
                    accounts.append(item)
    except ClientError as err:
        logger.error(err.response['Error']['Message'])
        raise
    else:
        return accounts

def get_cost_json(start, end):
    ce = boto3.client('ce')
    response = ce.get_cost_and_usage(
        TimePeriod={
            'Start': start,
            'End' :  end,
        },
        Granularity='DAILY',
        Metrics=[
            'NetUnblendedCost'
        ],
        GroupBy=[
            {
                'Type': 'DIMENSION',
                'Key': 'LINKED_ACCOUNT'
            }
        ]
    )
    return response['ResultsByTime']

def lambda_handler(event, context):
    today = datetime.date.today()
    start = today.replace(day=1).strftime('%Y-%m-%d')
    end = today.strftime('%Y-%m-%d')
    key = 'daily-cost-' + today.strftime('%Y-%m') + '.csv'
    output_file = '/tmp/output.csv'
    bucket = os.environ['BUCKET']
    account_list = pandas.DataFrame(list_accounts(), columns=['Account Id', 'Account Name'])
    daily_cost_list = get_cost_json(start, end)

    merged_cost = pandas.DataFrame(
        index=[],
        columns=['Account Id']
    )

    for index, item in enumerate(daily_cost_list):
        normalized_json = pandas.json_normalize(item['Groups'])
        split_keys = pandas.DataFrame(
            normalized_json['Keys'].tolist(),
            columns=['Account Id']
        )
        cost = pandas.concat(
            [split_keys, normalized_json['Metrics.NetUnblendedCost.Amount']],
            axis=1
        )
        renamed_cost = cost.rename(
            columns={'Metrics.NetUnblendedCost.Amount': item['TimePeriod']['Start']}
        )
        merged_cost = pandas.merge(merged_cost, renamed_cost, on='Account Id', how='outer')

    daily_cost = pandas.merge(account_list, merged_cost, on='Account Id', how='right')
    daily_cost.to_csv(output_file, index=False)
    upload_s3(output_file, key, bucket)

Now all that is left is setting an arbitrary startup schedule in EventBridge and a Lambda function as the target.

• Run actions-runner-controller on Private EKS cluster
• Communication with GitHub API is through on-premises proxy server

This post is validated with the following configuration.

• Amazon EKS: Kubernetes 1.22
• actions-runner-controller: v0.23.0
• cert-manager v1.8.0
• eksctl: 0.93.0
• kubectl: v1.21.2
• AWS CLI: 2.5.7

What is actions-runner-controller?

Controller for operating GitHub Actions self-hosted runners on Kubernetes cluster.

https://github.com/actions-runner-controller/actions-runner-controller

The following describes the flow of building actions-runner-controller on private EKS cluster.

Create private EKS Cluster

Set up kubectl, eksctl, and AWS CLI in your working environment in advance. See below for detailed instructions.

kubectl: https://kubernetes.io/docs/tasks/tools/

eksctl: https://docs.aws.amazon.com/eks/latest/userguide/eksctl.html

AWS CLI: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html

Please refer to the following document to create a private cluster with eksctl.

https://eksctl.io/usage/eks-private-cluster/

In this case, created an EKS cluster using the following config file.

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: actions-runner
  region: ap-northeast-1
  version: "1.22"

privateCluster:
  enabled: true
  skipEndpointCreation: true

vpc:
  subnets:
    private:
      ap-northeast-1a:
        id: subnet-xxxxxxxxxxxxxxxxx
      ap-northeast-1c:
        id: subnet-yyyyyyyyyyyyyyyyy
      ap-northeast-1d:
        id: subnet-zzzzzzzzzzzzzzzzz

For a private EKS cluster, the following VPC endpoints are required.

• ECR (ecr.api, ecr.dkr)
• S3 (Gateway)
• EC2
• STS
• CloudWatch Logs

If you enable a private cluster with eksctl (privateCluster.enabled), these endpoints will be created automatically.

In this case, I had a requirement to use a VPC already connected via Direct Connect, so skip endpoint creation with privateCluster.skipEndpointCreation to use the existing VPC and endpoints.

NOTE: if you specify an existing VPC, eksctl will edit the route table.

However, if the subnet is associated with the main route table, eksctl will not edit the route table and will fail to create the cluster. Therefore, it is necessary to create and associate a route table explicitly.

eksctl communicates with the AWS API via a proxy server. However, communication to private EKS cluster endpoints does not use a proxy server, so the following environment variable should be set.

export https_proxy=http://xxx.xxx.xxx.xxx:yyyy
export no_proxy=.eks.amazonaws.com

The following command creates a cluster. Creating a private cluster takes longer than creating a regular cluster. This configuration took about 22 minutes, but if you are creating VPC or VPC endpoints, it will take even longer, so adjust the timeout value.

$ eksctl create cluster --timeout 30m -f private-cluster.yaml

Register Managed Node Groups

Add UserData to the launch template used by the managed node group and configure the Docker daemon to use a proxy.

This template is based on the following blog.

https://yomon.hatenablog.com/entry/2020/10/eks_docker_inside_proxy#%E3%83%9E%E3%83%8D%E3%83%BC%E3%82%B8%E3%83%89%E5%9E%8B%E3%83%8E%E3%83%BC%E3%83%89%E3%82%B0%E3%83%AB%E3%83%BC%E3%83%97%E3%81%ABProxy%E8%A8%AD%E5%AE%9A

AWSTemplateFormatVersion: "2010-09-09"
Parameters:
  ClusterIp:
    Type: String
  ProxyIP:
    Type: String
  ProxyPort:
    Type: Number
  SshKeyPairName:
    Type: String

Resources:
  EKSManagedNodeInPrivateNetworkLaunchTemplate:
    Type: AWS::EC2::LaunchTemplate
    Properties: 
      LaunchTemplateName: eks-managednodes-in-private-network
      LaunchTemplateData: 
        InstanceType: t3.small
        KeyName: !Ref SshKeyPairName
        TagSpecifications: 
          - ResourceType: instance
            Tags:
              - Key: Name 
                Value: actions-runner-nodegroup
          - ResourceType: volume
            Tags:
              - Key: Name
                Value: actions-runner-nodegroup

        UserData: !Base64 
          "Fn::Sub": |
            Content-Type: multipart/mixed; boundary="==BOUNDARY=="
            MIME-Version:  1.0

            --==BOUNDARY==
            Content-Type: text/cloud-boothook; charset="us-ascii"

            #Set the proxy hostname and port
            PROXY=${ProxyIP}:${ProxyPort}
            MAC=$(curl -s http://169.254.169.254/latest/meta-data/mac/)
            VPC_CIDR=$(curl -s http://169.254.169.254/latest/meta-data/network/interfaces/macs/$MAC/vpc-ipv4-cidr-blocks | xargs | tr ' ' ',')

            #Create the docker systemd directory
            mkdir -p /etc/systemd/system/docker.service.d

            #Configure docker with the proxy
            cloud-init-per instance docker_proxy_config tee <<EOF /etc/systemd/system/docker.service.d/http-proxy.conf >/dev/null
            [Service]
            Environment="HTTP_PROXY=http://$PROXY"
            Environment="HTTPS_PROXY=http://$PROXY"
            Environment="NO_PROXY=${ClusterIp},$VPC_CIDR,localhost,127.0.0.1,169.254.169.254,.internal,s3.amazonaws.com,.s3.ap-northeast-1.amazonaws.com,api.ecr.ap-northeast-1.amazonaws.com,dkr.ecr.ap-northeast-1.amazonaws.com,ec2.ap-northeast-1.amazonaws.com,ap-northeast-1.eks.amazonaws.com"
            EOF

            #Reload the daemon and restart docker to reflect proxy configuration at launch of instance
            cloud-init-per instance reload_daemon systemctl daemon-reload 
            cloud-init-per instance enable_docker systemctl enable --now --no-block docker
            --==BOUNDARY==

After creating the launch template, create a Managed Node Group with the following config file.

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: actions-runner
  region: ap-northeast-1

managedNodeGroups:
- name: t3s-proxy
  desiredCapacity: 1
  privateNetworking: true
  launchTemplate:
    id: lt-xxxxxxxxxxxxxxxxx
    version: "1"

Run create nodegroup.

$ eksctl create nodegroup -f nodegroup.yaml

Installing actions-runner-controller

actions-runner-controller uses cert-manager to manage certificates for Admission Webhooks, so it must be installed in advance.

$ kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.yaml

To use actions-runner-controller in a proxy environment, it is necessary to configure the proxy server information in the following three locations.

1. manager container of controller-manager
2. runner container of RunnerDeployment
3. sidecar (dind) container of RunnerDeployment

I deployed actions-runner-controller with kubectl this time.

1 sets the https_proxy environment variable to the Deployment definition of controller-manager. 2 and 3 set the https_proxy environment variable to the definition of RunnerDeployment.

Edit the manifest file of actions-runner-controller as follows.

33836     spec:
33837       containers:
33838       - args:
33839         - --metrics-addr=127.0.0.1:8080
33840         - --enable-leader-election
33841         command:
33842         - /manager
31843         env:
31844         - name: GITHUB_TOKEN
31845           valueFrom:
31846             secretKeyRef:
31847               key: github_token
31848               name: controller-manager
31849               optional: true
31850         - name: GITHUB_APP_ID
31851           valueFrom:
31852             secretKeyRef:
31853               key: github_app_id
31854               name: controller-manager
31855               optional: true
31856         - name: GITHUB_APP_INSTALLATION_ID
31857           valueFrom:
31858             secretKeyRef:
31859               key: github_app_installation_id
31860               name: controller-manager
31861               optional: true
31862         - name: GITHUB_APP_PRIVATE_KEY
31863           value: /etc/actions-runner-controller/github_app_private_key
+ 31864       - name: http_proxy
+ 31865         value: "http://xxx.xxx.xxx.xxx:xxxx"
+ 31866       - name: https_proxy
+ 31867         value: "http://xxx.xxx.xxx.xxx:xxxx"
+ 31868       - name: no_proxy
+ 31869         value: "172.20.0.1,*.eks.amazonaws.com"
31870         image: summerwind/actions-runner-controller:v0.22.3
31871         name: manager

Run kubectl create.

$ kubectl create -f actions-runners-controller.yaml

NOTE: Due to a known issue with Kubernetes, we must use create (or replace when updating) instead of apply.

https://github.com/actions-runner-controller/actions-runner-controller/issues/1317

GitHub API can be authenticated using the GitHub App or Personal Access Token (PAT).

In this case, we use PAT authentication. Set the appropriate scope for the token depending on the type of runner and create a secret.

kubectl create secret generic controller-manager \
    -n actions-runner-system \
    --from-literal=github_token=${GITHUB_TOKEN}

NOTE: PAT must be pre-authorized before accessing an Organization with SAML SSO configured.

https://docs.github.com/ja/enterprise-cloud@latest/authentication/authenticating-with-saml-single-sign-on/authorizing-a-personal-access-token-for-use-with-saml-single-sign-on

RunnerDeployment is defined below.

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: runner-deploy
spec:
  template:
    spec:
      enterprise: <enterprise-name>
      labels:
        - runner-container
      env:
        - name: RUNNER_FEATURE_FLAG_EPHEMERAL
          value: "true"
        - name: http_proxy
          value: "http://xxx.xxx.xxx.xxx:yyyy"
        - name: https_proxy
          value: "http://xxx.xxx.xxx.xxx:yyyy"
        - name: no_proxy
          value: "172.20.0.1,10.1.2.0/24,*.eks.amazonaws.com"
      dockerEnv:
        - name: http_proxy
          value: "http://xxx.xxx.xxx.xxx:yyyy"
        - name: https_proxy
          value: "http://xxx.xxx.xxx.xxx:yyyy"
        - name: no_proxy
          value: "172.20.0.1,10.1.2.0/24,*.eks.amazonaws.com"
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: runner-deployment-autoscaler
spec:
  scaleDownDelaySecondsAfterScaleOut: 300
  scaleTargetRef:
    name: runner-deploy
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - type: PercentageRunnersBusy
    scaleUpThreshold: '0.75'
    scaleDownThreshold: '0.25'
    scaleUpFactor: '2'
    scaleDownFactor: '0.5'

When a container is used in a GitHub Actions job, the docker container for dind, launched as a RunnerDeployment sidecar, is used. Specify dockerEnv so that this dind container can perform image pulls.

After deployment, confirm that the pod is successfully started and is visible from GitHub.

$ kubectl apply -f RunnerDeployment.yaml

$ kubectl get pod
NAME                            READY   STATUS    RESTARTS   AGE
pod/runner-deploy-xxxxx-xxxxx   2/2     Running   0          53s

If controller-manager does not start properly, check the settings in 1 or GitHub; if Runner does not start properly, check the settings in 2 or 3.

With the update on 2/2/2021, you can now access Amazon S3 via AWS PrivateLink.

🚀 AWS PrivateLink for Amazon S3 is Now Generally Available

The following is a description of the advantages and points to note compared to the existing Gateway-type VPC endpoint.

Good points

Access from on-premises

I think this is what we have been waiting for.

In the case of the gateway endpoint, the S3 endpoint remains a global IP address. The route table configuration ensures that communication to S3 in the VPC is directed to the VPC endpoint.

Due to this specification, it was not possible to directly communicate with S3 from an on-premises environment connected using AWS Direct Connect or VPN.

How did we handle it so far?
It was necessary to build a proxy server on EC2. Naturally, there are operating and maintenance costs associated with this proxy server.

For Interface type endpoints with AWS PrivateLink, an ENI is created in the VPC. The private IP assigned to the ENI can be used to make a private connection directly from the on-premises environment to S3!

Access from another VPC or region

The concept is the same as on-premise.
Since it can be accessed via the private IP of the VPC, it can also be accessed by VPCs in other regions connected via VPC Peering.

It does not mean that a single endpoint can access S3 buckets in multiple regions. As with the Gateway VPC endpoint, the S3 buckets to be accessed must reside in the same region. VPCs and VPC endpoints in the Tokyo region can only access S3 buckets that reside in the Tokyo region.

Use Gateway and Interface endpoints together

S3 Gateway endpoint and Interface endpoint can use together. For example, an application on a VPC can continue to use the Gateway endpoint, while an application on-premises can use the Interface endpoint.

As discussed in more detail below, this configuration has advantages because Interface endpoints must take into account data transfer charges and changes to the endpoint URL.

Points to consider

Usage fee

Gateway VPC endpoints are free of charge, but Interface VPC endpoints are charged per endpoint (USD/hour) and per GB of processed data .

Pricing in Tokyo Region

If you have a redundant configuration with 3 AZs, you will have to pay for 3 endpoints.
Pricing per GB data processed is also not very expensive, but if you are exchanging large amounts of data, you may need to consider the price beforehand.

Specifying endpoints

The S3 Interface endpoint does not support the private DNS feature. The checkbox is grayed out when creating the endpoint.

The Private DNS feature resolves the default DNS name of the service (e.g. ec2.ap-northeast-1.amazonaws.com) to the private IP address assigned to the VPC endpoint.

This allows VPCs using Route 53 Resolver (formerly Amazon Provided DNS) and on-premises environments using Route 53 Resolver for Hybrid Clouds to use the default DNS hostname to send requests to VPC endpoints.

Since this feature is not available, you will need to access S3 using your VPC endpoint's unique DNS name.

AWS CLI Example

$ aws s3 ls s3://my-bucket/ --endpoint-url https://bucket.vpce-1a2b3c4d-5e6f.s3.ap-northeast-1.vpce.amazonaws.com

AWS SDK (boto3) example

import boto3

s3_client = boto3.client(
    's3',
    endpoint_url = 'https://bucket.vpce-1a2b3c4d-5e6f.s3.ap-northeast-1.vpce.amazonaws.com'
)

You need to change the subdomain depending on the API you want to operate. Specify ”bucket" for the subdomain if you want to perform API operations related to the S3 bucket. It's a "bucket", not a bucket name.

For example, if you directly specify the private IP of the endpoint, SSL validation failed will occur.

$ aws s3 ls --endpoint-url https://10.0.0.9
SSL validation failed for https://10.0.0.9/ ("hostname 'xxx.xxx.xxx.xxx' doesn't match either of 's3.ap-northeast-1.amazonaws.com', 
'*.accesspoint.vpce-1a2b3c4d-5e6f-ap-northeast-1d.s3.ap-northeast-1.vpce.amazonaws.com', 
'*.control.vpce-1a2b3c4d-5e6f-ap-northeast-1a.s3.ap-northeast-1.vpce.amazonaws.com', 
'*.accesspoint.vpce-1a2b3c4d-5e6f.s3.ap-northeast-1.vpce.amazonaws.com', 'bucket.vpce-1a2b3c4d-5e6f-ap-northeast-1a.s3.ap-northeast-1.vpce.amazonaws.com', 
'*.s3-control.ap-northeast-1.amazonaws.com', '*.control.vpce-1a2b3c4d-5e6f-ap-northeast-1d.s3.ap-northeast-1.vpce.amazonaws.com', 
'*.s3.ap-northeast-1.amazonaws.com', '*.s3-accesspoint.ap-northeast-1.amazonaws.com', '*.control.vpce-1a2b3c4d-5e6f.s3.ap-northeast-1.vpce.amazonaws.com', '*.bucket.vpce-1a2b3c4d-5e6f-ap-northeast-1d.s3.ap-northeast-1.vpce.amazonaws.com', 
'bucket.vpce-1a2b3c4d-5e6f-ap-northeast-1c.s3.ap-northeast-1.vpce.amazonaws.com', 'bucket.vpce-1a2b3c4d-5e6f.s3.ap-northeast-1.vpce.amazonaws.com',
'*.control.vpce-1a2b3c4d-5e6f-ap-northeast-1c.s3.ap-northeast-1.vpce.amazonaws.com', '*.bucket.vpce-1a2b3c4d-5e6f.s3.ap-northeast-1.vpce.amazonaws.com', '*.accesspoint.vpce-1a2b3c4d-5e6f-ap-northeast-1c.s3.ap-northeast-1.vpce.amazonaws.com', 
'*.bucket.vpce-1a2b3c4d-5e6f-ap-northeast-1c.s3.ap-northeast-1.vpce.amazonaws.com', 
'bucket.vpce-1a2b3c4d-5e6f-ap-northeast-1d.s3.ap-northeast-1.vpce.amazonaws.com', 
'*.accesspoint.vpce-1a2b3c4d-5e6f-ap-northeast-1a.s3.ap-northeast-1.vpce.amazonaws.com', 
'*.bucket.vpce-1a2b3c4d-5e6f-ap-northeast-1a.s3.ap-northeast-1.vpce.amazonaws.com'",)

Experiment

What if my application doesn't support endpoint URL changes?

I tried setting the private IP address of the interface VPC endpoint locally to the hosts file to force it to override the default DNS name locally. Then, I was able to perform the bucket-level operations. However, this kind of usage is not documented and will most likely not be supported by AWS.

10.0.0.9 s3.ap-northeast-1.amazonaws.com
10.0.0.9 <bucket-name>.s3.ap-northeast-1.amazonaws.com

$ aws s3 ls
$ aws s3 ls s3://my-bucket/

Please note that there are many disadvantages to using a hosts file, so please use it only as a reference.

e.g.

• No redundant configuration
• Need to add a DNS name for each bucket.

Official Document

Amazon S3 and interface VPC endpoints (AWS PrivateLink) https://docs.aws.amazon.com/AmazonS3/latest/userguide/privatelink-interface-endpoints.html

That's all. Happy storing!

kube-bench has been added to the 3rd party integrations in AWS Security Hub.

🚀 AWS Security Hub adds open source tool integrations with Kube-bench and Cloud Custodian

https://aws.amazon.com/jp/about-aws/whats-new/2020/12/aws-security-hub-adds-open-source-tool-integration-with-kube-bench-and-cloud-custodian/

With this integration, the results of the CIS Kubernetes Benchmark and CIS Amazon EKS Benchmark checks run by kube-bench can now be centrally managed in AWS Security Hub.

What is CIS Benchmark?

CIS Benchmark is a set of guidelines published by the Center for Internet Security (CIS), a non-profit organization in the United States, for strengthening various operating systems, servers, and cloud environments.

CIS Benchmark is referenced in compliance requirements such as the PCI DSS when it states "industry-accepted system hardening standards".

You can download over 140 CIS Benchmarks in PDF format from the CIS website. https://www.cisecurity.org/cis-benchmarks/

What is kube-bench?

kube-bench is a Go application that allows you to check if your environment complies with the recommendations listed in the CIS Kubernetes Benchmark.

kube-bench supports checking not only the CIS Kubernetes Benchmark, but also the CIS Amazon Elastic Kubernetes Service (EKS) Benchmark and CIS Google Kubernetes Engine (GKE) Benchmark.

What is AWS Security Hub?

AWS Security Hub is a service for aggregating and centrally managing various security data for the entire AWS environment.

Security Hub integrates with many 3rd party security products, as well as AWS services such as Amazon GuardDuty, Inspector, and Macie.
Data can be sent from Security Hub-enabled products to the Security Hub and vice versa.

Internally, the result information is managed in a JSON type format called AWS Security Finding Format (ASFF), so you can import your own data as long as it conforms to this format.

Prerequisites

I have confirmed that the following versions work.

• Amazon EKS: 1.17
• eksctl: 0.30.0
• kube-bench: 0.40.0

Ran the CIS Amazon EKS Benchmark v1.0 check in an EKS cluster.

Enable integration

Search for kube-bench from the Security Hub console integration and click "Accept findings" to see information about the IAM policies required to send the findings to Security Hub.

Select "Accept findings" again on the confirmation screen, and the status will be activated.

Configue IAM Roles

In order for kube-bench to run in an EKS cluster, the pod must have permissions to send check results to Security Hub.

There are two ways to assign access to AWS resources to a Pod.

• Use IAM Roles for Service Accounts (IRSA)
• Use IAM roles configured for node groups

Using IRSA, you can associate an IAM role with a Kubernetes service account.
This allows you to provide Security Hub access only to pods launched by kube-bench.

Note that if you use an IAM role that is configured for a node group, all Pods running under that group will be assigned access to the Security Hub.

This time we will use IRSA. If you haven't created an IAM OIDC Provider to use IRSA, do so first!

$ eksctl utils associate-iam-oidc-provider \
--cluster {CLUSTER_NAME} --approve --region ap-northeast-1

Next, create an IAM role with the following policies attached.
Replace the regions as needed.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "securityhub:BatchImportFindings",
            "Resource": [
                "arn:aws:securityhub:ap-northeast-1::product/aqua-security/kube-bench"
            ]
        }
    ]
}

Add the following policy to the trust relationship of the IAM role. Set <ACCOUNT_ID> and <OIDR_PROVIDER_ID> to your own.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::<ACCOUNT_ID>:oidc-provider/oidc.eks.ap-northeast-1.amazonaws.com/id/<OIDC_PROVIDER_ID>"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringLike": {
          "oidc.eks.ap-northeast-1.amazonaws.com/id/<OIDC_PROVIDER_ID>:sub": "system:serviceaccount:kube-bench:*",
          "oidc.eks.ap-northeast-1.amazonaws.com/id/<OIDC_PROVIDER_ID>:aud": "sts.amazonaws.com"
        }
      }
    }
  ]
}

This is written assuming that kube-bech is used for Namespace.

Create container image

We need to build a container image of kube-bench and push it to ECR.
First, let's create an ECR repository to store the image.

$ aws ecr create-repository --repository-name k8s/kube-bench --image-tag-mutability MUTABLE
{
    "repository": {
        "repositoryArn": "arn:aws:ecr:ap-northeast-1:123456789012:repository/k8s/kube-bench",
        "registryId": "123456789012",
        "repositoryName": "k8s/kube-bench",
        "repositoryUri": "123456789012.dkr.ecr.ap-northeast-1.amazonaws.com/k8s/kube-bench",
        "createdAt": 1607747704.0,
        "imageTagMutability": "MUTABLE",
        "imageScanningConfiguration": {
            "scanOnPush": false
        },
        "encryptionConfiguration": {
            "encryptionType": "AES256"
        }
    }
}

Clone the source code of kube-bench from GitHub.

$ git clone https://github.com/aquasecurity/kube-bench.git
Cloning into 'kube-bench'...
remote: Enumerating objects: 4115, done.
remote: Total 4115 (delta 0), reused 0 (delta 0), pack-reused 4115
Receiving objects: 100% (4115/4115), 7.69 MiB | 5.28 MiB/s, done.
Resolving deltas: 100% (2644/2644), done.

$ cd kube-bench

If you want to send the execution result to Security Hub, you need to edit cfg/eks-1.0/config.yaml and rewrite the AWS account, region and cluster name to be executed before building the image.

---
AWS_ACCOUNT: "123456789012"
AWS_REGION: "ap-northeast-1"
CLUSTER_ARN: "arn:aws:eks:ap-northeast-1:123456789012:cluster/{YOUR_CLUSTER_NAME}"

Build the container image and push it to ECR.

$ aws ecr get-login-password | docker login --username AWS --password-stdin https://123456789012.dkr.ecr.ap-northeast-1.amazonaws.com
Login Succeeded

$ docker build -t k8s/kube-bench .
...
Successfully built 6ad073f96455
Successfully tagged k8s/kube-bench:latest

$ docker tag k8s/kube-bench:latest 123456789012.dkr.ecr.ap-northeast-1.amazonaws.com/k8s/kube-bench:latest
$ docker push 123456789012.dkr.ecr.ap-northeast-1.amazonaws.com/k8s/kube-bench:latest
The push refers to repository [123456789012.dkr.ecr.ap-northeast-1.amazonaws.com/k8s/kube-bench]
bd6c279efeaa: Pushed 
3032a8c3bf7a: Pushed 
b0efa7564210: Pushed 
fe4cf80d4f2c: Pushed 
31c3d3db74eb: Pushed 
d2e36eff2b5d: Pushed 
f4666769fca7: Pushed 
latest: digest: sha256:da95de0edccad7adb6fd6c80137a65b5458142efe14efa94ac44cc5c6ce6b2ef size: 1782

Run kube-bench

Creating a Service Account

Create a service account for kube-bench using a manifest file like the following. Note that the annotations specifies the IAM role that we just created.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-bench-sa
  namespace: kube-bench
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/<IAM_ROLE_NAME>

Create a namespace kube-bench and apply it.

$ kubectl create ns kube-bench
kubectl create ns kube-bench

$ kubectl apply -f sa.yaml
serviceaccount/kube-bench-sa created

$ kubectl get sa -n kube-bench
NAME            SECRETS   AGE
default         1         6m32s
kube-bench-sa   1         11s

Run a job

Open jobs-eks.yaml cloned from GitHub, and edit image and command. Adding the --asff flag to the command will send the results to Security Hub.

Add the service account name that you just created.
The edited file should look like the following.

---
apiVersion: batch/v1
kind: Job
metadata:
  name: kube-bench
spec:
  template:
    spec:
      hostPID: true
      serviceAccountName: kube-bench-sa
      containers:
        - name: kube-bench
          image: 123456789012.dkr.ecr.region.amazonaws.com/k8s/kube-bench:latest
          command: ["kube-bench", "node", "--benchmark", "eks-1.0", "--asff"]
          volumeMounts:
            - name: var-lib-kubelet
              mountPath: /var/lib/kubelet
              readOnly: true
            - name: etc-systemd
              mountPath: /etc/systemd
              readOnly: true
            - name: etc-kubernetes
              mountPath: /etc/kubernetes
              readOnly: true
      restartPolicy: Never
      volumes:
        - name: var-lib-kubelet
          hostPath:
            path: "/var/lib/kubelet"
        - name: etc-systemd
          hostPath:
            path: "/etc/systemd"
        - name: etc-kubernetes
          hostPath:
            path: "/etc/kubernetes"

Run kube-bench as a Kubernetes job and make sure it exits successfully.

$ kubectl apply -f job-eks.yaml -n kube-bench
job.batch/kube-bench created

$ kubectl get all -n kube-bench                                                                                                                                                    
NAME                   READY   STATUS      RESTARTS   AGE
pod/kube-bench-q4sbd   0/1     Completed   0          55s

NAME                   COMPLETIONS   DURATION   AGE
job.batch/kube-bench   1/1           5s         55s

If the following message is output to the Pod's log and the job fails, please check whether the contents of cfg/eks-1.0/config.yaml are correct.

failed to output to ASFF: finding publish failed: MissingEndpoint: 'Endpoint' configuration is required for this service

Check the findings of Security Hub

Normally, the results of running kube-bench will be output to the Pod's log.

$ kubectl logs -f pod/kube-bench-2j2ss -n kube-bench
[INFO] 3 Worker Node Security Configuration
[INFO] 3.1 Worker Node Configuration Files
[PASS] 3.1.1 Ensure that the proxy kubeconfig file permissions are set to 644 or more restrictive (Scored)
[PASS] 3.1.2 Ensure that the proxy kubeconfig file ownership is set to root:root (Scored)
[PASS] 3.1.3 Ensure that the kubelet configuration file has permissions set to 644 or more restrictive (Scored)
[PASS] 3.1.4 Ensure that the kubelet configuration file ownership is set to root:root (Scored)
[INFO] 3.2 Kubelet
[PASS] 3.2.1 Ensure that the --anonymous-auth argument is set to false (Scored)
[PASS] 3.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Scored)
[PASS] 3.2.3 Ensure that the --client-ca-file argument is set as appropriate (Scored)
[PASS] 3.2.4 Ensure that the --read-only-port argument is set to 0 (Scored)
[PASS] 3.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Scored)
[PASS] 3.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Scored)
[PASS] 3.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Scored) 
[PASS] 3.2.8 Ensure that the --hostname-override argument is not set (Scored)
[WARN] 3.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture (Scored)
[PASS] 3.2.10 Ensure that the --rotate-certificates argument is not set to false (Scored)
[PASS] 3.2.11 Ensure that the RotateKubeletServerCertificate argument is set to true (Scored)

== Remediations node ==
3.2.9 If using a Kubelet config file, edit the file to set eventRecordQPS: to an appropriate level.
If using command line arguments, edit the kubelet service file
/etc/systemd/system/kubelet.service on each worker node and
set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable.
Based on your system, restart the kubelet service. For example:
systemctl daemon-reload
systemctl restart kubelet.service


== Summary node ==
14 checks PASS
0 checks FAIL
1 checks WARN
0 checks INFO

== Summary total ==
14 checks PASS
0 checks FAIL
1 checks WARN
0 checks INFO

If you specify the --asff flag and send the results to Security Hub, only the number of cases imported to Security Hub will be listed in the Pod's log.

$ kubectl logs -f pod/kube-bench-q4sbd -n kube-bench
2020/12/12 14:51:07 Number of findings that were successfully imported:1

In the case of this environment, the result was one.

Note that the results will be sent to Security Hub only for items whose check result is FAIL or WARN. If all checks are PASSed, no results will be generated to Security Hub.

If you check the findings in Security Hub console, you will see that the one result sent was successfully captured.

Using Custom Insights

Security Hub has a feature called Custom Insights that allows you to aggregate data by resource or environment using specific grouping condition and filters.

Custom Insights can be created in the Insights section of the Security Hub console by simply setting any grouping condition, filters

For example, if you are using Security Hub in a multi-account configuration, you can use the management account to aggregate the detection results for each member account.

Let's check out an example of a custom insight with the grouping condition set to AWS account ID and the product name Kube-bench set as a filter.
By clicking on the account ID, you will be able to instantly see the kube-bench results for the target account.

Even if you are operating with a single account, you can use it to list the detection results for each EKS cluster by setting the resource ID as a group by condition.

References

Integrating kube-bench with AWS Security Hub
https://github.com/aquasecurity/kube-bench/blob/master/docs/asff.md

Managing custom insights
https://docs.aws.amazon.com/securityhub/latest/userguide/securityhub-custom-insights.html

無料で利用可能な開発者向けのコンテンツ (ブログ) 作成プラットフォームおよびコミュニティです。
※海外のサイトなので、投稿されているコンテンツは英語がメインです。

https://hashnode.com/ の top ページより

Hashnode は様々な機能をもっているのですが、プラットフォームとしての特徴は Hashnode Co-Founder の Product Hunt への書き込みが表していると思いますので 以下に google 翻訳の上、引用させていただきます。

従来のパブリッシングプラットフォームは、コンテンツの所有権を犠牲にして可視性と エンゲージメントを提供することに気づきました。 一方、セルフホストソリューションを使用する場合、記事は適切な可視性と到達範囲を獲得できません。 Hashnodeは、両方の長所を兼ね備えています。カスタムドメインをマッピングし、 独自のブランドで記事を公開し、組み込みの開発者コミュニティに配布することができます。

投稿したコンテンツの所有権は完全にユーザーにあります。
一方でプラットフォームとしてコミュニティ機能を備えていますので
フォローしているユーザーやトレンドの記事を簡単に追うことができますし
自分の記事の共有や SEO といった観点でもメリットがあります。

機能の一部を紹介

Hashnode を実際に利用してみて、お気に入りの機能をいくつか紹介させてください。

Markdown エディタ

これは外せませんね！

カスタムドメイン

Hashnode の大きな特徴として、カスタムドメインを簡単に設定することができます。
アカウント作成後に払い出される URL は https://<ユーザー名>.hashnode.dev/ の形式です。
カスタムドメインの設定は Hashnode のダッシュボード上で、ドメインを指定し、
利用している DNS プロバイダに指定された CNAME レコードを追加するだけです。
デフォルトで HTTPS 化され、初回アクセス時に自動で Let's Encrypt の証明書が設定されます。

私のアカウントの例では以下のようになります。

• デフォルト: https://hayao-k.hashnode.dev/
• カスタムドメイン: https://hayao-k.dev/

たまたま「.dev」ドメインを利用しているため、短縮しただけのように見えてしまいますが
ご自身が所有するドメインを設定し、ブログを運用することができます。

※ ルート (Zone Apex) をカスタムドメインに指定するには利用する DNS プロバイダが
CNAME Flattening をサポートしている必要があります。
A レコードで登録することも可能ですが、その場合 Hashnode 組み込みの
グローバル CDN とエッジキャッシングが使用できないため、表示速度などに影響があります。

外観のカスタマイズ

ヘッダーの色や、ブログのロゴ、記事一覧レイアウトなどの見た目をカスタマイズできます。
また利用者はデフォルトのテーマとダークモードを自由に切り替えることができますが、
アクセス時のデフォルトのテーマをダークモードとすることも可能です。

記事ページ以外にも Markdown ベースで独自のページを追加することができます。
使い方としてはプロフィールページなどを追加しているユーザーが多いようです。
また Hashnode への投稿で特定の条件を満たすとバッジを獲得し、
BADGES ページに公開できます (非公開設定も可)。投稿のモチベーションの一つになります。

またブログ読者が登録可能なニュースレター機能も設定で有効無効を切り替えることできます。

GitHub Backup

Hashnode の GitHub アプリをインストールし、バックアップ先のリポジトリを指定することで
Story (記事) の追加、変更を画像込み自動で GitHub にバックアップできます。

Import & Export

Medium.com と Dev.to から既存のコンテンツを移行する機能が用意されています。
実際に Dev.to にポスト済みの記事で試してみましたが、画像込みで簡単に Import できました。
また Hashnode に投稿した全ての記事のデータを JSON 形式で Export する機能も用意されています。

無料 & 広告無し

将来的には 個人や Team 向けの Proプラン (月額サブスクリプション) を導入し
収益化することも検討しているようです。

3rd Party Integration

Hashnode 自体も記事毎の View 数などの Analytics 機能は持っているのですが
Google Analytics や Hotjar などのサイト分析ツールの ID を別途設定できます。

収益化 (投げ銭)

Hashnode 自体に収益化の機能はないのですが、前述の 3rd Party Integration の機能で
Web Monetization の Payment Pointer を埋め込むことが可能です。 これにより Coil というサービスの課金ユーザーがブログを訪れた場合に
自動的にチップを受け取ることができます。

Ambassador program

Hashnode を他のユーザーに紹介すると限定のカスタマイズ要素などをアンロックできるようです。
最後にこちらにリンクを張っておきます。

つ https://hashnode.com/@hayao-k/joinme

もしこの記事で興味を持っていただけたら、上記からポチッと登録いただけたら嬉しいです。

参考

Why Should You Start Your Dev Blog On Hashnode
Why Hashnode Is Different Than Other Blogging Platforms
Web Monetization on Hashnode

以上です。

AWS CDK allows you to create your own Construct Library and publish it to npm or PyPI.

Using projen makes the development of Construct Library very comfortable.

The content of this article has been tested with the following versions

• projen: v0.3.162
• AWS CDK: v1.73.0

What is projen?

projen is a tool for defining and managing increasingly complex project configurations in code.

https://github.com/projen/projen

With projen, you no longer need to manage files such as package.json by yourself.

projen does not only generate various files during project creation but also continuously updates and maintains these settings.

You can easily start a new project using the pre-defined project types.
As of November 2020, the following project types are supported.

Commands:
  projen new awscdk-app-ts     AWS CDK app in TypeScript.
  projen new awscdk-construct  AWS CDK construct library project.
  projen new cdk8s-construct   CDK8s construct library project.
  projen new jsii              Multi-language jsii library project.
  projen new nextjs            Next.js project without TypeScript.
  projen new nextjs-ts         Next.js project with TypeScript.
  projen new node              Node.js project.
  projen new project           Base project.
  projen new react             React project without TypeScript.
  projen new react-ts          React project with TypeScript.
  projen new typescript        TypeScript project.
  projen new typescript-app    TypeScript app.

awscdk-construct creates an environment for building Contruct using jsii.
jsii allows you to generate libraries from TypeScript code to work in Python, Java, and .NET.

Create project

Create a Construct Library project with projen new awscdk-construct.

$ mkdir cdk-sample-lib && cd cdk-sample-lib

$ npx projen new awscdk-construct
npx: installed 63 in 8.37s
🤖 Created .projenrc.js for AwsCdkConstructLibrary
🤖 Synthesizing project...
$GIT_USER_NAME is not defined

Under the project directory, .projenrc.js has been created.

const { AwsCdkConstructLibrary } = require('projen');

const project = new AwsCdkConstructLibrary({
  authorAddress: "user@domain.com",
  authorName: $GIT_USER_NAME,
  cdkVersion: "1.60.0",
  name: "cdk-sample-lib",
  repository: "https://github.com/user/cdk-sample-lib.git",
});

project.synth();

You can add dependencies on AWS CDKs and other modules to be used.

  cdkDependencies: [
    '@aws-cdk/core',
    '@aws-cdk/aws-apigatewayv2',
    '@aws-cdk/aws-apigatewayv2-integrations',
    '@aws-cdk/aws-lambda'
  ],
  deps: [ 
    'super-useful-lib' 
  ]

If you want to cross-compile to languages other than TypeScript with jsii, add the target language.

  python: {
    distName: 'cdk-sample-lib',
    module: 'cdk_sample_lib',
  },

See API.md for other available options.
As an example, the modified file looks like this

const { AwsCdkConstructLibrary } = require('projen');

const PROJECT_NAME = "cdk-sample-lib" 

const project = new AwsCdkConstructLibrary({
  authorAddress: "hayaok333@gmail.com",
  authorName: "hayao-k",
  cdkVersion: "1.73.0",
  name: PROJECT_NAME,
  repository: "https://github.com/hayao-k/cdk-sample-lib.git",
  defaultReleaseBranch: 'main',
  cdkDependencies: [ 
    '@aws-cdk/core',
    '@aws-cdk/aws-apigatewayv2',
    '@aws-cdk/aws-apigatewayv2-integrations',
    '@aws-cdk/aws-lambda'
  ],
  python: {
    distName: PROJECT_NAME,
    module: 'cdk_sample_lib',
  },
});

project.synth();

Once you have edited .projenrc.js, run the projen command to reflect the changes (yarn is required).

$ npx projen
npx: installed 63 in 5.491s
🤖 Synthesizing project...
🤖 yarn install --check-files
yarn install v1.22.5
info No lockfile found.
[1/5] Validating package.json...
[2/5] Resolving packages...
[3/5] Fetching packages...
info fsevents@2.2.1: The platform "linux" is incompatible with this module.
info "fsevents@2.2.1" is an optional dependency and failed compatibility check. Excluding it from installation.
[4/5] Linking dependencies...
[5/5] Building fresh packages...
success Saved lockfile.
Done in 20.34s.
🤖 jest: * => ^26.6.3
🤖 @types/jest: * => ^26.0.15
🤖 ts-jest: * => ^26.4.4
🤖 eslint: * => ^7.13.0
🤖 eslint-import-resolver-node: * => ^0.3.4
🤖 eslint-import-resolver-typescript: * => ^2.3.0
🤖 eslint-plugin-import: * => ^2.22.1
🤖 json-schema: * => ^0.2.5
🤖 Synthesis complete

----------------------------------------------------------------------------------------------------
Commands:

BUILD
compile          Only compile
watch            Watch & compile in the background
build            Full release build (test+compile)

TEST
test             Run tests
test:watch       Run jest in watch mode
eslint           Runs eslint against the codebase

RELEASE
compat           Perform API compatibility check against latest version
release          Bumps version & push to master
docgen           Generate API.md from .jsii manifest
bump             Commits a bump to the package version based on conventional commits
package          Create an npm tarball

MAINTAIN
projen           Synthesize project configuration from .projenrc.js
projen:upgrade   upgrades projen to the latest version

MISC
start            Shows this menu

Tips:
💡 The VSCode jest extension watches in the background and shows inline test results
💡 Install Mergify in your GitHub repository to enable automatic merges of approved PRs
💡 Set `autoUpgradeSecret` to enable automatic projen upgrade pull requests
💡 `API.md` includes the API reference for your library
💡 Set "compat" to "true" to enable automatic API breaking-change validation

You will see that projen automatically generates the package.json, the .gitignore, .npmignore, eslint, jsii configuration, license files, etc., as well as the creation and installation of the package.json.

You no longer have to copy from an existing project every time you create a new project.

Whenever you edit these files, you need to modify the .projenrc.js file and re-run the projen command.
If you edit them manually, the build will fail.

$ tree -L 1 -a
.
├── .eslintrc.json
├── .github
├── .gitignore
├── LICENSE
├── .mergify.yml
├── node_modules
├── .npmignore
├── package.json
├── .projenrc.js
├── README.md
├── src
├── test
├── tsconfig.jest.json
├── version.json
├── .versionrc.json
└── yarn.lock

Development

Let's try a simple example of calling Hello World's Lambda from the API Gateway (HTTP API).
Please note that the HTTP API L2 Constructs has an Experimental status as of November 2020.

The following directory has already been created by projen.

.
├── lib/ 
├── src/
├── test/

The lib directory will contain the compiled files.

The code for the Lambda functions can also be inserted inline into the CDK code, but in this example create index.js in the functions directory.

• functions/index.js

exports.handler = async (event) => {
    const response = {
        statusCode: 200,
        body: JSON.stringify('Hello from Lambda!'),
    };
    return response;
};

Create the following two files in the src directory

• src/index.ts

import { HttpApi } from '@aws-cdk/aws-apigatewayv2';
import { LambdaProxyIntegration } from '@aws-cdk/aws-apigatewayv2-integrations';
import { Code, Function, Runtime } from '@aws-cdk/aws-lambda';
import * as cdk from '@aws-cdk/core';

export class CdkSampleLib extends cdk.Construct {
  constructor(scope: cdk.Construct, id: string) {
    super(scope, id);

    const handler = new Function(this, 'HelloWorld', {
      handler: 'index.handler',
      code: Code.fromAsset('functions'),
      runtime: Runtime.NODEJS_12_X,
    });

    const api = new HttpApi(this, 'API', {
      defaultIntegration: new LambdaProxyIntegration({ handler }),
    });

    new cdk.CfnOutput(this, 'ApiURL', { value: api.url! });
  }
}

• src/integ.default.ts

import * as cdk from '@aws-cdk/core';
import { CdkSampleLib } from './index';

const app = new cdk.App();
const stack = new cdk.Stack(app, 'MyStack');

new CdkSampleLib(stack, 'Cdk-Sample-Lib');

Create the following files in the test directory

• test/hello.test.ts

import * as cdk from '@aws-cdk/core';
import { CdkSampleLib } from '../src/index';
import '@aws-cdk/assert/jest';

test('create app', () => {
  const app = new cdk.App();
  const stack = new cdk.Stack(app);
  new CdkSampleLib(stack, 'TestStack');
  expect(stack).toHaveResource('AWS::Lambda::Function');
  expect(stack).toHaveResource('AWS::ApiGatewayV2::Api');
  expect(stack).toHaveResource('AWS::ApiGatewayV2::Integration');
});

Unit Test

Various scripts are predefined in the package.json generated from projen.

Run the test with yarn test. yarn build also run test, so omit the example output here.

Build

Run yarn build and compile TypeScript to the jsii module.

jsii-docgen generates API documentation (API.md) from comments in the code.

In addition, jsii-pacmak creates language-specific public packages in the dist directory.

$ yarn build
yarn run v1.22.5
$ yarn run test && yarn run compile && yarn run package
$ rm -fr lib/ && jest --passWithNoTests --updateSnapshot && yarn run eslint
 PASS  test/hello.test.ts
  ✓ create app (197 ms)

----------|---------|----------|---------|---------|-------------------
File      | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
----------|---------|----------|---------|---------|-------------------
All files |     100 |      100 |     100 |     100 |                   
 index.ts |     100 |      100 |     100 |     100 |                   
----------|---------|----------|---------|---------|-------------------
Test Suites: 1 passed, 1 total
Tests:       1 passed, 1 total
Snapshots:   0 total
Time:        4.713 s, estimated 7 s
Ran all test suites.
$ eslint --ext .ts,.tsx --fix --no-error-on-unmatched-pattern src test
$ jsii --silence-warnings=reserved-word --no-fix-peer-dependencies && jsii-docgen
$ jsii-pacmak
Done in 51.38s.

Once the build is successful, let's try deploying locally.

$ cdk deploy --app='./lib/integ.default.js'
This deployment will make potentially sensitive changes according to your current security approval level (--require-approval broadening).
Please confirm you intend to make the following modifications:

IAM Statement Changes
┌───┬──────────────────────────────────────────────┬────────┬───────────────────────┬──────────────────────────────────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│   │ Resource                                     │ Effect │ Action                │ Principal                        │ Condition                                                                                                                           │
├───┼──────────────────────────────────────────────┼────────┼───────────────────────┼──────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ + │ ${Cdk-Sample-Lib/HelloWorld.Arn}             │ Allow  │ lambda:InvokeFunction │ Service:apigateway.amazonaws.com │ "ArnLike": {                                                                                                                        │
│   │                                              │        │                       │                                  │   "AWS:SourceArn": "arn:${AWS::Partition}:execute-api:${AWS::Region}:${AWS::AccountId}:${CdkSampleLibAPI6FD5D6E6}/*/*"              │
│   │                                              │        │                       │                                  │ }                                                                                                                                   │
├───┼──────────────────────────────────────────────┼────────┼───────────────────────┼──────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ + │ ${Cdk-Sample-Lib/HelloWorld/ServiceRole.Arn} │ Allow  │ sts:AssumeRole        │ Service:lambda.amazonaws.com     │                                                                                                                                     │
└───┴──────────────────────────────────────────────┴────────┴───────────────────────┴──────────────────────────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
IAM Policy Changes
┌───┬──────────────────────────────────────────┬────────────────────────────────────────────────────────────────────────────────┐
│   │ Resource                                 │ Managed Policy ARN                                                             │
├───┼──────────────────────────────────────────┼────────────────────────────────────────────────────────────────────────────────┤
│ + │ ${Cdk-Sample-Lib/HelloWorld/ServiceRole} │ arn:${AWS::Partition}:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole │
└───┴──────────────────────────────────────────┴────────────────────────────────────────────────────────────────────────────────┘
(NOTE: There may be security-related changes not in this list. See https://github.com/aws/aws-cdk/issues/1299)

Do you wish to deploy these changes (y/n)? y
MyStack: deploying...
[0%] start: Publishing 20472c64d312cc547a9359d36b04cfc75633027c0b13c3c07b96dfdf1b1c428f:current
[100%] success: Published 20472c64d312cc547a9359d36b04cfc75633027c0b13c3c07b96dfdf1b1c428f:current
MyStack: creating CloudFormation changeset...
[██████████████████████████████████████████████████████████] (9/9)

 ✅  MyStack

Outputs:
MyStack.CdkSampleLibApiURL32C6192A = https://4p9zte6ny8.execute-api.ap-northeast-1.amazonaws.com/

Stack ARN:
arn:aws:cloudformation:ap-northeast-1:123456789012:stack/MyStack/8fc38650-2a78-11eb-8d53-0a6e476ede30

You can check the response of the Lambda function from the output API URL.

$ curl https://4p9zte6ny8.execute-api.ap-northeast-1.amazonaws.com/
"Hello from Lambda!"

To remove it, run the cdk destory.

$ cdk destroy --app='./lib/integ.default.js'                                        
Are you sure you want to delete: MyStack (y/n)? y
MyStack: destroying...
5:27:29 PM | DELETE_IN_PROGRESS   | AWS::CloudFormation::Stack | MyStack

 ✅  MyStack: destroyed

Release

First, commit the changes.

$ git add . 
$ git commit -m "feat: release 0.0.1"

yarn release bumps the version and automatically updates CANGELOG.md. Then push to the release branch of GitHub.

$ yarn release
yarn run v1.22.5
$ yarn run --silent no-changes || (yarn run bump && git push --follow-tags origin main)
$ yarn run --silent no-changes || standard-version
✔ bumping version in version.json from 0.0.0 to 0.0.1
✔ Running lifecycle script "postbump"
ℹ - execute command: "yarn run projen && git add ."
🤖 yarn install --check-files
🤖 Synthesis complete

----------------------------------------------------------------------------------------------------
Commands:

BUILD
compile          Only compile
watch            Watch & compile in the background
build            Full release build (test+compile)

TEST
test             Run tests
test:watch       Run jest in watch mode
eslint           Runs eslint against the codebase

RELEASE
compat           Perform API compatibility check against latest version
release          Bumps version & push to main
docgen           Generate API.md from .jsii manifest
bump             Commits a bump to the package version based on conventional commits
package          Create an npm tarball

MAINTAIN
projen           Synthesize project configuration from .projenrc.js
projen:upgrade   upgrades projen to the latest version

MISC
start            Shows this menu

Tips:
💡 The VSCode jest extension watches in the background and shows inline test results
💡 Install Mergify in your GitHub repository to enable automatic merges of approved PRs
💡 Set `autoUpgradeSecret` to enable automatic projen upgrade pull requests
💡 `API.md` includes the API reference for your library
💡 Set "compat" to "true" to enable automatic API breaking-change validation

✔ created CHANGELOG.md
✔ outputting changes to CHANGELOG.md
✔ committing version.json and CHANGELOG.md and all staged files
✔ tagging release v0.0.1
ℹ Run `git push --follow-tags origin master` to publish

Or you can bump to any version by running yarn bump.

$ yarn bump --release-as 1.0.0 && git push --follow-tags origin main

The Github Actions workflow definition is also generated when the projen command is executed, which makes it easy to automate the release to the package repository.

• Build workflow (.github/workflows/build.yaml) Runs when a pull request is created.
  Builds the library and checks for tampering (i.e., manual modification).

• Release workflow (.github/workflows/release.yaml): It is triggered by push to the release branch.
  After building in the release branch, it automatically publishes to the repositories, such as npm and PyPI by jsii-release.

In order for the Release job to work properly, you need to register the Secrets for the language you want to publish in the GitHub repository.

• npm: NPM_TOKEN
• .NET: NUGET_API_KEY
• Java: MAVEN_GPG_PRIVATE_KEY, MAVEN_GPG_PRIVATE_KEY_PASSPHRASE, MAVEN_PASSWORD, MAVEN_USERNAME, MAVEN_STAGING_PROFILE_ID
• Python: TWINE_USERNAME, TWINE_PASSWORD

Try It!

With projen, you can focus on the implementation of the Construct Library (and of course, on the regular CDK App).

Do you want to understand how to use projen in videos?
The following video published by @pahud, an AWS Developer Advocate, is very helpful.

I recently published a construct library called cdk-ecr-image-scan-notify using projen.

I hope this article will help you.

https://github.com/hayao-k/aws-sso-cloudformation-sample

Introduction

On 9/10/2020, AWS SSO API (sso-admin) was finally added to AWS Single Sign-On and operations through the AWS CLI/SDK and CloudFormation are now supported.

🚀 AWS Single Sign-On adds account assignment APIs and AWS CloudFormation support to automate multi-account access management

Until now, operations such as permission sets and assigning users/groups to AWS accounts had to be configured manually in the console.
This update paves the way for automating account assignment settings and getting to IaC!

Supported CloudFormation resources

As of September 2020, CloiudFormation will support the following two resources.

• AWS::SSO::PermissionSet Specifies the permissions to be set on the AWS SSO instance.

• AWS::SSO::Assignment Assign access to your AWS account using the specified permission set.

Points to note

Creating an AWS SSO resource in CloudFormation requires various IDs such as InstanceArn, Identiy-Store-Id, UserId, and GroupId.
These IDs are not display in the AWS SSO console and must be obtained via API.
To use the AWS SSO API (sso-admin), make sure you have AWS CLI version 1.18.136 or 2.0.48 or higher.

Example of a permission set

This is an example of creating a simple set of permissions using AWS management policies.

PermissionSet:
  Type: AWS::SSO::PermissionSet
  Properties:
    InstanceArn: 'arn:aws:sso:::instance/ssoins-xxxxxxxxxxxxxxxx'
    Name: 'AdministratorAccess'
    ManagedPolicies:
    - 'arn:aws:iam::aws:policy/AdministratorAccess'

For InstanceArn, specify the ARN of the SSO instance. This value is obtained from list-instances in sso-admin.

$ aws sso-admin list-instances
{
    "Instances": [
        {
            "InstanceArn": "arn:aws:sso:::instance/ssoins-xxxxxxxxxxxxxxxx",
            "IdentityStoreId": "d-xxxxxxxxxx"
        }
    ]
}

Example of Account Assignment

Specify the AWS account, privilege set, and principal as follows.

Assignment: 
  Type: AWS::SSO::Assignment
  Properties: 
    InstanceArn: 'arn:aws:sso:::instance/ssoins-xxxxxxxxxxxxxxxx'
    PermissionSetArn: 'arn:aws:sso:::permissionSet/ssoins-xxxxxxxxxxxxxxxx/ps-xxxxxxxxxxxxxxxx'
    TargetId: '123456789012'
    TargetType: AWS_ACCOUNT
    PrincipalId: 'f81d4fae-7dec-11d0-a765-00a0c91e6bf6'
    PrincipalType: 'GROUP'

You can check the PermissionSetArn with sso-admin's describe-permission-set.

$ aws sso-admin list-permission-sets \
> --instance-arn arn:aws:sso:::instance/ssoins-xxxxxxxxxxxxxxxx 
{
    "PermissionSets": [
        "arn:aws:sso:::permissionSet/ssoins-xxxxxxxxxxxxxxxx/ps-xxxxxxxxxxxxxxxx",
        "arn:aws:sso:::permissionSet/ssoins-xxxxxxxxxxxxxxxx/ps-yyyyyyyyyyyyyyyy",
        "arn:aws:sso:::permissionSet/ssoins-xxxxxxxxxxxxxxxx/ps-zzzzzzzzzzzzzzzz"
    ]
}

$ aws sso-admin describe-permission-set \
> --instance-arn arn:aws:sso:::instance/ssoins-xxxxxxxxxxxxxxx \
> --permission-set-arn arn:aws:sso:::permissionSet/ssoins-xxxxxxxxxxxxxxx/ps-xxxxxxxxxxxxxxx
{
    "PermissionSet": {
        "Name": "AdministratorAccess",
        "PermissionSetArn": "arn:aws:sso:::permissionSet/ssoins-xxxxxxxxxxxxxxx/ps-xxxxxxxxxxxxxxx",
        "CreatedDate": "2020-09-09T19:01:06.758000+09:00",
        "SessionDuration": "PT1H"
    }
}

If you have created a permission set in CloudFormation, use the built-in Fn::GetAtt function.

    PermissionSetArn: !GetAtt LogicalId.PermissionSetArn

Only AWS_ACCOUNT can be specified for TargetType. Specify the AWS account ID to be assigned for TargetId.

    TargetId: '123456789012'
    TargetType: AWS_ACCOUNT

PrincipalType can be specified as either USER or GROUP.
PrincipalId must be specified as the GUID of the user/group to be assigned.

    PrincipalId: 'f81d4fae-7dec-11d0-a765-00a0c91e6bf6'
    PrincipalType: 'GROUP'

Many operations of the AWS SSO API (sso-admin) rely on user and group identifiers called principals.
Use the AWS SSO Identity Store API (identitystore) to get the GUIDs for a user/group.

Specify the identity store identifier obtained from list-instances in sso-admin for --identity-store-id.
For list-users you can specify a UserName and for list-group you can specify a DisplayName as filter.

$ aws identitystore list-users \
> --identity-store-id d-xxxxxxxxxx \
> --filters AttributePath=UserName,AttributeValue="user@example.com"
{
    "Users": [
        {
            "UserName": "userXX@examle.com",
            "UserId": "f81d4faxge-7dec11d8-a765-3at5-80e4-00a0c91e6bf6"
        }
    ]
}

$ aws identitystore list-groups \
> --identity-store-id d-xxxxxxxxxx \
> --filters AttributePath=DisplayName,AttributeValue="TestGroup"
{
    "Groups": [
        {
            "GroupId": "f81d4faxge-789fcfa5-005c-4379-89ba-10a11e641c17"
            "DisplayName": "TestGroup"
        }
    ]
}

See the GitHub link at the top for the entire template.

Summary

• The AWS SSO API (sso-admin) has been added, enabling partial automation of administrative tasks and IaC
• CloudFormation supports permission set creation and account assignment
• User/Group IDs needed to identify principals must be obtained through the IdentityStore API
  • Full automation seems to be difficult at the moment because of the need to search for IDs when assigning accounts
  • AWS SSO console does not display these IDs and ARNs, which is a bit inconvenient
• That said, it's exciting to be able to manage AWS SSO with APIs!

References

AWS CloudFormation User Guide - SSO resource type reference
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/AWS_SSO.html

AWS CLI Command Reference - sso-admin
https://awscli.amazonaws.com/v2/documentation/api/latest/reference/sso-admin/index.html

AWS CLI Command Reference - identitystore
https://awscli.amazonaws.com/v2/documentation/api/latest/reference/identitystore/index.html