Introduction
kube-bench has been added to the 3rd party integrations in AWS Security Hub.
🚀 AWS Security Hub adds open source tool integrations with Kube-bench and Cloud Custodian
https://aws.amazon.com/jp/about-aws/whats-new/2020/12/aws-security-hub-adds-open-source-tool-integration-with-kube-bench-and-cloud-custodian/
With this integration, the results of the CIS Kubernetes Benchmark and CIS Amazon EKS Benchmark checks run by kube-bench can now be centrally managed in AWS Security Hub.
What is CIS Benchmark?
CIS Benchmark is a set of guidelines published by the Center for Internet Security (CIS), a non-profit organization in the United States, for strengthening various operating systems, servers, and cloud environments.
CIS Benchmark is referenced in compliance requirements such as the PCI DSS when it states "industry-accepted system hardening standards".
You can download over 140 CIS Benchmarks in PDF format from the CIS website.
https://www.cisecurity.org/cis-benchmarks/
What is kube-bench?
kube-bench is a Go application that allows you to check if your environment complies with the recommendations listed in the CIS Kubernetes Benchmark.
kube-bench supports checking not only the CIS Kubernetes Benchmark, but also the CIS Amazon Elastic Kubernetes Service (EKS) Benchmark and CIS Google Kubernetes Engine (GKE) Benchmark.
What is AWS Security Hub?
AWS Security Hub is a service for aggregating and centrally managing various security data for the entire AWS environment.
Security Hub integrates with many 3rd party security products, as well as AWS services such as Amazon GuardDuty, Inspector, and Macie.
Data can be sent from Security Hub-enabled products to the Security Hub and vice versa.
Internally, the result information is managed in a JSON type format called AWS Security Finding Format (ASFF), so you can import your own data as long as it conforms to this format.
Prerequisites
I have confirmed that the following versions work.
- Amazon EKS: 1.17
- eksctl: 0.30.0
- kube-bench: 0.40.0
Ran the CIS Amazon EKS Benchmark v1.0 check in an EKS cluster.
Enable integration
Search for kube-bench from the Security Hub console integration and click "Accept findings" to see information about the IAM policies required to send the findings to Security Hub.
Select "Accept findings" again on the confirmation screen, and the status will be activated.
Configue IAM Roles
In order for kube-bench to run in an EKS cluster, the pod must have permissions to send check results to Security Hub.
There are two ways to assign access to AWS resources to a Pod.
Using IRSA, you can associate an IAM role with a Kubernetes service account.
This allows you to provide Security Hub access only to pods launched by kube-bench.
Note that if you use an IAM role that is configured for a node group, all Pods running under that group will be assigned access to the Security Hub.
This time we will use IRSA.
If you haven't created an IAM OIDC Provider to use IRSA, do so first!
$ eksctl utils associate-iam-oidc-provider \
--cluster {CLUSTER_NAME} --approve --region ap-northeast-1
Next, create an IAM role with the following policies attached.
Replace the regions as needed.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "securityhub:BatchImportFindings",
"Resource": [
"arn:aws:securityhub:ap-northeast-1::product/aqua-security/kube-bench"
]
}
]
}
Add the following policy to the trust relationship of the IAM role.
Set <ACCOUNT_ID> and <OIDR_PROVIDER_ID> to your own.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::<ACCOUNT_ID>:oidc-provider/oidc.eks.ap-northeast-1.amazonaws.com/id/<OIDC_PROVIDER_ID>"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringLike": {
"oidc.eks.ap-northeast-1.amazonaws.com/id/<OIDC_PROVIDER_ID>:sub": "system:serviceaccount:kube-bench:*",
"oidc.eks.ap-northeast-1.amazonaws.com/id/<OIDC_PROVIDER_ID>:aud": "sts.amazonaws.com"
}
}
}
]
}
This is written assuming that kube-bech is used for Namespace.
Create container image
We need to build a container image of kube-bench and push it to ECR.
First, let's create an ECR repository to store the image.
$ aws ecr create-repository --repository-name k8s/kube-bench --image-tag-mutability MUTABLE
{
"repository": {
"repositoryArn": "arn:aws:ecr:ap-northeast-1:123456789012:repository/k8s/kube-bench",
"registryId": "123456789012",
"repositoryName": "k8s/kube-bench",
"repositoryUri": "123456789012.dkr.ecr.ap-northeast-1.amazonaws.com/k8s/kube-bench",
"createdAt": 1607747704.0,
"imageTagMutability": "MUTABLE",
"imageScanningConfiguration": {
"scanOnPush": false
},
"encryptionConfiguration": {
"encryptionType": "AES256"
}
}
}
Clone the source code of kube-bench from GitHub.
$ git clone https://github.com/aquasecurity/kube-bench.git
Cloning into 'kube-bench'...
remote: Enumerating objects: 4115, done.
remote: Total 4115 (delta 0), reused 0 (delta 0), pack-reused 4115
Receiving objects: 100% (4115/4115), 7.69 MiB | 5.28 MiB/s, done.
Resolving deltas: 100% (2644/2644), done.
$ cd kube-bench
If you want to send the execution result to Security Hub, you need to edit cfg/eks-1.0/config.yaml and rewrite the AWS account, region and cluster name to be executed before building the image.
---
AWS_ACCOUNT: "123456789012"
AWS_REGION: "ap-northeast-1"
CLUSTER_ARN: "arn:aws:eks:ap-northeast-1:123456789012:cluster/{YOUR_CLUSTER_NAME}"
Build the container image and push it to ECR.
$ aws ecr get-login-password | docker login --username AWS --password-stdin https://123456789012.dkr.ecr.ap-northeast-1.amazonaws.com
Login Succeeded
$ docker build -t k8s/kube-bench .
...
Successfully built 6ad073f96455
Successfully tagged k8s/kube-bench:latest
$ docker tag k8s/kube-bench:latest 123456789012.dkr.ecr.ap-northeast-1.amazonaws.com/k8s/kube-bench:latest
$ docker push 123456789012.dkr.ecr.ap-northeast-1.amazonaws.com/k8s/kube-bench:latest
The push refers to repository [123456789012.dkr.ecr.ap-northeast-1.amazonaws.com/k8s/kube-bench]
bd6c279efeaa: Pushed
3032a8c3bf7a: Pushed
b0efa7564210: Pushed
fe4cf80d4f2c: Pushed
31c3d3db74eb: Pushed
d2e36eff2b5d: Pushed
f4666769fca7: Pushed
latest: digest: sha256:da95de0edccad7adb6fd6c80137a65b5458142efe14efa94ac44cc5c6ce6b2ef size: 1782
Run kube-bench
Creating a Service Account
Create a service account for kube-bench using a manifest file like the following. Note that the annotations specifies the IAM role that we just created.
apiVersion: v1
kind: ServiceAccount
metadata:
name: kube-bench-sa
namespace: kube-bench
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/<IAM_ROLE_NAME>
Create a namespace kube-bench and apply it.
$ kubectl create ns kube-bench
kubectl create ns kube-bench
$ kubectl apply -f sa.yaml
serviceaccount/kube-bench-sa created
$ kubectl get sa -n kube-bench
NAME SECRETS AGE
default 1 6m32s
kube-bench-sa 1 11s
Run a job
Open jobs-eks.yaml cloned from GitHub, and edit image and command. Adding the --asff flag to the command will send the results to Security Hub.
Add the service account name that you just created.
The edited file should look like the following.
---
apiVersion: batch/v1
kind: Job
metadata:
name: kube-bench
spec:
template:
spec:
hostPID: true
serviceAccountName: kube-bench-sa
containers:
- name: kube-bench
image: 123456789012.dkr.ecr.region.amazonaws.com/k8s/kube-bench:latest
command: ["kube-bench", "node", "--benchmark", "eks-1.0", "--asff"]
volumeMounts:
- name: var-lib-kubelet
mountPath: /var/lib/kubelet
readOnly: true
- name: etc-systemd
mountPath: /etc/systemd
readOnly: true
- name: etc-kubernetes
mountPath: /etc/kubernetes
readOnly: true
restartPolicy: Never
volumes:
- name: var-lib-kubelet
hostPath:
path: "/var/lib/kubelet"
- name: etc-systemd
hostPath:
path: "/etc/systemd"
- name: etc-kubernetes
hostPath:
path: "/etc/kubernetes"
Run kube-bench as a Kubernetes job and make sure it exits successfully.
$ kubectl apply -f job-eks.yaml -n kube-bench
job.batch/kube-bench created
$ kubectl get all -n kube-bench
NAME READY STATUS RESTARTS AGE
pod/kube-bench-q4sbd 0/1 Completed 0 55s
NAME COMPLETIONS DURATION AGE
job.batch/kube-bench 1/1 5s 55s
If the following message is output to the Pod's log and the job fails, please check whether the contents of cfg/eks-1.0/config.yaml are correct.
failed to output to ASFF: finding publish failed: MissingEndpoint: 'Endpoint' configuration is required for this service
Check the findings of Security Hub
Normally, the results of running kube-bench will be output to the Pod's log.
$ kubectl logs -f pod/kube-bench-2j2ss -n kube-bench
[INFO] 3 Worker Node Security Configuration
[INFO] 3.1 Worker Node Configuration Files
[PASS] 3.1.1 Ensure that the proxy kubeconfig file permissions are set to 644 or more restrictive (Scored)
[PASS] 3.1.2 Ensure that the proxy kubeconfig file ownership is set to root:root (Scored)
[PASS] 3.1.3 Ensure that the kubelet configuration file has permissions set to 644 or more restrictive (Scored)
[PASS] 3.1.4 Ensure that the kubelet configuration file ownership is set to root:root (Scored)
[INFO] 3.2 Kubelet
[PASS] 3.2.1 Ensure that the --anonymous-auth argument is set to false (Scored)
[PASS] 3.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Scored)
[PASS] 3.2.3 Ensure that the --client-ca-file argument is set as appropriate (Scored)
[PASS] 3.2.4 Ensure that the --read-only-port argument is set to 0 (Scored)
[PASS] 3.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Scored)
[PASS] 3.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Scored)
[PASS] 3.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Scored)
[PASS] 3.2.8 Ensure that the --hostname-override argument is not set (Scored)
[WARN] 3.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture (Scored)
[PASS] 3.2.10 Ensure that the --rotate-certificates argument is not set to false (Scored)
[PASS] 3.2.11 Ensure that the RotateKubeletServerCertificate argument is set to true (Scored)
== Remediations node ==
3.2.9 If using a Kubelet config file, edit the file to set eventRecordQPS: to an appropriate level.
If using command line arguments, edit the kubelet service file
/etc/systemd/system/kubelet.service on each worker node and
set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable.
Based on your system, restart the kubelet service. For example:
systemctl daemon-reload
systemctl restart kubelet.service
== Summary node ==
14 checks PASS
0 checks FAIL
1 checks WARN
0 checks INFO
== Summary total ==
14 checks PASS
0 checks FAIL
1 checks WARN
0 checks INFO
If you specify the --asff flag and send the results to Security Hub, only the number of cases imported to Security Hub will be listed in the Pod's log.
$ kubectl logs -f pod/kube-bench-q4sbd -n kube-bench
2020/12/12 14:51:07 Number of findings that were successfully imported:1
In the case of this environment, the result was one.
Note that the results will be sent to Security Hub only for items whose check result is FAIL or WARN. If all checks are PASSed, no results will be generated to Security Hub.
If you check the findings in Security Hub console, you will see that the one result sent was successfully captured.
Using Custom Insights
Security Hub has a feature called Custom Insights that allows you to aggregate data by resource or environment using specific grouping condition and filters.
Custom Insights can be created in the Insights section of the Security Hub console by simply setting any grouping condition, filters
For example, if you are using Security Hub in a multi-account configuration, you can use the management account to aggregate the detection results for each member account.
Let's check out an example of a custom insight with the grouping condition set to AWS account ID and the product name Kube-bench set as a filter.
By clicking on the account ID, you will be able to instantly see the kube-bench results for the target account.
Even if you are operating with a single account, you can use it to list the detection results for each EKS cluster by setting the resource ID as a group by condition.
References
Integrating kube-bench with AWS Security Hub
https://github.com/aquasecurity/kube-bench/blob/master/docs/asff.md
Managing custom insights
https://docs.aws.amazon.com/securityhub/latest/userguide/securityhub-custom-insights.html