Question 17: Audit Log Policy

Problem Statement

Solve this question on: ssh cks3477

Audit Logging has been enabled in the cluster with an Audit Policy located at /etc/kubernetes/audit/policy.yaml on cks3477.

Tasks:

  1. Change the configuration so that only one backup of the logs is stored
  2. Alter the Policy in a way that it only stores logs:
    • From Secret resources, level Metadata
    • From "system:nodes" userGroups, level RequestResponse
  3. After you altered the Policy make sure to empty the log file so it only contains entries according to your changes
You can use jq to render json more readable, like cat data.json | jq
Use sudo -i to become root which may be required for this question

Solution

Step 1: Update API Server Configuration

First, let's check and update the API server configuration: 

➜ ssh cks3477

➜ candidate@cks3477:~# sudo -i

➜ root@cks3477:~# cp /etc/kubernetes/manifests/kube-apiserver.yaml ~/17_kube-apiserver.yaml # backup

➜ root@cks3477:~# vim /etc/kubernetes/manifests/kube-apiserver.yaml

Update the audit log configuration to keep only one backup: 

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --audit-policy-file=/etc/kubernetes/audit/policy.yaml
    - --audit-log-path=/etc/kubernetes/audit/logs/audit.log
    - --audit-log-maxsize=5
    - --audit-log-maxbackup=1                                    # CHANGE
    - --advertise-address=192.168.100.21
    - --allow-privileged=true
...
Important: When creating a backup, don't place it in the /etc/kubernetes/manifests directory. The kubelet will try to create the backup file as a static Pod since it looks for any files in that directory.
Step 2: Update Audit Policy

Now, let's update the audit policy:

➜ root@cks3477:~# vim /etc/kubernetes/audit/policy.yaml

Update the policy to match the requirements:

apiVersion: audit.k8s.io/v1
kind: Policy
rules:

# log Secret resources audits, level Metadata
- level: Metadata
  resources:
  - group: ""
    resources: ["secrets"]

# log node related audits, level RequestResponse
- level: RequestResponse
  userGroups: ["system:nodes"]

# for everything else don't log anything
- level: None
Step 3: Apply Changes

Restart the API server to apply the changes:

➜ root@cks3477:~# cd /etc/kubernetes/manifests/

➜ root@cks3477:/etc/kubernetes/manifests# mv kube-apiserver.yaml ..

➜ root@cks3477:/etc/kubernetes/manifests# watch crictl ps # wait for apiserver gone

➜ root@cks3477:/etc/kubernetes/manifests# echo > /etc/kubernetes/audit/logs/audit.log

➜ root@cks3477:/etc/kubernetes/manifests# mv ../kube-apiserver.yaml .

➜ root@cks3477:/etc/kubernetes/manifests# watch crictl ps # wait for apiserver created
Step 4: Verify the Changes

Check the audit logs to verify the changes:

➜ root@cks3477:~# cat /etc/kubernetes/audit/logs/audit.log | tail | jq

Example of a Secret resource audit log entry:

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "Metadata",
  "auditID": "aac47b4d-d1fe-4ab8-a0eb-f7843a89560f",
  "stage": "RequestReceived",
  "requestURI": "/api/v1/namespaces/restricted/secrets?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsecret1&resourceVersion=9028&timeout=9m45s&timeoutSeconds=585&watch=true",
  "verb": "watch",
  "user": {
    "username": "system:node:cks3477-node1",
    "groups": [ 
      "system:nodes",
      "system:authenticated"
    ]
  },
  "sourceIPs": [
    "192.168.100.22"
  ],
  "userAgent": "kubelet/v1.32.1 (linux/amd64) kubernetes/a51b3b7",
  "objectRef": {
    "resource": "secrets",
    "namespace": "restricted",
    "name": "secret1",
    "apiVersion": "v1"
  },
  "requestReceivedTimestamp": "2024-09-08T12:20:43.920816Z",
  "stageTimestamp": "2024-09-08T12:20:43.920816Z"
}

Example of a system:nodes audit log entry:

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "RequestResponse",
  "auditID": "80577862-91da-4bc4-bb9d-b1ebffdc0dda",
  "stage": "ResponseComplete",
  "requestURI": "/api/v1/nodes/cks3477?resourceVersion=0&timeout=10s",
  "verb": "get",
  "user": {
    "username": "system:node:cks3477",
    "groups": [
      "system:nodes",
      "system:authenticated"
    ]
  },
  "sourceIPs": [
    "192.168.100.21"
  ],
  "userAgent": "kubelet/v1.32.1 (linux/amd64) kubernetes/a51b3b7",
  "objectRef": {
    "resource": "nodes",
    "name": "cks3477",
    "apiVersion": "v1"
  },
  "responseStatus": {
    "metadata": {},
    "code": 200
  },
  "responseObject": {
    ...
  },
  "requestReceivedTimestamp": "2024-09-08T12:20:43.961117Z",
  "stageTimestamp": "2024-09-08T12:20:43.991929Z",
  "annotations": {
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": ""
  }
}
Step 5: Verify Policy Compliance

Run some verification commands to ensure the policy is working as expected: 

# shows Secret entries
cat audit.log | grep '"resource":"secrets"' | wc -l

# confirms Secret entries are only of level Metadata
cat audit.log | grep '"resource":"secrets"' | grep -v '"level":"Metadata"' | wc -l

# shows RequestResponse level entries
cat audit.log | grep -v '"level":"RequestResponse"' | wc -l

# shows RequestResponse level entries are only for system:nodes
cat audit.log | grep '"level":"RequestResponse"' | grep -v "system:nodes" | wc -l
Summary of changes:
  • Updated API server configuration to keep only one backup of audit logs
  • Modified audit policy to:
    • Log Secret resources at Metadata level
    • Log system:nodes userGroups at RequestResponse level
    • Disable logging for all other resources
  • Cleared existing audit logs
  • Verified the changes through log inspection
Security Note: Proper audit logging is crucial for security monitoring and compliance. The audit policy should be carefully configured to capture relevant events while managing log volume and storage requirements.
Back to Questions List