Question 25: Audit Log Analysis

Problem Statement

Solve this question on: ssh cks3477

Namespace security contains five Secrets of type Opaque which can be considered highly confidential. The latest Incident-Prevention-Investigation revealed that ServiceAccount p.auster had too broad access to the cluster for some time. This SA should never had access to any Secrets in that Namespace.

Tasks:

  1. Find out which Secrets in Namespace security this SA did access by looking at the Audit Logs under /opt/course/p2/audit.log
  2. Change the password to any new string of only those Secrets that were accessed by this SA

Note: You can use jq to render json more readable, like cat data.json | jq

Solution

Step 1: Identify the Secrets

First, let's list the Opaque Secrets in the security namespace: 

➜ ssh cks3477

➜ candidate@cks3477:~# k -n security get secret | grep Opaque
kubeadmin-token       Opaque                                1      37m
mysql-admin           Opaque                                1      37m
postgres001           Opaque                                1      37m
postgres002           Opaque                                1      37m
vault-token           Opaque                                1      37m
Step 2: Analyze Audit Logs

Navigate to the audit log directory and check its size:

➜ candidate@cks3477:~# cd /opt/course/p2
➜ candidate@cks3477:/opt/course/p2$ ls -lh
audit.log

➜ candidate@cks3477:/opt/course/p2$ cat audit.log | wc -l
4448

Filter logs for ServiceAccount p.auster:

➜ candidate@cks3477:/opt/course/p2$ cat audit.log | grep "p.auster" | wc -l
28

➜ candidate@cks3477:/opt/course/p2$ cat audit.log | grep "p.auster" | grep Secret | wc -l
2

Check for specific actions:

➜ candidate@cks3477:/opt/course/p2$ cat audit.log | grep "p.auster" | grep Secret | grep list | wc -l
0

➜ candidate@cks3477:/opt/course/p2$ cat audit.log | grep "p.auster" | grep Secret | grep get | wc -l
2

Examine the specific Secret access events:

cat audit.log | grep "p.auster" | grep Secret | grep get | jq
{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "RequestResponse",
  "auditID": "74fd9e03-abea-4df1-b3d0-9cfeff9ad97a",
  "stage": "ResponseComplete",
  "requestURI": "/api/v1/namespaces/security/secrets/vault-token",
  "verb": "get",
  "user": {
    "username": "system:serviceaccount:security:p.auster",
    "uid": "29ecb107-c0e8-4f2d-816a-b16f4391999c",
    "groups": [
      "system:serviceaccounts",
      "system:serviceaccounts:security",
      "system:authenticated"
    ]
  },
  "objectRef": {
    "resource": "secrets",
    "namespace": "security",
    "name": "vault-token",
    "apiVersion": "v1"
  }
}
{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "RequestResponse",
  "auditID": "aed6caf9-5af0-4872-8f09-ad55974bb5e0",
  "stage": "ResponseComplete",
  "requestURI": "/api/v1/namespaces/security/secrets/mysql-admin",
  "verb": "get",
  "user": {
    "username": "system:serviceaccount:security:p.auster",
    "uid": "29ecb107-c0e8-4f2d-816a-b16f4391999c",
    "groups": [
      "system:serviceaccounts",
      "system:serviceaccounts:security",
      "system:authenticated"
    ]
  },
  "objectRef": {
    "resource": "secrets",
    "namespace": "security",
    "name": "mysql-admin",
    "apiVersion": "v1"
  }
}
Step 3: Update Compromised Secrets

Generate new base64-encoded passwords:

➜ candidate@cks3477:/opt/course/p2$ echo -n new-vault-pass | base64
bmV3LXZhdWx0LXBhc3M=

➜ candidate@cks3477:/opt/course/p2$ echo -n new-mysql-pass | base64
bmV3LW15c3FsLXBhc3M=

Update the compromised Secrets:

➜ candidate@cks3477:/opt/course/p2$ k -n security edit secret vault-token
➜ candidate@cks3477:/opt/course/p2$ k -n security edit secret mysql-admin
Summary of findings:
  • Identified 5 Opaque Secrets in the security namespace
  • Found 2 Secrets accessed by p.auster: vault-token and mysql-admin
  • Updated passwords for both compromised Secrets
Security Note: Audit logs can contain sensitive information. It's recommended to configure audit policies to only store metadata-level information for sensitive resources like Secrets, rather than their complete content.
Back to Questions List