Question 14: Syscall Activity

Problem Statement

Solve this question on: ssh cks7262

There are Pods in Namespace team-yellow. A security investigation noticed that some processes running in these Pods are using the Syscall kill, which is forbidden by an internal policy of Team Yellow.

Tasks:

  1. Find the offending Pod(s) that are using the forbidden kill syscall
  2. Remove these by reducing the replicas of the parent Deployment to 0
You can connect to the worker nodes using ssh cks7262-node1 and ssh cks7262-node2 from cks7262.

Solution

Step 1: Check Pod Distribution

First, let's check which nodes the Pods are running on:

➜ ssh cks7262

➜ candidate@cks7262:~# k -n team-yellow get pod -owide
NAME                             ...       NODE               NOMINATED NODE   ...
collector1-8d9dbc99f-hswfn       ...       cks7262-node1              
collector1-8d9dbc99f-kwjtf       ...       cks7262-node1              
collector2-66547ddfb5-5mvtz      ...       cks7262-node1              
collector3-6ffb899c79-kwcxv      ...       cks7262-node1              
collector3-6ffb899c79-lxm79      ...       cks7262-node1

All Pods are running on cks7262-node1, so we'll focus our investigation there.

Step 2: Investigate collector1 Processes

Let's check the processes for the first Deployment collector1:

➜ candidate@cks7262:~# ssh cks7262-node1

➜ candidate@cks7262-node1:~# sudo -i

➜ root@cks7262-node1:~# crictl pods --name collector1
POD ID              CREATED             STATE        NAME                         ...
a61e29997e607       17 minutes ago      Ready        collector1-8d9dbc99f-kwjtf   ...
8b0c315bf5ccd       17 minutes ago      Ready        collector1-8d9dbc99f-hswfn   ...

➜ root@cks7262-node1:~# crictl ps --pod a61e29997e607
CONTAINER ID        IMAGE            ...    POD ID          POD
e18e766d288ac       71136cb0add32    ...    a61e29997e607   collector1-8d9dbc99f-kwjtf

➜ root@cks7262-node1:~# crictl inspect e18e766d288ac | grep args -A1
        "args": [
          "./collector1-process"

Let's check the process PIDs:

➜ root@cks7262-node1:~# ps aux | grep collector1-process
root       13980  0.0  0.0 702216   384 ?   ...    ./collector1-process
root       14079  0.0  0.0 702216   512 ?   ...    ./collector1-process
Step 3: Check Syscalls for collector1

Using strace to monitor syscalls:

➜ root@cks7262-node1:~# strace -p 14079
strace: Process 14079 attached
epoll_pwait(3, [], 128, 529, NULL, 1)   = 0
epoll_pwait(3, [], 128, 995, NULL, 1)   = 0
epoll_pwait(3, [], 128, 999, NULL, 1)   = 0
...
futex(0x4d7e68, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
kill(666, SIGTERM)                      = -1 ESRCH (No such process)
futex(0x4d7e68, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
kill(666, SIGTERM)                      = -1 ESRCH (No such process)
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1, si_uid=0} ---

Found it! The collector1 process is using the forbidden kill syscall.

Step 4: Verify Other Deployments

Let's check collector2 and collector3 to ensure they're not using the forbidden syscall:

➜ root@cks7262-node1:~# ps aux | grep collector2-process
root       14046  0.0  0.0 702224   512 ?        Ssl  10:55   0:00 ./collector2-process

➜ root@cks7262-node1:~# strace -p 14046
strace: Process 14046 attached
futex(0x4d9e68, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x4d9e68, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
epoll_pwait(3, [], 128, 999, NULL, 1)   = 0
...
➜ root@cks7262-node1:~# ps aux | grep collector3-process
root       14013  0.0  0.0 702480   640 ?        Ssl  10:55   0:00 ./collector3-process
root       14216  0.0  0.0 702480   640 ?        Ssl  10:55   0:00 ./collector3-process

➜ root@cks7262-node1:~# strace -p 14013
strace: Process 14013 attached
epoll_pwait(3, [], 128, 762, NULL, 1)   = 0
epoll_pwait(3, [], 128, 999, NULL, 1)   = 0
...

Both collector2 and collector3 are clean - they don't use the kill syscall.

Step 5: Scale Down Offending Deployment

Scale down the collector1 Deployment to 0 replicas:

➜ root@cks7262:~# k -n team-yellow scale deploy collector1 --replicas 0
Summary of findings:
  • Found that collector1 processes were using the forbidden kill syscall
  • Verified that collector2 and collector3 were not using the forbidden syscall
  • Successfully scaled down the collector1 Deployment to 0 replicas
Security Note: Syscalls are used by processes running in Userspace to communicate with the Linux Kernel. It's important to restrict syscalls for container processes using tools like Seccomp or AppArmor to enhance security.
Back to Questions List