CPU 100% with kswapd0 process, although no swap is needed

Question

I have this small virtual server (1GB RAM, 1 vcore), which served me well untill it suddenly started reporting 100% CPU usage because of kswapd0. Memory consumption is between 50% and 60%. I've set swappiness to 0 just to check if it releases CPU resources, but it did not. I've had this issue on 16.04, and now on 18.04 after upgrade. How to get rid of this kswapd0 CPU hog?

Answer 1

In case someone faces the same issues - the reason was malware: Multios.Coinminer.Miner.

kswapd0 was a binary file located in /root/.configrc/a/kswapd0. What you need to do is:

1. Clear crontab jobs referring to /root/.configrc
2. Clear ssh keys
3. Delete /root/.configrc

Answer 2

This is also relevant for 20.04. I caught from a Steam user adding content to the KillingFloor2 Dedicated server. Here's what it looks like.

Results of crontab -e:

Results of htop:

How to solve this problem:

1. I commented lines in the cronjob
2. Rebooted the server (CPU calmed down)
3. Deleted the .configrc folder
4. Checked again the paths of the crown and cleared it.

Answer 3

I changed swappiness from 60 to 0, but it didn't work. According to the answer from Paul, I found VIRT/RES/SHR non-zero for those CPU-hungry kswapd0 tasks, suggesting the existence of malwares.

Recent updates on similar questions also show the existence of malwares that hide as kswapd0:

1. kswapd0 is taking a lot of cpu
2. kswapd0 in ubuntu slows down the system
3. 记录被kswapd0这个挖矿病毒折磨的一天

Below is my solution. (Please use sudo, killall, rm -rf carefully)

1. Find where the fake kswapd0 is.

$ sudo find /home -name kswapd0
/home/user/.configrc5/a/kswapd0

2. Since the new server has few accounts, I double-checked that nobody is using this account, and the .configrc5 folder is relatively new (created in about one week).

$ sudo ls /home/user -al
total 224
drwxr-xr-x 18 user user   4096  四  10 23:56 .
drwxr-xr-x  8 root root   4096  四  10 20:44 ..
-rw-r--r--  1 user user    220  一  13 09:10 .bash_logout
-rw-r--r--  1 user user   3771  一  13 09:10 .bashrc
drwxrwxr-x 14 user user   4096  二  10 17:40 .cache
drwx------ 12 user user   4096  二  24 15:09 .config
drwxrwxr-x  4 user user   4096  四   4 00:48 .configrc5
...
-rw-r--r--  1 root root 131953  二  26 23:34 plm
-rw-r--r--  1 root root   6300  二  26 23:34 plm2
-rw-r--r--  1 user user    807  一  13 09:10 .profile
drwxr-xr-x  2 user user   4096  一  13 09:34 Public
drwx------  3 user user   4096  二  10 16:09 snap
drwx------  2 user user   4096  四  10 23:56 .ssh
-rw-r--r--  1 user user      0  一  13 09:51 .sudo_as_admin_successful
drwxr-xr-x  2 user user   4096  一  13 09:34 Templates
drwxr-xr-x  2 user user   4096  一  13 09:34 Videos

3. (Can skip) I thought I could kill the user account once and for all, and check that it had sudo privilege, but sudo userdel user is rejected.
4. Kill all the processes run by user

$ sudo killall -u user

5. Remove all files under .configrc5

$ sudo rm -rf /home/user/.configrc5

6. Check htop again and no strange CPU activities anymore.

Answer 4

Malware running as guest

If you have ever enabled Ubuntu's guest account and later enabled SSH, you may have malware running using your guest account.

sudo find /home -f kswapd0

It can be found in any user's home directory, such as /home/guest/.configrc/

Joining any network (conference, cafe, city) that has a compromised machine, or if you use a service like ngrok, all while having SSH open on your system leaves your computer exposed to this simple guest vulnerability.

Because privileges were never elevated (guest can't write files outside /home/guest, nor sudo without password), I was able to delete the guest account and everything is running normally.

Answer 5

I encountered the issue of kswapd0 occupying 100% CPU and looked up a lot of information, trying various solutions, most of which were not very helpful. Below is my final recommendation.

Direct Cause of High CPU Usage

kswapd0 is trying to move infrequently used program data to swap, a process that consumes extremely high CPU. This can last from a few minutes to several tens of minutes.

Indirect Cause

A certain process is requesting too much memory, causing system memory insufficiency, which triggers kswapd0 to move memory data to free up space.

Root Cause

Is the process occupying excessive memory really supposed to use this much? There are several possible reasons:

• On a computer with small memory, opening applications that require a lot of memory: for example, opening a very large number of web pages, or opening CAD, PS, virtual machines, very large files, etc.
• The program has a memory leak (or uses memory unreasonably) and will linearly and continuously request more memory.
• Infected with a virus, disguised as a kswapd0 process. It is a non-root process and can be killed with "killall -9 kswapd0".
• A Linux kernel bug that can be triggered under certain conditions (there's a thread from 2016 discussing a related bug, which was fixed in 2022).

A More Reasonable Solution?

When encountering insufficient memory, there are generally two measures to choose from:

1. Use kswapd0 to free up idle data and release more memory.
2. Kill the process that occupies the most memory.

When faced with insufficient memory, Windows generally kills the process occupying the most memory, so you might sometimes encounter OOM errors, but you won't experience system freezes similar to those caused by kswapd0.

Linux, on the other hand, considering that OOM terminating processes might cause significant data loss or corruption, defaults to using kswapd0 to free up more memory space.

However, computer hardware memory is always limited. Freeing up memory might help, or it might not. More often, it leads to system freezes with high CPU usage for up to ten minutes or more.

In the problem I encountered, I was using code for a quant strategy generated by an LLM (banbot) that was automatically compiled and executed for backtesting. Most of the time, the code it generated was fine, but occasionally, the generated code would cause memory leaks. I have now addressed this by adjusting prompts and incorporating built-in checks. However, as LLM-generated code becomes more prevalent, memory leaks due to code defects may become more common.

Therefore, I feel that when memory insufficiency is detected, killing the process that requested the most memory might also be a good option. Below is a shell script that should be configured in crontab to run every minute:

1. Check if a kswapd0 process currently exists.
2. If the kswapd0 process is not a root user, kill it directly.
3. If the kswapd0 process is a root user and its CPU usage exceeds 70%, find the process with the highest memory usage and kill it.

Before deciding to use this script, please first investigate and clarify the root cause of your problem. Only use it if you are sure it fits your scenario.

#!/bin/bash
# https://github.com/anyongjin/linux_oom_killer
# mail  anyongjin163@163.com

Function to log with timestamp
log_with_timestamp() {
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1"
}
Function to check and possibly kill kswapd0 and the highest memory usage process
check_and_kill() {
# Check if kswapd0 process exists
kswapd_pid=$(pgrep -x kswapd0)
if [ -z "$kswapd_pid" ]; then
exit 0
fi
# Check if current user is root
if [ "$EUID" -ne 0 ]; then
    # Non-root user, directly attempt to kill kswapd0 (Note: kernel processes cannot be killed, this operation may fail)
    kill -9 "$kswapd_pid" 2>/dev/null
    if [ $? -eq 0 ]; then
        log_with_timestamp "kswapd0 process killed (non-root user)."
    else
        log_with_timestamp "Failed to kill kswapd0 (likely due to permissions or kernel process)."
    fi
    return
fi

# For root user, check kswapd0's CPU usage
# Use top with shorter interval for faster but still accurate reading
# Run top for 1 second instead of 2 seconds
cpu_usage=$(top -b -n 2 -d 0.5 -p "$kswapd_pid" | tail -n 1 | awk '{print int($9)}')

if [ "$cpu_usage" -gt 10 ]; then
    log_with_timestamp "kswapd0 CPU usage is ${cpu_usage}% (threshold: 70%), $([ "$cpu_usage" -gt 70 ] && echo "taking action..." || echo "no action needed.")"
fi

if [ "$cpu_usage" -gt 70 ]; then
    # Find the process with the highest memory usage (exclude header row, using ps aux --sort=-%mem)
    max_mem_pid=$(ps aux --sort=-%mem | awk 'NR==2 {print $2}')
    if [ -n "$max_mem_pid" ]; then
        kill -9 "$max_mem_pid"
        log_with_timestamp "Killed process $max_mem_pid with highest memory usage."
    else
        log_with_timestamp "No processes found to kill."
    fi
fi
}
Main script logic
Check for the correct number of arguments
if [ "$#" -ne 2 ]; then
log_with_timestamp "Usage: $0 <number_of_runs> <interval_in_seconds>"
exit 1
fi
Assign command line arguments to variables
runs=$1
interval=$2
Loop to run the check_and_kill function the specified number of times
for (( i=1; i<=$runs; i++ )); do
check_and_kill
# Sleep for the specified interval, unless it's the last run
if [ $i -lt $runs ]; then
sleep "$interval"
fi
done

Grant execute permissions: chmod +x oom_killer.sh

Then, configure this script to run every 10 seconds in crontab -e (start once every minute, then execute 6 checks at 9-second intervals, with each check taking 1 second).

* * * * * /root/oom_killer.sh 6 9 >> /var/log/oom_killer.log 2>&1

To make it as convenient as possible to use, I have written an automatic installation script. You can execute the following command:

curl -sL https://banbot.site/set_oom_killer.sh | sudo bash

Alternatively, you can visit the github repository for more detailed information.