DFIR Blue Team Tips— Finding Evil Process In Linux OS
Linux OS — Finding Evil Process
Inspired from the Sans Poster: Find Evil — Know Normal — i.e Knowing what’s normal running process on a Windows OS host helps cut through the noise to quickly locate potential malware.
Refer: https://www.sans.org/security-resources/posters/hunt-evil/35/download
So I thought of implementing the same in the Linux OS side.
Though the windows OS is more targeted by APT other threat groups( as it’s mostly used one above 80%).
But in recent years, there was many APT etc groups are targeting the Linux OS platforms as well.
Reference: Intezer’s APT chart
https://www.intezer.com/blog/cloud-security/looking-back-on-the-last-decade-of-linux-apt-attacks/
As Blue teamer we should be knowing the common OS process and also identifying process masquerading, hidden process(explained in detail later in upcoming blogs.) etc.
Challenges
Linux has different OS distributions -Suse, OpenSUSE, Debian, Ubuntu, CentOS, Arch, Fedora, RHEL, AWS Linux all are common Linux distribution names.Some will be have same commands, only kernel variations etc.
Linux OS distribution used in this blog: AWS Linux.
Process chart: (For easy understanding)
Notes:
1) [kthreadd] thread process will have many child process like [migration/0] ….. [crypto] etc with random PID 3–16,30,174–184 etc.
2) Image Path: these kernel thread process will not have any path/elf file associated with them, if there is any path associated then it’s a suspicious one, could be a process masquerading attempt.
3) Systemd or init is the parent process for those circular process structure (above screenshot)
Process details & description:
Note: sha256sum is subject to change for different OS versions.
1) Init or Systemd
Image Path: /usr/lib/systemd/systemd — switched-root — system — deserialize 21 (systemd)
Parent Process: None
Number of Instances: 1
Process ID: 1
Parent process ID: 0
Session ID: 1
User Account: root
Start Time: Within few min of system boot
sha256sum:8dcedc65c32ff5e149343015798c7613254ff1659e133e8a6f51725bdf1afd2e (aws linux)
Description:
As per wiki: Thus, the kernel initializes devices, mounts the root filesystem specified by the boot loader as read only, and runs Init (/sbin/init) which is designated as the first process run by the system (PID = 1)
init is a abbreviation for Initialization. The init is a daemon process which starts as soon as the computer starts and continue running till, it is shutdown. In-fact init is the first process that starts when a computer boots, making it the parent of all other running processes directly or indirectly and hence typically it is assigned “pid=1“.
A systemd is a System Management Daemon named with UNIX convention to add ‘d‘ at the end of daemon. So, that they can be easily recognized. Initially it was released under GNU General Public License, but now the releases are made under GNU Lesser General Public License. Similar to init, systemd is the parent of all other processes directly or indirectly and is the first process that starts at boot hence typically assigned a “pid=1“.
Systemd has many advantages from the init (faster boot, parallel process etc)
A systemd, may refer to all the packages, utilities and libraries around daemon. It was designed to overcome the shortcomings of init. It itself is a background processes which is designed to start processes in parallel, thus reducing the boot time and computational overhead. It has a lot other features as compared to init.
Some legacy features were kept intact (such as init.d to some degree)
2) Kernel thread process — which will have a brackers [“process name”] ex: [kthreadd]
Kthreadd is a worker thread in kernel address space started by the kernel.
A kernel thread is a kernel task running only in kernel mode; it usually has not been created by fork() or clone() system calls.
Image Path: these kernel thread process will not have any path associated with them, if there is any path associated then it’s a suspicious one, could be a process masquerading attempt.
One of the way to find the kernel threads is running this command “strings /proc/$pid/cmdline” is empty for kernel threads.
To augment the need for running background operations, the kernel spawns threads (similar to processes). These kernel threads are similar to regular processes, in that they are represented by a task structure and assigned a PID. Unlike user processes, they do not have any address space mapped, and run exclusively in kernel mode, which makes them non-interactive. Various kernel subsystems use kthreads to run periodic and asynchronous operations.
The kthreadd process is created by idle through kernel_thread, and always runs in the kernel space, responsible for the scheduling and management of all kernel threads
[kthreadd] is the parent process for all the kernel threads.
Image Path: N/A
Parent Process: None
Number of Instances: 1
Process ID: 2
Parent process ID: 0
Session ID: 0
User Account: root
Start Time: Within few min of system boot
Sha256sum: N/A (because no path/elf associated with this)
Description:
[kthreadd] It will have many child process like [migration/0] etc with random PID 3–16,30,174–184 etc
Ex: [migration/0] is part of the kernel, and is a thread that is responsible for moving processes to another CPU to balance out the workload.
[crypto] — It provides access to the kernel crypto API designed to handle “transformations” of data between “states” (encrypted and unencrypted).It is where cryptographic functions live that are used by kernel modules.Kernel modules are kernel processes that manage your hardware, or filesystems. Most filesystems that use cryptographic functions, use the kernel crypto functions.This is for some security software that lets the Linux kernel manage keys for encryption and decryption, and it will get increasingly relevant as trusted computing evolves and computer hardware can fulfil more and more security functions.
[kworker/0:0] — is a placeholder process for kernel worker threads, which perform most of the actual processing for the kernel, especially in cases where there are interrupts, timers, I/O, etc.
Kernel threads are not children of init or systemd because they can be started before all the userspace processes.
They are typically used to manage hardware that’s why they are directly handled by the kernel and have high priority.
For a process to be child of init it needs to be cloned from init and Kthreads aren’t that’s why their parent PID is 0 meaning “no-one”.
Credits:
https://stackoverflow.com/questions/17988526/what-is-kthreadd-process-and-children-and-how-it-is-different-from-init-and-chil
A kernel thread is a kernel entity, like processes and interrupt handlers; it is the entity handled by the system scheduler. A kernel thread runs within a process, but can be referenced by any other thread in the system. The programmer has no direct control over these threads, unless you are writing kernel extensions or device drivers.
3) systemd-journald
Image Path: /usr/lib/systemd/systemd-journald
Parent Process: Systemd
Number of Instances: 1
Process ID: subject to change
Parent process ID: 1
User Account: root
Start Time: Within few min of system boot
Sha256sum:6040be31c23b5d45ab5ba168499abfb699330b4b3643785578d3e1e23bbcace0 (aws linux)
Description:
systemd-journald is a system service that collects and stores logging data
4)lvmetad -f
Image Path: /usr/sbin/lvmetad -f
Parent Process: Systemd
Number of Instances: 1
Process ID: subject to change
Parent process ID: 1
User Account: root
Start Time: Within few min of system boot
Sha256sum:c9cc0b4648bb5ae7b18225a76d20e814ed951080b0b8eefacaab6f8ba3f03735
Description:
lvmetad is a metadata caching daemon for LVM. The daemon receives notifications from udev rules (which must be installed for LVM to work correctly when lvmetad is in use). Through these notifications, lvmetad has an up-to-date and consistent image of the volume groups available in the system. By default, lvmetad, even if running, is not used by LVM. See lvm.conf(5).
5) systemd-udevd
Image Path: /usr/lib/systemd/systemd-udevd
Parent Process: Systemd
Number of Instances: 1
Process ID: subject to change
Parent process ID: 1
User Account: root
Start Time: Within few min of system boot
sha256sum:30c86e74ca507f246f532a66a72edf693621ac9b9ad5bca6fc2442d38f90b38a
Description:
systemd-udevd listens to kernel uevents. For every event, systemd-udevd executes matching instructions specified in udev rules. The behavior of the daemon can be configured using udev. conf
6) auditd
Image Path: /sbin/auditd
Parent Process: Systemd
Number of Instances: 1
Process ID: subject to change
Parent process ID: 1
User Account: root
Start Time: Within few min of system boot
sha256sum: c8f2e9de73129a5a6ccbfaf1782884e8aac0943dd40443afc73dca5183b3d849Description:
auditd is the userspace component to the Linux Auditing System.It’s responsible for writing audit records to the disk. Viewing the logs is done with the ausearch or aureport utilities.Configuring the audit system or loading rules is done with the auditctl utility. During startup, the rules in /etc/audit/audit.rules
The Linux Auditing System provides kernel-resident logging of system calls and user space tools to collect and view the logs. The auditd daemon writes the logging records to disk. auditd is configurable to allow control over what information is written to the logs.
7) lsmd -d
Image Path: /usr/bin/lsmd -d
Parent Process: Systemd
Number of Instances: 1
Process ID: subject to change
Parent process ID: 1
User Account: libstor+
Start Time: Within few min of system boot
sha256sum:58a1eb48c1cdd5c31d448d5933eaf088642b05d7e605cdcf3c8b771a68c8be75
Description:
The daemon (lsmd) of libstoragemgmt require to check existence of ‘libstoragemgmt’ user and group before droping privilege to it.The libStorageMgmt plugin daemon (lsmd) will read the lsmd.conf file in the folder defined via — confdir (default is /etc/lsm/).A lightweight service management daemon for linuxi based on System V Init.
Features:
Managing services by CLI.
Monitoring and do define action by service status.
Grouping service by type (system, applications, network, user, database).
Possibility of communication between servers (basic service HA feature) — sending status of service and making scenario.
Creating instances of services eg. you can create a several instances of httpd for several users.
Allow user to create services and instances of application (branch in /applications) with user privileges (running services as user).
8) dbus
Image Path: /usr/bin/dbus-daemon — system — address=systemd: — nofork — nopidfile — systemd-activation
Parent Process: Systemd
Number of Instances: 1
Process ID: subject to change
Parent process ID: 1
User Account: dbus
Start Time: Within few min of system boot
sha256sum: 2928d0edf4abcc73cec7468932e7a3647a6153e4842fa2e7a136d00616920e6b
Description:
DBus system usually has two buses: a system bus and a session bus.Session bus is started per user (in your case for root and xralf), lines 3 to 6. Line 2 is a dbus service that was requested by your window manager.A system bus is needed for system-wide message exchange.
dbus-daemon is the D-Bus message bus daemon. See http://www.freedesktop.org/software/dbus/ for more information about the big picture. D-Bus is first a library that provides one-to-one communication between any two applications; dbus-daemon is an application that uses this library to implement a message bus daemon. Multiple programs connect to the message bus daemon and can exchange messages with one another.
There are two standard message bus instances: the systemwide message bus (installed on many systems as the “messagebus” init service) and the per-user-login-session message bus (started each time a user logs in). dbus-daemon is used for both of these instances, but with a different configuration file.
9) systemd-logind
Image Path: /usr/lib/systemd/systemd-logind
Parent Process: Systemd
Number of Instances: 1
Process ID: subject to change
Parent process ID: 1
User Account: root
Start Time: Within few min of system boot
sha256sum: f479984b2276bff417692e059228a39a0d9ac50b345916f6195b487d117f2504
Description:
systemd-logind is a system service that manages user logins. It is responsible for:
• Keeping track of users and sessions, their processes and their idle state
• Providing PolicyKit-based access for users to operations such as system shutdown or sleep
• Implementing a shutdown/sleep inhibition logic for applications
• Handling of power/sleep hardware keys
• Multi-seat management
• Session switch management
• Device access management for users
• Automatic spawning of text logins (gettys) on virtual console activation and user runtime directory management
10) rngd
Image Path: /sbin/rngd -f — fill-watermark=0 — exclude=jitter
Parent Process: systemd
Number of Instances: 1
Process ID: subject to change
Parent process ID: 1
User Account: rngd
Start Time: Within few min of system boot
sha256sum: 6b3226f920a4059d8aa1d83ca4fa41b22374ea9f0949f3395984f8d0f5334b28
Description:This daemon feeds data from a random number generator to the kernel’s random number entropy pool, after first checking the data to ensure that it is properly random.
11) gssproxy
Image Path: /usr/sbin/gssproxy -D
Parent Process: systemd
Number of Instances: 1
Process ID: subject to change
Parent process ID: 1
User Account: root
Start Time: Within few min of system boot
sha256sum: e2db3d1bf22fdfbe63f2d78e31321710871af11edc685ede33bdcb35904acf8aDescription:
12) chronyd
Image Path: /usr/sbin/chronyd
Parent Process: systemd
Number of Instances: 1
Process ID:subject to change
Parent process ID: 1
User Account: chrony
Start Time: Within few min of system boot
sha256sum: 9ee5d51bd7b1ef35159ddb35947e9244e3682a79c16a0231d96b0d03af0e4565
Description:
Chrony is an implementation of the Network Time Protocol (NTP). You can use Chrony:
to synchronize the system clock with NTP servers,
to synchronize the system clock with a reference clock, for example a GPS receiver,
to synchronize the system clock with a manual time input,
as an NTPv4(RFC 5905) server or peer to provide a time service to other computers in the network.
13) dhclient
Image Path:/sbin/dhclient -q -lf /var/lib/dhclient/dhclient — eth0.lease -pf /var/run/dhclient-eth0.pid eth0
&
sbin/dhclient -6 -nw -lf /var/lib/dhclient/dhclient6 — eth0.lease -pf /var/run/dhclient6-eth0.pid eth0
Parent Process: systemd
Number of Instances: 1
Process ID: subject to change
Parent process ID: 1
User Account: root
Start Time: Within few min of system boot
sha256sum:a73d6a147a85b0de2100863523f1c2af73a0b4fc834cdf0eea24c8e0a21bf0b9
Description:
DHCP Client https://medium.com/swlh/setting-up-a-basic-dhcp-client-and-server-in-linux-9005457df607
14) postfix master
Image Path: /usr/libexec/postfix/master -w
Parent Process: systemd
Number of Instances: 1
Process ID: subject to change
Parent process ID: 1
User Account:postfix
Start Time: Within few min of system boot
sha256sum: 8211b07ffcecf987151fcfe8fee1c7d94683197e1d07c9b4170a199442c02e6bDescription:
The Postfix master daemon /usr/lib/postfix/master is started directly by Postfix when the system is booted and is terminated only when the system goes down or if Postfix ends.
The Postfix master daemon is normally configured once only when as the email system is set up, and is usually never changed.
The master daemon, which monitors the entire mail system.
Refer: https://www.halolinux.us/migrating-from-redhat-to-suse/configure-the-postfix-master-daemon.html
15) postfix pickup
Image Path: pickup -l -t unix -u
Parent Process: systemd
Number of Instances: 1
Process ID: subject to change
Parent process ID: postfix master
User Account: postfix
Start Time: Within few min of system boot
sha256sum:a0095760f143f767787b5052c21e8924ed8651b0e0416202427e6847956336b6
Description:
The pickup daemon is basically a process which watches a directory (maildrop) where other processes place files. It doesn’t talk with the outside world (like smtpd or others), it just waits for a process like postdrop
16) postfix qmgr
Image Path: qmgr -l -t unix -u
Parent Process: systemd
Number of Instances: 1
Process ID: subject to change
Parent process ID: postfix master
User Account: postfix
Start Time: Within few min of system boot
sha256sum: adf673f2077e6c850b9f0d922a331048625f21868ebab8d17cb6ce4727271342
Description:
The qmgr daemon awaits the arrival of incoming mail and arranges for its delivery via
Postfix delivery processes. The actual mail routing strategy is delegated to the
trivial-rewrite daemon. This program expects to be run from the master process manager.
17) rsyslogd
Image Path: /usr/sbin/rsyslogd -n
Parent Process: systemd
Number of Instances: 1
Process ID: subject to change
Parent process ID: 1
User Account: root
Start Time: Within few min of system boot
sha256sum: 27cee57613fa7b3a9e6260e3f793c7a0141289c72ffa4acff2c57898155df2f8
Description: Rsyslogd is a system utility providing support for message logging. Support of both internet and unix domain sockets enables this utility to support both local and remote logging.
The main rsyslog configuration file is located at /etc/rsyslog. conf, which loads modules, defines the global directives, contains rules for processing log messages and it also includes all config files in /etc/rsyslog.
18)AWS SSM Agent
Image Path: /usr/bin/amazon-ssm-agent
Parent Process: systemd
Number of Instances: 1
Process ID: subject to change
Parent process ID: 1
User Account: root
Start Time: Within few min of system boot
sha256sum: 975084132b5e5bb8a7a033be585a7f65d1ac04ddbf9c274c9b57b1c5d3bbf55c
Description:
AWS Agent process — Starting with version 2.3.50.0 of SSM Agent, the agent creates a local user account called ssm-user and adds it to the /etc/sudoers.d directory (Linux and macOS) or to the Administrators group (Windows Server). On agent versions before 2.3.612.0, the account is created the first time SSM Agent starts or restarts after installation. On version 2.3.612.0 and later, the ssm-user account is created the first time a session is started on an instance
19) atd
Image Path:/usr/sbin/atd -f
Parent Process: systemd
Number of Instances: 1
Process ID: subject to change
Parent process ID: 1
User Account: root
Start Time: Within few min of system boot
sha256sum:d065a34ef6b931de8c77a086fcd713136a36a88bd1705e0567a905aca25132ab
Description:
atd is a job scheduler daemon that runs jobs scheduled for later execution. These jobs are one-time task(not recurring) at a specific time scheduled using ‘at’ or ‘batch’ utility.
20)crond -n
Image Path: /usr/sbin/crond -n
Parent Process: systemd
Number of Instances: 1
Process ID: subject to change
Parent process ID: 1
User Account: root
Start Time: Within few min of system boot
sha256sum:e9aa913ca46ce5ea058f618ef420422af2862ca81d53d8c88407a26dce4667b0
Description:
crond is responsible for scanning the crontab files and running their commands at the appropriate time. It always synchronizes to the top of the minute, matching the current time against its internal list of parsed crontabs. That list is stored so that it can be scanned very quickly, and crond can deal with several hundred crontabs with several thousand entries without using noticeable CPU.
Cron jobs are not re-executed if a previous instance of them is still running. For example, if you have a crontab command sleep 70, that you request to be run every minute, crond will skip this job when it sees it is still running. So the job won’t be run more frequently than once every two minutes. If you do not like this feature, you can run your commands in the background with an &.
crond automatically detects when the clock has been changed, during its per-minute scans.
21) agetty
Image Path: /sbin/agetty — noclear tty1 linux & /sbin/agetty — keep-baud 115200,38400,9600 ttyS0 vt220
Parent Process: systemd
Number of Instances: 1
Process ID: subject to change
TTY: tty1 & ttys0
Parent process ID: 1
User Account: root
Start Time: Within few min of system boot
sha256sum: b74d9ba6a01c3dd2c81960c523f040fc83eaf31605d9f41144a87da72ef37dafDescription:
agetty is a Linux version of getty. getty short for “get tty” is a Unix program running on a host computer that manages physical or virtual terminals to allow multi-user access.
https://security.stackexchange.com/questions/92286/strange-agetty-process-running-on-my-vps/92291
22) AWS ssm-agent-worker
Image Path: /usr/bin/ssm-agent-worker
Parent Process: amazon-ssm-agent
Number of Instances: 1
Process ID: subject to change
Parent process ID: amazon-ssm-agent
User Account: root
Start Time: Within few min of system boot
sha256sum: 0d490a6b580d86615c5ea7a126c41a396efa34e8b18762a6fcf820c4cf6f4a7bDescription:
AWS SSM Agent process
When you update to version 3.0, the system renames amazon-ssm-agent on your managed instance to ssm-agent-worker
yum info amazon-ssm-agent to get the version
23) /usr/bin/ssm-session-worker
Image Path: /usr/bin/ssm-session-worker
Parent Process: /usr/bin/ssm-agent-worker
Number of Instances: multiple based on the logged ssm session
Process ID: subject to change
Parent process ID: /usr/bin/ssm-session-worker
User Account: root
Start Time: Within few min of system boot
sha256sum:391a14827a342cf4b991294b21c80b1d7cc71dc825ba9daf5bfb58949aa7a510
Description:
AWS SSM session manager process
Each /usr/bin/ssm-session-worker will have a child process called sh, TTY: pts/0
24) sshd -D
Image Path:/usr/sbin/sshd -D
Parent Process: systemd
Number of Instances: A new process is created for each new SSH session.
Process ID: Subject to change
Parent process ID: 1
User Account: root
Start Time: Within few min of system boot
sha256sum: d76838f8e55510c49761612618e89b0cecbeeec54863450afff15aa440bb234Description:
sshd is the OpenSSH server process. It listens to incoming connections using the SSH protocol and acts as the server for the protocol. It handles user authentication, encryption, terminal connections, file transfers, and tunneling
It is also the parent process of all the other sshd processes
The other sshd processes are child processes that serve a single connection. A new process is created for each new SSH session.
If the SSH server is upgraded or restarted, only the master server is generally restarte
On most systems, syslog is configured to log SSH-related messages by default into files under /var/log/. On Debian-derived systems, the default log file is usually /var/log/auth.log. On Red Hat derived systems, the default log file is usually /var/log/secure.
Finding Evil Tips:
1)These kernel thread process ( [“ “]) will not have any path, binaries associated with them and PPID will be zero, if there is any path,binaries associated then it’s a suspicious one, could be a process masquerading attempt, ex:
threat actor named Rocke has used this technique.
Refer: Craig H Rowland’s blog on detecting process masquerading
https://www.sandflysecurity.com/blog/detecting-linux-kernel-process-masquerading-with-command-line-forensics/
https://redcanary.com/blog/rocke-cryptominer/
2) Attackers would name the process in a way to camouflage or blend in plain sight.
Always check commandline, running from unusual paths like /tmp etc, arguments make sense, is the parent process expected on, processes started near time of attack, process name spelled incorrectly, whether it was started by the expected user, unknown process consuming high CPU resource(could be a miner), unsual network connection (netstat) and also check process memory map ( This location is normally where processes show libraries they are linking to and where they mapped to in memory)
About /proc directory, why it’s useful?
procfs is a virtual file system ( It doesn’t really exist) which provides access to kernel structures. procfs provides information about processes, networking, devices, processors, memory, and other system statistics.
It will be really useful for forensics and incident-response.
Note: When your taking full disk image, the proc directory will not be copied, as mentioned it’s a virtual file system.
Some of the useful proc commands for DFIR:
i)Process directory listing
cd /proc/$pid/ and do ls -al
ii)Process cmdline Parameters
cd /proc/$pid/ then do
cat comm — exposes the process’s comm value — that is, the command name associated with the process
strings cmdline — holds the complete command line for the process, unless the process is a zombie
iii)Process path:
ls -al /proc/<PID>/exe
iv)Process Maps — This can be used to see which libraries a process is using.
cat /proc/<PID>/maps.
v)Process Environment variables
strings /proc/<PID>/environ
vi)File descriptors(Similar to lsof)- see what files the process has opened/handle
ls -al /proc/<PID>/fd
vii)Status of the process
/proc/PID/stat and /proc/PID/status
viii)I/O statistics for the process,
cat /proc/PID/io
ix) This is a symbolic link to the process. you can copy these if the file has been deleted.
cp /proc/PID/exe /tmp/test
x)Current working directory:
ls -alR /proc/PID/cwd
xi)Netstat with PID info
netstat -plant
xii)Top usage of CPU/RAM:
top
xiii)Process tree
ps -auxwf
xiv)Process hash (sha256sum)
sha256sum /proc/$pid/exe
3) Fileless attacks are increasing -, injecting malicious payloads into benign system processes, using lolbins for process execution, hide in memory and hide process using rootkit techniques
— We can use volatility to detect the hidden process, injected process (will create a separate blog).
Refer: https://www.sandflysecurity.com/blog/detecting-linux-memfd_create-fileless-malware-with-command-line-forensics/
4) Always check the hash of the malicious/running process with the Virustotal as well. I know that Malwares can evade AV/EDRs/Sandbox as well. So in addition to the VT, Sandbox, It’s always better to do a malware analysis of the ELF executables, as well.(will create a separate blog on the ELF, Malware analysis etc).
5) Check if the process is packed or not. Some malware for Linux is packed or encrypted and shows very high entropy.
Refer: https://www.sandflysecurity.com/blog/sandfly-filescan-open-source-file-entropy-scanner-for-linux/
6) Sometimes malware will delete itself after it starts, check for the deleted binaries that are still running.
using : ls -alR /proc/*/exe 2> /dev/null | grep deleted ( watch out for False positives)
Please comment your thoughts or if I missed anything etc.
Special thanks to Craig H Rowland for the support.
Please follow for more updates.
