Suspicious process: /tmp/init -c /tmp/init12.cfg

[khitai]

Soon after the 2020 new year's holiday, I found a suspicious process on my web server:

11777 ? Ssl 30038:48 /tmp/init -c /tmp/init12.cfg

It was using almost 100% cpu, and it was now shown in "netstat -plnt" output.

> ls -l /tmp/init
-rwxr-xr-x 1 root root 902084 Jul 16 2015 /tmp/init

> file /tmp/init
/tmp/init: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, stripped

These was another file in /tmp:

> ls -l javax
-rwxr-xr-x 1 root root 3 Jan 19 01:47 javax
> file javax
javax: ASCII text, with no line terminators
> cat javax
748

And there were a bunch of immutable empty directories in /tmp:

Demon.x86
NoIr_x.86
Ouija_x.86
Yui.x86
a
a_thk
ai.x86
baby
config.json
cross.sh
efjins
evolutions
httpdz
ini
initdz
kerberods
kh
kionai
ksoftirqds
kthrotlds
kworkerds
lilpip
log_rot
mcoin
mcoin-ankit
migrations
networkservice
php
pvds
pvds2
pvds3
r.sh
racks_s
rogue_s
rzx
seasame
skfednw46d.mips
skfednw46d.x86
sqlcan
syscb
sysguard
sysnpmc
sysr.sh
sysupdate
update.sh
watchbog
watchdogs
x86
x86_64
xdsf
xfsalloc
xmrig_s

They all belonged to root user and had a mode of r-xr-xr-x. I had to run "chattr -i -a" and "chmod 755" on them before removing them.

I killed the /tmp/init process and removed all the above files and directories in /tmp, and things seemed to be ok. But today, I found that exactly the same thing happened again, a "/tmp/init -c /tmp/init12.cfg" process using 99% to 100% cpu, and seemingly the same immutable empty directories in /tmp.

Here is the "netstat -plnt" output of my nginx web server:

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 6519/master
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 21043/nginx
tcp 0 0 0.0.0.0:26725 0.0.0.0:* LISTEN 17922/sshd
tcp 0 0 0.0.0.0:18983 0.0.0.0:* LISTEN 1336/java
tcp 0 0 127.0.0.1:9000 0.0.0.0:* LISTEN 2340/php-fpm.conf)
tcp 0 0 127.0.0.1:9001 0.0.0.0:* LISTEN 2340/php-fpm.conf)
tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN 26649/mysqld
tcp 0 0 127.0.0.1:7983 0.0.0.0:* LISTEN 1336/java
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 21043/nginx
tcp 0 0 0.0.0.0:41490 0.0.0.0:* LISTEN 1336/java
tcp 0 0 0.0.0.0:8983 0.0.0.0:* LISTEN 1336/java
tcp6 0 0 :::26725 :::* LISTEN 17922/sshd

This is Ubuntu 14.04.5 LTS x86_64 with kernel 3.13.0-45-generic, nginx 1.4.6-1ubuntu3.5, php 5.6.23, openssl 1.0.2h, openjdk-7-jre 7u111-2.6.7-0ubuntu0.14.04.3.

Any idea about the root cause or security hole, the hacking method used, measures for security hardening is apreciated, thanks!

 

[Rob]

Hey there @khitai - welcome to linux.org!

Well, i'd say there's something going on there.. first thing i would do is re-mount /tmp so executables can't be run from there with:

mount -o remount,noexec /tmp

Then, in your /etc/fstab, add the noexec to the /tmp line with something like:

/dev/sda4       /tmp              ext4  noexec,nosuid           0       2

Then, delete everything in /tmp

I'd look at all of the files in the documentroot of your site to make sure only your files exist - someone got on there and put their code on your system.

Next, start setting up a new server with updated software - what software does your website use? (wordpress, etc..)

You could also try something like cxs (configserver.com) to look for malicious files on your server.. or a 'rootkit hunter' type script until you have your new server set up.

 

[khitai]

@Rob

Thank you for your advices! About /tmp, yes, you are totally right, I should remount it with noexec,nosuid. And I am now also monitoring /tmp using auditd. I checked all the sites configured in nginx, didn't find any unknown file.

Actually I may have found the cause: there is a solr service running on my webserver, and it has https://www.tenable.com/cve/CVE-2019-12409. The fix is here.

 

[Rob]

Ok - sounds good.. Also, i was going to say if you looked at the init12.cfg file, it'd prob give some info about what it was doing since it was used as a config looks like.