Debian Webserver keeps getting hacked

[z0q]

Hi everyone,

Since a few months our webserver keeps getting hacked. When this happens a lot of files are added or modified in our webroot. These

files and modifications include mailbots. This results in our webserver sending spam mails and the exim mail queue fills up. Another

thing that happens after an attack is that we cannot create new files, because all the inodes are being used. After every attack we clean

the server and it works again, but it is, of course, very annoying.

We have already tried multiple things. Such as:
- Change the CHMOD of all folders in the webroot to 775 and all the files to 664.
- Run a virusscan with ClamAV
- Run RKHunter
- Install Apache Mod Security, but we have disabled it again, because some of our websites did not work properly anymore with it
- Empty the mail queue and delete infected files
- Installing SFTP instead of FTP
- Block Root access via SSH
- Upgrade the server
- Restart the server
- And more

Is anyone experienced in this area? Or does anyone know how we can secure our server?

We would really appreciate your help .

Sincerely,

Z0q

 

[unixfish]

Have you updated Debian / ssh? This sounds like it could be a shellshock issue, although you have mentioned some things that is probably something else.

It sounds like there is an open port or a default password that someone is leveraging. You may want to install a firewall and lock the site down as well.

I don't know of an "easy" answer here - this is kind of like saying "my car won't start, how do I fix it?"

We need a bit more info: What version of Debian, what patches are applied, what version of Apache, what applications is Apache running, etc. There are some vulnerabilities with some languages running over Apache as CGI, etc.

Help us narrow this down a bit and we can probably get you in the right ballpark.

 

[z0q]

Have you updated Debian / ssh? This sounds like it could be a shellshock issue, although you have mentioned some things that is probably something else.

It sounds like there is an open port or a default password that someone is leveraging. You may want to install a firewall and lock the site down as well.

I don't know of an "easy" answer here - this is kind of like saying "my car won't start, how do I fix it?"

We need a bit more info: What version of Debian, what patches are applied, what version of Apache, what applications is Apache running, etc. There are some vulnerabilities with some languages running over Apache as CGI, etc.

Help us narrow this down a bit and we can probably get you in the right ballpark.

Thank you for your quick reply.

The version of Debian is:
Debian GNU/Linux 7.8 (wheezy)

The version of Apache is:
Apache/2.2.22 (Debian)

Application of Apache:
Web Applications using PHP/MySQL. Some websites use Wordpress. I've read that often there are leaks within Wordpress. But if the webserver is secure from itself, then the websites running on it shouldn't be the problem, right? I think that on a proper Webhosting it shouldn't matter what kind of website a user uploads. The server should be secure enough and not crash from it, right?

 

[unixfish]

Not necessarily. The server can be secure, but if there is an issue in the web application, crackers can still get in.

Think of locking your car and leaving a key in a magnetic box under it. If someone knows that magnetic box is there and it holds a key, it's not really secure.

Have you set a password on the root account, changed the default passwords on MySQL, and tested for shellshock (env x='() { :;}; echo vulnerable' bash -c "echo this is a test")? That may require some re-config and patching. You will also want to search for security issues with your web site / version, and consider whether you need to upgrade / firewall / update. Also google for PHP application vulnerabilities.

I had to upgrade an Ubuntu / Debian based web server because Shellshock was allowing crackers to use this server as a spamming jump point. Spending too much time blacklisting - had to bite the bullet and do the full upgrade.

 

[z0q]

Not necessarily. The server can be secure, but if there is an issue in the web application, crackers can still get in.

Think of locking your car and leaving a key in a magnetic box under it. If someone knows that magnetic box is there and it holds a key, it's not really secure.

Have you set a password on the root account, changed the default passwords on MySQL, and tested for shellshock (env x='() { :;}; echo vulnerable' bash -c "echo this is a test")? That may require some re-config and patching. You will also want to search for security issues with your web site / version, and consider whether you need to upgrade / firewall / update. Also google for PHP application vulnerabilities.

I had to upgrade an Ubuntu / Debian based web server because Shellshock was allowing crackers to use this server as a spamming jump point. Spending too much time blacklisting - had to bite the bullet and do the full upgrade.

Not necessarily. The server can be secure, but if there is an issue in the web application, crackers can still get in.
But you can make it impossible for the magnetic box to hold the key, right?

Have you set a password on the root account, changed the default passwords on MySQL, and tested for shellshock (env x='() { :;}; echo vulnerable' bash -c "echo this is a test")?
I did all of the above

 

[debian_guy]

You should need to know the reason first. Search the host for newly created suspicious files and search them in the internet for kind of the hack.

- Close all unnecessary ports of your server with iptables or else
- Change your ssh port and give it some good password
- Reinstall all your systems if its possible with new files

 

[rstanley]

And change ALL passwords on the system to secure cryptic passwords, preferably at least 16 characters. If by doing all this and you still get hacked, then you should upgrade to Jessie on a bare hard drive and installing all software from scratch.

Also remove your authorized keys file.and generate all new public and private keys for anyone using ssh to log into the server

 

[Jim Laughlan]

Just a thought, are there any firewall issues that need to be looked at? Open ports they're getting through?