Total file system inode count; Web Server / VPS / Security

[askquestiongetanswernobs]

Hi, my first post here and hopefully asking a simple question will get me a simple or reasonably simple answer.

Using the Linux terminal how do I determine how many inodes the file system was setup with initially?

My server came under inode attack two days ago. I've never heard the phrase inode and it still makes no sense. It's a huge vulnerability where whoever setup the file system did so not using the maximum number of inodes. Apparently no one bothered to ensure that the inode system doesn't get taken advantage of...well, someone decided to take advantage of it. Here is a paragraph from this PDF file:

https://www-users.cse.umn.edu/~kjlu/papers/kara.pdf

The inode attack. In the inode attack, the victim-container keeps
allocating inode structures. Unfortunately, the mount namespace
does not isolate the inode. Neither the Linux kernel provides any
inode related control groups. As a result, all inodes on the partition
are exhausted. All operations consuming inodes fail, including
the ones from the victim-container or the host machine. In our
experiments, Alibaba Cloud is vulnerable to the inode attack. The
victim-container even gets evicted. Moreover, the host-machine
cannot create any new files either.

I'm a web developer and I know more than enough about Linux to have a mild conversation though (wildly) far away enough to not make it the OS on my PC. I'm not a security expert or specialize in security though I have most certainly come across many security issues in the past. The support team at my host decided to play off the inode attack as a misconfiguration of modsec. I'm not sure if modsec is Linux specific, distro specific or cPanel specific. I've had several exchanges with different people because apparently I'm a technological prostitute not worth more than 30 seconds for a pointless reply. This was clearly an attack as the host is otherwise specific about attacks, maintenance and other such related things. So I do not know where the security vulnerability is nor how to prevent it from happening. I recieved an email from cPanel about an 80% usage. An hour later it was around 87%. Before I got off the phone with a worthless level 1 tech who just wanted to placate me it was at 97%. Apparently, if the inode usage reaches 100% files written/updated/whatever can get corrupted. Then a day or two of attacking this program becomes a week or two.

The purpose of my question (using a user name I wouldn't use) is that I build monitoring software and I need to know in advance if there is going to be a problem so I can solve it while I'm working solo instead of say, in the middle of a sales presentation where having everything on f#$%ing fire or already in a pile of ashes would destroy my credibility because so many people can't do their jobs correctly. I install PHP modules and often use newer (not newest, newer) versions of things that software (that I hate) like cPanel is combative against. So I have to ensure that after a server downgr...er, "upgrade" that cPanel hasn't destroyed half of my custom server configuration. I color code this stuff and everything because I just need to know that everything is nominal so I can you know, move forward with my life. I love programming but I'm not programming for the sake of programming.

So the closest I've gotten (and no one better try to placate me with this as the answer) is using df -i.

A modified example of the output:

Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/ploop15222p3 216000 26949 159051 13% /
none 524288 1 54272 1% /sys/fs/cgroup
none 524288 7 54214 1% /dev
tmpfs 524288 2 54286 1% /dev/shm
tmpfs 524288 30 53988 1% /run

So the lazy person (who should be too lazy to reply) would say the first number of Inodes is the system total. But even if I didn't mess with those numbers the basic math doesn't work (and yes, I verified them properly). Unless of course there is some convoluted thing where inodes are assigned to directories instead of being in some general pool of availability. So no, I absolutely do not trust that number at all!

For anyone replying I can not overemphasize that my primary focus is to resolve things that are within my control. If I know an inode attack is happening I can call support and they'll at least resolve it while refusing to give me terminal commands or whatever they did in WHM/cPanel/etc. Which is fine because that is a very quick way to get me to switch hosts. When someone is starving you don't teach them to fish and not give them a fish or give them a fish and not teach them, you give them the fish so they don't starve now and you teach them to fish so they don't starve later either! So once I have the ability to know when, in whack-a-mole, a mole will pop-up that I have the ability to "whack" it by calling support - then I'm interested in hunting down the vulnerability. I am, secondary, interested in locking down useless or misused commands (so long as I know how to turn them back on). Or disabling ports or removing garbage software included that I don't need that just dump in their own vulnerabilities.

Lastly this is not my first time dealing with the consenquences of someone else's stupidity. So I'm committed to not doing the stereotypical "it wurked thanks-k-bye" useless replies so someone else who is also at the mercy of stupid people can hopefully resolve the same or similar-enough issue in much less time. From what searching I have done this is not a day-zero attack though it's not some wildly known attack. Hopefully replies will be useful and on-topic.

 

[osprey]

The "df -i" output appears unclear with the number: 15222p3, not actually being a number that the df command would normally output. If you could use code tags to make the output clear and exact, it would help. It can be copied and pasted for accuracy, for example:

[tom@min ~]$ df -i
Filesystem       Inodes  IUsed    IFree IUse% Mounted on
udev            1995447    470  1994977    1% /dev
tmpfs           2001599    887  2000712    1% /run
/dev/nvme0n1p3 29523968 398104 29125864    2% /
tmpfs           2001599      1  2001598    1% /dev/shm
tmpfs           2001599      5  2001594    1% /run/lock
efivarfs              0      0        0     - /sys/firmware/efi/efivars
/dev/nvme0n1p1        0      0        0     - /boot/efi
tmpfs            400319     69   400250    1% /run/user/1000

If unclear about code tags, you can have a look here:

BBCode - Scratch Wiki

en.scratch-wiki.info

at the "code" section.

If you could provide some information about the nature of the "inode attack" it would help, for example, what are the symptoms on the system and is it any of the following?:

a) Inode injection where an attacker may attempt to inject malicious data into inode structures to modify file permissions, ownership, or other attributes, enabling them to gain unauthorized access to sensitive files.
b) Inode corruption where an attacker may deliberately corrupt inode data structures to cause filesystem errors or crashes, potentially leading to denial of service or data loss.
c) Inode symlink attacks where an attacker may create symbolic links (symlinks) to manipulate file operations or trick users or applications into accessing unintended files or directories.
d) Inode-based privilege escalation where an attacker may exploit vulnerabilities in the filesystem or kernel to manipulate inode data structures and escalate their privileges on the system.

Or is it something else?

 

[askquestiongetanswernobs]

Thanks for the reply Osprey. My bad on not using the quote bit earlier:

Filesystem         Inodes  IUsed   IFree IUse% Mounted on
/dev/ploop34569p1 2016000 256952 1759048   13% /
none               524288     16  524272    1% /sys/fs/cgroup
none               524288     74  524214    1% /dev
tmpfs              524288      2  524286    1% /dev/shm
tmpfs              524288    294  523994    1% /run

Again, I'm not a security specialist nor am I a server administrator. So I'm not sure where exactly where to look and what to look for. I have root access on my VPS and can get around and find things if I have an idea of what I should be looking for though. I'm happy to go through the process to get meaningful confirmation though.

 

[dcbrown73]

Hi,

I had a bit of difficulty understanding your question as the post was quite long and talked about several different things.

By inode attack, what do you mean? You noted the Support staff at your hosting provided said it was a misconfiguration of mod_sec (Web Application Firewall) that caused the attack. That is a very broad statement so I cannot comment on that without more context.

You noted the numbers didn't add up when talking about your inodes, but they seem quite close to me.

Total Inodes = 2016000
Free iNodes = 1759048
1759048 / 2016000 = 87.254% or 12.746% used.

That's pretty close to the 13% df -i gave you above.

Anyhow, as Osprey commented. If you could explain what you mean by inode attack. We might be able to help you. That said. You can misconfigured mod_sec, but the most common misconfiguration that I've seen is disabling a key rule or not enabling an appropriate rule for the software you are running.

WAF's aren't perfect and if you are running an extremely common application like Magento, Wordpress and even cPanel. There are enormous amount of people out there trying specifically to exploit these applications. We have one at work and we basically block all non-US based access to it and that cut the attacks down by 700%. We still get idiots attacking, but GeoBlocking is a good thing!

 

[CaffeineAddict]

Using the Linux terminal how do I determine how many inodes the file system was setup with initially?

It's easy to know because all you need to know is file system type, ext2, ext3 and ext4 each provide different amount of inodes.
See this question for more info: https://unix.stackexchange.com/ques...ines-the-number-of-inodes-available-on-a-disk

As you can see from the answer (and also comments) it also depends on the cmd line arguments used to format specific file system and disk block size.

So the lazy person (who should be too lazy to reply) would say the first number of Inodes is the system total. But even if I didn't mess with those numbers the basic math doesn't work (and yes, I verified them properly). Unless of course there is some convoluted thing where inodes are assigned to directories instead of being in some general pool of availability. So no, I absolutely do not trust that number at all!

It is indeed system total if by "system" you mean file system and not OS use of inodes, but again it depends on which file system type? is it ext4 or something else? and how it was formatted?

If you want to calculate then run this command:
cat /etc/mke2fs.conf

From the output under [defaults] you'll see number which says inode_ratio = 16384 (or some other number, 16384 is the default)

Next step is to calculate how many bytes your file system is consuming, to do this run this command:
lsblk -b
From the output identify the partition you're interested in and then do the math.

bytes of partition from output above / inode_ratio
And you should get the same number that's reported by df -i per given partition.

This of course applies for default formatted partition, otherwise you need to update variables.

 

[osprey]

The output in post #3 looks fine as @dcbrown73 suggests.
Some general observations:

An inode attack would very likely involve root privileges on the system, and if an intruder had root privileges, there are a number of more productive malicious ways to mess with a system than attack the inodes.

A common error message when inodes are involved is: "checksum does not match inode" , but there's no indication such a message has appeared in any of the logs. Perhaps check all the logs in /var/log, and the journal. That sort of message is usually dealt with initially by using fsck to check the filesystem and repair it. There's no harm in checking the filesystem at any time with fsck, but the filesystem has to be unmounted to do that.