SOLVED: "Unable to fork: Cannot allocate memory" even 10GB RAM free excluding cache

[postcd]

Hello,

on a dedicated server i am using a bash script like this: https://github.com/slrslr/linux-bas...pses-access-logs/blob/master/vpslogscanner.sh

I believe it not returned any errors "Unable to fork: Cannot allocate memory" when it was executed.
But now, when i execute it, i see this "Cannot allocate memory" error.

I do free -m before it and after it and i see there is plenty of free RAM (when i not count cache):

it is not first time i am having trouble with this error, so it probably is not just "bad" commands executed from the bash script.

Please what should i do to discover the cause, why this happens?

 

[JasKinasis]

I don't have time to take a look at the example script right now, as I'm already running late for rehearsals. But off the top of my head - if you are sure that there is no way that your machine is running out of memory when running the script - the only major factor I can think of that could cause fork() to fail would be if the number of PIDs and/or threads that are running exceed the maximums set in the kernel.
If memory serves, these limits are set in /proc/sys/kernel/threads-max and /proc/sys/kernel/pid_max.

So perhaps when you run the script, it causes fork() to be called so many times that it eventually fails when the maximum number of PIDS/threads is reached.

There may be one or two other factors - scheduling perhaps? IDK
Take a look at the following man pages: fork(2), proc(5) and sched(7)

Otherwise, have you tried debugging the script to see which part of the script is causing the problem?

bash -x /path/to/script.sh

And see where the script is failing...

Alternatively, you could enable debugging by changing the shebang line of the script like so:

#!/bin/bash -x

You could also use the -v option to debug the shell input lines as they are read.

 

[Bethlehem]

Let me tell ya... these things happen when you get older kid. Your body just can't keep up any more, and your mind is worn and tired.
Sometimes the two argue with your mind saying "hell yeah let's do it" and your body is all like "oh hell no, just wait a minute!".
But my memory aint never been so bad that I couldn't allocate enough of it to remember how to f..... oh wait. I think I read that title wrong.
I'll just take a seat over there -->

 

[postcd]

@ JasKinasis


Thx, these are values you mentioned:
# cat /proc/sys/kernel/pid_max && cat /proc/sys/kernel/threads-max
1048576
383064

when i did bash -x scriptfilename
all seems to went thru OK (it is many lines i not spotted fork error. When did "bash -x scriptfilename 2>/dev/null" it returned no output)

But when i do ./scriptfilename or /bin/sh scriptfilename
It returns mentioned fork error

Here is: # ulimit -a

core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 191532
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 191532
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

Thank you

 

[JasKinasis]

when i did bash -x scriptfilename
all seems to went thru OK (it is many lines i not spotted fork error. When did "bash -x scriptfilename 2>/dev/null" it returned no output)

But when i do ./scriptfilename or /bin/sh scriptfilename
It returns mentioned fork error

Were you able to view all of the lines of output when you ran using 'bash -x scriptfilename'?
Some terminals limit the amount of output that is cached for scroll-back - often around 1000 lines or so. If the output from the script exceeds that limit, the lines of output regarding the fork error might be unavailable to you.

Redirecting stderrors output to the bit-bucket ('2> /dev/null') certainly will not help you to diagnose the problem!

However, perhaps redirecting ALL output from the script to a file might help:

 bash -x scriptfilename &> ~/OutputFromScript

When the script is finished, use grep to check the output file for any lines that contain the word 'fork'. If any occurrences are found - examine the lines of output immediately before each to see if you can get any clues about what part of the script is throwing the error.

 grep -ni --color fork ~/OutputFromScript

When running the script, it might also be worth having another terminal window open with something like 'top' running so you can see what's going on process/thread-wise while the script is running, to see if any of the limits are close to being reached.

Looking at the script in your OP, it's a little outside of my area of expertise. I have no experience with openvz and very limited experience with any kind of virtualisation. But I assume that the vzctl commands in the script are running commands on each host/server - If this is the case, is there any chance that it could be one of the other servers that is failing to fork when vzctl passes it a command to run?? Perhaps if it is under a heavy load at the time the script is being executed - could that explain it?

I'm just spit-balling ideas at this point. I've never seen a fork error on Linux before. As a C/C++ programmer, I've used fork() in programs and I'm aware of some of the issues that can cause fork to fail. But it's not something I've seen first-hand. Just trying some typical debug-fu!

 

[postcd]

@ JasKinasis

Your command
bash -x scriptfilename &> ~/OutputFromScript

helped, as i indeed found "Unable to fork: Cannot allocate memory" there and it appears right below the command executed over one of the virtual servers.

I see this VPS is indeed not responding. It has many fail count regarding memory allocation. And when the VPS is crashed this way, it shows "Unable to fork: Cannot allocate memory" when called from within OpenVZ virtualization utility "vzctl".

To remove error is to restart VPS and prevent its software to allocate too much RAM or increase RAm for the VPS.

Thank you for the helpfull command. SOLVED

 

[JasKinasis]

Excellent. Glad to have helped!

So my hunch about one of the servers throwing the fork errors turned out to be correct.. Perhaps I should finally take the plunge and have a pop at getting LPI certified. Maybe I should consider a career-switch to sys-admin!

Using the -x option with bash has helped me to diagnose problems with errant shell-scripts in the past.
The -v option can help too!