Bash: Influence amount of pipe data?

[peterfarge]

Hello Forum,

I have two programs connected with a pipe. The pipe writes every friday text lines (newline ended) to a second process who processes them. Sometimes it takes several hours. Is it possible to set the amount of textlines per second that goes through the pipe? Maybe by putting a third program in between.


Thanks

Peter

 

[JasKinasis]

If CPU usage is a problem, then surely - adding a third process is just going to increase the amount of CPU usage on the server, NOT reduce it?

One option could be to take a look at the algorithms you are using in the two programs and see if you can simplify, or optimise them to make them more efficient.

Otherwise, would it make sense for your initial process on the server to collect the data and then upload/store it somewhere, so you can download it and then process it on your local PC? That way CPU usage on the server is not as much of an issue!

 

[peterfarge]

Its a virtual private server and the terms of service say that I should not use the CPU at 100% for a long time (only for bursts). The process take lets say 3h. For me it would be ok to reduce the CPU load to 10%, so the process takes 30h. I found this solution:
cpulimit --limit=10 -b --exe=progB >/dev/null 2>&1 &

It would be really cool if there is a program which controls the number of text line per seconds, reads from stdin and writes to stdout:
progA | control -lines60s 1000 | progB
Then I could calculate at what time progB should be exactly finished. But I hope the cpulimit solution will work too.