parallel commands

[Diputs]

What are the options if you want to use parallelism for file operation commands (like CP) on huge amounts of files ?

Suppose I want to copy entire structures, but there's lots of files in the complete structure.
How can you copy these files in such a way that the whole of it goes significantly faster ?

Excluding the method to launch several commands based on different subdirectories or file specification, that's a too simple solution that only works in very specific conditions.
A good solutions needs to work for all conditions.

 

[JasKinasis]

Take a look at GNU parallel.
Using parallel, you can set up jobs that will use a pool of worker threads to run a command in parallel. You can specify the maximum number of threads/cores to use. Or if you don’t set a limit, it will use all available cores/threads.


Parallel should be in your installed distributions software repositories. So the best place to install will be via your usual package manager.

There’s a tutorial here:

Learn Multi-Threading Bash scripts with GNU Parallel

Learn how to speed up those Bash command and scripts with GNU Parallel with some handy GNU Parallel examples in this tutorial!

adamtheautomator.com

And official documentation here:

GNU Parallel - GNU Project - Free Software Foundation

www.gnu.org

 

[Diputs]

That sounds promising ... and my Linux is supported. Yeah !

Thanks

So it's not a parallel option for the command in question, but a tool to run anything in parallel. Gotta start somewhere.