Command for faster Process?

Support & Help Requests
#1

Hi. I have the following command to turn *.jps to *.dds for X-Plane

ls *.jpg | xargs -I TEX basename TEX .jpg | xargs --max-procs=4 -I TEX2 convert TEX2.jpg TEX2.dds

I was wondering if I can edit this command to make it faster? The webpage where this command comes from states that the "procs" controls the number of parallel processes; I was wondering if I can increase this for a faster conversion? I have a quad-core i5 3.2GHz and an Nvidia GTX 1650.

#2

Hi, @Anthony_Craig,

xargs manual says the following:

-P max-procs, --max-procs=max-procs
Run up to max-procs processes at a time; the default is 1.
If max-procs is 0, xargs will run as many processes as possible at a time.

I believe your i5 has 2 logical cores per physical core due to hyperthreading. You can run nproc in terminal to check that.

2 Likes
#3

Two things:

  1. Consider using find . -name '*.jpg' instead of ls
  2. look at gnu parallel https://www.gnu.org/software/parallel/ (a better version of xargs)
find . -name '*.jpg' | parallel -j4 script.sh {}
3 Likes
#4

Thanks ironfoot. So, the fastest would be to replace --max-procs=4 to --max-procs=0; is that right?

pavlos, someone from Ubuntu Forums told me that running this command using "ls" is not a good idea--they said that when it fails, it fails badly and silently; so the command would be:

find . -name '*.jpg' | xargs -I TEX basename TEX .jpg | xargs --max-procs=0 -I TEX2 convert TEX2.jpg TEX2.dds

Is this right? Thanks for the info. If it helps, this computer has one physical processor, four cores, and four threads.

1 Like
#5

@Anthony_Craig

ls is not a good idea to parse output in scripts, see ParsingLs - Greg's Wiki

4 Likes
#6

With --max-procs=0 it will simply run as many parallel convert tasks as possible, as stated. Is this the optimal processing strategy for you? It depends on the size and structure of your dataset. It also depends on whether the convert function uses parallelization itself.
If you are simply choosing between running your task single-threaded vs multi-threaded, then yes, use xargs with --max-procs=0. Since you have 4 cores without hyperthreading (as you claim), I don't expect any significant difference between --max-procs=4 and --max-procs=0. On the other hand, if you are looking for some optimal solution, you have to test different approaches as well: check other parallelization tools as @pavlos_kairis suggests, study parallelization in the image processing library that provides the convert function, etc.

3 Likes