HOME

Mini Shell 1.0
DIR:/usr/local/src/parallel-20231122/src/
Current File : //usr/local/src/parallel-20231122/src/parallel_book.texi
\input texinfo
@setfilename Why_should_you_read_this_book.info

@documentencoding utf-8

@settitle Why should you read this book?

@node Top
@top Why should you read this book?

@menu
* Why should you read this book?::
* Learn GNU Parallel in 5 minutes::
* Learn GNU Parallel in an hour::
* Replacement strings::
* Advanced usage::
@end menu

@node Why should you read this book?
@chapter Why should you read this book?

If you write shell scripts to do the same processing for different
input, then GNU @strong{parallel} will make your life easier and make your
scripts run faster.

The book is written so you get the juicy parts first: The goal is that
you read just enough to get you going. GNU @strong{parallel} has an
overwhelming amount of special features to help in different
situations, and to avoid overloading you with information, the most
used features are presented first.

All the examples are tested in Bash, and most will work in other
shells, too, but there are a few exceptions. So you are recommended to
use Bash while testing out the examples.

@node Learn GNU Parallel in 5 minutes
@chapter Learn GNU Parallel in 5 minutes

You just need to run commands in parallel. You do not care about fine
tuning.

To get going please run this to make some example files:

@verbatim
  # If your system does not have 'seq', replace 'seq' with 'jot'
  seq 5 | parallel seq {} '>' example.{}
@end verbatim

@menu
* Input sources::
* Building the command line::
* Controlling the output::
* Controlling the execution::
* Pipe mode::
* That's it::
@end menu

@node Input sources
@section Input sources

GNU @strong{parallel} reads values from input sources. One input source is
the command line. The values are put after @strong{:::} :

@verbatim
  parallel echo ::: 1 2 3 4 5
@end verbatim

This makes it easy to run the same program on some files:

@verbatim
  parallel wc ::: example.*
@end verbatim

If you give multiple @strong{:::}s, GNU @strong{parallel} will generate all
combinations:

@verbatim
  parallel wc ::: -l -c ::: example.*
@end verbatim

GNU @strong{parallel} can also read the values from stdin (standard input):

@verbatim
  seq 5 | parallel echo
@end verbatim

@node Building the command line
@section Building the command line

The command line is put before the @strong{:::}. It can contain contain a
command and options for the command:

@verbatim
  parallel wc -l ::: example.*
@end verbatim

The command can contain multiple programs. Just remember to quote
characters that are interpreted by the shell (such as @strong{;}):

@verbatim
  parallel echo counting lines';' wc -l ::: example.*
@end verbatim

The value will normally be appended to the command, but can be placed
anywhere by using the replacement string @strong{@{@}}:

@verbatim
  parallel echo counting {}';' wc -l {} ::: example.*
@end verbatim

When using multiple input sources you use the positional replacement
strings @strong{@{1@}} and @strong{@{2@}}:

@verbatim
  parallel echo count {1} in {2}';' wc {1} {2} ::: -l -c ::: example.*
@end verbatim

You can check what will be run with @strong{--dry-run}:

@verbatim
  parallel --dry-run echo count {1} in {2}';' wc {1} {2} ::: -l -c ::: example.*
@end verbatim

This is a good idea to do for every command until you are comfortable
with GNU @strong{parallel}.

@node Controlling the output
@section Controlling the output

The output will be printed as soon as the command completes. This
means the output may come in a different order than the input:

@verbatim
  parallel sleep {}';' echo {} done ::: 5 4 3 2 1
@end verbatim

You can force GNU @strong{parallel} to print in the order of the values with
@strong{--keep-order}/@strong{-k}. This will still run the commands in parallel.
The output of the later jobs will be delayed, until the earlier jobs
are printed:

@verbatim
  parallel -k sleep {}';' echo {} done ::: 5 4 3 2 1
@end verbatim

@node Controlling the execution
@section Controlling the execution

If your jobs are compute intensive, you will most likely run one job
for each core in the system. This is the default for GNU @strong{parallel}.

But sometimes you want more jobs running. You control the number of
job slots with @strong{-j}. Give @strong{-j} the number of jobs to run in
parallel:

@verbatim
  parallel -j50 \
    wget https://ftpmirror.gnu.org/parallel/parallel-{1}{2}22.tar.bz2 \
    ::: 2012 2013 2014 2015 2016 \
    ::: 01 02 03 04 05 06 07 08 09 10 11 12
@end verbatim

@node Pipe mode
@section Pipe mode

GNU @strong{parallel} can also pass blocks of data to commands on stdin
(standard input):

@verbatim
  seq 1000000 | parallel --pipe wc
@end verbatim

This can be used to process big text files. By default GNU @strong{parallel}
splits on \n (newline) and passes a block of around 1 MB to each job.

@node That's it
@section That's it

You have now learned the basic use of GNU @strong{parallel}. This will
probably cover most cases of your use of GNU @strong{parallel}.

The rest of this document will go into more details on each of the
sections and cover special use cases.

@node Learn GNU Parallel in an hour
@chapter Learn GNU Parallel in an hour

In this part we will dive deeper into what you learned in the first 5 minutes.

To get going please run this to make some example files:

@verbatim
  seq 6 > seq6
  seq 6 -1 1 > seq-6
@end verbatim

@menu
* Input sources 1::
* Building the command line 1::
@end menu

@node Input sources 1
@section Input sources

On top of the command line, input sources can also be stdin (standard
input or '-'), files and fifos and they can be mixed. Files are given
after @strong{-a} or @strong{::::}. So these all do the same:

@verbatim
  parallel echo Dice1={1} Dice2={2} ::: 1 2 3 4 5 6 ::: 6 5 4 3 2 1
  parallel echo Dice1={1} Dice2={2} :::: <(seq 6) :::: <(seq 6 -1 1)
  parallel echo Dice1={1} Dice2={2} :::: seq6 seq-6
  parallel echo Dice1={1} Dice2={2} :::: seq6 :::: seq-6
  parallel -a seq6 -a seq-6 echo Dice1={1} Dice2={2}
  parallel -a seq6 echo Dice1={1} Dice2={2} :::: seq-6
  parallel echo Dice1={1} Dice2={2} ::: 1 2 3 4 5 6 :::: seq-6
  cat seq-6 | parallel echo Dice1={1} Dice2={2} :::: seq6 -
@end verbatim

If stdin (standard input) is the only input source, you do not need the '-':

@verbatim
  cat seq6 | parallel echo Dice1={1}
@end verbatim

@menu
* Linking input sources::
@end menu

@node Linking input sources
@subsection Linking input sources

You can link multiple input sources with @strong{:::+} and @strong{::::+}:

@verbatim
  parallel echo {1}={2} ::: I II III IV V VI :::+ 1 2 3 4 5 6
  parallel echo {1}={2} ::: I II III IV V VI ::::+ seq6
@end verbatim

The @strong{:::+} (and @strong{::::+}) will link each value to the corresponding
value in the previous input source, so value number 3 from the first
input source will be linked to value number 3 from the second input
source.

You can combine @strong{:::+} and @strong{:::}, so you link 2 input sources, but
generate all combinations with other input sources:

@verbatim
  parallel echo Dice1={1}={2} Dice2={3}={4} ::: I II III IV V VI ::::+ seq6 \
    ::: VI V IV III II I ::::+ seq-6
@end verbatim

@node Building the command line 1
@section Building the command line

@menu
* The command::
* The replacement strings::
@end menu

@node The command
@subsection The command

The command can be a script, a binary or a Bash function if the
function is exported using @strong{export -f}:

@verbatim
  # Works only in Bash
  my_func() {
    echo in my_func "$1"
  }
  export -f my_func
  parallel my_func ::: 1 2 3
@end verbatim

If the command is complex, it often improves readability to make it
into a function.

@node The replacement strings
@subsection The replacement strings

GNU @strong{parallel} has some replacement strings to make it easier to
refer to the input read from the input sources.

If the input is @strong{mydir/mysubdir/myfile.myext} then:

@verbatim
  {} = mydir/mysubdir/myfile.myext
  {.} = mydir/mysubdir/myfile
  {/} = myfile.myext
  {//} = mydir/mysubdir
  {/.} = myfile
  {#} = the sequence number of the job
  {%} = the job slot number
@end verbatim

When a job is started it gets a sequence number that starts at 1 and
increases by 1 for each new job. The job also gets assigned a slot
number. This number is from 1 to the number of jobs running in
parallel. It is unique between the running jobs, but is re-used as
soon as a job finishes.

@menu
* The positional replacement strings::
@end menu

@node The positional replacement strings
@subsubsection The positional replacement strings

The replacement strings have corresponding positional replacement
strings. If the value from the 3rd input source is
@strong{mydir/mysubdir/myfile.myext}:

@verbatim
  {3} = mydir/mysubdir/myfile.myext
  {3.} = mydir/mysubdir/myfile
  {3/} = myfile.myext
  {3//} = mydir/mysubdir
  {3/.} = myfile
@end verbatim

So the number of the input source is simply prepended inside the @{@}'s.

@node Replacement strings
@chapter Replacement strings

--plus replacement strings

change the replacement string (-I --extensionreplace --basenamereplace --basenamereplace --dirnamereplace --basenameextensionreplace --seqreplace --slotreplace

--header with named replacement string

@{= =@}

Dynamic replacement strings

@menu
* Defining replacement strings::
* Copying environment::
* Controlling the output 1::
* Controlling the execution 1::
* Remote execution::
* Pipe mode 1::
* That's it 1::
@end menu

@node Defining replacement strings
@section Defining replacement strings

@node Copying environment
@section Copying environment

env_parallel

@node Controlling the output 1
@section Controlling the output

@menu
* parset::
@end menu

@node parset
@subsection parset

@strong{parset} is a shell function to get the output from GNU @strong{parallel}
into shell variables.

@strong{parset} is fully supported for @strong{Bash/Zsh/Ksh} and partially supported
for @strong{ash/dash}. I will assume you run @strong{Bash}.

To activate @strong{parset} you have to run:

@verbatim
  . `which env_parallel.bash`
@end verbatim

(replace @strong{bash} with your shell's name).

Then you can run:

@verbatim
  parset a,b,c seq ::: 4 5 6
  echo "$c"
@end verbatim

or:

@verbatim
  parset 'a b c' seq ::: 4 5 6
  echo "$c"
@end verbatim

If you give a single variable, this will become an array:

@verbatim
  parset arr seq ::: 4 5 6
  echo "${arr[1]}"
@end verbatim

@strong{parset} has one limitation: If it reads from a pipe, the output will
be lost.

@verbatim
  echo This will not work | parset myarr echo
  echo Nothing: "${myarr[*]}"
@end verbatim

Instead you can do this:

@verbatim
  echo This will work > tempfile
  parset myarr echo < tempfile
  echo ${myarr[*]}
@end verbatim

sql
cvs

@node Controlling the execution 1
@section Controlling the execution

--dryrun -v

@node Remote execution
@section Remote execution

For this section you must have @strong{ssh} access with no password to 2
servers: @strong{$server1} and @strong{$server2}.

@verbatim
  server1=server.example.com
  server2=server2.example.net
@end verbatim

So you must be able to do this:

@verbatim
  ssh $server1 echo works
  ssh $server2 echo works
@end verbatim

It can be setup by running 'ssh-keygen -t dsa; ssh-copy-id $server1'
and using an empty passphrase. Or you can use @strong{ssh-agent}.

@menu
* Workers::
* --transferfile::
* --return::
* --cleanup::
@end menu

@node Workers
@subsection Workers

@node --transferfile
@subsection --transferfile

@strong{--transferfile} @emph{filename} will transfer @emph{filename} to the
worker. @emph{filename} can contain a replacement string:

@verbatim
  parallel -S $server1,$server2 --transferfile {} wc ::: example.*
  parallel -S $server1,$server2 --transferfile {2} \
     echo count {1} in {2}';' wc {1} {2} ::: -l -c ::: example.*
@end verbatim

A shorthand for @strong{--transferfile @{@}} is @strong{--transfer}.

@node --return
@subsection --return

@node --cleanup
@subsection --cleanup

A shorthand for @strong{--transfer --return @{@} --cleanup} is @strong{--trc @{@}}.

@node Pipe mode 1
@section Pipe mode

--pipepart

@node That's it 1
@section That's it

@node Advanced usage
@chapter Advanced usage

parset fifo, cmd substitution, arrayelements, array with var names and cmds, env_parset

env_parallel

Interfacing with R.

Interfacing with JSON/jq

4dl() @{
  board="$(printf -- '%s' "$@{1@}" | cut -d '/' -f4)"
  thread="$(printf -- '%s' "$@{1@}" | cut -d '/' -f6)"
  wget -qO- "https://a.4cdn.org/$@{board@}/thread/$@{thread@}.json" |
    jq -r '
      .posts
      | map(select(.tim != null))
      | map((.tim | tostring) + .ext)
      | map("https://i.4cdn.org/'"$@{board@}"'/"+.)[]
    ' |
      parallel --gnu -j 0 wget -nv
@}

Interfacing with XML/?

Interfacing with HTML/?

@menu
* Controlling the execution 2::
* Remote execution 1::
@end menu

@node Controlling the execution 2
@section Controlling the execution

--termseq

@node Remote execution 1
@section Remote execution

seq 10 | parallel --sshlogin 'ssh -i "key.pem" a@@b.com' echo

seq 10 | PARALLEL_SSH='ssh -i "key.pem"' parallel --sshlogin a@@b.com echo

seq 10 | parallel --ssh 'ssh -i "key.pem"' --sshlogin a@@b.com echo

ssh-agent

The sshlogin file format

Check if servers are up

@bye