Trying to use GNU Parallel with sed

Question

Why does this line do nothing? I'm trying to run a "sed" command in parallel and it outputs nothing to "standard.txt"

$Filetemp = the file I'm stream editing

standard.txt = the file I'm outputting to

cat $Filetemp | parallel --pipe sed -e "s/[[:space:]]\+/ /g" > standard.txt

This is the original code that works just fine but takes way too long:

sed -e "s/[[:space:]]\+/ /g" $Filetmp > standard.txt

GNU Parallel Version: 20130922

Lubuntu 14.04

I have no idea how to use parallel so I don't know what works or doesn't — Roboman1723, Jun 23 '14 at 16:27
What are you trying to achieve? Did you want to replace one or more spaces to a single space? — Avinash Raj, Jun 23 '14 at 16:28
I would like to run sed -e "s/[[:space:]]\+/ /g" $Filetmp > standard.txt in parallel since the file I'm editing is huge. — Roboman1723, Jun 23 '14 at 16:29
sed parses the input line by line. I don't know the meaning of parallel. — Avinash Raj, Jun 23 '14 at 16:34
Option -q http://www.gnu.org/software/parallel/man.html#quoting If $Filetemp is a real file, then you may find the new option --pipepart interesting, too. — Ole Tange, Jun 23 '14 at 17:08

score 6 · Answer 1 · answered Jun 23 '14 at 16:35

6

When I run that on a random file, I got a syntax issue. You need to quote the command:

cat $Filetemp | parallel --pipe 'sed -e "s/[[:space:]]\+/ /g"' > standard.txt

If that still doesn't work for you, test echo $Filetemp (or pick a file manually) or remove the redirection so you can see the output as it happens (just in case there's some sort of weird overwriting issue).

answered Jun 23 '14 at 16:35

Oli

299,936

Thanks, all I needed was the single quotes. Oddly enough it seems to run slower than running on one core. Do you have any idea why? – Roboman1723 Jun 23 '14 at 18:44

score 0 · Answer 2 · answered Apr 04 '18 at 13:10

Since version 20140422 GNU Parallel has had --pipepart which is highly efficient:

parallel -a $Filetemp --pipepart 'sed -e "s/[[:space:]]\+/ /g"' > standard.txt

And from version 20161222 you can use --block -1 which will chop $Filetemp into one block per jobslot:

parallel -a $Filetemp --block -1 --pipepart 'sed -e "s/[[:space:]]\+/ /g"' > standard.txt

This can deliver more than 1 GB/s for each core, which means your are likely limited by I/O. You can see if this the case by looking at:

iostats -dkx 1

If the utilization is 100%, then the disk is the bottleneck.

Trying to use GNU Parallel with sed

2 Answers2