Creating a script from a complex bash command

Question

I need to run the following command in a folder containing a lot of gzipped files.

perl myscript.pl -d <path> -f <protocol> "<startdate time>"
 "<enddate time>" -o "0:00 23:59" -v -g -b <outputfilename>

My problem is that the command does not take gzipped files as input. So, I would need to unzip all those gzipped files on the fly and run this command on those unzipped files. These gzipped files are in another folder where I am not allowed to unzip them. I want a shell script that will take the path of the remote gzipped files and store it under path (which is also going to be a argument to the script). Do the unzipping and then run the above command.

N.B: The "protocol", "startdate time", "enddate time", "outputfilename" don't have to be arguments for now I will just put them directly in the script so that it is less complex.

aularon · Accepted Answer · 2010-09-07 13:07:51Z

3

You can do:

for fname in path/to/*.gz; do gunzip -c "$fname" | perl myscript.pl ; done

Expanded:

for fname in path/to/*.gz; do
  gunzip -c "$fname" | perl myscript.pl
done

And to make it accept filenames with spaces:

old_IFS=$IFS
IFS=$'\n'
for fname in path/to/*.gz; do
  gunzip -c "$fname" | perl myscript.pl -f <protocol> "<startdate time>" \
     "<enddate time>" -o "0:00 23:59" -v -g -b <outputfilename>
done
IFS=$old_IFS

This way, you make the script read standard input, which will contain the file content, without having to use temporary files.

EDIT: Here's a wrapper script that solves the problem like initially suggested in the question:

`myscriptwrapper`:
#!/bin/bash
gzip_path="$1"
temp_path="$2"
#loop thru files from gzip_pah\th
for fname in $gzip_path/*.gz; do
  basename=`basename $fname`
  #fill them in the target dir
  gunzip "$fname" -c > "$temp_path/$basename"
done

#finally, call our script
perl myscript.pl -d "$temp_path" -f <protocol> "<startdate time>" "<enddate time>" -o "0:00 23:59" -v -g -b <outputfilename>

EDIT 2: Using tar.gz files:

`myscriptwrapper`:
#!/bin/bash
gzip_path="$1"
temp_path="$2"
cd "$temp_path"
#loop thru files from gzip_pah\th
for fname in $gzip_path/*.tar.gz; do
    tar -xzf $fname
done

#finally, call our script
perl myscript.pl -d "$temp_path" -f <protocol> "<startdate time>" "<enddate time>" -o "0:00 23:59" -v -g -b <outputfilename>

edited Sep 7, 2010 at 13:07

answered Sep 7, 2010 at 11:31

aularon

11.1k4 gold badges38 silver badges41 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

sfactor Over a year ago

thanks for the reply. but as you can see the perl script takes a lot of arguments of which the path which contains these unzipped folders is only one (-d is the option switch for it). the solution of yours doesn't take that into consideration, does it?

aularon Over a year ago

@sfactor, I edited it so now it takes other arguments, but yes you have to modify the script to read a single file from the standard input. the advantage of this is that you don't need a temporary directory for files.

aularon Over a year ago

@sfactor I edited adding the solution you initially suggested, check.

sfactor Over a year ago

sorry but there was one thing it seems it didn't make clear, the files are not just *.gz but *.tar.gz. i tried replacing gunzip with tar -zxvf but doesn't work properly. have i missed something?

Collectives™ on Stack Overflow

Creating a script from a complex bash command

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related