Reading a file line by line using bash

Welcome to this weeks edition of practical shell scripting. This week I'm going to give a small simple example of a how to read a file one line at a time using bash. As with my other previous and hopefully future blog entries I'm trying to keep this down to just a single small topic and try to give a good example of it without clouding it with a lot of other things.

What prompted this weeks entry was a side project I had over the weekend to update one of my servers at home that required me to download a large number of files from an http server. However, I did not want to go to each file one at a time and click on it and download through the web so I viewed the source of the html file, saved it, and stripped out the html until I had a file that was nothing more than a list of all the files that I wanted to download. Once I had that file I fed it into a much more complicated script that retrieved those files from the http server using wget.

For the sake of this example we're going to be looking at just a tiny part of that and I've turned it into a generic example to show how bas reads line by line from a file called grocery.list.

First, here's what's in the file called grocery.list :

apples
oranges
bananas
pears
juice

Now here's the bash script to read it:

#!/bin/bash
###-------------------------------------------------###
#
# this is a simple bash script to show
# how to read from a file
#
###-------------------------------------------------###

INPUT_FILE="$1"

while read curline; do
echo $curline

done < "$INPUT_FILE"



....and that's all there is to it. So what we have here is a simple looping construct that goes one line at a time and steps through the input file that is given as the first argument or $1. In his example "curline" is the variable that the data is read into. This can be confusing because "curline" does not need to be declared a head of time, and in fact in all the shell scripts I have seen I can not recall a single instance of ever being declared up front. The other thing that has always been a little counter intuitive to me is that file being read from is always at the bottom of the while loop which is taking it's input in the done line. Most of us who come from a more formal language like C or C++ are used to having the file declared and read from someplace closer to the top, and this always seems odd to me.


So to execute this example you would type:
./ReadLineExample.sh grocery.list

And the screen should show exactly what is in the file. While there are a great many tools in bash for parsing and working with files this is one of the most versatile since it allows simple usage of a looping construct.

So that's it for this week. As always if you have any questions or comments please feel free to contact me. Happy coding.

Working with getopts and optargs in bash

Welcome to this weeks edition of Practical shell scripting. This weeks edition is about getopts and optargs in bash. In the last couple of weeks I was given a job to do at work that had a request to create a shell script that was capable of taking command line argument in the fashion of what C/C++ programmers would refer to as argv and argc. That is the arguments indicate what and how the following value will be used. This is typically depicted in a fashion like so:

#> ./foo -y 14

Where foo is the application or executable and -y tells you something about the next argument coming in. So maybe -y is for years or something else defined by the application foo internally and the value following -y is that value. This is easily done in the C programming language, but I had never done this in shell scripts, and while it seems straight forward there was a gotcha that I wanted to point out that hung me up a bit. So first here's a slice of example code to illustrate the basics of implementing getopts and optargs in a shell script. As with all shell scripts the # identifies lines which are not interpreted by bash either to be used as comments or to delineate the code in some fashion.


#!/bin/bash
###-----------------------------------------------------------###
# GetOptsExample.sh
# a small shell script to demonstrate getopts and optargs in bash
# --apg
#
###-----------------------------------------------------------------------------------###
PRODUCE=0

while getopts "a:o:b:" OPTION;
do
case $OPTION in
a)
## -a is for apples ##
echo "OPTION is a"
PRODUCE=$OPTARG
echo "The produce is $PRODUCE apples "
;;
o)
## -o is for oranges ##
echo "OPTION is o"
PRODUCE=$OPTARG
echo "The produce is $PRODUCE oranges"
;;
b)
## -b is for beans ##
echo "OPTION is b"
PRODUCE=$OPTARG
echo "The produce is $PRODUCE beans"
;;
esac
done


###--------------------------------------------------------------------------------------###

So what we see here is basically a simple case statement that uses the word OPTION as the flag for the case statement to differentiate which options are incoming as arguments. The optargs are then the parts that follow the options. This all seems fairly straight forward enough. However, the gotcha in my original attempts was quite sneaky. It seems that one critical aspect of the getopts line is the placement of the colon after each option. Each option will only accept an argument if that option is followed by a ":". So If you were to enter the line of code like this

while getopts "a:o:b" OPTION; #### -- note the missing ":" after the b option

any argument that you had hoped to pass along pertaining to the b would be lost. One easy way to test this out is to simple walk through each optarg section in the case and echo out what his optarg argument is.

So that's it for this week. As always if you have any questions or comments please feel free to.
contact me. Happy coding.