Bash Gotchas

(or got me, anyway)

Lately, I've been playing around with bash scripting, which is a lot of fun. However, I'm finding that I run into problems not covered by various books or howtos--either I wind up getting the answer through trial and error, asking friends who know more than I, searching deja, etc. So, I'm putting this page up as a reminder to myself. At some point, I'll probably make it available from the site's main menu, when I'm sure that I'm not overly embarrassing myself. Therefore, as I write these things, which are notes to myself, I'm trying to make them understandable to others as well.

I'm not a programmer--I think I have Attention Deficiency Disorder, and have a great deal of admiration for those who can write thousands of lines of code and keep it coherent. I remember reading once that learning programming is difficult because to understand a you must first understand b, but to understand b, you have to know part of a. I've found that to be so.

The information here has been gleaned from a variety of sources--BashProgramming--Introduction Howto and Advanced Bash Scripting Guide both online at the tldp.org site are quite useful. Another one that I have found helpful is an online guide by a gentleman with the first name of Vivek.

As each programmer, or scripter in this case, has their own habits, they sometimes give different ways, or slightly different syntax for doing the same thing. Figure out which one works, and which one you like best. A programmer friend defined elegant as striking the balance between being easy to type and easy to read. They aren't always the same thing, though they often are, as usually the shorter something is, the easier it is to both type and read.

So, below, in no particular order yet, are things that I had to research before figuring out how they worked. If you find something similar, then by all means drop me a line.

A bash script runs in its own separate shell

If you've ever made a dos batchfile, then clicked on the icon, or called it from a prompt, you've seen how a dos window will open and quickly close as soon as the command has been executed. Bash scripts can do something similar. I had thought this was because of the line at the top #!/bin/bash called a new shell, but leaving this line out doesn't change matters.

An example. I was tired of mounting a RedHat CDROM, then having to cd over to the RedHat/RPMS directory so I made a simple script.

#!/bin/bash
mount /dev/cdrom
cd /mnt/cdrom/RedHat/RPMS

The trouble was that when I called this script, I'd find myself right back at my command prompt in my home directory. The script had executed--the CDROM was mounted. However, it had done it in a separate shell, then closed that shell. The way around this one is to call the script with a . (period, then a space) first. So, if the script was called cdx at a command prompt type . cdx

This tells it to run in the current shell. Doing it this way would then put me in the /mnt/cdrom/RedHat/RPMS directory.

As typing period space <filename> is tedious, you might want to make an alias for any script which requires this. For instance, as it's usually root who will mount a RedHat CD to install a program, root's .bashrc contains

alias cdx='. cdx'

Therefore, whenever I call my cdx script, it comes out as . cdx and runs in the current shell.

This is also useful for another script I have, which will change the environment variables to Japanese, which I need from time to time. I had the same problem, it would perform the operation, close the shell it had spawned, and leave me back in an English environment. So, if a script seems to be executing, but nothing seems to be happening, test it that way.

Calling a script in a hurry with sh <scriptname>

I have a little scripting article, dealing with one and two line commands. Most of the article consists in telling the novice where to put a script and how to call it at will. This can be a bit of a nuisance when you're learning scripting and making a bunch of scripts that you won't save, but are just testing. This is one of those things that seems to be so well-known that many tutorials don't mention it. If you've just made a script and you want to test it, you can skip making a path, and even skip making it executable.

For example, say you've made your first script, that just prints Hello World on the screen. You save it as hello. Now, in the scripts directory, you can run it by just typing

sh hello

(Actually, in FreeBSD, this doesn't work--you'd have to do

bash hello

If you are trying something out, you can just run it at the command prompt, without making and saving a script. For example, I want to see how the "for" loop works, so I try something simple. At the command prompt

for friend in Bob Sue Mary John
(when you hit return, rather than the standard $ prompt, you'll see a > indicating that the command is not finished)
>do
>echo "Hello $friend"
>done

You'll then see
Hello Bob
Hello Sue
Hello Mary
Hello John

Then you'll be back at your command prompt.

Using the let keyword

Linux Programming for Dummies which is actually about bash scripting, suggests using let whenever declaring a numeric variable. Although that's not necessary, it can be used to save a bit of typing in examples like the following

a=10
b=20
sum=`expr $a + $b`

This can be more quickly typed with

let sum="$a + $b" or let sum=$a+$b or even let sum="$a+$b". Spacing can be tricky to remember---if I ever figure it out completely, I'll post it up here, but for the moment--the simplest way--that is, with the least typing, is let sum=variable+variable with no quotes and no spacing. If you do put a space between each variable and the + sign, you need double quotes.

Backticks (or backquotes, whatever you want to call them)

In the first example above, notice the mark around expr $a + $b. The reader's reflex is to type a single quote like this ' . (The one that is under the double quote). This confuses many people (including myself) when first starting on bash books or tutorials. It's a backquote, usually found under the tilde ~ to the left of the 1 key. (That's the number one, on the top row, if you don't count the F1, F2, etc keys, of a standard keyboard).

If you need a command executed in the middle of a line, you use the backtick. A single quote would simply create the variable sum with a string value of expr $a + $b. So if you did echo $sum, you'd get

expr $a + $b

Let's take a minute to review quotes.

Double quotes quote the expression, but process special symbols, such as a $. Single quotes quote and ignore special symbols. Backticks process the command.

So, we have our script

#!/bin/bash
a=10
b=20
sum=`expr $a + $b`
echo $sum

If you're not sure about quotes, take a minute to run the above script. With backticks, we get what we want. The variable "sum" adds the variables a and b and takes their added value of 20. The backticks have told it to run the command expr and process the information. If we had done it with single quotes ' then it would just echo, as said above,

expr $a + $b

That is, it would simply treat what was between the quotes as a string, and do no processing of its special characters, in this case, the $.

If done with double quotes " it will treat it as a string, but process the $a and $b. So echo $sum would result in

expr 10 + 20

Lastly, rather than backticks you can also enclose the command with $(). (Note that those are parentheses not brackets). So

sum=`expr $a + $b`
could also be written as sum=$(expr $a + $b). Use whatever is easier for you to type.

Using and not using the $ in front of a variable name.

This works. (This is from the Linux Documentation Project's beginner's bash howto
counter=0
while [ $counter -lt 10 ]; do
echo the counter is $counter
let counter=counter+1
done

This also works

counter=0
while [ $counter -lt 10 ]; do
echo the counter is $counter
let counter=$counter+1
done

This doesn't work

counter=0
while [ $counter -lt 10 ]; do
echo the counter is $counter
let $counter=$counter+1
done

Putting the $ in front of the variable refers to the current value of the variable, as opposed to the variable itself. So, you're not referring to the variable "counter" in that case--you're referring to the numeric value 0. Therefore, you're saying let 0=0+1. Try that at a command prompt--type let 0=0+1 and you'll get back an error message like

attempted assignment to non-variable (error token is "=0+1"). It's important to see the difference. If using the name of a numeric variable, and you put the $ in front of the name, you are referring to the present numeric value of that variable, and it is the same as if you had typed that number, be it 0, 1 or whatever.

Using = and -eq in [ ]

Here was an error from Linux Programming for Dummies (Hey, water seeks its own level).

if [ $1 -eq "Bob"]
It didn't work--I'd get a message that an integer or something numeric was expected. So, using if [ $1 = "Bob" ] worked. One could leave out the quotes as well. One cannot leave out the space between the = and Bob or you'll get a different type of error, unary expression expected.

The example in the book was in creating a verify function. $1 and $2 refer to first and second arguments--if you don't understand that then take a quick look at my simple scripts page. It was matching two entries, the name Bob and employee number 555. This works

if [ $1 = Bob ] && [ $2 -eq 555 ];
then
echo "Verified"
else
echo "REJECTED"
fi

Note that with the numeric expression of 555 we can use -eq. = will also work, actually, but again, you have to have the space between the = and the 555--it treats it as a string.

This can be confusing. If you're comparing numeric values, then you must use -eq -lt etc. In a test statement one of those

if [ $foo = $bar ]

the = sign is comparing strings. To compare the value of an integer you must use -eq. Many times it will work, but only because the string value is the same. Try this one

x=10
y=0010
if [ $x = $y ];
then
echo true
else
echo false
fi

It will echo false because they are two different strings, 10 and 0010. However, if you repeat it but change the test statement to read

if [ $x -eq $y ];

It will then echo true

Trapping ctrl +C and stopping ctrl + D

Someone actually asked me to write a script for them to use at work. It had to present a menu to the user, allowing them to do only three things--it also stopped them from accessing the standard shell prompt. The trouble was that they could reach the shell prompt by sending ctrl+c or ctrl+d.
Ctrl+c was fairly easy to solve with a quick look on deja. It sends a signal, so one has to trap that signal. That one was done with the trap command
trap 'echo "Please choose 1, 2 or 3"' 2 3
(If you do a trap -l, it shows you which signals are being affected--2 is SIGINT and 3 SIGQUIT. Stopping both of these fixed the ctrl+C problem.)
Ctrl+D is not a signal per se, but more of an EOF (End Of File) thing. So, I simply set it to null
stty eof ""
This kept the user from exiting with Ctl+D. Ctl+Z, for whatever reason, was never an issue.

Using # and % to strip characters

Daniel Robbins has a good article on bash scripting. His way of remembering which of these strips from the beginning of the line and which strips from the end is to look at your keyboard and see that the # comes before the %. So, the next question is, what good is this?

An example that I recently had to use on the job--we have a program that we use that can't detect an image with a jpg extension. So, I cd into the directory holding these images.

Then

for i in *.jpg
do
mv $i ${i%.jpg}
done

The % looks for a match at the end of the string and gets rid of it. So, let's go through this line by line.

for i in *.jpg
This examines every file in the directory that ends with .jpg--we're arbitrarily calling our variable i, you could call it bob if you want.

do
Hopefully you already know the syntax of a for loop--do indicates that it's going to do something :)
mv $i ${i%.jpg}
This is the tricky part. mv $i of course means that it's taking the variable, which will represent any file ending with .jpg and renaming it. Next, as it's easy to have problems with the various expansions that we're doing (for instance if you just did mv $i $i%.jpg you'll find that you now have files ending with .jpg.jpg%) we enclose the variable with {} and put the $ in front. Then, the %.jpg isolates the .jpg ending. It then strips the .jpg from the filename.

If you'd wanted to replace .jpg with .gif it would work this way. for i in *.jpg; do; mv $i ${i%.jpg}.gif; done

Now, you've renamed $i (any file ending with .jpg) to a file without the .jpg ending and added .gif to it. In other words, say a file was named test.jpg. mv $i (as the for loop goes for anything ending with .jpg includes our test.jpg). We're moving $i to a new variable ${i%.jpg} which means take $i and strip away anything after the percent sign, i.e mv test.jpg to test. We then add .gif after the variable so $i, test.jpg in this case is moved to test.gif. (Since the gif also has a dot in front, in this particular case mv $i ${i%jpg}gif would have also worked).

The # sign is similar save that it looks for a string at the beginning of a file.

Suppose you have a bunch of files named 3.122 3.12223 3.111,etc. Now, you want to remove the 3. at the beginning.

for i in 3.*
This will find all files beginning with 3.
do
mv $i ${i#3.}
done

In this case, it examined all files that had 3. in front of them and removed the beginning, rather than the end of the file. So, the files now have the 3. removed. If you wanted to substitute 3.122 etc with 4.122 you would have done mv $i 4.${i#3.} to insert your new number in front of the new file name.

Doing Arithmetic

Bash can do whole number calculations without problem. We've discussed the let keyword, backticks and $(). However, if we're performing more than one operation, the easiest way is to use $(()), double parentheses.

#!/bin/bash
echo "Please enter a number"
read NUM1
echo "Please enter a second number"
read NUM2
NUM3=$(( $NUM1 + $NUM2 ))
echo "The sum of the two numbers is $NUM3"

The $(( )) encloses the arithmetic. This can also usually be done with $[ ] but I've sometimes found that that caused errors (though it may have been because of some other mistypes on my part)

A few other shells, such as ksh and zsh can also do decimals. (Bash can't). For example:


#!/usr/local/bin/ksh93

PS3="Please choose conversion => "; export PS3
select opt in 'Farenheit to Celsius' 'Celsius to Farenheit' none

do
case $opt in 

'Farenheit to Celsius')
echo "Enter Farenheit Temperature"
read FAR
CEL=$(( ( $FAR - 32 ) * .55 ))
echo "$CEL Celsius"
break ;;

'Celsius to Farenheit')
echo "Enter Celsius temperature"
read CEL
FAR=$(( $CEL * 1.8 + 32 ))
echo "$FAR Farenheit"
break;; 

none)
break ;;

esac

done

The above is a simple temperature converter, using the select loop and case. I'm just putting it in to show ksh's ability to handle decimals. (The #/usr/local/bin/ksh93 at the top is because the above script is from a FreeBSD box).

Using ^ to search

This is from another problem I had at work. A large number of files had to have 3. as their beginning. Someone had not realized that, then caught their mistake and thought that she'd have to go through literally hundreds of files and rename any that didn't begin with 3. However, it's easily done with a bash script.

for i in $(ls | grep -v "^3\.")
do
mv $i 3.$i
done

This made sure that only files beginning with 3. were affected. The reader can try this if they want. Make a few files

touch 123 3.123 345 3345 333.5 You can then try it. Try leaving off the quotes around ^3\. Try leaving out the backslash to escape the period. Try it without the ^. You'll see that your results will vary. This is one way to learn. (By the way, sometimes, I've found that Cygwin's version of bash behaves differently than others)

A for loop performs its actions on every file in the loop

This might be obvious to the reader, but as I said, this page is primarily something to remind myself how to do things. In this case, I wanted to do the following--examine a group of files, make sure that none of them were 0 byte files, and, if I found a 0 byte file, send an email to someone about it.

My original attempt was

for i in *
do
[ ! -s "$i" ] then;
echo "this style is a 0 byte file and will have to be redone" > /home/srobbins/nobytes
ls >> nobytes
fi 
done
mail (to the person who had to redo it) < /home/srobbins/nobytes

This didn't work. Here's what the for loop would do--say it found 3 files that were 0 byte files. So, for file one it would echo "this style" etc to the /home/srobbins/nobytes. Since I was using > rather than >> it would overwrite whatever might have already been in the file. Then it would list the file name. If there was only one 0 byte file in there, this was fine. However, then the for loop continues--if it found a second 0 byte file, it would simply repeat the process and once again echo "this style" etc---again with a > so it would overwrite whatever was in the file already.

I could have made it work by changing the > to a >>. However, in that case, for each file, it would have the message "this style", etc--and that would be tedious and annoying for the user to read. A better way to do it was given me by Cameron Simpson (who often gives me a better way to do something in a bash script). I will heavily comment it here, so that the less experienced scripter can hopefully figure out what is happening.

#make a variable for the file that will be created
#It will be far easier to type $nob than /home/srobbins/nobytes
nob=/home/srobbins/nobytes

#wipe out whatever was in the file before, and create a new 0 byte file
> $nob
#run the for loop

for i in *
do
if [ ! -s "$i" ]
then
ls "$i" >> "$nob"
#If it finds any 0 byte files, it will list them in the /home/srobbins/nobytes
#file
fi 
done
#end the for loop--it's done what we needed it to do
#Now we see if anything was written to $nob--remember, it began as a
#0 byte file
if [ -s "$nob" ]
then
(echo "The following styles will have to be redone"
cat "$nob" 
) | mail user@blah.blah

What we've done is, inside the ( ) is create a subscript--this script first writes "The following styles" etc and then lists the contents of the $nob file. Then it mails the result to the user who will receive an email beginning with "These styles" etc and then the list of files. A much cleaner way to do it.

The main point of this section, however, is the first part. My big mistake was in not realizing that every time the for loop found a 0 byte file, having it echo something with a > would wipe out whatever had been previously written to that file.

Generating Random numbers

I first ran into this when making a Buffy the Vampire Slayer quote generator. (Yes, I agree, I need to get a life, but there are a LOT of funny quotes from that show.). I wanted a script to take a random number and generate a quote based on that number. Most of that was straightforward, using a case statement, but I wasn't sure how to generate the random number. FreeBSD has a program called jot included that will do it, but I wanted to know how to use Bash's built in random number generator.

As with many things, it turned out to be simple once I knew how to do it. The eventual script looked like this

#!/bin/bash
clear
case $((RANDOM%135+1)) in
1)echo "Xander: I don't like vampires. I'm going to take a
 stand and say they're not good."
;;
2)echo "Willow: Why couldn't he be possessed by a puppy, or
some ducks?"
;;
3)echo "Xander: I laugh in the face of danger. Then I hide until it goes away"
;;

The above only shows three quotes. However, the 135+1 indicates that there can be up to 135 random numbers generated. So, the syntax is

$((RANDOM%x+1))

where x is the amount of random numbers that you wish.

Interestingly, FreeBSD's version of bash didn't like that syntax. FreeBSD also includes jot, however, so there I simply substituted the case $((RANDOM%135 +1)) in with

number=`jot -r 1 1 135`
case $number in
and it worked without problem. I'm not sure if the problem with $RANDOM in FreeBSD is a difference in the spacing or what, but at time of writing, haven't had time to look into it.

Converting back and forth between decimal and binary

This isn't really the shell per se, but it's a neat little trick. You have to have the bc program installed--many distributions come with it by default, as does FreeBSD. To convert decimal to binary, if 7 was the decimal number
echo "obase=2; 7" | bc

That will give the result of 111. The obase is output base for the bc program. The piping is simply a shortcut to opening bc and then typing obase=2, hitting return and then hitting seven. The bc program assumes base 10 by default. In the next example, we indicate that the input is in base 2. So, to convert 111 back to seven. we're using ibase rather than obase.
echo "ibase=2; 111" |bc