From: (Joe Halpin)
Subject: FAQ - Answers to Frequently Asked Questions
Summary: This posting answers questions which come up with some
         frequency on It should be read by
         anyone with a question about shell programming before
         posting questions to the newsgroup.

Archive-name: unix-faq/shell/sh
Posting-Frequency: monthly
Version: $Id: cus-faq.html,v 1.11 2005/09/01 17:39:36 jhalpin Exp $
Maintainer: Joe Halpin 

This FAQ list contains the answers to some Frequently Asked Questions often seen in It spells "unix" in lower case letters to avoid arguments about whether or not Linux, FreeBSD, etc are unix. That's not the point of this FAQ, and I'm ignoring the issue.

This document as a whole is Copyright (c) 2003 Joe Halpin. It may be copied freely. Exceptions are noted in individual answers.

Suggestions, complaints, et al, should be sent to the maintainer at or posted to

There are two levels of questions about shells.

One is the use of the shell itself as an interface to the operating system. For example, "how do I run a program in the background, and go on with other things?". Or "how do I setup environmental variables when I log in?".

The other level is how to write shell scripts. This often involves having the shell execute unix utilities to perform part of the work the shell script needs to accomplish, and requires knowledge of these utilities, which isn't nominally in the scope of shell programming. However, unless the question involves something other than standard unix utilities, it should be included in this FAQ.

Standard unix utilities are defined by either POSIX or the Single Unix Specification. These are now joined and are normally abbreviated as "POSIX/SUS". This specification can be found at

The man pages found on that web page define standard behavior for any given utility (or the shell itself). However, you should also check the man page on your system for any utility or shell you need to use. There isn't always a perfect correspondence between the standard and a particular implementation (in fact, I'm not sure there's any case in which they perfectly correspond). There is also an Austin Group FAQ, which describes the standardization effort in more detail ls at

Other good web sites that provide information about shells and shell programming (including OS utilities) include:

This FAQ is available at

The predictable legal stuff

The answers given in this FAQ list are provided with the best intentions, but they may not be accurate for any particular shell/os. They may be completely wrong for any shell/os. If you don't test the answers, that's a bug in your procedures.

There are no guarantees for the answers or recommendations given in this document. In fact, I don't even claim to have tested any or all of them myself. Many of the answers here have been contributed by one or more regular participants in the newsgroup, who I believe to be competent (certainly more competent than I am), but THERE ARE NO GUARANTEES.

Did I really need to make that all uppercase? Hopefully not, but there are a lot of lawyers around with too much time on their hands, so I want to make it clear that THERE ARE NO GUARANTEES about the accuracy of answers in this FAQ list. This is, hopefully, an aid to people trying to learn shell programming, but it is NOT a supported product. You have to figure out for yourself whether or not the answers here work for what you're trying to do.

Under no circumstances will the maintainer of this FAQ list, or any contributors to it, be held liable for any mistakes in this document. If the answers work for you, well and good. If not, please tell me and I'll modify them appropriately so that this will be more useful.

If you don't agree to that, don't read any farther than this. Reading beyond this point indicates your agreement.

If you do test the answers and find a problem, please send email to the maintainer (see above), so it can be corrected, or (preferably) post a question to the newsgroup so it can be discussed and corrected if there's a problem.

A number of people have contributed to this FAQ, knowingly or unknowingly. Some of the answers were taken from previous postings in the group, and other people contributed questions and answers directly to the maintainer, which you are welcome to do as well.

Among the contributors is Heiner Steven, who also provided the momentum to get this FAQ list started. He maintains a web site about shell programming that has a lot of good stuff in it:


  • 0a. Glossary
        POSIX/SUS ("the standard")
        race condition



    Some contributors may copyright their submissions and license them differently than this document.


    Google is one of the search engines on the Internet. It took over dejanews some years ago, and now is the standard reference when directing someone to a past thread one some topic. This is a very good place to start when researching a question about shell programming (and just about anything else).

    POSIX/SUS ("the standard")
    POSIX (Portable Operating System Interface) and SUS (Single Unix Specification) have been joined into one standard. This is what people usually mean when they refer to "the standard" in discussions about unix. When people in this group refer to the POSIX shell, they are talking about the shell prescribed by this specification. You can find this standard at
    This is short for "Useless use of cat". It's used to point out that some example script has used cat when it could have used redirection instead. It's more efficient to redirect input than it is to spawn a process to run cat. For example
    $ cat file | tr -d 'xyz'
    runs two processes, one for cat and one for tr. This is less efficient than
    $ tr -d 'xyz' < file
    In general, "cat file | somecommand" can be more efficiently replaced by "somecommand < file" or (especially for multi-file input)
    $ somecommand file [file ...]
    but check the man page for "somecommand" to find out if it will accept this syntax.

    For more details about this, as well as other things like it, see

    This refers to a file which starts with '.' (a dot). These files are not shown in directory listings without the -a (or -A in newer versions of ls - check the man page on your system) option to ls. Often they are configuration files, subdirectories used by applications to store configuration files, NFS swap files, et al.
    The word "portable" means different things to different people, in different situations, which is to say, there isn't one definition of "portable".

    At one extreme, a portable script is one which will work under any shell, on any operating system. At this end of the spectrum, there is no such thing as a portable shell script (some operating systems don't even have shells). If we confine the operating system to unix (which would make sense since this is, the only truly portable scripts are those which make no use of built-in shell facilities or syntax, but which only call external utilities. For example

    echo Hello World

    would probably qualify. However, that doesn't do anyone much good.

    Given that there are probably few (if any) scripts which have to meet such a standard, a more frequent use of the word "portable" indicates the degree to which a script will run under different shells and/or different operating environments.

    For example, if you're writing an installation script for an application, and the platforms on which that application runs are defined, then the problem is pretty well bounded. The choice of shell is one which is available on all required platforms, and the syntax to be used is the smallest subset of all the variants of that shell on the target platforms.

    The degree to which your shell script needs to be portable has to be determined by you, or the requirements you've been given for the script.

    race condition
    This is a situation in which two entities (processes, threads, etc) are trying to access a shared resource, or perform the same action, and the result depends on the order of execution of the two entities.
    This is the first line of a shell script, which indicates to the operating system which interpreter (shell) it should invoke to interpret the script. It has the form
    #!/path/to/shell [ argument ]
    where /path/to/shell might be /bin/sh, /usr/local/bin/bash, etc.

    This line is only interpreted by the operating system. That is, if a shell script ( is executable and run from the command line by typing its name and giving the script as an argument, as in

    $ sh
    then sh interprets For sh, the shebang line is simply a comment, and is ignored.
    A number of shells are discussed in this group, including
    • sh
    • csh
    • pdksh
    • ksh88
    • ksh93
    • tcsh
    • zsh
    • rc
    • es
    • bash
    • ash
    • dash
    These (and more) are names of shells which are referenced in the group. A comparison of some of these is available at

    However, it does not make specific the differences between ksh88, ksh93 and pdksh, which are not entirely compatible.

    A. top posting
    Q. What's the most irritating way to respond on usenet?

    Please see the following:

    0b. Notes about using echo

    This isn't really a FAQ, but discussions about using echo come up often enough that it seems reasonable to have something about it in the FAQ list.

    The echo command is not consistent in the handling of its arguments from implementation to implementation. Sometimes a string with backslash quoted characters will be interpreted in one way, and sometimes another.

    Also, if the string being echoed wasn't built into the script itself, then it could have shell metacharacters in it, which could confuse things. In cases where external input is used to build a string to be echoed the string typically should be quoted.

    For example

    s="a string with\na newline and\ta tab"
    Following are some results with various shells:
       $ echo "$s"
       a string with\na newline and\ta tab
       $ echo -e "$s"
       a string with
       a newline and        a tab
       $ echo "$s"
       a string with
       a newline and        a tab
       $ echo -e "$s"
       $ echo "$s"
       a string with
       a newline and        a tab
       $ echo "$s"
       a string with
       a newline and        a tab
       $ echo -e "$s"
       -e a string with
       a newline and        a tab
       $ echo "$s"
       a string with\na newline and\ta tab
       $ echo -e "$s"
       -e a string with\na newline and\ta tab
    Note that ksh93 makes the handling of arguments system dependent when they contain '\', and/or the first argument begins with '-'.

    POSIX does not allow the -e option. It also makes the result of using -n or any string with '\' in it implementation-defined. However, on XSI-conforming systems, it disallows options, and defines the use of backslash-quoted characters.

    In general, the behavior of echo is system and/or shell dependent if its arguments contain a backslash, or its first argument is -n or -e.

    The biggest problem with echo is when using it to output strings that the script got externally (e.g. user input, or reading from a file). These strings may have '\' characters in them for example. In this case, results may not be what you expect.

    print is available in some shells, although printf(1) is perhaps more portable. Additionally, a here document will give predictable results in that it will not expand escape sequences.

       cat <<EOF
    a string with\na newline and\ta tab
    So consider not using echo unless you are sure what will happen, given the shell you're using.

    1. How can I send e-mails with attached files?

       a. Use uuencode
          This is the simplest way to do this. For example
          $ uuencode surfing.jpeg surfing.jpeg | mail someone@some.where
          To send regular text as well
          $ (cat mailtext; uuencode surfing.jpeg surfing.jpeg) |
            mail someone@some.where
       b. Use MIME
          $ metasend -b -t someone@some.where -s "Hear our son!" \
            -m audio/basic -f
          These examples are adapted from
          goes into much more detail about this.
       c. Use pine (with a patch) or mutt

    2. How can I generate random numbers in shell scripts?

    This depends on the shell, and the facilities available from the OS. a. Some shells have a variable called RANDOM, which evaluates to a different value every time you dereference it. If your shell has this variable, $ number=$RANDOM will produce a random number. b. Some systems have a /dev/urandom device, which generates a stream of bits. This can be accessed using the dd(1) utility. An example of this (from a more extensive discussion of different techniques at n=`dd if=/dev/urandom bs=1 count=4 2>/dev/null | od -t u4 | \ awk 'NR==1 {print $2}'` also: od -vAn -N4 -tu4 < /dev/urandom c. Use a utility such as awk(1), which has random number generation included. This approach is the most portable between shells and operating systems. awk 'BEGIN {srand();print rand()}' Note that this doesn't work with older versions of awk. This requires a version supporting the POSIX spec for srand(). For example, on Solaris this will not work with /usr/bin/awk, but will with nawk or /usr/xpg4/bin/awk. Also, if you call this line more than once within the same second, you'll get the same number you did the previous time. ======================================================================

    3. How can I automatically transfer files using FTP with error checking?

    First, there are tools to do that: curl, wget, lftp, ncftp. But, they are generally not part of the base system (you need to install them). zsh (version 4 and above) provides a FTP facility, see "info -f zsh -n 'zsh/zftp Module'" #! /usr/bin/zsh zftp open host user passwd || exit zftp get /remote/file > /local/file; r=$? zftp close && exit r With your system "ftp" command, two ways: 1- using "ftp -n". Without the -n option, ftp expects user interaction to enter the password, so you'd need to use "expect". With "-n", you provide the user and password as any other FTP command. #! /bin/sh ftp -n << EOF open user anonymous ${LOGNAME:-`who am i`}@ binary get /remote/file /local/file bye EOF The error checking can't be made correctly (if "open" fails, the "user" command will be still sent even if it shouldn't). 2- using ~/.netrc If you put: << machine login mylogin password mypasswd macdef init binary get /remote/file /local/file bye >> (with the trailing empty line) in your ~/.netrc (ensure it's not world readable) and then run "ftp", ftp will find the matching "machine" entry in your ~/.netrc and use the parameters provided there to make the ftp transaction. Those work at least on Linux, FreeBSD, Solaris, HPUX ======================================================================

    4. How can I remove whitespace characters within file names?

    File names in unix can contain all kinds of whitespace characters, not just spaces. The following examples only work with spaces, adjust accordingly. a. Use the substitution capabilities of awk, sed, et al. f=`printf '%s\n' "$filename" | sed 's/ /_/g'` f=`printf '%s\n' "$filename" | awk '{gsub(" ","_");print $0}'` f=`printf '%s\n' "$filename" | tr ' ' _` Add characters to the tr command line as needed (see the man page for tr to find out the available escape sequences). Additionally (although not exactly a one-liner) f=`tr ' ' _ <<EOF $filename EOF ` See section 0a "Notes about using echo" for why echo is not used here. b. Use the substitution capabilities of the shell if it has them. Check the man page for your shell (probably under a section named something like "Parameter expansion") to see. For example: f=${filename// /_} With zsh: autoload -U zmv zmv '* *' '$f:gs/ /_/' It should be noted that the zmv solution renames the files (call mv internally and adress several problems that may arise) while the other solutions only update a variable (and then, renaming the files may involve a quite complicated script to do it reliably). ======================================================================

    5. How can I automate a telnet session?

    This is outside the realm of shell programing, per se. You need a more special purpose scripting language such as expect. See Perl scripts can also do this with the Telnet module from CPAN. ======================================================================

    6. How do I do date arithmetic?

    This depends on exactly what you have in mind. a. Finding yesterday's date The GNU version of date has some nice features in this respect. For example To find yesterday's date $ date --date yesterday To find tomorrow's date $ date --date tomorrow See the man page for GNU date for other options. It can also provide dates more than one day in the past/future. The FreeBSD version of date also provides extensions that can do things like this. $ date Wed Oct 22 13:48:29 CDT 2003 $ date -v-1d Tue Oct 21 13:45:16 CDT 2003 Playing with the TZ variable isn't a reliable method. If you need to do something like this, but don't have GNU or FreeBSD date available, see section g. "Arbitrary date arithmetic". b. Determining relative ages of files If you want to determine whether or not one file is older than another, you can (with bash, pdksh, ksh93) do $ [[ file1 -ot file2 ]] && echo file1 is older or you can use find to search a directory tree for files that are newer/older than some file: $ find . -name '*.c' -newer test.c c. Finding elapsed time If you want to find elapsed time, perhaps because you want to know when some operation has timed out, some shells (bash, ksh zsh [,??]) have a SECONDS variable which tell how many seconds have elapsed since the invocation of the shell, or since the last time it was set. ksh93 has a floating point SECONDS which is locale dependent. In zsh 4.1 and above one can be made floating point with: float SECONDS zsh 4.1 and above also has $EPOCHSECONDS for seconds since 1970-1-1 0:0:0 UTC (see zsh/datetime module). d. Determining leap year A leap year in the Gregorian calendar is defined as a year which is evenly divisible by 4, however, if it's also evenly divisible by 100 then it's not a leap year unless it's also evenly divisible by 400. It gets worse than that, actually, but this is as far as I go :-). In the Julain calendar which was used before in Europe, only the years divisible by 4 where leap years. The standard "cal" utility performed the switch between Julian and Gregorian calendar in september 1752 (see cal 9 1752) which corresponds to the date used in England. The Gregorian calendar (created by Pope Gregory III) was first used in 1582 in many other countries. One possibility for a ksh function to do this (after 1600 AD/CE) is isleap() { y=$1 four=$(( $y % 4 )) hundred=$(( $y % 100 )) fourhundred=$(( $y % 400 )) if [ $four -eq 0 ];then if [ $hundred -eq 0 ];then if [ $fourhundred -eq 0 ];then echo leap year else echo not a leap year fi else echo leap year fi else echo not a leap year fi } Or, valid with any date with the same calendar switch day as POSIX cal's (POSIX syntax): is_leap_year() # args: year # NB: year before year 1 is year -1, not 0. { [ "$1" -lt 0 ] && set -- "$(($1 + 1))" [ "$(($1 % 4))" -eq 0 ] && { [ "$(($1 % 100))" -ne 0 ] || [ "$(($1 % 400))" -eq 0 ] \ || [ "$1" -le 1752 ] } } Or in any Bourne shell: is_leap_year() { ## USAGE: is_leap_year [year] isl_year=${1:-`date +%Y`} case $isl_year in *0[48] |\ *[2468][048] |\ *[13579][26] |\ *[13579][26]0|\ *[2468][048]00 |\ *[13579][26]00 ) _IS_LEAP_YEAR=1 return 0 ;; *) _IS_LEAP_YEAR=0 return 1 ;; esac } On FreeBSD, use the -f option to date(1) to pass in the (supposed) February 29 in the current year and then print it the day of the month again to see if there really is such a date (note that you need -j as well as -f, otherwise date(1) thinks you want to set the clock): if [ $(date -jf%Y%m%d $(date +%Y0229) +%d) = 29 ]; then echo Leap year! fi e. Determining the last day of a month. There are a number of possibilities for doing this which have been mentioned in the group. The following is a sampling: In any Bourne-type shell (in conjunction with is_leap_year() as given above, when month is February): days_in_month() { ## USAGE: days_in_month [month [year]] if [ -n "$1" ] then dim_m=$1 dim_y=$2 else eval `date "+dim_m=%m dim_y=%Y"` fi case $dim_m in 9|09|4|04|6|06|11) _DAYS_IN_MONTH=30 ;; 1|01|3|03|5|05|7|07|8|08|10|12) _DAYS_IN_MONTH=31 ;; 2|02) is_leap_year ${dim_y:-`date +%Y`} && _DAYS_IN_MONTH=29 || _DAYS_IN_MONTH=28 ;; esac [ ${SILENT_FUNCS:-0} -eq 1 ] || echo $_DAYS_IN_MONTH } With GNU date: year=2003 month=9 date -d "$year/$month/1 +1 month -1 day" +%d With FreeBSD date use the -v-1d option to date(1) to get the day before the first day of the next month: $ MONTH=12 $ date -v-1d -jf%Y-%m-%d $(date +%Y-$(((MONTH+1)%12))-01) +%d 31 In the shell using cal (But beware of implementations of cal which print more than one month at a time): month=9 ; year=2003 # adjust ## for lday in `cal $month $year` ; do : ; done echo $lday ## or set -- `cal $month $year` ; eval lday=\${$#} echo $lday In ksh, bash and zsh: : $(cal) days_in_month=$_ In zsh: days_in_month=${$(cal)[-1]} f. Determining the day of the week for a given date. This algorithm is known as Zeller's congruence. An explanation of it is available from the Dictionary of Algorithms and Data Structures web page at NIST: Also, a fuller explanation is available at An example in C, with a short explanation, is given at A shell (ksh93) implementation of a homework assignment (given for illustration only - don't turn this in as yours - you might be sorry if it's wrong :-) dayofweek() { # Implementation of a homework assignment given at # # # call with day: 1 - 31 # month: March = 1, Jan & Feb are months 11 and # 12 of the previous year. # year: The year of the century # c: The previous century # # For example, for July 4, 1989, # m = 5, d = 4, y = 89, and c = 19, # while for January 25, 1989, # m = 11, d = 25, y = 88, and c = 19. # # The output is the day of the week with Sunday = 0, # Monday = 1, etc. d=$1 m=$2 y=$3 c=$4 A=$(( ($m * 13 - 1) / 5 )) B=$(( $y / 4 )) C=$(( $c / 4 )) D=$(( $A + $B + $C + $d + $y - ($c * 2) )) echo $(( $D % 7 )) } On FreeBSD, use the -f option to date(1) to pass in the date of interest and +%A to print the day of the week: $ date -jf%Y-%m-%d 2000-01-01 +%A Saturday (Use +%u or +%w if you want the weekday as a number. See the strftime(3) manpage for details.) g. Arbitrary date arithmetic To do arbitrary date calculations is more complicated. One possibility is to call an external utility, or a program in another scripting language, which has this built in. For example, perl has wrappers for the unix time functions built in, so it can provide some relief in this regard. C programs can also be easily written to do date arithmetic (see the examples section). One thing to keep in mind, however, is that unix time functions are, strictly speaking, limited to the range of time between January 1 1970 at midnight, and January 19, 2038 at 3:14:07. C/Perl programs which calculate dates outside this range might work, or they might not, that would depend on the implementation. To do arbitrary date arithmetic in the shell itself is also possible. An article provided on the web by SysAdmin magazine describes one way to do this. Another possibility is given in the examples section, from See also On FreeBSD, the -f and -v options to date(1) cover most things you might want to do, with the caveat that only dates within the range mentioned above are defined. Dates outside that range are not guaranteed to work. Also, zsh 4.1 and above has the zsh/datetime module that provides the $EPOCHSECONDS and the strftime function. h. Getting the number of seconds since the epoch - GNU date has the %s format option which returns the epoch time. - More portably, use awk. awk 'BEGIN {srand(); printf("%d\n", srand())}' This works because srand() sets its seed value with the current epoch time if not given an argument. It also returns the previous seed value, so the second call gives the epoch time. Note that this doesn't work with older versions of awk. This requires a version supporting the POSIX spec for srand(). For example, on Solaris this will not work with /usr/bin/awk, but will with nawk or /usr/xpg4/bin/awk. Depending on scheduling, when the call is actually executed, etc, this might be off by a second. - Another way is to use perl if you have it. perl -le 'print time' - Also, zsh 4.1 and above has the zsh/datetime module that provides the $EPOCHSECONDS and the strftime function. ======================================================================

    7. Why did someone tell me to RTFM?

    Because you didn't :-) RTFM is part of Usenet lingo, and means "Read The F-ing Manual". Generally people say this when someone asks a question that is asked so often, and is answered plainly in some relevant man page, that they're tired of seeing it asked. So RTFM, and the FAQs first before asking. Also, if you're new to the group, search Google Groups before asking questions. And please don't post your homework questions to the group unless you've tried to figure them out, and have some specific questions. People will generally be happy to help you with your homework if you post what you've got and ask specific questions. ======================================================================

    8. How do I create a lock file?

    Very carefully :-) The scheduler can stop one process in the middle of a non-atomic operation, and run another one, which wants to perform the same operation. The second one, having a full timeslice, might finish the operation. When control returns to the first process, confusion will reign. The trick is to do something atomic, so that this won't happen. There are a couple ways to do this. One is to create a directory instead of a file, the other is to create a symbolic link. Both operations are defined to be atomic by POSIX/SUS, by virtue of the fact that they both require invocation of the corresponding system calls, which are atomic. Beware of trying to create ANY kind of lock file on an NFS partition. NFS pretty much eliminates anything like atomicity. If you're going to create a lock file, make sure you're doing it on a local partition, such as /tmp. Netscape/Mozilla uses the symbolic link method for its lockfile (in spite of the fact that it creates it in the user's home directory, which may be NFS mounted). When it starts up it creates a file named for the IP address of the machine it's running on, and the pid of the creating process. Then it tries to create a symbolic link named "lock", which points to that file. If this symlink already exists, link(2) will return an error. In a script this would work something like

       touch /tmp/xxx
       ln -s /tmp/xxx /tmp/lockfile 2>/dev/null
       rm /tmp/xxx
       if [ $ret -ne 0 ];then
         echo lockfile already exists
         exit 1
         echo success

    If you have procmail installed, another possibility is the lockfile(1) command that comes with it. ======================================================================

    9. How can I convert DOS text files to unix, and vice versa?

    Unix text files consist of lines delimited by an LF ("line-feed") character (ASCII 10). DOS uses the two characters CR LF ("carriage return", "line feed"; ASCII 13, 10) for the same purpose. To convert a DOS text into unix text format, the CR characters (control-M) at the end of a line have to be removed. To create a DOS text file, the CR character should be added. A couple ways to remove CR characters: sed 's/^M$//' dos.txt > unix.txt tr -d '\r' < dosfile > unixfile To add them: sed 's/$/^M/' unix.txt > dos.txt Note that "^M" in this case is an embedded control character, (CR, ASCII 13). Many shells allow embedding control characters by entering ^V first (control-V), resulting in the sequence ^V^M for entering "^M". However, zsh, bash or ksh93 allow for: sed $'s/$/\r/' There is one special case to be considered: DOS text files sometimes contain an explicit end-of-file character ^Z (ASCII 26, or octal 32), which has no correspondent character for unix text files, where the end-of-file condition is determined implicitly. To remove that as well as the CR characters: tr -d '\r\032' < dosfile > unixfile Note that sed does not understand that notation, but awk does, and one simple way to do the opposite conversion is $ awk '{printf "%s\r\n" $0}END{printf "%c", 26}' unixfile > dosfile This assume a not-quite-ancient awk, in practice anything but Solaris /bin/awk (use nawk or /usr/xpg4/bin/awk in Solaris). Finally, your system may come with utilities named something like dos2unix and unix2dos, or d2u dos2unix fromdos non-standard utilities, and GNU recode: recode /CRLF ======================================================================

    10. How can a shell prompt be set up to change the title of xterm?

    Gives escape sequences for xterm. For example, to change the name of the current window to "XXX" (in bash), do $ echo -en "\033]2;XXX\007" or, more portably: $ printf '%b' '\e]2;XXX\a' See also "Why doesn't echo do what I want?"