From: j.p.h@comcast.net (Joe Halpin)
Newsgroups: comp.unix.shell
Subject: comp.unix.shell FAQ - Answers to Frequently Asked Questions
         Part 2
Summary: This posting answers questions which come up with some
         frequency on comp.unix.shell. It should be read by
         anyone with a question about shell programming before
         posting questions to the newsgroup.
Followup-To: comp.unix.shell

Archive-name: unix-faq/shell/sh
Posting-Frequency: monthly
Version: $Id: cus-faq-2.html,v 1.4 2005/09/01 17:39:36 jhalpin Exp $
Maintainer: Joe Halpin 


Please read the introduction in the first section of this document. This section assumes you have read that introduction.

====================================================================== 11. How do I get the exit code of cmd1 in cmd1|cmd2 First, note that cmd1 exit code could be non-zero and still don't mean an error. This happens for instance in cmd | head -1 you might observe a 141 (or 269 with ksh93) exit status of cmd1, but it's because cmd was interrupted by a SIGPIPE signal when "head -1" terminated after having read one line. To know the exit status of the elements of a pipeline cmd1 | cmd2 | cmd3 a. with zsh: The exit codes are provided in the pipestatus special array. cmd1 exit code is in $pipestatus[1], cmd3 exit code in $pipestatus[3], so that $? is always the same as $pipestatus[-1]. b. with bash: The exit codes are provided in the PIPESTATUS special array. cmd1 exit code is in `, fd4 goes to the pipe # whose other end is read and passed to eval; # fd1 is the normal standard output preserved # the line before with exec 3>&1 exec 4>&1 >&3 3>&- { cmd1 4>&-; echo "ec1=$?;" >&4 } | { cmd2 4>&-; echo "ec2=$?;" >&4 } | cmd3 echo "ec3=$?;" >&4 ` d. with a POSIX shell You can use this function to make it easier: run() { j=1 while eval "\${pipestatus_$j+:} false"; do unset pipestatus_$j j=$(($j+1)) done j=1 com= k=1 l= for a; do if [ "x$a" = 'x|' ]; then com="$com { $l "'3>&- echo "pipestatus_'$j'=$?" >&3 } 4>&- |' j=$(($j+1)) l= else l="$l \"\$$k\"" fi k=$(($k+1)) done com="$com $l"' 3>&- >&4 4>&- echo "pipestatus_'$j'=$?"' exec 4>&1 eval "$(exec 3>&1; eval "$com")" exec 4>&- j=1 while eval "\${pipestatus_$j+:} false"; do eval "[ \$pipestatus_$j -eq 0 ]" || return 1 j=$(($j+1)) done return 0 } use it as: run cmd1 \| cmd2 \| cmd3 exit codes are in $pipestatus_1, $pipestatus_2, $pipestatus_3 ====================================================================== 12. Why do I get "script.sh: not found" a. While script starts with "#!/bin/sh" (^M issue) That's the kind of error that occurs when you transfer a file by FTP from a MS Windows machine. On those systems, the line separator is the CRLF sequence, while on unix the line separator is LF alone, CR being just another ordinary character (the problem is that it is an invisible one on your terminal (where it actually moves the cursor to the beginning of the line) or in most text editors or pagers). So, if a MSDOS line is "#!/bin/sh", when on a Unix system, it becomes "#!/bin/sh<CR>" (other names for <CR> are \r, \015, ^M, <Ctrl-M>). So, if you run the file as a script, the system will look in /bin for an interpreter named "sh<CR>", and report it doesn't exist. $ sed 'l;d;q' < script.sh #!/bin/sh\r$ shows you the problem ($ marks the end of line, \r is the CR character). b. PATH issue Sometimes a shell is installed someplace other than /bin or /usr/bin. For example, a shell which was not part of the OS installation might be installed into /usr/local/bin. If the script was written on a machine which had ksh located in /usr/bin, but was run on a machine where ksh was located in /usr/local/bin, the shebang line would not resolve correctly. This is unlikely to occur when using sh. However, if the shell is bash, zsh, et al, it might be installed in different places on different machines. One way around this is to use the env command in the shebang line. So instead of #!/bin/sh use #!/usr/bin/env sh Of course, env might itself live in some other directory than /usr/bin, but it's not likely. ====================================================================== 13. Why doesn't echo do what I want? See also section 0a "Notes about using echo" The echo command is not consistent from shell to shell. For example, some shells (bash, pdksh [,?]) use the following arguments -n suppress newline at the end of argument list -e interpret backslash-escaped characters -E disable interpretation of backslash-escaped characters, even on systems where interpretation is the default. However, pdksh also allows using \c to disable a newline at the end of the argument list. POSIX only allows \c to be used to suppress newlines, and doesn't accept any of the above arguments. ksh88 and ksh93 leave the interpretation of backslash-escaped characters up to the implementation. [descriptions of behavior of other shells welcome] In short, you have to know how echo works in any environment you choose to use it in, and its use can therefore be problemmatic. If available, print(1) or printf(1) would be better. ====================================================================== 14. How do I loop through files with spaces in their name? So, you're going to loop through a list of files? How is this list stored? If it's stored as text, there probably was already an assumption about the characters allowed in a filename. Every character except '\0' (NUL) is allowed in a file path on Unix. So the only way to store a list of file names in a file is to separate them by a '\0' character (if you don't use a quoting mechanism as for xargs input). Unfortunately most shells (except zsh) and most standard unix text utilities (except GNU ones) can't cope with "\0" characters. Moreover, many tools, like "ls", "find", "grep -l" output a \n separated list of files. So, if you want to postprocess this output, the simpler is to assume that the filenames don't contain newline characters (but beware that once you make that assumption, you can't pretend anymore your code is reliable (and thus can't be exploited)). So, if you've got a newline separated list of files in a list.txt file, Here are two ways to process it: 1- while IFS= read -r file <&3; do something with "$file" # be sure to quote "$file" done 3< list.txt (if your read doesn't have the "-r" option, either make another assumption that filenames don't contain backslashes, or use: exec 3<&0 sed 's/\\/&&/g' < list.txt | while IFS= read file; do something with "$file" <&3 3<&- done ) 2- IFS=" " # set the internal field separator to the newline character # instead of the default "<space><tab><NL>". set -f # disable filename generation (or make the assumption that # filenames don't contain *, [ or ? characters (maybe more # depending on your shell)). for file in $(cat < list.txt); do something with "$file" # it's less a problem if you forget to # quote $file here. done Now, beware that there are things you can do before building this list.txt. There are other ways to store filenames. For instance, you have the positional parameters. with: set -- ./*.txt you have the list of txt files in the current directory, and no problem with weird characters. Looping through them is just a matter of: for file do something with "$file" done You can also escape the separator. For instance, with find . -exec sh -c 'printf %s\\n "$1" | sed -n '"':1 \$!{N;b1 } s/|/|p/g;s/\n/|n/g;p'" '{}' '{}' \; instead of find . -print you have the same list of files except that the \n in filenames are changed to "|n" and the "|" to "|p". So that you're sure there's one filename per line and you have to convert back "|n" to "\n" and "|p" to "|" before referring to the file. ====================================================================== 15. how do I change my login shell? See http://www.faqs.org/faqs/unix-faq/shell/shell-differences Unless you have a very good reason to do so, do not change root's default login shell. By "default login shell" is meant the shell recorded in /etc/passwd. Note that "I login as root but don't like the default shell" isn't a good reason. The default shell for root is one which will work in single user mode, when only the root partition is mounted. This is one of the contexts root works in, and the default shell must accommodate this. So if you change it to a dynamically linked shell which depends on libraries that are not in the root partition, you're asking for trouble. The safest way of changing root's shell is to login as root and then # SHELL=/preferred/shell; export SHELL # exec <your preferred shell with login flag> e.g. # SHELL=/usr/bin/ksh; export SHELL # exec $SHELL -l Another possibility is to add something to root's .profile or .login which checks to see if the preferred shell is runnable, and then execs it. This is more complicated and has more pitfalls than simply typing "exec <shell>" when you login though. For example, one of the libraries that the desired shell relies on might have been mangled, etc. One suggestion that has been made is if [ -x /usr/bin/ksh ]; then SHELL=/usr/bin/ksh; export SHELL ENV=/root/.kshrc; export ENV /usr/bin/ksh -l && exit fi A safer way is to try to run a command with the preferred shell before you try to exec it. This will lessen the possibility that the shell or one of the libraries it depends on has been corrupted, or that one of the libraries it depends on is not in the available mounted partitions. if [ -x /usr/bin/ksh ]; then /usr/bin/ksh -c echo >/dev/null 2>&1 if [ $? -eq 0 ];then SHELL=/usr/bin/ksh; export SHELL ENV=/root/.kshrc; export ENV /usr/bin/ksh -l && exit fi fi Another common approach is to create another user with UID 0. For example, FreeBSD systems commonly create an account named toor, which can be setup however you like. This bypasses the controversy. ====================================================================== 16. When should I use a shell instead of perl/python/ruby/tcl... a. Portability In many cases it can't be assumed that perl/python/etc are installed on the target machine. Many customer sites do not allow installation of such things. In cases like this, writing a shell script is more likely to be successful. In the extreme, writing a pure Bourne shell script is most likely to succeed. b. Maintainability If the script is one which serves some important purpose, and will need to be maintained after you get promoted, it's more likely that a maintainer can be found for a shell script than for other scripting languages (especially less used ones such as ruby, rexx, etc). c. Policy Sometimes you're just told what to use :-) ====================================================================== 17. Why shouldn't I use csh? http://www.grymoire.com/Unix/CshTop10.txt http://www.grymoire.com/Unix/Csh.html#uh-0 http://www.faqs.org/faqs/unix-faq/shell/csh-whynot/ ====================================================================== 18. How do I reverse a file? Non-standard commands to do so are GNU tac and "tail -r". sed '1!G;h;$!d' is subject to sed limitation on the size of its hold space and is generally slow. The awk equivalent would be: awk '{l[n++]=$0}END{while(n--)print l[n]}' It stores the whole file in memory. The best approach in terms of efficiency portability and resource cosumption seems to be: cat -n | sort -rn | cut -f2- "cat -n" is not POSIX but appears to be fairly portable. Alternatives are "grep -n '^'", "awk '{print NR,$0}'". Also, nl can be used as nl -ba -d' ' i.e. NL as the delimiter. ====================================================================== 19. how do I remove the last n lines? First we need to tell the code how many lines we want to cut from the bottom of a file. X=10 Then We can do this: head -n $(( $(wc -l < file ) - $X )) file >$$ \ && cat $$ >file && rm $$ The break down: 1) $(wc -l < file) Find out how many lines are in the file. Need to use redirection so wc won't print the file name. 2) $(( $lines_in_file - $X )) Take the output from step one and do some math to find out how many lines we want to have when all is said and done. 3) head -$lines_when_said_and_done file extracts all but the unwanted lines from the file, and >$$ puts those lines into a temp file that has the name of the pid of the current shell. 4) && cat $$ > file if everything has worked so far then cat the temp file into the original file. This is better than mv or cp because it insures that the permissions of the temp file do not override with the perms of the original file. 5) && rm $$ Remove the temp file. AWK solutions: awk 'NR<=(count-12)' count="`awk 'END{print NR}' file`" file awk 'NR>n{print a[NR%n]} {a[NR%n]=$0}' n=12 file awk 'BEGIN{n=12} NR>n{print a[NR%n]} {a[NR%n]=$0}' file Whenever a line is read, the line that came 12 lines ago is printed, and then overwritten with the newly read line, using an rolling array indexed 0..11. See also question 26. for information about setting awk variables on the command line. $SHELL/sed/mv solutions: L=`wc -l <file` DL=`expr $L - 11` sed "$DL,\$d" file L=`wc -l <file` DL=`expr $L - 12` sed "${DL}q" file sed "`expr \`wc -l <file\` - 12`q" file sed -n -e :a -e '1,12{N;ba' -e '}' -e 'P;N;D' file The last solution is basically same algorithm as the rolling array awk solutions, and shares with them the advantage that the file is only read once - they will even work in a pipe. There may be limitations in sed's pattern space which would make this unusable however. PERL solution: perl -ne' print shift @x if @x == 12; push @x, $_ ' file Using GNU dd: ls -l file.txt | { IFS=" " read z z z z sz z last=`tail -10 file.txt | wc -c` dd bs=1 seek=`expr $sz - $last` if=/dev/null of=file.txt } ====================================================================== 20. how do I get file size, or file modification time? If your system has stat(1), use it. On Linux, for example: filesize=$(stat -c %s -- filename) or use cut, awk, etc on the output. Probably the most portable solution is to use wc filesize=`wc -c < "$file"` ls may be able to tell you what you want to know. From the man page for ls we learn about "ls -l" the file mode, the number of links to the file, the owner name, the group name, the size of the file (in bytes), the timestamp, and the filename. For the file size in human readable formate use the "-h" option. For example: $ ls -l timeTravel.html -rw-rw-r-- 1 user user 20624 Jun 19 2002 timeTravel1.html so to get the file size: $ set -- `ls -l timeTravel1.html` $ echo $5 20624 Note that ls doesn't always give the date in the same format. Check the man page for ls on your system if that matters. If you're interested in the file modification time. Another possibility is to use GNU ls, which has a -T option giving complete time information for the file, including month, day, hour, minute, second and year. See also GNU find (-printf), GNU stat, GNU date (-r) and zsh stat (+mtime). On FreeBSD 4, you can use the -lT option to ls(1) to get the full modification time and the -f option to date(1) to parse it, for example: $ FILE=/etc/motd $ date -jf'%b %d %T %Y' +%Y-%m-%dT%T \ $(ls -lT $FILE|tr -s ' ' \\t|cut -f6-9) 2003-09-09T16:04:06 Adjust syntax as needed if your shell is FreeBSD sh ====================================================================== 21. How do I get a process id given a process name? Or, how do I find out if a process is still running, given a process ID? There isn't a reliable way to to this portably in the shell. Some systems reuse process ids much like file descriptors. That is, they use the lowest numbered pid which is not currently in use when starting a new process. That means that the pid you're looking for is there, but might not refer to the process you think it does. The usual approach is to parse the output of ps, but that involves a race condition, since the pid you find that way may not refer to the same process when you actually do something with that pid. There's no good way around that in a shell script though, so be advised that you might be stepping into a trap. One suggestion is to use pgrep if on Solaris, and 'ps h -o pid -C $STRING' if not, and your ps supports that syntax, but neither of those are perfect or ubiquitous. The normal solution when writing C programs is to create a pid file, and then lock it with fcntl(2). Then, if another program wants to know if that program is really running, it can attempt to gain a lock on the file. If the lock attempt fails, then it knows the file is still running. We don't have options in the shell like that, unless we can supply a C program which can try the lock for the script. Even so, the race condition described above still exists. ====================================================================== 22. How do I get a script to update my current environment? Processes in unix cannot update the environment of the process that spawned them. Consequently you cannot run another process normally and expect it to do that, since it will be a child of the running process. There are a couple ways it can be done though. a. source the script This means that you use whatever syntax your shell has to read the desired script into the current environment. In Bourne derived shells (sh/ksh/bash/POSIX/etc) the syntax would be $ . script In csh type shells this would be $ source script b. use eval The eval command constructs a command by evaluating and then executing a set of arguments. If those arguments evaluate to a shell variable assignment, the current environment will be updated. For example --- exportFoo #!/bin/ksh echo export FOO=bar If you run this like eval "`exportFoo`" the value of FOO will be set to 'bar' in the calling shell. Note that the quotes are recommended as they will preserve any whitespace that may be present in the variables being set. However, be aware that eval'ing a script written in another shell could turn out to be the wrong thing to do. For example, eval'ing this from a ksh script #!/bin/csh echo setenv FOO bar Would not do what you expect. It would produce an error, because ksh doesn't have a setenv command. ====================================================================== 23. How do I rename *.foo to *.bar? Naive examples in ksh/bash (which may or may not work many times) $ ls *.foo | while read f;do mv "$f" "${f%.*}".bar More generically $ ls *.foo | while read f;do mv "$f" `basename "$f" .foo`.bar However, these examples contain a potentially unnecessary use of ls (ie, if the number of files is small enough to not overflow the command line buffer), and will fail if any file names contain a newline, or if there are leading or trailing spaces. An alternative is: for file in *.foo do mv -- "$file" "`basename -- \"$file\" .foo`.bar" done Also, tests for existence of files should also be incorporated, e.g.: for file in ./*.foo do newfile=`basename "$file" .foo`.bar [ -f "$file" ] || continue [ -f "$newfile" -o -d "$newfile" ] && continue mv "$file" "$newfile" done In some linux distributions you may be able to use the rename command $ rename .foo .bar * If not (Debian, for one, comes with a perl version of rename that won't work with that command line) try $ rename 's/.foo/.bar/' *.foo More options, and much more discussion about this, is available from http://www.faqs.org/faqs/unix-faq/faq/part2/section-6.html Note that for file specifications which don't match existing files, the shell usually responds with something like "ls: *.foo: No such file or directory", which will mess up your processing of file names. One possibility is #! /bin/sh set x [*].foo ./*.foo case "$2$3" in "[*].foo./*.foo") ;; *) shift 2 for file do repl=`basename "$file" .foo`.bar mv "$file" "$repl" done;; esac Except that contrary to (zsh) mmv or zmv it doesn't check for file overwriting and fails for filenames with NLs before the "." and doesn't handle dotfiles. ====================================================================== 24. How do I use shell variables in awk scripts Short answer = either of these, where "svar" is a shell variable and "avar" is an awk variable: awk -v avar="$svar" '... avar ...' file awk 'BEGIN{avar=ARGV[1];ARGV[1]=""}... avar ...' "$svar" file depending on your requirements for handling backslashes and handling ARGV[] if it contains a null string (see below for details). Long answer = There are several ways of passing the values of shell variables to awk scripts depending on which version of awk (and to a much lesser extent which OS) you're using. For this discussion, we'll consider the following 4 awk versions: oawk (old awk, /usr/bin/awk and /usr/bin/oawk on Solaris) nawk (new awk, /usr/bin/nawk on Solaris) sawk (non-standard name for /usr/xpg4/bin/awk on Solaris) gawk (GNU awk, downloaded from http://www.gnu.org/software/gawk) If you wanted to find all lines in a given file that match text stored in a shell variable "svar" then you could use one of the following: a) awk -v avar="$svar" '$0 == avar' file b) awk -vavar="$svar" '$0 == avar' file c) awk '$0 == avar' avar="$svar" file d) awk 'BEGIN{avar=ARGV[1];ARGV[1]=""}$0 == avar' "$svar" file e) awk 'BEGIN{avar=ARGV[1];ARGC--}$0 == avar' "$svar" file f) svar="$svar" awk 'BEGIN{avar=ENVIRON["svar"]}$0 == avar' file g) awk '$0 == '"$svar"'' file The following list shows which version is supported by which awk on Solaris (which should also apply to most other OSs): oawk = c, g nawk = a, c, d, f, g sawk = a, c, d, f, g gawk = a, b, c, d, f, g Notes: 1) Old awk only works with forms "c" and "g", both of which have problems. 2) GNU awk is the only one that works with form "b" (no space between "-v" and "var="). Since gawk also supports form "a", as do all the other new awks, you should avoid form "b" for portability between newer awks. 3) In form "c", ARGV[1] is still getting populated, but because it contains an equals sign (=), awk changes it's normal behavior of assuming that arguments are file names and now instead assumes this is a variable assignment so you don't need to clear ARGV[1] as in form "d". 4) In light of "3)" above, this raises the interesting question of how to pass awk a file name that contains an equals sign - the answer is to do one of the following: i) Specify a path, e.g. for a file named "abc=def" in the current directory, you'd use: awk '...' ./abc=def Note that that won't work with older versions of gawk or with sawk. ii) Redirect the input from a file so it's opend by the shell rather than awk having to parse the file name as an argument and then open it: awk '...' < abc=def Note that you will not have access to the file name in the FILENAME variable in this case. 5) An alternative to setting ARGV[1]="" in form "d" is to delete that array entry, e.g.: awk 'BEGIN{avar=ARGV[1];delete ARGV[1]}$0 == avar' "$svar" file This is slightly misleading, however since although ARGV[1] does get deleted in the BEGIN section and remains deleted for any files that preceed the deleted variable assignment, the ARGV[] entry is recreated by awk when it gets to that argument during file processing, so in the case above when parsing "file", ARGV[1] would actually exist with a null string value just like if you'd done ARGV[1]="". Given that it's misleading and introduces inconsistency of ARGV[] settings between files based on command-line order, it is not recommended. 6) An alternative to setting svar="$svar" on the command line prior to invoking awk in form "f" is to export svar first, e.g.: export svar awk 'BEGIN{avar=ENVIRON["svar"]}$0 == avar' file Since this forces you to export variables that you wouldn't normally export and so risk interfering with the environment of other commands invoked from your shell, it is not recommended. 7) When you use form "d", you end up with a null string in ARGV[1], so if at the end of your program you want to print out all the file names then instead of doing: END{for (i in ARGV) print ARGV[i]} you need to check for a null string before printing. or store FILENAMEs in a different array during processing. Note that the above loop as written would also print the script name stored in ARGV[0]. 8) When you use form "a", "b", or "c", the awk variable assignment gets processed during awks lexical analaysis stage (i.e. when the internal awk program gets built) and any backslashes present in the shell variable may get expanded so, for example, if svar contains "hi\there" then avar could contain "hi<tab>there" with a literal tab character. This behavior depends on the awk version as follows: oawk: does not print a warning and sets avar="hi\there" sawk: does not print a warning and sets avar="hi<tab>here" nawk: does not print a warning and sets avar="hi<tab>here" gawk: does not print a warning and sets avar="hi<tab>here" If the backslash preceeds a character that has no special meaning to awk then the backslash may be discarded with or without a warning, e.g. if svar contained "hi\john" then the backslash preceeds "j" and "\j" has no special meaning so the various new awks each would behave differently as follows: oawk: does not print a warning and sets avar="hi\john" sawk: does not print a warning and sets avar="hi\john" nawk: does not print a warning and sets avar="hijohn" gawk: prints a warning and sets avar="hijohn" 9) None of the awk versions discussed here work with form "e" but it is included above as there are older (i.e. pre-POSIX) versions of awk that will treat form "d" as if it's intended to access a file named "" so you instead need to use form "e". If you find yourself with that or any other version of "old awk", you need to get a new awk to avoid future headaches and they will not be discussed further here. So, the forms accepted by all 3 newer awks under discussion (nawk, sawk, and gawk) are a, c, d, f, and g. The main differences between each of these forms is as follows: |-------|-------|----------|-----------|-----------|--------| | BEGIN | files | requires | accepts | expands | null | | avail | set | access | backslash | backslash | ARGV[] | |-------|-------|----------|-----------|-----------|--------| a) | y | all | n | n | y | n | c) | n | sub | n | n | y | n | d) | y | all | n | n | n | y | f) | y | all | y | n | n | n | g) | y | all | n | y | n/a | n | |-------|-------|----------|-----------|-----------|--------| where the columns mean: BEGIN avail = y: variable IS available in the BEGIN section BEGIN avail = n: variable is NOT available in the BEGIN section files set = all: variable is set for ALL files regardless of command-line order. files set = sub: variable is ONLY set for those files subsequent to the definition of the variable on the command line requires access = y: variable DOES need to be exported or set on the command line requires access = n: shell variable does NOT need to be exported or set on the command line accepts backslash = y: variable CAN contain a backslash without causing awk to fail with a syntax error accepts backslash = n: variable can NOT contain a backslash without causing awk to fail with a syntax error expands backslash = y: if the variable contains a backslash, it IS expanded before execution begins expands backslash = n: if the variable contains a backslash, it is NOT expanded before execution begins null ARGV[] = y: you DO end up with a null entry in the ARGV[] array null ARGV[] = n: you do NOT end up with a null entry in the ARGV[] array For most applications, form "a" and "d" provide the most intuitive functionality. The only functional differences between the 2 are: 1) Whether or not backslashes get expanded on variable assignment. 2) Whether or not ARGV[] ends up containing a null string. so which one you choose to use depends on your requirements for these 2 situations. ====================================================================== 25. How do I get input from the user with a timeout? In bash or ksh93 you can use the read built-in with the "-t" option. In zsh, use the zsh/zselect module. You can also use your terminal capability to do that. { s=$(stty -g) stty -icanon min 0 time 100 var=$(head -n 1) stty "$s" } For a 10 second timeout (reset at each key press). ====================================================================== 26. How do I get one character input from the user? In bash this can be done with the "-n" option to read. In ksh93 it's read -N In zsh it's read -k More portably: OLDSTTY=$(stty -g) # save our terminal settings stty cbreak # enable independent processing of each input character ONECHAR=$(dd bs=1 count=1 2>/dev/null) # read one byte from standard in stty "$OLDSTTY" # restore the terminal settings Use the `something` format if your shell doesn't understand $(something). This reads from standard input, which may or may not be desirable. If you want to read from the terminal regardless of where standard input is, add "if=$(tty)" to the dd command. ====================================================================== 27. why isn't my .profile read? ~/.profile is only read for login shells. In short if you don't see a login prompt then your ~/.profile isn't being read. You can fix this by either porting all of the things in your ~/.profile to /etc/profile or your shells rc script such as ~/.bashrc or ~/.zshrc. You may have to set the ENV variable in your login shell to get the .*rc shell read. See the man page for your shell to understand how it works. ====================================================================== 28. why do I get "[5" not found in "[$1 -eq 2]"? Because you didn't RTFM :-) "[" is an alias for the "test" command. As such, it's called by a script like any other command (this applies even if test is builtin). Since the command line uses spaces to separate a command from its arguments, you have to put a space between '[' and its argument. So: $ [ -f xxx ] isn't the same as $ [-f xxx ] In the latter case, the shell will think that "[-f" is the command, not "[" with arguments "-f xxx ] ====================================================================== 29. How do I exactly display the content of $var (with a \n appended). A: on POSIX systems or with shells with builtin printf (bash2, ksh93, zsh4.1, dash...) printf '%s\n' "$var" (except for memory/environment full errors, should be expected to work at least if $var is not longer than LINE_MAX (supposed to be at least _POSIX2_LINE_MAX == 2048), no hardcoded limits in zsh/ksh/bash/dash builtins) ksh, zsh: print -r -- "$var" zsh: echo -E - "$var" Other bourne like shells: cat << EOF $var EOF (creates a temporary file and forks a process) expr "x$var" : 'x\(.*\)' (limited to 126 characters with some exprs, may return a non-null exit code). With ash: (unset a; ${a?$var}) 2>&1 printf %s "$var" # posix print -rn -- "$var" # zsh/ksh echo -nE - "$var" # zsh awk 'NR>1{print ""}{printf("%s",$0)}' << EOF $var EOF ====================================================================== 30. How do I split a pathname into the directory and file? The most portable way of doing this is to use the external commands dirname(1) and basename(1), as in pathname='/path/to/some/file' dir=`dirname "$pathname"` file=`basename "$pathname"` However, since this executes an external command, it's slower than using shell builtins (if your shell has them). For ksh, bash, zsh and POSIX shells the following will do the same thing more efficiently: pathname=/path/to/some/file file=${pathname##*/} To get the directory using the shell builtin, you should first ensure that the path has a '/' in it. case $pathname in */*) dir=${pathname%/*};; *) dir='' esac In zsh, (abd csh, tcsh), you have ${pathname:h} (head) ${pathname:t} (tail). ====================================================================== 31. How do I make an alias take an argument? In Bourne-derived shells aliases cannot take arguments, so if you need to be able to do that, define a shell function rather than an alias. Aliases are often used to reduce the need to type long command lines: alias prog='/opt/bin/prog -x -y -z --long-option' Or to make a command default to using certain parameters: alias ls='ls -F' Shell functions must be used when arguments are needed. For example, this will move one or more files to a Trash directory: trash() { mv -- "$@" ~/.Trash; } ====================================================================== 32. How do I deal with a file whose name begins with a weird character Do something to hide the weird character from the command being used. Assuming that command is rm, try things like rm ./-foo rm -- -foo rm -i -- * (and then decide what you want to delete interactively) If the weird character is not printable, the last option may be your best bet. Another possibility in that case is to pipe the output of ls into od -c and figure out what the weird character is. Then use sed to isolate it and nuke it. However, the rm -i approach is probably much safer. For more particulars, see the rm man page for your system. ====================================================================== 33. Why do I lose the value of global variables that are set in a loop. Given the following program #!/bin/sh x="this is the initial value of x" cat dataFile | while read line;do x="$line" done echo x = $x You may get the following for output x = this is the initial value of x This is because in the Bourne shell redirected control structures run in a subshell, so the value of x only gets changed in the subshell, and is lost when the loop ends. In other shells the same result may be seen because of the way pipelines are handled. In shells other than ksh (not pdksh) and zsh elements of a pipeline are run in subshells. In ksh and zsh, the last element of the pipeline is run in the current shell. An alternative for non-Bourne shells is to use redirection instead of the pipeline #!/bin/sh x="this is the initial value of x" while read line;do x="$line" done < dataFile echo x = $x With a Bourne shell you need to reassign file descriptors, so no pipline or redirection in the loop is involved. exec 3<&0 # save stdin exec < file while read line; do x=$line done exec 0<&3 # restore stdin Note that putting #!/bin/sh at the top of a script doesn't guarantee you're using the Bourne shell. Some systems link /bin/sh to some other shell. Check your system documentation to find out what shell you're really getting in this case. ====================================================================== 34. How do I batch a FTP download/upload? The best way to handle this is with ncftpput and ncftpget which are part of the ncftp program. ncftpput -u username -p password somewebsite.com /pics *jpg The above usage of the username and password is not recomend though as it will be seen by anyone using "ps" while the script is running. ncftp has a way to handle that as well. Just create a file with the information in the following formate: host somewebsite.com user username pass password Then just use the -f option on the ncftp program: ncftpput -f /home/username/somefile somewebsite.com /pics *jpg ncftp can be found at http://freshmeat.net/projects/ncftp/ If you want to do this interactively, there's no need to keep the password in a file. For example, if you're building a program on one machine, but testing it on another, and you have to keep ftp'ing the files, you can cut down on typing by doing something like #!/bin/ksh ftp -n $remote_host <<EOF user <username> cd <path to build directory> prompt bin mget "$@" bye EOF The ftp program will automatically ask you for the password, then do the rest for you.