Chapter 19. I/O Redirection

There are always three default "files" open, stdin (the keyboard), stdout (the screen), and stderr (error messages output to the screen). These, and any other open files, can be redirected. Redirection simply means capturing output from a file, command, program, script, or even code block within a script (see Example 3-1 and Example 3-2) and sending it as input to another file, command, program, or script.

Each open file gets assigned a file descriptor. [1] The file descriptors for stdin, stdout, and stderr are 0, 1, and 2, respectively. For opening additional files, there remain descriptors 3 to 9. It is sometimes useful to assign one of these additional file descriptors to stdin, stdout, or stderr as a temporary duplicate link. [2] This simplifies restoration to normal after complex redirection and reshuffling (see Example 19-1).

   1    COMMAND_OUTPUT >
   2       # Redirect stdout to a file.
   3       # Creates the file if not present, otherwise overwrites it.
   4 
   5       ls -lR > dir-tree.list
   6       # Creates a file containing a listing of the directory tree.
   7 
   8    : > filename
   9       # The > truncates file "filename" to zero length.
  10       # If file not present, creates zero-length file (same effect as 'touch').
  11       # The : serves as a dummy placeholder, producing no output.
  12 
  13    > filename    
  14       # The > truncates file "filename" to zero length.
  15       # If file not present, creates zero-length file (same effect as 'touch').
  16       # (Same result as ": >", above, but this does not work with some shells.)
  17 
  18    COMMAND_OUTPUT >>
  19       # Redirect stdout to a file.
  20       # Creates the file if not present, otherwise appends to it.
  21 
  22 
  23       # Single-line redirection commands (affect only the line they are on):
  24       # --------------------------------------------------------------------
  25 
  26    1>filename
  27       # Redirect stdout to file "filename."
  28    1>>filename
  29       # Redirect and append stdout to file "filename."
  30    2>filename
  31       # Redirect stderr to file "filename."
  32    2>>filename
  33       # Redirect and append stderr to file "filename."
  34    &>filename
  35       # Redirect both stdout and stderr to file "filename."
  36       #
  37       #  Note that   &>>filename
  38       #+ -- attempting to redirect and *append*
  39       #+ stdout and stderr to file "filename" --
  40       #+ fails with the error message,
  41       #+ syntax error near unexpected token `>'.
  42 
  43    M>N
  44      # "M" is a file descriptor, which defaults to 1, if not explicitly set.
  45      # "N" is a filename.
  46      # File descriptor "M" is redirect to file "N."
  47    M>&N
  48      # "M" is a file descriptor, which defaults to 1, if not set.
  49      # "N" is another file descriptor.
  50 
  51       #==============================================================================
  52 
  53       # Redirecting stdout, one line at a time.
  54       LOGFILE=script.log
  55 
  56       echo "This statement is sent to the log file, \"$LOGFILE\"." 1>$LOGFILE
  57       echo "This statement is appended to \"$LOGFILE\"." 1>>$LOGFILE
  58       echo "This statement is also appended to \"$LOGFILE\"." 1>>$LOGFILE
  59       echo "This statement is echoed to stdout, and will not appear in \"$LOGFILE\"."
  60       # These redirection commands automatically "reset" after each line.
  61 
  62 
  63 
  64       # Redirecting stderr, one line at a time.
  65       ERRORFILE=script.errors
  66 
  67       bad_command1 2>$ERRORFILE       #  Error message sent to $ERRORFILE.
  68       bad_command2 2>>$ERRORFILE      #  Error message appended to $ERRORFILE.
  69       bad_command3                    #  Error message echoed to stderr,
  70                                       #+ and does not appear in $ERRORFILE.
  71       # These redirection commands also automatically "reset" after each line.
  72       #=======================================================================

   1    2>&1
   2       # Redirects stderr to stdout.
   3       # Error messages get sent to same place as standard output.
   4 
   5    i>&j
   6       # Redirects file descriptor i to j.
   7       # All output of file pointed to by i gets sent to file pointed to by j.
   8 
   9    >&j
  10       # Redirects, by default, file descriptor 1 (stdout) to j.
  11       # All stdout gets sent to file pointed to by j.

   1    0< FILENAME
   2     < FILENAME
   3       # Accept input from a file.
   4       # Companion command to ">", and often used in combination with it.
   5       #
   6       # grep search-word <filename
   7 
   8 
   9    [j]<>filename
  10       #  Open file "filename" for reading and writing,
  11       #+ and assign file descriptor "j" to it.
  12       #  If "filename" does not exist, create it.
  13       #  If file descriptor "j" is not specified, default to fd 0, stdin.
  14       #
  15       #  An application of this is writing at a specified place in a file. 
  16       echo 1234567890 > File    # Write string to "File".
  17       exec 3<> File             # Open "File" and assign fd 3 to it.
  18       read -n 4 <&3             # Read only 4 characters.
  19       echo -n . >&3             # Write a decimal point there.
  20       exec 3>&-                 # Close fd 3.
  21       cat File                  # ==> 1234.67890
  22       #  Random access, by golly.
  23 
  24 
  25 
  26    |
  27       # Pipe.
  28       # General purpose process and command chaining tool.
  29       # Similar to ">", but more general in effect.
  30       # Useful for chaining commands, scripts, files, and programs together.
  31       cat *.txt | sort | uniq > result-file
  32       # Sorts the output of all the .txt files and deletes duplicate lines,
  33       # finally saves results to "result-file".

Multiple instances of input and output redirection and/or pipes can be combined in a single command line.
   1 command < input-file > output-file
   2 
   3 command1 | command2 | command3 > output-file
See Example 15-31 and Example A-15.

Multiple output streams may be redirected to one file.
   1 ls -yz >> command.log 2>&1
   2 #  Capture result of illegal options "yz" in file "command.log."
   3 #  Because stderr is redirected to the file,
   4 #+ any error messages will also be there.
   5 
   6 #  Note, however, that the following does *not* give the same result.
   7 ls -yz 2>&1 >> command.log
   8 #  Outputs an error message and does not write to file.
   9 
  10 #  If redirecting both stdout and stderr,
  11 #+ the order of the commands makes a difference.

Closing File Descriptors

n<&-

Close input file descriptor n.

0<&-, <&-

Close stdin.

n>&-

Close output file descriptor n.

1>&-, >&-

Close stdout.

Child processes inherit open file descriptors. This is why pipes work. To prevent an fd from being inherited, close it.
   1 # Redirecting only stderr to a pipe.
   2 
   3 exec 3>&1                              # Save current "value" of stdout.
   4 ls -l 2>&1 >&3 3>&- | grep bad 3>&-    # Close fd 3 for 'grep' (but not 'ls').
   5 #              ^^^^   ^^^^
   6 exec 3>&-                              # Now close it for the remainder of the script.
   7 
   8 # Thanks, S.C.

For a more detailed introduction to I/O redirection see Appendix E.

19.1. Using exec

An exec <filename command redirects stdin to a file. From that point on, all stdin comes from that file, rather than its normal source (usually keyboard input). This provides a method of reading a file line by line and possibly parsing each line of input using sed and/or awk.


Example 19-1. Redirecting stdin using exec

   1 #!/bin/bash
   2 # Redirecting stdin using 'exec'.
   3 
   4 
   5 exec 6<&0          # Link file descriptor #6 with stdin.
   6                    # Saves stdin.
   7 
   8 exec < data-file   # stdin replaced by file "data-file"
   9 
  10 read a1            # Reads first line of file "data-file".
  11 read a2            # Reads second line of file "data-file."
  12 
  13 echo
  14 echo "Following lines read from file."
  15 echo "-------------------------------"
  16 echo $a1
  17 echo $a2
  18 
  19 echo; echo; echo
  20 
  21 exec 0<&6 6<&-
  22 #  Now restore stdin from fd #6, where it had been saved,
  23 #+ and close fd #6 ( 6<&- ) to free it for other processes to use.
  24 #
  25 # <&6 6<&-    also works.
  26 
  27 echo -n "Enter data  "
  28 read b1  # Now "read" functions as expected, reading from normal stdin.
  29 echo "Input read from stdin."
  30 echo "----------------------"
  31 echo "b1 = $b1"
  32 
  33 echo
  34 
  35 exit 0

Similarly, an exec >filename command redirects stdout to a designated file. This sends all command output that would normally go to stdout to that file.

Important

exec N > filename affects the entire script or current shell. Redirection in the PID of the script or shell from that point on has changed. However . . .

N > filename affects only the newly-forked process, not the entire script or shell.

Thank you, Ahmed Darwish, for pointing this out.


Example 19-2. Redirecting stdout using exec

   1 #!/bin/bash
   2 # reassign-stdout.sh
   3 
   4 LOGFILE=logfile.txt
   5 
   6 exec 6>&1           # Link file descriptor #6 with stdout.
   7                     # Saves stdout.
   8 
   9 exec > $LOGFILE     # stdout replaced with file "logfile.txt".
  10 
  11 # ----------------------------------------------------------- #
  12 # All output from commands in this block sent to file $LOGFILE.
  13 
  14 echo -n "Logfile: "
  15 date
  16 echo "-------------------------------------"
  17 echo
  18 
  19 echo "Output of \"ls -al\" command"
  20 echo
  21 ls -al
  22 echo; echo
  23 echo "Output of \"df\" command"
  24 echo
  25 df
  26 
  27 # ----------------------------------------------------------- #
  28 
  29 exec 1>&6 6>&-      # Restore stdout and close file descriptor #6.
  30 
  31 echo
  32 echo "== stdout now restored to default == "
  33 echo
  34 ls -al
  35 echo
  36 
  37 exit 0


Example 19-3. Redirecting both stdin and stdout in the same script with exec

   1 #!/bin/bash
   2 # upperconv.sh
   3 # Converts a specified input file to uppercase.
   4 
   5 E_FILE_ACCESS=70
   6 E_WRONG_ARGS=71
   7 
   8 if [ ! -r "$1" ]     # Is specified input file readable?
   9 then
  10   echo "Can't read from input file!"
  11   echo "Usage: $0 input-file output-file"
  12   exit $E_FILE_ACCESS
  13 fi                   #  Will exit with same error
  14                      #+ even if input file ($1) not specified (why?).
  15 
  16 if [ -z "$2" ]
  17 then
  18   echo "Need to specify output file."
  19   echo "Usage: $0 input-file output-file"
  20   exit $E_WRONG_ARGS
  21 fi
  22 
  23 
  24 exec 4<&0
  25 exec < $1            # Will read from input file.
  26 
  27 exec 7>&1
  28 exec > $2            # Will write to output file.
  29                      # Assumes output file writable (add check?).
  30 
  31 # -----------------------------------------------
  32     cat - | tr a-z A-Z   # Uppercase conversion.
  33 #   ^^^^^                # Reads from stdin.
  34 #           ^^^^^^^^^^   # Writes to stdout.
  35 # However, both stdin and stdout were redirected.
  36 # Note that the 'cat' can be omitted.
  37 # -----------------------------------------------
  38 
  39 exec 1>&7 7>&-       # Restore stout.
  40 exec 0<&4 4<&-       # Restore stdin.
  41 
  42 # After restoration, the following line prints to stdout as expected.
  43 echo "File \"$1\" written to \"$2\" as uppercase conversion."
  44 
  45 exit 0

I/O redirection is a clever way of avoiding the dreaded inaccessible variables within a subshell problem.


Example 19-4. Avoiding a subshell

   1 #!/bin/bash
   2 # avoid-subshell.sh
   3 # Suggested by Matthew Walker.
   4 
   5 Lines=0
   6 
   7 echo
   8 
   9 cat myfile.txt | while read line;
  10                  do {
  11                    echo $line
  12                    (( Lines++ ));  #  Incremented values of this variable
  13                                    #+ inaccessible outside loop.
  14                                    #  Subshell problem.
  15                  }
  16                  done
  17 
  18 echo "Number of lines read = $Lines"     # 0
  19                                          # Wrong!
  20 
  21 echo "------------------------"
  22 
  23 
  24 exec 3<> myfile.txt
  25 while read line <&3
  26 do {
  27   echo "$line"
  28   (( Lines++ ));                   #  Incremented values of this variable
  29                                    #+ accessible outside loop.
  30                                    #  No subshell, no problem.
  31 }
  32 done
  33 exec 3>&-
  34 
  35 echo "Number of lines read = $Lines"     # 8
  36 
  37 echo
  38 
  39 exit 0
  40 
  41 # Lines below not seen by script.
  42 
  43 $ cat myfile.txt
  44 
  45 Line 1.
  46 Line 2.
  47 Line 3.
  48 Line 4.
  49 Line 5.
  50 Line 6.
  51 Line 7.
  52 Line 8.

Notes

[1]

A file descriptor is simply a number that the operating system assigns to an open file to keep track of it. Consider it a simplified type of file pointer. It is analogous to a file handle in C.

[2]

Using file descriptor 5 might cause problems. When Bash creates a child process, as with exec, the child inherits fd 5 (see Chet Ramey's archived e-mail, SUBJECT: RE: File descriptor 5 is held open). Best leave this particular fd alone.