This is the third installment of our Dig Into Unix series, an ongoing look into the deep, geeky insides of the core of OS X. In the first part, we got to fire up the Terminal and take a look around the filesystem as the OS sees it, which is slightly different from how the rest of us see it through the Finder. In the second installment, we took a look at vi, the ancient text editor fit for kings.
Today, I’d like to cover a very basic, but very powerful, aspect of Unix, the three standard streams: standard in, standard out, and standard error. These three are normally abbreviated as stdin, stdout and stderr.
When interacting with an application on the command line, it’s good to imagine that application with three pipes. One is a big funnel, right on top — this is stdin, accepting input from the keyboard. The second pipe is on the bottom, stdout, sending text to the Terminal. The third, stderr, sticks out the side, and normally also sends its output to the Terminal. I say “normally” because the standard streams can be redirected by certain key characters. For example, if you use “cat” to display a document, you are sending stdout to the Terminal. However, if you use “cat” like this, you actually send the output of the cat command to another file, redirecting stdout to a new text file.
cat someFile 1> someCopy.txt
The number one in the command above represents stdout. The right arrow is telling the shell that you’d like to redirect stdout to a new file. If that file already exists, this command will replace that file with the contents of “someFile”. If you’d like to append the text to someCopy.txt, you could do this as well:
cat someOtherFile 1>> someCopy.txt
You’ll notice that I used two right arrows that time. Using the double right arrows tells the shell to append the text to the end of the file. This works great for keeping a running log of regularly scheduled tasks.
Sometimes you’ll want to either keep error messages out of the way, or you’ll like to keep a log of error messages. In this case, you can redirect stderr to a file like this:
grep something somewhere 2> grep_error_log
The grep command is another Unix power tool that we’ll touch on later in the series. For now, just know that grep searches for strings of text in files. In the command above, I told grep to search for the string “something” in the file “somewhere” and then redirect stderr “2” to a file named “grep_error_log.” Run this command without the redirect on it, then add the redirect and run it again. You can compare the contents of the error log to what you saw on the terminal and see that they are the same.
Being able to redirect output is handy, but not really a “power tool.” The real power is in the pipe, the key on the far right above the return key, that looks like this: “|”, and it’s probably a key that you may have looked at and wondered…“why is that there?” The pipe takes the stdout from one command and feeds it to the stdin of another command. Think about the picture above; now imagine plugging lots of different commands into each other in a long string. The nature of Unix is that it is continually used in ways that its creators never imagined. It is truly a system limited only by your creativity and curiosity.
During one of my early classes on Unix, after we had entered in a command very similar to the one below, the instructor stood up and said, “Congratulations, you are all programmers!” We weren’t, not by a long shot, but the idea was that by entering commands and stringing them together in the Terminal, we were programming the computer to perform a specific task. We were not using a traditional “programming” language like C or even Java, but we were programming directly to the shell.
To put this idea into action, consider this command:
cat someFile | grep someText 1> foundText
First of all, yes, I’m aware that this is a useless use of cat. That’s not the point. The point is that the above command connects cat’s stdout, which normally points to the Terminal, to stdin of grep, which then searches what it has been handed for the string “someText,” and then, just for good measure, redirect the output of grep to a file named “foundText.”
The simple illustrations I’ve given here work in all kinds of situations. Personally, I use the pipe and stream redirection to parse the results of commands that search logs, retrieve information from the Internet, and reformat data into comma separated values for importing into Numbers, Excel or OpenOffice. There are very few limitations to what you can do with plain text and the tools lurking under the hood of your Mac.