Time again to pop a shell and dig into the deep, geeky Unix internals of OS X with Dig Into Unix. Today we are going to look at two top-shelf power tools for text editing: sed and awk.

Sed is a Stream EDitor, and if you recall our previous Dig Into Unix installment concerning standard streams, you’ll understand that the streams we are talking about are actually just text from one source or another. Sed’s bread and butter is text search and replace, very similary to the “Edit” and “Find…” functions in TextEdit and many other GUI text editors. Unlike those text editors though, sed, by default, will write its output to the screen, or stdout.

As an example, try some basic operations on this string of text:
The quick brown fox jumped over the lazy dog's back.

Save the string of text as a file named test.txt, and type this into the Terminal:
sed s/quick/slow/g test.txt

The fox is now slow on the screen, but not changed in the file itself. To follow the stream, the text came from the file, through sed, and to the screeen. The best set of examples I’ve found for getting right into sed and starting to play with it is the collection of sed one liners hosted at Sourceforge.

Personally, I use sed when I’ve got a large number of configuration files that need to be edited. For example, it might be decided that we do not need our Nagios monitoring system alerting on the a certain statistic. I could go into 100 different files and perform the same action on all of them, or I could rely on a simple shell script and sed to do it for me.


for each in `ls *.cfg`; do
mv $each $each.bak #Safety First!
sed '30,35s/^/#/g' $each.bak > $each
done


This will plow through all of the config files in a certain directory and add a # sign at the beginning of lines 30 through 35, commenting those lines out. Then I can restart Nagios, and if all goes well, delete all of the .bak files created as backups by the script.

While sed operates on lines and regular expressions (the subject of a future Dig Into Unix article!), awk works with fields. When given a stream of text, either from a text file or piped in from another application, awk can manipulate the text and rearrange the words. By default, awk separates the text fields by a space character, but you can use any other character you’d like.

Like sed, awk also has a great collection of one liners, this collection here is a great resource collected by Eric Pement. In my day to day activities, I call on awk when I want to format text for a report or to be input into another application.

To use our quick brown fox example again, we can print only the fourth, third, and second words, in reverse order, with this command:
awk '{ print $4, $3, $2 }' test

That will print out “fox brown quick”. Not very practical or useful. Something more practical might be to manipulate a list of comma separated values. Using the “-F” flag in awk, you can tell awk to separate it’s fields based on the comma, as in:
awk -F, '{ print $1 }' test

Since there are no commas in the test file, this will print the entire string of text. So, we could run this command to take care of that:
sed s/ /,/g test > test2

This will use sed to replace all spaces with commas. The backslash is there to escape a special character, so the space is interpreted literally and not as part of a command. Now, you could use awk to manipulate the string of text as needed.
awk -F, '{ print $7, $8, $9, $10, $5, $6, $1, $2, $3, $4 " did" }' test2

This article has just barely skimmed the surface of what sed and awk can do. There are some rather hefty books dedicated to the pair, this one from O’Reilly has been on my desk for years now. Awk is an entire programming language, but the point of this series is not to teach the in-depth details, it’s just to get your feet wet, and maybe, just maybe leave you thirsting for more. The real rub is that everything that sed and awk can do can also be done, at times more efficiently, with the practical extraction and report language…better known as Perl, which is the subject of a future Dig Into Unix article.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Why iPad 2 Will Lead Consumers Into the Post-PC Era
• The Near-Term Evolution of Social Commerce
• Content Farms: The Players, The Benefits, The Risks

This is the third installment of our Dig Into Unix series, an ongoing look into the deep, geeky insides of the core of OS X. In the first part, we got to fire up the Terminal and take a look around the filesystem as the OS sees it, which is slightly different from how the rest of us see it through the Finder. In the second installment, we took a look at vi, the ancient text editor fit for kings.

Today, I’d like to cover a very basic, but very powerful, aspect of Unix, the three standard streams: standard in, standard out, and standard error. These three are normally abbreviated as stdin, stdout and stderr.

When interacting with an application on the command line, it’s good to imagine that application with three pipes. One is a big funnel, right on top — this is stdin, accepting input from the keyboard. The second pipe is on the bottom, stdout, sending text to the Terminal. The third, stderr, sticks out the side, and normally also sends its output to the Terminal. I say “normally” because the standard streams can be redirected by certain key characters. For example, if you use “cat” to display a document, you are sending stdout to the Terminal. However, if you use “cat” like this, you actually send the output of the cat command to another file, redirecting stdout to a new text file.

cat someFile 1> someCopy.txt

The number one in the command above represents stdout. The right arrow is telling the shell that you’d like to redirect stdout to a new file. If that file already exists, this command will replace that file with the contents of “someFile”. If you’d like to append the text to someCopy.txt, you could do this as well:

cat someOtherFile 1>> someCopy.txt

You’ll notice that I used two right arrows that time. Using the double right arrows tells the shell to append the text to the end of the file. This works great for keeping a running log of regularly scheduled tasks.

Sometimes you’ll want to either keep error messages out of the way, or you’ll like to keep a log of error messages. In this case, you can redirect stderr to a file like this:

grep something somewhere 2> grep_error_log

The grep command is another Unix power tool that we’ll touch on later in the series. For now, just know that grep searches for strings of text in files. In the command above, I told grep to search for the string “something” in the file “somewhere” and then redirect stderr “2” to a file named “grep_error_log.” Run this command without the redirect on it, then add the redirect and run it again. You can compare the contents of the error log to what you saw on the terminal and see that they are the same.

Being able to redirect output is handy, but not really a “power tool.” The real power is in the pipe, the key on the far right above the return key, that looks like this: “|”, and it’s probably a key that you may have looked at and wondered…“why is that there?” The pipe takes the stdout from one command and feeds it to the stdin of another command. Think about the picture above; now imagine plugging lots of different commands into each other in a long string. The nature of Unix is that it is continually used in ways that its creators never imagined. It is truly a system limited only by your creativity and curiosity.

During one of my early classes on Unix, after we had entered in a command very similar to the one below, the instructor stood up and said, “Congratulations, you are all programmers!” We weren’t, not by a long shot, but the idea was that by entering commands and stringing them together in the Terminal, we were programming the computer to perform a specific task. We were not using a traditional “programming” language like C or even Java, but we were programming directly to the shell.

To put this idea into action, consider this command:

cat someFile | grep someText 1> foundText

First of all, yes, I’m aware that this is a useless use of cat. That’s not the point. The point is that the above command connects cat’s stdout, which normally points to the Terminal, to stdin of grep, which then searches what it has been handed for the string “someText,” and then, just for good measure, redirect the output of grep to a file named “foundText.”

The simple illustrations I’ve given here work in all kinds of situations. Personally, I use the pipe and stream redirection to parse the results of commands that search logs, retrieve information from the Internet, and reformat data into comma separated values for importing into Numbers, Excel or OpenOffice. There are very few limitations to what you can do with plain text and the tools lurking under the hood of your Mac.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Why iPad 2 Will Lead Consumers Into the Post-PC Era
• The Near-Term Evolution of Social Commerce
• Content Farms: The Players, The Benefits, The Risks

Continuing our Dig Into Unix series, we’ve now covered the absolute basics of launching Terminal.app, moving around the file system, looking at files with cat, and learning about commands with man. Now, I’d like to introduce you to the power of vi.

vi (pronounced vee-eye) is a text editor for the command line, and was originally written by Bill Joy in 1976. vi is now part of every single Unix system, including OS X. The version included with Leopard is vim, which stands for Vi iMproved. One of the major problems that vi has faced over the years is that it has a major learning curve. Every action that you take in vi you do with keyboard combos. Since there are so many actions that you can do, it takes awhile to get used to the different keystrokes, but once you do, they become ingrained in your fingers. From time to time, when typing in some other app, I find myself hitting the escape key followed by jjj or hhh trying to move the curser.

Getting Started

To get started, open up Terminal.app and type vi. This starts the editor in command mode. Type i to enter insert mode, and then you can begin typing text into the file. Copy the text from this article and paste it into vi, regular CMD-C / CMD-V will work just fine. This will give you something to work with.

Now that you’ve got some text, it’d probably be good to save it. Hit the escape key to put the editor back into command mode, and type :w newfile.txt. This will create a text file named newfile.txt in your home directory. Press i again, and you can begin typing, starting where you left off. Press escape again.

Navigating Lines

To move to any line of text in vi, you can press the number of the line you want to go to and G. For example, to go to the first line, type 1G. If you type that and the characters “1G” show up on your screen, you are not in command mode, don’t forget to press escape first. To get back to the end of the file, you can just press G by itself. That will bring you to the start of the last line in the file.

• To get to the end of a line, you can press $, and your curser will jump to the end of the line.
• To go to the beginning of the line, you press ^. Now you can jump around the file.
• To move the curser one character at a time, use the four keys h j k and l. h moves one character to the left, j moves one character to the right, k to moves one character up, and l moves one character down.
• To open a line below, you can press o, and to open a line above, O.

Configuration File

These basic text editing commands form the basis of the real strengths of vi, and the reason I continue to use it every day. But to really take advantage of the power of vi, you need to create a configuration file for shortcuts.

Exit out of vi and write the file to disk at the same time by typing :wq! while in command mode. Now, create a new file, and open the file in vi for editing by typing vi .vimrc. The .vimrc file is the configuration file vi reads when starting. The important thing to remember about this configuration file is that you can add any of the commands above into the file, including movement, inserting text, and opening lines, and then map those commands to a keyboard combo. If you are familiar with TextExpander, you’ll follow right along with this concept, but it’s even stronger than that.

We are going to create a single line of text in the file that looks like this:
map ,a 1GiJack Doe^MGuy in Charge^MACME Inc.^M111 Main St.^MAnytown, USA

^M^M^M^MDear Sir,^M^M^M^MSincerely,^M^M^MJohn Doe^[10G i

The line above has been separated into two lines to fit on the page. When you see ^M, that’s not really two characters, it’s only one. It’s the return key, captured in raw form in the config file for use in generating our commands. To capture it, you press Control-V, followed by return. The esc key is also captured above by pressing Control-V and then hitting the escape key. That creates the ^[ character above, and allows us to enter some of the vi commands above into the config. Enter in that line of text above, using the Control-V trick to capture the return and esc characters.

Now, save the new file by typing :wq! in command mode, open up a new file with vi file, and test out your new ,a shortcut. If everything went well, it should look just like the video below.

Worth Learning

The learning curve with vi is steep, but as with most things worth learning, the payoff is significant. This short article has barely scratched the surface of what vi can do. vi can be a full-fledged IDE, or you could use it to write your next book. The mapping features above are especially helpful if you do any programming, or if you manage any Unix or Linux servers. If you’d like to learn more, I’d suggest starting with man vi, and maybe picking up a copy of the O’Reilly book “Learning the vi and Vim Editors.” Of course, I’d be happy to help out as well.

Back when I was first learning Unix, I asked an old graybeard why I should bother learning such a complicated, ancient text editor. He told me that since everything in Unix is a file, and almost all files in Unix are text, I needed some way to edit those files to control the system. Someday, he said, I would find myself without any other text editor, and without any other way to change a file. He was right, since I’m now working as a Unix systems administrator, I’ve been in that position for several years. Over time, I’ve come to appreciate all that vi can do for me.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Why iPad 2 Will Lead Consumers Into the Post-PC Era
• The Near-Term Evolution of Social Commerce
• Content Farms: The Players, The Benefits, The Risks

When Apple revamped its operating system and adopted Nextstep as the base of OS X, they brought along with it an extremely powerful version of Unix based on the open-source project FreeBSD, now known as Darwin.

Unix has a long history, one that started in the basements of Bell Labs by a group of AT&T engineers some 40 years ago. A professor in a C programming course I took once said that they were supposed to be writing drivers for the AT&T hardware, but instead, they wanted some way to use the system to play games, so they invented Unix.

Unix is now a mature and robust operating system, and since OS X is based on Unix, it has inherited all of its power, and some of its complexity. The beautiful aqua interface that we are used to seeing is really all that is needed, but if you would like to take a look at what makes your computer tick, Apple included Terminal.app to act as a window into the GUI and into the Unix soul of OS X.

Nick started a great series back in January 2007 on this subject, and now I’d like to cover some of the basics again, and maybe bring a different point of view to the table as well.

Open Terminal.app (found in Applications → Utilities), and you’ll see a window with a prompt waiting for you to start typing.

At this point, it’s important to understand a few things about the Terminal. For one, the commands that you can type are interpreted and carried out immediately, no waiting around. So if you tell it to remove a file, it will do it right then, with no easy way of recovering it. There isn’t a recycle bin on the command line (not without a little coaxing, anyway). Secondly, since Unix was developed decades ago, many of the commands seem a bit archaic. Back when most of these utilities were written, they were all abbreviated to save space and cut down on the number of keystrokes you’d need to type. Below is a list of a few essentials, and another list from Nick’s post is here.

• ls (list): Probably one of the most important commands, it lets you see what’s in your current directory.
• cd (change directory): This is how you move about the filesystem in Unix, for example, to move from Library to Documents.
• file: This one isn’t short for anything, but it will give you a brief description of what a particular file is.
• cat (concatenate): Or “Grab everything in this file and let me read it.”
• man (manual): The online manual will describe most commands that you’re interested in; for example, typing man ls will give you the manual page about the ls command.

It might be helpful to open a Finder window and put it right next to the Terminal. Open the Finder so it shows your home directory. Type ls in the Terminal to see the same files that you see in the Finder. Try another command: touch. touch is designed to change the last accessed timestamp of a file, but it will also create a blank file. You can see the file created in the Finder as well. You can cat the file, and see that there is nothing in it.

Now that you’ve created a file, and looked at its (blank) contents, you can remove the file using the rm command. This is one of those dangerous commands that, if used carelessly, can really screw things up. For this example, carefully type rm file into the terminal, and watch the file disappear in the Finder. You’ll notice that your Trash stays empty — that file is goners.

The Unix filesystem is a nested hierarchy, with each directory separated by a forward slash (/). The current working directory is symbolized as a dot (.), and the parent directory is symbolized by two dots (..). The top of the hierarchy, known as the root, is symbolized by a single forward slash.

So, to move to the parent directory, you could type cd ... If you were in the Library directory and you typed this command, you would then be moved into your home directory. To see the very top of the hierarchy, you would type cd /. Type this command now.

Also, move in the Finder to the hard drive where you have OS X installed. I have mine named, originally, “OS X”. Type ls in the Terminal, and you’ll notice a few more files than you can see in the Finder. These files are important Unix system files, and should not be touched unless you really know what you’re doing…and really, not even then. To illustrate the importance of these files, you could type file mach_kernel.

This is the kernel, the core of the operating system. Do not mess with this file. Several of the other files are directories. You can change directories into /bin, for example, and type ls to list the contents of that directory. You’ll find a file in that directory named “ls,” which is the executable for the ls command. When you type ls into the terminal, it executes this tiny app. A great place to learn about Unix is to get a list of this directory, and then read the man page for every file listed.

To exit the Terminal, just CMD-Q like any other app, and you are back in the comforts of OS X.

This has been a very brief overview of how to go from absolutely no Unix knowledge whatsoever to the smallest amount of Unix knowledge necessary to poke around a bit. There’s a lot of power under there, but before you can really start to harness it, you need to get a decent understanding of the hows and whys.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Why iPad 2 Will Lead Consumers Into the Post-PC Era
• The Near-Term Evolution of Social Commerce
• Content Farms: The Players, The Benefits, The Risks