Sed: Learn This Classic Text Processing Tool

Sed is a utility program and simple scripting language used for editing text. The name is an abbreviation of “stream editor,” and its purpose really is the editing of input streams — typically individual lines in a file, or a single stream from stdin (standard input).

The language is not remarkably powerful. Though sed is Turing complete, it is not at all efficient for any real programming tasks — it only has two available variables, for example. It is, however, very useful for programmatically finding and editing patterns of text in files, from the command line or from a shell script.

The most important sed command is s, which is the substitution command. The command takes two arguments, plus an options flag. For example, if you wanted to correct every instance of the British-spelling “colour” to the American “color,” and write the corrected version to a new file, you might do the following from the command line:

sed s/colour/color/g <old.file >new.file

A quick explanation:

• The s/ begins the substitution command.
• Whatever is between the first pair of slashes is the pattern to match. In this case, it’s the word “colour,” but it could also be a regular expression — a pattern, rather than a specific string of characters.
• After the pattern to match is the replacement string.
• The g means “global,” or “do this every time, not just the first time.”

Both the pattern to match and the substitution can be variable. If the pattern to match is a regular expression, and you want to amend it (not just replace it) you can used the ampersand ( & ) to indicate the matched pattern. For example, suppose you wanted to turn all plain-text URLs into HTML hyperlinks. You might do this:

sed 's_http[:]//[^ ]*_<a href="&">&</a>_'

• This uses the underscore delimiter instead of the slash, to make it easier to read with all the slashes already present in URLs.
• The pattern to match is a regular expression that matches well-formed URLs (although in a limited way).
• The substitution creates an anchor tag with the matched pattern (the URL) in the href attribute and text of the tag.

This sort of find-and-replace was the original reason sed was invented, and continues to be the most common usage.

Sed Online Resources and Tutorials

• Sed — an Introduction and Tutorial at the Unix Wizard’s Grymoire, provides a great intro and reference to sed;
• Sed Tutorial is a decent introduction that explains some of the behind-the-scenes work that sed is doing;
• The Sed Tutorial at Tutorials Point is a very clear and methodical walk-through of the most important sed concepts;
• Using Sed is a great two-part tutorial from Digital Ocean;
• YouTuber Kris Occhipinti has an 18-part Sed Video Tutorial;
• Sed-Tutorial from LinuxHowtos is a small, but helpful, tutorial on sed;
• Sed Subreddit is not very active, but many sed (and related) questions show up on the regex and commandline subreddits;

Books on Sed

• Sed & Awk (1997) by Dougherty and Robbins, is the classic text on the two most important command-line text tools; if you can only read one thing on sed, this is probably the one to read; also check out the handy companion Aed & Awk Pocket Reference;
• Definitive Guide to Sed: Tutorial and Reference (2013) by Daniel A Goldman, provides a much more contemporary perspective on using sed, with a particular emphasis on GNU sed (gsed);
• The Ksh, Awk, and Sed Script Collection: Mastering Unix Programming Through Practical Examples () by Steve Myers, is a cookbook of useful shell scripts.

Sed vs Awk and Other Options

The sed tool is just one of many options for searching and manipulating text from the command line or in shell scripts. The most frequently compared tool is Awk, but there is also grep. Additionally, you can use more robust scripting languages such Perl or Python.

Here are a few resources to help you determine which tool is the best for your text-searching and in-place editing needs:

• Grep vs Sed vs Awk: What a Proficient/Advanced Linux Shell User Should Know is a fantastic article describing the three main text-processing tools, with a lot of detail about when to use which one;
• When to Use Sed/Awk Over Perl or Python is a forum posting with a great conversation about when to use full-on scripting languages vs the simpler text processing tools;
• The Functional and Performance Differences of Sed, Awk, and Other Unix Parsing Utilities provides a detailed technical comparison, including notes on speed;
• What Is the Difference between Sed and Awk? was considered “off-topic” at StackExchange, but fortunately a few users provided very helpful answers anyway; if you don’t have time for the more detailed explanations above, this page provides the best 30-second overview.

Books and Resources on Related Topics

To use sed effectively, you need skills in two related areas: regular expressions and shell scripting in general. Here are some books and online tutorials to help you round out your sed abilities.

In addition to these resources, you should also check out our pages on Awk and Bash.

Books

• Introducing Regular Expressions (2012) by Michael Fitzgerald is a very helpful intro text that makes this potentially intimidating topic easy to understand; it also includes sed scripting in particular, as well as info on using regex in Perl
• Mastering Regular Expressions (2006) by Jeffrey Friedl is your next-level regex resource, after the introductory text above;
• Classic Shell Scripting (2005) by Robbins and Beebe is a book that covers the most important Unix command line and scripting tools, including sed; Arnold Robbins, one of the authors, is also the co-author of the classic Sed & Awk book listed above.

Online Resources

There are a lot of online resources on these topics. Here’s just a few of the most helpful:

• RegExr is an online tool to learn, build, and test regular expressions;
• RegEx101 is another online regular expression tester, with a built-in cheat sheet and helpful parsing feedback;
• Regex Cheat Sheet provides a comprehensive reference, for when you need to know which seemingly-random string of characters you need to type to find the seemingly-random string of characters you are looking for;
• The Grymoire, mentioned above, is a trove of information for Unix scripting;
• Linux Shell Scripting Tutorial is one of the best online tutorials for learning Bash.

Summary

There was a time when virtually all serious computer users knew command-line tools like sed very well. For good or bad, that time is long past. Today, even many advanced developers don’t know how to use these fundamental tools.

However, if you’re interested in really maximizing your efficiency as a power user — especially in a Linux, Unix, or Mac environment — having a good handle on the command line is essential. This is particularly helpful if you do any web development or administration, since most of the web runs on Linux servers, and most remote-server management is done at the command line.

Sed, which is included in nearly every Linux and Unix-like operating system, can help you find what you are looking for. When you don’t have the user-friendly search tools of a desktop computer, this can become essential. It isn’t that hard to learn, and you’ll never be sorry you know how to use this classic tool.

Further Reading and Resources

We have more guides, tutorials, and infographics related to computers:

• Linux Programming Introduction and Resources: this deep dive into Linux programming gets down into the kernel where all the action is.
• Bash Programming Resources: get started with the most important Unix command line shell.
• Awk Resources: Learn all about sed’s sister tool Awk.

The Ultimate List of Webmaster Tools A-Z

Sed and Awk and Bash are just part of a large collection of tools that allow system administrators to manage operating systems. The Ultimate List of Webmaster Tools A-Z will provide you with a lot of help in doing your work.

The Ultimate List of Webmaster Tools A-Z