<!doctypehtml><html class="sidebar-visible no-js light"lang=en><head><meta charset=UTF-8><title>awk scripts - CLI text processing with GNU awk</title><meta content="text/html; charset=utf-8"http-equiv=Content-Type><meta content="Example based guide to mastering GNU awk"name=description><meta content=width=device-width,initial-scale=1 name=viewport><meta content=#ffffff name=theme-color><meta content="CLI text processing with GNU awk"property=og:title><meta content=website property=og:type><meta content="Example based guide to mastering GNU awk"property=og:description><meta content=https://learnbyexample.github.io/learn_gnuawk/ property=og:url><meta content=https://raw.githubusercontent.com/learnbyexample/learn_gnuawk/master/images/gawk_ls.png property=og:image><meta content=1280 property=og:image:width><meta content=720 property=og:image:height><meta content=summary_large_image property=twitter:card><meta content=@learn_byexample property=twitter:site><link href=favicon.svg rel=icon><link rel="shortcut icon"href=favicon.png><link href=css/variables.css rel=stylesheet><link href=css/general.css rel=stylesheet><link href=css/chrome.css rel=stylesheet><link href=FontAwesome/css/font-awesome.css rel=stylesheet><link href=fonts/fonts.css rel=stylesheet><link href=highlight.css rel=stylesheet><link href=tomorrow-night.css rel=stylesheet><link href=ayu-highlight.css rel=stylesheet><link href=style.css rel=stylesheet><body><script>var path_to_root = "";

var default_theme = window.matchMedia("(prefers-color-scheme: dark)").matches ? "navy" : "light";</script><script>try {

var theme = localStorage.getItem('mdbook-theme');

var sidebar = localStorage.getItem('mdbook-sidebar');

if (theme.startsWith('"') && theme.endsWith('"')) {

localStorage.setItem('mdbook-theme', theme.slice(1, theme.length - 1));

}

if (sidebar.startsWith('"') && sidebar.endsWith('"')) {

localStorage.setItem('mdbook-sidebar', sidebar.slice(1, sidebar.length - 1));

}

} catch (e) { }</script><script>var theme;

try { theme = localStorage.getItem('mdbook-theme'); } catch(e) { }

if (theme === null || theme === undefined) { theme = default_theme; }

var html = document.querySelector('html');

html.classList.remove('no-js')

html.classList.remove('light')

html.classList.add(theme);

html.classList.add('js');</script><script>var html = document.querySelector('html');

var sidebar = 'hidden';

if (document.body.clientWidth >= 1080) {

try { sidebar = localStorage.getItem('mdbook-sidebar'); } catch(e) { }

sidebar = sidebar || 'visible';

}

html.classList.remove('sidebar-visible');

html.classList.add("sidebar-" + sidebar);</script><nav aria-label="Table of contents"class=sidebar id=sidebar><div class=sidebar-scrollbox><ol class=chapter><li class="chapter-item expanded affix"><a href=cover.html>Cover</a><li class="chapter-item expanded affix"><a href=buy.html>Buy PDF/EPUB versions</a><li class="chapter-item expanded"><a href=preface.html><strong aria-hidden=true>1.</strong> Preface</a><li class="chapter-item expanded"><a href=installation-and-documentation.html><strong aria-hidden=true>2.</strong> Installation and Documentation</a><li class="chapter-item expanded"><a href=awk-introduction.html><strong aria-hidden=true>3.</strong> awk introduction</a><li class="chapter-item expanded"><a href=regular-expressions.html><strong aria-hidden=true>4.</strong> Regular Expressions</a><li class="chapter-item expanded"><a href=field-separators.html><strong aria-hidden=true>5.</strong> Field separators</a><li class="chapter-item expanded"><a href=record-separators.html><strong aria-hidden=true>6.</strong> Record separators</a><li class="chapter-item expanded"><a href=in-place-file-editing.html><strong aria-hidden=true>7.</strong> In-place file editing</a><li class="chapter-item expanded"><a href=using-shell-variables.html><strong aria-hidden=true>8.</strong> Using shell variables</a><li class="chapter-item expanded"><a href=control-structures.html><strong aria-hidden=true>9.</strong> Control Structures</a><li class="chapter-item expanded"><a href=built-in-functions.html><strong aria-hidden=true>10.</strong> Built-in functions</a><li class="chapter-item expanded"><a href=multiple-file-input.html><strong aria-hidden=true>11.</strong> Multiple file input</a><li class="chapter-item expanded"><a href=processing-multiple-records.html><strong aria-hidden=true>12.</strong> Processing multiple records</a><li class="chapter-item expanded"><a href=two-file-processing.html><strong aria-hidden=true>13.</strong> Two file processing</a><li class="chapter-item expanded"><a href=dealing-with-duplicates.html><strong aria-hidden=true>14.</strong> Dealing with duplicates</a><li class="chapter-item expanded"><a class=active href=awk-scripts.html><strong aria-hidden=true>15.</strong> awk scripts</a><li class="chapter-item expanded"><a href=gotchas-and-tips.html><strong aria-hidden=true>16.</strong> Gotchas and Tips</a><li class="chapter-item expanded"><a href=further-reading.html><strong aria-hidden=true>17.</strong> Further Reading</a><li class="chapter-item expanded"><a href=Exercise_solutions.html><strong aria-hidden=true>18.</strong> Exercise Solutions</a></li><br><hr><li class="chapter-item expanded"><i class="fa fa-github"id=git-repository-button></i><a href=https://github.com/learnbyexample/learn_gnuawk>   Source code</a><li class="chapter-item expanded"><i class="fa fa-home"id=home-button></i><a href=https://learnbyexample.github.io/>   My Blog</a><li class="chapter-item expanded"><i class="fa fa-book"id=book-button></i><a href=https://learnbyexample.github.io/books/>   My Books</a><li class="chapter-item expanded"><i class="fa fa-twitter"id=twitter-button></i><a href=https://twitter.com/learn_byexample>   Twitter</a></ol></div><div class=sidebar-resize-handle id=sidebar-resize-handle></div></nav><div class=page-wrapper id=page-wrapper><div class=page><div id=menu-bar-hover-placeholder></div><div class="menu-bar sticky bordered"id=menu-bar><div class=left-buttons><button aria-label="Toggle Table of Contents"title="Toggle Table of Contents"aria-controls=sidebar class=icon-button id=sidebar-toggle type=button><i class="fa fa-bars"></i></button><button aria-label="Change theme"title="Change theme"aria-controls=theme-list aria-expanded=false aria-haspopup=true class=icon-button id=theme-toggle type=button><i class="fa fa-paint-brush"></i></button><ul aria-label=Themes class=theme-popup id=theme-list role=menu><li role=none><button class=theme id=light role=menuitem>Light (default)</button><li role=none><button class=theme id=rust role=menuitem>Rust</button><li role=none><button class=theme id=coal role=menuitem>Coal</button><li role=none><button class=theme id=navy role=menuitem>Navy</button><li role=none><button class=theme id=ayu role=menuitem>Ayu</button></ul><button aria-label="Toggle Searchbar"title="Search. (Shortkey: s)"aria-controls=searchbar aria-expanded=false aria-keyshortcuts=S class=icon-button id=search-toggle type=button><i class="fa fa-search"></i></button></div><h1 class=menu-title>CLI text processing with GNU awk</h1><div class=right-buttons><a aria-label=Blog href=https://learnbyexample.github.io title=Blog> <i class="fa fa-home"id=home-button></i> </a><a aria-label=Twitter href=https://twitter.com/learn_byexample title=Twitter> <i class="fa fa-twitter"id=twitter-button></i> </a><a aria-label="Git repository"title="Git repository"href=https://github.com/learnbyexample/learn_gnuawk> <i class="fa fa-github"id=git-repository-button></i> </a></div></div><div class=hidden id=search-wrapper><form class=searchbar-outer id=searchbar-outer><input placeholder="Search this book ..."aria-controls=searchresults-outer aria-describedby=searchresults-header id=searchbar name=searchbar type=search></form><div class="searchresults-outer hidden"id=searchresults-outer><div class=searchresults-header id=searchresults-header></div><ul id=searchresults></ul></div></div><script>document.getElementById('sidebar-toggle').setAttribute('aria-expanded', sidebar === 'visible');

document.getElementById('sidebar').setAttribute('aria-hidden', sidebar !== 'visible');

Array.from(document.querySelectorAll('#sidebar a')).forEach(function(link) {

link.setAttribute('tabIndex', sidebar === 'visible' ? 0 : -1);

});</script><div class=content id=content><main><div class=sidetoc><nav class=pagetoc></nav></div><h1 id=awk-scripts><a class=header href=#awk-scripts>awk scripts</a></h1><p>So far, you've only seen how to provide <code>awk</code> scripts directly on the command line. In this chapter, you'll see basic examples for executing scripts saved in files.<blockquote><p><img alt=info src=images/info.svg> The <a href=https://github.com/learnbyexample/learn_gnuawk/tree/master/example_files>example_files</a> directory has all the files used in the examples.</blockquote><h2 id=-f-option><a class=header href=#-f-option>-f option</a></h2><p>The <code>-f</code> command line option allows you to pass the <code>awk</code> script via files instead of writing everything on the command line. Here's an one-liner seen earlier that's been converted to a multiline script. Note that <code>;</code> is no longer necessary to separate the commands, newline will do that too.<pre><code class=language-bash>$ cat buf.awk

/error/{

f = 1

buf = $0

next

}

f{

buf = buf ORS $0

}

/state/{

if(f)

print buf

f = 0

}

$ awk -f buf.awk broken.txt

error 2

1234

6789

state 1

error 4

abcd

state 3

</code></pre><p>Another advantage is that single quotes can be freely used.<pre><code class=language-bash>$ echo 'cue us on this example' | awk -v q="'" '{gsub(/\w+/, q "&" q)} 1'

'cue' 'us' 'on' 'this' 'example'

# the above solution is simpler to write as a script

$ cat quotes.awk

{

gsub(/\w+/, "'&'")

}

1

$ echo 'cue us on this example' | awk -f quotes.awk

'cue' 'us' 'on' 'this' 'example'

</code></pre><h2 id=-o-option><a class=header href=#-o-option>-o option</a></h2><p>If the code has been first tried out on the command line, you can use the <code>-o</code> option to get a pretty printed version. Output filename can be passed along as an argument to this option. By default, <code>awkprof.out</code> will be used as the filename.<pre><code class=language-bash># adding -o after the one-liner has been tested

# input filenames and -v would be simply ignored

$ awk -o -v OFS='\t' 'NR==FNR{r[$1]=$2; next}

{$(NF+1) = FNR==1 ? "Role" : r[$2]} 1' role.txt marks.txt

# pretty printed version

$ cat awkprof.out

NR == FNR {

r[$1] = $2

next

}

{

$(NF + 1) = FNR == 1 ? "Role" : r[$2]

}

1 {

print

}

# calling the script

# note that other command line options have to be provided as usual

$ awk -v OFS='\t' -f awkprof.out role.txt marks.txt

Dept Name Marks Role

ECE Raj 53 class_rep

ECE Joel 72

EEE Moi 68

CSE Surya 81

EEE Tia 59 placement_rep

ECE Om 92

CSE Amy 67 sports_rep

</code></pre><h2 id=summary><a class=header href=#summary>Summary</a></h2><p>So, now you know how to write program files for <code>awk</code> instead of just the one-liners. And about the <code>-o</code> option, which helps to convert complicated one-liners to pretty printed program files.<p>Next chapter will discuss a few gotchas and tricks.<h2 id=exercises><a class=header href=#exercises>Exercises</a></h2><blockquote><p><img alt=info src=images/info.svg> The <a href=https://github.com/learnbyexample/learn_gnuawk/tree/master/exercises>exercises</a> directory has all the files used in this section.</blockquote><p><strong>1)</strong> Before explaining the problem statement, here's an example of markdown headers and their converted link version. Note the use of <code>-1</code> for the second occurrence of the <code>Summary</code> header. Also note that this sample doesn't illustrate every rule explained below.<pre><code class=language-bash># Field separators

## Summary

# Gotchas and Tips

## Summary

* [Field separators](#field-separators)

* [Summary](#summary)

* [Gotchas and Tips](#gotchas-and-tips)

* [Summary](#summary-1)

</code></pre><p>For the input file <code>gawk.md</code>, construct a Table of Content section as per the details described below:<ul><li>Identify all header lines <ul><li>there are two types of header lines, one starting with <code># </code> and the other starting with <code>## </code><li>lines starting with <code>#</code> inside code blocks defined by <code>```bash</code> and <code>```</code> markers should be ignored</ul><li>The headers lines should then be converted as per the following rules: <ul><li>content is defined as the portion of the header ignoring the initial <code>#</code> or <code>##</code> characters and the space character<li><code>##</code> should be replaced with four spaces and a <code>*</code> character<li>else, <code>#</code> should be replaced with <code>*</code> character<li>create a copy of the content, change it to all lowercase, replace all space characters with the <code>-</code> character and then enclose it within <code>(#</code> and <code>)</code> <ul><li>if there are multiple headers with the same content, append <code>-1</code>, <code>-2</code>, etc respectively for the second header, third header, etc</ul><li>surround the original content with <code>[]</code> and then append the string obtained from the previous step</ul><li>Note that the output should have only the converted headers, all other input lines should not be present</ul><p>The script file should be named as <code>toc.awk</code> and save the output in <code>out.md</code>.<pre><code class=language-bash>$ awk -f toc.awk gawk.md > out.md

$ diff -sq out.md toc_expected.md

Files out.md and toc_expected.md are identical

</code></pre><p><strong>2)</strong> For the input file <code>odd.txt</code>, surround the first two whole words of each line with <code>{}</code> that start and end with the same word character. Assume that the input file will not require case insensitive comparison. This is a contrived exercise that needs around 10 instructions and makes you use various features presented in this book.<pre><code class=language-bash>$ cat odd.txt

-oreo-not:a _a2_ roar<=>took%22

RoaR to wow-

$ awk -f same.awk odd.txt

-{oreo}-not:{a} _a2_ roar<=>took%22

{RoaR} to {wow}-

</code></pre></main><nav aria-label="Page navigation"class=nav-wrapper><a aria-label="Previous chapter"class="mobile-nav-chapters previous"title="Previous chapter"aria-keyshortcuts=Left href=dealing-with-duplicates.html rel=prev> <i class="fa fa-angle-left"></i> </a><a aria-label="Next chapter"class="mobile-nav-chapters next"title="Next chapter"aria-keyshortcuts=Right href=gotchas-and-tips.html rel=next> <i class="fa fa-angle-right"></i> </a><div style="clear: both"></div></nav></div></div><nav aria-label="Page navigation"class=nav-wide-wrapper><a aria-label="Previous chapter"class="nav-chapters previous"title="Previous chapter"aria-keyshortcuts=Left href=dealing-with-duplicates.html rel=prev> <i class="fa fa-angle-left"></i> </a><a aria-label="Next chapter"class="nav-chapters next"title="Next chapter"aria-keyshortcuts=Right href=gotchas-and-tips.html rel=next> <i class="fa fa-angle-right"></i> </a></nav></div><script>window.playground_copyable = true;</script><script charset=utf-8 src=elasticlunr.min.js></script><script charset=utf-8 src=mark.min.js></script><script charset=utf-8 src=searcher.js></script><script charset=utf-8 src=clipboard.min.js></script><script charset=utf-8 src=highlight.js></script><script charset=utf-8 src=book.js></script><script src=sidebar.js></script>