<!doctypehtml><html class="sidebar-visible no-js light"lang=en><head><meta charset=UTF-8><title>Multiple file input - CLI text processing with GNU awk</title><meta content="text/html; charset=utf-8"http-equiv=Content-Type><meta content="Example based guide to mastering GNU awk"name=description><meta content=width=device-width,initial-scale=1 name=viewport><meta content=#ffffff name=theme-color><meta content="CLI text processing with GNU awk"property=og:title><meta content=website property=og:type><meta content="Example based guide to mastering GNU awk"property=og:description><meta content=https://learnbyexample.github.io/learn_gnuawk/ property=og:url><meta content=https://raw.githubusercontent.com/learnbyexample/learn_gnuawk/master/images/gawk_ls.png property=og:image><meta content=1280 property=og:image:width><meta content=720 property=og:image:height><meta content=summary_large_image property=twitter:card><meta content=@learn_byexample property=twitter:site><link href=favicon.svg rel=icon><link rel="shortcut icon"href=favicon.png><link href=css/variables.css rel=stylesheet><link href=css/general.css rel=stylesheet><link href=css/chrome.css rel=stylesheet><link href=FontAwesome/css/font-awesome.css rel=stylesheet><link href=fonts/fonts.css rel=stylesheet><link href=highlight.css rel=stylesheet><link href=tomorrow-night.css rel=stylesheet><link href=ayu-highlight.css rel=stylesheet><link href=style.css rel=stylesheet><body><script>var path_to_root = "";

var default_theme = window.matchMedia("(prefers-color-scheme: dark)").matches ? "navy" : "light";</script><script>try {

var theme = localStorage.getItem('mdbook-theme');

var sidebar = localStorage.getItem('mdbook-sidebar');

if (theme.startsWith('"') && theme.endsWith('"')) {

localStorage.setItem('mdbook-theme', theme.slice(1, theme.length - 1));

}

if (sidebar.startsWith('"') && sidebar.endsWith('"')) {

localStorage.setItem('mdbook-sidebar', sidebar.slice(1, sidebar.length - 1));

}

} catch (e) { }</script><script>var theme;

try { theme = localStorage.getItem('mdbook-theme'); } catch(e) { }

if (theme === null || theme === undefined) { theme = default_theme; }

var html = document.querySelector('html');

html.classList.remove('no-js')

html.classList.remove('light')

html.classList.add(theme);

html.classList.add('js');</script><script>var html = document.querySelector('html');

var sidebar = 'hidden';

if (document.body.clientWidth >= 1080) {

try { sidebar = localStorage.getItem('mdbook-sidebar'); } catch(e) { }

sidebar = sidebar || 'visible';

}

html.classList.remove('sidebar-visible');

html.classList.add("sidebar-" + sidebar);</script><nav aria-label="Table of contents"class=sidebar id=sidebar><div class=sidebar-scrollbox><ol class=chapter><li class="chapter-item expanded affix"><a href=cover.html>Cover</a><li class="chapter-item expanded affix"><a href=buy.html>Buy PDF/EPUB versions</a><li class="chapter-item expanded"><a href=preface.html><strong aria-hidden=true>1.</strong> Preface</a><li class="chapter-item expanded"><a href=installation-and-documentation.html><strong aria-hidden=true>2.</strong> Installation and Documentation</a><li class="chapter-item expanded"><a href=awk-introduction.html><strong aria-hidden=true>3.</strong> awk introduction</a><li class="chapter-item expanded"><a href=regular-expressions.html><strong aria-hidden=true>4.</strong> Regular Expressions</a><li class="chapter-item expanded"><a href=field-separators.html><strong aria-hidden=true>5.</strong> Field separators</a><li class="chapter-item expanded"><a href=record-separators.html><strong aria-hidden=true>6.</strong> Record separators</a><li class="chapter-item expanded"><a href=in-place-file-editing.html><strong aria-hidden=true>7.</strong> In-place file editing</a><li class="chapter-item expanded"><a href=using-shell-variables.html><strong aria-hidden=true>8.</strong> Using shell variables</a><li class="chapter-item expanded"><a href=control-structures.html><strong aria-hidden=true>9.</strong> Control Structures</a><li class="chapter-item expanded"><a href=built-in-functions.html><strong aria-hidden=true>10.</strong> Built-in functions</a><li class="chapter-item expanded"><a class=active href=multiple-file-input.html><strong aria-hidden=true>11.</strong> Multiple file input</a><li class="chapter-item expanded"><a href=processing-multiple-records.html><strong aria-hidden=true>12.</strong> Processing multiple records</a><li class="chapter-item expanded"><a href=two-file-processing.html><strong aria-hidden=true>13.</strong> Two file processing</a><li class="chapter-item expanded"><a href=dealing-with-duplicates.html><strong aria-hidden=true>14.</strong> Dealing with duplicates</a><li class="chapter-item expanded"><a href=awk-scripts.html><strong aria-hidden=true>15.</strong> awk scripts</a><li class="chapter-item expanded"><a href=gotchas-and-tips.html><strong aria-hidden=true>16.</strong> Gotchas and Tips</a><li class="chapter-item expanded"><a href=further-reading.html><strong aria-hidden=true>17.</strong> Further Reading</a><li class="chapter-item expanded"><a href=Exercise_solutions.html><strong aria-hidden=true>18.</strong> Exercise Solutions</a></li><br><hr><li class="chapter-item expanded"><i class="fa fa-github"id=git-repository-button></i><a href=https://github.com/learnbyexample/learn_gnuawk>   Source code</a><li class="chapter-item expanded"><i class="fa fa-home"id=home-button></i><a href=https://learnbyexample.github.io/>   My Blog</a><li class="chapter-item expanded"><i class="fa fa-book"id=book-button></i><a href=https://learnbyexample.github.io/books/>   My Books</a><li class="chapter-item expanded"><i class="fa fa-twitter"id=twitter-button></i><a href=https://twitter.com/learn_byexample>   Twitter</a></ol></div><div class=sidebar-resize-handle id=sidebar-resize-handle></div></nav><div class=page-wrapper id=page-wrapper><div class=page><div id=menu-bar-hover-placeholder></div><div class="menu-bar sticky bordered"id=menu-bar><div class=left-buttons><button aria-label="Toggle Table of Contents"title="Toggle Table of Contents"aria-controls=sidebar class=icon-button id=sidebar-toggle type=button><i class="fa fa-bars"></i></button><button aria-label="Change theme"title="Change theme"aria-controls=theme-list aria-expanded=false aria-haspopup=true class=icon-button id=theme-toggle type=button><i class="fa fa-paint-brush"></i></button><ul aria-label=Themes class=theme-popup id=theme-list role=menu><li role=none><button class=theme id=light role=menuitem>Light (default)</button><li role=none><button class=theme id=rust role=menuitem>Rust</button><li role=none><button class=theme id=coal role=menuitem>Coal</button><li role=none><button class=theme id=navy role=menuitem>Navy</button><li role=none><button class=theme id=ayu role=menuitem>Ayu</button></ul><button aria-label="Toggle Searchbar"title="Search. (Shortkey: s)"aria-controls=searchbar aria-expanded=false aria-keyshortcuts=S class=icon-button id=search-toggle type=button><i class="fa fa-search"></i></button></div><h1 class=menu-title>CLI text processing with GNU awk</h1><div class=right-buttons><a aria-label=Blog href=https://learnbyexample.github.io title=Blog> <i class="fa fa-home"id=home-button></i> </a><a aria-label=Twitter href=https://twitter.com/learn_byexample title=Twitter> <i class="fa fa-twitter"id=twitter-button></i> </a><a aria-label="Git repository"title="Git repository"href=https://github.com/learnbyexample/learn_gnuawk> <i class="fa fa-github"id=git-repository-button></i> </a></div></div><div class=hidden id=search-wrapper><form class=searchbar-outer id=searchbar-outer><input placeholder="Search this book ..."aria-controls=searchresults-outer aria-describedby=searchresults-header id=searchbar name=searchbar type=search></form><div class="searchresults-outer hidden"id=searchresults-outer><div class=searchresults-header id=searchresults-header></div><ul id=searchresults></ul></div></div><script>document.getElementById('sidebar-toggle').setAttribute('aria-expanded', sidebar === 'visible');

document.getElementById('sidebar').setAttribute('aria-hidden', sidebar !== 'visible');

Array.from(document.querySelectorAll('#sidebar a')).forEach(function(link) {

link.setAttribute('tabIndex', sidebar === 'visible' ? 0 : -1);

});</script><div class=content id=content><main><div class=sidetoc><nav class=pagetoc></nav></div><h1 id=multiple-file-input><a class=header href=#multiple-file-input>Multiple file input</a></h1><p>You have already seen blocks like <code>BEGIN</code>, <code>END</code> and statements like <code>next</code>. This chapter will discuss features that are useful to make decisions around each file when there are multiple files passed as input.<blockquote><p><img alt=info src=images/info.svg> The <a href=https://github.com/learnbyexample/learn_gnuawk/tree/master/example_files>example_files</a> directory has all the files used in the examples.</blockquote><h2 id=beginfile-endfile-and-filename><a class=header href=#beginfile-endfile-and-filename>BEGINFILE, ENDFILE and FILENAME</a></h2><ul><li><code>BEGINFILE</code> — this block gets executed before the start of each input file<li><code>ENDFILE</code> — this block gets executed after processing each input file<li><code>FILENAME</code> — special variable having the filename of the current input file</ul><p>Here are some examples:<pre><code class=language-bash># can also use: awk 'BEGINFILE{printf "--- %s ---\n", FILENAME} 1'

$ awk 'BEGINFILE{print "--- " FILENAME " ---"} 1' greeting.txt table.txt

--- greeting.txt ---

Hi there

Have a nice day

Good bye

--- table.txt ---

brown bread mat hair 42

blue cake mug shirt -7

yellow banana window shoes 3.14

# same as: tail -q -n1 greeting.txt table.txt

$ awk 'ENDFILE{print $0}' greeting.txt table.txt

Good bye

yellow banana window shoes 3.14

</code></pre><h2 id=nextfile><a class=header href=#nextfile>nextfile</a></h2><p>The <code>nextfile</code> statement helps to skip the remaining records from the current file being processed and move on to the next file. Note that the <code>ENDFILE</code> block will still be executed, if present.<pre><code class=language-bash># print filename if it contains 'I' anywhere in the file

# same as: grep -l 'I' f[1-3].txt greeting.txt

$ awk '/I/{print FILENAME; nextfile}' f[1-3].txt greeting.txt

f1.txt

f2.txt

# print filename if it contains both 'o' and 'at' anywhere in the file

$ awk 'BEGINFILE{m1=m2=0} /o/{m1=1} /at/{m2=1}

m1 && m2{print FILENAME; nextfile}' f[1-3].txt greeting.txt

f2.txt

f3.txt

# print filename if it contains 'at' but not 'o'

$ awk 'BEGINFILE{m1=m2=0} /o/{m1=1; nextfile} /at/{m2=1}

ENDFILE{if(!m1 && m2) print FILENAME}' f[1-3].txt greeting.txt

f1.txt

</code></pre><blockquote><p><img alt=warning src=images/warning.svg> <code>nextfile</code> cannot be used in the <code>BEGIN</code> or <code>END</code> or <code>ENDFILE</code> blocks. See <a href=https://www.gnu.org/software/gawk/manual/gawk.html#Nextfile-Statement>gawk manual: nextfile</a> for more details, how it affects <code>ENDFILE</code> and other special cases.</blockquote><h2 id=argc-and-argv><a class=header href=#argc-and-argv>ARGC and ARGV</a></h2><p>The <code>ARGC</code> special variable contains the total number of arguments passed to the <code>awk</code> command, including <code>awk</code> itself as an argument. The <code>ARGV</code> special array contains the arguments themselves.<pre><code class=language-bash># note that the index starts with '0' here

$ awk 'BEGIN{for(i=0; i&LTARGC; i++) print ARGV[i]}' f[1-3].txt greeting.txt

awk

f1.txt

f2.txt

f3.txt

greeting.txt

</code></pre><p>Similar to manipulating <code>NF</code> and modifying <code>$N</code> field contents, you can change the values of <code>ARGC</code> and <code>ARGV</code> to control how the arguments should be processed.<p>However, not all arguments are necessarily filenames. <code>awk</code> allows assigning variable values without the <code>-v</code> option if it is done in the place where you usually provide file arguments. For example:<pre><code class=language-bash>$ awk 'BEGIN{for(i=0; i&LTARGC; i++) print ARGV[i]}' table.txt n=5 greeting.txt

awk

table.txt

n=5

greeting.txt

</code></pre><p>In the above example, the variable <code>n</code> will get a value of <code>5</code> after <code>awk</code> has finished processing the <code>table.txt</code> file. Here's an example where <code>FS</code> is changed between two files.<pre><code class=language-bash>$ cat table.txt

brown bread mat hair 42

blue cake mug shirt -7

yellow banana window shoes 3.14

$ cat books.csv

Harry Potter,Mistborn,To Kill a Mocking Bird

Matilda,Castle Hangnail,Jane Eyre

# for table.txt, FS will be the default value

# for books.csv, FS will be the comma character

# OFS is comma for both the files

$ awk -v OFS=, 'NF=2' table.txt FS=, books.csv

brown,bread

blue,cake

yellow,banana

Harry Potter,Mistborn

Matilda,Castle Hangnail

</code></pre><blockquote><p><img alt=info src=images/info.svg> See <a href=https://stackoverflow.com/a/64427745/4082052>stackoverflow: extract positions 2-7 from a fasta sequence</a> for a practical example of changing field/record separators between the files being processed.</blockquote><h2 id=summary><a class=header href=#summary>Summary</a></h2><p>This chapter introduced a few more special blocks and variables that are handy for processing multiple file inputs. These will show up in examples in the coming chapters as well.<p>Next chapter will discuss use cases where you need to take decisions based on multiple input records.<h2 id=exercises><a class=header href=#exercises>Exercises</a></h2><blockquote><p><img alt=info src=images/info.svg> The <a href=https://github.com/learnbyexample/learn_gnuawk/tree/master/exercises>exercises</a> directory has all the files used in this section.</blockquote><p><strong>1)</strong> Print the last field of the first two lines for the input files <code>table.txt</code>, <code>scores.csv</code> and <code>fw.txt</code>. The field separators for these files are space, comma and fixed width respectively. To make the output more informative, print filenames and a separator as shown in the output below. Assume that the input files will have at least two lines.<pre><code class=language-bash>$ awk ##### add your solution here

>table.txt<

>scores.csv<

>fw.txt<

</code></pre><p><strong>2)</strong> For the input files <code>sample.txt</code>, <code>secrets.txt</code>, <code>addr.txt</code> and <code>table.txt</code>, display only the names of files that contain <code>in</code> or <code>at</code> or <code>fun</code> in the third field. Assume space as the field separator. The output should not show a matching filename more than once.<pre><code class=language-bash>$ awk ##### add your solution here sample.txt secrets.txt addr.txt table.txt

secrets.txt

addr.txt

table.txt

</code></pre></main><nav aria-label="Page navigation"class=nav-wrapper><a aria-label="Previous chapter"class="mobile-nav-chapters previous"title="Previous chapter"aria-keyshortcuts=Left href=built-in-functions.html rel=prev> <i class="fa fa-angle-left"></i> </a><a aria-label="Next chapter"class="mobile-nav-chapters next"title="Next chapter"aria-keyshortcuts=Right href=processing-multiple-records.html rel=next> <i class="fa fa-angle-right"></i> </a><div style="clear: both"></div></nav></div></div><nav aria-label="Page navigation"class=nav-wide-wrapper><a aria-label="Previous chapter"class="nav-chapters previous"title="Previous chapter"aria-keyshortcuts=Left href=built-in-functions.html rel=prev> <i class="fa fa-angle-left"></i> </a><a aria-label="Next chapter"class="nav-chapters next"title="Next chapter"aria-keyshortcuts=Right href=processing-multiple-records.html rel=next> <i class="fa fa-angle-right"></i> </a></nav></div><script>window.playground_copyable = true;</script><script charset=utf-8 src=elasticlunr.min.js></script><script charset=utf-8 src=mark.min.js></script><script charset=utf-8 src=searcher.js></script><script charset=utf-8 src=clipboard.min.js></script><script charset=utf-8 src=highlight.js></script><script charset=utf-8 src=book.js></script><script src=sidebar.js></script>