<!doctypehtml><html class="sidebar-visible no-js light"lang=en><head><meta charset=UTF-8><title>Dealing with duplicates - CLI text processing with GNU awk</title><meta content="text/html; charset=utf-8"http-equiv=Content-Type><meta content="Example based guide to mastering GNU awk"name=description><meta content=width=device-width,initial-scale=1 name=viewport><meta content=#ffffff name=theme-color><meta content="CLI text processing with GNU awk"property=og:title><meta content=website property=og:type><meta content="Example based guide to mastering GNU awk"property=og:description><meta content=https://learnbyexample.github.io/learn_gnuawk/ property=og:url><meta content=https://raw.githubusercontent.com/learnbyexample/learn_gnuawk/master/images/gawk_ls.png property=og:image><meta content=1280 property=og:image:width><meta content=720 property=og:image:height><meta content=summary_large_image property=twitter:card><meta content=@learn_byexample property=twitter:site><link href=favicon.svg rel=icon><link rel="shortcut icon"href=favicon.png><link href=css/variables.css rel=stylesheet><link href=css/general.css rel=stylesheet><link href=css/chrome.css rel=stylesheet><link href=FontAwesome/css/font-awesome.css rel=stylesheet><link href=fonts/fonts.css rel=stylesheet><link href=highlight.css rel=stylesheet><link href=tomorrow-night.css rel=stylesheet><link href=ayu-highlight.css rel=stylesheet><link href=style.css rel=stylesheet><body><script>var path_to_root = "";

var default_theme = window.matchMedia("(prefers-color-scheme: dark)").matches ? "navy" : "light";</script><script>try {

var theme = localStorage.getItem('mdbook-theme');

var sidebar = localStorage.getItem('mdbook-sidebar');

if (theme.startsWith('"') && theme.endsWith('"')) {

localStorage.setItem('mdbook-theme', theme.slice(1, theme.length - 1));

}

if (sidebar.startsWith('"') && sidebar.endsWith('"')) {

localStorage.setItem('mdbook-sidebar', sidebar.slice(1, sidebar.length - 1));

}

} catch (e) { }</script><script>var theme;

try { theme = localStorage.getItem('mdbook-theme'); } catch(e) { }

if (theme === null || theme === undefined) { theme = default_theme; }

var html = document.querySelector('html');

html.classList.remove('no-js')

html.classList.remove('light')

html.classList.add(theme);

html.classList.add('js');</script><script>var html = document.querySelector('html');

var sidebar = 'hidden';

if (document.body.clientWidth >= 1080) {

try { sidebar = localStorage.getItem('mdbook-sidebar'); } catch(e) { }

sidebar = sidebar || 'visible';

}

html.classList.remove('sidebar-visible');

html.classList.add("sidebar-" + sidebar);</script><nav aria-label="Table of contents"class=sidebar id=sidebar><div class=sidebar-scrollbox><ol class=chapter><li class="chapter-item expanded affix"><a href=cover.html>Cover</a><li class="chapter-item expanded affix"><a href=buy.html>Buy PDF/EPUB versions</a><li class="chapter-item expanded"><a href=preface.html><strong aria-hidden=true>1.</strong> Preface</a><li class="chapter-item expanded"><a href=installation-and-documentation.html><strong aria-hidden=true>2.</strong> Installation and Documentation</a><li class="chapter-item expanded"><a href=awk-introduction.html><strong aria-hidden=true>3.</strong> awk introduction</a><li class="chapter-item expanded"><a href=regular-expressions.html><strong aria-hidden=true>4.</strong> Regular Expressions</a><li class="chapter-item expanded"><a href=field-separators.html><strong aria-hidden=true>5.</strong> Field separators</a><li class="chapter-item expanded"><a href=record-separators.html><strong aria-hidden=true>6.</strong> Record separators</a><li class="chapter-item expanded"><a href=in-place-file-editing.html><strong aria-hidden=true>7.</strong> In-place file editing</a><li class="chapter-item expanded"><a href=using-shell-variables.html><strong aria-hidden=true>8.</strong> Using shell variables</a><li class="chapter-item expanded"><a href=control-structures.html><strong aria-hidden=true>9.</strong> Control Structures</a><li class="chapter-item expanded"><a href=built-in-functions.html><strong aria-hidden=true>10.</strong> Built-in functions</a><li class="chapter-item expanded"><a href=multiple-file-input.html><strong aria-hidden=true>11.</strong> Multiple file input</a><li class="chapter-item expanded"><a href=processing-multiple-records.html><strong aria-hidden=true>12.</strong> Processing multiple records</a><li class="chapter-item expanded"><a href=two-file-processing.html><strong aria-hidden=true>13.</strong> Two file processing</a><li class="chapter-item expanded"><a class=active href=dealing-with-duplicates.html><strong aria-hidden=true>14.</strong> Dealing with duplicates</a><li class="chapter-item expanded"><a href=awk-scripts.html><strong aria-hidden=true>15.</strong> awk scripts</a><li class="chapter-item expanded"><a href=gotchas-and-tips.html><strong aria-hidden=true>16.</strong> Gotchas and Tips</a><li class="chapter-item expanded"><a href=further-reading.html><strong aria-hidden=true>17.</strong> Further Reading</a><li class="chapter-item expanded"><a href=Exercise_solutions.html><strong aria-hidden=true>18.</strong> Exercise Solutions</a></li><br><hr><li class="chapter-item expanded"><i class="fa fa-github"id=git-repository-button></i><a href=https://github.com/learnbyexample/learn_gnuawk>   Source code</a><li class="chapter-item expanded"><i class="fa fa-home"id=home-button></i><a href=https://learnbyexample.github.io/>   My Blog</a><li class="chapter-item expanded"><i class="fa fa-book"id=book-button></i><a href=https://learnbyexample.github.io/books/>   My Books</a><li class="chapter-item expanded"><i class="fa fa-twitter"id=twitter-button></i><a href=https://twitter.com/learn_byexample>   Twitter</a></ol></div><div class=sidebar-resize-handle id=sidebar-resize-handle></div></nav><div class=page-wrapper id=page-wrapper><div class=page><div id=menu-bar-hover-placeholder></div><div class="menu-bar sticky bordered"id=menu-bar><div class=left-buttons><button aria-label="Toggle Table of Contents"title="Toggle Table of Contents"aria-controls=sidebar class=icon-button id=sidebar-toggle type=button><i class="fa fa-bars"></i></button><button aria-label="Change theme"title="Change theme"aria-controls=theme-list aria-expanded=false aria-haspopup=true class=icon-button id=theme-toggle type=button><i class="fa fa-paint-brush"></i></button><ul aria-label=Themes class=theme-popup id=theme-list role=menu><li role=none><button class=theme id=light role=menuitem>Light (default)</button><li role=none><button class=theme id=rust role=menuitem>Rust</button><li role=none><button class=theme id=coal role=menuitem>Coal</button><li role=none><button class=theme id=navy role=menuitem>Navy</button><li role=none><button class=theme id=ayu role=menuitem>Ayu</button></ul><button aria-label="Toggle Searchbar"title="Search. (Shortkey: s)"aria-controls=searchbar aria-expanded=false aria-keyshortcuts=S class=icon-button id=search-toggle type=button><i class="fa fa-search"></i></button></div><h1 class=menu-title>CLI text processing with GNU awk</h1><div class=right-buttons><a aria-label=Blog href=https://learnbyexample.github.io title=Blog> <i class="fa fa-home"id=home-button></i> </a><a aria-label=Twitter href=https://twitter.com/learn_byexample title=Twitter> <i class="fa fa-twitter"id=twitter-button></i> </a><a aria-label="Git repository"title="Git repository"href=https://github.com/learnbyexample/learn_gnuawk> <i class="fa fa-github"id=git-repository-button></i> </a></div></div><div class=hidden id=search-wrapper><form class=searchbar-outer id=searchbar-outer><input placeholder="Search this book ..."aria-controls=searchresults-outer aria-describedby=searchresults-header id=searchbar name=searchbar type=search></form><div class="searchresults-outer hidden"id=searchresults-outer><div class=searchresults-header id=searchresults-header></div><ul id=searchresults></ul></div></div><script>document.getElementById('sidebar-toggle').setAttribute('aria-expanded', sidebar === 'visible');

document.getElementById('sidebar').setAttribute('aria-hidden', sidebar !== 'visible');

Array.from(document.querySelectorAll('#sidebar a')).forEach(function(link) {

link.setAttribute('tabIndex', sidebar === 'visible' ? 0 : -1);

});</script><div class=content id=content><main><div class=sidetoc><nav class=pagetoc></nav></div><h1 id=dealing-with-duplicates><a class=header href=#dealing-with-duplicates>Dealing with duplicates</a></h1><p>Often, you need to eliminate duplicates from an input file. This could be based on the entire line content or based on certain fields. These are typically solved with the <code>sort</code> and <code>uniq</code> commands. Advantages with <code>awk</code> include regexp based field and record separators, input doesn't have to be sorted, and in general more flexibility because it is a programming language.<blockquote><p><img alt=info src=images/info.svg> The <a href=https://github.com/learnbyexample/learn_gnuawk/tree/master/example_files>example_files</a> directory has all the files used in the examples.</blockquote><h2 id=whole-line-duplicates><a class=header href=#whole-line-duplicates>Whole line duplicates</a></h2><p><code>awk '!a[$0]++'</code> is one of the most famous <code>awk</code> one-liners. It eliminates line based duplicates while retaining the input order. The following example shows it in action along with an illustration of how the logic works.<pre><code class=language-bash>$ cat purchases.txt

coffee

tea

washing powder

coffee

toothpaste

tea

soap

tea

$ awk '{print +a[$0] "\t" $0; a[$0]++}' purchases.txt

0 coffee

0 tea

0 washing powder

1 coffee

0 toothpaste

1 tea

0 soap

2 tea

# only those entries with zero in the first column will be retained

$ awk '!a[$0]++' purchases.txt

coffee

tea

washing powder

toothpaste

soap

</code></pre><blockquote><p><img alt=info src=images/info.svg> See also <a href=https://github.com/koraa/huniq>huniq</a>, a faster alternative for removing line based duplicates.</blockquote><h2 id=column-wise-duplicates><a class=header href=#column-wise-duplicates>Column wise duplicates</a></h2><p>Removing field based duplicates is simple for a single field comparison. Just change <code>$0</code> to the required field number after setting the appropriate field separator.<pre><code class=language-bash>$ cat duplicates.txt

brown,toy,bread,42

dark red,ruby,rose,111

blue,ruby,water,333

dark red,sky,rose,555

yellow,toy,flower,333

white,sky,bread,111

light red,purse,rose,333

# based on the last field

$ awk -F, '!seen[$NF]++' duplicates.txt

brown,toy,bread,42

dark red,ruby,rose,111

blue,ruby,water,333

dark red,sky,rose,555

</code></pre><p>For multiple fields comparison, separate the fields with <code>,</code> so that <code>SUBSEP</code> is used to combine the field values to generate the key. As mentioned before, <code>SUBSEP</code> has a default value of <code>\034</code> non-printing character, which is typically not used in text files.<pre><code class=language-bash># based on the first and third fields

$ awk -F, '!seen[$1,$3]++' duplicates.txt

brown,toy,bread,42

dark red,ruby,rose,111

blue,ruby,water,333

yellow,toy,flower,333

white,sky,bread,111

light red,purse,rose,333

</code></pre><h2 id=duplicate-count><a class=header href=#duplicate-count>Duplicate count</a></h2><p>In this section, how many times a duplicate record is found plays a role in determining the output.<p>First up, printing only a specific numbered duplicate.<pre><code class=language-bash># print only the second occurrence of duplicates based on the second field

$ awk -F, '++seen[$2]==2' duplicates.txt

blue,ruby,water,333

yellow,toy,flower,333

white,sky,bread,111

# print only the third occurrence of duplicates based on the last field

$ awk -F, '++seen[$NF]==3' duplicates.txt

light red,purse,rose,333

</code></pre><p>Next, printing only the last copy of duplicates. Since the count isn't known, the <code>tac</code> command comes in handy again.<pre><code class=language-bash># reverse the input line-wise, retain the first copy and then reverse again

$ tac duplicates.txt | awk -F, '!seen[$NF]++' | tac

brown,toy,bread,42

dark red,sky,rose,555

white,sky,bread,111

light red,purse,rose,333

</code></pre><p>To get all the records based on a duplicate count, you can pass the input file twice. Then use the two file processing trick to make decisions.<pre><code class=language-bash># all duplicates based on the last column

$ awk -F, 'NR==FNR{a[$NF]++; next} a[$NF]>1' duplicates.txt duplicates.txt

dark red,ruby,rose,111

blue,ruby,water,333

yellow,toy,flower,333

white,sky,bread,111

light red,purse,rose,333

# all duplicates based on the last column, minimum 3 duplicates

$ awk -F, 'NR==FNR{a[$NF]++; next} a[$NF]>2' duplicates.txt duplicates.txt

blue,ruby,water,333

yellow,toy,flower,333

light red,purse,rose,333

# only unique lines based on the third column

$ awk -F, 'NR==FNR{a[$3]++; next} a[$3]==1' duplicates.txt duplicates.txt

blue,ruby,water,333

yellow,toy,flower,333

</code></pre><h2 id=summary><a class=header href=#summary>Summary</a></h2><p>This chapter showed how to work with duplicate contents for records and fields. If you don't need regexp based separators and if your input is too big to handle, then specialized command line tools like <code>sort</code> and <code>uniq</code> will be better suited compared to <code>awk</code>.<p>Next chapter will show how to write <code>awk</code> scripts instead of the usual one-liners.<h2 id=exercises><a class=header href=#exercises>Exercises</a></h2><blockquote><p><img alt=info src=images/info.svg> The <a href=https://github.com/learnbyexample/learn_gnuawk/tree/master/exercises>exercises</a> directory has all the files used in this section.</blockquote><p><strong>1)</strong> Retain only the first copy of a line for the input file <code>lines.txt</code>. Case should be ignored while comparing the lines. For example, <code>hi there</code> and <code>HI TheRE</code> should be considered as duplicates.<pre><code class=language-bash>$ cat lines.txt

Go There

come on

go there

---

2 apples and 5 mangoes

come on!

---

2 Apples

COME ON

$ awk ##### add your solution here

Go There

come on

---

2 apples and 5 mangoes

come on!

2 Apples

</code></pre><p><strong>2)</strong> Retain only the first copy of a line for the input file <code>twos.txt</code>. Assume space as the field separator with exactly two fields per line. Compare the lines irrespective of the order of the fields. For example, <code>hehe haha</code> and <code>haha hehe</code> should be considered as duplicates.<pre><code class=language-bash>$ cat twos.txt

hehe haha

door floor

haha hehe

6;8 3-4

true blue

hehe bebe

floor door

3-4 6;8

tru eblue

haha hehe

$ awk ##### add your solution here

hehe haha

door floor

6;8 3-4

true blue

hehe bebe

tru eblue

</code></pre><p><strong>3)</strong> For the input file <code>twos.txt</code>, create a file <code>uniq.txt</code> with all the unique lines and <code>dupl.txt</code> with all the duplicate lines. Assume space as the field separator with exactly two fields per line. Compare the lines irrespective of the order of the fields. For example, <code>hehe haha</code> and <code>haha hehe</code> should be considered as duplicates.<pre><code class=language-bash>$ awk ##### add your solution here

$ cat uniq.txt

true blue

hehe bebe

tru eblue

$ cat dupl.txt

hehe haha

door floor

haha hehe

6;8 3-4

floor door

3-4 6;8

haha hehe

</code></pre></main><nav aria-label="Page navigation"class=nav-wrapper><a aria-label="Previous chapter"class="mobile-nav-chapters previous"title="Previous chapter"aria-keyshortcuts=Left href=two-file-processing.html rel=prev> <i class="fa fa-angle-left"></i> </a><a aria-label="Next chapter"class="mobile-nav-chapters next"title="Next chapter"aria-keyshortcuts=Right href=awk-scripts.html rel=next> <i class="fa fa-angle-right"></i> </a><div style="clear: both"></div></nav></div></div><nav aria-label="Page navigation"class=nav-wide-wrapper><a aria-label="Previous chapter"class="nav-chapters previous"title="Previous chapter"aria-keyshortcuts=Left href=two-file-processing.html rel=prev> <i class="fa fa-angle-left"></i> </a><a aria-label="Next chapter"class="nav-chapters next"title="Next chapter"aria-keyshortcuts=Right href=awk-scripts.html rel=next> <i class="fa fa-angle-right"></i> </a></nav></div><script>window.playground_copyable = true;</script><script charset=utf-8 src=elasticlunr.min.js></script><script charset=utf-8 src=mark.min.js></script><script charset=utf-8 src=searcher.js></script><script charset=utf-8 src=clipboard.min.js></script><script charset=utf-8 src=highlight.js></script><script charset=utf-8 src=book.js></script><script src=sidebar.js></script>