1

After asking a question on ubuntuforums.org and not getting a satisfying answer, I've decided to ask the question again here on Ask Ubuntu. I need the answer to be very detailed. Specifically, I need to know which lines get compared every time a line is printed using uniq for the following two examples:

file1.txt:

$ cat -A file1.txt
aaa^Iupc$
b$
c$
aaa^Iztp$
b$
c$
C$
A$
B$
B$
b$

$ sort file1.txt | uniq -f 1
A
aaa    upc
aaa    ztp
b

and file2.txt:

$ cat -A file2.txt
aaa^Iupc$
b$
c$
aaa^Iztp$
b$
c$
C$
A$
B$
B$
bbb^Ixpz$

$ sort file2.txt | uniq -f 1
A
aaa    upc
aaa    ztp
b
bbb    xpz
c

I'm confused about the second example. I don't understand how come uppercase B doesn't end up in the final output. Shouldn't the line with uppercase B be printed given that lines B and bbb xpz are both adjacent to each other? If:

B ---> (empty)

and

bbb ---> xpz

an empty value and xpz are both unique so both lines should be printed. Or am I missing something?

muru
  • 207,970
John_Patrick_Mason
  • 149

2 Answers2

2

The answer lies in the sorting order and what does uniq uses for a field value when less than the given field number (N) exists while using -f N.

As seen you have ASCII charsets, so the sorting order is much predictable:

% sort file.txt            
A
aaa upc
aaa ztp
b
b
B
B
bbb xpz
c
c
C

Now, let's use uniq -f 1 to get unique lines with skipping the (whitespace separated) first field of each line while checking:

% sort file.txt | uniq -f 1
A
aaa upc
aaa ztp
b
bbb xpz
c

Now, the important thing to note that, uniq uses null string for lines that has has less than the fields mentioned, 1 in this case; so, all the lines that has only one field would be treated as having null strings for other fields while comparing with other lines having >=2 fields.

So, from the sort file2.txt output:

b
b
B
B

would all be treated as same and only the first line containing b would be preserved, hence you have a b in the output.

Similarly, from:

c
c
C

only the first c would end up in the uniq's output.

heemayl
  • 94,145
  • OK, but how come typing sort file2.txt | uniq -f 1 yields uppercase A and lowercase b in the second file? Shouldn't both be left out of comparison and not influence the output? – John_Patrick_Mason Jun 09 '17 at 01:56
  • @John_Patrick_Mason Expanded, check now. – heemayl Jun 09 '17 at 02:20
  • I think I get it now. I replaced every alphabetical character that has only one field with the letter z to better visualize what's happening. If I understand correctly, the lines that have only one field mess up what uniq sees. If I'm correct, it's as if I hadn't used the sort command at all and passed through uniq an unsorted file. – John_Patrick_Mason Jun 09 '17 at 03:19
  • @John_Patrick_Mason No, not quite. Without sort, you would not get the liens sorted at the first place. SO, only the successive lines that have single field in the original file would result in only the first line. – heemayl Jun 09 '17 at 03:22
  • @heemaul When I said I replaced every alphabetical character, with only one field, with the letter z, to better visualize, I meant that I created a new file, first, based on the output of: sort file2.txt. Does that make sense? – John_Patrick_Mason Jun 09 '17 at 03:46
  • @John_Patrick_Mason No, it does not. Now you are modifying the collation order of the whole file as those lines now end in z. To get a better view, run sort file2.txt with and without adding z. – heemayl Jun 09 '17 at 03:51
  • The new file that I created -- let's call it file3.txt, for the sake of simplicity -- was not passed through uniq. z was only meant to represent a null string, as a visual aid. Once I created file3.txt, I typed: cat file3.txt, then typed: sort file2.txt (to compare it), then typed: sort file2.txt | uniq -f 1. I know a null string is not technically the same thing as the letter z. – John_Patrick_Mason Jun 09 '17 at 04:41
0

Here's a table that may help you to work through the process:

----------------+---------------+----------+----------------+
    sort        |     Remove    | Adjacent |                |
 (C locale)     |    field #1   |  match?  |    Output      |
----------------+---------------+----------+----------------+
A               |               |    N*    |A               |
B               |               |    Y     |                |
B               |               |    Y     |                |
C               |               |    Y     |                |
aaa     upc     |   upc         |    N     |aaa     upc     |
aaa     ztp     |   ztp         |    N     |aaa     ztp     |
b               |               |    N     |b               |
b               |               |    Y     |                |
bbb     xpz     |   xpz         |    N     |bbb     xpz     |
c               |               |    N     |c               |
c               |               |    Y     |                |
----------------+---------------+----------+----------------+
* the first line has no adjacent above, so is always output
steeldriver
  • 143,099