uniq not working

uniq not working

[Felipe Vieira]

Dear mailing list,

the uniq program seems to be faulty on my cygwin:

/tmp » cat u.txt
1
2
3
4
5
1
2
3
6
7
8


/tmp » uniq -c u.txt
      1 1
      1 2
      1 3
      1 4
      1 5
      1 1
      1 2
      1 3
      1 6
      1 7
      1 8
      1
/tmp »

As you can see it does not eliminate duplicate lines.
This file was created with vim. Same results if created with windows notepad.
What am I missing?

Best,

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: uniq not working

[Csaba Raduly]

Hi Felipe,

On Tue, Oct 11, 2016 at 12:43 PM, Felipe Vieira  wrote:
> Dear mailing list,
>
> the uniq program seems to be faulty on my cygwin:
>
> /tmp » cat u.txt
> 1
> 2
> 3
> 4
> 5
> 1
> 2
> 3
> 6
> 7
> 8
>
>
> /tmp » uniq -c u.txt
>       1 1
>       1 2
>       1 3
>       1 4
>       1 5
>       1 1
>       1 2
>       1 3
>       1 6
>       1 7
>       1 8
>       1
> /tmp »
>
> As you can see it does not eliminate duplicate lines.
> This file was created with vim. Same results if created with windows notepad.
> What am I missing?

You are missing an important step: reading the manual.

$ man uniq
UNIQ(1)
                         User Commands
                                                        UNIQ(1)

NAME
       uniq - report or omit repeated lines

SYNOPSIS
       uniq [OPTION]... [INPUT [OUTPUT]]

DESCRIPTION
       Filter adjacent matching lines from INPUT (or standard input),
writing to OUTPUT (or standard output).
               ^^^^^^^^^

uniq collapses identical lines only if they are consecutive.
The typical way to ensure this is to sort the file first.

$ sort c.txt | uniq

Csaba
-- 
GCS a+ e++ d- C++ ULS$ L+$ !E- W++ P+++$ w++$ tv+ b++ DI D++ 5++
The Tao of math: The numbers you can count are not the real numbers.
Life is complex, with real and imaginary parts.
"Ok, it boots. Which means it must be bug-free and perfect. " -- Linus Torvalds
"People disagree with me. I just ignore them." -- Linus Torvalds

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: uniq not working

[Henry S. Thompson]

You may find the following bash function useful:

sus () 
{ 
    sort "$@" | uniq -c | sort -k1nr,1
}

With you data:

> sus u.txt

      2 1
      2 2
      2 3
      1 4
      1 5
      1 6
      1 7
      1 8

ht
-- 
       Henry S. Thompson, School of Informatics, University of Edinburgh
      10 Crichton Street, Edinburgh EH8 9AB, SCOTLAND -- (44) 131 650-4440
                Fax: (44) 131 650-4587, e-mail: ht@inf.ed.ac.uk
                       URL: http://www.ltg.ed.ac.uk/~ht/
 [mail from me _always_ has a .sig like this -- mail without it is forged spam]

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: uniq not working

[Andrey Repin]

Greetings, Henry S. Thompson!

> You may find the following bash function useful:

> sus () 
> { 
>     sort "$@" | uniq -c | sort -k1nr,1
> }

Why not sort -u ?

> With you data:

>> sus u.txt

>       2 1
>       2 2
>       2 3
>       1 4
>       1 5
>       1 6
>       1 7
>       1 8

> ht


-- 
With best regards,
Andrey Repin
Tuesday, October 11, 2016 17:18:39

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: uniq not working

[Henry S. Thompson]

Andrey Repin writes:

> Greetings, Henry S. Thompson!
>
>> You may find the following bash function useful:
>
>> sus () 
>> { 
>>     sort "$@" | uniq -c | sort -k1nr,1
>> }
>
> Why not sort -u ?

Because then all the counts will be 1.

ht
-- 
       Henry S. Thompson, School of Informatics, University of Edinburgh
      10 Crichton Street, Edinburgh EH8 9AB, SCOTLAND -- (44) 131 650-4440
                Fax: (44) 131 650-4587, e-mail: ht@inf.ed.ac.uk
                       URL: http://www.ltg.ed.ac.uk/~ht/
 [mail from me _always_ has a .sig like this -- mail without it is forged spam]

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: uniq not working

[Markus Schönhaber]

Am 11.10.2016 um 12:43 schrieb Felipe Vieira:

> the uniq program seems to be faulty on my cygwin:
> 
[...]
> /tmp » uniq -c u.txt
>       1 1
>       1 2
>       1 3
>       1 4
>       1 5
>       1 1
>       1 2
>       1 3
>       1 6
>       1 7
>       1 8
>       1
> /tmp »
> 
> As you can see it does not eliminate duplicate lines.
> This file was created with vim. Same results if created with windows notepad.
> What am I missing?

From man uniq:

| Note:  'uniq' does not detect repeated lines unless they are
| adjacent.  You may want to sort the input first, or use 'sort -u'
| without 'uniq'.

-- 
Regards
  mks


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: uniq not working

[Felipe Vieira]

On Tue, Oct 11, 2016 at 7:43 AM, Felipe Vieira <fmv1992@gmail.com> wrote:
> Dear mailing list,
>
> the uniq program seems to be faulty on my cygwin:
>
> /tmp » cat u.txt
> 1
> 2
> 3
> 4
> 5
> 1
> 2
> 3
> 6
> 7
> 8
>
>
> /tmp » uniq -c u.txt
>       1 1
>       1 2
>       1 3
>       1 4
>       1 5
>       1 1
>       1 2
>       1 3
>       1 6
>       1 7
>       1 8
>       1
> /tmp »
>
> As you can see it does not eliminate duplicate lines.
> This file was created with vim. Same results if created with windows notepad.
> What am I missing?
>
> Best,

Thanks Csaba.

I have read the manual but did not understand it completely; missed
one word (adjacent) and that changed everything.

Best regards : )

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple