Unix uniq command
-
Sorry no response in IT Support and Tech Corner forum so come here post. Need help in below.
I have a problem. I need to use Unix uniq command to remove duplicating lines but with an extra twist.
Example
1 apple a
2 apple b
3 orange c
4 orange d
....
I wanna the command to consider the middle field only (i.e apple, orange) to do removing of duplicates. But it seem uniq is treating the whole line to do comparison when remove duplicates. Any idea how?
Dun ask me use M$ Excel as the command is supposed to be in a Unix script to be executed on commandline by data centre operators in a Unix environment.
TIA -
you mean you're actually writing a shellscript to parse a text file or something bro ? ....Originally posted by sohguanh:Sorry no response in IT Support and Tech Corner forum so come here post. Need help in below.
I have a problem. I need to use Unix uniq command to remove duplicating lines but with an extra twist.
Example
1 apple a
2 apple b
3 orange c
4 orange d
....
I wanna the command to consider the middle field only (i.e apple, orange) to do removing of duplicates. But it seem uniq is treating the whole line to do comparison when remove duplicates. Any idea how?
Dun ask me use M$ Excel as the command is supposed to be in a Unix script to be executed on commandline by data centre operators in a Unix environment.
TIA
if it's a shellscript you're writing, you can try parsing the middle field with regular expressions, then work on from there ...
tell me if it works ...
-
if it's a shellscript you're writing, you can try parsing the middle field with regular expressions, then work on from there ...
Do you mean by first parsing the middle field and then do the uniq command? I wan performance and hence I tot of using juz a single uniq command to do it. If all fails I guess I must use two commands and pipe them together. -
I just logged in and saw your thread, but the problem is that I don't program in Unix... only VB6... sorry.
-
Originally posted by sohguanh:
yes, that's what I meant, if you're just parsing a simple text file then you can filter out all the unneccessary bits then pipe the output to uniq ... and frankly, i don't think there's much difference performance wise ....
[b]if it's a shellscript you're writing, you can try parsing the middle field with regular expressions, then work on from there ...
Do you mean by first parsing the middle field and then do the uniq command? I wan performance and hence I tot of using juz a single uniq command to do it. If all fails I guess I must use two commands and pipe them together.[/b]
I just "man uniq" .... not very helpful there .... let me know if my idea works yah ...
(ahhhh ... hang on ... you may want to tinker around with -f or -s ....
) -
bro ... got the answer ...
you go uniq -f 1 [input file] [outputfile] , where 1 is the first field of each input line
i did a little experiment, cat >> test.txt a whole string of text, seperated by white space ....
and yes, it filters out the first token ....
cheers ! ... -
(ahhhh ... hang on ... you may want to tinker around with -f or -s .... Razz )
I just check the -f option is is always skip the FIRST fields so it means it cannot skip selected fields like say 1st and 3rd field during comparison for removing duplicates -
and yes, it filters out the first token ....
Correct but what I want is to filter out the first and the third token when doing comparison. So "1 apple a" should be same as "2 apple b"
-
Originally posted by sohguanh:
so you just want the middle token eh ? ...
[b]and yes, it filters out the first token ....
Correct but what I want is to filter out the first and the third token when doing comparison. So "1 apple a" should be same as "2 apple b" [/b]
well ... a dirty hack would be to simply parse the output file again with -f 3 again lor ...
"uniq -f 1 [input file] | uniq -f 3 " ..... -
"uniq -f 1 [input file] | uniq -f 3 " .....
Errrr.... do you actually execute those commands you recommend? Do you have a linux or unix accessible to you? I tried yours and it does not work. I will try to post my working solution later assuming I do in two or more commands -
Originally posted by sohguanh:
i did it on my apple terminal and it works ....
[b]"uniq -f 1 [input file] | uniq -f 3 " .....
Errrr.... do you actually execute those commands you recommend? Do you have a linux or unix accessible to you? I tried yours and it does not work. I will try to post my working solution later assuming I do in two or more commands [/b]
should be no problem leh ... try again and see ? ... the above command outputs to standard out ... maybe you try piping to your desired output file and cat it and see how ? ... -
ah ... I see what you mean ... the output doesn't seem to be correct ...
sorry bro ... that i couldn't be much help ... but post the solution up yah ... i'd like to take a look at the final shellscript .. -
cannot use AWK ?
-
bro ...
i just couldn't put this down ... re-read the man again ...
the reason it's not working is cos the -f option ignores the first "x" number of fields, so -f 3 ignores the entire line, instead of ignoring the third token instead ...
hope this helps you with your solution ... good luck -
the reason it's not working is cos the -f option ignores the first "x" number of fields, so -f 3 ignores the entire line, instead of ignoring the third token instead ...
I think I did mention in my earlier post I did say the FIRST "x" number of fields..
anyway me gotta it all work out but i uses 3 process of which 2 is awk maybe someone can help me tune it further to 2? The input file is called a.txt
Contents of a.txt
1 apple a
2 apple b
3 orange c
4 orange d
5 pear e
6 pear f
Result is
1 apple a
3 orange c
5 pear ecode:
awk -F " " '{print $1" "$3" "$2;}' a.txt|uniq -f 2| \
awk -F " " '{print $1" "$3" "$2;}' -
hey sorry guys
given a certain string [11to200]
how can i grab out the 11 and 200 to do a comparision based on a certain value being passed in ??
I need to test conditions where a certain return codes maps to a certain number or range ...similar to one above . -
given a certain string [11to200]
how can i grab out the 11 and 200 to do a comparision based on a certain value being passed in ??
You should give an example of what is the input and what you want the output to look like. Then it is clearer cuz by describing in english sentence sometimes ppl can misinterpret especially for english lousy ppl like me -
Originally posted by sohguanh:
okay cause i got an INI file which requires me to map specfic values to user defined ranges similarly above
[b]given a certain string [11to200]
how can i grab out the 11 and 200 to do a comparision based on a certain value being passed in ??
You should give an example of what is the input and what you want the output to look like. Then it is clearer cuz by describing in english sentence sometimes ppl can misinterpret especially for english lousy ppl like me [/b]
so lets say
criticalerror=[11to20],69
warningerror=[200to234]
given a control script that execute a sub sciript and returns 18,the control script has to map the return code 18 to a critical error which is bounded by the range [11to20] -
# New variables
set UHOMEDIR=`(cat /etc/passwd | awk -F":" '{print $6}')`
set SORTED_UHOMEDIR=`echo $UHOMEDIR | tr " " "\n" | sort -b -t" " -u`
i written something similar
the above will take the 6th field from /etc/passwd
and then sort with the unique entries
hope it helps