Search code examples
bashfilesortingduplicatesin-place

How to remove duplicates from a file and write to the same file?


I know my title is not much self-explanatory but let me try to explain it here.

I have a file name test.txt which has some duplicate lines. Now, what I want to do is remove those duplicate lines and at the same time update test.txt with the new content.

test.txt

AAAA
BBBB
AAAA
CCCC

I know I can use sort -u test.txt to remove the duplicates but to update the file with new content how do I redirect it's output to the same file. The below command doesn't work.

sort -u test.txt > test.txt

So, why the above command is not working and whats the correct way?

Also is there any other way like

sort_and_update_file test.txt

which sorts and automatically updates my file without any need of redirection.


Solution

  • This might work for you:

    sort -u -o test.txt test.txt