deleting a duplicate line in a .txt file using the command line

anniehswong

New Member
Joined
Aug 26, 2017
Messages
1
Reaction score
0
Credits
0
Would someone please advise how I can delete a duplicate line in a .txt file using the command line.

I have a file called Lab1.txt and me.txt

I was suppose to append the content in me.txt to Lab1.txt but somehow I did it twice and now my name is showing twice. I googled and used the command line sed '4d' Lab1.txt to delete the duplicate line, it worked at first but when I typed cat Lab1.txt to review the content, the changes did not save. I'm not exactly sure where I went wrong and I'm pretty new to linux. Would someone please advise? Thanks!

sed '4d' Lab1.txt
 

Attachments

  • Screen Shot 2017-08-25 at 11.27.01 PM.png
    Screen Shot 2017-08-25 at 11.27.01 PM.png
    54.4 KB · Views: 665


Make sure you create a backup of the file first then try this one:

sort garbage.txt | uniq -u
 
The -u will remove all of the matching lines.. they prob just want to:
sort Lab1.txt | uniq

Code:
cat test.txt
1
1
2
3
4
1

cat test.txt | uniq
1
2
3
4
1

sort test.txt | uniq -u
2
3
4

sort test.txt | uniq
1
2
3
4
 
The problem with both of the above solutions is: once you have used sort - the lines in the data file are completely out of order, which wouldn't have fixed the OP's problem.

The best option for the OP would be to use sed with the -i option to edit the file in place.
Code:
sed -i '4d' lab1.txt
The -i option will cause the changes to be written to the original file. Without it, sed just reads the file, makes the specified changes and outputs the altered text to the terminal.

Alternatively - the OP could use redirection to redirect the output from sed to a new file:
Code:
sed '4d' lab1.txt > lab1-fixed.txt
 
  • Like
Reactions: Rob

Members online


Top