Limit lines in document to only so many characters, How?

None-yet

Member
Credits
576
I am not sure this even exists. I have a large wordlist file. Anyone who has worked with a wordlist will understand what I am going to ask here. This file is over 4 gb. I rarely open it. It has mixed in lines of very long text that appear to be random numbers and letters with like 100 characters. I would like to remove these using command line or something because to open the file takes a while and those lines are mixed here and there. I would take a good long effort to do by hand.

Does anyone know of a way I can delete these lines with a command line interface. I was thinking a command that would be able to limit each line to only a certain number of characters or something. If anyone knows how I can do this please also give me a breakdown of what you think will work. I am here to learn. Thanks
 


f33dm3bits

Gold Member
Gold Supporter
Credits
3,480
Do you have a few examples of words in that list that belong on the list and a few of those that you want removed.
 

None-yet

Member
Credits
576
Sorry for it taking so long. Client came in.

The ones that belong are just standard dictionary words. The line that don't belong. Below.


++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

donesyq54c8s9w63w5s96w2z6w2q6s265q6ssddcd63c6dcd3c2d6ewd6df2d3363d36ew6fdf3ac33636d3333e3de3sa3JohnTheRipperf6f5vdfv1dgb4gfsd+c4+dsv4vsv4f4gb+fv+ g4bt+y4hr9t4hgf6 _1cxz3ca
 

None-yet

Member
Credits
576
I greatly appreciate the answer. I went here and here. Didn't find an example of what I want to do. Plus with all of these you have to have the file open. This file being so large I am at the point of not wanting to open it because it takes so long. I was hoping for something that would take a command line and scan the document while it is closed and delete the unwanted lines. I may have to leave them in there but have never asked about doing this so I wanted to give this a shot.

Any suggestions?
 

None-yet

Member
Credits
576
That should give me something to go on. Did a quick look and it looks like it should work. Thank You Sir! If it doesn't or I need some help since I know very little about sed and awk I may post back. If anyone see's this that knows these post me a hello.

One question though. What would be a good start for me to study both sed and awk? I would love to read up on either or both to get a good understanding of them.

Thanks again gvisoc!
 

f33dm3bits

Gold Member
Gold Supporter
Credits
3,480
That should give me something to go on. Did a quick look and it looks like it should work. Thank You Sir! If it doesn't or I need some help since I know very little about sed and awk I may post back. If anyone see's this that knows these post me a hello.

One question though. What would be a good start for me to study both sed and awk? I would love to read up on either or both to get a good understanding of them.

Thanks again gvisoc!
Like the previous two posters said, vim, awk or sed should be able to do the job. My experience with them is limited so wouldn't know what to tell you. I would search do a google search with something around the lines of learn sed and awk.
 

None-yet

Member
Credits
576
I am actually trying to open the file now in vim to check something. However after 10 minutes it has not opened or it just may still be loading. It is a very large file. I am going to look through my library in a few to see if I already have something on sed and awk. I like the way awk works. Use it with Nmap.

Thanks
 

gvisoc

Member
Credits
901
O-yea, just checked, I have UNIX Power Tools 2nd Addition sed & awk, The AWK Manual, Essential awk so these should get me going good.
I got some of those books too. I was going to recommend to have a look to humble bundle periodically, as sometimes they bundle (slightly old editions of) very good O’Reilly Unix books at ridiculously low prices.
 

None-yet

Member
Credits
576
I get digital books by the ftp full. I have an in with a few publishers so new or old they upload almost daily. I love reading and took a speed read class way back. I can read a 350 pg book in roughly two maybe three hours. It depends what else I have on my mind. I have four tb of books, novels, reference, white papers, manuals, text books-high schools, college, trade. Will not say I have everything but enough. So many I don't know what I have till I need something like the books I mentioned. I didn't know they were there. My mom said I was always weird with books. Reading newspapers when I was 5. My dad would quiz me on the news daily.

In case anyone needs or just enjoys reading and needs something I would be happy to provide if I am able.
 



Latest posts

Top