Need to Merge, remove Duplicate Domains or Duplicate URL’s from large text files?
ScrapeBox DupeRemove is a small, fast, lightweight and free tool that allows you to merge multiple text file URL lists in to one large file. Also it can remove duplicate URL’s and duplicate domains from files as large as 180 Million lines long in just a few seconds.
To merge files, click the “Select Source Files to Merge” button then highlight all the files you would like to merge. Then click the “Select Target Files” button and give your new merged file a name. Now simply click “Merge Files” and it’s done, all your files are merged in to one.
To remove duplicate URL’s or Domains from a text file, click the “Select Source File” button and select the text file you would like to remove duplicates from. Then click the “Select Target File” button, and give your new dupe free file a name. Now simply click “Remove Dup URL’s” or “Remove Dup Domains” and it’s done your duplicate free file will be saved.
The tool was designed to help manage ScrapeBox text files, however we are releasing it Free for anyone.
But other than that there’s no opt-in, so you will not have to divulge your email and put up with a barrage of spam emails, there’s no registration involved, the software never expires and it will always work and there’s no other catches what so ever. It’s also “portable” so you don’t have to install it and it doesn’t need .net framework, it will run on XP, Vista, Windows 7 and Windows Server 2003/2008.
Update 9th August 2011: Added ability to split large files.