Movies and music could be shared faster over the net thanks to a system pioneered by researchers in the US.
The speed of current services, like BitTorrent, is limited by the number of people sharing a specific file, such as a movie or song.
Similarity-Enhanced Transfer (SET) works by spotting chunks of identical data in files that are an exact or near match to the one needed.
Using SET the researchers have seen speed increases of up to 500%.
Looks familiar
The findings are outlined in a paper, Exploiting Similarity for Multi-Source Downloads Using File Handprints, written by David Andersen of Carnegie Mellon University, Himabindu Pucha, of Purdue University, and Michael Kaminsky of Intel Research.
Current file-sharing systems, like BitTorrent work best when there are multiple sources of a specific shared file.
When a file is shared it is divided into chunks and those parcels of data are distributed to groups of people who are searching for that file.
The more sources of those chunks there are, the more information there is that can be sent to a user, resulting in faster download speeds.
Identical pieces
But these services often fail to deliver fast speeds because there are not enough users sharing the chunks of the specific file wanted.
"A big limitation of BitTorrent is that it only lets clients share data if they're downloading the exact same file," said Professor Andersen. "This means that the available client pool for any particular file is smaller than it needs to be."
SET works faster because the trio realised that many files being shared on the net contain identical pieces of data even though they appear to be different.
Professor Andersen said he was "shocked" by this discovery.
"In retrospect, of course, the causes of the similarity (different audio tracks, different artist names, etc.) make a lot of sense, but I honestly didn't expect it at all," he told the BBC News website.
The SET system assigns files a similarity ranking and can take chunks of data from files which are both identical and similar to the one being searched for.
The lower similarity ranking that SET searches for, the more sources for that data are likely to be found.
"The extra overhead of locating these sources does not out-weigh the benefit of using them to help saturate the recipient's available bandwidth," wrote the scientists. "Indeed, exploiting similar sources can significantly improve download time," it added.
SET uses a technique called handprinting - which has been used to filter junk e-mails - to seek out files that contain some of the data needed by the one a file-sharing program has requested.
For example, a search for a particular Madonna song may result in a wide range of titles because the tags, or labels, have been filled in differently, or incorrectly.
SET knows that the music data of a number of differently-labelled Madonna songs is identical and that chunks of data from those files can be shared between users.
In tests, SET improved the transfer time of an MP3 music file by 71% and a 55Mb movie trailer went 30% faster using the researchers' techniques to draw from movie trailers that were 47% similar.
Prof Andersen said that SET could help most with less popular files.
"It gives clients a larger pool of other clients to draw from," he said "This probably won't matter much for super-popular data, where there is already a huge set of people downloading it, but our experiments suggest that in the other cases, SET can help a lot."
Retro-fitting BitTorrent to use SET could be quite straightforward, said Prof Andersen, though he added that some of its techniques would need much more work to make it very good at finding files that have some, but not all, of the data that a user wants.
He said he hoped developers would take the ideas and build on them.
"This is a technique that I would like people to steal," he said and added that he hopes to use SET in a service for sharing software or academic papers.
"It would make P2P transfers faster and more efficient," he added, "and developers should just take the idea and use it in their own systems."
Millions of files are being shared via peer to peer sites
