This 3 times faster than the previous known record.
For those who don’t know, Jim Gray et al established a series of tests, including the 1TB sort, in order to give database vendors a playground for honest comparisons. The results are maintained online. Here are the two related papers:
- “A Measure of Transaction Processing”. Note how it’s signed as “anon et al”. If I remember well, Jim didn’t want for the paper to be associated with a particular vendor. I could be wrong.
- “A Measure of Transaction Processing 20 Years Later”.
Google managed to sort 1TB in 68secs using their MapReduce infrastructure on 1,000 machines. Then, they attempted to sort 1PB of data on 4,000 machines. It’s interesting how when sorting 1PB of data one hits the hard disk failure rates.
Interesting stuff. I am looking forward to the paper.
One response to “1TB sort done in 68 seconds by Google”
That is impressive; I can see an arms race developing here. What I’d like to see is an estimate of the number or megajoules used in the operation, possibly along with the number of MJ to upload the data and keep it stored for a month. Then we can measure performance not just by wall time, but energy efficiency.