Dropbox Interview Question

How to find duplicate files in storage

Interview Answer

Anonymous

Dec 8, 2016

Create a SortedDictionary> and fill it with all of the length of the file, and file paths. Walk the list to find files with the same length. For files of the same length - which could be N in number which can make binary file to file comparison slow, read each file and compute MD5. Assume you have M with same MD5. Take first and do binary compare with the remaining M-1. Find the first bytes that are different in them (if none, then you have a n-tuple). If they are different then go to the next file and check the byte at where the first file are different. Deposit the next file into one of the following baskets (Match-1, Match-2, Match-3). For speed we do not walk the entire file but use the fact that we have the same MD5 to quickly and cheaply drop each file into a basket. Once all files have been processed, repeat for each file in each basket... It is really unlikely that given the same file size and the same MD5 hash (odds are around 2^128), that the files will be different.... but the above provide that for that rare possibility. Compute time: P files size read, N files have MD5 computed (only those with the same length), M (<=N) files are binary compared.

3