{"date_published":"2006-08-01T00:00:00Z","article_processing_charge":"No","quality_controlled":"1","day":"01","year":"2006","status":"public","abstract":[{"lang":"eng","text":"Broder et al.'s [3] shingling algorithm and Charikar's [4] random projection based approach are considered \"state-of-the-art\" algorithms for finding near-duplicate web pages. Both algorithms were either developed at or used by popular web search engines. We compare the two algorithms on a very large scale, namely on a set of 1.6B distinct web pages. The results show that neither of the algorithms works well for finding near-duplicate pairs on the same site, while both achieve high precision for near-duplicate pairs on different sites. Since Charikar's algorithm finds more near-duplicate pairs on different sites, it achieves a better precision overall, namely 0.50 versus 0.38 for Broder et al.'s algorithm. We present a combined algorithm which achieves precision 0.79 with 79% of the recall of the other algorithms."}],"date_created":"2022-08-19T07:20:31Z","date_updated":"2023-02-17T13:49:03Z","publication":"29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval","page":"284-291","user_id":"2DF688A6-F248-11E8-B48F-1D18A9856A87","extern":"1","author":[{"orcid":"0000-0002-5008-6530","first_name":"Monika H","full_name":"Henzinger, Monika H","last_name":"Henzinger","id":"540c9bbd-f2de-11ec-812d-d04a5be85630"}],"language":[{"iso":"eng"}],"publication_status":"published","conference":{"name":"SIGIR: International Conference on Research and Development in Information Retrieval","end_date":"2006-08-11","location":"Seattle, WA, United States","start_date":"2006-08-06"},"scopus_import":"1","publisher":"Association for Computing Machinery","citation":{"short":"M.H. Henzinger, in:, 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Association for Computing Machinery, 2006, pp. 284–291.","apa":"Henzinger, M. H. (2006). Finding near-duplicate web pages: A large-scale evaluation of algorithms. In <i>29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval</i> (pp. 284–291). Seattle, WA, United States: Association for Computing Machinery. <a href=\"https://doi.org/10.1145/1148170.1148222\">https://doi.org/10.1145/1148170.1148222</a>","ama":"Henzinger MH. Finding near-duplicate web pages: A large-scale evaluation of algorithms. In: <i>29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval</i>. Association for Computing Machinery; 2006:284-291. doi:<a href=\"https://doi.org/10.1145/1148170.1148222\">10.1145/1148170.1148222</a>","ista":"Henzinger MH. 2006. Finding near-duplicate web pages: A large-scale evaluation of algorithms. 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR: International Conference on Research and Development in Information Retrieval, 284–291.","ieee":"M. H. Henzinger, “Finding near-duplicate web pages: A large-scale evaluation of algorithms,” in <i>29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval</i>, Seattle, WA, United States, 2006, pp. 284–291.","chicago":"Henzinger, Monika H. “Finding Near-Duplicate Web Pages: A Large-Scale Evaluation of Algorithms.” In <i>29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval</i>, 284–91. Association for Computing Machinery, 2006. <a href=\"https://doi.org/10.1145/1148170.1148222\">https://doi.org/10.1145/1148170.1148222</a>.","mla":"Henzinger, Monika H. “Finding Near-Duplicate Web Pages: A Large-Scale Evaluation of Algorithms.” <i>29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval</i>, Association for Computing Machinery, 2006, pp. 284–91, doi:<a href=\"https://doi.org/10.1145/1148170.1148222\">10.1145/1148170.1148222</a>."},"type":"conference","oa_version":"None","month":"08","title":"Finding near-duplicate web pages: A large-scale evaluation of algorithms","doi":"10.1145/1148170.1148222","_id":"11929"}