{"year":"2024","tmp":{"image":"/images/cc_by.png","short":"CC BY (4.0)","name":"Creative Commons Attribution 4.0 International Public License (CC-BY 4.0)","legal_code_url":"https://creativecommons.org/licenses/by/4.0/legalcode"},"article_number":"27","scopus_import":"1","abstract":[{"lang":"eng","text":"Large-scale software repositories are a source of insights for software engineering. They offer an unmatched window into the software development process at scale. Their sheer number and size holds the promise of broadly applicable results. At the same time, that very size presents practical challenges for scaling tools and algorithms to millions of projects. A reasonable approach is to limit studies to representative samples of the population of interest. Broadly applicable conclusions can then be obtained by generalizing to the entire population. The contribution of this paper is a standardized experimental design methodology for choosing the inputs of studies working with large-scale repositories. We advocate for a methodology that clearly lays out what the population of interest is, how to sample it, and that fosters reproducibility. Along the way, we discourage researchers from using extrinsic attributes of projects such as stars, that measure some unclear notion of popularity."}],"date_updated":"2024-10-07T11:21:15Z","_id":"18175","article_processing_charge":"No","title":"The fault in our stars: Designing reproducible large-scale code analysis experiments","publication_status":"published","status":"public","date_published":"2024-09-01T00:00:00Z","citation":{"ieee":"P. Maj, S. Muroya Lei, K. Siek, L. Di Grazia, and J. Vitek, “The fault in our stars: Designing reproducible large-scale code analysis experiments,” in <i>38th European Conference on Object-Oriented Programming</i>, Vienna, Austria, 2024, vol. 313.","ista":"Maj P, Muroya Lei S, Siek K, Di Grazia L, Vitek J. 2024. The fault in our stars: Designing reproducible large-scale code analysis experiments. 38th European Conference on Object-Oriented Programming. ECOOP: European Conference on Object-Oriented Programming, LIPIcs, vol. 313, 27.","mla":"Maj, Petr, et al. “The Fault in Our Stars: Designing Reproducible Large-Scale Code Analysis Experiments.” <i>38th European Conference on Object-Oriented Programming</i>, vol. 313, 27, Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2024, doi:<a href=\"https://doi.org/10.4230/LIPIcs.ECOOP.2024.27\">10.4230/LIPIcs.ECOOP.2024.27</a>.","apa":"Maj, P., Muroya Lei, S., Siek, K., Di Grazia, L., &#38; Vitek, J. (2024). The fault in our stars: Designing reproducible large-scale code analysis experiments. In <i>38th European Conference on Object-Oriented Programming</i> (Vol. 313). Vienna, Austria: Schloss Dagstuhl - Leibniz-Zentrum für Informatik. <a href=\"https://doi.org/10.4230/LIPIcs.ECOOP.2024.27\">https://doi.org/10.4230/LIPIcs.ECOOP.2024.27</a>","ama":"Maj P, Muroya Lei S, Siek K, Di Grazia L, Vitek J. The fault in our stars: Designing reproducible large-scale code analysis experiments. In: <i>38th European Conference on Object-Oriented Programming</i>. Vol 313. Schloss Dagstuhl - Leibniz-Zentrum für Informatik; 2024. doi:<a href=\"https://doi.org/10.4230/LIPIcs.ECOOP.2024.27\">10.4230/LIPIcs.ECOOP.2024.27</a>","chicago":"Maj, Petr, Stefanie Muroya Lei, Konrad Siek, Luca Di Grazia, and Jan Vitek. “The Fault in Our Stars: Designing Reproducible Large-Scale Code Analysis Experiments.” In <i>38th European Conference on Object-Oriented Programming</i>, Vol. 313. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2024. <a href=\"https://doi.org/10.4230/LIPIcs.ECOOP.2024.27\">https://doi.org/10.4230/LIPIcs.ECOOP.2024.27</a>.","short":"P. Maj, S. Muroya Lei, K. Siek, L. Di Grazia, J. Vitek, in:, 38th European Conference on Object-Oriented Programming, Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2024."},"month":"09","language":[{"iso":"eng"}],"author":[{"first_name":"Petr","last_name":"Maj","full_name":"Maj, Petr"},{"first_name":"Stefanie","last_name":"Muroya Lei","full_name":"Muroya Lei, Stefanie","id":"a376de31-8972-11ed-ae7b-d0251c13c8ff"},{"last_name":"Siek","first_name":"Konrad","full_name":"Siek, Konrad"},{"last_name":"Di Grazia","first_name":"Luca","full_name":"Di Grazia, Luca"},{"first_name":"Jan","last_name":"Vitek","full_name":"Vitek, Jan"}],"file":[{"date_updated":"2024-10-07T11:10:55Z","access_level":"open_access","relation":"main_file","success":1,"date_created":"2024-10-07T11:10:55Z","content_type":"application/pdf","file_name":"2024_LIPICs_Maj.pdf","file_size":1764222,"checksum":"2e75d305a8c817d76a0c7f136ce34f86","file_id":"18184","creator":"dernst"}],"publication_identifier":{"isbn":["9783959773416"],"issn":["1868-8969"]},"quality_controlled":"1","has_accepted_license":"1","oa":1,"conference":{"start_date":"2024-09-16","end_date":"2024-09-20","location":"Vienna, Austria","name":"ECOOP: European Conference on Object-Oriented Programming"},"volume":313,"type":"conference","ddc":["000"],"date_created":"2024-10-06T22:01:12Z","acknowledgement":"This work was supported by the Czech Ministry of Education, Youth and Sports under\r\nprogram ERC-CZ, grant agreement LL2325, BigCode (reg. no. CZ.02.1.01/0.0/0.0/15_003/0000421). NSF grants CCF-1910850, CNS-1925644, and CCF-2139612, as well as the GACR EXPRO grant 23-07580X. We would like to thank Digital Ocean for their involuntary contribution of computational resources during the early data gathering phase of our research. We acknoweldge the reviewers of ICSE’22, and thank the reviewers of ECOOP’23 for their encouragments and for sticking around until 2024.","alternative_title":["LIPIcs"],"intvolume":"       313","file_date_updated":"2024-10-07T11:10:55Z","user_id":"2DF688A6-F248-11E8-B48F-1D18A9856A87","oa_version":"Published Version","department":[{"_id":"ToHe"}],"doi":"10.4230/LIPIcs.ECOOP.2024.27","publisher":"Schloss Dagstuhl - Leibniz-Zentrum für Informatik","day":"01","publication":"38th European Conference on Object-Oriented Programming"}