{"year":"2021","article_processing_charge":"No","month":"05","user_id":"2DF688A6-F248-11E8-B48F-1D18A9856A87","department":[{"_id":"GradSch"},{"_id":"ChLa"}],"day":"01","main_file_link":[{"open_access":"1","url":"https://openreview.net/pdf?id=krz7T0xU9Z_"}],"file":[{"file_size":502356,"relation":"main_file","file_id":"9417","file_name":"iclr2021_conference.pdf","creator":"bphuong","content_type":"application/pdf","date_updated":"2021-05-24T11:15:57Z","date_created":"2021-05-24T11:15:57Z","access_level":"open_access","checksum":"f34ff17017527db5ba6927f817bdd125"}],"related_material":{"record":[{"id":"9418","relation":"dissertation_contains","status":"public"}]},"quality_controlled":"1","title":"The inductive bias of ReLU networks on orthogonally separable data","citation":{"chicago":"Phuong, Mary, and Christoph Lampert. “The Inductive Bias of ReLU Networks on Orthogonally Separable Data.” In <i>9th International Conference on Learning Representations</i>, 2021.","ista":"Phuong M, Lampert C. 2021. The inductive bias of ReLU networks on orthogonally separable data. 9th International Conference on Learning Representations.  ICLR: International Conference on Learning Representations.","apa":"Phuong, M., &#38; Lampert, C. (2021). The inductive bias of ReLU networks on orthogonally separable data. In <i>9th International Conference on Learning Representations</i>. Virtual.","short":"M. Phuong, C. Lampert, in:, 9th International Conference on Learning Representations, 2021.","mla":"Phuong, Mary, and Christoph Lampert. “The Inductive Bias of ReLU Networks on Orthogonally Separable Data.” <i>9th International Conference on Learning Representations</i>, 2021.","ieee":"M. Phuong and C. Lampert, “The inductive bias of ReLU networks on orthogonally separable data,” in <i>9th International Conference on Learning Representations</i>, Virtual, 2021.","ama":"Phuong M, Lampert C. The inductive bias of ReLU networks on orthogonally separable data. In: <i>9th International Conference on Learning Representations</i>. ; 2021."},"date_published":"2021-05-01T00:00:00Z","type":"conference","has_accepted_license":"1","scopus_import":"1","_id":"9416","oa":1,"ddc":["000"],"publication":"9th International Conference on Learning Representations","language":[{"iso":"eng"}],"publication_status":"published","author":[{"id":"3EC6EE64-F248-11E8-B48F-1D18A9856A87","full_name":"Bui Thi Mai, Phuong","last_name":"Bui Thi Mai","first_name":"Phuong"},{"id":"40C20FD2-F248-11E8-B48F-1D18A9856A87","full_name":"Lampert, Christoph","last_name":"Lampert","orcid":"0000-0001-8622-7887","first_name":"Christoph"}],"conference":{"start_date":"2021-05-03","location":"Virtual","end_date":"2021-05-07","name":" ICLR: International Conference on Learning Representations"},"date_created":"2021-05-24T11:16:46Z","status":"public","date_updated":"2023-09-07T13:29:50Z","file_date_updated":"2021-05-24T11:15:57Z","abstract":[{"text":"We study the inductive bias of two-layer ReLU networks trained by gradient flow. We identify a class of easy-to-learn (`orthogonally separable') datasets, and characterise the solution that ReLU networks trained on such datasets converge to. Irrespective of network width, the solution turns out to be a combination of two max-margin classifiers: one corresponding to the positive data subset and one corresponding to the negative data subset. The proof is based on the recently introduced concept of extremal sectors, for which we prove a number of properties in the context of orthogonal separability. In particular, we prove stationarity of activation patterns from some time  onwards, which enables a reduction of the ReLU network to an ensemble of linear subnetworks.","lang":"eng"}],"oa_version":"Published Version"}