MARLIN: Mixed-precision auto-regressive parallel inference on Large Language Models
Frantar E, Castro R, Chen J, Hoefler T, Alistarh D-A. 2024. MARLIN: Mixed-precision auto-regressive parallel inference on Large Language Models, Zenodo, 10.5281/ZENODO.14213091.
Download (ext.)
Research Data Reference
Creator
Corresponding author has ISTA affiliation
Department
Abstract
This is Marlin, a Mixed Auto-Regressive Linear kernel (and the name of one of the planet's fastest fish), an extremely optimized FP16xINT4 matmul kernel aimed at LLM inference that can deliver close to ideal (4x) speedups up to batchsizes of 16-32 tokens (in contrast to the 1-2 tokens of prior work with comparable speedup).
Additionally, it includes Sparse-Marlin, an extension of the MARLIN kernels adding support to 2:4 weight sparsity, achieving 5.3x speedups on NVIDIA GPUs (Ampere/Ada).
Publishing Year
Date Published
2024-11-24
Publisher
Zenodo
IST-REx-ID
Cite this
Frantar E, Castro R, Chen J, Hoefler T, Alistarh D-A. MARLIN: Mixed-precision auto-regressive parallel inference on Large Language Models. 2024. doi:10.5281/ZENODO.14213091
Frantar, E., Castro, R., Chen, J., Hoefler, T., & Alistarh, D.-A. (2024). MARLIN: Mixed-precision auto-regressive parallel inference on Large Language Models. Zenodo. https://doi.org/10.5281/ZENODO.14213091
Frantar, Elias, Roberto Castro, Jiale Chen, Torsten Hoefler, and Dan-Adrian Alistarh. “MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models.” Zenodo, 2024. https://doi.org/10.5281/ZENODO.14213091.
E. Frantar, R. Castro, J. Chen, T. Hoefler, and D.-A. Alistarh, “MARLIN: Mixed-precision auto-regressive parallel inference on Large Language Models.” Zenodo, 2024.
Frantar E, Castro R, Chen J, Hoefler T, Alistarh D-A. 2024. MARLIN: Mixed-precision auto-regressive parallel inference on Large Language Models, Zenodo, 10.5281/ZENODO.14213091.
Frantar, Elias, et al. MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models. Zenodo, 2024, doi:10.5281/ZENODO.14213091.
All files available under the following license(s):
Creative Commons Attribution 4.0 International Public License (CC-BY 4.0):
Link(s) to Main File(s)
Access Level
Open Access
Material in ISTA:
Used for analysis in
Google Scholar