---
OA_place: publisher
OA_type: free access
_id: '22830'
abstract:
- lang: eng
text: "We introduce AutoJudge, a method that accelerates large language model (LLM)
inference with task-specific lossy speculative decoding. Instead of matching the
original model output distribution token-by-token, we identify the generated tokens
that affect the downstream quality of the response, relaxing the distribution
match guarantee so that the \"unimportant\" tokens can be generated faster. Our
approach relies on a semi‑greedy search algorithm to test which of the mismatches
between target and draft models should be corrected to preserve quality and which
ones may be skipped. We then train a lightweight classifier based on existing
LLM embeddings to predict, at inference time, which mismatching tokens can be
safely accepted without compromising the final answer quality. We evaluate AutoJudge
with multiple draft/target model pairs on mathematical reasoning and programming
benchmarks, achieving significant speedups at the cost of a minor accuracy reduction.
Notably, on GSM8K with the Llama 3.1 70B target model, our approach achieves up
to \r\n≈\r\n2\r\n×\r\n speedup \\textit{over speculative decoding} at the cost
of a \r\n≤\r\n1\r\n%\r\n drop in accuracy. When applied to the LiveCodeBench benchmark,
AutoJudge automatically detects programming-specific important tokens, accepting
\r\n≥\r\n25\r\n tokens per speculation cycle at a\r\n \r\n2\r\n%\r\n drop in Pass@1.
Our approach requires no human annotation and is easy to integrate with modern
LLM inference frameworks."
acknowledgement: "We would like to express our sincere gratitude to Denis Mazur for
his valuable contributions to the\r\nimplementation of API calls used in Algorithm
1 and for supporting inference with the Llama 405B\r\nmodel. We are also thankful
for his positive influence on the overall atmosphere and team morale\r\nthroughout
the course of this project."
alternative_title:
- Advances in Neural Information Processing Systems
article_processing_charge: No
arxiv: 1
author:
- first_name: Roman
full_name: Garipov, Roman
last_name: Garipov
- first_name: Fedor
full_name: Velikonivtsev, Fedor
last_name: Velikonivtsev
- first_name: Ivan
full_name: Ermakov, Ivan
last_name: Ermakov
- first_name: Ruslan
full_name: Svirschevski, Ruslan
last_name: Svirschevski
- first_name: Vage
full_name: Egiazarian, Vage
id: 77451e76-92b2-11ef-a4d1-8dbaa06e16ad
last_name: Egiazarian
- first_name: Max
full_name: Ryabinin, Max
last_name: Ryabinin
citation:
ama: 'Garipov R, Velikonivtsev F, Ermakov I, Svirschevski R, Egiazarian V, Ryabinin
M. AutoJudge: Judge decoding without manual annotation. In: 39th Annual Conference
on Neural Information Processing Systems. Vol 38. Neural Information Processing
Systems Foundation; 2025:104904-104941. doi:10.52202/085713-3163'
apa: 'Garipov, R., Velikonivtsev, F., Ermakov, I., Svirschevski, R., Egiazarian,
V., & Ryabinin, M. (2025). AutoJudge: Judge decoding without manual annotation.
In 39th Annual Conference on Neural Information Processing Systems (Vol.
38, pp. 104904–104941). San Diego, CA, United States: Neural Information Processing
Systems Foundation. https://doi.org/10.52202/085713-3163'
chicago: 'Garipov, Roman, Fedor Velikonivtsev, Ivan Ermakov, Ruslan Svirschevski,
Vage Egiazarian, and Max Ryabinin. “AutoJudge: Judge Decoding without Manual Annotation.”
In 39th Annual Conference on Neural Information Processing Systems, 38:104904–41.
Neural Information Processing Systems Foundation, 2025. https://doi.org/10.52202/085713-3163.'
ieee: 'R. Garipov, F. Velikonivtsev, I. Ermakov, R. Svirschevski, V. Egiazarian,
and M. Ryabinin, “AutoJudge: Judge decoding without manual annotation,” in 39th
Annual Conference on Neural Information Processing Systems, San Diego, CA,
United States, 2025, vol. 38, pp. 104904–104941.'
ista: 'Garipov R, Velikonivtsev F, Ermakov I, Svirschevski R, Egiazarian V, Ryabinin
M. 2025. AutoJudge: Judge decoding without manual annotation. 39th Annual Conference
on Neural Information Processing Systems. NeurIPS: Neural Information Processing
Systems, Advances in Neural Information Processing Systems, vol. 38, 104904–104941.'
mla: 'Garipov, Roman, et al. “AutoJudge: Judge Decoding without Manual Annotation.”
39th Annual Conference on Neural Information Processing Systems, vol. 38,
Neural Information Processing Systems Foundation, 2025, pp. 104904–41, doi:10.52202/085713-3163.'
short: R. Garipov, F. Velikonivtsev, I. Ermakov, R. Svirschevski, V. Egiazarian,
M. Ryabinin, in:, 39th Annual Conference on Neural Information Processing Systems,
Neural Information Processing Systems Foundation, 2025, pp. 104904–104941.
conference:
end_date: 2025-12-07
location: San Diego, CA, United States
name: 'NeurIPS: Neural Information Processing Systems'
start_date: 2025-12-02
das_tickbox: '0'
date_created: 2026-09-06T22:02:00Z
date_published: 2025-12-02T00:00:00Z
date_updated: 2026-09-10T11:08:33Z
day: '02'
department:
- _id: DaAl
doi: 10.52202/085713-3163
external_id:
arxiv:
- '2504.20039'
fulldoi: https://doi.org/10.52202/085713-3163
intvolume: ' 38'
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://doi.org/10.52202/085713-3163
month: '12'
oa: 1
oa_version: Published Version
page: 104904-104941
publication: 39th Annual Conference on Neural Information Processing Systems
publication_identifier:
isbn:
- '9798331338275'
issn:
- 1049-5258
publication_status: published
publisher: Neural Information Processing Systems Foundation
quality_controlled: '1'
researchdata_availability: no
scopus_import: '1'
status: public
supplementarymaterial: yes
title: 'AutoJudge: Judge decoding without manual annotation'
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 38
year: '2025'
...