---
OA_place: publisher
OA_type: gold
_id: '22826'
abstract:
- lang: eng
text: "Large Language Models (LLMs) have demonstrated the ability to tackle increasingly
complex tasks through advanced reasoning, long-form content generation,\r\nand
tool use. Solving these tasks often involves long inference-time computations.\r\nIn
human problem solving, a common strategy to expedite work is collaboration: by\r\ndividing
the problem into sub-tasks, exploring different strategies concurrently, etc.\r\nRecent
research has shown that LLMs can also operate in parallel by implementing\r\nexplicit
cooperation frameworks, such as voting mechanisms or the explicit creation of
independent sub-tasks that can be executed in parallel. However, each of\r\nthese
frameworks may not be suitable for all types of tasks, which can hinder their\r\napplicability.
In this work, we propose a different design approach: we run LLM\r\n“workers”
in parallel , allowing them to synchronize via a concurrently-updated\r\nattention
cache and prompt these workers to decide how best to collaborate. Our\r\napproach
allows the LLM instances to come up with their own collaboration strategy for
the problem at hand, all the while “seeing” each other’s memory in the\r\nconcurrent
KV cache. We implement this approach via Hogwild! Inference: a\r\nparallel LLM
inference engine where multiple instances of the same LLM run in\r\nparallel with
the same attention cache, with “instant” access to each other’s memory.1 Hogwild!
Inference takes advantage of Rotary Position Embeddings (RoPE)\r\nto avoid recomputation
while improving parallel hardware utilization. We find that\r\nmodern reasoning-capable
LLMs can perform inference with shared Key-Value\r\ncache out of the box, without
additional fine-tuning."
acknowledgement: "We thank Vladimir Malinovskii for his help with brainstorming, helpful\r\nfeedback
and suggesting future work directions. We also thank Philip Zmushko for proofreading."
alternative_title:
- Advances in Neural Information Processing Systems
article_processing_charge: No
author:
- first_name: Gleb
full_name: Rodionov, Gleb
last_name: Rodionov
- first_name: Roman
full_name: Garipov, Roman
last_name: Garipov
- first_name: Alina
full_name: Shutova, Alina
last_name: Shutova
- first_name: George
full_name: Yakushev, George
last_name: Yakushev
- first_name: Erik
full_name: Schultheis, Erik
id: 2786b299-e6b0-11f0-91da-9243fe3ef96b
last_name: Schultheis
- first_name: Vage
full_name: Egiazarian, Vage
id: 77451e76-92b2-11ef-a4d1-8dbaa06e16ad
last_name: Egiazarian
- first_name: Anton
full_name: Sinitsin, Anton
last_name: Sinitsin
- first_name: Denis
full_name: Kuznedelev, Denis
last_name: Kuznedelev
- first_name: Dan-Adrian
full_name: Alistarh, Dan-Adrian
id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
last_name: Alistarh
orcid: 0000-0003-3650-940X
citation:
ama: 'Rodionov G, Garipov R, Shutova A, et al. Hogwild! Inference: Parallel LLM
generation via concurrent attention. In: 39th Conference on Neural Information
Processing Systems. Vol 38. Neural Information Processing Systems Foundation;
2025:52014-52055. doi:10.52202/085713-1551'
apa: 'Rodionov, G., Garipov, R., Shutova, A., Yakushev, G., Schultheis, E., Egiazarian,
V., … Alistarh, D.-A. (2025). Hogwild! Inference: Parallel LLM generation via
concurrent attention. In 39th Conference on Neural Information Processing Systems
(Vol. 38, pp. 52014–52055). San Diego, CA, United States: Neural Information Processing
Systems Foundation. https://doi.org/10.52202/085713-1551'
chicago: 'Rodionov, Gleb, Roman Garipov, Alina Shutova, George Yakushev, Erik Schultheis,
Vage Egiazarian, Anton Sinitsin, Denis Kuznedelev, and Dan-Adrian Alistarh. “Hogwild!
Inference: Parallel LLM Generation via Concurrent Attention.” In 39th Conference
on Neural Information Processing Systems, 38:52014–55. Neural Information
Processing Systems Foundation, 2025. https://doi.org/10.52202/085713-1551.'
ieee: 'G. Rodionov et al., “Hogwild! Inference: Parallel LLM generation via
concurrent attention,” in 39th Conference on Neural Information Processing
Systems, San Diego, CA, United States, 2025, vol. 38, pp. 52014–52055.'
ista: 'Rodionov G, Garipov R, Shutova A, Yakushev G, Schultheis E, Egiazarian V,
Sinitsin A, Kuznedelev D, Alistarh D-A. 2025. Hogwild! Inference: Parallel LLM
generation via concurrent attention. 39th Conference on Neural Information Processing
Systems. NeurIPS: Neural Information Processing Systems, Advances in Neural Information
Processing Systems, vol. 38, 52014–52055.'
mla: 'Rodionov, Gleb, et al. “Hogwild! Inference: Parallel LLM Generation via Concurrent
Attention.” 39th Conference on Neural Information Processing Systems, vol.
38, Neural Information Processing Systems Foundation, 2025, pp. 52014–55, doi:10.52202/085713-1551.'
short: G. Rodionov, R. Garipov, A. Shutova, G. Yakushev, E. Schultheis, V. Egiazarian,
A. Sinitsin, D. Kuznedelev, D.-A. Alistarh, in:, 39th Conference on Neural Information
Processing Systems, Neural Information Processing Systems Foundation, 2025, pp.
52014–52055.
conference:
end_date: 2025-12-07
location: San Diego, CA, United States
name: 'NeurIPS: Neural Information Processing Systems'
start_date: 2025-12-02
das_tickbox: '0'
date_created: 2026-09-06T22:01:59Z
date_published: 2025-12-02T00:00:00Z
date_updated: 2026-09-10T07:06:31Z
day: '02'
ddc:
- '000'
department:
- _id: DaAl
doi: 10.52202/085713-1551
fulldoi: https://doi.org/10.52202/085713-1551
has_accepted_license: '1'
intvolume: ' 38'
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://doi.org/10.52202/085713-1551
month: '12'
oa: 1
oa_version: Published Version
page: 52014-52055
publication: 39th Conference on Neural Information Processing Systems
publication_identifier:
isbn:
- '9798331338275'
issn:
- 1049-5258
publication_status: published
publisher: Neural Information Processing Systems Foundation
quality_controlled: '1'
researchdata_availability: no
scopus_import: '1'
status: public
supplementarymaterial: yes
title: 'Hogwild! Inference: Parallel LLM generation via concurrent attention'
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 38
year: '2025'
...
---
OA_place: publisher
OA_type: free access
_id: '22830'
abstract:
- lang: eng
text: "We introduce AutoJudge, a method that accelerates large language model (LLM)
inference with task-specific lossy speculative decoding. Instead of matching the
original model output distribution token-by-token, we identify the generated tokens
that affect the downstream quality of the response, relaxing the distribution
match guarantee so that the \"unimportant\" tokens can be generated faster. Our
approach relies on a semi‑greedy search algorithm to test which of the mismatches
between target and draft models should be corrected to preserve quality and which
ones may be skipped. We then train a lightweight classifier based on existing
LLM embeddings to predict, at inference time, which mismatching tokens can be
safely accepted without compromising the final answer quality. We evaluate AutoJudge
with multiple draft/target model pairs on mathematical reasoning and programming
benchmarks, achieving significant speedups at the cost of a minor accuracy reduction.
Notably, on GSM8K with the Llama 3.1 70B target model, our approach achieves up
to \r\n≈\r\n2\r\n×\r\n speedup \\textit{over speculative decoding} at the cost
of a \r\n≤\r\n1\r\n%\r\n drop in accuracy. When applied to the LiveCodeBench benchmark,
AutoJudge automatically detects programming-specific important tokens, accepting
\r\n≥\r\n25\r\n tokens per speculation cycle at a\r\n \r\n2\r\n%\r\n drop in Pass@1.
Our approach requires no human annotation and is easy to integrate with modern
LLM inference frameworks."
acknowledgement: "We would like to express our sincere gratitude to Denis Mazur for
his valuable contributions to the\r\nimplementation of API calls used in Algorithm
1 and for supporting inference with the Llama 405B\r\nmodel. We are also thankful
for his positive influence on the overall atmosphere and team morale\r\nthroughout
the course of this project."
alternative_title:
- Advances in Neural Information Processing Systems
article_processing_charge: No
arxiv: 1
author:
- first_name: Roman
full_name: Garipov, Roman
last_name: Garipov
- first_name: Fedor
full_name: Velikonivtsev, Fedor
last_name: Velikonivtsev
- first_name: Ivan
full_name: Ermakov, Ivan
last_name: Ermakov
- first_name: Ruslan
full_name: Svirschevski, Ruslan
last_name: Svirschevski
- first_name: Vage
full_name: Egiazarian, Vage
id: 77451e76-92b2-11ef-a4d1-8dbaa06e16ad
last_name: Egiazarian
- first_name: Max
full_name: Ryabinin, Max
last_name: Ryabinin
citation:
ama: 'Garipov R, Velikonivtsev F, Ermakov I, Svirschevski R, Egiazarian V, Ryabinin
M. AutoJudge: Judge decoding without manual annotation. In: 39th Annual Conference
on Neural Information Processing Systems. Vol 38. Neural Information Processing
Systems Foundation; 2025:104904-104941. doi:10.52202/085713-3163'
apa: 'Garipov, R., Velikonivtsev, F., Ermakov, I., Svirschevski, R., Egiazarian,
V., & Ryabinin, M. (2025). AutoJudge: Judge decoding without manual annotation.
In 39th Annual Conference on Neural Information Processing Systems (Vol.
38, pp. 104904–104941). San Diego, CA, United States: Neural Information Processing
Systems Foundation. https://doi.org/10.52202/085713-3163'
chicago: 'Garipov, Roman, Fedor Velikonivtsev, Ivan Ermakov, Ruslan Svirschevski,
Vage Egiazarian, and Max Ryabinin. “AutoJudge: Judge Decoding without Manual Annotation.”
In 39th Annual Conference on Neural Information Processing Systems, 38:104904–41.
Neural Information Processing Systems Foundation, 2025. https://doi.org/10.52202/085713-3163.'
ieee: 'R. Garipov, F. Velikonivtsev, I. Ermakov, R. Svirschevski, V. Egiazarian,
and M. Ryabinin, “AutoJudge: Judge decoding without manual annotation,” in 39th
Annual Conference on Neural Information Processing Systems, San Diego, CA,
United States, 2025, vol. 38, pp. 104904–104941.'
ista: 'Garipov R, Velikonivtsev F, Ermakov I, Svirschevski R, Egiazarian V, Ryabinin
M. 2025. AutoJudge: Judge decoding without manual annotation. 39th Annual Conference
on Neural Information Processing Systems. NeurIPS: Neural Information Processing
Systems, Advances in Neural Information Processing Systems, vol. 38, 104904–104941.'
mla: 'Garipov, Roman, et al. “AutoJudge: Judge Decoding without Manual Annotation.”
39th Annual Conference on Neural Information Processing Systems, vol. 38,
Neural Information Processing Systems Foundation, 2025, pp. 104904–41, doi:10.52202/085713-3163.'
short: R. Garipov, F. Velikonivtsev, I. Ermakov, R. Svirschevski, V. Egiazarian,
M. Ryabinin, in:, 39th Annual Conference on Neural Information Processing Systems,
Neural Information Processing Systems Foundation, 2025, pp. 104904–104941.
conference:
end_date: 2025-12-07
location: San Diego, CA, United States
name: 'NeurIPS: Neural Information Processing Systems'
start_date: 2025-12-02
das_tickbox: '0'
date_created: 2026-09-06T22:02:00Z
date_published: 2025-12-02T00:00:00Z
date_updated: 2026-09-10T11:08:33Z
day: '02'
department:
- _id: DaAl
doi: 10.52202/085713-3163
external_id:
arxiv:
- '2504.20039'
fulldoi: https://doi.org/10.52202/085713-3163
intvolume: ' 38'
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://doi.org/10.52202/085713-3163
month: '12'
oa: 1
oa_version: Published Version
page: 104904-104941
publication: 39th Annual Conference on Neural Information Processing Systems
publication_identifier:
isbn:
- '9798331338275'
issn:
- 1049-5258
publication_status: published
publisher: Neural Information Processing Systems Foundation
quality_controlled: '1'
researchdata_availability: no
scopus_import: '1'
status: public
supplementarymaterial: yes
title: 'AutoJudge: Judge decoding without manual annotation'
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 38
year: '2025'
...