---
OA_place: publisher
OA_type: gold
_id: '22826'
abstract:
- lang: eng
text: "Large Language Models (LLMs) have demonstrated the ability to tackle increasingly
complex tasks through advanced reasoning, long-form content generation,\r\nand
tool use. Solving these tasks often involves long inference-time computations.\r\nIn
human problem solving, a common strategy to expedite work is collaboration: by\r\ndividing
the problem into sub-tasks, exploring different strategies concurrently, etc.\r\nRecent
research has shown that LLMs can also operate in parallel by implementing\r\nexplicit
cooperation frameworks, such as voting mechanisms or the explicit creation of
independent sub-tasks that can be executed in parallel. However, each of\r\nthese
frameworks may not be suitable for all types of tasks, which can hinder their\r\napplicability.
In this work, we propose a different design approach: we run LLM\r\n“workers”
in parallel , allowing them to synchronize via a concurrently-updated\r\nattention
cache and prompt these workers to decide how best to collaborate. Our\r\napproach
allows the LLM instances to come up with their own collaboration strategy for
the problem at hand, all the while “seeing” each other’s memory in the\r\nconcurrent
KV cache. We implement this approach via Hogwild! Inference: a\r\nparallel LLM
inference engine where multiple instances of the same LLM run in\r\nparallel with
the same attention cache, with “instant” access to each other’s memory.1 Hogwild!
Inference takes advantage of Rotary Position Embeddings (RoPE)\r\nto avoid recomputation
while improving parallel hardware utilization. We find that\r\nmodern reasoning-capable
LLMs can perform inference with shared Key-Value\r\ncache out of the box, without
additional fine-tuning."
acknowledgement: "We thank Vladimir Malinovskii for his help with brainstorming, helpful\r\nfeedback
and suggesting future work directions. We also thank Philip Zmushko for proofreading."
alternative_title:
- Advances in Neural Information Processing Systems
article_processing_charge: No
author:
- first_name: Gleb
full_name: Rodionov, Gleb
last_name: Rodionov
- first_name: Roman
full_name: Garipov, Roman
last_name: Garipov
- first_name: Alina
full_name: Shutova, Alina
last_name: Shutova
- first_name: George
full_name: Yakushev, George
last_name: Yakushev
- first_name: Erik
full_name: Schultheis, Erik
id: 2786b299-e6b0-11f0-91da-9243fe3ef96b
last_name: Schultheis
- first_name: Vage
full_name: Egiazarian, Vage
id: 77451e76-92b2-11ef-a4d1-8dbaa06e16ad
last_name: Egiazarian
- first_name: Anton
full_name: Sinitsin, Anton
last_name: Sinitsin
- first_name: Denis
full_name: Kuznedelev, Denis
last_name: Kuznedelev
- first_name: Dan-Adrian
full_name: Alistarh, Dan-Adrian
id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
last_name: Alistarh
orcid: 0000-0003-3650-940X
citation:
ama: 'Rodionov G, Garipov R, Shutova A, et al. Hogwild! Inference: Parallel LLM
generation via concurrent attention. In: 39th Conference on Neural Information
Processing Systems. Vol 38. Neural Information Processing Systems Foundation;
2025:52014-52055. doi:10.52202/085713-1551'
apa: 'Rodionov, G., Garipov, R., Shutova, A., Yakushev, G., Schultheis, E., Egiazarian,
V., … Alistarh, D.-A. (2025). Hogwild! Inference: Parallel LLM generation via
concurrent attention. In 39th Conference on Neural Information Processing Systems
(Vol. 38, pp. 52014–52055). San Diego, CA, United States: Neural Information Processing
Systems Foundation. https://doi.org/10.52202/085713-1551'
chicago: 'Rodionov, Gleb, Roman Garipov, Alina Shutova, George Yakushev, Erik Schultheis,
Vage Egiazarian, Anton Sinitsin, Denis Kuznedelev, and Dan-Adrian Alistarh. “Hogwild!
Inference: Parallel LLM Generation via Concurrent Attention.” In 39th Conference
on Neural Information Processing Systems, 38:52014–55. Neural Information
Processing Systems Foundation, 2025. https://doi.org/10.52202/085713-1551.'
ieee: 'G. Rodionov et al., “Hogwild! Inference: Parallel LLM generation via
concurrent attention,” in 39th Conference on Neural Information Processing
Systems, San Diego, CA, United States, 2025, vol. 38, pp. 52014–52055.'
ista: 'Rodionov G, Garipov R, Shutova A, Yakushev G, Schultheis E, Egiazarian V,
Sinitsin A, Kuznedelev D, Alistarh D-A. 2025. Hogwild! Inference: Parallel LLM
generation via concurrent attention. 39th Conference on Neural Information Processing
Systems. NeurIPS: Neural Information Processing Systems, Advances in Neural Information
Processing Systems, vol. 38, 52014–52055.'
mla: 'Rodionov, Gleb, et al. “Hogwild! Inference: Parallel LLM Generation via Concurrent
Attention.” 39th Conference on Neural Information Processing Systems, vol.
38, Neural Information Processing Systems Foundation, 2025, pp. 52014–55, doi:10.52202/085713-1551.'
short: G. Rodionov, R. Garipov, A. Shutova, G. Yakushev, E. Schultheis, V. Egiazarian,
A. Sinitsin, D. Kuznedelev, D.-A. Alistarh, in:, 39th Conference on Neural Information
Processing Systems, Neural Information Processing Systems Foundation, 2025, pp.
52014–52055.
conference:
end_date: 2025-12-07
location: San Diego, CA, United States
name: 'NeurIPS: Neural Information Processing Systems'
start_date: 2025-12-02
das_tickbox: '0'
date_created: 2026-09-06T22:01:59Z
date_published: 2025-12-02T00:00:00Z
date_updated: 2026-09-10T07:06:31Z
day: '02'
ddc:
- '000'
department:
- _id: DaAl
doi: 10.52202/085713-1551
fulldoi: https://doi.org/10.52202/085713-1551
has_accepted_license: '1'
intvolume: ' 38'
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://doi.org/10.52202/085713-1551
month: '12'
oa: 1
oa_version: Published Version
page: 52014-52055
publication: 39th Conference on Neural Information Processing Systems
publication_identifier:
isbn:
- '9798331338275'
issn:
- 1049-5258
publication_status: published
publisher: Neural Information Processing Systems Foundation
quality_controlled: '1'
researchdata_availability: no
scopus_import: '1'
status: public
supplementarymaterial: yes
title: 'Hogwild! Inference: Parallel LLM generation via concurrent attention'
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 38
year: '2025'
...