---
_id: '7122'
abstract:
- lang: eng
text: Data-rich applications in machine-learning and control have motivated an intense
research on large-scale optimization. Novel algorithms have been proposed and
shown to have optimal convergence rates in terms of iteration counts. However,
their practical performance is severely degraded by the cost of exchanging high-dimensional
gradient vectors between computing nodes. Several gradient compression heuristics
have recently been proposed to reduce communications, but few theoretical results
exist that quantify how they impact algorithm convergence. This paper establishes
and strengthens the convergence guarantees for gradient descent under a family
of gradient compression techniques. For convex optimization problems, we derive
admissible step sizes and quantify both the number of iterations and the number
of bits that need to be exchanged to reach a target accuracy. Finally, we validate
the performance of different gradient compression techniques in simulations. The
numerical results highlight the properties of different gradient compression algorithms
and confirm that fast convergence with limited information exchange is possible.
article_number: '8619625'
article_processing_charge: No
author:
- first_name: Sarit
full_name: Khirirat, Sarit
last_name: Khirirat
- first_name: Mikael
full_name: Johansson, Mikael
last_name: Johansson
- first_name: Dan-Adrian
full_name: Alistarh, Dan-Adrian
id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
last_name: Alistarh
orcid: 0000-0003-3650-940X
citation:
ama: 'Khirirat S, Johansson M, Alistarh D-A. Gradient compression for communication-limited
convex optimization. In: 2018 IEEE Conference on Decision and Control.
IEEE; 2019. doi:10.1109/cdc.2018.8619625'
apa: 'Khirirat, S., Johansson, M., & Alistarh, D.-A. (2019). Gradient compression
for communication-limited convex optimization. In 2018 IEEE Conference on Decision
and Control. Miami Beach, FL, United States: IEEE. https://doi.org/10.1109/cdc.2018.8619625'
chicago: Khirirat, Sarit, Mikael Johansson, and Dan-Adrian Alistarh. “Gradient Compression
for Communication-Limited Convex Optimization.” In 2018 IEEE Conference on
Decision and Control. IEEE, 2019. https://doi.org/10.1109/cdc.2018.8619625.
ieee: S. Khirirat, M. Johansson, and D.-A. Alistarh, “Gradient compression for communication-limited
convex optimization,” in 2018 IEEE Conference on Decision and Control,
Miami Beach, FL, United States, 2019.
ista: 'Khirirat S, Johansson M, Alistarh D-A. 2019. Gradient compression for communication-limited
convex optimization. 2018 IEEE Conference on Decision and Control. CDC: Conference
on Decision and Control, 8619625.'
mla: Khirirat, Sarit, et al. “Gradient Compression for Communication-Limited Convex
Optimization.” 2018 IEEE Conference on Decision and Control, 8619625, IEEE,
2019, doi:10.1109/cdc.2018.8619625.
short: S. Khirirat, M. Johansson, D.-A. Alistarh, in:, 2018 IEEE Conference on Decision
and Control, IEEE, 2019.
conference:
end_date: 2018-12-19
location: Miami Beach, FL, United States
name: 'CDC: Conference on Decision and Control'
start_date: 2018-12-17
date_created: 2019-11-26T15:07:49Z
date_published: 2019-01-21T00:00:00Z
date_updated: 2023-09-06T11:14:55Z
day: '21'
department:
- _id: DaAl
doi: 10.1109/cdc.2018.8619625
external_id:
isi:
- '000458114800023'
isi: 1
language:
- iso: eng
month: '01'
oa_version: None
publication: 2018 IEEE Conference on Decision and Control
publication_identifier:
isbn:
- '9781538613955'
issn:
- 0743-1546
publication_status: published
publisher: IEEE
quality_controlled: '1'
scopus_import: '1'
status: public
title: Gradient compression for communication-limited convex optimization
type: conference
user_id: c635000d-4b10-11ee-a964-aac5a93f6ac1
year: '2019'
...
---
_id: '7201'
abstract:
- lang: eng
text: Applying machine learning techniques to the quickly growing data in science
and industry requires highly-scalable algorithms. Large datasets are most commonly
processed "data parallel" distributed across many nodes. Each node's contribution
to the overall gradient is summed using a global allreduce. This allreduce is
the single communication and thus scalability bottleneck for most machine learning
workloads. We observe that frequently, many gradient values are (close to) zero,
leading to sparse of sparsifyable communications. To exploit this insight, we
analyze, design, and implement a set of communication-efficient protocols for
sparse input data, in conjunction with efficient machine learning algorithms which
can leverage these primitives. Our communication protocols generalize standard
collective operations, by allowing processes to contribute arbitrary sparse input
data vectors. Our generic communication library, SparCML1, extends MPI to support
additional features, such as non-blocking (asynchronous) operations and low-precision
data representations. As such, SparCML and its techniques will form the basis
of future highly-scalable machine learning frameworks.
article_number: a11
article_processing_charge: No
arxiv: 1
author:
- first_name: Cedric
full_name: Renggli, Cedric
last_name: Renggli
- first_name: Saleh
full_name: Ashkboos, Saleh
id: 0D0A9058-257B-11EA-A937-9341C3D8BC8A
last_name: Ashkboos
- first_name: Mehdi
full_name: Aghagolzadeh, Mehdi
last_name: Aghagolzadeh
- first_name: Dan-Adrian
full_name: Alistarh, Dan-Adrian
id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
last_name: Alistarh
orcid: 0000-0003-3650-940X
- first_name: Torsten
full_name: Hoefler, Torsten
last_name: Hoefler
citation:
ama: 'Renggli C, Ashkboos S, Aghagolzadeh M, Alistarh D-A, Hoefler T. SparCML: High-performance
sparse communication for machine learning. In: International Conference for
High Performance Computing, Networking, Storage and Analysis, SC. ACM; 2019.
doi:10.1145/3295500.3356222'
apa: 'Renggli, C., Ashkboos, S., Aghagolzadeh, M., Alistarh, D.-A., & Hoefler,
T. (2019). SparCML: High-performance sparse communication for machine learning.
In International Conference for High Performance Computing, Networking, Storage
and Analysis, SC. Denver, CO, Unites States: ACM. https://doi.org/10.1145/3295500.3356222'
chicago: 'Renggli, Cedric, Saleh Ashkboos, Mehdi Aghagolzadeh, Dan-Adrian Alistarh,
and Torsten Hoefler. “SparCML: High-Performance Sparse Communication for Machine
Learning.” In International Conference for High Performance Computing, Networking,
Storage and Analysis, SC. ACM, 2019. https://doi.org/10.1145/3295500.3356222.'
ieee: 'C. Renggli, S. Ashkboos, M. Aghagolzadeh, D.-A. Alistarh, and T. Hoefler,
“SparCML: High-performance sparse communication for machine learning,” in International
Conference for High Performance Computing, Networking, Storage and Analysis, SC,
Denver, CO, Unites States, 2019.'
ista: 'Renggli C, Ashkboos S, Aghagolzadeh M, Alistarh D-A, Hoefler T. 2019. SparCML:
High-performance sparse communication for machine learning. International Conference
for High Performance Computing, Networking, Storage and Analysis, SC. SC: Conference
for High Performance Computing, Networking, Storage and Analysis, a11.'
mla: 'Renggli, Cedric, et al. “SparCML: High-Performance Sparse Communication for
Machine Learning.” International Conference for High Performance Computing,
Networking, Storage and Analysis, SC, a11, ACM, 2019, doi:10.1145/3295500.3356222.'
short: C. Renggli, S. Ashkboos, M. Aghagolzadeh, D.-A. Alistarh, T. Hoefler, in:,
International Conference for High Performance Computing, Networking, Storage and
Analysis, SC, ACM, 2019.
conference:
end_date: 2019-11-19
location: Denver, CO, Unites States
name: 'SC: Conference for High Performance Computing, Networking, Storage and Analysis'
start_date: 2019-11-17
date_created: 2019-12-22T23:00:42Z
date_published: 2019-11-17T00:00:00Z
date_updated: 2025-07-10T11:54:21Z
day: '17'
department:
- _id: DaAl
doi: 10.1145/3295500.3356222
ec_funded: 1
external_id:
arxiv:
- '1802.08021'
isi:
- '000545976800011'
isi: 1
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://arxiv.org/abs/1802.08021
month: '11'
oa: 1
oa_version: Preprint
project:
- _id: 268A44D6-B435-11E9-9278-68D0E5697425
call_identifier: H2020
grant_number: '805223'
name: Elastic Coordination for Scalable Machine Learning
publication: International Conference for High Performance Computing, Networking,
Storage and Analysis, SC
publication_identifier:
eissn:
- 2167-4337
isbn:
- '9781450362290'
issn:
- 2167-4329
publication_status: published
publisher: ACM
quality_controlled: '1'
scopus_import: '1'
status: public
title: 'SparCML: High-performance sparse communication for machine learning'
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
year: '2019'
...
---
_id: '7228'
abstract:
- lang: eng
text: "Traditional concurrent programming involves manipulating shared mutable state.
Alternatives to this programming style are communicating sequential processes
(CSP) and actor models, which share data via explicit communication. These models
have been known for almost half a century, and have recently had started to gain
significant traction among modern programming languages. The common abstraction
for communication between several processes is the channel. Although channels
are similar to producer-consumer data structures, they have different semantics
and support additional operations, such as the select expression. Despite their
growing popularity, most known implementations of channels use lock-based data
structures and can be rather inefficient.\r\n\r\nIn this paper, we present the
first efficient lock-free algorithm for implementing a communication channel for
CSP programming. We provide implementations and experimental results in the Kotlin
and Go programming languages. Our new algorithm outperforms existing implementations
on many workloads, while providing non-blocking progress guarantee. Our design
can serve as an example of how to construct general communication data structures
for CSP and actor models. "
alternative_title:
- LNCS
article_processing_charge: No
author:
- first_name: Nikita
full_name: Koval, Nikita
id: 2F4DB10C-F248-11E8-B48F-1D18A9856A87
last_name: Koval
- first_name: Dan-Adrian
full_name: Alistarh, Dan-Adrian
id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
last_name: Alistarh
orcid: 0000-0003-3650-940X
- first_name: Roman
full_name: Elizarov, Roman
last_name: Elizarov
citation:
ama: 'Koval N, Alistarh D-A, Elizarov R. Scalable FIFO channels for programming
via communicating sequential processes. In: 25th Anniversary of Euro-Par.
Vol 11725. Springer Nature; 2019:317-333. doi:10.1007/978-3-030-29400-7_23'
apa: 'Koval, N., Alistarh, D.-A., & Elizarov, R. (2019). Scalable FIFO channels
for programming via communicating sequential processes. In 25th Anniversary
of Euro-Par (Vol. 11725, pp. 317–333). Göttingen, Germany: Springer Nature.
https://doi.org/10.1007/978-3-030-29400-7_23'
chicago: Koval, Nikita, Dan-Adrian Alistarh, and Roman Elizarov. “Scalable FIFO
Channels for Programming via Communicating Sequential Processes.” In 25th Anniversary
of Euro-Par, 11725:317–33. Springer Nature, 2019. https://doi.org/10.1007/978-3-030-29400-7_23.
ieee: N. Koval, D.-A. Alistarh, and R. Elizarov, “Scalable FIFO channels for programming
via communicating sequential processes,” in 25th Anniversary of Euro-Par,
Göttingen, Germany, 2019, vol. 11725, pp. 317–333.
ista: 'Koval N, Alistarh D-A, Elizarov R. 2019. Scalable FIFO channels for programming
via communicating sequential processes. 25th Anniversary of Euro-Par. Euro-Par:
European Conference on Parallel Processing, LNCS, vol. 11725, 317–333.'
mla: Koval, Nikita, et al. “Scalable FIFO Channels for Programming via Communicating
Sequential Processes.” 25th Anniversary of Euro-Par, vol. 11725, Springer
Nature, 2019, pp. 317–33, doi:10.1007/978-3-030-29400-7_23.
short: N. Koval, D.-A. Alistarh, R. Elizarov, in:, 25th Anniversary of Euro-Par,
Springer Nature, 2019, pp. 317–333.
conference:
end_date: 2019-08-30
location: Göttingen, Germany
name: 'Euro-Par: European Conference on Parallel Processing'
start_date: 2019-08-26
date_created: 2020-01-05T23:00:46Z
date_published: 2019-08-13T00:00:00Z
date_updated: 2023-09-06T14:53:59Z
day: '13'
department:
- _id: DaAl
doi: 10.1007/978-3-030-29400-7_23
external_id:
isi:
- '000851061400023'
intvolume: ' 11725'
isi: 1
language:
- iso: eng
month: '08'
oa_version: None
page: 317-333
publication: 25th Anniversary of Euro-Par
publication_identifier:
eissn:
- 1611-3349
isbn:
- 978-3-0302-9399-4
issn:
- 0302-9743
publication_status: published
publisher: Springer Nature
quality_controlled: '1'
scopus_import: '1'
status: public
title: Scalable FIFO channels for programming via communicating sequential processes
type: conference
user_id: c635000d-4b10-11ee-a964-aac5a93f6ac1
volume: 11725
year: '2019'
...
---
_id: '7437'
abstract:
- lang: eng
text: 'Most of today''s distributed machine learning systems assume reliable networks:
whenever two machines exchange information (e.g., gradients or models), the network
should guarantee the delivery of the message. At the same time, recent work exhibits
the impressive tolerance of machine learning algorithms to errors or noise arising
from relaxed communication or synchronization. In this paper, we connect these
two trends, and consider the following question: Can we design machine learning
systems that are tolerant to network unreliability during training? With this
motivation, we focus on a theoretical problem of independent interest-given a
standard distributed parameter server architecture, if every communication between
the worker and the server has a non-zero probability p of being dropped, does
there exist an algorithm that still converges, and at what speed? The technical
contribution of this paper is a novel theoretical analysis proving that distributed
learning over unreliable network can achieve comparable convergence rate to centralized
or distributed learning over reliable networks. Further, we prove that the influence
of the packet drop rate diminishes with the growth of the number of parameter
servers. We map this theoretical result onto a real-world scenario, training deep
neural networks over an unreliable network layer, and conduct network simulation
to validate the system improvement by allowing the networks to be unreliable.'
article_processing_charge: No
arxiv: 1
author:
- first_name: Chen
full_name: Yu, Chen
last_name: Yu
- first_name: Hanlin
full_name: Tang, Hanlin
last_name: Tang
- first_name: Cedric
full_name: Renggli, Cedric
last_name: Renggli
- first_name: Simon
full_name: Kassing, Simon
last_name: Kassing
- first_name: Ankit
full_name: Singla, Ankit
last_name: Singla
- first_name: Dan-Adrian
full_name: Alistarh, Dan-Adrian
id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
last_name: Alistarh
orcid: 0000-0003-3650-940X
- first_name: Ce
full_name: Zhang, Ce
last_name: Zhang
- first_name: Ji
full_name: Liu, Ji
last_name: Liu
citation:
ama: 'Yu C, Tang H, Renggli C, et al. Distributed learning over unreliable networks.
In: 36th International Conference on Machine Learning, ICML 2019. Vol 2019-June.
IMLS; 2019:12481-12512.'
apa: 'Yu, C., Tang, H., Renggli, C., Kassing, S., Singla, A., Alistarh, D.-A., …
Liu, J. (2019). Distributed learning over unreliable networks. In 36th International
Conference on Machine Learning, ICML 2019 (Vol. 2019–June, pp. 12481–12512).
Long Beach, CA, United States: IMLS.'
chicago: Yu, Chen, Hanlin Tang, Cedric Renggli, Simon Kassing, Ankit Singla, Dan-Adrian
Alistarh, Ce Zhang, and Ji Liu. “Distributed Learning over Unreliable Networks.”
In 36th International Conference on Machine Learning, ICML 2019, 2019–June:12481–512.
IMLS, 2019.
ieee: C. Yu et al., “Distributed learning over unreliable networks,” in 36th
International Conference on Machine Learning, ICML 2019, Long Beach, CA, United
States, 2019, vol. 2019–June, pp. 12481–12512.
ista: 'Yu C, Tang H, Renggli C, Kassing S, Singla A, Alistarh D-A, Zhang C, Liu
J. 2019. Distributed learning over unreliable networks. 36th International Conference
on Machine Learning, ICML 2019. ICML: International Conference on Machine Learning
vol. 2019–June, 12481–12512.'
mla: Yu, Chen, et al. “Distributed Learning over Unreliable Networks.” 36th International
Conference on Machine Learning, ICML 2019, vol. 2019–June, IMLS, 2019, pp.
12481–512.
short: C. Yu, H. Tang, C. Renggli, S. Kassing, A. Singla, D.-A. Alistarh, C. Zhang,
J. Liu, in:, 36th International Conference on Machine Learning, ICML 2019, IMLS,
2019, pp. 12481–12512.
conference:
end_date: 2019-06-15
location: Long Beach, CA, United States
name: 'ICML: International Conference on Machine Learning'
start_date: 2019-06-10
date_created: 2020-02-02T23:01:06Z
date_published: 2019-06-01T00:00:00Z
date_updated: 2023-09-06T15:21:48Z
day: '01'
department:
- _id: DaAl
external_id:
arxiv:
- '1810.07766'
isi:
- '000684034307036'
isi: 1
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://arxiv.org/abs/1810.07766
month: '06'
oa: 1
oa_version: Preprint
page: 12481-12512
publication: 36th International Conference on Machine Learning, ICML 2019
publication_identifier:
isbn:
- '9781510886988'
publication_status: published
publisher: IMLS
quality_controlled: '1'
scopus_import: '1'
status: public
title: Distributed learning over unreliable networks
type: conference
user_id: c635000d-4b10-11ee-a964-aac5a93f6ac1
volume: 2019-June
year: '2019'
...
---
_id: '7542'
abstract:
- lang: eng
text: We present a novel class of convolutional neural networks (CNNs) for set functions,i.e.,
data indexed with the powerset of a finite set. The convolutions are derivedas
linear, shift-equivariant functions for various notions of shifts on set functions.The
framework is fundamentally different from graph convolutions based on theLaplacian,
as it provides not one but several basic shifts, one for each element inthe ground
set. Prototypical experiments with several set function classificationtasks on
synthetic datasets and on datasets derived from real-world hypergraphsdemonstrate
the potential of our new powerset CNNs.
article_processing_charge: No
arxiv: 1
author:
- first_name: Chris
full_name: Wendler, Chris
last_name: Wendler
- first_name: Dan-Adrian
full_name: Alistarh, Dan-Adrian
id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
last_name: Alistarh
orcid: 0000-0003-3650-940X
- first_name: Markus
full_name: Püschel, Markus
last_name: Püschel
citation:
ama: 'Wendler C, Alistarh D-A, Püschel M. Powerset convolutional neural networks.
In: Vol 32. Neural Information Processing Systems Foundation; 2019:927-938.'
apa: 'Wendler, C., Alistarh, D.-A., & Püschel, M. (2019). Powerset convolutional
neural networks (Vol. 32, pp. 927–938). Presented at the NIPS: Conference on Neural
Information Processing Systems, Vancouver, Canada: Neural Information Processing
Systems Foundation.'
chicago: Wendler, Chris, Dan-Adrian Alistarh, and Markus Püschel. “Powerset Convolutional
Neural Networks,” 32:927–38. Neural Information Processing Systems Foundation,
2019.
ieee: 'C. Wendler, D.-A. Alistarh, and M. Püschel, “Powerset convolutional neural
networks,” presented at the NIPS: Conference on Neural Information Processing
Systems, Vancouver, Canada, 2019, vol. 32, pp. 927–938.'
ista: 'Wendler C, Alistarh D-A, Püschel M. 2019. Powerset convolutional neural networks.
NIPS: Conference on Neural Information Processing Systems vol. 32, 927–938.'
mla: Wendler, Chris, et al. Powerset Convolutional Neural Networks. Vol.
32, Neural Information Processing Systems Foundation, 2019, pp. 927–38.
short: C. Wendler, D.-A. Alistarh, M. Püschel, in:, Neural Information Processing
Systems Foundation, 2019, pp. 927–938.
conference:
end_date: 2019-12-14
location: Vancouver, Canada
name: 'NIPS: Conference on Neural Information Processing Systems'
start_date: 2019-12-08
date_created: 2020-02-28T10:03:24Z
date_published: 2019-12-01T00:00:00Z
date_updated: 2025-04-14T07:49:16Z
day: '01'
department:
- _id: DaAl
ec_funded: 1
external_id:
arxiv:
- '1909.02253'
isi:
- '000534424300084'
intvolume: ' 32'
isi: 1
language:
- iso: eng
main_file_link:
- open_access: '1'
url: http://papers.nips.cc/paper/8379-powerset-convolutional-neural-networks
month: '12'
oa: 1
oa_version: Published Version
page: 927-938
project:
- _id: 268A44D6-B435-11E9-9278-68D0E5697425
call_identifier: H2020
grant_number: '805223'
name: Elastic Coordination for Scalable Machine Learning
publication_identifier:
issn:
- 1049-5258
publication_status: published
publisher: Neural Information Processing Systems Foundation
quality_controlled: '1'
status: public
title: Powerset convolutional neural networks
type: conference
user_id: c635000d-4b10-11ee-a964-aac5a93f6ac1
volume: 32
year: '2019'
...
---
_id: '6673'
abstract:
- lang: eng
text: Several classic problems in graph processing and computational geometry are
solved via incremental algorithms, which split computation into a series of small
tasks acting on shared state, which gets updated progressively. While the sequential
variant of such algorithms usually specifies a fixed (but sometimes random) order
in which the tasks should be performed, a standard approach to parallelizing such
algorithms is to relax this constraint to allow for out-of-order parallel execution.
This is the case for parallel implementations of Dijkstra's single-source shortest-paths
(SSSP) algorithm, and for parallel Delaunay mesh triangulation. While many software
frameworks parallelize incremental computation in this way, it is still not well
understood whether this relaxed ordering approach can still provide any complexity
guarantees. In this paper, we address this problem, and analyze the efficiency
guarantees provided by a range of incremental algorithms when parallelized via
relaxed schedulers. We show that, for algorithms such as Delaunay mesh triangulation
and sorting by insertion, schedulers with a maximum relaxation factor of k in
terms of the maximum priority inversion allowed will introduce a maximum amount
of wasted work of O(łog n poly(k)), where n is the number of tasks to be executed.
For SSSP, we show that the additional work is O(poly(k), dmax / wmin), where dmax
is the maximum distance between two nodes, and wmin is the minimum such distance.
In practical settings where n >> k, this suggests that the overheads of relaxation
will be outweighed by the improved scalability of the relaxed scheduler. On the
negative side, we provide lower bounds showing that certain algorithms will inherently
incur a non-trivial amount of wasted work due to scheduler relaxation, even for
relatively benign relaxed schedulers.
article_processing_charge: No
arxiv: 1
author:
- first_name: Dan-Adrian
full_name: Alistarh, Dan-Adrian
id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
last_name: Alistarh
orcid: 0000-0003-3650-940X
- first_name: Giorgi
full_name: Nadiradze, Giorgi
id: 3279A00C-F248-11E8-B48F-1D18A9856A87
last_name: Nadiradze
orcid: 0000-0001-5634-0731
- first_name: Nikita
full_name: Koval, Nikita
id: 2F4DB10C-F248-11E8-B48F-1D18A9856A87
last_name: Koval
citation:
ama: 'Alistarh D-A, Nadiradze G, Koval N. Efficiency guarantees for parallel incremental
algorithms under relaxed schedulers. In: 31st ACM Symposium on Parallelism
in Algorithms and Architectures. ACM; 2019:145-154. doi:10.1145/3323165.3323201'
apa: 'Alistarh, D.-A., Nadiradze, G., & Koval, N. (2019). Efficiency guarantees
for parallel incremental algorithms under relaxed schedulers. In 31st ACM Symposium
on Parallelism in Algorithms and Architectures (pp. 145–154). Phoenix, AZ,
United States: ACM. https://doi.org/10.1145/3323165.3323201'
chicago: Alistarh, Dan-Adrian, Giorgi Nadiradze, and Nikita Koval. “Efficiency Guarantees
for Parallel Incremental Algorithms under Relaxed Schedulers.” In 31st ACM
Symposium on Parallelism in Algorithms and Architectures, 145–54. ACM, 2019.
https://doi.org/10.1145/3323165.3323201.
ieee: D.-A. Alistarh, G. Nadiradze, and N. Koval, “Efficiency guarantees for parallel
incremental algorithms under relaxed schedulers,” in 31st ACM Symposium on
Parallelism in Algorithms and Architectures, Phoenix, AZ, United States, 2019,
pp. 145–154.
ista: 'Alistarh D-A, Nadiradze G, Koval N. 2019. Efficiency guarantees for parallel
incremental algorithms under relaxed schedulers. 31st ACM Symposium on Parallelism
in Algorithms and Architectures. SPAA: Symposium on Parallelism in Algorithms
and Architectures, 145–154.'
mla: Alistarh, Dan-Adrian, et al. “Efficiency Guarantees for Parallel Incremental
Algorithms under Relaxed Schedulers.” 31st ACM Symposium on Parallelism in
Algorithms and Architectures, ACM, 2019, pp. 145–54, doi:10.1145/3323165.3323201.
short: D.-A. Alistarh, G. Nadiradze, N. Koval, in:, 31st ACM Symposium on Parallelism
in Algorithms and Architectures, ACM, 2019, pp. 145–154.
conference:
end_date: 2019-06-24
location: Phoenix, AZ, United States
name: 'SPAA: Symposium on Parallelism in Algorithms and Architectures'
start_date: 2019-06-22
date_created: 2019-07-24T08:59:36Z
date_published: 2019-06-01T00:00:00Z
date_updated: 2026-04-08T07:00:45Z
day: '01'
department:
- _id: DaAl
doi: 10.1145/3323165.3323201
ec_funded: 1
external_id:
arxiv:
- '2003.09363'
isi:
- '000507618500018'
isi: 1
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://arxiv.org/abs/2003.09363
month: '06'
oa: 1
oa_version: Preprint
page: 145-154
project:
- _id: 268A44D6-B435-11E9-9278-68D0E5697425
call_identifier: H2020
grant_number: '805223'
name: Elastic Coordination for Scalable Machine Learning
publication: 31st ACM Symposium on Parallelism in Algorithms and Architectures
publication_identifier:
isbn:
- '9781450361842'
publication_status: published
publisher: ACM
quality_controlled: '1'
related_material:
record:
- id: '10429'
relation: dissertation_contains
status: public
scopus_import: '1'
status: public
title: Efficiency guarantees for parallel incremental algorithms under relaxed schedulers
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
year: '2019'
...
---
_id: '7214'
abstract:
- lang: eng
text: "Background: Many cancer genomes are extensively rearranged with highly aberrant
chromosomal karyotypes. Structural and copy number variations in cancer genomes
can be determined via abnormal mapping of sequenced reads to the reference genome.
Recently it became possible to reconcile both of these types of large-scale variations
into a karyotype graph representation of the rearranged cancer genomes. Such a
representation, however, does not directly describe the linear and/or circular
structure of the underlying rearranged cancer chromosomes, thus limiting possible
analysis of cancer genomes somatic evolutionary process as well as functional
genomic changes brought by the large-scale genome rearrangements.\r\n\r\nResults:
Here we address the aforementioned limitation by introducing a novel methodological
framework for recovering rearranged cancer chromosomes from karyotype graphs.
For a cancer karyotype graph we formulate an Eulerian Decomposition Problem (EDP)
of finding a collection of linear and/or circular rearranged cancer chromosomes
that are determined by the graph. We derive and prove computational complexities
for several variations of the EDP. We then demonstrate that Eulerian decomposition
of the cancer karyotype graphs is not always unique and present the Consistent
Contig Covering Problem (CCCP) of recovering unambiguous cancer contigs from the
cancer karyotype graph, and describe a novel algorithm CCR capable of solving
CCCP in polynomial time. We apply CCR on a prostate cancer dataset and demonstrate
that it is capable of consistently recovering large cancer contigs even when underlying
cancer genomes are highly rearranged.\r\n\r\nConclusions: CCR can recover rearranged
cancer contigs from karyotype graphs thereby addressing existing limitation in
inferring chromosomal structures of rearranged cancer genomes and advancing our
understanding of both patient/cancer-specific as well as the overall genetic instability
in cancer."
article_number: '641'
article_processing_charge: No
article_type: original
author:
- first_name: Sergey
full_name: Aganezov, Sergey
last_name: Aganezov
- first_name: Ilya
full_name: Zban, Ilya
last_name: Zban
- first_name: Vitalii
full_name: Aksenov, Vitalii
id: 2980135A-F248-11E8-B48F-1D18A9856A87
last_name: Aksenov
- first_name: Nikita
full_name: Alexeev, Nikita
last_name: Alexeev
- first_name: Michael C.
full_name: Schatz, Michael C.
last_name: Schatz
citation:
ama: Aganezov S, Zban I, Aksenov V, Alexeev N, Schatz MC. Recovering rearranged
cancer chromosomes from karyotype graphs. BMC Bioinformatics. 2019;20.
doi:10.1186/s12859-019-3208-4
apa: Aganezov, S., Zban, I., Aksenov, V., Alexeev, N., & Schatz, M. C. (2019).
Recovering rearranged cancer chromosomes from karyotype graphs. BMC Bioinformatics.
BMC. https://doi.org/10.1186/s12859-019-3208-4
chicago: Aganezov, Sergey, Ilya Zban, Vitalii Aksenov, Nikita Alexeev, and Michael
C. Schatz. “Recovering Rearranged Cancer Chromosomes from Karyotype Graphs.” BMC
Bioinformatics. BMC, 2019. https://doi.org/10.1186/s12859-019-3208-4.
ieee: S. Aganezov, I. Zban, V. Aksenov, N. Alexeev, and M. C. Schatz, “Recovering
rearranged cancer chromosomes from karyotype graphs,” BMC Bioinformatics,
vol. 20. BMC, 2019.
ista: Aganezov S, Zban I, Aksenov V, Alexeev N, Schatz MC. 2019. Recovering rearranged
cancer chromosomes from karyotype graphs. BMC Bioinformatics. 20, 641.
mla: Aganezov, Sergey, et al. “Recovering Rearranged Cancer Chromosomes from Karyotype
Graphs.” BMC Bioinformatics, vol. 20, 641, BMC, 2019, doi:10.1186/s12859-019-3208-4.
short: S. Aganezov, I. Zban, V. Aksenov, N. Alexeev, M.C. Schatz, BMC Bioinformatics
20 (2019).
date_created: 2019-12-29T23:00:46Z
date_published: 2019-12-17T00:00:00Z
date_updated: 2026-04-16T08:35:00Z
day: '17'
ddc:
- '570'
department:
- _id: DaAl
doi: 10.1186/s12859-019-3208-4
external_id:
isi:
- '000511618800007'
file:
- access_level: open_access
checksum: 7a30357efdcf8f66587ed495c0927724
content_type: application/pdf
creator: dernst
date_created: 2020-01-02T16:10:58Z
date_updated: 2020-07-14T12:47:54Z
file_id: '7221'
file_name: 2019_BMCBioinfo_Aganezov.pdf
file_size: 1917374
relation: main_file
file_date_updated: 2020-07-14T12:47:54Z
has_accepted_license: '1'
intvolume: ' 20'
isi: 1
language:
- iso: eng
license: https://creativecommons.org/licenses/by/4.0/
month: '12'
oa: 1
oa_version: Published Version
publication: BMC Bioinformatics
publication_identifier:
eissn:
- 1471-2105
publication_status: published
publisher: BMC
quality_controlled: '1'
scopus_import: '1'
status: public
title: Recovering rearranged cancer chromosomes from karyotype graphs
tmp:
image: /images/cc_by.png
legal_code_url: https://creativecommons.org/licenses/by/4.0/legalcode
name: Creative Commons Attribution 4.0 International Public License (CC-BY 4.0)
short: CC BY (4.0)
type: journal_article
user_id: ba8df636-2132-11f1-aed0-ed93e2281fdd
volume: 20
year: '2019'
...
---
_id: '7812'
abstract:
- lang: eng
text: Deep neural networks (DNNs) continue to make significant advances, solving
tasks from image classification to translation or reinforcement learning. One
aspect of the field receiving considerable attention is efficiently executing
deep models in resource-constrained environments, such as mobile or embedded devices.
This paper focuses on this problem, and proposes two new compression methods,
which jointly leverage weight quantization and distillation of larger teacher
networks into smaller student networks. The first method we propose is called
quantized distillation and leverages distillation during the training process,
by incorporating distillation loss, expressed with respect to the teacher, into
the training of a student network whose weights are quantized to a limited set
of levels. The second method, differentiable quantization, optimizes the location
of quantization points through stochastic gradient descent, to better fit the
behavior of the teacher model. We validate both methods through experiments on
convolutional and recurrent architectures. We show that quantized shallow students
can reach similar accuracy levels to full-precision teacher models, while providing
order of magnitude compression, and inference speedup that is linear in the depth
reduction. In sum, our results enable DNNs for resource-constrained environments
to leverage architecture and accuracy advances developed on more powerful devices.
acknowledgement: "We would like to thank Ce Zhang (ETH Zurich), Hantian Zhang (ETH
Zurich) and Martin Jaggi ´\r\n(EPFL) for their support with experiments and valuable
feedback.\r\n"
article_processing_charge: No
arxiv: 1
author:
- first_name: Antonio
full_name: Polino, Antonio
last_name: Polino
- first_name: Razvan
full_name: Pascanu, Razvan
last_name: Pascanu
- first_name: Dan-Adrian
full_name: Alistarh, Dan-Adrian
id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
last_name: Alistarh
orcid: 0000-0003-3650-940X
citation:
ama: 'Polino A, Pascanu R, Alistarh D-A. Model compression via distillation and
quantization. In: 6th International Conference on Learning Representations.
; 2018.'
apa: Polino, A., Pascanu, R., & Alistarh, D.-A. (2018). Model compression via
distillation and quantization. In 6th International Conference on Learning
Representations. Vancouver, Canada.
chicago: Polino, Antonio, Razvan Pascanu, and Dan-Adrian Alistarh. “Model Compression
via Distillation and Quantization.” In 6th International Conference on Learning
Representations, 2018.
ieee: A. Polino, R. Pascanu, and D.-A. Alistarh, “Model compression via distillation
and quantization,” in 6th International Conference on Learning Representations,
Vancouver, Canada, 2018.
ista: 'Polino A, Pascanu R, Alistarh D-A. 2018. Model compression via distillation
and quantization. 6th International Conference on Learning Representations. ICLR:
International Conference on Learning Representations.'
mla: Polino, Antonio, et al. “Model Compression via Distillation and Quantization.”
6th International Conference on Learning Representations, 2018.
short: A. Polino, R. Pascanu, D.-A. Alistarh, in:, 6th International Conference
on Learning Representations, 2018.
conference:
end_date: 2018-05-03
location: Vancouver, Canada
name: 'ICLR: International Conference on Learning Representations'
start_date: 2018-04-30
date_created: 2020-05-10T22:00:51Z
date_published: 2018-05-01T00:00:00Z
date_updated: 2025-06-30T10:04:44Z
day: '01'
ddc:
- '000'
department:
- _id: DaAl
external_id:
arxiv:
- '1802.05668'
file:
- access_level: open_access
checksum: a4336c167978e81891970e4e4517a8c3
content_type: application/pdf
creator: dernst
date_created: 2020-05-26T13:02:00Z
date_updated: 2020-07-14T12:48:03Z
file_id: '7894'
file_name: 2018_ICLR_Polino.pdf
file_size: 308339
relation: main_file
file_date_updated: 2020-07-14T12:48:03Z
has_accepted_license: '1'
language:
- iso: eng
month: '05'
oa: 1
oa_version: Published Version
publication: 6th International Conference on Learning Representations
publication_status: published
quality_controlled: '1'
scopus_import: '1'
status: public
title: Model compression via distillation and quantization
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
year: '2018'
...
---
_id: '5961'
abstract:
- lang: eng
text: "The area of machine learning has made considerable progress over the past
decade, enabled by the widespread availability of large datasets, as well as by
improved algorithms and models. Given the large computational demands of machine
learning workloads, parallelism, implemented either through single-node concurrency
or through multi-node distribution, has been a third key ingredient to advances
in machine learning.\r\nThe goal of this tutorial is to provide the audience with
an overview of standard distribution techniques in machine learning, with an eye
towards the intriguing trade-offs between synchronization and communication costs
of distributed machine learning algorithms, on the one hand, and their convergence,
on the other.The tutorial will focus on parallelization strategies for the fundamental
stochastic gradient descent (SGD) algorithm, which is a key tool when training
machine learning models, from classical instances such as linear regression, to
state-of-the-art neural network architectures.\r\nThe tutorial will describe the
guarantees provided by this algorithm in the sequential case, and then move on
to cover both shared-memory and message-passing parallelization strategies, together
with the guarantees they provide, and corresponding trade-offs. The presentation
will conclude with a broad overview of ongoing research in distributed and concurrent
machine learning. The tutorial will assume no prior knowledge beyond familiarity
with basic concepts in algebra and analysis.\r\n"
article_processing_charge: No
author:
- first_name: Dan-Adrian
full_name: Alistarh, Dan-Adrian
id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
last_name: Alistarh
orcid: 0000-0003-3650-940X
citation:
ama: 'Alistarh D-A. A brief tutorial on distributed and concurrent machine learning.
In: Proceedings of the 2018 ACM Symposium on Principles of Distributed Computing
- PODC ’18. ACM; 2018:487-488. doi:10.1145/3212734.3212798'
apa: 'Alistarh, D.-A. (2018). A brief tutorial on distributed and concurrent machine
learning. In Proceedings of the 2018 ACM Symposium on Principles of Distributed
Computing - PODC ’18 (pp. 487–488). Egham, United Kingdom: ACM. https://doi.org/10.1145/3212734.3212798'
chicago: Alistarh, Dan-Adrian. “A Brief Tutorial on Distributed and Concurrent Machine
Learning.” In Proceedings of the 2018 ACM Symposium on Principles of Distributed
Computing - PODC ’18, 487–88. ACM, 2018. https://doi.org/10.1145/3212734.3212798.
ieee: D.-A. Alistarh, “A brief tutorial on distributed and concurrent machine learning,”
in Proceedings of the 2018 ACM Symposium on Principles of Distributed Computing
- PODC ’18, Egham, United Kingdom, 2018, pp. 487–488.
ista: 'Alistarh D-A. 2018. A brief tutorial on distributed and concurrent machine
learning. Proceedings of the 2018 ACM Symposium on Principles of Distributed Computing
- PODC ’18. PODC: Principles of Distributed Computing, 487–488.'
mla: Alistarh, Dan-Adrian. “A Brief Tutorial on Distributed and Concurrent Machine
Learning.” Proceedings of the 2018 ACM Symposium on Principles of Distributed
Computing - PODC ’18, ACM, 2018, pp. 487–88, doi:10.1145/3212734.3212798.
short: D.-A. Alistarh, in:, Proceedings of the 2018 ACM Symposium on Principles
of Distributed Computing - PODC ’18, ACM, 2018, pp. 487–488.
conference:
end_date: 2018-07-27
location: Egham, United Kingdom
name: 'PODC: Principles of Distributed Computing'
start_date: 2018-07-23
date_created: 2019-02-13T09:48:55Z
date_published: 2018-07-27T00:00:00Z
date_updated: 2025-06-03T11:56:33Z
day: '27'
department:
- _id: DaAl
doi: 10.1145/3212734.3212798
external_id:
isi:
- '000458186900063'
isi: 1
language:
- iso: eng
month: '07'
oa_version: None
page: 487-488
publication: Proceedings of the 2018 ACM Symposium on Principles of Distributed Computing -
PODC '18
publication_identifier:
isbn:
- '9781450357951'
publication_status: published
publisher: ACM
quality_controlled: '1'
scopus_import: '1'
status: public
title: A brief tutorial on distributed and concurrent machine learning
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
year: '2018'
...
---
_id: '5962'
abstract:
- lang: eng
text: Stochastic Gradient Descent (SGD) is a fundamental algorithm in machine learning,
representing the optimization backbone for training several classic models, from
regression to neural networks. Given the recent practical focus on distributed
machine learning, significant work has been dedicated to the convergence properties
of this algorithm under the inconsistent and noisy updates arising from execution
in a distributed environment. However, surprisingly, the convergence properties
of this classic algorithm in the standard shared-memory model are still not well-understood.
In this work, we address this gap, and provide new convergence bounds for lock-free
concurrent stochastic gradient descent, executing in the classic asynchronous
shared memory model, against a strong adaptive adversary. Our results give improved
upper and lower bounds on the "price of asynchrony'' when executing the fundamental
SGD algorithm in a concurrent setting. They show that this classic optimization
tool can converge faster and with a wider range of parameters than previously
known under asynchronous iterations. At the same time, we exhibit a fundamental
trade-off between the maximum delay in the system and the rate at which SGD can
converge, which governs the set of parameters under which this algorithm can still
work efficiently.
article_processing_charge: No
arxiv: 1
author:
- first_name: Dan-Adrian
full_name: Alistarh, Dan-Adrian
id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
last_name: Alistarh
orcid: 0000-0003-3650-940X
- first_name: Christopher
full_name: De Sa, Christopher
last_name: De Sa
- first_name: Nikola H
full_name: Konstantinov, Nikola H
id: 4B9D76E4-F248-11E8-B48F-1D18A9856A87
last_name: Konstantinov
citation:
ama: 'Alistarh D-A, De Sa C, Konstantinov NH. The convergence of stochastic gradient
descent in asynchronous shared memory. In: Proceedings of the 2018 ACM Symposium
on Principles of Distributed Computing - PODC ’18. ACM; 2018:169-178. doi:10.1145/3212734.3212763'
apa: 'Alistarh, D.-A., De Sa, C., & Konstantinov, N. H. (2018). The convergence
of stochastic gradient descent in asynchronous shared memory. In Proceedings
of the 2018 ACM Symposium on Principles of Distributed Computing - PODC ’18
(pp. 169–178). Egham, United Kingdom: ACM. https://doi.org/10.1145/3212734.3212763'
chicago: Alistarh, Dan-Adrian, Christopher De Sa, and Nikola H Konstantinov. “The
Convergence of Stochastic Gradient Descent in Asynchronous Shared Memory.” In
Proceedings of the 2018 ACM Symposium on Principles of Distributed Computing
- PODC ’18, 169–78. ACM, 2018. https://doi.org/10.1145/3212734.3212763.
ieee: D.-A. Alistarh, C. De Sa, and N. H. Konstantinov, “The convergence of stochastic
gradient descent in asynchronous shared memory,” in Proceedings of the 2018
ACM Symposium on Principles of Distributed Computing - PODC ’18, Egham, United
Kingdom, 2018, pp. 169–178.
ista: 'Alistarh D-A, De Sa C, Konstantinov NH. 2018. The convergence of stochastic
gradient descent in asynchronous shared memory. Proceedings of the 2018 ACM Symposium
on Principles of Distributed Computing - PODC ’18. PODC: Principles of Distributed
Computing, 169–178.'
mla: Alistarh, Dan-Adrian, et al. “The Convergence of Stochastic Gradient Descent
in Asynchronous Shared Memory.” Proceedings of the 2018 ACM Symposium on Principles
of Distributed Computing - PODC ’18, ACM, 2018, pp. 169–78, doi:10.1145/3212734.3212763.
short: D.-A. Alistarh, C. De Sa, N.H. Konstantinov, in:, Proceedings of the 2018
ACM Symposium on Principles of Distributed Computing - PODC ’18, ACM, 2018, pp.
169–178.
conference:
end_date: 2018-07-27
location: Egham, United Kingdom
name: 'PODC: Principles of Distributed Computing'
start_date: 2018-07-23
date_created: 2019-02-13T09:58:58Z
date_published: 2018-07-23T00:00:00Z
date_updated: 2025-06-03T11:56:41Z
day: '23'
department:
- _id: DaAl
doi: 10.1145/3212734.3212763
external_id:
arxiv:
- '1803.08841'
isi:
- '000458186900022'
isi: 1
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://arxiv.org/abs/1803.08841
month: '07'
oa: 1
oa_version: Preprint
page: 169-178
publication: Proceedings of the 2018 ACM Symposium on Principles of Distributed Computing -
PODC '18
publication_identifier:
isbn:
- '9781450357951'
publication_status: published
publisher: ACM
quality_controlled: '1'
scopus_import: '1'
status: public
title: The convergence of stochastic gradient descent in asynchronous shared memory
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
year: '2018'
...
---
_id: '5963'
abstract:
- lang: eng
text: 'There has been significant progress in understanding the parallelism inherent
to iterative sequential algorithms: for many classic algorithms, the depth of
the dependence structure is now well understood, and scheduling techniques have
been developed to exploit this shallow dependence structure for efficient parallel
implementations. A related, applied research strand has studied methods by which
certain iterative task-based algorithms can be efficiently parallelized via relaxed
concurrent priority schedulers. These allow for high concurrency when inserting
and removing tasks, at the cost of executing superfluous work due to the relaxed
semantics of the scheduler. In this work, we take a step towards unifying these
two research directions, by showing that there exists a family of relaxed priority
schedulers that can efficiently and deterministically execute classic iterative
algorithms such as greedy maximal independent set (MIS) and matching. Our primary
result shows that, given a randomized scheduler with an expected relaxation factor
of k in terms of the maximum allowed priority inversions on a task, and any graph
on n vertices, the scheduler is able to execute greedy MIS with only an additive
factor of \poly(k) expected additional iterations compared to an exact (but not
scalable) scheduler. This counter-intuitive result demonstrates that the overhead
of relaxation when computing MIS is not dependent on the input size or structure
of the input graph. Experimental results show that this overhead can be clearly
offset by the gain in performance due to the highly scalable scheduler. In sum,
we present an efficient method to deterministically parallelize iterative sequential
algorithms, with provable runtime guarantees in terms of the number of executed
tasks to completion.'
article_processing_charge: No
arxiv: 1
author:
- first_name: Dan-Adrian
full_name: Alistarh, Dan-Adrian
id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
last_name: Alistarh
orcid: 0000-0003-3650-940X
- first_name: Trevor A
full_name: Brown, Trevor A
id: 3569F0A0-F248-11E8-B48F-1D18A9856A87
last_name: Brown
- first_name: Justin
full_name: Kopinsky, Justin
last_name: Kopinsky
- first_name: Giorgi
full_name: Nadiradze, Giorgi
last_name: Nadiradze
citation:
ama: 'Alistarh D-A, Brown TA, Kopinsky J, Nadiradze G. Relaxed schedulers can efficiently
parallelize iterative algorithms. In: Proceedings of the 2018 ACM Symposium
on Principles of Distributed Computing - PODC ’18. ACM; 2018:377-386. doi:10.1145/3212734.3212756'
apa: 'Alistarh, D.-A., Brown, T. A., Kopinsky, J., & Nadiradze, G. (2018). Relaxed
schedulers can efficiently parallelize iterative algorithms. In Proceedings
of the 2018 ACM Symposium on Principles of Distributed Computing - PODC ’18
(pp. 377–386). Egham, United Kingdom: ACM. https://doi.org/10.1145/3212734.3212756'
chicago: Alistarh, Dan-Adrian, Trevor A Brown, Justin Kopinsky, and Giorgi Nadiradze.
“Relaxed Schedulers Can Efficiently Parallelize Iterative Algorithms.” In Proceedings
of the 2018 ACM Symposium on Principles of Distributed Computing - PODC ’18,
377–86. ACM, 2018. https://doi.org/10.1145/3212734.3212756.
ieee: D.-A. Alistarh, T. A. Brown, J. Kopinsky, and G. Nadiradze, “Relaxed schedulers
can efficiently parallelize iterative algorithms,” in Proceedings of the 2018
ACM Symposium on Principles of Distributed Computing - PODC ’18, Egham, United
Kingdom, 2018, pp. 377–386.
ista: 'Alistarh D-A, Brown TA, Kopinsky J, Nadiradze G. 2018. Relaxed schedulers
can efficiently parallelize iterative algorithms. Proceedings of the 2018 ACM
Symposium on Principles of Distributed Computing - PODC ’18. PODC: Principles
of Distributed Computing, 377–386.'
mla: Alistarh, Dan-Adrian, et al. “Relaxed Schedulers Can Efficiently Parallelize
Iterative Algorithms.” Proceedings of the 2018 ACM Symposium on Principles
of Distributed Computing - PODC ’18, ACM, 2018, pp. 377–86, doi:10.1145/3212734.3212756.
short: D.-A. Alistarh, T.A. Brown, J. Kopinsky, G. Nadiradze, in:, Proceedings of
the 2018 ACM Symposium on Principles of Distributed Computing - PODC ’18, ACM,
2018, pp. 377–386.
conference:
end_date: 2018-07-27
location: Egham, United Kingdom
name: 'PODC: Principles of Distributed Computing'
start_date: 2018-07-23
date_created: 2019-02-13T10:03:25Z
date_published: 2018-07-23T00:00:00Z
date_updated: 2025-06-03T11:56:49Z
day: '23'
department:
- _id: DaAl
doi: 10.1145/3212734.3212756
external_id:
arxiv:
- '1808.04155'
isi:
- '000458186900048'
isi: 1
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://arxiv.org/abs/1808.04155
month: '07'
oa: 1
oa_version: Preprint
page: 377-386
publication: Proceedings of the 2018 ACM Symposium on Principles of Distributed Computing -
PODC '18
publication_identifier:
isbn:
- '9781450357951'
publication_status: published
publisher: ACM
quality_controlled: '1'
scopus_import: '1'
status: public
title: Relaxed schedulers can efficiently parallelize iterative algorithms
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
year: '2018'
...
---
_id: '5964'
abstract:
- lang: eng
text: A standard design pattern found in many concurrent data structures, such as
hash tables or ordered containers, is an alternation of parallelizable sections
that incur no data conflicts and critical sections that must run sequentially
and are protected with locks. A lock can be viewed as a queue that arbitrates
the order in which the critical sections are executed, and a natural question
is whether we can use stochastic analysis to predict the resulting throughput.
As a preliminary evidence to the affirmative, we describe a simple model that
can be used to predict the throughput of coarse-grained lock-based algorithms.
We show that our model works well for CLH lock, and we expect it to work for other
popular lock designs such as TTAS, MCS, etc.
article_processing_charge: No
author:
- first_name: Vitaly
full_name: Aksenov, Vitaly
last_name: Aksenov
- first_name: Dan-Adrian
full_name: Alistarh, Dan-Adrian
id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
last_name: Alistarh
orcid: 0000-0003-3650-940X
- first_name: Petr
full_name: Kuznetsov, Petr
last_name: Kuznetsov
citation:
ama: 'Aksenov V, Alistarh D-A, Kuznetsov P. Brief Announcement: Performance prediction
for coarse-grained locking. In: Proceedings of the 2018 ACM Symposium on Principles
of Distributed Computing - PODC ’18. ACM; 2018:411-413. doi:10.1145/3212734.3212785'
apa: 'Aksenov, V., Alistarh, D.-A., & Kuznetsov, P. (2018). Brief Announcement:
Performance prediction for coarse-grained locking. In Proceedings of the 2018
ACM Symposium on Principles of Distributed Computing - PODC ’18 (pp. 411–413).
Egham, United Kingdom: ACM. https://doi.org/10.1145/3212734.3212785'
chicago: 'Aksenov, Vitaly, Dan-Adrian Alistarh, and Petr Kuznetsov. “Brief Announcement:
Performance Prediction for Coarse-Grained Locking.” In Proceedings of the 2018
ACM Symposium on Principles of Distributed Computing - PODC ’18, 411–13.
ACM, 2018. https://doi.org/10.1145/3212734.3212785.'
ieee: 'V. Aksenov, D.-A. Alistarh, and P. Kuznetsov, “Brief Announcement: Performance
prediction for coarse-grained locking,” in Proceedings of the 2018 ACM Symposium
on Principles of Distributed Computing - PODC ’18, Egham, United Kingdom,
2018, pp. 411–413.'
ista: 'Aksenov V, Alistarh D-A, Kuznetsov P. 2018. Brief Announcement: Performance
prediction for coarse-grained locking. Proceedings of the 2018 ACM Symposium on
Principles of Distributed Computing - PODC ’18. PODC: Principles of Distributed
Computing, 411–413.'
mla: 'Aksenov, Vitaly, et al. “Brief Announcement: Performance Prediction for Coarse-Grained
Locking.” Proceedings of the 2018 ACM Symposium on Principles of Distributed
Computing - PODC ’18, ACM, 2018, pp. 411–13, doi:10.1145/3212734.3212785.'
short: V. Aksenov, D.-A. Alistarh, P. Kuznetsov, in:, Proceedings of the 2018 ACM
Symposium on Principles of Distributed Computing - PODC ’18, ACM, 2018, pp. 411–413.
conference:
end_date: 2018-07-27
location: Egham, United Kingdom
name: 'PODC: Principles of Distributed Computing'
start_date: 2018-07-23
date_created: 2019-02-13T10:08:19Z
date_published: 2018-07-23T00:00:00Z
date_updated: 2025-06-03T11:58:00Z
day: '23'
department:
- _id: DaAl
doi: 10.1145/3212734.3212785
external_id:
isi:
- '000458186900052'
isi: 1
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://hal-univ-lyon3.archives-ouvertes.fr/INRIA/hal-01887733v1
month: '07'
oa: 1
oa_version: Submitted Version
page: 411-413
publication: Proceedings of the 2018 ACM Symposium on Principles of Distributed Computing -
PODC '18
publication_identifier:
isbn:
- '9781450357951'
publication_status: published
publisher: ACM
quality_controlled: '1'
scopus_import: '1'
status: public
title: 'Brief Announcement: Performance prediction for coarse-grained locking'
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
year: '2018'
...
---
_id: '5966'
abstract:
- lang: eng
text: 'The transactional conflict problem arises in transactional systems whenever
two or more concurrent transactions clash on a data item. While the standard solution
to such conflicts is to immediately abort one of the transactions, some practical
systems consider the alternative of delaying conflict resolution for a short interval,
which may allow one of the transactions to commit. The challenge in the transactional
conflict problem is to choose the optimal length of this delay interval so as
to minimize the overall running time penalty for the conflicting transactions.
In this paper, we propose a family of optimal online algorithms for the transactional
conflict problem. Specifically, we consider variants of this problem which arise
in different implementations of transactional systems, namely "requestor wins''''
and "requestor aborts'''' implementations: in the former, the recipient of a coherence
request is aborted, whereas in the latter, it is the requestor which has to abort.
Both strategies are implemented by real systems. We show that the requestor aborts
case can be reduced to a classic instance of the ski rental problem, while the
requestor wins case leads to a new version of this classical problem, for which
we derive optimal deterministic and randomized algorithms. Moreover, we prove
that, under a simplified adversarial model, our algorithms are constant-competitive
with the offline optimum in terms of throughput. We validate our algorithmic results
empirically through a hardware simulation of hardware transactional memory (HTM),
showing that our algorithms can lead to non-trivial performance improvements for
classic concurrent data structures.'
article_processing_charge: No
arxiv: 1
author:
- first_name: Dan-Adrian
full_name: Alistarh, Dan-Adrian
id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
last_name: Alistarh
orcid: 0000-0003-3650-940X
- first_name: Syed Kamran
full_name: Haider, Syed Kamran
last_name: Haider
- first_name: Raphael
full_name: Kübler, Raphael
last_name: Kübler
- first_name: Giorgi
full_name: Nadiradze, Giorgi
last_name: Nadiradze
citation:
ama: 'Alistarh D-A, Haider SK, Kübler R, Nadiradze G. The transactional conflict
problem. In: Proceedings of the 30th on Symposium on Parallelism in Algorithms
and Architectures - SPAA ’18. ACM; 2018:383-392. doi:10.1145/3210377.3210406'
apa: 'Alistarh, D.-A., Haider, S. K., Kübler, R., & Nadiradze, G. (2018). The
transactional conflict problem. In Proceedings of the 30th on Symposium on
Parallelism in Algorithms and Architectures - SPAA ’18 (pp. 383–392). Vienna,
Austria: ACM. https://doi.org/10.1145/3210377.3210406'
chicago: Alistarh, Dan-Adrian, Syed Kamran Haider, Raphael Kübler, and Giorgi Nadiradze.
“The Transactional Conflict Problem.” In Proceedings of the 30th on Symposium
on Parallelism in Algorithms and Architectures - SPAA ’18, 383–92. ACM, 2018.
https://doi.org/10.1145/3210377.3210406.
ieee: D.-A. Alistarh, S. K. Haider, R. Kübler, and G. Nadiradze, “The transactional
conflict problem,” in Proceedings of the 30th on Symposium on Parallelism in
Algorithms and Architectures - SPAA ’18, Vienna, Austria, 2018, pp. 383–392.
ista: 'Alistarh D-A, Haider SK, Kübler R, Nadiradze G. 2018. The transactional conflict
problem. Proceedings of the 30th on Symposium on Parallelism in Algorithms and
Architectures - SPAA ’18. SPAA: Symposium on Parallelism in Algorithms and Architectures,
383–392.'
mla: Alistarh, Dan-Adrian, et al. “The Transactional Conflict Problem.” Proceedings
of the 30th on Symposium on Parallelism in Algorithms and Architectures - SPAA
’18, ACM, 2018, pp. 383–92, doi:10.1145/3210377.3210406.
short: D.-A. Alistarh, S.K. Haider, R. Kübler, G. Nadiradze, in:, Proceedings of
the 30th on Symposium on Parallelism in Algorithms and Architectures - SPAA ’18,
ACM, 2018, pp. 383–392.
conference:
end_date: 2018-07-18
location: Vienna, Austria
name: 'SPAA: Symposium on Parallelism in Algorithms and Architectures'
start_date: 2018-07-16
date_created: 2019-02-13T10:26:07Z
date_published: 2018-07-16T00:00:00Z
date_updated: 2025-06-03T11:58:22Z
day: '16'
department:
- _id: DaAl
doi: 10.1145/3210377.3210406
external_id:
arxiv:
- '1804.00947'
isi:
- '000545269600046'
isi: 1
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://arxiv.org/abs/1804.00947
month: '07'
oa: 1
oa_version: Preprint
page: 383-392
publication: Proceedings of the 30th on Symposium on Parallelism in Algorithms and
Architectures - SPAA '18
publication_identifier:
isbn:
- '9781450357999'
publication_status: published
publisher: ACM
quality_controlled: '1'
scopus_import: '1'
status: public
title: The transactional conflict problem
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
year: '2018'
...
---
_id: '6001'
abstract:
- lang: eng
text: "The concurrent memory reclamation problem is that of devising a way for a
deallocating thread to verify that no other concurrent threads hold references
to a memory block being deallocated. To date, in the absence of automatic garbage
collection, there is no satisfactory solution to this problem; existing tracking
methods like hazard pointers, reference counters, or epoch-based techniques like
RCU are either prohibitively expensive or require significant programming expertise
to the extent that implementing them efficiently can be worthy of a publication.
None of the existing techniques are automatic or even semi-automated.\r\nIn this
article, we take a new approach to concurrent memory reclamation. Instead of manually
tracking access to memory locations as done in techniques like hazard pointers,
or restricting shared accesses to specific epoch boundaries as in RCU, our algorithm,
called ThreadScan, leverages operating system signaling to automatically detect
which memory locations are being accessed by concurrent threads.\r\nInitial empirical
evidence shows that ThreadScan scales surprisingly well and requires negligible
programming effort beyond the standard use of Malloc and Free."
article_number: '18'
author:
- first_name: Dan-Adrian
full_name: Alistarh, Dan-Adrian
id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
last_name: Alistarh
orcid: 0000-0003-3650-940X
- first_name: William
full_name: Leiserson, William
last_name: Leiserson
- first_name: Alexander
full_name: Matveev, Alexander
last_name: Matveev
- first_name: Nir
full_name: Shavit, Nir
last_name: Shavit
citation:
ama: 'Alistarh D-A, Leiserson W, Matveev A, Shavit N. ThreadScan: Automatic and
scalable memory reclamation. ACM Transactions on Parallel Computing. 2018;4(4).
doi:10.1145/3201897'
apa: 'Alistarh, D.-A., Leiserson, W., Matveev, A., & Shavit, N. (2018). ThreadScan:
Automatic and scalable memory reclamation. ACM Transactions on Parallel Computing.
Association for Computing Machinery. https://doi.org/10.1145/3201897'
chicago: 'Alistarh, Dan-Adrian, William Leiserson, Alexander Matveev, and Nir Shavit.
“ThreadScan: Automatic and Scalable Memory Reclamation.” ACM Transactions on
Parallel Computing. Association for Computing Machinery, 2018. https://doi.org/10.1145/3201897.'
ieee: 'D.-A. Alistarh, W. Leiserson, A. Matveev, and N. Shavit, “ThreadScan: Automatic
and scalable memory reclamation,” ACM Transactions on Parallel Computing,
vol. 4, no. 4. Association for Computing Machinery, 2018.'
ista: 'Alistarh D-A, Leiserson W, Matveev A, Shavit N. 2018. ThreadScan: Automatic
and scalable memory reclamation. ACM Transactions on Parallel Computing. 4(4),
18.'
mla: 'Alistarh, Dan-Adrian, et al. “ThreadScan: Automatic and Scalable Memory Reclamation.”
ACM Transactions on Parallel Computing, vol. 4, no. 4, 18, Association
for Computing Machinery, 2018, doi:10.1145/3201897.'
short: D.-A. Alistarh, W. Leiserson, A. Matveev, N. Shavit, ACM Transactions on
Parallel Computing 4 (2018).
date_created: 2019-02-14T13:24:11Z
date_published: 2018-09-01T00:00:00Z
date_updated: 2023-02-23T13:17:54Z
day: '01'
department:
- _id: DaAl
doi: 10.1145/3201897
intvolume: ' 4'
issue: '4'
language:
- iso: eng
month: '09'
oa_version: None
publication: ACM Transactions on Parallel Computing
publication_identifier:
issn:
- 2329-4949
publication_status: published
publisher: Association for Computing Machinery
quality_controlled: '1'
related_material:
record:
- id: '779'
relation: earlier_version
status: public
scopus_import: 1
status: public
title: 'ThreadScan: Automatic and scalable memory reclamation'
type: journal_article
user_id: 3E5EF7F0-F248-11E8-B48F-1D18A9856A87
volume: 4
year: '2018'
...
---
_id: '6031'
abstract:
- lang: eng
text: We introduce Clover, a new library for efficient computation using low-precision
data, providing mathematical routines required by fundamental methods in optimization
and sparse recovery. Our library faithfully implements variants of stochastic
quantization that guarantee convergence at low precision, and supports data formats
from 4-bit quantized to 32-bit IEEE-754 on current Intel processors. In particular,
we show that 4-bit can be implemented efficiently using Intel AVX despite the
lack of native support for this data format. Experimental results with dot product,
matrix-vector multiplication (MVM), gradient descent (GD), and iterative hard
thresholding (IHT) demonstrate that the attainable speedups are in many cases
close to linear with respect to the reduction of precision due to reduced data
movement. Finally, for GD and IHT, we show examples of absolute speedup achieved
by 4-bit versus 32-bit, by iterating until a given target error is achieved.
article_number: '8598402'
article_processing_charge: No
author:
- first_name: Alen
full_name: Stojanov, Alen
last_name: Stojanov
- first_name: Tyler Michael
full_name: Smith, Tyler Michael
last_name: Smith
- first_name: Dan-Adrian
full_name: Alistarh, Dan-Adrian
id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
last_name: Alistarh
orcid: 0000-0003-3650-940X
- first_name: Markus
full_name: Puschel, Markus
last_name: Puschel
citation:
ama: 'Stojanov A, Smith TM, Alistarh D-A, Puschel M. Fast quantized arithmetic on
x86: Trading compute for data movement. In: 2018 IEEE International Workshop
on Signal Processing Systems. Vol 2018-October. IEEE; 2018. doi:10.1109/SiPS.2018.8598402'
apa: 'Stojanov, A., Smith, T. M., Alistarh, D.-A., & Puschel, M. (2018). Fast
quantized arithmetic on x86: Trading compute for data movement. In 2018 IEEE
International Workshop on Signal Processing Systems (Vol. 2018–October). Cape
Town, South Africa: IEEE. https://doi.org/10.1109/SiPS.2018.8598402'
chicago: 'Stojanov, Alen, Tyler Michael Smith, Dan-Adrian Alistarh, and Markus Puschel.
“Fast Quantized Arithmetic on X86: Trading Compute for Data Movement.” In 2018
IEEE International Workshop on Signal Processing Systems, Vol. 2018–October.
IEEE, 2018. https://doi.org/10.1109/SiPS.2018.8598402.'
ieee: 'A. Stojanov, T. M. Smith, D.-A. Alistarh, and M. Puschel, “Fast quantized
arithmetic on x86: Trading compute for data movement,” in 2018 IEEE International
Workshop on Signal Processing Systems, Cape Town, South Africa, 2018, vol.
2018–October.'
ista: 'Stojanov A, Smith TM, Alistarh D-A, Puschel M. 2018. Fast quantized arithmetic
on x86: Trading compute for data movement. 2018 IEEE International Workshop on
Signal Processing Systems. SiPS: Workshop on Signal Processing Systems vol. 2018–October,
8598402.'
mla: 'Stojanov, Alen, et al. “Fast Quantized Arithmetic on X86: Trading Compute
for Data Movement.” 2018 IEEE International Workshop on Signal Processing Systems,
vol. 2018–October, 8598402, IEEE, 2018, doi:10.1109/SiPS.2018.8598402.'
short: A. Stojanov, T.M. Smith, D.-A. Alistarh, M. Puschel, in:, 2018 IEEE International
Workshop on Signal Processing Systems, IEEE, 2018.
conference:
end_date: 2018-10-24
location: Cape Town, South Africa
name: 'SiPS: Workshop on Signal Processing Systems'
start_date: 2018-10-21
date_created: 2019-02-17T22:59:25Z
date_published: 2018-12-31T00:00:00Z
date_updated: 2023-09-19T14:41:51Z
day: '31'
department:
- _id: DaAl
doi: 10.1109/SiPS.2018.8598402
external_id:
isi:
- '000465106800060'
isi: 1
language:
- iso: eng
month: '12'
oa_version: None
publication: 2018 IEEE International Workshop on Signal Processing Systems
publication_status: published
publisher: IEEE
quality_controlled: '1'
scopus_import: '1'
status: public
title: 'Fast quantized arithmetic on x86: Trading compute for data movement'
type: conference
user_id: c635000d-4b10-11ee-a964-aac5a93f6ac1
volume: 2018-October
year: '2018'
...
---
_id: '6558'
abstract:
- lang: eng
text: This paper studies the problem of distributed stochastic optimization in an
adversarial setting where, out of m machines which allegedly compute stochastic
gradients every iteration, an α-fraction are Byzantine, and may behave adversarially.
Our main result is a variant of stochastic gradient descent (SGD) which finds
ε-approximate minimizers of convex functions in T=O~(1/ε²m+α²/ε²) iterations.
In contrast, traditional mini-batch SGD needs T=O(1/ε²m) iterations, but cannot
tolerate Byzantine failures. Further, we provide a lower bound showing that, up
to logarithmic factors, our algorithm is information-theoretically optimal both
in terms of sample complexity and time complexity.
article_processing_charge: No
arxiv: 1
author:
- first_name: Dan-Adrian
full_name: Alistarh, Dan-Adrian
id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
last_name: Alistarh
orcid: 0000-0003-3650-940X
- first_name: Zeyuan
full_name: Allen-Zhu, Zeyuan
last_name: Allen-Zhu
- first_name: Jerry
full_name: Li, Jerry
last_name: Li
citation:
ama: 'Alistarh D-A, Allen-Zhu Z, Li J. Byzantine stochastic gradient descent. In:
Advances in Neural Information Processing Systems. Vol 2018. Neural Information
Processing Systems Foundation; 2018:4613-4623.'
apa: 'Alistarh, D.-A., Allen-Zhu, Z., & Li, J. (2018). Byzantine stochastic
gradient descent. In Advances in Neural Information Processing Systems
(Vol. 2018, pp. 4613–4623). Montreal, Canada: Neural Information Processing Systems
Foundation.'
chicago: Alistarh, Dan-Adrian, Zeyuan Allen-Zhu, and Jerry Li. “Byzantine Stochastic
Gradient Descent.” In Advances in Neural Information Processing Systems,
2018:4613–23. Neural Information Processing Systems Foundation, 2018.
ieee: D.-A. Alistarh, Z. Allen-Zhu, and J. Li, “Byzantine stochastic gradient descent,”
in Advances in Neural Information Processing Systems, Montreal, Canada,
2018, vol. 2018, pp. 4613–4623.
ista: 'Alistarh D-A, Allen-Zhu Z, Li J. 2018. Byzantine stochastic gradient descent.
Advances in Neural Information Processing Systems. NeurIPS: Conference on Neural
Information Processing Systems vol. 2018, 4613–4623.'
mla: Alistarh, Dan-Adrian, et al. “Byzantine Stochastic Gradient Descent.” Advances
in Neural Information Processing Systems, vol. 2018, Neural Information Processing
Systems Foundation, 2018, pp. 4613–23.
short: D.-A. Alistarh, Z. Allen-Zhu, J. Li, in:, Advances in Neural Information
Processing Systems, Neural Information Processing Systems Foundation, 2018, pp.
4613–4623.
conference:
end_date: 2018-12-08
location: Montreal, Canada
name: 'NeurIPS: Conference on Neural Information Processing Systems'
start_date: 2018-12-02
date_created: 2019-06-13T08:22:37Z
date_published: 2018-12-01T00:00:00Z
date_updated: 2023-09-19T15:12:45Z
day: '01'
department:
- _id: DaAl
external_id:
arxiv:
- '1803.08917'
isi:
- '000461823304061'
intvolume: ' 2018'
isi: 1
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://arxiv.org/abs/1803.08917
month: '12'
oa: 1
oa_version: Published Version
page: 4613-4623
publication: Advances in Neural Information Processing Systems
publication_status: published
publisher: Neural Information Processing Systems Foundation
quality_controlled: '1'
scopus_import: '1'
status: public
title: Byzantine stochastic gradient descent
type: conference
user_id: c635000d-4b10-11ee-a964-aac5a93f6ac1
volume: 2018
year: '2018'
...
---
_id: '6589'
abstract:
- lang: eng
text: Distributed training of massive machine learning models, in particular deep
neural networks, via Stochastic Gradient Descent (SGD) is becoming commonplace.
Several families of communication-reduction methods, such as quantization, large-batch
methods, and gradient sparsification, have been proposed. To date, gradient sparsification
methods--where each node sorts gradients by magnitude, and only communicates a
subset of the components, accumulating the rest locally--are known to yield some
of the largest practical gains. Such methods can reduce the amount of communication
per step by up to \emph{three orders of magnitude}, while preserving model accuracy.
Yet, this family of methods currently has no theoretical justification. This is
the question we address in this paper. We prove that, under analytic assumptions,
sparsifying gradients by magnitude with local error correction provides convergence
guarantees, for both convex and non-convex smooth objectives, for data-parallel
SGD. The main insight is that sparsification methods implicitly maintain bounds
on the maximum impact of stale updates, thanks to selection by magnitude. Our
analysis and empirical validation also reveal that these methods do require analytical
conditions to converge well, justifying existing heuristics.
article_processing_charge: No
arxiv: 1
author:
- first_name: Dan-Adrian
full_name: Alistarh, Dan-Adrian
id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
last_name: Alistarh
orcid: 0000-0003-3650-940X
- first_name: Torsten
full_name: Hoefler, Torsten
last_name: Hoefler
- first_name: Mikael
full_name: Johansson, Mikael
last_name: Johansson
- first_name: Nikola H
full_name: Konstantinov, Nikola H
id: 4B9D76E4-F248-11E8-B48F-1D18A9856A87
last_name: Konstantinov
- first_name: Sarit
full_name: Khirirat, Sarit
last_name: Khirirat
- first_name: Cedric
full_name: Renggli, Cedric
last_name: Renggli
citation:
ama: 'Alistarh D-A, Hoefler T, Johansson M, Konstantinov NH, Khirirat S, Renggli
C. The convergence of sparsified gradient methods. In: Advances in Neural Information
Processing Systems 31. Vol Volume 2018. Neural Information Processing Systems
Foundation; 2018:5973-5983.'
apa: 'Alistarh, D.-A., Hoefler, T., Johansson, M., Konstantinov, N. H., Khirirat,
S., & Renggli, C. (2018). The convergence of sparsified gradient methods.
In Advances in Neural Information Processing Systems 31 (Vol. Volume 2018,
pp. 5973–5983). Montreal, Canada: Neural Information Processing Systems Foundation.'
chicago: Alistarh, Dan-Adrian, Torsten Hoefler, Mikael Johansson, Nikola H Konstantinov,
Sarit Khirirat, and Cedric Renggli. “The Convergence of Sparsified Gradient Methods.”
In Advances in Neural Information Processing Systems 31, Volume 2018:5973–83.
Neural Information Processing Systems Foundation, 2018.
ieee: D.-A. Alistarh, T. Hoefler, M. Johansson, N. H. Konstantinov, S. Khirirat,
and C. Renggli, “The convergence of sparsified gradient methods,” in Advances
in Neural Information Processing Systems 31, Montreal, Canada, 2018, vol.
Volume 2018, pp. 5973–5983.
ista: 'Alistarh D-A, Hoefler T, Johansson M, Konstantinov NH, Khirirat S, Renggli
C. 2018. The convergence of sparsified gradient methods. Advances in Neural Information
Processing Systems 31. NeurIPS: Conference on Neural Information Processing Systems
vol. Volume 2018, 5973–5983.'
mla: Alistarh, Dan-Adrian, et al. “The Convergence of Sparsified Gradient Methods.”
Advances in Neural Information Processing Systems 31, vol. Volume 2018,
Neural Information Processing Systems Foundation, 2018, pp. 5973–83.
short: D.-A. Alistarh, T. Hoefler, M. Johansson, N.H. Konstantinov, S. Khirirat,
C. Renggli, in:, Advances in Neural Information Processing Systems 31, Neural
Information Processing Systems Foundation, 2018, pp. 5973–5983.
conference:
end_date: 2018-12-08
location: Montreal, Canada
name: 'NeurIPS: Conference on Neural Information Processing Systems'
start_date: 2018-12-02
corr_author: '1'
date_created: 2019-06-27T09:32:55Z
date_published: 2018-12-01T00:00:00Z
date_updated: 2025-06-26T12:23:06Z
day: '01'
department:
- _id: DaAl
- _id: ChLa
ec_funded: 1
external_id:
arxiv:
- '1809.10505'
isi:
- '000461852000047'
isi: 1
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://arxiv.org/abs/1809.10505
month: '12'
oa: 1
oa_version: Preprint
page: 5973-5983
project:
- _id: 2564DBCA-B435-11E9-9278-68D0E5697425
call_identifier: H2020
grant_number: '665385'
name: International IST Doctoral Program
publication: Advances in Neural Information Processing Systems 31
publication_status: published
publisher: Neural Information Processing Systems Foundation
quality_controlled: '1'
scopus_import: '1'
status: public
title: The convergence of sparsified gradient methods
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: Volume 2018
year: '2018'
...
---
_id: '7116'
abstract:
- lang: eng
text: 'Training deep learning models has received tremendous research interest recently.
In particular, there has been intensive research on reducing the communication
cost of training when using multiple computational devices, through reducing the
precision of the underlying data representation. Naturally, such methods induce
system trade-offs—lowering communication precision could de-crease communication
overheads and improve scalability; but, on the other hand, it can also reduce
the accuracy of training. In this paper, we study this trade-off space, and ask:Can
low-precision communication consistently improve the end-to-end performance of
training modern neural networks, with no accuracy loss?From the performance point
of view, the answer to this question may appear deceptively easy: compressing
communication through low precision should help when the ratio between communication
and computation is high. However, this answer is less straightforward when we
try to generalize this principle across various neural network architectures (e.g.,
AlexNet vs. ResNet),number of GPUs (e.g., 2 vs. 8 GPUs), machine configurations(e.g.,
EC2 instances vs. NVIDIA DGX-1), communication primitives (e.g., MPI vs. NCCL),
and even different GPU architectures(e.g., Kepler vs. Pascal). Currently, it is
not clear how a realistic realization of all these factors maps to the speed up
provided by low-precision communication. In this paper, we conduct an empirical
study to answer this question and report the insights.'
article_processing_charge: No
author:
- first_name: Demjan
full_name: Grubic, Demjan
last_name: Grubic
- first_name: Leo
full_name: Tam, Leo
last_name: Tam
- first_name: Dan-Adrian
full_name: Alistarh, Dan-Adrian
id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
last_name: Alistarh
orcid: 0000-0003-3650-940X
- first_name: Ce
full_name: Zhang, Ce
last_name: Zhang
citation:
ama: 'Grubic D, Tam L, Alistarh D-A, Zhang C. Synchronous multi-GPU training for
deep learning with low-precision communications: An empirical study. In: Proceedings
of the 21st International Conference on Extending Database Technology. OpenProceedings;
2018:145-156. doi:10.5441/002/EDBT.2018.14'
apa: 'Grubic, D., Tam, L., Alistarh, D.-A., & Zhang, C. (2018). Synchronous
multi-GPU training for deep learning with low-precision communications: An empirical
study. In Proceedings of the 21st International Conference on Extending Database
Technology (pp. 145–156). Vienna, Austria: OpenProceedings. https://doi.org/10.5441/002/EDBT.2018.14'
chicago: 'Grubic, Demjan, Leo Tam, Dan-Adrian Alistarh, and Ce Zhang. “Synchronous
Multi-GPU Training for Deep Learning with Low-Precision Communications: An Empirical
Study.” In Proceedings of the 21st International Conference on Extending Database
Technology, 145–56. OpenProceedings, 2018. https://doi.org/10.5441/002/EDBT.2018.14.'
ieee: 'D. Grubic, L. Tam, D.-A. Alistarh, and C. Zhang, “Synchronous multi-GPU training
for deep learning with low-precision communications: An empirical study,” in Proceedings
of the 21st International Conference on Extending Database Technology, Vienna,
Austria, 2018, pp. 145–156.'
ista: 'Grubic D, Tam L, Alistarh D-A, Zhang C. 2018. Synchronous multi-GPU training
for deep learning with low-precision communications: An empirical study. Proceedings
of the 21st International Conference on Extending Database Technology. EDBT: Conference
on Extending Database Technology, 145–156.'
mla: 'Grubic, Demjan, et al. “Synchronous Multi-GPU Training for Deep Learning with
Low-Precision Communications: An Empirical Study.” Proceedings of the 21st
International Conference on Extending Database Technology, OpenProceedings,
2018, pp. 145–56, doi:10.5441/002/EDBT.2018.14.'
short: D. Grubic, L. Tam, D.-A. Alistarh, C. Zhang, in:, Proceedings of the 21st
International Conference on Extending Database Technology, OpenProceedings, 2018,
pp. 145–156.
conference:
end_date: 2018-03-29
location: Vienna, Austria
name: 'EDBT: Conference on Extending Database Technology'
start_date: 2018-03-26
corr_author: '1'
date_created: 2019-11-26T14:19:11Z
date_published: 2018-03-26T00:00:00Z
date_updated: 2024-10-09T20:59:05Z
day: '26'
ddc:
- '000'
department:
- _id: DaAl
doi: 10.5441/002/EDBT.2018.14
file:
- access_level: open_access
checksum: ec979b56abc71016d6e6adfdadbb4afe
content_type: application/pdf
creator: dernst
date_created: 2019-11-26T14:23:04Z
date_updated: 2020-07-14T12:47:49Z
file_id: '7118'
file_name: 2018_OpenProceedings_Grubic.pdf
file_size: 1603204
relation: main_file
file_date_updated: 2020-07-14T12:47:49Z
has_accepted_license: '1'
language:
- iso: eng
license: https://creativecommons.org/licenses/by-nc-nd/4.0/
month: '03'
oa: 1
oa_version: Published Version
page: 145-156
publication: Proceedings of the 21st International Conference on Extending Database
Technology
publication_identifier:
isbn:
- '9783893180783'
issn:
- 2367-2005
publication_status: published
publisher: OpenProceedings
quality_controlled: '1'
scopus_import: 1
status: public
title: 'Synchronous multi-GPU training for deep learning with low-precision communications:
An empirical study'
tmp:
image: /images/cc_by_nc_nd.png
legal_code_url: https://creativecommons.org/licenses/by-nc-nd/4.0/legalcode
name: Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International
(CC BY-NC-ND 4.0)
short: CC BY-NC-ND (4.0)
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
year: '2018'
...
---
_id: '7123'
abstract:
- lang: eng
text: "Population protocols are a popular model of distributed computing, in which
n agents with limited local state interact randomly, and cooperate to collectively
compute global predicates. Inspired by recent developments in DNA programming,
an extensive series of papers, across different communities, has examined the
computability and complexity characteristics of this model. Majority, or consensus,
is a central task in this model, in which agents need to collectively reach a
decision as to which one of two states A or B had a higher initial count. Two
metrics are important: the time that a protocol requires to stabilize to an output
decision, and the state space size that each agent requires to do so. It is known
that majority requires Ω(log log n) states per agent to allow for fast (poly-logarithmic
time) stabilization, and that O(log2 n) states are sufficient. Thus, there is
an exponential gap between the space upper and lower bounds for this problem.
This paper addresses this question.\r\n\r\nOn the negative side, we provide a
new lower bound of Ω(log n) states for any protocol which stabilizes in O(n1–c)
expected time, for any constant c > 0. This result is conditional on monotonicity
and output assumptions, satisfied by all known protocols. Technically, it represents
a departure from previous lower bounds, in that it does not rely on the existence
of dense configurations. Instead, we introduce a new generalized surgery technique
to prove the existence of incorrect executions for any algorithm which would contradict
the lower bound. Subsequently, our lower bound also applies to general initial
configurations, including ones with a leader. On the positive side, we give a
new algorithm for majority which uses O(log n) states, and stabilizes in O(log2
n) expected time. Central to the algorithm is a new leaderless phase clock technique,
which allows agents to synchronize in phases of Θ(n log n) consecutive interactions
using O(log n) states per agent, exploiting a new connection between population
protocols and power-of-two-choices load balancing mechanisms. We also employ our
phase clock to build a leader election algorithm with a state space of size O(log
n), which stabilizes in O(log2 n) expected time."
article_processing_charge: No
arxiv: 1
author:
- first_name: Dan-Adrian
full_name: Alistarh, Dan-Adrian
id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
last_name: Alistarh
orcid: 0000-0003-3650-940X
- first_name: James
full_name: Aspnes, James
last_name: Aspnes
- first_name: Rati
full_name: Gelashvili, Rati
last_name: Gelashvili
citation:
ama: 'Alistarh D-A, Aspnes J, Gelashvili R. Space-optimal majority in population
protocols. In: Proceedings of the 29th Annual ACM-SIAM Symposium on Discrete
Algorithms. ACM; 2018:2221-2239. doi:10.1137/1.9781611975031.144'
apa: 'Alistarh, D.-A., Aspnes, J., & Gelashvili, R. (2018). Space-optimal majority
in population protocols. In Proceedings of the 29th Annual ACM-SIAM Symposium
on Discrete Algorithms (pp. 2221–2239). New Orleans, LA, United States: ACM.
https://doi.org/10.1137/1.9781611975031.144'
chicago: Alistarh, Dan-Adrian, James Aspnes, and Rati Gelashvili. “Space-Optimal
Majority in Population Protocols.” In Proceedings of the 29th Annual ACM-SIAM
Symposium on Discrete Algorithms, 2221–39. ACM, 2018. https://doi.org/10.1137/1.9781611975031.144.
ieee: D.-A. Alistarh, J. Aspnes, and R. Gelashvili, “Space-optimal majority in population
protocols,” in Proceedings of the 29th Annual ACM-SIAM Symposium on Discrete
Algorithms, New Orleans, LA, United States, 2018, pp. 2221–2239.
ista: 'Alistarh D-A, Aspnes J, Gelashvili R. 2018. Space-optimal majority in population
protocols. Proceedings of the 29th Annual ACM-SIAM Symposium on Discrete Algorithms.
SODA: Symposium on Discrete Algorithms, 2221–2239.'
mla: Alistarh, Dan-Adrian, et al. “Space-Optimal Majority in Population Protocols.”
Proceedings of the 29th Annual ACM-SIAM Symposium on Discrete Algorithms,
ACM, 2018, pp. 2221–39, doi:10.1137/1.9781611975031.144.
short: D.-A. Alistarh, J. Aspnes, R. Gelashvili, in:, Proceedings of the 29th Annual
ACM-SIAM Symposium on Discrete Algorithms, ACM, 2018, pp. 2221–2239.
conference:
end_date: 2018-01-10
location: New Orleans, LA, United States
name: 'SODA: Symposium on Discrete Algorithms'
start_date: 2018-01-07
date_created: 2019-11-26T15:10:55Z
date_published: 2018-01-30T00:00:00Z
date_updated: 2024-10-21T06:02:41Z
day: '30'
department:
- _id: DaAl
doi: 10.1137/1.9781611975031.144
external_id:
arxiv:
- '1704.04947'
isi:
- '000483921200145'
isi: 1
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://arxiv.org/abs/1704.04947
month: '01'
oa: 1
oa_version: Preprint
page: 2221-2239
publication: Proceedings of the 29th Annual ACM-SIAM Symposium on Discrete Algorithms
publication_identifier:
isbn:
- '9781611975031'
publication_status: published
publisher: ACM
quality_controlled: '1'
scopus_import: '1'
status: public
title: Space-optimal majority in population protocols
type: conference
user_id: c635000d-4b10-11ee-a964-aac5a93f6ac1
year: '2018'
...
---
_id: '76'
abstract:
- lang: eng
text: 'Consider a fully-connected synchronous distributed system consisting of n
nodes, where up to f nodes may be faulty and every node starts in an arbitrary
initial state. In the synchronous C-counting problem, all nodes need to eventually
agree on a counter that is increased by one modulo C in each round for given C>1.
In the self-stabilising firing squad problem, the task is to eventually guarantee
that all non-faulty nodes have simultaneous responses to external inputs: if a
subset of the correct nodes receive an external “go” signal as input, then all
correct nodes should agree on a round (in the not-too-distant future) in which
to jointly output a “fire” signal. Moreover, no node should generate a “fire”
signal without some correct node having previously received a “go” signal as input.
We present a framework reducing both tasks to binary consensus at very small cost.
For example, we obtain a deterministic algorithm for self-stabilising Byzantine
firing squads with optimal resilience f<n/3, asymptotically optimal stabilisation
and response time O(f), and message size O(log f). As our framework does not restrict
the type of consensus routines used, we also obtain efficient randomised solutions.'
article_processing_charge: Yes (via OA deal)
author:
- first_name: Christoph
full_name: Lenzen, Christoph
last_name: Lenzen
- first_name: Joel
full_name: Rybicki, Joel
id: 334EFD2E-F248-11E8-B48F-1D18A9856A87
last_name: Rybicki
orcid: 0000-0002-6432-6646
citation:
ama: Lenzen C, Rybicki J. Near-optimal self-stabilising counting and firing squads.
Distributed Computing. 2018. doi:10.1007/s00446-018-0342-6
apa: Lenzen, C., & Rybicki, J. (2018). Near-optimal self-stabilising counting
and firing squads. Distributed Computing. Springer. https://doi.org/10.1007/s00446-018-0342-6
chicago: Lenzen, Christoph, and Joel Rybicki. “Near-Optimal Self-Stabilising Counting
and Firing Squads.” Distributed Computing. Springer, 2018. https://doi.org/10.1007/s00446-018-0342-6.
ieee: C. Lenzen and J. Rybicki, “Near-optimal self-stabilising counting and firing
squads,” Distributed Computing. Springer, 2018.
ista: Lenzen C, Rybicki J. 2018. Near-optimal self-stabilising counting and firing
squads. Distributed Computing.
mla: Lenzen, Christoph, and Joel Rybicki. “Near-Optimal Self-Stabilising Counting
and Firing Squads.” Distributed Computing, Springer, 2018, doi:10.1007/s00446-018-0342-6.
short: C. Lenzen, J. Rybicki, Distributed Computing (2018).
corr_author: '1'
date_created: 2018-12-11T11:44:30Z
date_published: 2018-09-12T00:00:00Z
date_updated: 2025-04-15T06:53:15Z
day: '12'
ddc:
- '000'
department:
- _id: DaAl
doi: 10.1007/s00446-018-0342-6
external_id:
isi:
- '000475627800005'
file:
- access_level: open_access
checksum: 872db70bba9b401500abe3c6ae2f1a61
content_type: application/pdf
creator: dernst
date_created: 2018-12-17T14:21:22Z
date_updated: 2020-07-14T12:48:01Z
file_id: '5711'
file_name: 2018_DistributedComputing_Lenzen.pdf
file_size: 799337
relation: main_file
file_date_updated: 2020-07-14T12:48:01Z
has_accepted_license: '1'
isi: 1
language:
- iso: eng
month: '09'
oa: 1
oa_version: Published Version
project:
- _id: B67AFEDC-15C9-11EA-A837-991A96BB2854
name: IST Austria Open Access Fund
publication: Distributed Computing
publication_status: published
publisher: Springer
publist_id: '7978'
quality_controlled: '1'
scopus_import: '1'
status: public
title: Near-optimal self-stabilising counting and firing squads
tmp:
image: /images/cc_by.png
legal_code_url: https://creativecommons.org/licenses/by/4.0/legalcode
name: Creative Commons Attribution 4.0 International Public License (CC-BY 4.0)
short: CC BY (4.0)
type: journal_article
user_id: c635000d-4b10-11ee-a964-aac5a93f6ac1
year: '2018'
...