---
_id: '6031'
abstract:
- lang: eng
text: We introduce Clover, a new library for efficient computation using low-precision
data, providing mathematical routines required by fundamental methods in optimization
and sparse recovery. Our library faithfully implements variants of stochastic
quantization that guarantee convergence at low precision, and supports data formats
from 4-bit quantized to 32-bit IEEE-754 on current Intel processors. In particular,
we show that 4-bit can be implemented efficiently using Intel AVX despite the
lack of native support for this data format. Experimental results with dot product,
matrix-vector multiplication (MVM), gradient descent (GD), and iterative hard
thresholding (IHT) demonstrate that the attainable speedups are in many cases
close to linear with respect to the reduction of precision due to reduced data
movement. Finally, for GD and IHT, we show examples of absolute speedup achieved
by 4-bit versus 32-bit, by iterating until a given target error is achieved.
article_number: '8598402'
article_processing_charge: No
author:
- first_name: Alen
full_name: Stojanov, Alen
last_name: Stojanov
- first_name: Tyler Michael
full_name: Smith, Tyler Michael
last_name: Smith
- first_name: Dan-Adrian
full_name: Alistarh, Dan-Adrian
id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
last_name: Alistarh
orcid: 0000-0003-3650-940X
- first_name: Markus
full_name: Puschel, Markus
last_name: Puschel
citation:
ama: 'Stojanov A, Smith TM, Alistarh D-A, Puschel M. Fast quantized arithmetic on
x86: Trading compute for data movement. In: 2018 IEEE International Workshop
on Signal Processing Systems. Vol 2018-October. IEEE; 2018. doi:10.1109/SiPS.2018.8598402'
apa: 'Stojanov, A., Smith, T. M., Alistarh, D.-A., & Puschel, M. (2018). Fast
quantized arithmetic on x86: Trading compute for data movement. In 2018 IEEE
International Workshop on Signal Processing Systems (Vol. 2018–October). Cape
Town, South Africa: IEEE. https://doi.org/10.1109/SiPS.2018.8598402'
chicago: 'Stojanov, Alen, Tyler Michael Smith, Dan-Adrian Alistarh, and Markus Puschel.
“Fast Quantized Arithmetic on X86: Trading Compute for Data Movement.” In 2018
IEEE International Workshop on Signal Processing Systems, Vol. 2018–October.
IEEE, 2018. https://doi.org/10.1109/SiPS.2018.8598402.'
ieee: 'A. Stojanov, T. M. Smith, D.-A. Alistarh, and M. Puschel, “Fast quantized
arithmetic on x86: Trading compute for data movement,” in 2018 IEEE International
Workshop on Signal Processing Systems, Cape Town, South Africa, 2018, vol.
2018–October.'
ista: 'Stojanov A, Smith TM, Alistarh D-A, Puschel M. 2018. Fast quantized arithmetic
on x86: Trading compute for data movement. 2018 IEEE International Workshop on
Signal Processing Systems. SiPS: Workshop on Signal Processing Systems vol. 2018–October,
8598402.'
mla: 'Stojanov, Alen, et al. “Fast Quantized Arithmetic on X86: Trading Compute
for Data Movement.” 2018 IEEE International Workshop on Signal Processing Systems,
vol. 2018–October, 8598402, IEEE, 2018, doi:10.1109/SiPS.2018.8598402.'
short: A. Stojanov, T.M. Smith, D.-A. Alistarh, M. Puschel, in:, 2018 IEEE International
Workshop on Signal Processing Systems, IEEE, 2018.
conference:
end_date: 2018-10-24
location: Cape Town, South Africa
name: 'SiPS: Workshop on Signal Processing Systems'
start_date: 2018-10-21
date_created: 2019-02-17T22:59:25Z
date_published: 2018-12-31T00:00:00Z
date_updated: 2023-09-19T14:41:51Z
day: '31'
department:
- _id: DaAl
doi: 10.1109/SiPS.2018.8598402
external_id:
isi:
- '000465106800060'
fulldoi: https://doi.org/10.1109/SiPS.2018.8598402
isi: 1
language:
- iso: eng
month: '12'
oa_version: None
publication: 2018 IEEE International Workshop on Signal Processing Systems
publication_status: published
publisher: IEEE
quality_controlled: '1'
scopus_import: '1'
status: public
title: 'Fast quantized arithmetic on x86: Trading compute for data movement'
type: conference
user_id: c635000d-4b10-11ee-a964-aac5a93f6ac1
volume: 2018-October
year: '2018'
...
---
_id: '76'
abstract:
- lang: eng
text: 'Consider a fully-connected synchronous distributed system consisting of n
nodes, where up to f nodes may be faulty and every node starts in an arbitrary
initial state. In the synchronous C-counting problem, all nodes need to eventually
agree on a counter that is increased by one modulo C in each round for given C>1.
In the self-stabilising firing squad problem, the task is to eventually guarantee
that all non-faulty nodes have simultaneous responses to external inputs: if a
subset of the correct nodes receive an external “go” signal as input, then all
correct nodes should agree on a round (in the not-too-distant future) in which
to jointly output a “fire” signal. Moreover, no node should generate a “fire”
signal without some correct node having previously received a “go” signal as input.
We present a framework reducing both tasks to binary consensus at very small cost.
For example, we obtain a deterministic algorithm for self-stabilising Byzantine
firing squads with optimal resilience f<n/3, asymptotically optimal stabilisation
and response time O(f), and message size O(log f). As our framework does not restrict
the type of consensus routines used, we also obtain efficient randomised solutions.'
article_processing_charge: Yes (via OA deal)
author:
- first_name: Christoph
full_name: Lenzen, Christoph
last_name: Lenzen
- first_name: Joel
full_name: Rybicki, Joel
id: 334EFD2E-F248-11E8-B48F-1D18A9856A87
last_name: Rybicki
orcid: 0000-0002-6432-6646
citation:
ama: Lenzen C, Rybicki J. Near-optimal self-stabilising counting and firing squads.
Distributed Computing. 2018. doi:10.1007/s00446-018-0342-6
apa: Lenzen, C., & Rybicki, J. (2018). Near-optimal self-stabilising counting
and firing squads. Distributed Computing. Springer. https://doi.org/10.1007/s00446-018-0342-6
chicago: Lenzen, Christoph, and Joel Rybicki. “Near-Optimal Self-Stabilising Counting
and Firing Squads.” Distributed Computing. Springer, 2018. https://doi.org/10.1007/s00446-018-0342-6.
ieee: C. Lenzen and J. Rybicki, “Near-optimal self-stabilising counting and firing
squads,” Distributed Computing. Springer, 2018.
ista: Lenzen C, Rybicki J. 2018. Near-optimal self-stabilising counting and firing
squads. Distributed Computing.
mla: Lenzen, Christoph, and Joel Rybicki. “Near-Optimal Self-Stabilising Counting
and Firing Squads.” Distributed Computing, Springer, 2018, doi:10.1007/s00446-018-0342-6.
short: C. Lenzen, J. Rybicki, Distributed Computing (2018).
corr_author: '1'
date_created: 2018-12-11T11:44:30Z
date_published: 2018-09-12T00:00:00Z
date_updated: 2025-04-15T06:53:15Z
day: '12'
ddc:
- '000'
department:
- _id: DaAl
doi: 10.1007/s00446-018-0342-6
external_id:
isi:
- '000475627800005'
file:
- access_level: open_access
checksum: 872db70bba9b401500abe3c6ae2f1a61
content_type: application/pdf
creator: dernst
date_created: 2018-12-17T14:21:22Z
date_updated: 2020-07-14T12:48:01Z
file_id: '5711'
file_name: 2018_DistributedComputing_Lenzen.pdf
file_size: 799337
relation: main_file
file_date_updated: 2020-07-14T12:48:01Z
fulldoi: https://doi.org/10.1007/s00446-018-0342-6
has_accepted_license: '1'
isi: 1
language:
- iso: eng
month: '09'
oa: 1
oa_version: Published Version
project:
- _id: B67AFEDC-15C9-11EA-A837-991A96BB2854
name: IST Austria Open Access Fund
publication: Distributed Computing
publication_status: published
publisher: Springer
publist_id: '7978'
quality_controlled: '1'
scopus_import: '1'
status: public
title: Near-optimal self-stabilising counting and firing squads
tmp:
image: /images/cc_by.png
legal_code_url: https://creativecommons.org/licenses/by/4.0/legalcode
name: Creative Commons Attribution 4.0 International Public License (CC-BY 4.0)
short: CC BY (4.0)
type: journal_article
user_id: c635000d-4b10-11ee-a964-aac5a93f6ac1
year: '2018'
...
---
_id: '7812'
abstract:
- lang: eng
text: Deep neural networks (DNNs) continue to make significant advances, solving
tasks from image classification to translation or reinforcement learning. One
aspect of the field receiving considerable attention is efficiently executing
deep models in resource-constrained environments, such as mobile or embedded devices.
This paper focuses on this problem, and proposes two new compression methods,
which jointly leverage weight quantization and distillation of larger teacher
networks into smaller student networks. The first method we propose is called
quantized distillation and leverages distillation during the training process,
by incorporating distillation loss, expressed with respect to the teacher, into
the training of a student network whose weights are quantized to a limited set
of levels. The second method, differentiable quantization, optimizes the location
of quantization points through stochastic gradient descent, to better fit the
behavior of the teacher model. We validate both methods through experiments on
convolutional and recurrent architectures. We show that quantized shallow students
can reach similar accuracy levels to full-precision teacher models, while providing
order of magnitude compression, and inference speedup that is linear in the depth
reduction. In sum, our results enable DNNs for resource-constrained environments
to leverage architecture and accuracy advances developed on more powerful devices.
acknowledgement: "We would like to thank Ce Zhang (ETH Zurich), Hantian Zhang (ETH
Zurich) and Martin Jaggi ´\r\n(EPFL) for their support with experiments and valuable
feedback.\r\n"
article_processing_charge: No
arxiv: 1
author:
- first_name: Antonio
full_name: Polino, Antonio
last_name: Polino
- first_name: Razvan
full_name: Pascanu, Razvan
last_name: Pascanu
- first_name: Dan-Adrian
full_name: Alistarh, Dan-Adrian
id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
last_name: Alistarh
orcid: 0000-0003-3650-940X
citation:
ama: 'Polino A, Pascanu R, Alistarh D-A. Model compression via distillation and
quantization. In: 6th International Conference on Learning Representations.
; 2018.'
apa: Polino, A., Pascanu, R., & Alistarh, D.-A. (2018). Model compression via
distillation and quantization. In 6th International Conference on Learning
Representations. Vancouver, Canada.
chicago: Polino, Antonio, Razvan Pascanu, and Dan-Adrian Alistarh. “Model Compression
via Distillation and Quantization.” In 6th International Conference on Learning
Representations, 2018.
ieee: A. Polino, R. Pascanu, and D.-A. Alistarh, “Model compression via distillation
and quantization,” in 6th International Conference on Learning Representations,
Vancouver, Canada, 2018.
ista: 'Polino A, Pascanu R, Alistarh D-A. 2018. Model compression via distillation
and quantization. 6th International Conference on Learning Representations. ICLR:
International Conference on Learning Representations.'
mla: Polino, Antonio, et al. “Model Compression via Distillation and Quantization.”
6th International Conference on Learning Representations, 2018.
short: A. Polino, R. Pascanu, D.-A. Alistarh, in:, 6th International Conference
on Learning Representations, 2018.
conference:
end_date: 2018-05-03
location: Vancouver, Canada
name: 'ICLR: International Conference on Learning Representations'
start_date: 2018-04-30
date_created: 2020-05-10T22:00:51Z
date_published: 2018-05-01T00:00:00Z
date_updated: 2025-06-30T10:04:44Z
day: '01'
ddc:
- '000'
department:
- _id: DaAl
external_id:
arxiv:
- '1802.05668'
file:
- access_level: open_access
checksum: a4336c167978e81891970e4e4517a8c3
content_type: application/pdf
creator: dernst
date_created: 2020-05-26T13:02:00Z
date_updated: 2020-07-14T12:48:03Z
file_id: '7894'
file_name: 2018_ICLR_Polino.pdf
file_size: 308339
relation: main_file
file_date_updated: 2020-07-14T12:48:03Z
has_accepted_license: '1'
language:
- iso: eng
month: '05'
oa: 1
oa_version: Published Version
publication: 6th International Conference on Learning Representations
publication_status: published
quality_controlled: '1'
scopus_import: '1'
status: public
title: Model compression via distillation and quantization
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
year: '2018'
...
---
_id: '85'
abstract:
- lang: eng
text: Concurrent accesses to shared data structures must be synchronized to avoid
data races. Coarse-grained synchronization, which locks the entire data structure,
is easy to implement but does not scale. Fine-grained synchronization can scale
well, but can be hard to reason about. Hand-over-hand locking, in which operations
are pipelined as they traverse the data structure, combines fine-grained synchronization
with ease of use. However, the traditional implementation suffers from inherent
overheads. This paper introduces snapshot-based synchronization (SBS), a novel
hand-over-hand locking mechanism. SBS decouples the synchronization state from
the data, significantly improving cache utilization. Further, it relies on guarantees
provided by pipelining to minimize synchronization that requires cross-thread
communication. Snapshot-based synchronization thus scales much better than traditional
hand-over-hand locking, while maintaining the same ease of use.
acknowledgement: Trevor Brown was supported in part by the ISF (grants 2005/17 & 1749/14)
and by a NSERC post-doctoral fellowship.
alternative_title:
- LNCS
article_processing_charge: No
author:
- first_name: Eran
full_name: Gilad, Eran
last_name: Gilad
- first_name: Trevor A
full_name: Brown, Trevor A
id: 3569F0A0-F248-11E8-B48F-1D18A9856A87
last_name: Brown
- first_name: Mark
full_name: Oskin, Mark
last_name: Oskin
- first_name: Yoav
full_name: Etsion, Yoav
last_name: Etsion
citation:
ama: 'Gilad E, Brown TA, Oskin M, Etsion Y. Snapshot based synchronization: A fast
replacement for Hand-over-Hand locking. In: Vol 11014. Springer; 2018:465-479.
doi:10.1007/978-3-319-96983-1_33'
apa: 'Gilad, E., Brown, T. A., Oskin, M., & Etsion, Y. (2018). Snapshot based
synchronization: A fast replacement for Hand-over-Hand locking (Vol. 11014, pp.
465–479). Presented at the Euro-Par: European Conference on Parallel Processing,
Turin, Italy: Springer. https://doi.org/10.1007/978-3-319-96983-1_33'
chicago: 'Gilad, Eran, Trevor A Brown, Mark Oskin, and Yoav Etsion. “Snapshot Based
Synchronization: A Fast Replacement for Hand-over-Hand Locking,” 11014:465–79.
Springer, 2018. https://doi.org/10.1007/978-3-319-96983-1_33.'
ieee: 'E. Gilad, T. A. Brown, M. Oskin, and Y. Etsion, “Snapshot based synchronization:
A fast replacement for Hand-over-Hand locking,” presented at the Euro-Par: European
Conference on Parallel Processing, Turin, Italy, 2018, vol. 11014, pp. 465–479.'
ista: 'Gilad E, Brown TA, Oskin M, Etsion Y. 2018. Snapshot based synchronization:
A fast replacement for Hand-over-Hand locking. Euro-Par: European Conference on
Parallel Processing, LNCS, vol. 11014, 465–479.'
mla: 'Gilad, Eran, et al. Snapshot Based Synchronization: A Fast Replacement
for Hand-over-Hand Locking. Vol. 11014, Springer, 2018, pp. 465–79, doi:10.1007/978-3-319-96983-1_33.'
short: E. Gilad, T.A. Brown, M. Oskin, Y. Etsion, in:, Springer, 2018, pp. 465–479.
conference:
end_date: 2018-08-31
location: Turin, Italy
name: 'Euro-Par: European Conference on Parallel Processing'
start_date: 2018-08-27
date_created: 2018-12-11T11:44:33Z
date_published: 2018-08-01T00:00:00Z
date_updated: 2026-04-16T09:53:41Z
day: '01'
ddc:
- '000'
department:
- _id: DaAl
doi: 10.1007/978-3-319-96983-1_33
external_id:
isi:
- '000851042300031'
file:
- access_level: open_access
checksum: 13a3f250be8878405e791b53c19722ad
content_type: application/pdf
creator: dernst
date_created: 2019-02-12T07:40:40Z
date_updated: 2020-07-14T12:48:14Z
file_id: '5954'
file_name: 2018_Brown.pdf
file_size: 665372
relation: main_file
file_date_updated: 2020-07-14T12:48:14Z
fulldoi: https://doi.org/10.1007/978-3-319-96983-1_33
has_accepted_license: '1'
intvolume: ' 11014'
isi: 1
language:
- iso: eng
month: '08'
oa: 1
oa_version: Preprint
page: 465 - 479
project:
- _id: 26450934-B435-11E9-9278-68D0E5697425
name: NSERC Postdoctoral fellowship
publication_identifier:
issn:
- 0302-9743
publication_status: published
publisher: Springer
publist_id: '7969'
quality_controlled: '1'
scopus_import: '1'
status: public
title: 'Snapshot based synchronization: A fast replacement for Hand-over-Hand locking'
type: conference
user_id: ba8df636-2132-11f1-aed0-ed93e2281fdd
volume: 11014
year: '2018'
...
---
_id: '6558'
abstract:
- lang: eng
text: This paper studies the problem of distributed stochastic optimization in an
adversarial setting where, out of m machines which allegedly compute stochastic
gradients every iteration, an α-fraction are Byzantine, and may behave adversarially.
Our main result is a variant of stochastic gradient descent (SGD) which finds
ε-approximate minimizers of convex functions in T=O~(1/ε²m+α²/ε²) iterations.
In contrast, traditional mini-batch SGD needs T=O(1/ε²m) iterations, but cannot
tolerate Byzantine failures. Further, we provide a lower bound showing that, up
to logarithmic factors, our algorithm is information-theoretically optimal both
in terms of sample complexity and time complexity.
article_processing_charge: No
arxiv: 1
author:
- first_name: Dan-Adrian
full_name: Alistarh, Dan-Adrian
id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
last_name: Alistarh
orcid: 0000-0003-3650-940X
- first_name: Zeyuan
full_name: Allen-Zhu, Zeyuan
last_name: Allen-Zhu
- first_name: Jerry
full_name: Li, Jerry
last_name: Li
citation:
ama: 'Alistarh D-A, Allen-Zhu Z, Li J. Byzantine stochastic gradient descent. In:
Advances in Neural Information Processing Systems. Vol 2018. Neural Information
Processing Systems Foundation; 2018:4613-4623.'
apa: 'Alistarh, D.-A., Allen-Zhu, Z., & Li, J. (2018). Byzantine stochastic
gradient descent. In Advances in Neural Information Processing Systems
(Vol. 2018, pp. 4613–4623). Montreal, Canada: Neural Information Processing Systems
Foundation.'
chicago: Alistarh, Dan-Adrian, Zeyuan Allen-Zhu, and Jerry Li. “Byzantine Stochastic
Gradient Descent.” In Advances in Neural Information Processing Systems,
2018:4613–23. Neural Information Processing Systems Foundation, 2018.
ieee: D.-A. Alistarh, Z. Allen-Zhu, and J. Li, “Byzantine stochastic gradient descent,”
in Advances in Neural Information Processing Systems, Montreal, Canada,
2018, vol. 2018, pp. 4613–4623.
ista: 'Alistarh D-A, Allen-Zhu Z, Li J. 2018. Byzantine stochastic gradient descent.
Advances in Neural Information Processing Systems. NeurIPS: Conference on Neural
Information Processing Systems vol. 2018, 4613–4623.'
mla: Alistarh, Dan-Adrian, et al. “Byzantine Stochastic Gradient Descent.” Advances
in Neural Information Processing Systems, vol. 2018, Neural Information Processing
Systems Foundation, 2018, pp. 4613–23.
short: D.-A. Alistarh, Z. Allen-Zhu, J. Li, in:, Advances in Neural Information
Processing Systems, Neural Information Processing Systems Foundation, 2018, pp.
4613–4623.
conference:
end_date: 2018-12-08
location: Montreal, Canada
name: 'NeurIPS: Conference on Neural Information Processing Systems'
start_date: 2018-12-02
date_created: 2019-06-13T08:22:37Z
date_published: 2018-12-01T00:00:00Z
date_updated: 2026-06-18T19:08:25Z
day: '01'
ddc:
- '000'
department:
- _id: DaAl
external_id:
arxiv:
- '1803.08917'
isi:
- '000461823304061'
intvolume: ' 2018'
isi: 1
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://arxiv.org/abs/1803.08917
month: '12'
oa: 1
oa_version: Published Version
page: 4613-4623
publication: Advances in Neural Information Processing Systems
publication_status: published
publisher: Neural Information Processing Systems Foundation
quality_controlled: '1'
scopus_import: '1'
status: public
title: Byzantine stochastic gradient descent
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 2018
year: '2018'
...
---
_id: '7116'
abstract:
- lang: eng
text: 'Training deep learning models has received tremendous research interest recently.
In particular, there has been intensive research on reducing the communication
cost of training when using multiple computational devices, through reducing the
precision of the underlying data representation. Naturally, such methods induce
system trade-offs—lowering communication precision could de-crease communication
overheads and improve scalability; but, on the other hand, it can also reduce
the accuracy of training. In this paper, we study this trade-off space, and ask:Can
low-precision communication consistently improve the end-to-end performance of
training modern neural networks, with no accuracy loss?From the performance point
of view, the answer to this question may appear deceptively easy: compressing
communication through low precision should help when the ratio between communication
and computation is high. However, this answer is less straightforward when we
try to generalize this principle across various neural network architectures (e.g.,
AlexNet vs. ResNet),number of GPUs (e.g., 2 vs. 8 GPUs), machine configurations(e.g.,
EC2 instances vs. NVIDIA DGX-1), communication primitives (e.g., MPI vs. NCCL),
and even different GPU architectures(e.g., Kepler vs. Pascal). Currently, it is
not clear how a realistic realization of all these factors maps to the speed up
provided by low-precision communication. In this paper, we conduct an empirical
study to answer this question and report the insights.'
article_processing_charge: No
author:
- first_name: Demjan
full_name: Grubic, Demjan
last_name: Grubic
- first_name: Leo
full_name: Tam, Leo
last_name: Tam
- first_name: Dan-Adrian
full_name: Alistarh, Dan-Adrian
id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
last_name: Alistarh
orcid: 0000-0003-3650-940X
- first_name: Ce
full_name: Zhang, Ce
last_name: Zhang
citation:
ama: 'Grubic D, Tam L, Alistarh D-A, Zhang C. Synchronous multi-GPU training for
deep learning with low-precision communications: An empirical study. In: Proceedings
of the 21st International Conference on Extending Database Technology. OpenProceedings;
2018:145-156. doi:10.5441/002/EDBT.2018.14'
apa: 'Grubic, D., Tam, L., Alistarh, D.-A., & Zhang, C. (2018). Synchronous
multi-GPU training for deep learning with low-precision communications: An empirical
study. In Proceedings of the 21st International Conference on Extending Database
Technology (pp. 145–156). Vienna, Austria: OpenProceedings. https://doi.org/10.5441/002/EDBT.2018.14'
chicago: 'Grubic, Demjan, Leo Tam, Dan-Adrian Alistarh, and Ce Zhang. “Synchronous
Multi-GPU Training for Deep Learning with Low-Precision Communications: An Empirical
Study.” In Proceedings of the 21st International Conference on Extending Database
Technology, 145–56. OpenProceedings, 2018. https://doi.org/10.5441/002/EDBT.2018.14.'
ieee: 'D. Grubic, L. Tam, D.-A. Alistarh, and C. Zhang, “Synchronous multi-GPU training
for deep learning with low-precision communications: An empirical study,” in Proceedings
of the 21st International Conference on Extending Database Technology, Vienna,
Austria, 2018, pp. 145–156.'
ista: 'Grubic D, Tam L, Alistarh D-A, Zhang C. 2018. Synchronous multi-GPU training
for deep learning with low-precision communications: An empirical study. Proceedings
of the 21st International Conference on Extending Database Technology. EDBT: Conference
on Extending Database Technology, 145–156.'
mla: 'Grubic, Demjan, et al. “Synchronous Multi-GPU Training for Deep Learning with
Low-Precision Communications: An Empirical Study.” Proceedings of the 21st
International Conference on Extending Database Technology, OpenProceedings,
2018, pp. 145–56, doi:10.5441/002/EDBT.2018.14.'
short: D. Grubic, L. Tam, D.-A. Alistarh, C. Zhang, in:, Proceedings of the 21st
International Conference on Extending Database Technology, OpenProceedings, 2018,
pp. 145–156.
conference:
end_date: 2018-03-29
location: Vienna, Austria
name: 'EDBT: Conference on Extending Database Technology'
start_date: 2018-03-26
corr_author: '1'
date_created: 2019-11-26T14:19:11Z
date_published: 2018-03-26T00:00:00Z
date_updated: 2024-10-09T20:59:05Z
day: '26'
ddc:
- '000'
department:
- _id: DaAl
doi: 10.5441/002/EDBT.2018.14
file:
- access_level: open_access
checksum: ec979b56abc71016d6e6adfdadbb4afe
content_type: application/pdf
creator: dernst
date_created: 2019-11-26T14:23:04Z
date_updated: 2020-07-14T12:47:49Z
file_id: '7118'
file_name: 2018_OpenProceedings_Grubic.pdf
file_size: 1603204
relation: main_file
file_date_updated: 2020-07-14T12:47:49Z
fulldoi: https://doi.org/10.5441/002/EDBT.2018.14
has_accepted_license: '1'
language:
- iso: eng
month: '03'
oa: 1
oa_version: Published Version
page: 145-156
publication: Proceedings of the 21st International Conference on Extending Database
Technology
publication_identifier:
isbn:
- '9783893180783'
issn:
- 2367-2005
publication_status: published
publisher: OpenProceedings
quality_controlled: '1'
scopus_import: 1
status: public
title: 'Synchronous multi-GPU training for deep learning with low-precision communications:
An empirical study'
tmp:
image: /images/cc_by_nc_nd.png
legal_code_url: https://creativecommons.org/licenses/by-nc-nd/4.0/legalcode
name: Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International
(CC BY-NC-ND 4.0)
short: CC BY-NC-ND (4.0)
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
year: '2018'
...
---
_id: '7123'
abstract:
- lang: eng
text: "Population protocols are a popular model of distributed computing, in which
n agents with limited local state interact randomly, and cooperate to collectively
compute global predicates. Inspired by recent developments in DNA programming,
an extensive series of papers, across different communities, has examined the
computability and complexity characteristics of this model. Majority, or consensus,
is a central task in this model, in which agents need to collectively reach a
decision as to which one of two states A or B had a higher initial count. Two
metrics are important: the time that a protocol requires to stabilize to an output
decision, and the state space size that each agent requires to do so. It is known
that majority requires Ω(log log n) states per agent to allow for fast (poly-logarithmic
time) stabilization, and that O(log2 n) states are sufficient. Thus, there is
an exponential gap between the space upper and lower bounds for this problem.
This paper addresses this question.\r\n\r\nOn the negative side, we provide a
new lower bound of Ω(log n) states for any protocol which stabilizes in O(n1–c)
expected time, for any constant c > 0. This result is conditional on monotonicity
and output assumptions, satisfied by all known protocols. Technically, it represents
a departure from previous lower bounds, in that it does not rely on the existence
of dense configurations. Instead, we introduce a new generalized surgery technique
to prove the existence of incorrect executions for any algorithm which would contradict
the lower bound. Subsequently, our lower bound also applies to general initial
configurations, including ones with a leader. On the positive side, we give a
new algorithm for majority which uses O(log n) states, and stabilizes in O(log2
n) expected time. Central to the algorithm is a new leaderless phase clock technique,
which allows agents to synchronize in phases of Θ(n log n) consecutive interactions
using O(log n) states per agent, exploiting a new connection between population
protocols and power-of-two-choices load balancing mechanisms. We also employ our
phase clock to build a leader election algorithm with a state space of size O(log
n), which stabilizes in O(log2 n) expected time."
article_processing_charge: No
arxiv: 1
author:
- first_name: Dan-Adrian
full_name: Alistarh, Dan-Adrian
id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
last_name: Alistarh
orcid: 0000-0003-3650-940X
- first_name: James
full_name: Aspnes, James
last_name: Aspnes
- first_name: Rati
full_name: Gelashvili, Rati
last_name: Gelashvili
citation:
ama: 'Alistarh D-A, Aspnes J, Gelashvili R. Space-optimal majority in population
protocols. In: Proceedings of the 29th Annual ACM-SIAM Symposium on Discrete
Algorithms. ACM; 2018:2221-2239. doi:10.1137/1.9781611975031.144'
apa: 'Alistarh, D.-A., Aspnes, J., & Gelashvili, R. (2018). Space-optimal majority
in population protocols. In Proceedings of the 29th Annual ACM-SIAM Symposium
on Discrete Algorithms (pp. 2221–2239). New Orleans, LA, United States: ACM.
https://doi.org/10.1137/1.9781611975031.144'
chicago: Alistarh, Dan-Adrian, James Aspnes, and Rati Gelashvili. “Space-Optimal
Majority in Population Protocols.” In Proceedings of the 29th Annual ACM-SIAM
Symposium on Discrete Algorithms, 2221–39. ACM, 2018. https://doi.org/10.1137/1.9781611975031.144.
ieee: D.-A. Alistarh, J. Aspnes, and R. Gelashvili, “Space-optimal majority in population
protocols,” in Proceedings of the 29th Annual ACM-SIAM Symposium on Discrete
Algorithms, New Orleans, LA, United States, 2018, pp. 2221–2239.
ista: 'Alistarh D-A, Aspnes J, Gelashvili R. 2018. Space-optimal majority in population
protocols. Proceedings of the 29th Annual ACM-SIAM Symposium on Discrete Algorithms.
SODA: Symposium on Discrete Algorithms, 2221–2239.'
mla: Alistarh, Dan-Adrian, et al. “Space-Optimal Majority in Population Protocols.”
Proceedings of the 29th Annual ACM-SIAM Symposium on Discrete Algorithms,
ACM, 2018, pp. 2221–39, doi:10.1137/1.9781611975031.144.
short: D.-A. Alistarh, J. Aspnes, R. Gelashvili, in:, Proceedings of the 29th Annual
ACM-SIAM Symposium on Discrete Algorithms, ACM, 2018, pp. 2221–2239.
conference:
end_date: 2018-01-10
location: New Orleans, LA, United States
name: 'SODA: Symposium on Discrete Algorithms'
start_date: 2018-01-07
date_created: 2019-11-26T15:10:55Z
date_published: 2018-01-30T00:00:00Z
date_updated: 2024-10-21T06:02:41Z
day: '30'
department:
- _id: DaAl
doi: 10.1137/1.9781611975031.144
external_id:
arxiv:
- '1704.04947'
isi:
- '000483921200145'
fulldoi: https://doi.org/10.1137/1.9781611975031.144
isi: 1
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://arxiv.org/abs/1704.04947
month: '01'
oa: 1
oa_version: Preprint
page: 2221-2239
publication: Proceedings of the 29th Annual ACM-SIAM Symposium on Discrete Algorithms
publication_identifier:
isbn:
- '9781611975031'
publication_status: published
publisher: ACM
quality_controlled: '1'
scopus_import: '1'
status: public
title: Space-optimal majority in population protocols
type: conference
user_id: c635000d-4b10-11ee-a964-aac5a93f6ac1
year: '2018'
...
---
_id: '397'
abstract:
- lang: eng
text: 'Concurrent sets with range query operations are highly desirable in applications
such as in-memory databases. However, few set implementations offer range queries.
Known techniques for augmenting data structures with range queries (or operations
that can be used to build range queries) have numerous problems that limit their
usefulness. For example, they impose high overhead or rely heavily on garbage
collection. In this work, we show how to augment data structures with highly efficient
range queries, without relying on garbage collection. We identify a property of
epoch-based memory reclamation algorithms that makes them ideal for implementing
range queries, and produce three algorithms, which use locks, transactional memory
and lock-free techniques, respectively. Our algorithms are applicable to more
data structures than previous work, and are shown to be highly efficient on a
large scale Intel system. '
alternative_title:
- PPoPP
article_processing_charge: No
author:
- first_name: Maya
full_name: Arbel Raviv, Maya
last_name: Arbel Raviv
- first_name: Trevor A
full_name: Brown, Trevor A
id: 3569F0A0-F248-11E8-B48F-1D18A9856A87
last_name: Brown
citation:
ama: 'Arbel Raviv M, Brown TA. Harnessing epoch-based reclamation for efficient
range queries. In: Vol 53. ACM; 2018:14-27. doi:10.1145/3178487.3178489'
apa: 'Arbel Raviv, M., & Brown, T. A. (2018). Harnessing epoch-based reclamation
for efficient range queries (Vol. 53, pp. 14–27). Presented at the PPoPP: Principles
and Practice of Parallel Programming, Vienna, Austria: ACM. https://doi.org/10.1145/3178487.3178489'
chicago: Arbel Raviv, Maya, and Trevor A Brown. “Harnessing Epoch-Based Reclamation
for Efficient Range Queries,” 53:14–27. ACM, 2018. https://doi.org/10.1145/3178487.3178489.
ieee: 'M. Arbel Raviv and T. A. Brown, “Harnessing epoch-based reclamation for efficient
range queries,” presented at the PPoPP: Principles and Practice of Parallel Programming,
Vienna, Austria, 2018, vol. 53, no. 1, pp. 14–27.'
ista: 'Arbel Raviv M, Brown TA. 2018. Harnessing epoch-based reclamation for efficient
range queries. PPoPP: Principles and Practice of Parallel Programming, PPoPP,
vol. 53, 14–27.'
mla: Arbel Raviv, Maya, and Trevor A. Brown. Harnessing Epoch-Based Reclamation
for Efficient Range Queries. Vol. 53, no. 1, ACM, 2018, pp. 14–27, doi:10.1145/3178487.3178489.
short: M. Arbel Raviv, T.A. Brown, in:, ACM, 2018, pp. 14–27.
conference:
end_date: 2018-02-28
location: Vienna, Austria
name: 'PPoPP: Principles and Practice of Parallel Programming'
start_date: 2018-02-24
date_created: 2018-12-11T11:46:14Z
date_published: 2018-02-10T00:00:00Z
date_updated: 2023-09-11T14:10:25Z
day: '10'
department:
- _id: DaAl
doi: 10.1145/3178487.3178489
external_id:
isi:
- '000446161100002'
fulldoi: https://doi.org/10.1145/3178487.3178489
intvolume: ' 53'
isi: 1
issue: '1'
language:
- iso: eng
month: '02'
oa_version: None
page: 14 - 27
publication_identifier:
isbn:
- 978-1-4503-4982-6
publication_status: published
publisher: ACM
publist_id: '7430'
quality_controlled: '1'
scopus_import: '1'
status: public
title: Harnessing epoch-based reclamation for efficient range queries
type: conference
user_id: c635000d-4b10-11ee-a964-aac5a93f6ac1
volume: 53
year: '2018'
...
---
_id: '43'
abstract:
- lang: eng
text: 'The initial amount of pathogens required to start an infection within a susceptible
host is called the infective dose and is known to vary to a large extent between
different pathogen species. We investigate the hypothesis that the differences
in infective doses are explained by the mode of action in the underlying mechanism
of pathogenesis: Pathogens with locally acting mechanisms tend to have smaller
infective doses than pathogens with distantly acting mechanisms. While empirical
evidence tends to support the hypothesis, a formal theoretical explanation has
been lacking. We give simple analytical models to gain insight into this phenomenon
and also investigate a stochastic, spatially explicit, mechanistic within-host
model for toxin-dependent bacterial infections. The model shows that pathogens
secreting locally acting toxins have smaller infective doses than pathogens secreting
diffusive toxins, as hypothesized. While local pathogenetic mechanisms require
smaller infective doses, pathogens with distantly acting toxins tend to spread
faster and may cause more damage to the host. The proposed model can serve as
a basis for the spatially explicit analysis of various virulence factors also
in the context of other problems in infection dynamics.'
acknowledgement: J.R. and J.V.A. were also supported by the Academy of Finland Grants
1273253 and 267541.
article_processing_charge: No
author:
- first_name: Joel
full_name: Rybicki, Joel
id: 334EFD2E-F248-11E8-B48F-1D18A9856A87
last_name: Rybicki
orcid: 0000-0002-6432-6646
- first_name: Eva
full_name: Kisdi, Eva
last_name: Kisdi
- first_name: Jani
full_name: Anttila, Jani
last_name: Anttila
citation:
ama: Rybicki J, Kisdi E, Anttila J. Model of bacterial toxin-dependent pathogenesis
explains infective dose. Proceedings of the National Academy of Sciences of
the United States of America. 2018;115(42):10690-10695. doi:10.1073/pnas.1721061115
apa: Rybicki, J., Kisdi, E., & Anttila, J. (2018). Model of bacterial toxin-dependent
pathogenesis explains infective dose. Proceedings of the National Academy of
Sciences of the United States of America. National Academy of Sciences. https://doi.org/10.1073/pnas.1721061115
chicago: Rybicki, Joel, Eva Kisdi, and Jani Anttila. “Model of Bacterial Toxin-Dependent
Pathogenesis Explains Infective Dose.” Proceedings of the National Academy
of Sciences of the United States of America. National Academy of Sciences,
2018. https://doi.org/10.1073/pnas.1721061115.
ieee: J. Rybicki, E. Kisdi, and J. Anttila, “Model of bacterial toxin-dependent
pathogenesis explains infective dose,” Proceedings of the National Academy
of Sciences of the United States of America, vol. 115, no. 42. National Academy
of Sciences, pp. 10690–10695, 2018.
ista: Rybicki J, Kisdi E, Anttila J. 2018. Model of bacterial toxin-dependent pathogenesis
explains infective dose. Proceedings of the National Academy of Sciences of the
United States of America. 115(42), 10690–10695.
mla: Rybicki, Joel, et al. “Model of Bacterial Toxin-Dependent Pathogenesis Explains
Infective Dose.” Proceedings of the National Academy of Sciences of the United
States of America, vol. 115, no. 42, National Academy of Sciences, 2018, pp.
10690–95, doi:10.1073/pnas.1721061115.
short: J. Rybicki, E. Kisdi, J. Anttila, Proceedings of the National Academy of
Sciences of the United States of America 115 (2018) 10690–10695.
date_created: 2018-12-11T11:44:19Z
date_published: 2018-10-02T00:00:00Z
date_updated: 2025-06-03T11:16:28Z
day: '02'
ddc:
- '570'
- '577'
department:
- _id: DaAl
doi: 10.1073/pnas.1721061115
ec_funded: 1
external_id:
isi:
- '000447491300057'
file:
- access_level: open_access
checksum: df7ac544a587c06b75692653b9fabd18
content_type: application/pdf
creator: dernst
date_created: 2019-04-09T08:02:50Z
date_updated: 2020-07-14T12:46:26Z
file_id: '6258'
file_name: 2018_PNAS_Rybicki.pdf
file_size: 4070777
relation: main_file
file_date_updated: 2020-07-14T12:46:26Z
fulldoi: https://doi.org/10.1073/pnas.1721061115
has_accepted_license: '1'
intvolume: ' 115'
isi: 1
issue: '42'
language:
- iso: eng
month: '10'
oa: 1
oa_version: Submitted Version
page: 10690 - 10695
project:
- _id: 260C2330-B435-11E9-9278-68D0E5697425
call_identifier: H2020
grant_number: '754411'
name: ISTplus - Postdoctoral Fellowships
publication: Proceedings of the National Academy of Sciences of the United States
of America
publication_identifier:
eissn:
- 1091-6490
issn:
- 0027-8424
publication_status: published
publisher: National Academy of Sciences
publist_id: '8011'
pubrep_id: '1063'
quality_controlled: '1'
scopus_import: '1'
status: public
title: Model of bacterial toxin-dependent pathogenesis explains infective dose
type: journal_article
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 115
year: '2018'
...
---
_id: '6589'
abstract:
- lang: eng
text: Distributed training of massive machine learning models, in particular deep
neural networks, via Stochastic Gradient Descent (SGD) is becoming commonplace.
Several families of communication-reduction methods, such as quantization, large-batch
methods, and gradient sparsification, have been proposed. To date, gradient sparsification
methods--where each node sorts gradients by magnitude, and only communicates a
subset of the components, accumulating the rest locally--are known to yield some
of the largest practical gains. Such methods can reduce the amount of communication
per step by up to \emph{three orders of magnitude}, while preserving model accuracy.
Yet, this family of methods currently has no theoretical justification. This is
the question we address in this paper. We prove that, under analytic assumptions,
sparsifying gradients by magnitude with local error correction provides convergence
guarantees, for both convex and non-convex smooth objectives, for data-parallel
SGD. The main insight is that sparsification methods implicitly maintain bounds
on the maximum impact of stale updates, thanks to selection by magnitude. Our
analysis and empirical validation also reveal that these methods do require analytical
conditions to converge well, justifying existing heuristics.
alternative_title:
- Advances in Neural Information Processing Systems
article_processing_charge: No
arxiv: 1
author:
- first_name: Dan-Adrian
full_name: Alistarh, Dan-Adrian
id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
last_name: Alistarh
orcid: 0000-0003-3650-940X
- first_name: Torsten
full_name: Hoefler, Torsten
last_name: Hoefler
- first_name: Mikael
full_name: Johansson, Mikael
last_name: Johansson
- first_name: Nikola H
full_name: Konstantinov, Nikola H
id: 4B9D76E4-F248-11E8-B48F-1D18A9856A87
last_name: Konstantinov
orcid: 0009-0009-5204-7621
- first_name: Sarit
full_name: Khirirat, Sarit
last_name: Khirirat
- first_name: Cedric
full_name: Renggli, Cedric
last_name: Renggli
citation:
ama: 'Alistarh D-A, Hoefler T, Johansson M, Konstantinov NH, Khirirat S, Renggli
C. The convergence of sparsified gradient methods. In: 32nd Conference on Neural
Information Processing Systems. Neural Information Processing Systems Foundation;
2018:5973-5983.'
apa: 'Alistarh, D.-A., Hoefler, T., Johansson, M., Konstantinov, N. H., Khirirat,
S., & Renggli, C. (2018). The convergence of sparsified gradient methods.
In 32nd Conference on Neural Information Processing Systems (pp. 5973–5983).
Montreal, Canada: Neural Information Processing Systems Foundation.'
chicago: Alistarh, Dan-Adrian, Torsten Hoefler, Mikael Johansson, Nikola H Konstantinov,
Sarit Khirirat, and Cedric Renggli. “The Convergence of Sparsified Gradient Methods.”
In 32nd Conference on Neural Information Processing Systems, 5973–83. Neural
Information Processing Systems Foundation, 2018.
ieee: D.-A. Alistarh, T. Hoefler, M. Johansson, N. H. Konstantinov, S. Khirirat,
and C. Renggli, “The convergence of sparsified gradient methods,” in 32nd Conference
on Neural Information Processing Systems, Montreal, Canada, 2018, pp. 5973–5983.
ista: 'Alistarh D-A, Hoefler T, Johansson M, Konstantinov NH, Khirirat S, Renggli
C. 2018. The convergence of sparsified gradient methods. 32nd Conference on Neural
Information Processing Systems. NeurIPS: Conference on Neural Information Processing
Systems, Advances in Neural Information Processing Systems, , 5973–5983.'
mla: Alistarh, Dan-Adrian, et al. “The Convergence of Sparsified Gradient Methods.”
32nd Conference on Neural Information Processing Systems, Neural Information
Processing Systems Foundation, 2018, pp. 5973–83.
short: D.-A. Alistarh, T. Hoefler, M. Johansson, N.H. Konstantinov, S. Khirirat,
C. Renggli, in:, 32nd Conference on Neural Information Processing Systems, Neural
Information Processing Systems Foundation, 2018, pp. 5973–5983.
conference:
end_date: 2018-12-08
location: Montreal, Canada
name: 'NeurIPS: Conference on Neural Information Processing Systems'
start_date: 2018-12-02
corr_author: '1'
das_tickbox: '1'
date_created: 2019-06-27T09:32:55Z
date_published: 2018-12-01T00:00:00Z
date_updated: 2026-07-08T05:48:21Z
day: '01'
department:
- _id: DaAl
- _id: ChLa
ec_funded: 1
external_id:
arxiv:
- '1809.10505'
isi:
- '000461852000047'
isi: 1
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://arxiv.org/abs/1809.10505
month: '12'
oa: 1
oa_version: Preprint
page: 5973-5983
project:
- _id: 2564DBCA-B435-11E9-9278-68D0E5697425
call_identifier: H2020
grant_number: '665385'
name: International IST Doctoral Program
publication: 32nd Conference on Neural Information Processing Systems
publication_identifier:
issn:
- 1049-5258
publication_status: published
publisher: Neural Information Processing Systems Foundation
quality_controlled: '1'
scopus_import: '1'
status: public
title: The convergence of sparsified gradient methods
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
year: '2018'
...
---
_id: '487'
abstract:
- lang: eng
text: In this paper we study network architecture for unlicensed cellular networking
for outdoor coverage in TV white spaces. The main technology proposed for TV white
spaces is 802.11af, a Wi-Fi variant adapted for TV frequencies. However, 802.11af
is originally designed for improved indoor propagation. We show that long links,
typical for outdoor use, exacerbate known Wi-Fi issues, such as hidden and exposed
terminal, and significantly reduce its efficiency. Instead, we propose CellFi,
an alternative architecture based on LTE. LTE is designed for long-range coverage
and throughput efficiency, but it is also designed to operate in tightly controlled
and centrally managed networks. CellFi overcomes these problems by designing an
LTE-compatible spectrum database component, mandatory for TV white space networking,
and introducing an interference management component for distributed coordination.
CellFi interference management is compatible with existing LTE mechanisms, requires
no explicit communication between base stations, and is more efficient than CSMA
for long links. We evaluate our design through extensive real world evaluation
on of-the-shelf LTE equipment and simulations. We show that, compared to 802.11af,
it increases coverage by 40% and reduces median flow completion times by 2.3x.
article_processing_charge: No
author:
- first_name: Ghufran
full_name: Baig, Ghufran
last_name: Baig
- first_name: Bozidar
full_name: Radunovic, Bozidar
last_name: Radunovic
- first_name: Dan-Adrian
full_name: Alistarh, Dan-Adrian
id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
last_name: Alistarh
orcid: 0000-0003-3650-940X
- first_name: Matthew
full_name: Balkwill, Matthew
last_name: Balkwill
- first_name: Thomas
full_name: Karagiannis, Thomas
last_name: Karagiannis
- first_name: Lili
full_name: Qiu, Lili
last_name: Qiu
citation:
ama: 'Baig G, Radunovic B, Alistarh D-A, Balkwill M, Karagiannis T, Qiu L. Towards
unlicensed cellular networks in TV white spaces. In: Proceedings of the 2017
13th International Conference on Emerging Networking EXperiments and Technologies.
ACM; 2017:2-14. doi:10.1145/3143361.3143367'
apa: 'Baig, G., Radunovic, B., Alistarh, D.-A., Balkwill, M., Karagiannis, T., &
Qiu, L. (2017). Towards unlicensed cellular networks in TV white spaces. In Proceedings
of the 2017 13th International Conference on emerging Networking EXperiments and
Technologies (pp. 2–14). Incheon, South Korea: ACM. https://doi.org/10.1145/3143361.3143367'
chicago: Baig, Ghufran, Bozidar Radunovic, Dan-Adrian Alistarh, Matthew Balkwill,
Thomas Karagiannis, and Lili Qiu. “Towards Unlicensed Cellular Networks in TV
White Spaces.” In Proceedings of the 2017 13th International Conference on
Emerging Networking EXperiments and Technologies, 2–14. ACM, 2017. https://doi.org/10.1145/3143361.3143367.
ieee: G. Baig, B. Radunovic, D.-A. Alistarh, M. Balkwill, T. Karagiannis, and L.
Qiu, “Towards unlicensed cellular networks in TV white spaces,” in Proceedings
of the 2017 13th International Conference on emerging Networking EXperiments and
Technologies, Incheon, South Korea, 2017, pp. 2–14.
ista: 'Baig G, Radunovic B, Alistarh D-A, Balkwill M, Karagiannis T, Qiu L. 2017.
Towards unlicensed cellular networks in TV white spaces. Proceedings of the 2017
13th International Conference on emerging Networking EXperiments and Technologies.
CoNEXT: Conference on emerging Networking EXperiments and Technologies, 2–14.'
mla: Baig, Ghufran, et al. “Towards Unlicensed Cellular Networks in TV White Spaces.”
Proceedings of the 2017 13th International Conference on Emerging Networking
EXperiments and Technologies, ACM, 2017, pp. 2–14, doi:10.1145/3143361.3143367.
short: G. Baig, B. Radunovic, D.-A. Alistarh, M. Balkwill, T. Karagiannis, L. Qiu,
in:, Proceedings of the 2017 13th International Conference on Emerging Networking
EXperiments and Technologies, ACM, 2017, pp. 2–14.
conference:
end_date: 2017-12-15
location: Incheon, South Korea
name: 'CoNEXT: Conference on emerging Networking EXperiments and Technologies'
start_date: 2017-12-12
corr_author: '1'
date_created: 2018-12-11T11:46:45Z
date_published: 2017-11-28T00:00:00Z
date_updated: 2025-09-18T09:50:43Z
day: '28'
department:
- _id: DaAl
doi: 10.1145/3143361.3143367
external_id:
isi:
- '000526087500002'
fulldoi: https://doi.org/10.1145/3143361.3143367
isi: 1
language:
- iso: eng
month: '11'
oa_version: None
page: 2 - 14
publication: Proceedings of the 2017 13th International Conference on emerging Networking
EXperiments and Technologies
publication_identifier:
isbn:
- 978-145035422-6
publication_status: published
publisher: ACM
publist_id: '7333'
quality_controlled: '1'
scopus_import: '1'
status: public
title: Towards unlicensed cellular networks in TV white spaces
type: conference
user_id: 317138e5-6ab7-11ef-aa6d-ffef3953e345
year: '2017'
...
---
_id: '791'
abstract:
- lang: eng
text: 'Consider the following random process: we are given n queues, into which
elements of increasing labels are inserted uniformly at random. To remove an element,
we pick two queues at random, and remove the element of lower label (higher priority)
among the two. The cost of a removal is the rank of the label removed, among labels
still present in any of the queues, that is, the distance from the optimal choice
at each step. Variants of this strategy are prevalent in state-of-the-art concurrent
priority queue implementations. Nonetheless, it is not known whether such implementations
provide any rank guarantees, even in a sequential model. We answer this question,
showing that this strategy provides surprisingly strong guarantees: Although the
single-choice process, where we always insert and remove from a single randomly
chosen queue, has degrading cost, going to infinity as we increase the number
of steps, in the two choice process, the expected rank of a removed element is
O(n) while the expected worst-case cost is O(n log n). These bounds are tight,
and hold irrespective of the number of steps for which we run the process. The
argument is based on a new technical connection between "heavily loaded"
balls-into-bins processes and priority scheduling. Our analytic results inspire
a new concurrent priority queue implementation, which improves upon the state
of the art in terms of practical performance.'
article_processing_charge: No
arxiv: 1
author:
- first_name: Dan-Adrian
full_name: Alistarh, Dan-Adrian
id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
last_name: Alistarh
orcid: 0000-0003-3650-940X
- first_name: Justin
full_name: Kopinsky, Justin
last_name: Kopinsky
- first_name: Jerry
full_name: Li, Jerry
last_name: Li
- first_name: Giorgi
full_name: Nadiradze, Giorgi
id: 3279A00C-F248-11E8-B48F-1D18A9856A87
last_name: Nadiradze
orcid: 0000-0001-5634-0731
citation:
ama: 'Alistarh D-A, Kopinsky J, Li J, Nadiradze G. The power of choice in priority
scheduling. In: Proceedings of the ACM Symposium on Principles of Distributed
Computing. Vol Part F129314. ACM; 2017:283-292. doi:10.1145/3087801.3087810'
apa: 'Alistarh, D.-A., Kopinsky, J., Li, J., & Nadiradze, G. (2017). The power
of choice in priority scheduling. In Proceedings of the ACM Symposium on Principles
of Distributed Computing (Vol. Part F129314, pp. 283–292). Washington, WA,
USA: ACM. https://doi.org/10.1145/3087801.3087810'
chicago: Alistarh, Dan-Adrian, Justin Kopinsky, Jerry Li, and Giorgi Nadiradze.
“The Power of Choice in Priority Scheduling.” In Proceedings of the ACM Symposium
on Principles of Distributed Computing, Part F129314:283–92. ACM, 2017. https://doi.org/10.1145/3087801.3087810.
ieee: D.-A. Alistarh, J. Kopinsky, J. Li, and G. Nadiradze, “The power of choice
in priority scheduling,” in Proceedings of the ACM Symposium on Principles
of Distributed Computing, Washington, WA, USA, 2017, vol. Part F129314, pp.
283–292.
ista: 'Alistarh D-A, Kopinsky J, Li J, Nadiradze G. 2017. The power of choice in
priority scheduling. Proceedings of the ACM Symposium on Principles of Distributed
Computing. PODC: Principles of Distributed Computing vol. Part F129314, 283–292.'
mla: Alistarh, Dan-Adrian, et al. “The Power of Choice in Priority Scheduling.”
Proceedings of the ACM Symposium on Principles of Distributed Computing,
vol. Part F129314, ACM, 2017, pp. 283–92, doi:10.1145/3087801.3087810.
short: D.-A. Alistarh, J. Kopinsky, J. Li, G. Nadiradze, in:, Proceedings of the
ACM Symposium on Principles of Distributed Computing, ACM, 2017, pp. 283–292.
conference:
end_date: 2017-07-27
location: Washington, WA, USA
name: 'PODC: Principles of Distributed Computing'
start_date: 2017-07-25
date_created: 2018-12-11T11:48:31Z
date_published: 2017-07-26T00:00:00Z
date_updated: 2025-06-04T09:44:47Z
day: '26'
department:
- _id: DaAl
doi: 10.1145/3087801.3087810
external_id:
arxiv:
- '1706.04178'
isi:
- '000462995000035'
fulldoi: https://doi.org/10.1145/3087801.3087810
isi: 1
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://arxiv.org/abs/1706.04178
month: '07'
oa: 1
oa_version: Submitted Version
page: 283 - 292
publication: Proceedings of the ACM Symposium on Principles of Distributed Computing
publication_identifier:
isbn:
- 978-145034992-5
publication_status: published
publisher: ACM
publist_id: '6864'
quality_controlled: '1'
scopus_import: '1'
status: public
title: The power of choice in priority scheduling
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: Part F129314
year: '2017'
...
---
_id: '431'
abstract:
- lang: eng
text: 'Parallel implementations of stochastic gradient descent (SGD) have received
significant research attention, thanks to its excellent scalability properties.
A fundamental barrier when parallelizing SGD is the high bandwidth cost of communicating
gradient updates between nodes; consequently, several lossy compresion heuristics
have been proposed, by which nodes only communicate quantized gradients. Although
effective in practice, these heuristics do not always converge. In this paper,
we propose Quantized SGD (QSGD), a family of compression schemes with convergence
guarantees and good practical performance. QSGD allows the user to smoothly trade
off communication bandwidth and convergence time: nodes can adjust the number
of bits sent per iteration, at the cost of possibly higher variance. We show that
this trade-off is inherent, in the sense that improving it past some threshold
would violate information-theoretic lower bounds. QSGD guarantees convergence
for convex and non-convex objectives, under asynchrony, and can be extended to
stochastic variance-reduced techniques. When applied to training deep neural networks
for image classification and automated speech recognition, QSGD leads to significant
reductions in end-to-end training time. For instance, on 16GPUs, we can train
the ResNet-152 network to full accuracy on ImageNet 1.8 × faster than the full-precision
variant. '
alternative_title:
- Advances in Neural Information Processing Systems
article_processing_charge: No
arxiv: 1
author:
- first_name: Dan-Adrian
full_name: Alistarh, Dan-Adrian
id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
last_name: Alistarh
orcid: 0000-0003-3650-940X
- first_name: Demjan
full_name: Grubic, Demjan
last_name: Grubic
- first_name: Jerry
full_name: Li, Jerry
last_name: Li
- first_name: Ryota
full_name: Tomioka, Ryota
last_name: Tomioka
- first_name: Milan
full_name: Vojnović, Milan
last_name: Vojnović
citation:
ama: 'Alistarh D-A, Grubic D, Li J, Tomioka R, Vojnović M. QSGD: Communication-efficient
SGD via gradient quantization and encoding. In: Vol 2017. Neural Information Processing
Systems Foundation; 2017:1710-1721.'
apa: 'Alistarh, D.-A., Grubic, D., Li, J., Tomioka, R., & Vojnović, M. (2017).
QSGD: Communication-efficient SGD via gradient quantization and encoding (Vol.
2017, pp. 1710–1721). Presented at the NIPS: Neural Information Processing System,
Long Beach, CA, United States: Neural Information Processing Systems Foundation.'
chicago: 'Alistarh, Dan-Adrian, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan
Vojnović. “QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding,”
2017:1710–21. Neural Information Processing Systems Foundation, 2017.'
ieee: 'D.-A. Alistarh, D. Grubic, J. Li, R. Tomioka, and M. Vojnović, “QSGD: Communication-efficient
SGD via gradient quantization and encoding,” presented at the NIPS: Neural Information
Processing System, Long Beach, CA, United States, 2017, vol. 2017, pp. 1710–1721.'
ista: 'Alistarh D-A, Grubic D, Li J, Tomioka R, Vojnović M. 2017. QSGD: Communication-efficient
SGD via gradient quantization and encoding. NIPS: Neural Information Processing
System, Advances in Neural Information Processing Systems, vol. 2017, 1710–1721.'
mla: 'Alistarh, Dan-Adrian, et al. QSGD: Communication-Efficient SGD via Gradient
Quantization and Encoding. Vol. 2017, Neural Information Processing Systems
Foundation, 2017, pp. 1710–21.'
short: D.-A. Alistarh, D. Grubic, J. Li, R. Tomioka, M. Vojnović, in:, Neural Information
Processing Systems Foundation, 2017, pp. 1710–1721.
conference:
end_date: 2017-12-09
location: Long Beach, CA, United States
name: 'NIPS: Neural Information Processing System'
start_date: 2017-12-04
corr_author: '1'
date_created: 2018-12-11T11:46:26Z
date_published: 2017-01-01T00:00:00Z
date_updated: 2025-09-18T10:07:20Z
day: '01'
department:
- _id: DaAl
external_id:
arxiv:
- '1610.02132'
isi:
- '000452649401072'
intvolume: ' 2017'
isi: 1
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://arxiv.org/abs/1610.02132
month: '01'
oa: 1
oa_version: Submitted Version
page: 1710-1721
publication_identifier:
issn:
- 1049-5258
publication_status: published
publisher: Neural Information Processing Systems Foundation
publist_id: '7392'
quality_controlled: '1'
status: public
title: 'QSGD: Communication-efficient SGD via gradient quantization and encoding'
type: conference
user_id: 317138e5-6ab7-11ef-aa6d-ffef3953e345
volume: 2017
year: '2017'
...
---
_id: '432'
abstract:
- lang: eng
text: 'Recently there has been significant interest in training machine-learning
models at low precision: by reducing precision, one can reduce computation and
communication by one order of magnitude. We examine training at reduced precision,
both from a theoretical and practical perspective, and ask: is it possible to
train models at end-to-end low precision with provable guarantees? Can this lead
to consistent order-of-magnitude speedups? We mainly focus on linear models, and
the answer is yes for linear models. We develop a simple framework called ZipML
based on one simple but novel strategy called double sampling. Our ZipML framework
is able to execute training at low precision with no bias, guaranteeing convergence,
whereas naive quanti- zation would introduce significant bias. We val- idate our
framework across a range of applica- tions, and show that it enables an FPGA proto-
type that is up to 6.5 × faster than an implemen- tation using full 32-bit precision.
We further de- velop a variance-optimal stochastic quantization strategy and show
that it can make a significant difference in a variety of settings. When applied
to linear models together with double sampling, we save up to another 1.7 × in
data movement compared with uniform quantization. When training deep networks
with quantized models, we achieve higher accuracy than the state-of-the- art XNOR-Net. '
alternative_title:
- PMLR Press
article_processing_charge: No
author:
- first_name: Hantian
full_name: Zhang, Hantian
last_name: Zhang
- first_name: Jerry
full_name: Li, Jerry
last_name: Li
- first_name: Kaan
full_name: Kara, Kaan
last_name: Kara
- first_name: Dan-Adrian
full_name: Alistarh, Dan-Adrian
id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
last_name: Alistarh
orcid: 0000-0003-3650-940X
- first_name: Ji
full_name: Liu, Ji
last_name: Liu
- first_name: Ce
full_name: Zhang, Ce
last_name: Zhang
citation:
ama: 'Zhang H, Li J, Kara K, Alistarh D-A, Liu J, Zhang C. ZipML: Training linear
models with end-to-end low precision, and a little bit of deep learning. In: Proceedings
of Machine Learning Research. Vol 70. ML Research Press; 2017:4035-4043.'
apa: 'Zhang, H., Li, J., Kara, K., Alistarh, D.-A., Liu, J., & Zhang, C. (2017).
ZipML: Training linear models with end-to-end low precision, and a little bit
of deep learning. In Proceedings of Machine Learning Research (Vol. 70,
pp. 4035–4043). Sydney, Australia: ML Research Press.'
chicago: 'Zhang, Hantian, Jerry Li, Kaan Kara, Dan-Adrian Alistarh, Ji Liu, and
Ce Zhang. “ZipML: Training Linear Models with End-to-End Low Precision, and a
Little Bit of Deep Learning.” In Proceedings of Machine Learning Research,
70:4035–43. ML Research Press, 2017.'
ieee: 'H. Zhang, J. Li, K. Kara, D.-A. Alistarh, J. Liu, and C. Zhang, “ZipML: Training
linear models with end-to-end low precision, and a little bit of deep learning,”
in Proceedings of Machine Learning Research, Sydney, Australia, 2017, vol.
70, pp. 4035–4043.'
ista: 'Zhang H, Li J, Kara K, Alistarh D-A, Liu J, Zhang C. 2017. ZipML: Training
linear models with end-to-end low precision, and a little bit of deep learning.
Proceedings of Machine Learning Research. ICML: International Conference on Machine
Learning, PMLR Press, vol. 70, 4035–4043.'
mla: 'Zhang, Hantian, et al. “ZipML: Training Linear Models with End-to-End Low
Precision, and a Little Bit of Deep Learning.” Proceedings of Machine Learning
Research, vol. 70, ML Research Press, 2017, pp. 4035–43.'
short: H. Zhang, J. Li, K. Kara, D.-A. Alistarh, J. Liu, C. Zhang, in:, Proceedings
of Machine Learning Research, ML Research Press, 2017, pp. 4035–4043.
conference:
end_date: 2017-08-11
location: Sydney, Australia
name: 'ICML: International Conference on Machine Learning'
start_date: 2017-08-06
corr_author: '1'
date_created: 2018-12-11T11:46:26Z
date_published: 2017-01-01T00:00:00Z
date_updated: 2025-09-18T10:06:02Z
day: '01'
ddc:
- '000'
department:
- _id: DaAl
external_id:
isi:
- '000683309504015'
file:
- access_level: open_access
checksum: 86156ba7f4318e47cef3eb9092593c10
content_type: application/pdf
creator: dernst
date_created: 2019-01-22T08:23:58Z
date_updated: 2020-07-14T12:46:26Z
file_id: '5869'
file_name: 2017_ICML_Zhang.pdf
file_size: 849345
relation: main_file
file_date_updated: 2020-07-14T12:46:26Z
has_accepted_license: '1'
isi: 1
language:
- iso: eng
month: '01'
oa: 1
oa_version: Submitted Version
page: 4035 - 4043
publication: Proceedings of Machine Learning Research
publication_identifier:
isbn:
- 978-151085514-4
publication_status: published
publisher: ML Research Press
publist_id: '7391'
quality_controlled: '1'
scopus_import: '1'
status: public
title: 'ZipML: Training linear models with end-to-end low precision, and a little
bit of deep learning'
type: conference
user_id: 317138e5-6ab7-11ef-aa6d-ffef3953e345
volume: ' 70'
year: '2017'
...