A data augmentation perspective on diffusion models and retrieval

Burg MF, Wenzel F, Zietlow D, Horn M, Makansi O, Locatello F, Russell C. A data augmentation perspective on diffusion models and retrieval. arXiv, 2304.10253.

Download (ext.)

Preprint | Submitted | English
Author
Burg, Max F.; Wenzel, Florian; Zietlow, Dominik; Horn, Max; Makansi, Osama; Locatello, FrancescoISTA ; Russell, Chris
Department
Abstract
Diffusion models excel at generating photorealistic images from text-queries. Naturally, many approaches have been proposed to use these generative abilities to augment training datasets for downstream tasks, such as classification. However, diffusion models are themselves trained on large noisily supervised, but nonetheless, annotated datasets. It is an open question whether the generalization capabilities of diffusion models beyond using the additional data of the pre-training process for augmentation lead to improved downstream performance. We perform a systematic evaluation of existing methods to generate images from diffusion models and study new extensions to assess their benefit for data augmentation. While we find that personalizing diffusion models towards the target data outperforms simpler prompting strategies, we also show that using the training data of the diffusion model alone, via a simple nearest neighbor retrieval procedure, leads to even stronger downstream performance. Overall, our study probes the limitations of diffusion models for data augmentation but also highlights its potential in generating new training data to improve performance on simple downstream vision tasks.
Publishing Year
Date Published
2023-04-20
Journal Title
arXiv
Article Number
2304.10253
IST-REx-ID

Cite this

Burg MF, Wenzel F, Zietlow D, et al. A data augmentation perspective on diffusion models and retrieval. arXiv. doi:10.48550/arXiv.2304.10253
Burg, M. F., Wenzel, F., Zietlow, D., Horn, M., Makansi, O., Locatello, F., & Russell, C. (n.d.). A data augmentation perspective on diffusion models and retrieval. arXiv. https://doi.org/10.48550/arXiv.2304.10253
Burg, Max F., Florian Wenzel, Dominik Zietlow, Max Horn, Osama Makansi, Francesco Locatello, and Chris Russell. “A Data Augmentation Perspective on Diffusion Models and Retrieval.” ArXiv, n.d. https://doi.org/10.48550/arXiv.2304.10253.
M. F. Burg et al., “A data augmentation perspective on diffusion models and retrieval,” arXiv. .
Burg MF, Wenzel F, Zietlow D, Horn M, Makansi O, Locatello F, Russell C. A data augmentation perspective on diffusion models and retrieval. arXiv, 2304.10253.
Burg, Max F., et al. “A Data Augmentation Perspective on Diffusion Models and Retrieval.” ArXiv, 2304.10253, doi:10.48550/arXiv.2304.10253.
All files available under the following license(s):
Copyright Statement:
This Item is protected by copyright and/or related rights. [...]

Link(s) to Main File(s)
Access Level
OA Open Access

Export

Marked Publications

Open Data ISTA Research Explorer

Sources

arXiv 2304.10253

Search this title in

Google Scholar