---
_id: '17374'
abstract:
- lang: eng
text: Achieving photorealistic 3D view synthesis and relighting of human portraits
is pivotal for advancing AR/VR applications. Existing methodologies in portrait
relighting demonstrate substantial limitations in terms of generalization and
3D consistency, coupled with inaccuracies in physically realistic lighting and
identity preservation. Furthermore, personalization from a single view is difficult
to achieve and often requires multiview images during the testing phase or involves
slow optimization processes. This paper introduces Lite2Relight , a novel technique
that can predict 3D consistent head poses of portraits while performing physically
plausible light editing at interactive speed. Our method uniquely extends the
generative capabilities and efficient volumetric representation of EG3D, leveraging
a lightstage dataset to implicitly disentangle face reflectance and perform relighting
under target HDRI environment maps. By utilizing a pre-trained geometry-aware
encoder and a feature alignment module, we map input images into a relightable
3D space, enhancing them with a strong face geometry and reflectance prior. Through
extensive quantitative and qualitative evaluations, we show that our method outperforms
the state-of-the-art methods in terms of efficacy, photorealism, and practical
application. This includes producing 3D-consistent results of the full head, including
hair, eyes, and expressions. Lite2Relight paves the way for large-scale adoption
of photorealistic portrait editing in various domains, offering a robust, interactive
solution to a previously constrained problem.
acknowledgement: This work was supported by the ERC Consolidator Grant 4DReply (770784).
We extend our gratitude to Shrisha Bharadwaj for providing feedback and constant
support.
article_number: '41'
article_processing_charge: Yes (in subscription journal)
arxiv: 1
author:
- first_name: Pramod
full_name: Rao, Pramod
last_name: Rao
- first_name: Gereon
full_name: Fox, Gereon
last_name: Fox
- first_name: Abhimitra
full_name: Meka, Abhimitra
last_name: Meka
- first_name: B. R.
full_name: Mallikarjun, B. R.
last_name: Mallikarjun
- first_name: Fangneng
full_name: Zhan, Fangneng
last_name: Zhan
- first_name: Tim
full_name: Weyrich, Tim
last_name: Weyrich
- first_name: Bernd
full_name: Bickel, Bernd
id: 49876194-F248-11E8-B48F-1D18A9856A87
last_name: Bickel
orcid: 0000-0001-6511-9385
- first_name: Hanspeter
full_name: Pfister, Hanspeter
last_name: Pfister
- first_name: Wojciech
full_name: Matusik, Wojciech
last_name: Matusik
- first_name: Mohamed
full_name: Elgharib, Mohamed
last_name: Elgharib
- first_name: Christian
full_name: Theobalt, Christian
last_name: Theobalt
citation:
ama: 'Rao P, Fox G, Meka A, et al. Lite2Relight: 3D-aware single image portrait
relighting. In: Proceedings - SIGGRAPH 2024 Conference Papers. Association
for Computing Machinery; 2024. doi:10.1145/3641519.3657470'
apa: 'Rao, P., Fox, G., Meka, A., Mallikarjun, B. R., Zhan, F., Weyrich, T., … Theobalt,
C. (2024). Lite2Relight: 3D-aware single image portrait relighting. In Proceedings
- SIGGRAPH 2024 Conference Papers. Denver, CO, United States: Association
for Computing Machinery. https://doi.org/10.1145/3641519.3657470'
chicago: 'Rao, Pramod, Gereon Fox, Abhimitra Meka, B. R. Mallikarjun, Fangneng Zhan,
Tim Weyrich, Bernd Bickel, et al. “Lite2Relight: 3D-Aware Single Image Portrait
Relighting.” In Proceedings - SIGGRAPH 2024 Conference Papers. Association
for Computing Machinery, 2024. https://doi.org/10.1145/3641519.3657470.'
ieee: 'P. Rao et al., “Lite2Relight: 3D-aware single image portrait relighting,”
in Proceedings - SIGGRAPH 2024 Conference Papers, Denver, CO, United States,
2024.'
ista: 'Rao P, Fox G, Meka A, Mallikarjun BR, Zhan F, Weyrich T, Bickel B, Pfister
H, Matusik W, Elgharib M, Theobalt C. 2024. Lite2Relight: 3D-aware single image
portrait relighting. Proceedings - SIGGRAPH 2024 Conference Papers. SIGGRAPH:
Computer Graphics and Interactive Techniques Conference, 41.'
mla: 'Rao, Pramod, et al. “Lite2Relight: 3D-Aware Single Image Portrait Relighting.”
Proceedings - SIGGRAPH 2024 Conference Papers, 41, Association for Computing
Machinery, 2024, doi:10.1145/3641519.3657470.'
short: P. Rao, G. Fox, A. Meka, B.R. Mallikarjun, F. Zhan, T. Weyrich, B. Bickel,
H. Pfister, W. Matusik, M. Elgharib, C. Theobalt, in:, Proceedings - SIGGRAPH
2024 Conference Papers, Association for Computing Machinery, 2024.
conference:
end_date: 2024-08-01
location: Denver, CO, United States
name: 'SIGGRAPH: Computer Graphics and Interactive Techniques Conference'
start_date: 2024-07-27
das_tickbox: '0'
date_created: 2024-08-04T22:01:21Z
date_published: 2024-07-13T00:00:00Z
date_updated: 2026-09-24T11:00:21Z
day: '13'
ddc:
- '000'
department:
- _id: BeBi
doi: 10.1145/3641519.3657470
external_id:
arxiv:
- '2407.10487'
isi:
- '001282218200076'
file:
- access_level: open_access
checksum: 4650f6d1419e675929133e46a91ca177
content_type: application/pdf
creator: dernst
date_created: 2024-08-05T08:25:18Z
date_updated: 2024-08-05T08:25:18Z
file_id: '17387'
file_name: 2024_SIGGRAPH_Rao.pdf
file_size: 59683257
relation: main_file
success: 1
file_date_updated: 2024-08-05T08:25:18Z
fulldoi: https://doi.org/10.1145/3641519.3657470
has_accepted_license: '1'
isi: 1
language:
- iso: eng
license: https://creativecommons.org/licenses/by/4.0/
month: '07'
oa: 1
oa_version: Published Version
publication: Proceedings - SIGGRAPH 2024 Conference Papers
publication_identifier:
isbn:
- '9798400705250'
publication_status: published
publisher: Association for Computing Machinery
quality_controlled: '1'
researchdata_availability: no
scopus_import: '1'
status: public
supplementarymaterial: yes
title: 'Lite2Relight: 3D-aware single image portrait relighting'
tmp:
image: /images/cc_by.png
legal_code_url: https://creativecommons.org/licenses/by/4.0/legalcode
name: Creative Commons Attribution 4.0 International Public License (CC-BY 4.0)
short: CC BY (4.0)
type: conference
user_id: 317138e5-6ab7-11ef-aa6d-ffef3953e345
year: '2024'
...