---
_id: '14488'
abstract:
- lang: eng
  text: 'Portrait viewpoint and illumination editing is an important problem with
    several applications in VR/AR, movies, and photography. Comprehensive knowledge
    of geometry and illumination is critical for obtaining photorealistic results.
    Current methods are unable to explicitly model in 3D while handling both viewpoint
    and illumination editing from a single image. In this paper, we propose VoRF,
    a novel approach that can take even a single portrait image as input and relight
    human heads under novel illuminations that can be viewed from arbitrary viewpoints.
    VoRF represents a human head as a continuous volumetric field and learns a prior
    model of human heads using a coordinate-based MLP with individual latent spaces
    for identity and illumination. The prior model is learned in an auto-decoder manner
    over a diverse class of head shapes and appearances, allowing VoRF to generalize
    to novel test identities from a single input image. Additionally, VoRF has a reflectance
    MLP that uses the intermediate features of the prior model for rendering One-Light-at-A-Time
    (OLAT) images under novel views. We synthesize novel illuminations by combining
    these OLAT images with target environment maps. Qualitative and quantitative evaluations
    demonstrate the effectiveness of VoRF for relighting and novel view synthesis,
    even when applied to unseen subjects under uncontrolled illumination. This work
    is an extension of Rao et al. (VoRF: Volumetric Relightable Faces 2022). We provide
    extensive evaluation and ablative studies of our model and also provide an application,
    where any face can be relighted using textual input.'
acknowledgement: Open Access funding enabled and organized by Projekt DEAL.
article_processing_charge: Yes (via OA deal)
article_type: original
author:
- first_name: Pramod
  full_name: Rao, Pramod
  last_name: Rao
- first_name: B. R.
  full_name: Mallikarjun, B. R.
  last_name: Mallikarjun
- first_name: Gereon
  full_name: Fox, Gereon
  last_name: Fox
- first_name: Tim
  full_name: Weyrich, Tim
  last_name: Weyrich
- first_name: Bernd
  full_name: Bickel, Bernd
  id: 49876194-F248-11E8-B48F-1D18A9856A87
  last_name: Bickel
  orcid: 0000-0001-6511-9385
- first_name: Hanspeter
  full_name: Pfister, Hanspeter
  last_name: Pfister
- first_name: Wojciech
  full_name: Matusik, Wojciech
  last_name: Matusik
- first_name: Fangneng
  full_name: Zhan, Fangneng
  last_name: Zhan
- first_name: Ayush
  full_name: Tewari, Ayush
  last_name: Tewari
- first_name: Christian
  full_name: Theobalt, Christian
  last_name: Theobalt
- first_name: Mohamed
  full_name: Elgharib, Mohamed
  last_name: Elgharib
citation:
  ama: Rao P, Mallikarjun BR, Fox G, et al. A deeper analysis of volumetric relightiable
    faces. <i>International Journal of Computer Vision</i>. 2024;132:1148-1166. doi:<a
    href="https://doi.org/10.1007/s11263-023-01899-3">10.1007/s11263-023-01899-3</a>
  apa: Rao, P., Mallikarjun, B. R., Fox, G., Weyrich, T., Bickel, B., Pfister, H.,
    … Elgharib, M. (2024). A deeper analysis of volumetric relightiable faces. <i>International
    Journal of Computer Vision</i>. Springer Nature. <a href="https://doi.org/10.1007/s11263-023-01899-3">https://doi.org/10.1007/s11263-023-01899-3</a>
  chicago: Rao, Pramod, B. R. Mallikarjun, Gereon Fox, Tim Weyrich, Bernd Bickel,
    Hanspeter Pfister, Wojciech Matusik, et al. “A Deeper Analysis of Volumetric Relightiable
    Faces.” <i>International Journal of Computer Vision</i>. Springer Nature, 2024.
    <a href="https://doi.org/10.1007/s11263-023-01899-3">https://doi.org/10.1007/s11263-023-01899-3</a>.
  ieee: P. Rao <i>et al.</i>, “A deeper analysis of volumetric relightiable faces,”
    <i>International Journal of Computer Vision</i>, vol. 132. Springer Nature, pp.
    1148–1166, 2024.
  ista: Rao P, Mallikarjun BR, Fox G, Weyrich T, Bickel B, Pfister H, Matusik W, Zhan
    F, Tewari A, Theobalt C, Elgharib M. 2024. A deeper analysis of volumetric relightiable
    faces. International Journal of Computer Vision. 132, 1148–1166.
  mla: Rao, Pramod, et al. “A Deeper Analysis of Volumetric Relightiable Faces.” <i>International
    Journal of Computer Vision</i>, vol. 132, Springer Nature, 2024, pp. 1148–66,
    doi:<a href="https://doi.org/10.1007/s11263-023-01899-3">10.1007/s11263-023-01899-3</a>.
  short: P. Rao, B.R. Mallikarjun, G. Fox, T. Weyrich, B. Bickel, H. Pfister, W. Matusik,
    F. Zhan, A. Tewari, C. Theobalt, M. Elgharib, International Journal of Computer
    Vision 132 (2024) 1148–1166.
das_tickbox: '1'
dataavailabilitystatement: 'Due to privacy concerns, we cannot make the dataset used
  in our project publicly available. However, to demonstrate the effectiveness of
  our proposed method, we evaluate our approach using publicly available datasets
  such as H3DS (Ramon et al., 2021): https://github.com/CrisalixSA/h3ds, FFHQ (Karras
  et al., 2021): https://github.com/NVlabs/ffhq-dataset, and CelebA (Liu et al., 2015):
  https://mmlab.ie.cuhk.edu.hk/projects/CelebA.html. Using these datasets allows us
  to evaluate the generalization ability of our proposed method on unseen data. H3DS
  provides high-quality 3D scans of human faces, FFHQ contains high-resolution facial
  images, and CelebA is a large-scale dataset of celebrity faces. We use these datasets
  to evaluate the performance of our proposed method in various scenarios, such as
  face rotation and relighting, and compare them with state-of-the-art methods.'
date_created: 2023-11-05T23:00:54Z
date_published: 2024-04-01T00:00:00Z
date_updated: 2026-09-24T10:59:50Z
day: '01'
ddc:
- '000'
department:
- _id: BeBi
doi: 10.1007/s11263-023-01899-3
external_id:
  isi:
  - '001091935600002'
  pmid:
  - '38549787'
file:
- access_level: open_access
  checksum: 5eef1d920f6fe700d7856098000d05f1
  content_type: application/pdf
  creator: dernst
  date_created: 2024-07-22T11:07:14Z
  date_updated: 2024-07-22T11:07:14Z
  file_id: '17304'
  file_name: 2024_IJCV_Rao.pdf
  file_size: 9942520
  relation: main_file
  success: 1
file_date_updated: 2024-07-22T11:07:14Z
fulldoi: https://doi.org/10.1007/s11263-023-01899-3
has_accepted_license: '1'
intvolume: '       132'
isi: 1
language:
- iso: eng
month: '04'
oa: 1
oa_version: Published Version
page: 1148-1166
pmid: 1
publication: International Journal of Computer Vision
publication_identifier:
  eissn:
  - 1573-1405
  issn:
  - 0920-5691
publication_status: published
publisher: Springer Nature
quality_controlled: '1'
researchdata_availability: no (restricted access)
scopus_import: '1'
status: public
supplementarymaterial: no
title: A deeper analysis of volumetric relightiable faces
tmp:
  image: /images/cc_by.png
  legal_code_url: https://creativecommons.org/licenses/by/4.0/legalcode
  name: Creative Commons Attribution 4.0 International Public License (CC-BY 4.0)
  short: CC BY (4.0)
type: journal_article
user_id: 317138e5-6ab7-11ef-aa6d-ffef3953e345
volume: 132
year: '2024'
...
