---
OA_place: publisher
OA_type: free access
_id: '18222'
abstract:
- lang: eng
text: "STUDY QUESTION: What is the accuracy and agreement of embryologists when
assessing the implantation probability of blastocysts using time-lapse imaging
(TLI), and can it be improved with a data-driven algorithm?\r\n\r\nSUMMARY ANSWER:
The overall interobserver agreement of a large panel of embryologists was moderate
and prediction accuracy was modest, while the purpose-built artificial intelligence
model generally resulted in higher performance metrics.\r\n\r\nWHAT IS KNOWN ALREADY:
Previous studies have demonstrated significant interobserver variability amongst
embryologists when assessing embryo quality. However, data concerning embryologists’
ability to predict implantation probability using TLI is still lacking. Emerging
technologies based on data-driven tools have shown great promise for improving
embryo selection and predicting clinical outcomes.\r\n\r\nSTUDY DESIGN, SIZE,
DURATION: TLI video files of 136 embryos with known implantation data were retrospectively
collected from two clinical sites between 2018 and 2019 for the performance assessment
of 36 embryologists and comparison with a deep neural network (DNN).\r\n\r\nPARTICIPANTS/MATERIALS,
SETTING, METHODS: We recruited 39 embryologists from 13 different countries. All
participants were blinded to clinical outcomes. A total of 136 TLI videos of embryos
that reached the blastocyst stage were used for this experiment. Each embryo’s
likelihood of successfully implanting was assessed by 36 embryologists, providing
implantation probability grades (IPGs) from 1 to 5, where 1 indicates a very low
likelihood of implantation and 5 indicates a very high likelihood. Subsequently,
three embryologists with over 5 years of experience provided Gardner scores. All
136 blastocysts were categorized into three quality groups based on their Gardner
scores. Embryologist predictions were then converted into predictions of implantation
(IPG ≥ 3) and no implantation (IPG ≤ 2). Embryologists’ performance and agreement
were assessed using Fleiss kappa coefficient. A 10-fold cross-validation DNN was
developed to provide IPGs for TLI video files. The model’s performance was compared
to that of the embryologists.\r\n\r\nMAIN RESULTS AND THE ROLE OF CHANCE: Logistic
regression was employed for the following confounding variables: country of residence,
academic level, embryo scoring system, log years of experience and experience
using TLI. None were found to have a statistically significant impact on embryologist
performance at α = 0.05. The average implantation prediction accuracy for the
embryologists was 51.9% for all embryos (N = 136). The average accuracy of the
embryologists when assessing top quality and poor quality embryos (according to
the Gardner score categorizations) was 57.5% and 57.4%, respectively, and 44.6%
for fair quality embryos. Overall interobserver agreement was moderate (κ = 0.56,
N = 136). The best agreement was achieved in the poor + top quality group (κ = 0.65,
N = 77), while the agreement in the fair quality group was lower (κ = 0.25, N = 59).
The DNN showed an overall accuracy rate of 62.5%, with accuracies of 62.2%, 61%
and 65.6% for the poor, fair and top quality groups, respectively. The AUC for
the DNN was higher than that of the embryologists overall (0.70 DNN vs 0.61 embryologists)
as well as in all of the Gardner groups (DNN vs embryologists—Poor: 0.69 vs 0.62;
Fair: 0.67 vs 0.53; Top: 0.77 vs 0.54).\r\n\r\nLIMITATIONS, REASONS FOR CAUTION:
Blastocyst assessment was performed using video files acquired from time-lapse
incubators, where each video contained data from a single focal plane. Clinical
data regarding the underlying cause of infertility and endometrial thickness before
the transfer was not available, yet may explain implantation failure and lower
accuracy of IPGs. Implantation was defined as the presence of a gestational sac,
whereas the detection of fetal heartbeat is a more robust marker of embryo viability.
The raw data were anonymized to the extent that it was not possible to quantify
the number of unique patients and cycles included in the study, potentially masking
the effect of bias from a limited patient pool. Furthermore, the lack of demographic
data makes it difficult to draw conclusions on how representative the dataset
was of the wider population. Finally, embryologists were required to assess the
implantation potential, not embryo quality. Although this is not the traditional
approach to embryo evaluation, morphology/morphokinetics as a means of assessing
embryo quality is believed to be strongly correlated with viability and, for some
methods, implantation potential.\r\n\r\nWIDER IMPLICATIONS OF THE FINDINGS: Embryo
selection is a key element in IVF success and continues to be a challenge. Improving
the predictive ability could assist in optimizing implantation success rates and
other clinical outcomes and could minimize the financial and emotional burden
on the patient. This study demonstrates moderate agreement rates between embryologists,
likely due to the subjective nature of embryo assessment. In particular, we found
that average embryologist accuracy and agreement were significantly lower for
fair quality embryos when compared with that for top and poor quality embryos.
Using data-driven algorithms as an assistive tool may help IVF professionals increase
success rates and promote much needed standardization in the IVF clinic. Our results
indicate a need for further research regarding technological advancement in this
field."
article_processing_charge: No
article_type: original
author:
- first_name: Daniel E
full_name: Fordham, Daniel E
last_name: Fordham
- first_name: Dror
full_name: Rosentraub, Dror
last_name: Rosentraub
- first_name: Avital L
full_name: Polsky, Avital L
last_name: Polsky
- first_name: Talia
full_name: Aviram, Talia
last_name: Aviram
- first_name: Yotam
full_name: Wolf, Yotam
last_name: Wolf
- first_name: Oriel
full_name: Perl, Oriel
last_name: Perl
- first_name: Asnat
full_name: Devir, Asnat
last_name: Devir
- first_name: Shahar
full_name: Rosentraub, Shahar
last_name: Rosentraub
- first_name: David H
full_name: Silver, David H
last_name: Silver
- first_name: Yael
full_name: Gold Zamir, Yael
last_name: Gold Zamir
- first_name: Alexander
full_name: Bronstein, Alexander
id: 58f3726e-7cba-11ef-ad8b-e6e8cb3904e6
last_name: Bronstein
orcid: 0000-0001-9699-8730
- first_name: Miguel
full_name: Lara Lara, Miguel
last_name: Lara Lara
- first_name: Jara
full_name: Ben Nagi, Jara
last_name: Ben Nagi
- first_name: Adrian
full_name: Alvarez, Adrian
last_name: Alvarez
- first_name: Santiago
full_name: Munné, Santiago
last_name: Munné
citation:
ama: 'Fordham DE, Rosentraub D, Polsky AL, et al. Embryologist agreement when assessing
blastocyst implantation probability: Is data-driven prediction the solution to
embryo assessment subjectivity? Human Reproduction. 2022;37(10):2275-2290.
doi:10.1093/humrep/deac171'
apa: 'Fordham, D. E., Rosentraub, D., Polsky, A. L., Aviram, T., Wolf, Y., Perl,
O., … Munné, S. (2022). Embryologist agreement when assessing blastocyst implantation
probability: Is data-driven prediction the solution to embryo assessment subjectivity?
Human Reproduction. Oxford University Press. https://doi.org/10.1093/humrep/deac171'
chicago: 'Fordham, Daniel E, Dror Rosentraub, Avital L Polsky, Talia Aviram, Yotam
Wolf, Oriel Perl, Asnat Devir, et al. “Embryologist Agreement When Assessing Blastocyst
Implantation Probability: Is Data-Driven Prediction the Solution to Embryo Assessment
Subjectivity?” Human Reproduction. Oxford University Press, 2022. https://doi.org/10.1093/humrep/deac171.'
ieee: 'D. E. Fordham et al., “Embryologist agreement when assessing blastocyst
implantation probability: Is data-driven prediction the solution to embryo assessment
subjectivity?,” Human Reproduction, vol. 37, no. 10. Oxford University
Press, pp. 2275–2290, 2022.'
ista: 'Fordham DE, Rosentraub D, Polsky AL, Aviram T, Wolf Y, Perl O, Devir A, Rosentraub
S, Silver DH, Gold Zamir Y, Bronstein AM, Lara Lara M, Ben Nagi J, Alvarez A,
Munné S. 2022. Embryologist agreement when assessing blastocyst implantation probability:
Is data-driven prediction the solution to embryo assessment subjectivity? Human
Reproduction. 37(10), 2275–2290.'
mla: 'Fordham, Daniel E., et al. “Embryologist Agreement When Assessing Blastocyst
Implantation Probability: Is Data-Driven Prediction the Solution to Embryo Assessment
Subjectivity?” Human Reproduction, vol. 37, no. 10, Oxford University Press,
2022, pp. 2275–90, doi:10.1093/humrep/deac171.'
short: D.E. Fordham, D. Rosentraub, A.L. Polsky, T. Aviram, Y. Wolf, O. Perl, A.
Devir, S. Rosentraub, D.H. Silver, Y. Gold Zamir, A.M. Bronstein, M. Lara Lara,
J. Ben Nagi, A. Alvarez, S. Munné, Human Reproduction 37 (2022) 2275–2290.
date_created: 2024-10-08T12:53:20Z
date_published: 2022-10-01T00:00:00Z
date_updated: 2024-10-14T09:54:40Z
day: '01'
doi: 10.1093/humrep/deac171
extern: '1'
fulldoi: https://doi.org/10.1093/humrep/deac171
intvolume: ' 37'
issue: '10'
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://doi.org/10.1093/humrep/deac171
month: '10'
oa: 1
oa_version: Published Version
page: 2275-2290
publication: Human Reproduction
publication_identifier:
eissn:
- 1460-2350
issn:
- 0268-1161
publication_status: published
publisher: Oxford University Press
quality_controlled: '1'
scopus_import: '1'
status: public
title: 'Embryologist agreement when assessing blastocyst implantation probability:
Is data-driven prediction the solution to embryo assessment subjectivity?'
type: journal_article
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 37
year: '2022'
...