Research Output
From the Virtual to the RealWorld: Referring to Objects in Real-World Spatial Scenes
  Predicting the success of referring expressions (RE) is vital for real world applications such as navigation systems. Traditionally, research has focused on studying Referring Expression Generation (REG) in virtual, controlled environments. In this paper, we describe a novel study of spatial references from real scenes rather than virtual. First, we investigate how humans describe objects in open, uncontrolled scenarios and compare our findings to those reported in virtual environments. We show that REs in real-world scenarios differ significantly to those in virtual worlds. Second, we propose a novel approach to quantifying image complexity when complete annotations are not present (e.g. due to poor object recognition capabitlities), and third, we present a model for success prediction of REs for objects in real scenes. Finally, we discuss implications for Natural Language Generation (NLG) systems and future directions.

  • Date:

    17 September 2015

  • Publication Status:


  • Publisher

    Association for Computational Linguistics (ACL)

  • DOI:


  • Library of Congress:

    QA75 Electronic computers. Computer science

  • Dewey Decimal Classification:

    006.3 Artificial intelligence

  • Funders:

    Engineering and Physical Sciences Research Council


Gkatzia, D., Rieser, V., Bartie, P., & Mackaness, W. (2015). From the Virtual to the RealWorld: Referring to Objects in Real-World Spatial Scenes. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 1936-1942.



Referring expressions, RE, navigation systems, virtual environments, natural language generation, NLG.

Monthly Views:

Available Documents