Proposal of a Generative Model of Event-based Representations for Grounded Language Understanding

Simon Brodeur, Luca Celotti, Jean Rouat


Grounding is the problem of correspondence between the symbolic concepts of language and the physical environment. The research direction that we propose to tackle language acquisition and grounding is based on multimodal event-based representations and probabilistic generative modeling. First, we establish a new multimodal dataset recorded from a mobile robot and describe how such multimodal signals can be efficiently encoded into compact, event-based representations using sparse coding. We highlight how they could be better suited to ground concepts. We then describe a generative probabilistic model based on those event-based representations. We discuss possible applications of this probabilistic framework in the context of a cognitive agent, such as detecting novelty at the inputs or reasoning by building internal simulations of the environment. While this work is still in progress, this could open new perspectives on how representational learning can play a key role in the ability to map structures of the multimodal scene to language.


 DOI: 10.21437/GLU.2017-14

Cite as: Brodeur, S., Celotti, L., Rouat, J. (2017) Proposal of a Generative Model of Event-based Representations for Grounded Language Understanding. Proc. GLU 2017 International Workshop on Grounding Language Understanding, 68-72, DOI: 10.21437/GLU.2017-14.


@inproceedings{Brodeur2017,
  author={Simon Brodeur and Luca Celotti and Jean Rouat},
  title={Proposal of a Generative Model of Event-based Representations for Grounded Language Understanding},
  year=2017,
  booktitle={Proc. GLU 2017 International Workshop on Grounding Language Understanding},
  pages={68--72},
  doi={10.21437/GLU.2017-14},
  url={http://dx.doi.org/10.21437/GLU.2017-14}
}