LKG-Corpus – Linguistic, Kinematic and Gaze Information in Task Descriptions

The LKG Corpus comprises multimodal recordings from 22 persons performing pick and place (take, put, push) actions, using eye tracking, motion capture, and recordings of the self-explaining utterances made while the actions are performed. The motivation for recording this kind of dataset was to study potential correlations between visible motor activities, eye gaze and linguistic structuring.

The data are licensed under Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0). This license allows you to use the data free of charge for non-commercial purposes. You may modify and redistribute the data as long as you keep the attribution to the original in all the files, publish your work under the same licencse and cite the LREC 2020 paper listed below.

The dataset consists of the following information:

• manual orthographic transcriptions of speech (conducted in Praat)
• a video from the participants perspective, automatically overlaid with a circle indicating the target of the participants’ gaze by SMI software
• schematics of the gaze vector every 5 frames (see Reinboth et al. 2020, Fig. 4), also compiled into an animation of gaze behaviour
• information on gaze behaviours, for example whether a participant is looking at/close to an object (manually annotated)
• information on whether the user’s dominant hand touches an object (manually annotated)
• information on when an object is being moved
• information on when participants are looking at the point they will next move an object to (manually annotated)
• information on when participants’ gaze was stable on one point in space, indicating a visual ‘fixation’

Publications

Tim Reinboth, Stephanie Gross, Laura Bishop, Brigitte Krenn. "Linguistic, Kinematic and Gaze Information in Task descriptions. The LKG-Corpus." LREC 2020.

Acknowledgments

Data collection was conducted at the Performance Science Lab of the Department of Music Acoustics – Wiener Klangstil, University of Music and Performing Arts Vienna (mdw).

Authors

• Tim Reinboth
• Brigitte Krenn
• Stephanie Gross
• Laura Bishop

Licence

Sponsors