Lego Audio and Braille Building Instructions

Inspired by blind entrepreneur, Matthew Shifrin, the new building instructions provided by the LEGO Group will help children with vision impairment build and learn through play using LEGO bricks. This project was developed in collaboration with the Austrian Research Institute for Artificial Intelligence (OFAI) in Vienna. OFAI can look back on 35 successful years in applied research in many fields of AI.

The idea comes from Matthew Shifrin, who was born blind. As a child, he developed a strong passion for LEGO play. However, he always needed assistance when it came to specific LEGO building instructions. He had a friend, Lilya Finkel, who started to write down all the building steps for him so that he could upload them into a system that allowed him to read the building steps on a Braille reader through his fingers. By that he was able to build LEGO models on his own - even really large ones such as the London Tower Bridge. Matthew made the instructions available for the public and got an impressive echo from the community, asking for more instructions.

Through a friend at the MIT (Massachusetts Institute of Technology) Media Lab, he was then introduced to the Creative Play Lab at the LEGO Group. The team took his idea to the Austrian Research Institute for Artificial Intelligence who developed new AI software to translate LXFML data (LEGO Exchange Format Mel Script) from the visual digital building instructions to text based descriptions for braille and voice commanded instructions. Speech synthesis company, CereProc then provided the Software Development Kit and the voice that reads the instructions. The automatically generated instructions are only one part of a greater whole - it is essential to provide the users with a narrative with detailed descriptions of the individual components: from describing how a graffiti wall looks like to suggestions what you could do with a small cart.

Available in English as a free service for all through the website, the first four instructions have been made accessible on Aug. 28th 2019: LEGO® Audio & Braille Building Instructions. OFAI is very glad to have been able to contribute to the success of this important project.

Links

• Try out: LEGO® Classic Bricks & Ideas
• More info: LEGO Group: Pilot Project Launch

Gallery

Research staff

• Susanne Höfler
• Friedrich Neubarth
• Marcin Skowron