As the leading advocate for the Deaf and hard of hearing community, the National Foundation for Deaf and Hard of Hearing (NFDHH) has partnered with award-winning Kara Technologies to enable them to create AI avatars sign or translate in NZSL.
Accessibility in the Deaf community is about to be transformed, with the announcement that NZ Sign Language (NZSL) will be made available in a digital format.
As the leading advocate for the Deaf and hard of hearing community, the National Foundation for Deaf and Hard of Hearing (NFDHH) has partnered with award-winning Kara Technologies to enable them to create AI avatars sign or translate in NZSL.
The foundation has also committed to a world-first, ensuring the NZSL database will also include some of the well-known te reo Māori concepts used by the community, making all three official languages of New Zealand more cohesive.
“This is a vital project for our community to have equitable access to crucial information,” NFDHH chief executive Natasha Gallardo, who has severe hearing loss, says.
“Kara’s platform will bring an everyday communication used by more than 24,500 people into the digital age, 20 years after NZSL was made an official language of New Zealand.”
Kara, a Kiwi start up, will create a library of more than 10,000 NZSL signs, similar to work it has done digitising ASL (American Sign Language) and has underway for Auslan (Australian Sign Language).
Those signs cannot be used in Aotearoa as NZSL is unique. English, even in written form, is the second language of the Deaf community and NZSL’s culture, vocabulary, grammar and sentence structure differs. This can mean other digital tools such as Google and Apple translations are a challenge.
It is estimated there are more than 4,500 Deaf people who rely on NZSL as their first language, and the Stats NZ 2023 Census showed more than 24,500 people use NZSL to communicate with Deaf family, friends and colleagues.
NFDHH has followed Kara’s progress since it presented its technology at the foundation’s 2023 Listen Up conference and partnered with them earlier this year to scope this ambitious project.
The AI-enabled programme is the brainchild of engineer Arash Tayebi after he contracted Meniere’s Disease – an inner-ear condition that causes vertigo attacks and hearing loss – when he was a University of Auckland PhD student. He now has no hearing in his left ear.
“We want to get to a point where anyone around the world who is Deaf or uses any form of sign language can access any and all information in a matter of seconds,” Tayebi says.
“If our digital signing avatars can give generic information, we free up interpreters to do other work that’s really necessary – such as face-to-face doctor appointments and court interviews. We will never replace human interpreters, they are so valuable,” he says.
Kara’s communication and Deaf culture manager Grace Covey has a Cochlear implant, a device that stimulates sound in one ear. She feels privileged to have helped build the initial translations, as they will benefit her community – and her family.
“My two children are profoundly Deaf. When I think about their future, I think about how to improve the world for them and give them access that I didn’t have, how to remove the barriers I faced,” she says.
“This work means the Deaf community will finally have access to all the content and information that’s out there. We don’t have to wait for another Deaf person or interpreter to translate particular parts of information.”
Gallardo anticipates the library of signs will go live early next year. NFDHH and Kara are encouraging the Deaf community to give feedback on the user experience of three avatar prototypes – including the translation, accuracy and clarity, along with their facial expression and naturalness.
To view the avatar prototypes, go to nfdhh.org.nz
Kara uses motion capture (mo-cap) from a Deaf person signing, captured using 12 infrared cameras mapping movements of small reflective markers on a body-hugging suit. There are 72 data point dots in the hands alone.
Each sign is then edited and clipped into individual signs. The Deaf person reviews the content to make sure the signs are accurate and culturally appropriate.
The individual files are named and given parameters – actions, emotional context and examples of usage, so AI can learn how to apply them. Kara’s patented technology then blends these signs together so the digital avatar signs smoothly and naturally, like a fluent NZSL user.
