Tech4Good Accessibility Award 2018: Facebook reaches finals with Automatic Alt Text and image recognition features

More than one billion photos are shared on Facebook every day, and artificial intelligence is now making many more of those images accessible to Facebook users with the development of Automatic Alt Text.

Through research with the vision loss community, Facebook learned that for users of screen readers, who are blind or have vision loss, it wasn't straightforward to know what was in a photo that arrived in their News Feed. 

So in 2016, Facebook launched Automatic Alt Text (AAT) - a feature that uses object recognition technology to describe photos to people who are blind or who have low vision and use screen readers. And at the end of 2017, the social media platform also launched a Face Recognition tool that tells people using screen readers who appears in photos in their News Feed, even if they aren’t tagged (as long as that person has allowed this option in their settings).

Facebook labelling an image using Automatic Alt Text

Facebook’s efforts to make the social media platform more accessible to people who have sight loss and those who are blind has earned them a place in the finals of the Tech4Good Accessibility Award - with the winner announced at BT Centre on 17 July.

Now in their eighth year the awards are supported by BT and celebrate some of the amazing people who use tech to help make the world a better place.

Robin Christopherson, head of digital inclusion at AbilityNet commented: "In a popular social media platform like Facebook, it’s impossible to expect every user to add descriptions to their images. This smart use of AI gets around that problem and is a game changer for disabled users."

Using artificial intelligence and machine-learning for accessibility

AAT was developed by programming machines using AI and was updated and refined based on feedback from multiple rounds of user research. The system can currently detect more than one hundred concepts, such as the number of people in a photo, whether people are smiling and physical objects like a “car”, “tree”, “mountain”, and others. Currently, about 75% of photos on Facebook now have at least one image identified by AAT.

The platform's Face Recognition technology analyses the pixels in photos and videos, such as a user’s profile picture and photos and videos that the user has been tagged in, to calculate a unique number, which is called a template.  When photos and videos are uploaded to Facebook’s systems, those images are compared to the template to find matches.  With this technology, people using screen readers can know who appears in photos in their News Feed, even if they aren’t tagged.

Manual alt text

The traditional mechanism for describing photos to people with vision loss is the use of alt text. Traditionally, alt text requires that the content creator/ person who uploads an image include a secondary description (as well a written post) for each photo, which is then read by a screen reader.  This is time-consuming for the person posting and is also extremely uncommon as too many people don't know about the value of alt-text.

To address this challenge, Facebook created automatic tools powered by AI to describe photos on Facebook, which allow Facebook to dramatically increase the number of photos that have supplemental text descriptions. Facebook told Tech4Good judges that as it continues to improve its object and face recognition services, AAT and Face Recognition will continue to provide more descriptive narratives for visual content.

Vote now in the Tech4Good People's Award

The People’s Award is one of Tech4Good's most sought Awards because it is chosen by the public, voting online via Instagram, Twitter or on the tech4Good website.