What These Engineers Are Doing To Make Facebook More Accessible To Visually Impaired Users

For many of us, accessing our favorite social media sites is as easy as opening our browser or mobile app. We log in and expect it to simply work. But for blind and visually impaired users who work with screen reader software, accessibility is far from simple.

With websites like Facebook, accessibility has to be coded into the back end. Their content must be able to be interpreted by a screen reader, which outputs the content either by reading it out loud, or by showing it on a refreshable Braille display. That's why, when Facebook engineer Matt King, who is blind, joined Facebook as a user in 2009, the site was virtually impossible for him to use, he tells Bustle. At the time, Facebook was becoming more and more visual, and King's screen reader software could not describe to him posts that were thick with images. So when Facebook recruited him onto its accessibility team in 2015, he became one of the people who works every day to make Facebook more accessible for people with disabilities, including helping to launch a game-changing photo description system that has now been deployed across Facebook products.

Without back-end accessibility coding, screen readers often produce "gobbledygook" for large swathes of websites, King says. Before Facebook began throwing serious developmental muscle behind accessibility, users like King would have to deal with their screen readers interpreting pictures as "photo" or something like "GX_23789ABCD.jpg, because it was the name of the image that was there," King explains.

But with the system King helped to launch, users with screen readers are provided with descriptions of photos. At first launch in 2016, the system was only able to provide phrases like "image may contain two people outdoors, [...] water," King tells Bustle. "Pretty crude phrase, but a lot better than 'photo'."

This early model was a big enough step in the right direction that positive feedback poured in from users, and the team expanded the capability from Newsfeed to other products like Groups and Files. Now, the system is available in 25 languages "across most of the products that Facebook offers," King says.

And Facebook's accessibility team continues to flesh it out. In early 2017, they were able to give the AI the capability to recognize actions, so the system now recognizes 17 different actions like "walking, eating, playing musical instrument, onstage, and a few other things like that," King explains. So now the phrase "image may contain two people outdoors" has been upgraded to, "two people standing outdoors," he says.

Most recently, in December 2017, King's team added the ability for users to hear the names of people who are in photos thanks to facial recognition. King clarifies that only a user's friends who have facial recognition turned on in their settings will be detected by the AI, and says it can be a huge help for people who are visually impaired. "If you're my friend, please have that facial recognition setting enabled," he says.

Part of sighted users improving accessibility for their blind and visually impaired friends involves understanding that blind and visually impaired people use computers in the first place — which not all non-disabled people necessarily understand. King says people often seem incredulous when he tells them he works with computers. "I started using a computer as a blind person back in 1985," he says. He used computers in college, and then as an electrical engineer at IBM, where he says he became aware of just how simple it was for sighted people to do the same work that required King to have very specialized knowledge.

"It bothered me that there were so many blind people that I was trying to help, like peers at IBM, for example," he explains. "I was just trying to help them do their job better, and they had to learn so much just to do some of the basics on the computer."

Screen reader software is an essential tool for visually impaired computer users, but having a screen reader doesn't make up for websites that don't consider accessibility. On websites where a screen reader can't pick up on much of the content, particularly visual content, "You just have no concept of what's in [images], so there's no way to interact and be part of the conversation," King explains. "It's like there's this room with glass walls and you're on the outside, and everyone else is inside having their conversation. You can see the conversation going on, but you have no idea what they're talking about."

With programs like word processors, screen readers don't have much heavy lifting to do, but when it comes to websites, if accessibility isn't done on the back end, using what King calls the "plumbing of accessibility," then screen readers will be stymied, rendering content inaccessible to blind and visually impaired users.

Turning on facial recognition is one of the ways Facebook users can help their blind and visually impaired friends right now. And though King and his team are continually building up the photo description system, users can also take a few moments to write their own descriptions in photo posts. A simple sentence like, "My daughter riding a gray horse in a field on a sunny day," helps improve visually impaired users' experience.

On the Facebook accessibility team's "roadmap for the future" is adding the names of celebrities to facial recognition in photos, adding "concepts" to the photo description system so it will be able to recognize and describe more things, and potentially offering a functionality where a user would be able to have a conversation with Facebook's AI about a photo.

King says that feature is "a long way from being a product," but it would involve being able to "ask the picture a question like, 'What color are the curtains?' Just any arbitrary question, and the system would try to answer that," he explains.

And when it comes to spreading the word about how accessible technology works, King says, "Anything that raises the awareness of accessibility, this whole idea that we can bring the media world to life, getting everybody to spread awareness of accessibility all at once [...] It really helps if the general public understands how important equal access to information and technology is for people with disabilities."

Folks like King work on improving accessibility every day, but it's also up to non-disabled people to be aware of the challenges created by a world that is rarely accessible to all.