Justin, a man in a red shirt, sun glasses and baseball cap is stood holding on to his dog's lead. Behind Justin is a mountain view, and he is standing on a patch of ground surrounded by shrubs. PJ, the seeing eye dog, is a black lab, sniffing the ground excitedly.

Seeing AI: One App, Multiple Uses

Last year, Microsoft released Seeing AI, an app designed to assist blind people with common tasks. This app not only performs tasks that a blind person used to have to have multiple apps installed to use, it made all of these functionalities free.

One App to Rule Them All?

When I lost my sight and moved to the iPhone, I was introduced to the plethora of apps that were available to assist blind people with everyday tasks. Many of the apps that were available at the time, and still are, were paid apps, ranging from $10 to $100 for the KNFB Reader app.

Then, last year, Microsoft unveiled Seeing AI for the iPhone. This was a game changer. Not only did the app replace what I had used multiple apps to do in the past, it was free.

When the app initially came out, they released this video with it:

Channels

As the video above shows, someone can choose between different modes in the app by simply changing the channel. With VoiceOver enabled, all you have to do to change the channel is find the channel selector, then swipe up and down. Some of the channel options are as follows:

Short Text

The short text channel may be one of the most useful functions of the app. Simply point the camera at something and it will read any text it finds. No having to take a picture or anything, it just reads. This can be really useful for sorting through mail, reading signs, or anything where you don’t need an entire document read to you.

Document

The document channel is your full-fledged OCR (optical character recognition) option. The app will assist you in lining up a printed document so the camera captures the entire image, then say “hold steady” and take a picture automatically when it sees the entire document. It will then process the image and read the text to you.

Product

Not sure what that can you found in your cabinet is? Product can help with that. The app will make noises to help you locate the barcode on the can, or any other product, then automatically capture a picture of it and tell you what it is. I demonstrated this feature in a meeting at work a few months ago with a bottle of Dr. Pepper out of our vending machine and it only took a few seconds to tell us what it is.

Person

This channel will allow you to use the front or rear facing camera to identify people for you. It can be trained to recognize certain people, or will describe features of the person if it doesn’t recognize them.

When I was demonstrating this app in that meeting at work, my supervisor volunteered to let me take a picture of him. It said he was a 50 year old man with a beard wearing glasses. He’s actually 48, so it was pretty close that time around. The age guesses, however, aren’t always accurate. It generally thinks I’m in my mid to late 40’s, and I’m 33. It also said it thought my wife was 24 once. I’m not going to say how old she really is, but let’s just say we were both very happy with that number.

Currency

The currency channel will identify paper money for you. It currently supports US and Canadian Dollars, British Pounds, Euros, and indian Rupees. Just hold the bill in front of the camera and the phone will tell you the denomination. This comes in really handy for me at times, as US paper money does not currently have any tactile features.

Scene Preview

This allows you to take a picture of something and have it described to you. The best way to explain this one is to simply include a picture here and give you the description that the app gave me.

Justin, a man in a red shirt, sun glasses and baseball cap is stood holding on to his dog's lead. Behind Justin is a mountain view, and he is standing on a patch of ground surrounded by shrubs. PJ, the seeing eye dog, is a black lab, sniffing the ground excitedly.

My wife took this picture earlier this year when we were on vacation near Jackson Hole, Wyoming. The Grand Tetons are in the background. The App described this picture as “A brown and black dog standing on top of a mountain.” It’s not a ton of detail, but you at least get an idea what the picture is of.

Color Preview

With this channel, you can simply point the camera at something and it will tell you what color it sees. Keep in mind that colors may look different to the camera based on lighting or other conditions, so the color it gives you may not always be 100% accurate.

Handwriting Preview

OCR technology has always struggled with recognizing handwriting, but Seeing AI will still try to read it. This experimental channel will try to read handwriting. Note that the accuracy will vary based on the handwriting style of the person doing the writing and may not always be accurate.

Light

While on this channel, the phone will constantly emit a tone to indicate how bright what the camera sees is. The tone will be a lower pitch in darker areas, higher in brighter ones. This could be useful when trying to figure out if the lights are on or something like that.

Importing a Picture

Another really handy feature Seeing AI offers is the ability to import a picture from another app and recognize it. To do so, just pull up the “Share” menu for the image and select “Recognize with Seeing AI.” The app will process the image and provide information it finds. It will identify any people it finds, try to read any text to you, and describe the photo. This can come in really handy on places like Twitter, where you can share an image that hasn’t been described into the app and it can tell you all about it.

Amy always provides image descriptions in her tweets with pictures, but let’s run an image from one of her tweets through Seeing AI just to see what it says. Here’s the tweet:

Here’s how she described the image:

A man in sunglasses and an orange hi vis vest is kneeling on the ground. He is handling some stencil boards with cut out letters. On the pavement in front of him to his left a chalk message reads “want to help a disabled passenger #JustAskDontGrab”. It is a busy street and in the background feet, luggage and the road is visible.

And here’s what Seeing AI came up with:

Text al

Want to help

disabled passenger

#JustAskDontGrab

Worship TheGround

Scene probably a group of people on a street

As you can see, Amy’s description is much more detailed, but Seeing AI did pick up some text in the image and gave a brief description of the scene. While it’s far from ideal, it’s really not all that bad considering a computer came up with all that. I didn’t expect it to pick up that much text, considering it was all chalked on the ground. It will, though, pick up text in a meme or something like that much better.

Conclusion

As you can see from all the information above, the Seeing AI app provides a ton of features that can come in handy for someone who is blind or visually impaired. It’s available to download for free in the App Store.

About Justin: Justin works as an accessibility specialist for a Phoenix-area community college, specializing in digital accessibility. He lives in Phoenix with his wife Jennifer and Seeing Eye Dog PJ. He can be found on Twitter at @FatElvis04.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s