Invisible accessibility: Smartphones and AI

Apple focuses on accessibility through a closed system of its own chips and coordinated software. The data is processed directly on the device (on-device processing). This avoids delays in critical moments and ensures that sensitive health data and biometric information do not leave the iPhone.

One of the most fascinating examples is the LiDAR scanner in the Pro models of the iPhone and iPad. With the “Lupe” (Magnifier) app, the camera essentially becomes a 3D radar thanks to laser pulses. The “Door Detection” feature solves a major problem for blind people: navigating the last few meters. The system calculates the exact distance to the door, reads signs using optical recognition, and tells you whether you need to push or pull the door.

For people with paralysis, Apple has integrated eye-tracking technology. You no longer need expensive external sensors – the iPhone simply uses the front camera and machine learning to track the micro-movements of your pupils. You move the virtual cursor across the screen with just your gaze. By focusing on a point, you simulate a finger tap, maintaining full control over your apps. To help people with atypical speech, there are now “Vocal Shortcuts” (voice shortcuts). This allows you to define your own sounds or phrases that the iPhone recognizes as complex system commands.

Another hidden superpower of Apple hardware is the Taptic Engine, a brilliant vibration motor. With “Music Haptics,” Apple has made it possible for deaf and hard-of-hearing people to truly “feel” music. The system translates rhythm, bass, and vocals into perceptible impulses in real time. This transforms listening to music on your phone into a completely new, multisensory experience.

“iks.haus / AI-generated with DALL-E 3”

When someone loses the ability to speak, for example, due to diseases like ALS or surgery, it is a significant loss of autonomy. You lose not only the means of communication but also a part of your identity that is contained in your own voice. Apple’s response to this is “Personal Voice” (your own voice).

You only need to read randomly generated sentences for just under 15 minutes. The system then uses complex deep neural networks to analyze your speech patterns and create a unique artificial clone of your voice that accurately reproduces your intonation. This clone is seamlessly integrated into the live speech output. You simply type what you want to say, and your iPhone or Mac will speak it during phone calls, FaceTime, or in direct conversations using your own “living” voice.

Unlike Apple’s closed system, where hardware and software work perfectly together and data is processed directly on the device, Google relies on the enormous computing power of the cloud and brings open-source AI to Android devices worldwide.

The screen reader TalkBack has received a massive upgrade through the direct integration of the Gemini Nano model. Instead of simply telling you “photo of a dog,” TalkBack now provides blind people with extremely detailed, multimodal photo descriptions. It can identify the material of clothing, the emotions people are showing in photos, and the structure of buildings – and all of this is done completely offline to protect privacy.

In the real world, Google is going even further with Project Astra. Developed in collaboration with the translation service Aira, this AI uses the phone’s camera or a smart glasses to check the surroundings in real time in a video stream. Unlike with photos, Astra can tell a blind person in real time exactly where a desired object is located, warn of obstacles in the way, and have intelligent conversations about the surroundings. At the same time, Guided Frame takes on the role of the photographer and helps blind people to perfectly position their faces for a selfie or a group photo using audio cues and vibrations.

One of the most significant technological breakthroughs is Project Relate. This app was specifically developed for people with dysarthria and severely impaired speech, such as after a stroke or with muscular dystrophy. Colin Hughes, a former BBC producer with muscular dystrophy who uses a breathing mask, said that conventional dictation devices were useless to him. However, after Project Relate was trained on his specific voice, it could easily ignore the humming of his ventilator and translate his words into clean text with remarkable accuracy.

Android operating systems are increasingly integrating with medical aids. Android 16 and 17 brought significant improvements to LE Audio (Low Energy Audio). People who are deaf or hard of hearing can now transmit system sounds directly from their mobile phone to their hearing aids. Even better, with Android 16, you can use your smartphone as an external directional microphone. Simply point it at the person you are talking to in a noisy restaurant, and the phone wirelessly transmits their clear voice to your hearing aid, while filtering out all the background noise.

iksHaus | KI-generiert nach redaktioneller Konzeption, 2026

“iks.haus / AI-generated with DALL-E 3”

Because blind users are completely reliant on the voice prompts from their phones, there is a safety issue on the street. If you wear traditional headphones that completely seal the ear canal, it can be life-threatening if you have to rely on your hearing to cross the street.

The solution originally comes from the military: wireless bone conduction headphones. They have a lightweight frame and sit in front of the ear, directly on the cheekbone. The sound is transmitted directly to the inner ear through micro-vibrations. The eardrum is completely bypassed, and the ear canal remains open. This allows a blind person to hear the GPS prompts from their phone very clearly, while still being able to hear all the cars, the instructions from their guide dog, and the tapping of their walking stick.

While these Silicon Valley innovations may sound like science fiction, the gap between users worldwide is constantly widening. The shift to large language models (LLMs) has significantly increased the digital language barrier. People who speak “major” languages like English or Spanish are being given access to futuristic tools, while users in markets like the Croatian-speaking region are being left behind.

Apple’s Personal Voice, which restores the voice to terminally ill individuals, simply does not exist in Croatian. The same applies to Google’s Project Relate, which brilliantly understands atypical language, but is officially still primarily limited to English-speaking countries. Users in these countries are being cut off from life-changing technological aids, simply because the tech giants lack the financial incentive to address local languages.

However, a significant ray of hope comes from the legal pressure exerted by the European Accessibility Act (EAA), whose strict rules will apply throughout Europe from June 28, 2025. This law obligates private companies and the public sector throughout the EU. All digital products – whether smartphones, online shops, ATMs, or public transport apps – must then meet high standards for accessibility (based on WCAG and EN 301 549). Those who do not comply risk hefty fines or being removed from the European market.

The EAA sends a clear message: AI and hidden tools in smartphones are no excuse for sloppy, non-inclusive app design. Even though modern AI can easily scan and describe photos, companies are still legally obligated to manually implement proper alt text and a clean, accessible architecture. Perfect “hidden technology” must never shift the responsibility onto the user with a disability. It should simply be an invisible helper that compensates for the errors of our society and guarantees everyone the right to an independent and dignified life.