Designing Audio-Haptic Experiences

I'm sure you're familiar with that sound.

It's been part of our life for years. In 2019 though, I think we can do better.

I'm Camille Moussette, interaction designer on the Apple design team.

And, I'm Hugo Verweij, sound designer on the design team.

This session is about designing great audio-haptic experiences. Our goal, with this talk, is for you to be inspired, and leave you with practical ideas about how to design great sound and haptics when used together in the right way, can bring a new dimension to your app.

During the next 30 minutes, we'll talk about three things.

First we'll introduce what is an audio-haptic experience? Then we'll look at three guiding principles to help you design those great experiences.

Lastly, we'll look at different techniques, and practical tips, to make those experiences great and truly compelling.

So, what is an audio-haptic experience? Well, let's start by listening to a sound.

OK, now let's lower the sound.

What happens if I lower it even further? Whoa.

It's so low I can't hear it any more.

You know, our ears just don't register it any more. But, if you would put your finger on the speaker, you could still feel it move back and forth.

We designed the Taptic Engine specifically to play these low frequencies that you can only feel.

Here it is in the iPhone.

And, next to it, the speaker module. The haptic sensations from the Taptic Engine are synchronized to the sounds coming from the speaker. And, the result is what we call an audio-haptic experience.

But, haptic sensations are meant to be felt, so because we are presenting this on stage, and on screen, we need your help in imaging what this would feel like.

We'll do our best to help you by visualizing the haptics like this.

Or, by playing a sound that resembles the haptic, like this.

We will also visualize these experiences on the timeline, and Camille will tell you some more about how that's done in a quick haptics design primer.

iOS 13 introduced a new API for designing your own custom haptics.

It's called Core Haptics.

This new API allows you developers to use the Taptic Engine fully in iPhone.

The Taptic Engine is capable of rendering a wide range of experiences. It can generate a custom vibration, like this.

Looks like this, and it should sound and feel like this.

So, as you've seen, we're using a waveform and sound to represent haptics. As Hugo said, you need to imagine this in your hand as a silent experience.

This should be felt and not heard. So, we can play these continuous experiences.

We can also have something that's more shorter and compact.

It's a single cycle, and we call this experience a transient.

It's much more momentary, and it feels like an impact, or a strike, or a tap.

Very momentary.

And then we actually can refine it further.

Moving forward, we'll use basic shapes to represent haptics in different patterns. So, our transient becomes our simple rectangle.

And, because our Taptic Engine is an exceptional piece of haptic engineering, we can modulate the experience in different ways.

First, we can modulate the intensity or the amplitude. We can also make it feel more round, or soft.

At the other extreme, we can make it more precise and crisp.

So, this experience is possible with the Taptic Engine.

So, in the end, this completes our quick introduction to haptic design, and what the Core Haptics API is all about.

We have one intensity you can modulate, and another design dimension, haptic sharpness, that you are in control for two types of event, continuous and transient.

Now, let's look at the three guiding principle that we want to share with you today. First is causality.

Then, we have harmony.

And, lastly we have utility.

These concept or approaches are used throughout the work that we do at Apple, and we think they can help you as well in your own app experience. For each of them, we'll look at the concept and explain through a few examples. Let's get started.

Causality: Causality is about, for feedback to be useful; it must be obvious what caused it. So, imagine being a soccer player, kicking this ball.

What would the experience feel like? There is a clear relationship, an obvious relationship between the cause-- the foot colliding with the ball-- and the effect-- the sound of the impact and the feel of the impact.

Now, what this experience sounds and feels like is determined by the qualities of the interacting objects. The material of the shoe, the material of the ball, and then, the dynamics of the action.

Is it a hard kick or a soft kick? And, the environment.

The acoustics of the stadium, or the soccer field.

Because we are so familiar with these things, it would not make sense at all to use a sound that is very different.

Let's try it out, and take it way over the top.

Very strange. That doesn't really work.

Now, when designing sounds for