Build for the iPadOS pointer

Hello and welcome to WWDC.

Hi. I'm Mohammed, a UIKit engineer. Later, I'll be joined by my colleague Joey from the iOS System UI team, and this is "How to Build for the iPadOS Pointer." In 13.4, iPadOS added general pointing device support to the iPad. This is a brand-new input method for an OS that has been primarily touch-based since its inception. Rather than just bring over the exact interaction model that exists on a Mac, we thought about what value a pointer can add to iPadOS and how it can coexist with a touch-based interface.

The result is a system built on a fluid adaptive pointer that morphs into controls when hovering over them to adapt its accuracy and provide a clear indication that you're about to interact with them.

In other contexts, the pointer changes its shape and motion characteristics to provide contextual hints.

When hovering over text, for example, the pointer changes into a beam and snaps to lines to make interacting with text easier. In this video, we'll discuss the strategy we followed to update iPadOS for pointer and how you can use the same strategy when updating your app.

In the course of the discussion, we'll talk about the pointer customization APIs introduced in iOS 13.4, and we'll talk about some of the best practices to keep in mind while updating your app.

We'll also touch on some of the basic design principles behind the pointer UI. For a more detailed discussion of these principles and some of the thought processes behind the pointer design, check out the session on "Design for the iPadOS Pointer." When you use an iPad with a pointing device connected, you'll notice that a lot of things work with the pointer automatically without any additional adoption from apps. This is because a lot of system components have pointer support built in. Controls like UIBarButtonItem, UISegmentedControl and UIMenuController, to name a few, have built-in pointer effects and behaviors.

Scroll views respond to scrolling with two fingers as well as mouse wheels and pinching to zoom on the trackpad.

And in addition to these scroll view basics, collection and table view now support two-finger panning for swipe actions.

UITextView and other components that use UITextInteraction support a suite of new quick text selection and editing gestures that should be familiar to anyone who's used a Mac.

UIDragInteraction now allows you to drag quickly via a click-and-drag instead of requiring a long press as it does with touch.

And UIContextMenuInteraction lets you invoke its menu in a new, compact appearance via a secondary click.

So, how do we go about updating an app for pointer? We found that it's best to take a top-down approach. Start with the higher-level APIs. They'll offer you the most polished experience with tuned behaviors and visuals that are consistent with similar UI in the rest of the system. There are a number of pointer-related APIs available to you at different levels of the stack. Many controls have built-in pointer effects. Some, like bar buttons and segmented controls, have their effects enabled by default, while others, like UIButton, offer API that allows you to enable and customize their effects.

With UIPointerInteraction, you can make your custom UI react to and interact with the pointer in a way that feels consistent with the rest of the system. Using the interaction, you can choose from one of a collection of system-vended effects to apply to your views. Or you can change the pointer's shape within an area of your app. UIHoverGestureRecognizer lets you respond to the pointer's motion more directly. This is great for any custom behavior that doesn't involve applying hover or highlight effects or modifying the pointer's appearance. For more details on this gesture and other pointer-related additions to gestures and events, check out the session on handling trackpad and mouse input. When updating your app, you want to aim for an experience that makes it feel consistent with the rest of the OS.

When deciding where to add pointer support, take your cues from the Human Interface Guidelines and built-in apps. To that end, a good starting point is ensuring that controls in your app's chrome have the appropriate effects.

By "chrome," we mean the application's top and bottom bars, so start with any bar buttons in the app.

UIBarButtonItems created using the system item, image or title APIs get appropriate pointer effects automatically.

If you're using the custom view API, you'll have to implement the pointer behavior yourself, either using the view's convenience API, if it has one, or by installing a pointer interaction and managing the effect yourself.

UIButtons that have been placed in bars via the custom view API have their built-in interactions enabled by default and are given an effect that the system has deemed appropriate for that button in its containing bar.

You'll be able to tweak this effect using UIButton's convenience APIs.

To make this really easy, UIButton has a two-stage convenience API. To enable the automatic effect, simply set the button's isPointer- InteractionEnabled property to "true." After enabling the effect, you may customize it using the button's pointerStyleProvider.

In this closure, the system offers you a proposed effect and shape that have been determined based on the appearance, size and contents of the button.

Here, you can customize either of these or replace them entirely and construct a new style.

Before we look at some examples, let's take a brief detour and talk about pointer styles.

All APIs that modify the appearance of the pointer use UIPointerStyle to describe the modifications they apply.

Styles fit into two categories.

The first is what we call a content effect.

Content effects usually cause the pointer to morph into a view in the app and apply some visual treatment to it.

A common example of this is the highlight effect applied to bar buttons where the pointer morphs into a rounded rectangle, slides under the button and applies a subtle parallax effect to it.

This style consists of a UIPointerEffect which describes the visual treatment applied to the view...

and a UIPointerShape which describes the shape to which the pointer will change.

To specify this effect, we construct a UIPointerStyle with the highlight effect and roundedRect as the pointer shape.

The second category is shape customizations, which are expressed via a UIPointerShape and a UIAxis mask.

When applied, the pointer morphs into the shape and is constrained along the specified axes within the current region.

A good example of this is the pointer behavior in text views where we use a verticalBeam as the shape and specify "vertical" as the constrainedAxes to make the pointer feel as if it's on a horizontal rail along the line of text.

So, with that basic overview of the control APIs and the pointer styles, I'm going to hand it over to Joey to demo how they can be applied to an existing app. Joey? Thanks, Mohammed. Today we're going to take a look at an app that I've been working on called Quilt Simulator. I'm not much of a quilter myself, but I've been learning about quilting as I built this app. Let me build and run to see what we've got so far.

The first thing we want to take a look at is this button in the lower right corner. It toggles a straight-line guide mode on and off to make stitching in a straight line easier.

You can see that it uses the highlight effect, but it doesn't seem right. The effect isn't the right size.

Let's switch back to Xcode and see what we can do to improve it.

Here we can see that the pointer interaction has already been enabled for the button which gives up the default effect. We can add a pointerStyleProvider for this ruler button to improve the shape.

First we create a rect that is outset from the button's bounds by a suitable amount...

then convert that rect to the coordinate space of the target of the preview of the proposed effect.

And finally, return a style that uses the proposed effect but with an improved pointer shape based on that rect. Let's take a look at the change.

Much better. This button looks right and feels more comfortable.

Next, let's turn our attention to the thread selector button in the upper right corner. This button changes color with the currently selected thread color.

I already customized this bar button to use the lift effect because it has an intrinsic shape. You can see here it doesn't look quite right. When the pointer approaches the button, we see a diffuse blur under the button. This is distracting and incorrect. Let's switch back to Xcode to address this problem.

Since it is a UIBarButtonItem, it already gets an effect by default. Again, we modify the pointerStyleProvider, this time to change the default effect to a lift effect.

This implementation isn't correct though, so let's fix it with the correct shape.

In this updated implementation, we create a new UITargetedPreview that's suitable to be used with the lift effect. It's created with the view, target as well as preview parameters which we create above.

The parameter's shadowPath property is set to be the path which matches the outline of the button.

Finally, the closure returns a pointer style using a lift effect created with this preview and a pointer shape that matches the button.

And now back to the iPad to see the improvement.

Now things are looking good. That spool of thread just comes alive as we approach it with the pointer.

Mohammed, back to you.

Thanks, Joey. Now that we've seen how to enhance controls, let's talk about custom UI.

When considering adding pointer behaviors to custom UI, focus on areas where you think they'll add utility and unique value.

Our Quilt Simulator app has this large area in the middle. You click on it once to begin a stitch, and then you click again in a different location to complete it. By implementing some custom pointer behaviors, we can do some things to improve the experience. Let's add two enhancements.

First, let's change the pointer's shape when hovering over the quilt area to make it clear that clicking in a location starts a stitch.

And let's add a guided mode for beginner quilters where the pointer is constrained along the vertical axis to make it easier to make perfectly horizontal stitches.

Since this is an entirely custom view, we'll use UIPointerInteraction directly to achieve our goals. As with other UI interactions, we create an instance of UIPointerInteraction and attach it to the view. Unlike others, however, this interaction's delegate is optional.

As we did with the controls in our app, we'll be using UIPointerStyles to specify the effects or shape changes we'd like to apply to the pointer.

Since we want fine-grained control over where the styles are active, we'll also be defining custom UIPointerRegions to indicate to the interaction where to apply the styles we provide.

By default, the interaction uses a region that covers the entire view.

If the delegate doesn't provide any styles, the interaction applies the automatic pointer effect to the entire view.

This is really convenient in simple cases where we want to apply an effect to a view and be done with it. But our scenario's a bit more specialized.

Since we want to define behavior within specific subregions of our view, we're going to implement UIPointerInteractionDelegate's regionFor request method.

This method is called to request new regions as the pointer moves within the interaction's view. For our implementation, we'll provide it with regions that correspond to our guides.

As soon as the pointer enters a region, the interaction calls its delegate's styleForRegion method to request a style, so we'll need to implement the method and return a shape customization style that provides this crosshair shape.

Now I'm gonna hand it back to Joey to see what this looks like in practice.

Thanks again, Mohammed. Let's take another look at Quilt Simulator to see how we can improve it. Once the quilter has selected a thread color, it's time to start quilting, which is easily done by clicking for each of the stitches on the patchwork area.