Building Visually Rich User Experiences

Good afternoon. Thank you for coming. I'm Noah. I'm here with my colleague Warren and we're going to talk to you about building visually-rich user experiences. So, there's three main areas that we're going to cover today.

The first is a platform overview. Kind of a quick tour, kind of the lay of the land of what the graphics frameworks are on our platforms, what they can do, how you kind of fit them together. Next, we're going to kind of dive deep into one of the more essential parts of the graphics platform coordination and some best practices that you can use to kind of avoid common pitfalls that people run into a lot with CA. And, finally, we're going to spend a little more time on some tips and tricks. Some kind of small, fun, like, useful things you can do to really add some cool visual flair to your apps.

So, if you spend any time digging through the documentation or just reading just like overflow or anything, you've probably seen a lot of concepts kind of come up. And there are a lot of them and it would be great to have time to kind of go into every single one of these and just basically spend all day here, but we aren't really allowed to do that. So, instead, we're going to keep it pretty high level. Let's talk about what we have on iOS, macOS, and tvOS. First off, some of you are probably already familiar with UIKit and AppKit are the frameworks with which you build the UIs on most of your applications. They kind of provide you a standard set of system controls. They're customizable to a degree, so you can make our sliders look way cooler than usual ones. And they also come with some really useful kind of built-in behaviors. You get these standard appearances and transition animations and that kind of thing, but you also get support for accessibility features and dynamic type right out of the box. So, I said that these let you customize things to some degree, but as soon as you want to kind of go beyond just setting a tint color or an image override on something, you might have to start dropping down into something like Core Animation.

And, as you might already be aware, Core Animation underlies UIKit on iOS and is also pretty heavily integrated into AppKit on the Mac. And so, a view on iOS is composed of a-- like has a layer and the stuff that appears in your application is this hierarchy of layers kind of all appearing together. And that's what CA is good for. It's kind of-- it's a thing to take existing content, put it together, display it in this hierarchy that can move around and animate and so on so forth and finally composite it all together and get it onscreen.

So, beyond that, there are a couple other frameworks that we have for some kind of more specialized purposes. Core Graphics is one that you also might have encountered. It lets you do kind of really elaborate customizable 2D drawings. So, you can take a path and some colors and turn them into these kind of nice message bubbles over here. And Core Graphics is what you use when you need to build images at runtime. For something-- for building it, like beforehand, you probably use something like Photoshop or Sketch or Illustrator, but Core Graphics lets you do that on the device, you know, as your code is running.

Which is really useful. So, Core Graphics is useful for drawing things, but there's another framework that we have for working with images that you already have and that is appropriately named Core Image.

And what that does is apply filters to things. You provide some set of input images, you hand that to Core Image, get a-- and get a filter that does some thing, and it produces the final rendered output of the thing-- of the filter, rather. So, you have obviously simple color transformation effects, like this kind of nice sepia-- that's not sepia-- I don't know what to call that. Don't worry about it. But you can also do kind of interesting stylization effects, like this cartoon drawing kind of thing. And what's neat about Core Image is that it's really heavily optimized and we kind of do a lot of the work for you. So, when you're doing something more complicated like chaining multiple filters together, we kind of can optimize that for you and instead of having to kind of go through this process over and over again on your images for each filter, we can kind of pipeline that. And what's even better is that all of this stuff can run on basically whatever parallelization hardware your device has. If you don't have much GPU resources available, it runs on the CPU and makes use of Grand Central Dispatch and all that. Completely under the hood, transparent to you. If it has Metal available, then it can run using that. So, that's pretty cool.

There are a couple other kinds of contents you might want to get into your apps that we have some really useful frameworks for.

And for 3D content, we have this thing called SceneKit which basically lets you get your 3D content into your app and has some really nicely integrated tools in Xcode that let you kind of manipulate it, set up scenes, do all this. And you'll see a little bit of what that looks like a bit later on.

SpriteKit is another framework that's kind of built heavily for games. It gives you a lot of animation primitives and that kind of thing for having things, you know, move around and interact with each other onscreen.

So, there's one more framework that I'd like to talk about before we move on and that is one you've probably heard of called Metal.

And what Metal is is kind of the fundamental graphics API on our platforms. It's what all of these other things are built on top of. I mentioned that Core Image can use Metal, but SceneKit draws using Metal on devices that support it, SpriteKit does the same, and Core Animation itself is using Metal as well.

And so, Metal is-- it's kind of hard to describe exactly what you use it for because you can do basically anything with it if you are willing to put the work in.

But what it lets you do is just kind of get the fastest possible access to the lowest level of the graphics hardware we have available. And so, a lot of the times you're not using Metal directly, you're using it through one of these other frameworks, but if you need to build something custom, something that isn't really supported by any of these, that's what you're using it for. As an example, on the iPhone 7-- sorry, the 7 Plus-- we have this kind of neat depth of field effect in the camera and that basically takes the camera image, does some kind of depth reconstruction stuff on it, some math that is way beyond me, and can figure out kind of the depth of the pixels in each-- at each point in the scene and then apply an intelligent kind of blur filter to them so you get this nice soft background. And that doesn't sound like any of the other frameworks that I told you about, right? So, that was built for iOS 10 and rebuilt for iOS 11. I don't know if you've noticed that the effects just got better in the betas on top of Metal.

So, that was kind of a quick tour. We just went down through all these layers and the idea here is not to kind of give you an exhaustive explanation of what each one does, but I just want to kind of have you know where things are. So, if you have some problem that you want to solve in your app, you can kind of look back and be like I think I heard that there's a firmware that does that. Let me check that out. So, to give you a little bit more of an idea of how you use these and how they work, I'd like to bring up my colleague, Warren, for a demo.

Thanks, Noah.

So, with this series of demos, I'm going to walk you through some of the frameworks that Noah just introduced and we'll start right at the beginning with UIKit customization. I want to say right from the start that it's not important for you to pay attention to every single line that I highlight or talk about. The sample is already available online and you can definitely pull it down and follow along on your own time. But, over the course of the next several minutes, I'll walk through some of the platform features that we have and how you can use them to enhance the user experience of your application.

So, to start, I'll just go ahead show you what this particular UIKit customization example does.

I have here a screen, perhaps it's from a video-editing application that allows me to control the white balance of a photograph. As you can see, I've already customized the UI sliders at the