Live Photo Editing and RAW Processing with Core Image

Thank you so much, and good morning. And my name is David Hayward and I'm here to talk to you today about editing Live Photos and processing RAW images with Core Image. We got a bunch of great stuff to talk about today.

First I'll give a brief introduction to Core Image for those of you who are new to the subject. Then we'll be talking about our three main subjects for this morning. First we'll be adjusting RAW images on iOS. Second, we'll be editing Live Photos. And third, we'll be talking about how to extend Core Image in a new way using CIImageProcessor nodes. So first, so a very brief introduction to Core Image.

The reason for Core Image is that it provides a very simple, high-performance API to apply filters to images. The basic idea is you start with an input image that may come from a JPEG or a file or memory, and you can choose to apply a filter to it and the result is an output image, and it's very, very easy to do this in your code. All you do is you take your image, call applyingFilter, and specify the name of the filter and any parameters that are appropriate for that filter. It's super easy. And, of course, you can do much more complex things. You can chain together multiple filters in either sequences or graphs and get very complex effects.

One of the great features of Core Image is it provides automatic color management, and this is very important these days. We now have a variety of devices that support wide gamut input and wide gamut output. And what Core Image will do is it will automatically insert the appropriate nodes into the render graph so that it will match your input image to the Core Image working space, and when it comes time to display, it will match from the working space to the display space.

And this is something you should be very much aware of because wide color images and wide color displays are common now and many open source libraries for doing image processing don't handle this automatically. So this is a great feature of Core Image because it takes care of all that for you in a very easy to use way.

Another thing to be aware of is that each filter actually has a little bit of code associated with it, a small subroutine called a kernel. And all of our built-in filters have these kernels and one of the great features is if you chain together a sequence of filters, Core Image will automatically concatenate these subroutines into a single program. The idea behind this is to improve performance by reducing the -- and quality, by reducing the number of intermediate buffers.

Core Image has over 180 built-in filters. They are the exact same filters on all of our platforms; macOS, tvOS and iOS. We have a few new ones this year which I'd like to talk about.

One is a new filter for generating hue saturation and value gradient. It creates a gradient in hue and saturation, and then you can specify, as a parameter, the brightness of the image and also specify the color space that the wheel is in. And as you might guess, this filter is now used on macOS as the basis of the color picker, which is now aware of the different types of display color spaces. Another new filter we have is CINinePartStretched and NinePartTiled. The idea behind this is you might have a small asset, like this picture frame here, and you want to stretch it up to fit an arbitrary size. This filter is very easy to use. You provide an input image and you provide four breakpoints, two horizontal and two vertical. Once you've specified those points, you can specify the size you want it to stretch to.

It's very easy to use. The third new filter is something that's also quite interesting. The idea is to start with a small input image. In this case it's an image containing color data, but it can also contain parametric data. So imagine you have a small set of colors or parameters and maybe it's only 6 by 7 pixels and you want to upsample that to the full size of an image.

The idea is to upsample the color image, the small color image, but respect the edges in the guide image. Now, if you weren't to respect the guide images, if you were just to stretch the small image up to the same size as the full image, you'd just get a blend of colors, but with this filter you can get more. You can actually get something that preserves the edges while also respecting the colors. And this is actually a useful feature for a lot of other types of algorithms. In fact, in the new version of Photos app we use this to improve the behavior of the light adjustment sliders. I look forward to seeing how you can use that in your application. We also have some new performance controls this year and do things that improve performance in Core Image this year. One is we have Metal turned on by default. So if you use any of our built-in 180 filters or your own custom kernels, all of those kernels will be converted to Metal on the fly for you. It's a great way of leveraging the power of Metal with very little effort on your part. We've also made some great improvements to a critical API, which is creating a UIImage from a CIImage, and this now produces much better performance than it has in the past. So you can actually use this very efficiently to animate an image in a UIImage view. Also another new feature is that Core Image now supports a feature that's new to Core Graphics, which is that Core Graphics supports half-floats. Let me just talk for a second about pixel formats because this brings up an interesting point.

We're all familiar with the conventional pixel format of RGBA8 and it takes just 4 bytes per pixel to store and has 8 bits of depth, and can encode values in the range of 0 to 1.

However, this format is not great for representing wide-colored data because it only has 8 bits and it's limited to the values in the range 0 to 1. So in the past the alternative has been to use RGBAfloat, which takes 16 bytes per pixel, so four times as much memory, but gives you all the depth and range you could ever hope for.

Another feature of the fact that it's using floats is that what quantization there is, it's distributed logarithmically, which is a good fit for the way the human eye perceives color. Well, there's a new format which Core Image has supported and now Core Graphics does as well, which I refer to as the Goldilocks pixel format, which is RGBAh, and this allows you to, in just 8 bytes per pixel, store data that is 10 bits of depth and allows values in the range of minus 65,000 to positive 65,000. And again, those values are quantized logarithmically, so it's great to store linear data in a way that won't be perceived as quantized. So I highly recommend this pixel format. There's another new format which I should mention, which is that Core Video supports a pixel format with the long name of 30RGBLittle PackedWideGamut, and this also supports 10 bits of depth, but stores it in an only 4 bytes per pixel by sacrificing the alpha channel. So there's many cases where this is useful as well and Core Image supports either rendering from or to CV pixel buffers in this format. So now I'd like to actually talk about the next major subject of our discussion today, which is adjusting RAW images with Core Image, and I'm really excited to talk about this today. We've been working on this for a long time. It's been a lot of hard work and I'm really excited about the fact that we've brought this to iOS. In talking about this, I'd like to discuss what is a RAW file, how to use the CIRAWFilter API, some notes on supporting wide-gamut output, and also tips for managing memory.

So first, what is a RAW file? Well, the way most cameras work is that they have two key parts; a color filter array and a sensor array. And the idea is light from the scene enters from the scene through the color filter array and it's counted by the sensor array.

And this data is actually part of a much larger image, of course, but in order to turn this data into a usable image, a lot of image processing is needed in order to produce a pleasing image for the user.

So I want to talk a little bit about that. But the main idea here is that if you take the data that was captured by the sensor, that is a RAW file. If you take the data that was captured after the image processing, that's a TIFF or a JPEG.

RAW files store the unprocessed scene data, and JPEG files store the processed output image.

Another way to think of it is that the RAW file stores the ingredients from which you can make an image; whereas, a JPEG stores the results of the ingredients