Core Image: Performance, Prototyping, and Python

All right .

Thank you . Good afternoon everyone and thank you for coming to our session today on Core Image. My name is David Hayward, and I'm really excited to be talking about the great new performance and prototyping features our team has been adding to Core Image over the last year. We have a lot to talk about, so let's get right into the agenda. So, the first thing we're going to be talking about today are some great new APIs we've added to Core Image to help improve the performance of your applications. After that, we're going to segue into another topic, which is how you can use Core Image to help prototype new algorithm development. And lastly, we're going to be talking about how you can use Core Image with various machine learning applications.

All right. So, let's get into this and start talking about performance APIs. There's two main areas where we've worked on performance this year. First of all, we've added some new controls for inserting intermediate buffers -- we'll talk about that in some detail. And the second thing is we'll be talking about some new CI kernel language features that you can take advantage of. So, let's start by talking about intermediate buffers.

As you are aware, if you've used Core Image before, Core Image allows you to easily chain together sequences of filters. Every filter in Core Image is made up of one or more kernels. And one of the great features that Core Image uses to improve performance is the ability to concatenate kernels in order to minimize the number of intermediate buffers. In many cases, to get the best performance you want to have the minimum number of buffers.

However, there are some scenarios where you don't want to concatenate as much as possible. For example, your application might have an expensive filter early on in the filter chain. And the user of your application at a given moment in time might be adjusting a filter that follows it in the graph. And this is a classic situation where it's a good idea to have an intermediate buffer at a location like this in between.

The idea is that by having an intermediate buffer here, the cost of the expensive filter does not have to be paid for again when you adjust a secondary filter. So, how do you do this in your application? We have a new API, very aptly named, inserting intermediate. So, let's talk about how this affects our results. What we do is instead of concatenating as much as possible, we will respect that location of the intermediate and concatenate as much as possible around it.

Some notes on this. One thing to keep in mind, is that by default, Core Image cashes all intermediate buffers so that the assumption is that a subsequent render can be made as fast as possible. There are, sometimes, however, when you might want to turn off caching of intermediates. So, for example, if you application is going to be doing a batch export of 100 images, there is little benefit of caching the first one, because you'll be rendering a completely different image afterwards. So, you can do that today in your application by using the context option cache intermediates and setting that value to false.

However, if you are also using this new API that we spoke about, you can still turn on caching of intermediates, even if this context option is turned off. So, this allows you to really make sure that we cache something and don't cache anything else.

The next subject I'd like to talk about is some new features we've added to the kernel language that allows us to apply image processing.

So, one thing to keep in mind is that we have two different ways of writing kernels in Core Image. The traditional way is to use the CI kernel language. And in this case, you have a string inside your source file; either your Swift code or your objective C code. And at run time you make a call, to say, kernel with source.

And later on, when you create an image based on that kernel, you can then render that to any type of Core Image context, whether that context is backed by Metal or open GL.

When it comes time to render, however, that source needs to be translated. It needs to be translated either to Metal or GLSL, and that step has a cost. Eventually then, that code is compiled to the GPU instruction set and then executed. Starting last year in iOS 11, we added a new way of writing CI kernels, which has some significant advantages. And that's CI kernels based on the Metal shading language. In this case, you have your source in your project and this is -- this source is complied at build time rather than at runtime.

As before, you substantiate a kernel based on this code by using the kernel with Metal function name and binary data.

The advantage here is that this data can be applied without paying the cost of an additional compile. The caveat, however, is it works on Metal backed CI context. But it gives a big performance advantage. So, starting in this release we're going to be marking the CI kernel language as deprecated, because while we will continue to support this language, we feel that the new way of writing Metal kernels offers a lot of advantages to you, the developer. For one thing, you get the performance advantage I outlined earlier, but it also gives you the advantage of getting build time syntax coloring on your code and great debugging tools when you're working with your Metal source.

So, great. So, with that in mind I want to talk about a few other things that we've added to our kernel language. For one thing, we have added half float support.