Ahhh - CIFilter ... bliss

So, after ranting and raving about QT based effects I decided to chuck them and go for CoreVideo. After getting the movie export working using a procedure based approach (so that I can mess with the frames before they are compressed) I turned to CIFilter.

Oh what joy. Ahhhh. Awesome.

So instead of having to write a billion lines of code to find out what effects exist, you can just do this:

 NSArray *filters = [CIFilter filterNamesInCategories:[NSArray arrayWithObject:kCICategoryBlur]];

Which will get you all the blur filters. The really nice thing about CIFilter is the metadata. It makes it reasonably simple to create a user interface on the fly, given any CIFilter attributes. I won't go over the whole thing here as it's well documented elsewhere suffice too say that you've got all the info you need to build a UI dynamically.

When I was still considering the use of QTEffects for Squish I had originally thought that I might hard code a few commonly used filter interfaces such as Gamma, Brightness/Contrast/Saturation, and so on. The primary reason was that Squish isn't a video editor. It doesn't handle transitions and it's not intended to be able to insert complex filter overlays onto video - the type of thing that iMovie/FCP are good at. But when I saw the list of filters, I thought "hmm. that's quite a few." and realized I was going to have to be a bit smarter.

So Squish now builds it's filter inspector based on the properties of the filter you're looking at. One part I really liked, which is a mix of standard NSObject and the CIFilter interface was the elegant way by which all filter attributes can be extracted/and or set on the filter. In three words: Key Value Coding.

For example - lets say you want to get a mutable dictionary of all filter attributes that matter to you, change some values and the set the filter values again so that you can observe the result. Step one - get the attributes that matter:

  CIFilter *filter = [CIFilter filterWithName:effectName];
  [filter setDefaults];
  NSDictionary *inputKeyDictionary = [filter dictionaryWithValuesForKeys:[filter inputKeys]];
  NSMutableDictionary *attrs = [NSMutableDictionary dictionaryWithDictionary:inputKeyDictionary];

Er, well - that was simple :-) Not only are they easy to get but they will have sensible defaults due to the setDefaults message. So what if you've now modified the attributes and want to set them back? Again, so so trivial since the CIFilter simply uses the existing KVC methods and we already have all of the input attributes. We can set them back on the filter like so:

  CIFilter *aFilter = [CIFilter filterWithName:effectName];
  [aFilter setValuesForKeysWithDictionary: attrs];

Anyone seasoned in Cocoa will see the above as trivial. It is, certainly. But it's such a joy to use having just jumped off the run away train that is QT Atoms that I feel the need to write something. It's also nice to praise a sensible elegant API when you see it.

What this all means in reality is that I've been able to construct a full working CoreImage effects based filtering mechanism in about 3 days total. That's pretty impressive if you ask me (Ok, yes, it'd be nice if there were a standard inspector available, but I'll live) - and getting it working with the visual context of QT was a breeze.

Having wrapped up the filter code, it was then possible to apply a filter to a frame by simply calling one helper method (whose implementation isn't that complex). In fact; there's more code to get the bitmap data out (so that I can hand it to QT) that there is to perform the filtering!

 CVPixelBufferRef ref = [currentMovieFrame pixelBufferRef];
 CVReturn lockResult = CVPixelBufferLockBaseAddress(ref, 0);
 if(lockResult == kCVReturnSuccess) {
  void *dataPtr = CVPixelBufferGetBaseAddress(ref);
  if([parameters effectStack]) {
   CIImage *image = [[CIImage imageWithCVImageBuffer:ref] autorelease];
   CIFilter *filter = [CIFilterFactory generateFromEffectStack:image effectStack:[parameters effectStack]];
   if(filter) {
    CIImage *outImage = [filter valueForKey:@"outputImage"];
    int width = CVPixelBufferGetWidth(ref);
    int height = CVPixelBufferGetHeight(ref);
    int bytesPerRow = CVPixelBufferGetBytesPerRow(ref);
    CGColorSpaceRef cSpace = CGColorSpaceCreateDeviceRGB();
      
    CGContextRef bitmapCtx = CGBitmapContextCreate(dataPtr, width, height, 8, bytesPerRow, cSpace, kCGImageAlphaPremultipliedFirst);
    CIContext *ciContext = [CIContext contextWithCGContext:bitmapCtx options:nil];
    CGImageRef cgImage = [ciContext createCGImage:outImage fromRect:[outImage extent]];
    CGContextDrawImage(bitmapCtx, [outImage extent], cgImage);
      
    CGContextRelease(bitmapCtx);
    CGImageRelease(cgImage);
    CGColorSpaceRelease(cSpace);
   }
  }
  long dataSize = CVPixelBufferGetDataSize(ref);
  (*outputImageDescription)->dataSize = dataSize;
  (*outputImageDescription)->depth = 32;
  CVPixelBufferUnlockBaseAddress(ref, 0);

    // etc etc - all QT based stuff from here on

Before you scream - Squish isn't going to use the method above - it's quite ineffient (it creates a bitmap and throws it away, for every video frame). However; it's simple to read - and good for demonstrating the technique.

   CIFilter *filter = [CIFilterFactory generateFromEffectStack:image effectStack:[parameters effectStack]];

The above gets us a fully initialized filter chain (In Squish, filters can be layered and thus are chained together). We simply get the output image and go through the steps above to extract some bitmap data for use with QT. Pretty impressive if you ask me.

In closing - it's been a week of productivity because an API exists that a) has been thought out, b) integrates well with CoreGraphics and c) "just works". It's really nice when this happens!