The sharpen tool has the following effect on this tweed pattern:

Notice the stripe through the middle where the edges are more strongly defined.

The blur tool has the opposite effect of sharpen on the tweed pattern:

Implementation Overview

Like the sponge tool, the sharpen and blur tools are subclasses of the FilterBrush class introduced in the dodge and burn tutorial. However, unlike sponge, they do require a slight modification to the GenericBrush class to work.

Since almost all of the code in this tutorial has been covered before, I’ll just highlight the single change to the GenericBrush class, and the new Sharpen and Blur classes. If you want to see everything, download the code.

Changing GenericBrush

If you recall, at the very beginning of this article I mentioned that sharpen and blur tools typically allow the user to modify the blend mode. This is the blend mode used when stamping the filtered image back on to the original image. GenericBrush contains the code that handles the stamping, so it will need to be modified.

We change the render:at: method for GenericBrush to be:

- (void) render:(Canvas *)canvas at:(NSPoint)point
{
	CGContextRef canvasContext = [canvas context];
	CGContextSaveGState(canvasContext);

	CGPoint bottomLeft = CGPointMake( point.x - CGImageGetWidth(mMask) * 0.5, point.y - CGImageGetHeight(mMask) * 0.5 );

	// Our brush has a shape and soft edges. These are replicated by using our
	//	brush tip as a mask to clip to. No matter what we render after this,
	//	it will be in the proper shape of our brush.
	CGRect brushBounds = CGRectMake(bottomLeft.x, bottomLeft.y, CGImageGetWidth(mMask), CGImageGetHeight(mMask));

	CGContextClipToMask(canvasContext, brushBounds, mMask);
	CGContextSetBlendMode(canvasContext, [self blendMode]);
	[self renderInCanvas:canvas bounds:brushBounds at:point];

	CGContextRestoreGState(canvasContext);
}

The only change made is the call to CGContextSetBlendMode that allows brush subclasses to determine the blend mode to use. The default implementation is simply:

- (CGBlendMode) blendMode
{
	return kCGBlendModeNormal;
}

That’s it for GenericBrush.

Sharpen Tool

The sharpen tool has just two parameters: mMode, which is the blend mode, and mStrength which determines how strong the sharpen effect is applied. They are initialized in init like so:

- (id) init
{
	self = [super init];
	if ( self != nil ) {
		// Set the default values for our parameters
		mMode = kCGBlendModeDarken;
		mStrength = 1.0;
	}
	return self;
}

Here are some examples of the sharpen tool parameters:

mMode	mStrength	Result
kCGBlendModeNormal	0.5
kCGBlendModeNormal	1.0
kCGBlendModeDarken	1.0
kCGBlendModeLighten	1.0

The blend modes typically offered for a sharpen tool are kCGBlendModeNormal, kCGBlendModeDarken, kCGBlendModeLighten, kCGBlendModeHue, kCGBlendModeSaturation, kCGBlendModeColor, and kCGBlendModeLuminosity. In the examples above, kCGBlendModeDarken causes the sharpening to only affect light pixels and kCGBlendModeLighten only affects dark pixels.

The heart of the sharpen tool is the filter it creates. The code for that is:

- (CIFilter *) filter
{
	// We need to create and configure our filter here
	CIFilter * filter = [CIFilter filterWithName:@"CIUnsharpMask"];
	[filter setDefaults];
	[filter setValue:[NSNumber numberWithFloat:mStrength * 100.0] forKey:@"inputRadius"];
	return filter;
}

You’ll notice that I’m using the unsharp mask filter to do the sharpening. I should point out that most image editors don’t use this filter to do the sharpening. Instead, they seem to use a convolution filter to do the sharpening. However, the convolution filter means it is easy for the user to overdue the effect. Unsharp mask does not have that problem, plus unsharp mask is provided by the system.

The final bit of the sharpen tool is returning the correct blend mode:

- (CGBlendMode) blendMode
{
	return mMode;
}

Blur Tool

The blur tool is very similar to the sharpen tool. It even takes the same parameters. The only real difference is the filter that it creates, so that’s what I’ll present first:

- (CIFilter *) filter
{
	// We need to create and configure our filter here.
	CIFilter * filter = [CIFilter filterWithName:@"CIBoxBlur"];
	[filter setDefaults];
	[filter setValue:[NSNumber numberWithFloat:mStrength * 7.0] forKey:@"inputRadius"];

	return filter;
}

Here I use the built in CIBoxBlur. mStrength ranges from 0 to 1, inclusive. I set the coefficient of 7 simply by trial and error until I got something that “looked right.”

Here are some examples of the blur tool:

mMode	mStrength	Result
kCGBlendModeNormal	0.5
kCGBlendModeNormal	1.0
kCGBlendModeDarken	1.0
kCGBlendModeLighten	1.0

Conclusion

Sharpen and blur are the last of the common filter brushes. Enjoy, and download the sample code.

Overview

In concept, the sponge tool is a simple tool. It either increases or decreases the saturation of an image. Saturation describes how intense a given color is. 0% saturation means the color is gray. In the implementation that I will show, the sponge tool will behave like a brush.

Thus, if I have a simple red square with about 50% saturation, the sponge tool in the saturation mode affects it like so:

The desaturation mode has the opposite effect on the same image:

In addition to the saturate/desaturate mode, the sponge tool has a “flow” parameter, which determines how much to saturate or desaturate the image.

Implementation Overview

The sponge tool is simply a subclass of the FilterBrush class, which I created for the dodge and burn tutorial. In fact the only difference between this tutorial and that one is the new subclass Sponge. For that reason I’ll only be covering the Sponge class in this tutorial. For an explanation as to how the rest of the code works, please refer back to the dodge and burn tutorial.

As always, I’ve provided sample code for this article.

Sponge

Parameters

The sponge tool has two parameters which are initialized in the init method:

- (id) init
{
	self = [super init];
	if ( self != nil ) {
		// Set the default values for our parameters
		mFlow = 0.5;
		mMode = kSpongeMode_Desaturate;
	}
	return self;
}

The two parameters for the sponge tool are mFlow and mMode.

• mFlow Flow determines how strong the saturation or desaturation effect is applied to the image. It ranges from 0.0 to 1.0, where 0.0 means the effect isn’t applied, and 1.0 is where the effect is at its strongest.

  Sponge in saturate mode, varying flow:

  mFlow	Result
  0.25
  0.5
  1.0

• mMode Mode determines if the sponge saturates or desaturates the image. It has only two settings: saturate or desaturate. Saturate increases the color’s saturation, while desaturate decreases saturation.

  Sponge examples:

  mMode	Result
  kSpongeMode_Saturate
  kSpongeMode_Desaturate

In the examples I used a red rectangle with about a 50% saturation. That way, both saturation and desaturation effects will show up on it.

Creating the filter

If you recall from the dodge and burn tutorial, the only real responsibility of a FilterBrush subclass is to create a filter that will be applied to each brush stamp.

- (CIFilter *) filter
{
	// We need to create and configure our filter here. CIColorControls has controls
	//	for saturation which is what we care about.
	CIFilter * filter = [CIFilter filterWithName:@"CIColorControls"];
	[filter setDefaults];
	if ( mMode == kSpongeMode_Saturate )
		[filter setValue:[NSNumber numberWithFloat:1 + mFlow] forKey:@"inputSaturation"];
	else
		[filter setValue:[NSNumber numberWithFloat:1 - mFlow] forKey:@"inputSaturation"];

	return filter;
}

Unlike the dodge and burn tools, I don’t use a custom filter. Instead, I use the system provided CIColorControls, which has a parameter for saturation. The saturation parameter has a range of 0 to 2, where 1 is the identity value. Thus, if I’m trying to saturate the image, I add 1.0 to mFlow, and if I’m trying to desaturate the image, I subtract mFlow from 1.0.

That’s it. The rest of the heavy lifting is done by the framework I created in the dodge and burn tutorial.

Conclusion

The main thing this tutorial demonstrates to me is that the time spent building a framework for filter brushes paid off. It should help in the future if I decide to tackle the blur and sharpen brushes as well.

Hopefully you found this post enlightening. Don’t forget to download the source code.

Overview

Dodge and burn are photographic processes that lighten or darken a specific part of an image, respectively. Dodge works by placing an object between the light and the photo, such that the area that should be lightened is blocked from the light. Burn works the same way, except that the card is placed such that it disallows light to every part of the photo except the one the developer wants to darken. If you’re interested, Wikipedia has more in depth coverage of dodge and burn.

Our dodge and burn tools are simply going to lighten and darken the image. As in most bitmap editors, our dodge and burn tools will act like brushes.

For example, if we have a simple black and white linear gradient, the dodge tool affects it like so:

The burn tool has the opposite effect on the same image:

In addition to being able to adjust the strength, often called exposure, of the dodge and burn tools, the user will also be able to determine the range of colors the tools will affect the most: highlights, midtones, or shadows. Although our dodge and burn tools do not exactly mimic the tools found in large commercial packages, I believe they get reasonably close.

Implementation Overview

Since dodge and burn are brushes, they inherit all the properties of our basic bitmap brush. The difference is, instead of using a solid color for the brush tip, they use the source image, after it has been altered by a Core Image filter.

Since the basic brush implementation has been covered before, we’re only going to cover the basic algorithms used in the Core Image filters for dodge and burn. Both the dodge and burn tools only affect a specific range of colors at a time: highlights, midtones, and shadows. Because of this, there will be six filter algorithms shown here (three for dodge and three for burn).

The filters affect each of the color components in the same way. i.e. The function applied to the red channel is the same as the function applied to the green and blue channels. In addition, the range parameter is also per color component. Highlights are the component values closest to 1.0, while midtones are around 0.5, and shadows are close to 0.0. Finally, we’re not going to cover how to implement the exposure parameter here in the overview, but instead show the general filter functions.

Dodge Highlights

The dodge highlights filter will lighten all components, but will affect the components near 1.0 the most. The function for this filter is:

new component = component + e ^ component – 1.0

where component is one of pixel component values (red, green, or blue) and e is the math constant.

This yields the graph:

As you can see, for component values closer to 0 (i.e. shadow values), the function is close to the identity function (new component = component, aka y = x). As the component values approach 1, the function diverges from the identity greatly in that the new component value reaches 1 much quicker. The result is that components closer to 1 are pushed towards 1 more quickly than they normally would. i.e. Highlights are lightened.

Dodge Midtones

Dodging midtones will lighten all components, but it will affect the components near 0.5 the most. The function for this filter is:

new component = component + 0.25 * sin(component * PI)

where component is one of pixel component values (red, green, or blue) and PI is the math constant. The constant of 0.25 is just to limit the amplitude.

This yields the graph:

As the graph demonstrates, the function intersects at (0, 0), and (1, 1), just like the identity function, but bows upward in the middle, near 0.5. The result is that components near 0.5 are pushed closer to 1, or in other words, lightened.

Dodge Shadows

Dodge shadows, like all the other dodge filters, will lighten all colors, but will affect the components closest to 0 the most. The function for this filter is linear:

new component = 0.5 * component + 0.5

where component is one of pixel component values (red, green, or blue).

This yields the graph:

The function intersects at (1, 1), like the identity function, but quickly diverges from it, as the component approaches 0. As it approaches 0, the new components are lightened, in relation to the identity.

Burn Highlights

Burning the highlights will darken all colors, but will effect the components closest to 1 the most. The function for this filter is:

new component = 0.25 * component

where component is one of pixel component values (red, green, or blue).

This yields the graph:

Note that the function intersects at (0, 0) like the identity function, but then skews the values closer to 1 to closer to 0. i.e. It darkens the highlights.

Burn Midtones

Burning midtones will darken all components, but affects the components closest to 0.5 the most. The function for this filter is almost identical to that for dodging midtones:

new component = component – 0.25 * sin(component * PI)

where component is one of pixel component values (red, green, or blue) and PI is the math constant. The constant of 0.25 is just to limit the amplitude.

This yields the graph:

As the graph demonstrates, the function intersects at (0, 0), and (1, 1), just like the identity function, but bows downward in the middle, near 0.5. The result is that components near 0.5 are pushed closer to 0, or in other words, darkened.

Burn Shadows

The burn shadows filter will darken all components, but will affect the components near 0 the most. The function for this filter is similar to that of dodge highlights:

new component = component + (1.0 – e ^ (component – 1.0))

where component is one of pixel component values (red, green, or blue) and e is the math constant.

This yields the graph:

As you can see, for component values closer to 1 (i.e. highlight values), the function is close to the identity function (new component = component aka y = x). As the component values approach 0, the function diverges from the identity greatly in that the new component value reaches 0 much quicker. The result is that components closer to 0 are pushed towards 0 more quickly than they normally would. i.e. Shadows are darkened.

Code Architecture

As always, I’ve provided sample code for this article. Unlike previous sample code, the code for dodge and burn is somewhat forward looking, in that it provides a framework for future filter brushes. In the future, we can simply discuss the subclasses of FilterBrush and the corresponding filter instead of covering all the other brush related code that just changed a little.

Although all the brushing code is heavily borrowed from the smudge tool, it has been refactored so that it can support generic filter brushes. We’ll go over the changes required to support that, but in general, we’ll ignore the code we’ve already covered in previous articles.

Because the code attempts to set up somewhat of a framework, there are nine classes that make up the sample code: MyDocument, CanvasView, Canvas, GenericBrush, FilterBrush, Dodge, Burn, BurnBrushFilter, and DodgeBrushFilter. Fortunately most of these are carry overs which we’ll ignore since we’ve already dealt with them.

MyDocument, CanvasView and Canvas are the same as they ever were, although Canvas has two new methods which I’ll present later. GenericBrush is all generic brushing code that we’ve carried over ever since the first brush article. It contains a couple of new hooks that subclasses can override.

FilterBrush is where the new code begins. It is an abstract subclass of GenericBrush and deals with rendering a filtered section if the canvas as a brush stamp. Concrete subclasses only need to overload a method to return a configured CIFilter to work.

Dodge and Burn are subclasses of FilterBrush. They load their respective filters, DodgeBrushFilter and BurnBrushFilter, and override a method to return instances to their FilterBrush base class.

DodgeBrushFilter and BurnBrushFilter are CIFilter subclasses, and are nearly identical except for which kernels they load up. They are responsible for actually applying the filter.

We’ll first cover the new methods in Canvas and GenericBrush, then move on to FilterBrush, Dodge, and DodgeBrushFilter. Since Burn and BurnBrushFilter are so similar to their dodge counterparts, we’ll ignore them. However, all the Core Image kernel code will be covered since it is different for each.

Canvas

There are only two new methods on the Canvas class and they’re just used for supporting applying filters to the canvas.

The first one returns the CGContextRef for the canvas:

- (CGContextRef) context
{
	// Just grab the context off the layer
	return CGLayerGetContext(mLayer);
}

Nothing to really discuss here: we just ask our backing CGLayerRef for the CGContextRef.

The next method gets a CIImage that can be used for the input of a CIFilter. Because of a bug, it’s a bit more complicated that it should be:

- (CIImage *) image
{
	// We should theoretically be able to just return [CIImage imageWithCGLayer:],
	//	but we're not able to and get the proper results. Core Image does something
	//	a little indeterminate with the pixels: the effect is applied, but the component
	//	values are shifted.
	// This seems to be caused by the fact we're using a CGLayerRef for both the
	//	source and destination of the effect. I asked on the quartz-dev list
	//	if this was supported or not, and received no answer.
	// The work around is create a deep copy of the layer in a bitmap context.
	//	It is heavy, but it works. Simply created a duplicate CGLayerRef here
	//	doesn't seem to work.
#if 1
	CGSize size = CGLayerGetSize(mLayer);
	size_t width = size.width;
	size_t height = size.height;
	size_t bitsPerComponent = 8;
	size_t bytesPerRow = (width * 4+ 0x0000000F) & ~0x0000000F; // 16 byte aligned is good
	size_t dataSize = bytesPerRow * height;
	void* data = calloc(1, dataSize);
	CGColorSpaceRef colorspace = CGColorSpaceCreateWithName(kCGColorSpaceGenericRGB);

	CGContextRef bitmapContext = CGBitmapContextCreate(data, width, height, bitsPerComponent, bytesPerRow, colorspace,  kCGImageAlphaPremultipliedFirst);

	CGColorSpaceRelease(colorspace);

	CGContextDrawLayerAtPoint(bitmapContext, CGPointMake(0, 0), mLayer);

	CGImageRef imageRef = CGBitmapContextCreateImage(bitmapContext);
	CGContextRelease(bitmapContext);
	free(data);

	CIImage* image = [CIImage imageWithCGImage:imageRef];
	CGImageRelease(imageRef);

	return image;
#else
	return [CIImage imageWithCGLayer:mLayer];
#endif
}

Technically speaking, we should be able to just call the static CIImage method imageWithCGLayer to get our CIImage. Unfortunately, there appears to be a bug in Core Image when you use the same CGLayerRef as both the source and destination of a CIFilter. In that case it tends to lighten the effect a lot more than it should. I am not sure why.

The workaround is to create a bitmap context, copy our layer into it, create a CGImageRef from that, then produce a CIImage from the CGImageRef. It’s a lot more involved, not to mention slower.

GenericBrush

The GenericBrush is the same brushing code that it has always been. In the sample code however, we’ve pulled it out into its own class and added a few methods that can be overridden by subclasses. By default, these methods do nothing:

// Subclass overrides so that they can know when we about to start and stop brushing
- (void) startBrush:(Canvas *)canvas;
- (void) stopBrush;

	// Override for sublcasses to implement brush render
- (void) renderInCanvas:(Canvas *)canvas bounds:(CGRect)bounds at:(NSPoint)point;

startBrush and stopBrush are optional overrides, used in case the brush needs to do special set up or take down before or after a brush stroke. startBrush is invoked inside the mouseDown handler before the first stamp is rendered, and stopBrush is invoked in the mouseUp handler after the last stamp is rendered.

renderInCanvas is a required override for subclasses. It renders a single brush stamp. Before invoking renderInCanvas, the GenericBrush class will have already set up the canvas with the proper clipping and whatnot. The subclass only needs to render a stamp at the specified point.

FilterBrush

The FilterBrush class derives from GenericBrush and provides a base class for all future filter-based brushes, which, in this case, will be Dodge and Burn. It does all the set up for Core Image and the actual rendering of the single brush stamp.

Initialization

Although FilterBrush doesn’t add any new properties to the brush, it does need to cache some data, as seen in the init method:

- (id) init
{
	self = [super init];
	if ( self != nil ) {
		// Initialize to nil
		mFilter = nil;
		mCIContext = nil;
	}
	return self;
}

In order to get decent performance out of brushing with a filter, we have to cache both the CIFilter we’re applying and the CIContext we’re rendering into. If we don’t, performance is bad enough that the brush is unusable at any size.

Setting up the Brush Stroke

In order to implement the caching, we use the startBrush and stopBrush methods we just covered in GenericBrush. This means we cache the filter data for one brush stroke. With some more complicated logic, we could potentially be more aggressive with the caching and get even better performance, by caching for more than one stroke.

First, let’s cover the startBrush method:

- (void) startBrush:(Canvas *)canvas
{
	// This gets called right at the mouse down, before we start stamping. We
	//	should do any set up here.

	// Ask the subclass for our filter. Cache it since it is expensive to keep
	//	creating for each stamp we render.
	mFilter = [[self filter] retain];

The first step is to get and cache our filter. We don’t know what the filter is, being an abstract class and all, but our subclass does, so ask them. We, as the base class, expect the filter to already be configured and ready to go, with the exception of the input image:

	// Ask the canvas for a representation of itself as an image. Pass that off
	//	to our filter to be applied to.
	CIImage* inputImage = [canvas image];
	[mFilter setValue:inputImage forKey:@"inputImage"];

We ask the canvas for a CIImage representation of itself, and then give that to the filter as input.

Finally, we need to cache the CIContext we’ll be rendering into:

	// Since we're a filter brush, we're rendering right back onto the canvas,
	//	so ask the canvas for the context to render into. Convert the normal
	//	context into a Core Image context, and cache it off (it's expensive to
	//	destory).
	CGContextRef context = [canvas context];
	mCIContext = [[CIContext contextWithCGContext:context options:nil] retain];
}

Not much here. We ask the canvas for its context, then build a CIContext from it.

Rendering a Stamp

Now that we have our filter and context set up, we’re going to start getting render requests from the base class. We handle that in renderInCanvas:

- (void) renderInCanvas:(Canvas *)canvas bounds:(CGRect)bounds at:(NSPoint)point
{
	// Render a single stamp. In our case, that simply means asking the Core Image
	//	context to draw the output of our filter, at the specified stamp location.
	//	Our base class that called us has already set up the mask clipping so
	//	our stamp will be properly shaped.
	[mCIContext drawImage:[mFilter valueForKey:@"outputImage"] atPoint:bounds.origin fromRect:bounds];
}

Since we already have the filter and context, and the base class has already set up the context and helpfully passed in the stamp bounds, all we do is ask the CIContext to render the resulting image. Easy as pie.

Tearing down the Brush Stroke

When we are done with a single brush stroke, we need to clean up our cache. We do that in stopBrush:

- (void) stopBrush
{
	// This gets called after the mouse up, and the last stamp is rendered. We
	//	should do any clean up here.

	// We're done with the filter and context, so free them up.
	[mFilter release];
	[mCIContext release];
	mFilter = nil;
	mCIContext = nil;
}

This is pretty self-explainatory: we release the cached filter and context. Note that this is a fairly expensive operation.

Spacing

As we saw with the smudge tool, stamp spacing changes based the kind of brush. The filter tools, dodge and burn, are no different. So we overload the spacing method:

- (float) spacing
{
	// By filter brushes typically want closer spacing so the effect is smoother
	return 1.0;
}

Through trial and error I discovered decreasing the spacing to one pixel improved the quality of the rendering, so we do that here.

Loading an Image Unit Plugin

Although not used in the FilterBrush class itself, FilterBrush does provide a helper method to load up image unit plugins. This is useful for subclassed brushes who have custom filters stored in the application bundle:

- (void) loadFilter:(NSString *)filterName fromPlugin:(NSString *)pluginName
{
	// Helper function to ensure the given filter is loaded, so we can use it.

	// Ask the system to load up all the plugins it knows about
	[CIPlugIn loadAllPlugIns];
	NSArray *filterList = [CIFilter filterNamesInCategories:nil];

	// Check to see if our filter is loaded (it should be if we added to a system
	//	path.). If it is in a plugin in the application bundle, then it won't
	//	be found.
	if( ![filterList containsObject:filterName]) {
		// It wasn't loaded by default, so manually load it

		// Construct the path to the plugin bundle. We assume it's in the application
		//	bundle, in the plugins folder.
		NSString *path = [[[NSBundle mainBundle] builtInPlugInsPath] stringByAppendingPathComponent:pluginName];

		// Explicitly load the plugin, given the path
		[CIPlugIn loadPlugIn:[NSURL fileURLWithPath:path] allowNonExecutable:NO];
	}
}

This is fairly standard Core Image code. We ask the system to load up all the known Core Image plugins, then look for our specific filter. If it’s there, then we’re done. If not, then we look in our Plug-Ins folder inside our application bundle for the plug-in, and ask CIPlugIn to manually load that plug-in.

Dodge

The Dodge class implements the dodge tool by deriving from the FilterBrush, and overriding the filter method. It has a couple of parameters, which are the same as the Burn tool.

Parameters

The two parameters are initialized in the init method (in addition to the parameters in the GenericBrush init):

- (id) init
{
	self = [super init];
	if ( self != nil ) {
		// First, make sure our filter is loaded. We don't have to do it now
		//	but its convienent here.
		[self loadFilter:@"DodgeBrushFilter" fromPlugin:@"Filters.plugin"];

		// Set the default values for our parameters
		mExposure = 1.0;
		mRange = kDodgeRange_Midtones;
	}
	return self;
}

There are two new parameters for the Dodge and Burn tools: mExposure and mRange.

• mExposure Exposure determines how strong the dodge or burn is applied. It ranges from 0.0 to 1.0, where 0.0 means the effect isn’t applied, and 1.0 is where the effect is at its strongest.

  Dodge examples:

  • mExposure = 0.25,
  • mExposure = 0.5,
  • mExposure = 1.0,
• mRange Range determines the range of pixels that the dodge tool is applied to. It is an enumeration that includes highlights, midtones, and shadows. Highlights are those closest to 1, midtones those closest to 0.5, and shadows are those closest to 0.

  Dodge examples:

  • mRange = kDodgeRange_Highlights,
  • mRange = kDodgeRange_Midtones,
  • mRange = kDodgeRange_Shadows,

We use a black to white linear gradient because it represents the entire component range, from shadows to highlights.

The only other thing to note about the init method is that we manually load up our filter, by invoking the parent class helper method.

Creating a filter

The only real task of the Dodge tool is to create the appropriate filter based on its parameters. This is accomplished in one method, filter:

- (CIFilter *) filter
{
	// We need to create and configure our filter here.

	// Pull out our special filter, and set the exposure to be exactly
	//	what was given to us.
	CIFilter * filter = [CIFilter filterWithName:@"DodgeBrushFilter"];
	[filter setDefaults];
	[filter setValue:[NSNumber numberWithFloat:mExposure] forKey:@"inputExposure"];

	// Configuring the range is slightly more complicated because we have to
	//	convert an enumeration to a straight number.
	switch ( mRange ) {
		case kDodgeRange_Highlights:
			[filter setValue:[NSNumber numberWithInt:kDodgeFilter_Highlights] forKey:@"inputRange"];
			break;
		case kDodgeRange_Midtones:
			[filter setValue:[NSNumber numberWithInt:kDodgeFilter_Midtones] forKey:@"inputRange"];
			break;
		case kDodgeRange_Shadows:
			[filter setValue:[NSNumber numberWithInt:kDodgeFilter_Shadows] forKey:@"inputRange"];
			break;
	}

	return filter;
}

Here we simply ask CIFilter for an instance of our dodge filter (not the system one), which we loaded up in our init method. Note that our dodge tool parameters have a one to one mapping with the dodge filter parameters, so we just pass them through unchanged. After we’re done configuring our filter, we return it.

It should be noted that the Burn class is identical to the Dodge class, except that it uses the burn filter instead of the dodge filter.

DodgeBrushFilter

DodgeBrushFilter is a CIFilter derived class that is housed in an image unit plugin that our main application loads. It is a fairly standard filter, whose main task is to select the correct kernel to apply to the image passed in.

Like most CIFilter classes, we load the kernels in the init method:

static NSArray	*sKernels = nil;

- (id) init
{
	// If we haven't loaded up our array of kernels, do so now
	if ( sKernels == nil ) {
		// Look for our kernel code file inside of our bundle
		NSBundle *bundle = [NSBundle bundleForClass:[self class]];
		NSString *code = [NSString stringWithContentsOfFile:[bundle pathForResource:@"DodgeBrushFilter" ofType:@"cikernel"]];

		// We have three kernels in the file: highlights, midtones, and shadows.
		//	The range parameter selects which one we'll use. Cache them.
		sKernels = [[CIKernel kernelsWithString:code] retain];
	}

	return [super init];
}

We keep our kernels in a static variable so we don’t waste time loading them each time. The kernels are all stored in one file, DodgeBrushFilter.cikernel, which is kept inside the Image Unit Plugin bundle. We load the cikernel file, convert it to an array of CIKernels, and cache those off. There are three kernels loaded: one for each range.

Our dodge filter has two custom parameters, which we declare in the customAttributes method:

- (NSDictionary *) customAttributes
{
	// Return the custom attributes, which, in our case, is just the
	//	exposure and range parameters.
	return [NSDictionary dictionaryWithObjectsAndKeys:

		[NSDictionary dictionaryWithObjectsAndKeys:
			[NSNumber numberWithFloat: 0.0], kCIAttributeMin,
			[NSNumber numberWithFloat: 1.0], kCIAttributeMax,
			[NSNumber numberWithFloat: 0.0], kCIAttributeSliderMin,
			[NSNumber numberWithFloat: 1.0], kCIAttributeSliderMax,
			[NSNumber numberWithFloat: 0.5], kCIAttributeDefault,
			[NSNumber numberWithFloat: 0.0], kCIAttributeIdentity,
			kCIAttributeTypeScalar, kCIAttributeType,
			nil], @"inputExposure",

		[NSDictionary dictionaryWithObjectsAndKeys:
			[NSNumber numberWithInt: kDodgeFilter_Highlights], kCIAttributeMin,
			[NSNumber numberWithInt: kDodgeFilter_Shadows], kCIAttributeMax,
			[NSNumber numberWithInt: kDodgeFilter_Highlights], kCIAttributeSliderMin,
			[NSNumber numberWithInt: kDodgeFilter_Shadows], kCIAttributeSliderMax,
			[NSNumber numberWithInt: kDodgeFilter_Midtones], kCIAttributeDefault,
			kCIAttributeTypeScalar, kCIAttributeType,
			nil], @"inputRange",

		nil];
}

As with the dodge brush, the parameters are exposure and range. They mean exactly the same thing they did in the Dodge class.

The last part of our filter is the outputImage method, which actually constructs the output CIImage:

- (CIImage *) outputImage
{
	// The idea is to apply the kernel selected by the range parameter, to
	//	the image passed in. We don't really do any fancy preprocessing.
	CISampler *sampler = [CISampler samplerWithImage:inputImage];
	CIKernel *kernel = [sKernels objectAtIndex: [inputRange intValue]];

	// Just apply the chosen kernel
	return [self apply:kernel, sampler, inputExposure, kCIApplyOptionDefinition, [sampler definition], nil];
}

This is also a reasonably simple method. We create a sampler from our input image, and pull a kernel out of our cached array using the range parameter as an index. We then apply the chosen kernel to our image, and we’re done.

Like with the brushes, the burn filter is identical to the dodge filter, except that it loads up the burn kernels and applies those instead.

Kernels

The real meat of all this code are the kernels. They are straightforward implementations of the functions presented in the overview section, so not a lot of explanation will be given. The only difference is the kernels deal with implementing the exposure parameter, which the function presented in the overview section ignored.

Dodge Highlights

kernel vec4 dodgeHighlights(sampler image, float exposure)
{
	vec4 source = unpremultiply(sample(image, samplerCoord(image)));

	float factor = exposure;

	source.rgb = source.rgb + factor * (exp(source.rgb) - 1.0);

	return premultiply(source);
}

Dodge highlights is almost identical to the overview function presented. Note that we apply the function to all three components (RGB) at once, and the exp() function is e raised to the value passed in. In this case, the exposure is used directly to scale the steepness of the function.

Dodge Midtones

kernel vec4 dodgeMidtones(sampler image, float exposure)
{
	vec4 source = unpremultiply(sample(image, samplerCoord(image)));

	float pi = radians(180.0);
	float factor = exposure * 0.25;

	source.rgb = source.rgb + factor * sin(source.rgb * pi);

	return premultiply(source);
}

Dodge midtones is fairly straightforward. We scale the exposure by a quarter just so we don’t push values to 1 too quickly.

Dodge Shadows

kernel vec4 dodgeShadows(sampler image, float exposure)
{
	vec4 source = unpremultiply(sample(image, samplerCoord(image)));
	float factor = (1.0 - exposure * 0.5);

	source.rgb = factor * source.rgb + (1.0 - factor);

	return premultiply(source);
}

For dodging shadows, we scale the exposure by half so we don’t push everything to 1 at its peak. Also, since we’re using a linear function, we have to make the y intercept dependent on the slope. i.e. Both slope and y intercept are based on the exposure.

Burn Highlights

kernel vec4 burnHighlights(sampler image, float exposure)
{
	vec4 source = unpremultiply(sample(image, samplerCoord(image)));

	float factor = (1.0 - exposure * 0.75);

	source.rgb = factor * source.rgb;

	return premultiply(source);
}

Note that burning highlights is also a linear function. However there is no y-intercept because we want it to intersect at (0, 0). Also note that we scale the exposure by 3/4.

Burn Midtones

kernel vec4 burnMidtones(sampler image, float exposure)
{
	vec4 source = unpremultiply(sample(image, samplerCoord(image)));

	float pi = radians(180.0);
	float factor = exposure * 0.25;

	source.rgb = source.rgb - factor * sin(source.rgb * pi);

	return premultiply(source);
}

Burning midtones is identical to dodging midtones, except that we subtract off the sine wave. Like before, we scale the exposure by 1/4 before using it.

Burn Shadows

kernel vec4 burnShadows(sampler image, float exposure)
{
	vec4 source = unpremultiply(sample(image, samplerCoord(image)));

	float factor = exposure;

	source.rgb = source.rgb + factor * (1.0 - exp(1.0 - source.rgb));

	return premultiply(source);
}

Burning shadows is the same idea as dodging highlights, except that we’ve flipped the function upside down.

Method to the Madness

It took me a long time to figure out the current kernels for each of the ranges. On the surface, dodge and burn sound easy: you just lighten or darken the image. But when I started playing with existing dodge and burn tools, I found that it wasn’t quite that simple.

I started out with simple linear functions for the highlights and shadows. With some tweaks this worked respectably well for dodging shadows and burning highlights, but was way off for dodging highlights and burning shadows. In other implementations these tools pushed components to the extreme (either 0 or 1 depending on the tool) very quickly. It seemed exponential to me, which lead to the current implementation eventually. I started with base 2 instead of e though, on the presumption that 2 was more likely because it would execute faster.

Dealing with midtones took me a while. I knew I needed a curve that would be highest in the middle, but wasn’t sure which function would yield the best results. I actually started out with a gaussian function, but I concluded that was way too complex for anyone to really use for a filter. I then played around with making a curve with smooth step, but I wasn’t happy with the results. I’m not sure why sin wasn’t the first function to pop into my head when I knew I needed a curve, but it turned out to be the one to yield the best results.

I’m not sure if this how the big graphics apps implement dodge and burn, but the results seem reasonably close to me. Unfortunately no one seems to publish how they implemented dodge and burn. As a result, this took me a long time to figure out. I literally started writing this article three times before, but stopped because I wasn’t happy with the results of the tools. I also spent a lot of time in Grapher, Apple’s graphing calculator program staring at the graphs of various functions.

Conclusion

In addition to showcasing some relatively complex brushes, this article introduces a framework that we can build off of to implement other filter brushes. For example: sharpen, blur, and sponge tools.

This is one of the more satisfying articles to write, just because of what it took to implement the dodge and burn tools. Enjoy, and be sure to download the sample code.

Overview

As I mentioned in the previous article, converting an image mask to a vector path is not exactly a cheap operation. Will suggested much simpler way that involved using strictly Core Image.

The basic idea is to use the CIStripesGenerator Core Image filter to generate us up some black and white vertical lines. We then do an edge detection filter, CIEdges, on our selection mask to calculate a new mask, representing where the marching ants should show up. We then do a multiply composite, using CIMultiplyCompositing, to merge the striped lines with our marching ants mask. The result is the stripes only show up at the edges of the selection mask. Ta-da, marching ants.

OK, its a little more complicated than that, but the previous paragraph should give you a pretty good idea what’s going on.

Code

Like before, I have sample code to go along with what I’m going to show. However, unlike before, this sample code is heavily based on the previous article’s sample code. In fact, I really only modified one class from the Magic Wand sample code. So instead of going through all that code again, I’m going to assume you know how it works, and only highlight the new stuff.

SelectionBuilder

OK, I have to admit right off the top I lied about only having to modify one class. I had to modify SelectionBuilder slightly in order to get the generated mask to work with Core Image.

Instead of generating a true image mask via CGImageMaskCreate, I had to create a CGImageRef with a grayscale colorspace and no alpha. This meant that I had to:

1. Flip the colors. In SelectionBuilder, black now means not in the selection, while white means in the selection. In the init method, the mask data is calloc’d and left zeroed out. When we mark a point in the selection, we set it to white.
2. Use CGImageCreate instead of CGImageMaskCreate.

Fortunately, CoreGraphics doesn’t care, outside of mask creation, if it’s really an image or an image mask. So no other classes or code had to be modified for this particular change.

CanvasView

CanvasView is really the class that had to change, and it was mainly in the drawRect method. Other than that, it was simply stripping out the mCachedPath member data since it wasn’t needed anymore. In fact, I’m only going to cover the drawRect method. If you would like to see how the rest of the code changed download the sample code.

The new drawRect method starts out normal enough:

- (void)drawRect:(NSRect)rect {
	// Simply ask the canvas to draw into the current context, given the
	//	rectangle specified. A more sophisticated view might draw a border
	//	around the canvas, or a pasteboard in the case that the view was
	//	bigger than the canvas.
	NSGraphicsContext* context = [NSGraphicsContext currentContext];

	[mCanvas drawRect:rect inContext:context];

	// If we don't have a selection, bail now
	if ( mSelection == nil )
		return;

We just draw the contents of the canvas. If we don’t have a selection, we can stop right here (but that wouldn’t be very interesting, now would it?).

The first thing we need to do is convert our selection mask into something Core Image can use:

	// Create a CIImage from our selection image. It's important that our mSelection
	//	has to be an actual image, not an image mask as created by CGImageMaskCreate.
	//	CIImage will not create the proper image with a CGImageRef created with
	//	CGImageMaskCreate.
	CIImage *selectionImage = [CIImage imageWithCGImage:mSelection];

	// The first thing we want to do is edge detection. We make the assumption
	//	that our mask has only two colors: black and white. If we were to do
	//	some antialiasing in it, we might have to do some posterization to
	//	reduce the number of colors before running the edges filter.
	CIFilter* edgesFilter = [CIFilter filterWithName:@"CIEdges"];
	[edgesFilter setDefaults];
	[edgesFilter setValue:selectionImage forKey:@"inputImage"];

	// In order to use our mask, convert it into an alpha channel
	CIFilter* maskToAlphaFilter = [CIFilter filterWithName:@"CIMaskToAlpha"];
	[maskToAlphaFilter setDefaults];
	[maskToAlphaFilter setValue:[edgesFilter valueForKey:@"outputImage"] forKey:@"inputImage"];

We also go ahead and do an edge detection on our mask. Since we know that our mask only ever has two colors, we don’t need to do any posterization on it beforehand. In a real system, we might have antialiasing, and might need to reduce the number of colors with posterization. We then convert our new mask into an alpha channel so we can use it in a compositing filter later.

To illustrate this, suppose our selection is this:

our image mask would then be:

After we apply the edges filter to our mask, it would be:

As you can see the mask is now white where we want our marching ants to appear. Applying the mask to alpha filter then means it has an alpha of 1.0 (opaque) where it is white, and an alpha of 0.0 (transparent) where it is black.

Now that we have our mask, we need to generate our stripes that we’re going to use for the ants:

	// Generate vertical black and white stripes that are 4 pixels wide.
	//	We animate the marching ants here by shifting the y axis to the right
	//	each time through.
	CIFilter* stripesFilter = [CIFilter filterWithName:@"CIStripesGenerator"];
	[stripesFilter setDefaults];
	[stripesFilter setValue: [CIColor colorWithRed:0.0 green:0.0 blue:0.0 alpha:1.0] forKey:@"inputColor0"];
	[stripesFilter setValue: [CIColor colorWithRed:1.0 green:1.0 blue:1.0 alpha:1.0] forKey:@"inputColor1"];
	[stripesFilter setValue: [NSNumber numberWithFloat:4.0] forKey:@"inputWidth"];
	[stripesFilter setValue: [CIVector vectorWithX:mPhase Y:150.0] forKey:@"inputCenter"];

We use the CIStripesGenerator to create some vertical black and white alternating lines. We set the width to four simply because that was the line dash width we used in the original marching ants algorithm. However, because of the next step, the line segments won’t exactly be four pixels wide everywhere.

We also implement the animation of the marching ants here. One of the parameters of the stripes filter is where the center of the generated lines are. By incrementing the x value of the center point, we shift the vertical lines to the right each time through the animation, which makes the ants “march.”

Our initial stripes filter image would look like this:

In order to get our stripes to show up on all edges of a selection correctly, we need to tilt the stripes in one direction:

	// We have vertical stripes, which will look good on the top and bottom edges of
	//	the selection, but will appear as a solid colored line on the left and right.
	//	So that most border shapes will appear dashed, rotate the vertical lines
	//	by 45 degrees.
	CIFilter *affineTransform = [CIFilter filterWithName:@"CIAffineTransform"];
	NSAffineTransform *rotateTransform = [NSAffineTransform transform];
	[rotateTransform rotateByDegrees:-45];
	[affineTransform setDefaults];
	[affineTransform setValue:[stripesFilter valueForKey:@"outputImage"] forKey:@"inputImage"];
	[affineTransform setValue:rotateTransform forKey:@"inputTransform"];

The problem with leaving the stripes vertical is that they wouldn’t look right on the vertical edges of the selection. The top and bottom edges of the selection would nicely alternate between black and white, but the left and right edges would be one solid color.

To fix this we apply an affine transform to rotate the lines 45 degrees. Our stripes now look like:

We now have our two parts: the stripes that will be our ants, and the mask that marks where they should go. We only need to combine them:

	// The last filter we apply combines our newly created stripes with our mask.
	CIFilter *multiplyFilter = [CIFilter filterWithName:@"CIMultiplyCompositing"];
	[multiplyFilter setDefaults];
	[multiplyFilter setValue:[maskToAlphaFilter valueForKey:@"outputImage"] forKey:@"inputImage"];
	[multiplyFilter setValue:[affineTransform valueForKey:@"outputImage"] forKey:@"inputBackgroundImage"];

	// Finally, render our creation to the view.
	CIContext *ciContext = [context CIContext];
	CGRect imageRect = CGRectMake(0, 0, CGImageGetWidth(mSelection), CGImageGetHeight(mSelection));
	[ciContext drawImage:[multiplyFilter valueForKey:@"outputImage"] inRect:imageRect fromRect:imageRect];

We use the multiply compositing filter to combine the images. This works because our edge mask’s alpha is 1.0 at the edges and 0.0 everywhere else. When you multiply the alpha channels of the two images together, it filters out everything but the edges, thus giving us ants around the selection.

Since we now have our fully formed ants, we render them to the screen using a CIContext created from our NSGraphicsContext.

Just for completeness, here’s the last bit of the drawRect fuction:

	// The marching ants need to animate, so fire off a timer half a second later.
	//	It will update the mPhase member and then invalidate the view so
	//	it will redraw.
	[NSTimer scheduledTimerWithTimeInterval:0.5 target:self selector:@selector(onSelectionTimer:) userInfo:nil repeats:NO];
}

Nothing new here: we just fire off a timer that will increment the phase member and invalidate the view so the ants march.

That’s it, we’re done. Much less involved for us than the previous algorithm.

Conclusion

Once again, I’d like to thank Will Thimbleby for suggesting this approach. I have to admit: I think using a NSBezierPath to render the ants looks better. However for sufficiently complex or large image masks, it may be prohibitively expensive to use.

Download the sample code

Anyway, if you’d like to hear someone (me, specifically) drone on about Core Image Filters and Image Unit Plugins, you should tune into Late Night Cocoa: Core Image Filters, if its not already in your iTunes feed. It has lots of detailed information on Core Image filters, spoken in a high nasally voice.

Just think of it as James Earl Jones’ voice in drag.

If only one of the previous examples is in your Image Unit folder, then everything works fine. The problem comes in when you have both of them in the Image Unit folder. In that case, neither show up in the UI. The problem is Core Image cannot uniquely identify the filters (i.e. it thinks they are the same, and gets confused.)

At first, I thought I just needed to add a unique bundle identifier to my Image Unit package. Unfortunately, that is not the case, which makes sense because you can have several filters in one Image Unit package.

The problem lies in the Description.plist. From the template, there are three keys which have the value “MyKernelFilter”, and one key which itself is “MyKernelFilter.” Two of the values and the key need to be changed to something unique.

If you look under the first key in the Description file, CIPlugInFilterList, you’ll see a dictionary of filter descriptions. You’ll note that in my examples, such as the HSB Mixer, I left the key for my filter description alone, as MyKernelFilter. I need to change it to be unique, like so:

...
<key>CIPlugInFilterList</key>
<dict>
<key>HSBMixer</key>
<dict>
...


Now that I’ve picked “HSBMixer” as my unique key, I need to replace MyKernelFilter with it in a couple of places, namely the CIFilterClass and CIKernelFile values. So at the bottom of the Description file:

...
<key>CIFilterClass</key>
<string>HSBMixer</string>
<key>CIHasCustomInterface</key>
<false/>
<key>CIKernelFile</key>
<string>HSBMixer</string>
...


CIHasCustomInterface didn’t change, I just left it in there because it was sandwiched between the two values that I did change.

That should cover it. If you rebuild and add both of the examples to your Image Units folder, they will both show up. This is pretty tedious work, so in future posts, I will just assume this is known and skip over it.

I’m going to build off what I learned from my previous Core Image example, this time adding user parameters, demonstrating how to build functions in the Core Image Kernel Language, and other exciting magic tricks.

This filter is going to simply allow the user to make adjustments to the image in the HSB color space. So I probably need to come up with a few definitions as to what HSB color space is.

HSB Color Space

HSB, aka HSV, is a color space that defines color by three components: hue, saturation, and brightness (sometimes called value). Hue describes the color type (e.g. red, green, blue, etc) and is measured in degrees, from 0 to 360. Saturation is the vibrancy of the color, typically measured from 0 to 100. The less saturation, the more gray a color appears. The value, or brightness of a color, is essentially what it sounds like, and typically ranges from 0 to 100.

The HSB color space is a cone, with the hue running the circumference of the base, saturation running from the center to the edge of the cone, and brightness running the height of the cone. It made a lot more sense to me when I looked a few models, which can be found at Wikipedia’s entry on the HSV color space.

Project and Resources

Like last time, I want to create a project from the “Image Unit Plug-in for Objective-C” template. I’ve named mine “HSB Mixer.”

Next, I need to modify the Description.plist file to tell Core Image about the parameters my filter has, and what category my filter belongs to, which effects where it shows up in the UI.

Under the CIAttributeFilterCategories key in the plist, I find the different default categories HSB Mixer belongs to. Although the HSB Mixer will work with both video and still images, it’s not really a “stylize” filter, but more of a color adjustment. To that end, I change CICategoryStylize to CICategoryColorAdjustment:

<key>CIAttributeFilterCategories</key>
<array>
<string>CICategoryColorAdjustment</string>
<string>CICategoryVideo</string>
<string>CICategoryStillImage</string>
</array>


Now I need to add my parameters. By default, the image to modify is the first parameter, followed by two number parameters. Since this isn’t a generation filter, I need the image parameter, but instead of the two default number parameters, I need three number parameters for hue, saturation, and brightness. So I delete the last two parameters (the last two dict elements) under the CIInputs key.

The first parameter that I add is hue:

<dict>
<key>CIAttributeClass</key>
<string>NSNumber</string>
<key>CIAttributeDefault</key>
<real>0</real>
<key>CIAttributeDisplayName</key>
<string>inputHue</string>
<key>CIAttributeIdentity</key>
<real>0</real>
<key>CIAttributeName</key>
<string>inputHue</string>
<key>CIAttributeSliderMax</key>
<real>6</real>
<key>CIAttributeSliderMin</key>
<real>-6</real>
</dict>


The first key is CIAttributeClass, and it simply states the corresponding Cocoa class to use for the given parameter. All the possible values can be found in the document Modify the Description Property List in Table 4-2.

The CIAttributeDefault key describes the default value of the parameter in the UI. I choose 0 as the default value, because it is also the identity value (see below).

CIAttributeDisplayName is a key into the Description.strings file, and determines the localized name that is shown to the user.

The CIAttributeIdentity key specifies a value for the parameter which produces the identity of the input image. i.e. It’s a value such that the filter doesn’t change the source image in any way. In this case, a value of zero means the filter doesn’t change the hue of the image.

CIAttributeName is an internal name for the parameter. Core Image Filters are “Live Effects” in Fireworks parlance, meaning they store their parameters so they can be modified later, and they don’t modify the source image directly. Instead, they generate a new image based on the source image plus the filter parameters. In other words, they’re editable. To that end, each of the parameters I define need to be stored in an NSDictionary, and CIAttributeName is simply the key for each parameter in that dictionary.

CIAttributeSliderMin and CIAttributeSliderMax define the range of the slider control used to represent the parameter. Although hue ranges from 0 to 360 degrees, I’m going to scale it down to 0 to 6, using real numbers. Since I want to be able to both add and subtract from the hue, so all values are achievable, I let the range be -6 to 6.

The parameters for saturation and value are the same as hue, except they range from -1 to 1, and obviously have different names:

<dict>
<key>CIAttributeClass</key>
<string>NSNumber</string>
<key>CIAttributeDefault</key>
<real>0</real>
<key>CIAttributeDisplayName</key>
<string>inputSaturation</string>
<key>CIAttributeIdentity</key>
<real>0</real>
<key>CIAttributeName</key>
<string>inputSaturation</string>
<key>CIAttributeSliderMax</key>
<real>1</real>
<key>CIAttributeSliderMin</key>
<real>-1</real>
</dict>
<dict>
<key>CIAttributeClass</key>
<string>NSNumber</string>
<key>CIAttributeDefault</key>
<real>0</real>
<key>CIAttributeDisplayName</key>
<string>inputValue</string>
<key>CIAttributeIdentity</key>
<real>0</real>
<key>CIAttributeName</key>
<string>inputValue</string>
<key>CIAttributeSliderMax</key>
<real>1</real>
<key>CIAttributeSliderMin</key>
<real>-1</real>
</dict>


Finally, we need to modify Description.strings to have localized versions of our parameter display names:

"MyKernelFilter" = "HSB Mixer";
"inputHue" = "Hue";
"inputSaturation" = "Saturation";
"inputValue" = "Brightness";


Kernel Code

The algorithm for the kernel is simple:

1. Sample the current pixel
2. Convert the current pixel from RGB to HSV
3. Add the hue, saturation, and value input parameters to the corresponding component in the current HSV pixel
4. Convert the adjusted HSV pixel back to RGB
5. Return the new RGB pixel

The kernel function is defined as:

kernel vec4 hueSaturationKernel(sampler image, float inputHue, float inputSaturation, float inputValue)
{
// Get source pixel
vec4 p = sample(image, samplerCoord(image));

// Convert back to RGB color space
return hsvToRgb(hsvPixel);
}


Note that my user parameters are just passed in, in the order I declared them in Description.plist. The kernel function itself is pretty straightforward. rgbToHsv() and hsvToRgb() are the only user defined functions, and they convert to the HSV color space and back. Notice that the vec4 data type always assumes rgba, so when you see hsvPixel.r here, it actually corresponds to the hue. Similarly hsvPixel.g corresponds to saturation and hsvPixel.b corresponds to value, although the alpha channel remains untouched.

The tricky part is dealing with HSV to RGB and RGB to HSV conversions. There is code all over the place to deal with this, but I used the C code from an ACM paper as a template. The algorithm there scales the hue down to be from 0 to 6, which is why my input parameters do as well. I won’t go into the math behind the conversions here, because it is covered in any graphics textbook, as well as several places on the web, including Wikipedia.

The easier of the conversions, code wise, is RGB to HSV:

vec4 rgbToHsv(vec4 rgb)
{
float x = min(rgb.r, min(rgb.g, rgb.b));
float v = max(rgb.r, max(rgb.g, rgb.b));
float f = (rgb.r == x) ? rgb.g - rgb.b : ((rgb.g == x) ? rgb.b - rgb.r : rgb.r - rgb.g);
float i = (rgb.r == x) ? 3.0 : ((rgb.g == x) ? 5.0 : 1.0);
float h = i - (f / (v - x));
float s = (v - x) / v;

return (v == x) ? vec4(-1, 0, v, rgb.a) : vec4(h, s, v, rgb.a);
}


Just about everything in this function is straightforward. min() and max() are built in functions that will work on any numerical type.

You might notice that there is potentially wasted computation here, in the special case of v == x. In that case, I just return v, meaning I didn’t need to compute f, i, h, or s. The reason for this is that the Core Image Kernel Language doesn’t support if statements that are data dependent. That is, the expression inside of an if statement can only contain a constant.

Correct:

if ( true ) v = x;
if ( false ) x = v;


Incorrect:

if ( v == x ) v = x; // Error!
bool val = true;
if ( val ) x = v; // Error!


The kernel language does however allow use of the ternary operator, so that’s what I’m forced to use here. The side effect of that is that I cannot return early, even if all the relevant calculations are complete. I’m guessing the reason for this restriction is related to the abilities of most GPU’s.

The HSV to RGB conversion is sightly more complicated:

vec4 hsvToRgb(vec4 hsv)
{
float h = hsv.r;
float s = hsv.g;
float v = hsv.b;
int i = int(floor(h));
float f = isOdd(i) ? h - float(i) : 1.0 - (h - float(i));
float m = v * (1.0 - s);
float n = v * (1.0 - s * f);
vec4 result = (i == 0) ? vec4(v, n, m, hsv.a) : ((i == 1) ? vec4(n, v, m, hsv.a) : ((i == 2) ? vec4(m, v, n, hsv.a) : ((i == 3) ? vec4(m, n, v, hsv.a) : ((i == 4) ? vec4(n, m, v, hsv.a) : ((i == 5) ? vec4(v, m, n, hsv.a) : vec4(v, n, m, hsv.a))))));

return (h == -1.0) ? vec4(v, v, v, hsv.a) : result;
}


The first thing to note is that all type conversions must be explicit. To get i I call floor(), a built in function, on h, then explicitly cast it to an int. I’m not sure of design decisions made here, but I would hazard to guess that float to int conversions, and vice versa, are expensive.

The only other thing to note here is the last user defined function, isOdd():

bool isOdd(int v)
{
float dividend = float(v) / 2.0;
return dividend != floor(dividend);
}


I call attention to this function because of its technique. Usually, to determine if a number is odd, I bitwise AND against a bitmask of 0×0001, or take the modulus by 2 and see if it has a remainder. However, the kernel language does not support any bitwise operators or modulus. Here, I divide by two and see if has fractional part by comparing the dividend against the floor of itself. This is error prone because of rounding errors. Another way would be to take the difference between the dividend and its floor, and see if it is greater than a specified small number.

Trying it out

Like before, I like using the Core Image Fun House for experimenting with my filters. I just need to copy my built filter to ~/Library/Graphics/Image Units/ to do that.

Here’s Flowers.jpg with the HSB Mixer applied to it. The parameters are Hue: 2.0, Saturation: -0.48, Brightness: 0.0.

Conclusion

Hopefully you’ve learned more about Core Image Filters from this post. We’ve learned that simply declaring input parameters in a plist file, Core Image clients can automatically construct UI for your filter. Also, we’ve learned that although the kernel language is powerful, it has some serious limitations that we have to work around.

Download HSB Mixer Source Code

It wasn’t until the Fireworks 8 cycle that I really got a chance to start playing with image filters. I wrote several, but because I was pulled off Fireworks just a few months into Fireworks 8, only a couple of them actually made it into the final product. That didn’t end my interest in filters, but it did end my chance of publishing them.

I’ve also come to the conclusion that Ted Turner got it backwards. Instead of colorizing old movies, he should have been making all the new movies grayscale. As any artist can tell you, making images and/or movies black and white automatically makes them artsy. At least, that’s what all the artists I just made up in my head say.

The epiphany came today when I realized I should combine these two ideas, and create a filter that removes color. Well, that and show how to write Core Image filters. That might be interesting too.

Creating a project

I want to create a standalone Image Unit Plug-in. Fortunately, Xcode has a template just for that. To get this artistic revolution started, I select File > New Project… from Xcode’s menu. From here, I pick the “Image Unit Plug-in for Objective-C” template.

Next, I name my project “Grayscale,” because that’s what the plugin will do, and because I’m not a crazy Brit who insists on spelling gray with an “e.” The way they prattle on about it, you’d think they’d invented the language or something.

Modifying the description

Now that I have a project, the first thing I want to do is modify the description property list. The Description.plist is what tells Core Image what the parameters for the filter are, the category of the filter, and where to find the kernel, which, surprisingly, isn’t always at KFC.

In order to find my filter later to use it, I need to give it a meaningful category. Apple has several predefined categories that I have to choose from, thus cramping my artistic style. Despite this, I can still improve on the default choice of CICategoryStylize, by changing it to CICategoryColorEffect.

After my change, the Description.plist category section looks like this:

<key>CIAttributeFilterCategories</key>
<array>
<string>CICategoryColorEffect</string>
<string>CICategoryVideo</string>
<string>CICategoryStillImage</string>
</array>


The most important thing to modify the in Description.plist is the description of the filter’s parameters. It turns out that I don’t need any input parameters for this colorless revolution, with the exception of the source image. The default Description.plist comes with three parameters, so I’ll delete the last two parameters.

That concludes everything I need to do with Description.plist. However, I still need to change the display name that shows up in the UI for my filter. For that I open up Description.strings:

"MyKernelFilter" = "Demo CIKernel Only Filter";
"inputScale" = "Scale";
"inputGreenWeight" = "Green Weight";


Since I don’t have any user inputs, I can ditch the two last strings, inputScale and inputGreenWeight. MyKernelFilter is the name of my filter, so I’ll change it to “Grayscale.”

"MyKernelFilter" = "Grayscale";


That does it for all the meta-information for my filter plug-in.

Kernel

Now that I’ve set up all the resources for my filter, I need to write the actual code. The kernel code is written in what is called Core Image Kernel Language, which is a subset of the OpenGL Shading Language.

To start, I open up MyKernelFilter.cikernel, and delete it’s current contents. In its place, I add the following code:

kernel vec4 grayscaleKernel(sampler image)
{
// Get source pixel
vec4 p = sample(image, samplerCoord(image));

// Set the destination pixel based on intensity
return vec4(intensity, intensity, intensity, p.a);
}


It doesn’t really matter what I name the function, as long as it is tagged with the kernel keyword. The kernel, in this case, acts as a map from the source pixel value, to the destination pixel value. You’ll note that it returns a vector of four elements (Red/Green/Blue/Alpha) and takes the source image as the only parameter. If I had other parameters, they would come after the image parameter, just like in the Description.plist file.

The first line simply retrieves the pixel from the source image:

vec4 p = sample(image, samplerCoord(image));


As established earlier, image is the source image passed in as a parameter. samplerCoord() returns the coordinates of the source pixel inside the source image. sample() does what it says, and will return a single pixel value given an image and coordinates.

The second line calculates the intensity of the source pixel:

float intensity = clamp(0.3* p.r + 0.59* p.g + 0.11* p.b, 0.0, 1.0);


To do this I add the weighted RGB components together. The reason behind the weights goes beyond what I want to cover here, but I use intensity calculation from the YIQ color model. For safety reasons I clamp the value to between 0 and 1, because each color component has to stay in that range.

Finally, I return the value of the destination pixel:

return vec4(intensity, intensity, intensity, p.a);


I’m simply setting each of the RGB components to the intensity, which will make the color a shade of gray. Also note that I’m leaving the alpha channel alone.

That’s it as far as the coding goes. I just need to select Build > Build from the menu to complete my filter.

Making cone cells obsolete

Before I can do anything with my Grayscale filter, I need to install it. Core Image looks in a couple of different places, but the easiest folder to stick it in is: ~/Library/Graphics/Image Units/.

Although I could use Quartz Composer to try out my filter, I like the Core Image Fun House, because it’s simpler and more to the point. It can be found at /Developer/Applications/Graphics Tools/. When I open the Fun House, I’m prompted to pick a source image. I usually pick the Wolf.jpg, whom I’ve nicknamed “Rentzsch”, for reasons that escape me.

After picking an image, I click the plus button, and pick my Grayscale filter from under the Color Effect category.

And there you have it. A very artistic, extraordinary, gray wolf, of whom there are none like. Hmmm… maybe I should have created a green filter.

Conclusion

Although I avoided Quartz Composer for this simple example, it is pretty much required for any filter of sufficient complexity. When building, Xcode doesn’t actually compile the kernel or do any type of checking. As far as I can tell, only the Core Image Kernel tool in Quartz Composer will do checking. Unfortunately, Quartz Composer doesn’t allow me to inspect variables or step through statements. It only does basic syntax and type checking.

Also, the shading language, especially with Apple’s restrictions, is pretty limiting. Even with simple kernels, I struggled to get them to work with the kernel language.

Despite the restrictions, I’m having a lot of fun playing around with Core Image kernels. I’ve written a few others that I may, at some point in the future, show off here. Until then, have fun viewing the world through the eyes of a dog.

Download Grayscale Core Image Filter