Archive for October, 2006

Objective-C 3.0

My previous post about Key-Value Coding and being able dynamically add transient attributes to NSManagedObject got me thinking. Why can’t I dynamically add methods? I don’t mean on just NSManagedObjects, I mean on NSObjects. I realize that categories allow that somewhat, but only as a group of functions and only at compile time.

Anyway, this train of thought got me to thinking about what I’d like to see in the next version of Objective-C. Since 2.0 is pretty much a done deal, I’m thinking forward to Objective-C 3.0. I also ended up thinking about what it would take to implement the proposed features, in hopes it would give me a better idea as to what was feasible and what wasn’t. That said, I could totally be smoking crack, and none of this may be feasible.

Anonymous Functions

In the new Objective-C, I want at least some basic anonymous function support. To use JavaScript as an example: = function() { alert('foo called'); }

For the JavaScript uninitiated, this adds a function called “foo” to the current object. In Objective-C, it would end up looking like:

[self addMethod: @function() { NSLog(@"foo called"); }

As shown, this should be easily implementable because the anonymous function only touches global symbols. How to access globals doesn’t change from a normal function to an anonymous function.

Accessing instance variables in anonymous functions

Things get more interesting, and more difficult to implement, if you allow the code block to touch instance variables and methods. For example:

@interface Foo {
	int count;
- (float) cost;

@implementation Foo

- (id) init {
	[self addMethod: @function() {
		NSLog(@"%d items at %f each",
			self->count, [self cost]);


In the above example, accessing the method, cost, should be pretty trivial to implement inside of a anonymous function. self is just a parameter to the function, and the compiler just needs to send a message to it.

Unfortunately, accessing the ivar, count, off of self, is probably impossible to implement. When the anonymous function is compiled, it doesn’t know which class it is attached to. This is a problem because in C, structure accesses are just byte offsets from the address of the structure. On any given class, count could be anywhere within the structure. Since the anonymous function doesn’t know which class to look at, it doesn’t know what byte offset to generate.

Fortunately, it looks like Objective-C 2.0 is going to add property support, which is basically syntatic sugar. Properties will automatically generate methods needed to access or mutate them. This brings properties into the realm of feasibility inside of anonymous functions because they can be treated as method calls.

Accessing local variables in anonymous functions

Although the ability to access and use member variables inside of anonymous functions is interesting, it’s also important to be able to access local variables. For example:

- (id) init {
	int count = 20;

	[self addMethod: @function() {
		NSLog(@"%d items", count);

	[self foo]; // indirectly call the anonymous function

- (void) foo {
	[self display]; // invoke the added method

I should point out immediately that this example is an extremely contrived case, and doesn’t show how useful accessing locals from anonymous functions are. I need to introduce another concept before the usefulness becomes apparent. So for now, I’ll assume that it’s important, and just delve into how this might be implemented.

In the example given, count is a local in init, and it is used in the anonymous function. The compiler needs a mechanism to “bind” the local count to the anonymous function such that every time the function is invoked it can locate the local in memory. The compiler cannot assume that the anonymous function will always be invoked from the function that it was created in, but might instead be invoked by a function further down the stack. e.g. init calls foo and foo calls display. This fact rules out the possibility of passing in the local as a parameter to the anonymous function.

Normally locals are accessed by an offset from the base of a function’s stack frame. So if the code in the anonymous function can find the right stack frame for init, it can easily find the local count. It is also easy for code to walk up the call stack, and thus iterate the stack frames.

However, there is nothing currently in the stack frame that would uniquely identify a function. So the compiler would have to generate a context id for each function that defines an anonymous function, and generate code to store it in the stack frame. At runtime, the anonymous function would iterate up the stack frame until it found the correct context id, and then use the corresponding stack frame pointer as a base for the local’s byte offset.

The problem with this implementation is that I don’t remember seeing a space in the stack frame to store a context id. It would have to be a fixed byte offset from the base of the stack frame in order to be easily found. This is based off my memory of the PowerPC calling conventions. I unfortunately don’t remember my Intel calling conventions from my college days.

If the context id can’t be stored directly in the stack frame, then it’s possible to create a parallel stack which contains only the context ids. It would have to be stored as pairs of context ids and stack frame pointers. The downside of this approach is care would have to be taken to ensure thread safety.

Finally, there’s an obvious problem with the lifetime of local data. After init returns, count ceases to exist, and any call to display will result in a crash. Although at first tempted to try to fix this, I actually think it’s OK to allow the crash at runtime. This wouldn’t fly in the interpreted language world (like Ruby), but Objective-C isn’t interpreted. Objective-C programmers know they shouldn’t return pointers to locals, so creating anonymous functions that reference locals that will go out of scope shouldn’t be foreign or surprising to them.


Of course, the whole reason I bring up anonymous functions is because I want closures. Let’s take an example from Ruby:

array.each { |element| print element }

For those unfamilar with Ruby, some explaination is required. array, is, um, an array, and each is a instance method on array. each walks each element in the array, and calls the anonymous function attached to it, which is everything inside the curly braces, passing in the element. Parameters to the anonymous function are declared inside the pipe characters.

In Objective-C, the code would look something like:

NSArray* array = ...;
[array each] @function (id element) {
	NSLog( @"element: %@", element );

Where the implementation of each would look like:

- (void) each {
	NSEnumerator* enumerator = [self objectEnumerator];
	id element = nil;
	while ( element = [enumerator nextObject] )
		[closure yield:element];

Most of the above code is self explanatory. The new stuff is the closure object. Like the self object, it is a hidden parameter that is passed into the method except that it points to the anonymous function. The yield method takes a variable argument list, and actually invokes the anonymous function.

The interesting thing about the above code is that it implies anonymous functions can be encapsulated by objects. I’m not sure how feasible this is, however. An alternate syntax might be:

- (void) each {
	NSEnumerator* enumerator = [self objectEnumerator];
	id element = nil;
	while ( element = [enumerator nextObject] )

The only gotcha to the new syntax is that it doesn’t allow the called function to determine if there’s a closure attached, and change its behavior. As an example as to why you might want this can be illustrated via an NSArray initWithCapacity method:

- (id) initWithCapacity:(unsigned) size {
	if ( closure == nil ) {
		// Normal initialization with capacity
	} else {
		for(unsigned i = 0; i < size; ++i)
			[self addObject: [closure yield:i]];

NSArray* uninitializedArray = [[NSArray alloc] initWithCapacity:20];
NSArray* initializedArray = [[NSArray alloc] initWithCapacity:20]
	@function(unsigned index) {
		return [NSNumber numberWithUnsignedInt: index * 2];

As you can see, initWithCapacity changes behavior depending on if an anonymous function is attached or not. If there isn't a function, then it simply allocates an array of a given size. If there is a closure, then it calls the function to generate each element.

So, as you can see, it's advantageous inside of a method to know if there is an attached anonymous function. If @yield is used, then you obviously lose this ability. There are ways around this, such as introducing a new hidden BOOL parameter, or another code flow construct. However, they aren't quite as elegant.


Earlier I mentioned that it is an interesting feature to allow anonymous functions to access local variables. This becomes more apparent when using closures. For example:

unsigned sum = 0;
[array each] @function(NSNumber* element) {
	sum += [element unsignedIntValue];

You also don't have to write separate functions for threads. With closures you could write something like:

[NSThread spawn] @function() {
	NSLog(@"Hello World from a thread.");

I can take my earlier example of dynamically adding a method and make it simpler with closures:

[self addMethod: @selector(foo)] @function() {
	NSLog(@"foo called");


If you go back to the Ruby example, you'll notice that it has a much nicer syntax. Anonymous functions look like normal code blocks in Ruby. I don't think that would be reproducible in Objective-C because of the backwards compatibility to C. Furthermore, Objective-C likes to prepend an '@' to all the Objective-C features. I'm not sure if it's for technical reasons or purely aesthetic reasons, but it's unlikely that Objective-C would drop it for new features. In an attempt to clean up the sum example, it could look like:

unsigned sum = 0;
[array each] @ {
	@| NSNumber* element |

	sum += [element unsignedIntValue];

However, I'm not sure how much of an improvement this really is. I had to retain the '@' character, so the anonymous function still doesn't look like normal code block. What's more, the use of pipe characters to declare parameters will probably look foreign to most people used to C. Using the C function style parameter list will probably feel more intuitive to more people.


Although I've focused on methods the entire article, there's no reason why the same thing couldn't apply to properties.

For example:

[self addProperty: @"cost"] @function() {
	return 30.0;

NSLog(@"cost %f", self.cost); // prints 30.0

Of course, there's no reason why the property has to be backed by a method. It could be as simple as adding an entry to an internal NSMutableDictionary. In fact, if the caller doesn't provide a closure, then addProperty: could just add a dictionary entry:

[self addProperty: @"cost"] ;

And to initialize it with a default value:

[self addProperty: @"cost" withValue: [NSNumber numberWithFloat:30.0] ] ;


When I started writing this article, I was just thinking about the direction that Apple was going with Core Data. I didn't intend to end up talking about adding closures to Objective-C, but that's the logical progression. Despite all the rambling on my part, I didn't even touch other interesting features, such as built-in concurrency support. I suppose that's another post for another day.

Objective-C is unique in that it combines the low-level functionality of C with some really high level features, such as garbage collection, not commonly found in compiled languages. The high level features really improve the productivity of programmers, but they haven't been updated in a while (Objective-C 2.0 not withstanding). I'm not sure if Apple will implement these features, or if they do it will be in the way I implied, but I do hope that the language will continue to progress.

Dynamically adding transient attributes in Core Data

One of the NNTP commands I wanted to implement in Wombat was XPAT. XPAT is a simple search command that searches a specified header on a range of messages for a given wildmat expression. For example:

XPAT subject 1-12 *fred*

searches messages 1 through 12 for any message that has a subject containing the word “fred.” Matching a given header against a wildmat expression (i.e. the *fred* in the previous example), was easy because wildmat expressions are easily convertible into regular expressions. The problem was the user could specify any header to search against.

Although NNTP specifies a group of standard headers, posts are not limited to those standard headers. Any header, as long as they are syntactically correct, can be added to a message. What’s more, each message is going to have a different set of headers. The first message might have an X-NNTP-Posting-Agent header, while the second one might not.

Since headers vary from post to post, and could be almost innumerable, I couldn’t add them to my Post entity as persistent attributes. Instead, I have one attribute, headers, which is a string with each header on its own line.

This caused a problem because I wanted to use NSPredicate’s MATCHES operator to compare a specific header against a regular expression. I wanted to specify the predicate query as:

subject MATCHES ".*fred.*"

Unfortunately NSPredicate wants an attribute name to compare against (subject in this case), and my NSManagedObject didn’t have one. It had the headers attribute, but that contained all the headers, not just one. Since, once again, the headers can vary, I needed a way to dynamically add transient attributes to my NSManagedObject derived class, Post.

Since NSPredicate uses Key Value Coding to obtain the value of an attribute, I figured there was a solution. My first thought was to simply overload valueForKey: on my Post class. However, I was worried I’d muck up the Key Value Coding mechanism, and break something important. Fortunately, by digging through the Key Value Coding documentation I discovered the valueForUndefinedKey: message. According to the documentation, if a key wasn’t found via the normal means, the valueForUndefinedKey: message was called. This is important because that means if I overload valueForUndefinedKey:, I don’t run the risk of masking built-in, and possibly important, keys.

By default, valueForUndefinedKey: simply throws an exception stating no such attribute exists. This is valid behavior that I wanted to keep when overloading it. However, on the other hand, no matter what header the client asked for, I had to pretend that it was there and at least return an empty string. This was on the chance that some of the messages had the header. In other words, I needed a way to determine if the caller of valueForKey: was just asking for Joe Random attribute or a header attribute.

The only easy way I came up with was to prepend header attributes with a common prefix, like “header_”. So the NSPredicate query above becomes:

header_subject MATCHES ".*fred.*"

This makes the processing in valueForUndefinedKey: pretty easy. The method, in general, can be defined as:

- (id)valueForUndefinedKey:(NSString *)key
if ( [key hasPrefix:@"header_"] )
return [self headerForName:key];

return [super valueForUndefinedKey:key];

As you can see, determining valid header requests from invalid attribute requests is easy. I call the super’s implementation of valueForUndefinedKey: so that the exception is thrown, as expected, in the case of bad attribute requests. Also it means that if the parent class does something other than throw in the future, that it will continue to work.

I have to admit this was much easier to implement than I thought it would be. I also like the idea that I can easily access all my header values via NSPredicate, which is very useful.