raganwald
Friday, July 11, 2008
  I've seen things you people wouldn't believe
Pete Forde of Unspace emailed to say that the very first time he tried andand, he discovered a bug:

nil
  => nil
nil.present?
  => false
nil.andand.present?
  => true
Reg: Embarrassing.

Pete: No sir. Not embarrassing, because no one’s ever going to find out they’re down here. Because you’re going to spot them, and you’re going to air them out.

#present? is defined in Edge Rails as:

class Object
    def present?
        !blank?
    end
end
#blank? is also defined in Rails in Object, String, and NilClass. Hmmm. It didn’t take long to figure out what was happening. Internally, andand uses a BlankSlate class to implement proxies for returning nil or for forwarding a method.

module AndAnd

    class BlankSlate
        instance_methods.each { |m| undef_method m unless m =~ /^__/ }
        def initialize(me)
          @me = me
        end
    end

    class MockReturningMe < BlankSlate
        def method_missing(*args)
            @me
        end
    end

end
When you write nil.andand.present? What happens is this: nil.andand returns MockReturningMe.new(nil), and that object returns nil whenever you send it a method it doesn’t understand. Since it inherits from BlankSlate, we don’t expect it to understand #present?, so we expect it to return nil.

(Before you ask, nil.nil? returns true, but nil.andand.nil? returns nil. That’s intentional, because nil && nil.nil? returns nil, not true.)

So back to the bug. The BlankSlate wipes all of its instance methods out when it is first initialized. But what happens if the #present? method is added to Object after andand initializes its BlankSlate? Ta da! The MockReturningMe class does handle the method, so it never gets to use its method_missing magic.

And as it happens, a MockReturningMe object is not nil, therefore it is not blank, therefore it is present and #present? returns true.

If, that is, #present? is added to Object after AndAnd::BlankSlate is created. If #present? is added to Object before AndAnd::BlankSlate, you get entirely different behaviour. WTF‽

I’m stating facts here, not rendering judgement: This is another example of multiple metaprogramming whatsis doohickeys all gleefully re-plumbing the same core classes and stepping on each other’s toes. Even when they aren’t redefining the same methods.

Reg: It seems you feel our work is not a benefit to the public.

Avdi: Replicants are like any other machine. They’re either a benefit or a hazard. If they’re a benefit, it’s not my problem.

Update: Coderrr pointed out that the version of BlankSlate included in Rails (amongst other gems) fixes the problem by hooking the Object class. There are some subtleties in how this works or does not work, especially when several different pieces of code are all hooking the same events. After discussion with Nathan Weizenbaum, I decided that I certainly didn’t want andand being that invasive, but on the other hand if you have already decided you can live with a BlankSlate class…

So version 1.3.1 works as follows: If you already have a BlankSlate class defined, such as if you have installed Rails and used BlankSlate, or if you explicitly require BlankSlate before you require andand, andand will make use of the existing class and whatever mechanism that class uses to avoid this problem.

If you don’t have a BlankSlate class defined but you do have a BasicObject class (such as from Ruby Facets), andand will use that instead. And if you have neither a BlankSlate nor a BasicObject, andand will roll its own fairly uninvasive BlankSlate that wipes instance methods when it is instantiated (this is a performance hit compared to wiping them when they are created, however no hooks are required).

Pete is now happy. But this really encourages me to redouble my efforts on rewrite. Opening core classes to add certain kinds of functionality is very cool. But I believe it is unsustainable.

Methods like #present?, #andand, and even #nil? really don’t seem to work as polymorphic methods on objects. They need to be deeper. For example, you can define your own class where #nil? return true:

class Nullo
    def nil?
        true
    end
end
But guess what? This makes little sense in Ruby because your Nullo class is still truthy. (And no, you cannot write class Nullo < NilClass.)

I’ve been kvetching a little lately about Ruby not being turtles all the way down. I guess I’m doing that here as well: it is very hard to write things that extend or modify Ruby’s semantics consistently and safely, which i why I’m looking at rewriting.

With rewrite, for example, both #present? and #andand can be expressed as rewrite rules rather than as methods. (#blank? is a little bit more complicated, since you may want the luxury of writing a #blank? method for your own classes.) When you build semantic abstractions our of rewrite rules instead of methods in core classes, you benefit from restricted scope and you benefit from being able to work directly with Ruby’s existing semantics.
 

Comments on “I've seen things you people wouldn't believe:
Actually the way the BlankSlate class deals with new methods is by hooking method_added so it is notified of all new methods defined on Object and then wiping them from BlankSlate.
 
Coderrr:

Thanks for your suggestion, I was discussing that very option with Nathan W.

The very fact that we can debate whether we should erase methods as they are added or erase them when BlankSlate instances are created suggests to me that the concept of a BlankSlate inheriting from Object is deeply flawed.

However, of course, that is something hard-wired into Ruby and cannot be changed until Matz changes it.

I have heard that 1.9 might bring us a proper BlankSlate class in core, however the deeper problem is that Ruby lacks a Meta-Object Protocol, therefore when people want to do things outside of how the language is designed, torturous and fragile hacks are required.
 
Coderrr:

I think I now understand why you said “The way BlankSlate handles this is...”: You are referring to the implementation shipped with Rails. Of course, andand must function independently of rails and I am trying to minimize its impact on core classes, so I prefer to avoid adding hooks into Object and Kernel above and beyond the #andand method.

So I will experiment with using BlankSlate if it is already defined but usinga low-impact version if it is not present.

Thanks again for your comment.

By the way, I found this post you might find interesting: Bug in Facets’ BasicObject’s method_added. By strange coïncidence, it was also written by someone called “Coderrr.”
 
Reg help me out here because I have not played with ruby I am interpreting how some of the language structures work based on your writing.

Let me assume that changing what you have done to a re-write is better. Does re-write have a global "all across my application" implication or is a re-write limited in scope (meaning I might control what it rewrites in a given class but not in all classes)? If it rewrites all calls for certain things always wouldn't that potentially had some unintentional complexity in that if you are new to a particular code base you may not realize that something is being re-written for you when you make a certain type of call?
 
reg:

Yea sorry that was really unclear. I was referring to the BlankSlate implementations I've seen in the past: rails, facets, and hpricot. All of which use the method_added technique.

And yes, thanks to 1.9's BasicObject which comes for free, fewer crazy hacks will be needed.
 
Joe:

The rewrite I’m prototyping is limited in scope. Briefly, you write:

with(a, b, c) do
  …
end

Where a, b, and c are rewriters, and only the code between do and end is rewritten.

It might be possible to make global rewriting rules, especially in a Ruby implementation like Rubinius, but for now I am focusing on the restricted scope behaviour.

For example, if someone wants to use andand in a gem they ship to other people, the rewrite version of andand wouldn’t add an andand method to the Object class.
 
Along somewhat similar lines, I've discovered a couple interesting Ruby implementation details while playing around with andand this week.

Consider the following:

>> "foo" << "bar".andand
=> "foobar"
>> "foo" << nil.andand
TypeError: AndAnd::MockReturningMe#to_str should return String
from (irb):12:in `<<'
from (irb):12

and this one:

>> nil.andand.to_s
=> nil
>> nil.andand.to_str
=> nil
>> "#{nil.andand}"
=> "#<AndAnd::MockReturningMe:0x42a2174>"

I'm not saying these are bugs -- these probably fall into the "unsupported" category (it's definitely a strange way to use the utility). But it led me down an interesting path learning about the differences between to_s, to_str, and string interpolation in MRI.
 
Benjamin:

Terrific!

Yes, that is definitely unsupported at this time, in my head it feels like writing foo << bar &&.

I conjecture that in the cases where it appears to work, what is happening is this: Ruby does foo.andand and gets a mock object back. It isn't a String, so Ruby tries to cast it to a string by sending it the to_s or to_str method, which it forwards as usual.

In the case of String interpolation, Ruby does that directly in C, and it obviously does something without bothering to send the object a method.

(This is probably why string interpolation is much faster than catenating strings with RUby methods).
 
I have updated the post to reflect the release of andand 1.3.0. You know what to do:

sudo gem install andand
 
This kind of problem is a perfect demonstration of why duck-punching is a flat-out problem. And doubly a problem when you're using "nil" to represent a bunch of different things, instead of having a type system which allows you to return meaningful values.
 




<< Home
Reg Braithwaite


Recent Writing
Homoiconic Technical Writing / raganwald.posterous.com

Books
What I‘ve Learned From Failure / Kestrels, Quirky Birds, and Hopeless Egocentricity

Share
rewrite_rails / andand / unfold.rb / string_to_proc.rb / dsl_and_let.rb / comprehension.rb / lazy_lists.rb

Beauty
IS-STRICTLY-EQUIVALENT-TO-A / Spaghetti-Western Coding / Golf is a good program spoiled / Programming conventions as signals / Not all functions should be object methods

The Not So Big Software Design / Writing programs for people to read / Why Why Functional Programming Matters Matters / But Y would I want to do a thing like this?

Work
The single most important thing you must do to improve your programming career / The Naïve Approach to Hiring People / No Disrespect / Take control of your interview / Three tips for getting a job through a recruiter / My favourite interview question

Management
Exception Handling in Software Development / What if powerful languages and idioms only work for small teams? / Bricks / Which theory fits the evidence? / Still failing, still learning / What I’ve learned from failure

Notation
The unary ampersand in Ruby / (1..100).inject(&:+) / The challenge of teaching yourself a programming language / The significance of the meta-circular interpreter / Block-Structured Javascript / Haskell, Ruby and Infinity / Closures and Higher-Order Functions

Opinion
Why Apple is more expensive than Amazon / Why we are the biggest obstacles to our own growth / Is software the documentation of business process mistakes? / We have lost control of the apparatus / What I’ve Learned From Sales I, II, III

Whimsey
The Narcissism of Small Code Differences / Billy Martin’s Technique for Managing his Manager / Three stories about The Tao / Programming Language Stories / Why You Need a Degree to Work For BigCo

History
06/04 / 07/04 / 08/04 / 09/04 / 10/04 / 11/04 / 12/04 / 01/05 / 02/05 / 03/05 / 04/05 / 06/05 / 07/05 / 08/05 / 09/05 / 10/05 / 11/05 / 01/06 / 02/06 / 03/06 / 04/06 / 05/06 / 06/06 / 07/06 / 08/06 / 09/06 / 10/06 / 11/06 / 12/06 / 01/07 / 02/07 / 03/07 / 04/07 / 05/07 / 06/07 / 07/07 / 08/07 / 09/07 / 10/07 / 11/07 / 12/07 / 01/08 / 02/08 / 03/08 / 04/08 / 05/08 / 06/08 / 07/08 /