raganwald

A Brief History of Dangerous Ideas

noreply@blogger.com (Reginald Braithwaite) — Tue, 22 Jul 2008 14:04:00 +0000

(Brief because (a) I really don’t know that much history, and (b) there’s really only one idea here with a few examples, and once you get the point, we’re done and can move to the outro.)

Astronomy and Celestial Mechanics were dangerous ideas because they undermined the most powerful organization of their day, The Church. That’s why people have been banned, tortured, and burned at the stake for talking about these ideas.

Crossbows were a dangerous idea because they allowed an untrained peasant to kill a knight. Longbows were not a dangerous idea, because only trained archers can kill a knight with a longbow, and the nobility were the only people who could compel peasants to practise yeomanry.

Cryptography is not a dangerous idea, because we really don’t know if any of our algorithms and protocols are resistant to the NSA. This was hi-lighted when researchers “discovered” differential cryptanalysis. When they looked at the DES algorithm IBM has been promoting since the 1976, they found that it had been specifically tuned to resist differential cryptanalysis. IBM ‘fessed up: the US government had told them to tune things that way without explaining why, leading us to conclude they had known about this attack for decades before it became public knowledge. Today, we have no idea whether what we think is strong is actually strong or whether it has vulnerabilities and back doors governments can exploit.

Zed and Reg at Rubyfringe exchanging dangerous ideas, photo by Libin Pan
Miles Davis was a walking, talking, trumpet-playing dangerous idea. Not because he reinvented Jazz five different times (Dear Steve: Apple II, Macintosh, iPod/iPhone, Pixar. One more for the tie, two for the record.) Miles was an egocentric, venal man who worked the system, not undermined the system. But he was still dangerous because he got white people directly interested in black music. There was no Elvis or Vanilla Ice or anyone else between his music and the mainstream audience. For a government intent on keeping America’s two dominant cultures divided through fear, anything uniting them was a threat.

People say Miles’ legacy is his music. To me, Miles’ lasting legacy is people like me, people with one parent from each culture who grew up dancing to the same music together. People who, incidentally, do not vote for governments that take a divide and conquer approach to culture.

And on to tech. Microcomputers were not a dangerous idea. But personal computers were dangerous. It took decades for IT departments to regain control over people bringing their own computers to work. They can thank Microsoft for helping them get back into the driver’s seat.

(This, incidentally, is why I really dislike Microsoft’s policies: it has nothing to do with their lack of taste, it has to do with their mission to make the computer on my desk belong to my IT department or the record label or the movie studio or—I suspect—the government.)

Web applications are dangerous. Never mind the fact that they make desktop applications obsolete. The people who built desktop applications just go and get jobs writing web applications. Same people, different shit. But as Giles Bowkett pointed out, web applications just might make venture capital obsolete! When you don’t need hundreds of programmers and distribution channels and all the other friction-managing elements of a company that ships old-school software, you need a lot less money to start a business.

And on to media. You know that the web is busy putting newspapers out of business. My wife and I watched YouTube last Saturday Night. I’m not talking about the advertising business: I think we would have been happy to watch ads to watch our Mitch Hedberg and Billy Connoly comedy clips. But the web lets us choose what we want to watch, when we want to watch it. The network can’t put their up-and-coming show on right after their hit to give it a boost. The new show has to compete on its own merits. That puts users in control, and that’s dangerous.

Joel Spolsky said a similar thing about pricing all music at 99 cents a track: it means the labels can’t kill an artist by sticking their CD in the $3.99 crapola bin. Users choose what they want to listen to. That’s dangerous, again because users are in control.

Okay, that’s enough. Dangerous ideas are the ones that subvert the existing hierarchy of control, not just the ones that shuffle people around in the same old chairs. Apple Macintosh with a GUI replacing a PC with a command line? Not dangerous. Apple Macintosh with a Laserwriter and Aldus Pagemaker allowing someone to launch a magazine in their basement that competes with a company employing dozens of layout artists? That’s dangerous, and that’s interesting.

Dangerous equals subversive equals interesting.

outro

Deep breath. Okay, the next thing is not particularly dangerous for the world at large, but it is for me. I am retiring from blogging and retiring from hacking on Ruby. Maybe I’ll un-retire one day. I don’t know, the future is not set.

Miles Davis wasn’t afraid to move on when the time was right, even if what he was doing seemed to make people happy. Respecting his legacy means seeking what he sought.

Remember how I said that microcomputers were not dangerous, but personal computers were? Right now, I would say this: Ruby is not dangerous, but Rails is dangerous and Merb is dangerous and Sinatra is dangerous. Rewriting for Ruby is interesting. I believe it is useful. But is it dangerous? No.

Likewise, I can say with a clear conscience that while writing is gratifying, and trying to write out and explain ideas has helped me understand things, the writing I’ve been doing is not dangerous. It doesn’t subvert.

So while hacking away on Ruby the language and blogging about software development is gratifying and useful, they are not dangerous activities. They are microcomputers, but they are not personal computers.

I am going on vacation from August 2nd through 10th. During that time I plan to do absolutely no thinking about computers. For what it’s worth, I will be engaging in activities with faux danger: wreck diving and sport climbing. (p.s. these are not solitary pursuits: if you want to try some of the world’s greatest wreck diving and sport climbing, get in touch).

When I return, I will give things some serious thought and hopefully, discover a way I can help make our world a more dangerous place.

Thank you ever so much for your support and interest and feedback. Especial thanks to my fellow bloggers like Joel, Joey, Giles, Damien, Obie and so many others who exchanged ideas with me and kept the debate alive. And Reddit? And Hacker News? You rock, you are the future. I can’t wait to see how your communities and technologies evolve.

Warmest regards,

Reginald Braithwaite

That not is not the not I meant

noreply@blogger.com (Reginald Braithwaite) — Tue, 22 Jul 2008 09:00:00 +0000

During my presentation at RubyFringe, I put this slide up while talking about “Adverbs:”

Judging by the fun we had discussing this point later, I really needed a better example or a new idea. But for thse who are a little curious about where I was trying to go with this…

more on why not is not the same as not

I saw the #not method somewhere, and the point was that the author felt it read better than writing “!blitz.blank?” As someone pointed out, Ruby also gives us a not keyword, so you could write “not blitz.blank?” if you prefer the word “not” to the punctuation “!”

So the argument is that:


blitz.not.blank? == !blitz.blank? # and also == (not blitz.blank?)

But there’s another point we’re missing comparing the word to the punctuation. Let’s switch to English and you’ll see what I mean. The sentence “It is not the case that blitz is blank’ does not mean the same thing as “blitz has the property of not being blank.”

Whaaaa?

The key here is to think about where the parentheses go when we parse the two ways to express ourselves. Imagine “blank?” and “not” are functions instead of methods or keyword. Then two forms would look like this:


not(blank?)(blitz) # == blitz.not.blank?
not(blank?(blitz)) # == not blitz.blank?

Very different. The first expression passes the blank? function to the not function. The result is another function, the inverse of blank? itself, and that’s what we use. The second form is our old not keyword, where we pass blitz to out blank function, which presumably returns true or false, and we pass the result to our not function which inverses the boolean.

So in one case we are taking the inverse of a function, in the other we are taking the inverse of a boolean. This is what I was feeling out with my digression into wondering why Ruby reads from right to left. We have ways of applying functions to functions, but we don’t have easy ways of writing methods that alter other methods directly. And we certainly don’t have an easy way of sending a message that invokes such a method on an arbitrary object.

I guess I wonder whether we could ever write this blitz.not(blank?) to express this idea, or perhaps blitz.(not blank?), or perhaps some other syntax. But the point would be that we want the inverse of a method, not the inverse of a boolean.

speaking of jazz, so what?

It seems very abstract when we look at something like #not.blank?, because the result of an inverted #blank? method would usually be identical to the result of inverting its result.

But let’s consider the case where blitz is actually a proxy for a value in a SQL database column. So you might have something like “my_model.blitz.not.blank?” Fair enough?

Now if you know SQL, I think you will agree that there are times when the following is true:


  (model1.blitz != model2.blitz) != (not model1.blitz == model2.blitz)

Although it looks like x.not.eql?(y) should always have the same semantics as not x.eql?(y), SQL has special semantics for testing NULLs. Specifically, NULL is neither = nor <> any other value. So if you carefully map your proxies to the underlying SQL behaviour, you want the above to have this behaviour.

(For those who are blessed with a lack of SQL in their lives, NULL is a special value that is not the same thing as a Ruby nil. For example NULL = NULL is false in SQL. So if we map SQL values to Ruby objects, the following is the case: NULL != anything is false, NULL == anything is also false, therefore not NULL == anything is true, and therefore (NULL != anything) != (not NULL == anything).

If this digression into SQL NULLs now makes you regard your current ORM framework with deep suspicion, congratulations. They all suck, the trick to choosing an ORM is finding one that sucks in ways that aren’t fatal to your project.)

Essentially, Ruby already has a not adverb as a special case for #not==. #!= is there because sometimes (not foo == bar) is not equal to (foo != bar). However, like many things in Ruby this is a one-off thing. There isn’t a bang-method construct, #== and #!= are two completely arbitrary symbols that we happen to associate with equality and its inverse.

But maybe if we had a way to derive #!= from #== using a not adverb, maybe if we could write #== and then write a #not for the method, we would be able to do interesting things like replicate SQL’s trivalent logic more accurately. We would know that foo.not < 5 is not the same thing as not foo < 5, because if foo is a SQL null, foo < 5 is false, as is foo.not < 5. And neither throws an exception!

So getting back to the #not method as presented, I don’t know if it’s necessary, and if it does nothing more than replicated the exact semantics of the bang punctuation or not keyword it is clearly a matter of taste.

But I do feel that there is a use case for “adverbs,” for a way to talk about modifying methods themselves. Unfortunately, right now the mechanisms for working directly with methods are somewhat arcane and thus we don’t naturally consider them. So discussions like this feel very fringe and “out there.”

But at least now I feel like I have properly explained the not I meant, and not the not I didn’t mean.

"L" is not a code smell

noreply@blogger.com (Reginald Braithwaite) — Mon, 21 Jul 2008 16:43:00 +0000

During my presentation at RubyFringe, I shared a question that has been swirling around in my brain for a while: Are IDE features really language smells?

I don’t think it’s an original thought. If nothing else, it’s a corollary to what I believe to be true about many of the GoF design patterns: Many of them are workarounds, ways to Greenspun missing language features. Now, this is probably not the case for all IDE features, and in truth it may be that there are some features which could be implemented in either the language or the IDE, but the IDE may be the best place to put them.

But there is a fairly large class of IDE features that strike me as language workarounds. One of them is definitely the ability to spit out a lot of boilerplate. If you need a lot of code written, you ought to be able to get your programming language to do it for you, not your IDE.

There is room for people to disagree about this. There are some who feel (Strawman alert!) that programs consisting of large numbers of simple elements are easier to understand than programs consisting of a small number of highly abstract elements. Those folks feel an IDE gives you the ease of writing a program quickly plus the ease of reading that same program quickly. They feel that abstractions make the program easier to write but harder to read.

I happen to disagree with this, and if you have been reading this weblog for more than a couple of days you have already read why my experience leads me down a different path. Although in deference to my colleagues with different views, I offer this quote:

All problems can be solved by adding another layer of abstraction, except the problem of having too many layers of abstraction.

Anyone who has dealt with an hammer factory will agree.

So back to “L.”

Two speakers before me, Giles Bowkett gave his excellent Archaeopteryx—um—presentation. I hesitated over that word, because I could just as easily say performance. Performances are terrific entertainment, but they sometimes obscure the message behind them. I want to say outright that while this is true of many other subjects, I felt it worked for Giles because the subject of his presentation was software development as a how rather than a what, and for Giles the “what” is performance.

(Giles got a standing “O,” and many people might be tempted to rush out and make their presentations just as stimulating (400+ in-your-face slides punctuated with loud, driving drum and bass). Be sure that your material matches your presentation style! If not, people may walk away saying “Wow, amazing, but what exactly did she say?” I think it worked for Giles and that’s quite an accomplishment.)

Now really, back to “L.”

Giles is one of the people using closures in Ruby. Meaning, he is passing functions around and storing functions in objects. I am not going to try to say exactly what Archaeopteryx does, so I will describe this style of programming using an imaginary companion program that creates walking bass lines. I will call it Troody.

Let’s simplify things greatly and say that Troody will only ever play in perfect 4-4 time and further that Troody only ever play one of the eight notes in a particular chord’s standard scale. The probability of playing each of those notes on any one “beta” could be represented as an array with eight elements, like this: [.35, .05, .1, .05, .25, .05, .1, .05 ]. You can imagine passing arrays like this around in Troody.

For example, we can pass this array to an object that actually plucks the strings: Plucker.new.start_plucking([ 0.35, 0.05, 0.1, 0.05, 0.25, 0.05, 0.1, 0.05 ]).

Let’s try writing a naïve Troody Plucker:


class Plucker
    def start_plucking(probs)
        while (self.tune.playing)
            if (Metronone.on_the_beat)
                r = rand
                cumulative_probs = probs.inject([]) { |cum, element| 
                    cum + [ cum[-1] && (cum[-1] + element) || element ] 
                }
                notes_to_cumulative_probs = (1..8).zip(cumulative_probs)
                note_to_play = notes_to_cumulative_probs.detect { |note, prob| prob >= r }
                self.pluck(note_to_play)
            end
        end
    end
end

You pass it a set of probabilities, it produces bass notes. But stop, that’s so procedural. Let’s learn from a flying creature, let’s learn from Archaeopteryx. Instead of passing arrays, let’s pass lambdas, like this: lambda { [.35, .05, .1, .05, .25, .05, .1, .05 ] }. Now whenever Troody needs the probability of something, we call the function with .call or Ruby’s [] alternative syntax. So now we write Plucker.new.start_plucking(lambda { [.35, .05, .1, .05, .25, .05, .1, .05 ] })

Our new Plucker code is the same as the old, except we write:


cumulative_probs = probs.call.inject([]) { |cum, element| 
    cum + [ cum[-1] && (cum[-1] + element) || element ] 
}

We now call the probs lambda when we need a note. That’s it, we’ve added a .call call. What does that get us? Well, here’s one thing: If we want the probability to change over time, our function can do that, and we don’t have to rewrite our start_plucking method to handle the idea.

For example, here’s a probability lambda that usually plays the same way but from time to time decides it ought to play pedal notes (refactoring to OO is an optional exercise):


probs = lambda { |bars_of_pedal, beat|
    lambda {
        if beat == 0
            if bars_of_pedal == 0
                bars_of_pedal = 1 if rand < .05
            elsif bars_of_pedal == 5
                if rand < .25
                    bars_of_pedal = 0 
                else
                    bars_of_pedal += 1
                end
            elsif bars_of_pedal == 9
                if rand < .5
                    bars_of_pedal = 0 
                else
                    bars_of_pedal += 1
                end
            elsif bars_of_pedal == 13
                bars_of_pedal = 0
            end
        end
        beat = (beat + 1) % 4
        if bars_of_pedal == 0
            [ 0.35, 0.05, 0.1, 0.05, 0.25, 0.05, 0.1, 0.05 ]
        else
            [ 1.0, 0, 0, 0, 0, 0, 0, 0 ]
        end
    }

}.call(0, 0)

Thanks to the way we’ve separated the probabilities from the plucking, we do not need to subclass Plucker to try a different playing style in Troody.

As Giles pointed out, this is the Strategy Pattern. We are making different kinds of pluckers by encapsulating the logic of what to pluck in something we pass to a plucker. Archaeopteryx appears to do this everywhere. There are lambdas paramaterized by lambdas, lambdas that return lambdas…

This creates a problem. Imagine a programing language where all the keywords are in upper case: IF foo THEN bar ELSE bizzat. Try reading such a program aloud, and you end up shouting the punctuation but speaking the words. This is wrong! We should be shouting the words and whispering the punctuation!

And the problem with Ruby’s lambdas is that if you use a lot of them the word lambda really starts to stand out. So Giles fixed this by aliasing it to L: and using [] instead of .call():


alias :L :lambda

L{ |a| a + a}[5].
    => 10

Much nicer, and as Giles pointed out, this is an example of Ruby’s strength. If you have a program that rarely uses lambdas, you probably want lambdas to stand out when you use them, so you don’t alias lambdas to L and you use the call method, not the square brackets. But if you use lambdas a lot, it’s a win to abbreviate things.

Okay, we’re talking about “L.” Good.

Now in my talk, I said that abbreviating lambda to L was a code smell. I was wrong! Giles, my bad!!

What I actually think is that needing to abbreviate lambda to L is a language smell. Very different. If you show me a Java program and you show me Strategy Pattern, I shouldn’t say it’s a code smell. I should say too bad for you that you need all that boilerplate when Ruby lets you do that with the word lambda and a pair of curly braces.

So now to “L:” Giles, if lambdas are integral to Archaeopteryx, if they are so woven into the fabric of what Archaeopteryx does that you want the keyword “lambda” to fade away, I honestly think this is a place where the language could help you.

For example, what if Ruby had call-by-name semantics? You could write:


Plucker.new.start_plucking([ 0.35, 0.05, 0.1, 0.05, 0.25, 0.05, 0.1, 0.05 ])

# or...

bars_of_pedal, beat = 0, 0
Plucker.new.start_plucking(
    if beat == 0
        if bars_of_pedal == 0
            bars_of_pedal = 1 if rand < .05
        elsif bars_of_pedal == 5
            if rand < .25
                bars_of_pedal = 0 
            else
                bars_of_pedal += 1
            end
        elsif bars_of_pedal == 9
            if rand < .5
                bars_of_pedal = 0 
            else
                bars_of_pedal += 1
            end
        elsif bars_of_pedal == 13
            bars_of_pedal = 0
        end
    end
    beat = (beat + 1) % 4
    if bars_of_pedal == 0
        [ 0.35, 0.05, 0.1, 0.05, 0.25, 0.05, 0.1, 0.05 ]
    else
        [ 1.0, 0, 0, 0, 0, 0, 0, 0 ]
    end
)

And you would get the same behaviour as if you were using lambdas.

That’s it, that’s what I should have said on stage: any time you are working around your language—whether in your IDE, or by modifying open classes, or by abbreviating things—that’s a place where we should step back, where we should ask if our language is missing something.

The answer may very well be “no.” But we ought to at least ask the question.

Raganwald on the Fringe of a Nervous Breakdown

noreply@blogger.com (Reginald Braithwaite) — Fri, 18 Jul 2008 15:40:00 +0000

Thanks to Joel, I now have a format for Sunday’s presentation at RubyFringe: Twenty slides, each on the screen for exactly twenty seconds.

What does the word "Quality" mean?

noreply@blogger.com (Reginald Braithwaite) — Thu, 17 Jul 2008 20:13:00 +0000

The phrase “Quality Code” came up in a conversation with some colleagues. Just so we’re clear on what I mean when I use the word “Quality” with respect to code or architecture or software design:

Quality is the antonym of “defective,” with the addition of time. Meaning, it has few known defects in its current state, and we have high confidence that we will not discover defects in its current state over time, and additionally that we will not create further defects in it as we add to or alter its functionality in the normal course of maintaining this software.

That’s a lot of blather suggesting that quality in a code base is all about making code that works as expected. Some code is designed to be extremely flexible or malleable to make it easier to add functionality in the future. This is an admirable property of a code base, however I do not think of that as affecting a code base’s quality except for the fact that a more flexible code base does sometimes allow adding functionality with a lower probability of introducing new bugs or regressions.

I am not attempting to sum up all of the desirable properties of a code base. For example, in some business circumstances, increasing velocity of development at the expense of quality is a good trade-off, such as when putting together a demo that wins a large piece of business or when creating prototypes that users can use to hone a design.

But when I talk about code having very high quality I mean that it has few defects and that I expect it to continue to have few defects over time. Code that is extremely brittle and highly coupled is not of high quality because I expect that any attempt to make changes will introduce regressions.

Thus, the word “Quality” is not a subjective, aesthetic judgment. It is an objective measure of one axis of value to its owners.

Ruby.rewrite(Ruby) in Ten... Nine... Eight... Seven Days...

noreply@blogger.com (Reginald Braithwaite) — Mon, 14 Jul 2008 18:45:00 +0000

Checking the RubyFringe schedule, I see I must somehow keep the audience from yawning for thirty minutes between 2:10 and 2:40 on Sunday afternoon. I have the unenviable task of appearing between two incredibly smart and hardworking people, Damien Katz and Chris Wanstrath.

Upon reflection, I think I’d better open my talk with this quote from Horse Feathers:

I’ve got to stay here, but there’s no reason why you folks shouldn’t go out into the lobby until this thing to blows over.

—Groucho Marx

See you at RubyFringe!

My analyst warned me, but metaprogramming was so beautiful I got another analyst

noreply@blogger.com (Reginald Braithwaite) — Sun, 13 Jul 2008 13:45:00 +0000

Try to imagine a world where every programmer you know is a wannabe language designer, bent on molding the language to their whims. When I close my eyes and imagine it, I have a vision of the apocalypse, a perfect, pitch-black storm of utterly incomprehensible, pathologically difficult to debug code.

—Jeff Atwood

Whereas I have a vision of a world not too much different than the one we live in today. A few people invent ridiculous things, incomprehensible things. Most people carry on doing what they’ve always done. And a few people invent new, worthwhile things that move us forward.

In my world, a few people inventing great things more than makes up for a few other people inventing ridiculous things. If we get Rails out of the deal, isn’t it worth giving Ick a bemused smile, tousling the creator’s hair and encouraging him to “keep trying?”

Jeff goes on to put some smart people on a pedestal, prostrating before their greatness. I admire them as well, but I don’t confer a priesthood upon them. Once upon a time, reading and writing was not something plebeians did. Even the bible was a closed book, you only heard what was recited in Church.

Does anyone doubt that democratizing the written word—through education and especially Gutenberg’s invention—has been a force for good?

I imagine that if we traveled back in time we would find monks bewailing what would happen if anyone was allowed to write. People—the monks would stammer—people might write doggerel! Pornography!! Graffiti!!!

But of course, they will also write beautiful things. And useful things. Like theories of Astronomy that violate what the priests “knew” to be true. And that is the point: if you leave things up to the priesthood, all you get is stuff that is neatly aligned with what they already know. You cannot make progress by subjecting new ideas to scrutiny in a committee, even a smart and well-educated committee. Maybe especially not a smart and well-educated committee.

If it were up to educated people, do you think there would be personal computers today?

Language design in programming is the same thing. Yes, it creates an opportunity for misunderstanding. Yes, you can make an amazing amount of trouble for yourself and for others. But to put it under lock and key in the Church where only the priest are permitted to mention it in hushed tones… this is wrong.

The message is clear: You and I are worker ants. We do not think. We do not question our tools. We simply use them as we are directed by those in control. I find that far more horrifying than just about anything else I can imagine coming out of giving people more freedom.

Come on. Really. Let’s get ahold of ourselves. As a wonderful movie’s last line went:

“You have to have a little faith in people.”

I've seen things you people wouldn't believe

noreply@blogger.com (Reginald Braithwaite) — Sat, 12 Jul 2008 02:47:00 +0000

Pete Forde of Unspace emailed to say that the very first time he tried andand, he discovered a bug:


nil
  => nil
nil.present?
  => false
nil.andand.present?
  => true

Reg: Embarrassing.

Pete: No sir. Not embarrassing, because no one’s ever going to find out they’re down here. Because you’re going to spot them, and you’re going to air them out.

#present? is defined in Edge Rails as:


class Object
    def present?
        !blank?
    end
end

#blank? is also defined in Rails in Object, String, and NilClass. Hmmm. It didn’t take long to figure out what was happening. Internally, andand uses a BlankSlate class to implement proxies for returning nil or for forwarding a method.


module AndAnd

    class BlankSlate
        instance_methods.each { |m| undef_method m unless m =~ /^__/ }
        def initialize(me)
          @me = me
        end
    end

    class MockReturningMe < BlankSlate
        def method_missing(*args)
            @me
        end
    end

end

When you write nil.andand.present? What happens is this: nil.andand returns MockReturningMe.new(nil), and that object returns nil whenever you send it a method it doesn’t understand. Since it inherits from BlankSlate, we don’t expect it to understand #present?, so we expect it to return nil.

(Before you ask, nil.nil? returns true, but nil.andand.nil? returns nil. That’s intentional, because nil && nil.nil? returns nil, not true.)

So back to the bug. The BlankSlate wipes all of its instance methods out when it is first initialized. But what happens if the #present? method is added to Object after andand initializes its BlankSlate? Ta da! The MockReturningMe class does handle the method, so it never gets to use its method_missing magic.

And as it happens, a MockReturningMe object is not nil, therefore it is not blank, therefore it is present and #present? returns true.

If, that is, #present? is added to Object after AndAnd::BlankSlate is created. If #present? is added to Object before AndAnd::BlankSlate, you get entirely different behaviour. WTF‽

I’m stating facts here, not rendering judgement: This is another example of multiple metaprogramming whatsis doohickeys all gleefully re-plumbing the same core classes and stepping on each other’s toes. Even when they aren’t redefining the same methods.

Reg: It seems you feel our work is not a benefit to the public.

Avdi: Replicants are like any other machine. They’re either a benefit or a hazard. If they’re a benefit, it’s not my problem.

Update: Coderrr pointed out that the version of BlankSlate included in Rails (amongst other gems) fixes the problem by hooking the Object class. There are some subtleties in how this works or does not work, especially when several different pieces of code are all hooking the same events. After discussion with Nathan Weizenbaum, I decided that I certainly didn’t want andand being that invasive, but on the other hand if you have already decided you can live with a BlankSlate class…

So version 1.3.1 works as follows: If you already have a BlankSlate class defined, such as if you have installed Rails and used BlankSlate, or if you explicitly require BlankSlate before you require andand, andand will make use of the existing class and whatever mechanism that class uses to avoid this problem.

If you don’t have a BlankSlate class defined but you do have a BasicObject class (such as from Ruby Facets), andand will use that instead. And if you have neither a BlankSlate nor a BasicObject, andand will roll its own fairly uninvasive BlankSlate that wipes instance methods when it is instantiated (this is a performance hit compared to wiping them when they are created, however no hooks are required).

Pete is now happy. But this really encourages me to redouble my efforts on rewrite. Opening core classes to add certain kinds of functionality is very cool. But I believe it is unsustainable.

Methods like #present?, #andand, and even #nil? really don’t seem to work as polymorphic methods on objects. They need to be deeper. For example, you can define your own class where #nil? return true:


class Nullo
    def nil?
        true
    end
end

But guess what? This makes little sense in Ruby because your Nullo class is still truthy. (And no, you cannot write class Nullo < NilClass.)

I’ve been kvetching a little lately about Ruby not being turtles all the way down. I guess I’m doing that here as well: it is very hard to write things that extend or modify Ruby’s semantics consistently and safely, which i why I’m looking at rewriting.

With rewrite, for example, both #present? and #andand can be expressed as rewrite rules rather than as methods. (#blank? is a little bit more complicated, since you may want the luxury of writing a #blank? method for your own classes.) When you build semantic abstractions our of rewrite rules instead of methods in core classes, you benefit from restricted scope and you benefit from being able to work directly with Ruby’s existing semantics.

Off topic: Antagonyms

noreply@blogger.com (Reginald Braithwaite) — Tue, 08 Jul 2008 16:30:00 +0000

An antagonym (also called a contranym) is a single word that has meanings that contradict each other.

For example, “Reg Braithwaite buckled his belt and lifted, hoping he wouldn’t buckle under the strain.” The word “buckle” has two entirely opposite meanings.

Antagonyms aren’t nearly as important to the world as finding a way to make our avocation our vocation, but all the same the very idea brings a pleasant smile to my face, and I hope you’re amused as well.

(via Hacker News)

Separating the concern of "what to do" from "how to do it quickly"

noreply@blogger.com (Reginald Braithwaite) — Mon, 07 Jul 2008 11:45:00 +0000

Over the week-end, I put together a rudimentary rewrite-by-example feature for the rewrite gem (please be patient, I won’t be adding this feature to the gem until it has gone through a few more design iterations). The first thing I tried to do with it was simulating unhygienic macros:


Unhygienic.from {

    __to_receiver.andand.__to_message(__splat_parameters)

}.to {

    lambda { |andand_temp|
      andand_temp.__to_message(__splat_parameters) if andand_temp
    }.call(__to_receiver)

}

What this code produces is a sexp-rewriter that does a global search-and-replace. It looks for code like this:


Person.find(:first, ...).andand.friends(true)

And replaces it inline with:


lambda { |andand_temp|
    andand_temp.friends(true) if andand_temp
}.call(Person.find(:first, ...))

Declarative rewriting by example is a darn sight better than hand-written sexp manipulation:


def process_call(exp)
  exp.shift
  receiver_sexp = exp.first
  if matches_andand_invocation(receiver_sexp)
    exp.shift
    mono_parameter = Rewrite.gensym()
    s(:call, 
      s(:iter, 
        s(:fcall, :lambda), 
        s(:dasgn_curr, mono_parameter), 
        s(:if, 
          s(:call, s(:dvar, mono_parameter), :nil?), 
          s(:nil), 
          begin
            s(:call, 
              s(:dvar, mono_parameter), 
              *(exp.map { |inner| process_inner_expr inner })
            )
          ensure
            exp.clear
          end
        )
      ), 
      :call, 
      s(:array, 
        process_inner_expr(receiver_sexp[1])
      )
    )
  else
    begin
      s(:call,
        *(exp.map { |inner| process_inner_expr inner })
      )
    ensure
      exp.clear
    end
  end
end

But back to rewriting by example:


Unhygienic.from {

    __to_receiver.andand.__to_message(__splat_parameters)

}.to {

    lambda { |andand_temp|
      andand_temp.__to_message(__splat_parameters) if andand_temp
    }.call(__to_receiver)

}

As you can deduce, it is using some magic symbols. In the “from” part of the definition, __to_receiver and __to_message mean “Match something here and name the result receiver and message respectively,” while __splat_parameters means “Match a list of things here and name the result parameters.”

In the “to” part of the definitions, those magic symbols insert whatever was matched in the from. This is a crude approximation of how regular expressions capture things with () and insert them with $1..$n, only there is no way to capture any arbitrary sub-pattern. You can use any names you want, as long as they have the magic prefixes __to_ or __splat_. (Magic symbols are a blight upon all that is right and good with code, suggestions for a better way to express capturing by name gratefully solicited!)

Speaking of select, inject, and other higher-order functions, The Seasoned Schemer is devoted to the myriad uses of first class functions. This book is approachable and a delight to read, but the ideas are provocative and when you close the back cover you will be able to compose programs from functions in powerful new ways.

The example above is very much like a Lisp macro. In Lisp, everything is an sexp, so of course macro invocations are sexps just as function calls are sexps. In Ruby, some things look like function calls, some things look like method invocations. In this particular case, “Person.find(:first, …).andand.friends(true)” looks like a method invocation, but it actually behaves like a macro invocation.

Although it looks like a method, the rewriter version of #andand is not polymorphic. You can’t override #andand in any classes, just as you can’t override other syntactic constructions like “&&” or “!”. This bothers OO purists, however I am a member of the OO radical fringe who take purity to another level, a level where overriding functionality is not allowed to violate Liskov Equivalence.

(I personally do not care for the idea that something like #andand sometimes means one thing and sometimes means another, just as many people would completely freak out if !foo didn’t always mean “not foo.”)

beyond macros

Rewriting goes beyond adding new functions and verbs (I personally consider #andand to be an adverb). Basically, what we have here is an extremely weak version of XSLT for Ruby code. Match this, turn it into that. IMO, XSLT transformations is a much better analogy than macro expansion. Rewriters can match and replace fairly arbitrary expressions, not just implement things that look like function calls and method calls.

Consider this hypothetical example:


Unhygienic.from {

    __to_receiver.select { |__to_x| 
        __to_select_body 
    }.inject(__to_seed) { |__to_acc, __to_y| 
        __to_inject_body
    }

}.to {

    __to_receiver.inject(__to_seed) { |__to_acc, __to_x|
        if __to_select_body
            __to_y = __to_x
            __to_inject_body
        else
            __to_acc
        end
    }

}

This would transformation code like this:


heads_of_state = locations.select { |any_loc| 
    zips_in_this_state.include? any_loc.zip_code
}.inject(0) { |heads, loc_in_state|
    heads + loc_in_state.head_count
}

Into this:


heads_of_state = locations.inject({}) { |heads, any_loc|
    if zips_in_this_state.include? any_loc.zip_code
        loc_in_state = any_loc
        heads + loc_in_state.head_count
    else
        heads
    end
}

This is an example of an optimization.

For a large class of expressions chaining a select and an inject (the ones that don’t rely on side effects), this transformation retains the original semantics while rewriting the code to only traverse the collection once and to eliminate the creation of an intermediate collection.

Of course, compilers do this kind of thing all the time for many types of optimizations, so it’s tempting to wait for someone else to write a sufficiently smart compiler that can figure these things out. There are two troubles with waiting for someone else to do it. First, we might be that someone—maybe we’re the one who ought to Just Do It, and waiting for someone else won’t work because nobody else is going to do it.

Second, many problems like this are intractable in the general case. It’s hard (in the mathematical sense) to know when select {…. }.inject { … } can be transformed like this in an imperative language without accidentally stepping on some hidden side-effect. But just because it’s hard in the general case doesn’t mean it isn’t easy in the specific case. For example, you might be the sort of person who never knowingly relies on side effects in select and inject expressions.

So you could use this optimization, while a compiler-level optimization would be a disaster: even if 99.9% of the code out there wouldn’t break, the programmers behind the 0.1% of the broken programs would be furiously blogging about how Ruby wasn’t ready for the Enterprise.

optimizing your code

The joy and the pain of optimizing your code is that you don’t need rewrite to perform that optimization. The joy is that if you discover chaining select and inject is a performance hog somewhere in your code, you simply rewrite the code yourself.

The pain is that the code is no longer in the form you originally decided best represents its intent. In the trivial example I gave above, a rewritten version looks reasonable, especially if you rewrite it with #each in an imperative style:


heads_of_state = 0
locations.each { |any_loc|
    if zips_in_this_state.include? any_loc.zip_code
        heads_of_state += any_loc.head_count
    end
}

Now I’m not going to say that this is necessarily more or less readable than:


heads_of_state = locations.select { |any_loc| 
    zips_in_this_state.include? any_loc.zip_code
}.inject(0) { |heads, loc_in_state|
    heads + loc_in_state.head_count
}

Some people actively dislike using #select and #inject, so they might feel the #each version is better. For once, let’s talk about something other than a bike shed. Let’s focus on the fact that foo.select {…}.inject {…} says one thing: “Filter this collection using this predicate, and then fold the result as follows.” Whereas foo.each {…} says “Iterate over this collection doing the following thing with each element.”

If you wrote this as a #select/#inject pair, you might have a good reason for doing so. Perhaps most of your program is written in a functional style. Perhaps you like to signal that the there are no side effects in those snippets of code and your team share this understanding of how selects and injects are written.

Granting that you believe that #select and #inject do a better job of communicating the code’s intent to your fellow team members, it’s a win to optimize the code (in the compiler or using a rewriter) behind the scenes rather than rewrite it yourself. The code retains its semantics and the form you have decided best expresses its intent, while using less memory and running faster.

separation of concerns

What we have just done with our trivial example is separated two concerns: The concerns of how to best express an algorithm and the concern of how to best implement an algorithm. If for whatever reason—furious hand-waving to avoid arguing how to write loops—we believe that the best way to express a certain algorithm for readability is not the best way to express a certain algorithm for performance, we have two separate concerns: How to write readable code and how to write ~~performant~~ fast code.

Doesn’t it make sense to separate those concerns? So that the code explaining what the algorithm is supposed to do is in one place and the code expressing how to make such things go fast is in another?

This is pure speculation here, but I am conjecturing that being able to rewrite arbitrary snippets of code could be used like a compiler optimization directive. When debugging, you don’t rewrite the code. But when things have stabilized and you need to tweak performance, instead of rewriting the code, you use rewriters to do it for you, separating the concern of “what to do” from the concern of “how to do it quickly.”

Sneak Peek: Unhygienic Rewriting Ruby by Example

noreply@blogger.com (Reginald Braithwaite) — Sun, 06 Jul 2008 01:59:00 +0000


def test_unhygienic_andand
  andand = Unhygienic.from {
    __to_receiver.andand.__to_message(__splat_parameters)
  }.to {
    lambda { |andand_temp|
      andand_temp.__to_message(__splat_parameters) if andand_temp
    }.call(__to_receiver)
  }
  assert_equal(
    'Hello' + ' World', 
    with(andand) do
        'Hello'.andand + ' World'
    end
  )
  assert_nil(
    with(andand) do
        nil.andand + ' World'
    end
  )
end

Let’s take a closer look:


Rewrite::With.expand(andand) do
  class Funky
    def hello_world
      'Hello'.andand + ' World'
    end
  end
end

produces:


class Funky
  def hello_world
    lambda { |andand_temp| 
        (andand_temp + " World") if andand_temp 
    }.call("Hello")
  end
end

As you can probably deduce, it’s unhygienic: Variables like andand_temp shadow variables declared elsewhere. This is a problem if you fail to choose a sufficiently obfuscated name or—more likely—nest constructions like #andand.

Both the Classically Metaprogrammed and the Sexp-Rewriting versions of #andand avoid this problem.

Next step: Better hygiene.

Process Theatre

noreply@blogger.com (Reginald Braithwaite) — Wed, 02 Jul 2008 20:00:00 +0000

You know how Bruce Schneier uses the term Security Theatre to describe measures designed to make us feel safer but not actually safer? I am going to start using the term Process Theatre. I trust you can grasp the meaning immediately.

A blatant exercise in pimping an affiliate link to Art of the Metaobject Protocol

noreply@blogger.com (Reginald Braithwaite) — Tue, 01 Jul 2008 23:42:00 +0000

The Art of the Metaobject Protocol is the best book written in computing in ten years.

—Alan Kay, as quoted in Report on OOPSLA97

The bottom line: Languages like Java give you an object protocol: There is one way to do things with objects and classes and interfaces, period. Anything else, like adding generics or annotations, must be done outside of the language, it must be done by the creators of the language.

Languages like Common Lisp and Smalltalk give you a meta object protocol: You can decide for yourself how to do things with objects, classes, interfaces, generic functions, whatever you want. You don’t need to wait for a committee to try something different.

People who believe that software is best created in a Soviet-era Bureaucracy celebrating process and mediocrity—all the while spouting propaganda about celebrating the common worker and the primacy of the proletariat, of course—fear and loathe this idea. They will tell you that standardization trumps innovation, and that the only thing worse than metaprogramming within a standard object system is metaprogramming the system itself.

But we saw how valuing process over people played out, and our minds are actually open to new ideas, even when those ideas are forty years old.

My Mixed Feelings about Ruby

noreply@blogger.com (Reginald Braithwaite) — Mon, 30 Jun 2008 14:42:00 +0000

There’s a lot of features of Ruby that I like, but there are some that just drive me nuts like blocks not taking blocks and the ampersand operator. Raganwald did a great job of explaining blocks, procs, and the ampersand in this blog post: Explanation of Ruby’s Unary operator. I came away with the feeling, “Wow! It took that much explanation just to tell how to send blocks around?” If blocks were first-class citizens, Ruby would be more elegant. Raganwald would not be writing huge blog posts on block vs. proc because it would be unneeded.

—Blaine Buxton, Too Complex?

I feel for Blaine, I really do. When I first looked at Ruby, quite honestly I hated it. Not hated it in a Blubb-dacious “It’s not the same as my popular language” way, but hated it in a “The language is full of Accidental Complexity” way.

To my eyes, Ruby as a language looked a lot like an internal IT app that is built as an aggregation of features. There might be a wonderful, coherent design in the implementation that I can’t see, but the interface I use seems like a bunch on one-off features that don’t play well together.

Ruby definitely isn’t turtles all the way down. And I have never stopped being troubled by this. I lamented the fact that Ruby isn’t written in Ruby. Twice. No, make that three times.

Deep breath. But.

There’s a part of my personality that craves purity and regularity in a system. Purity and regularity have practical benefits, to be sure, but at the same time there are practical benefits to being messy and having special cases optimizing for things people do frequently. This is why design is freakin’ hard. It’s not as easy as following a bunch of rules that produce a system with the smallest possible intellectual surface area, otherwise we’d have stopped with Scheme and its five axiomatic special forms.

A Rube Goldberg contraption does a simple thing (like making ramen) in an extremely complex way. It turns accidental complexity and irregularity into entertainment.
Nor is it as easy as piling more features on regardless of how well they fit or whether people will actually use them. Otherwise Windows would have 97% of the market and OS X 3%. (Oh wait.)

When it comes to design, sometimes you have to experience the result to judge it. You can decide whether a chair is attractive by looking at it, but you really need to sit in it a while to know whether you will feel comfortable using it. Even then, only long experience will tell you whether it is a keeper. Many things that are nice in the showroom are mediocre in day-to-day use.

Design is all about problem solving. What problem do you think programming languages solve as a group? What problem does Lisp solve? Smalltalk? Ruby? For whom?

Matz has said that Ruby is an attempt to solve the problem of making programmers happy. So maybe we aren’t happy with some of the accidental complexity. But can we be happy overall? Can we find a way to program in harmony with Ruby rather than trying to Greenspun it into Lisp?

(I speak as a man working on rewriting code for Ruby: By far the hardest part of this is trying to provide the power of macros in a mechanism that works in harmony with the Ruby Way, rather than bolting Lisp onto Ruby’s side.)

Programming languages also solve the deep problem of helping programmers think about the parts of a program that matter and not clutter their minds up with the parts that don’t matter. This is a very hard problem. Very, very hard. Make things too simple, have too few axioms and abstractions, and you end up with something where each element is extremely simple to understand, but any non-trivial program has too many elements with difficult-to-understand interactions and dependencies so as a whole programs are harder to understand.

Does a regular language help us understand the parts of programs that matter more than an irregular language? That is not an easy question to answer. When I’m struggling with some subtle difference between Proc.new and lambda, I want to shout YES, GODDAMIT, GIVE ME SOME CONSISTENCY.

I’m wary of trying to decide about languages based on infrequent edge cases. With a certain very popular language, I’ve made up my mind that the design choices don’t pay off, that the places where you work hard to express yourself aren’t places where the easy, obvious thing is easy to write and obvious to read. But so far in Ruby, the things that trouble me do seem to have some inherent trade-off merit. I can see how making blocks a special case makes certain eay, simple things easy. So is it overall the best possible design? I don’t know. is it a good design? So far it seems to be a reasonable design.

At my core, I believe axiomatically that there is no one “best” language. That is not an excuse for saying that every language has merit. While I think that there can be more than one good language with different approaches and styles, I do not exclude the possibility that 90% of the programming languages in existence are stinkers through and through.

Is Ruby better for me personally than Lisp or Smalltalk? I don’t know the answer to that question either.

I decided a while back that I would give Ruby an honest try. And while I give myself the freedom to express my misgivings about some of the choices Matz has made, I also try to keep an open mind about them.

It’s really freakin’ (I know, I used this word already) hard for me to to do, but I’m trying to find out if worse might be better. And I don’t mean, “Ruby is worse than Lisp, but it’s better for those n00bs over there.” That’s not embracing change. I mean I am trying to discover if worse might be better for me personally, and it is hard for me to open my mind to that possibility when I’m already invested in “better.”

At this moment in time I have extremely mixed feelings about Ruby. I sorely miss the elegance and purity of languages like Scheme and Smalltalk. But at the same time, I am trying to keep my mind open to some of the ways in which Ruby is a great programming language.

The unary ampersand in Ruby

noreply@blogger.com (Reginald Braithwaite) — Fri, 27 Jun 2008 20:07:00 +0000

A marshmallow toasting dragon slayer from Montréal asked about the unary &.

I like this question, because when I saw it, I realized immediately that although I’ve figured out how to use it to accomplish certain things, writing an answer serves the excellent purpose of forcing myself to learn more.

Let’s start with what I do know, and what everyone can figure out from grepping source code: It has something to do with converting things to and from blocks. If you take nothing else away from this, remember that when you see a unary “&” in Ruby, you are looking at making something into a block, or making a block into something.

Now, blocks in Ruby are not first-class entities. You cannot store them in variables. They are not objects. There is no Block.new expression. The only way to make a block in Ruby syntax is to write one as part of a method call, like this:


[1, 2, 3, 4, 5].map { |x| x * x  }
    => [1, 4, 9, 16, 25]

As we said above, you cannot assign a block to a variable:


square_block = { |x| x * x  }
    => syntax error, unexpected '}', expecting $end

What do we do with these blocks? Well, inside of a method, we can yield to a block. Yielding to a block is something very much like calling a function. The value of an expression yielding to a block is the value of the expression in the block, paramaterized by anything you pass in the yield expression.

Oh heck, an example is much better:


def take_a_guess(range) # expects a block testing a guess
    guess = range.begin + rand(range.end - range.begin)
    if yield(guess)
        "Yay, I guessed correctly"
    else
        "Boo hoo, my guess was wrong"
    end
end

take_a_guess(1..10) { |x| x == 6 }
    => "Boo hoo, my guess was wrong"
take_a_guess(1..10) { |x| x == 6 }
    => "Boo hoo, my guess was wrong"
take_a_guess(1..10) { |x| x == 6 }
    => "Yay, I guessed correctly"

This method plays a guessing game with you: it takes a range (“guess a number from one to ten”) and a block for testing whether the guess was correct. It takes a guess and yield the guess to the block. It then exclaims its joy to the world if it guesses correctly.

Notice that there is nothing in the method signature saying it expects a block. There is no name for the block. You have to look for the yield keyword to figure out what is going on if the programmer doesn’t add a comment.

converting blocks to procs

So… Let’s talk conversions. If you want to do anything besides invoke a block with yield, you really want a Proc, not a block. For example:


class Guesser
    attr_reader :range, :tester
    def initialize(range, &tester)
        @range, @tester = range, tester
    end
    def take_a_guess
        guess = range.begin + rand(range.end - range.begin)
        if tester.call(guess)
            "Yay, I guessed #{guess} correctly"
        else
            "Boo hoo, my guess of #{guess} was wrong"
        end
    end
end

foo = Guesser.new(1..10) { |n| n == 6 }
foo.take_a_guess
    => "Boo hoo, my guess of 2 was wrong"
foo.take_a_guess
    => "Boo hoo, my guess of 8 was wrong"
foo.take_a_guess
    => "Yay, I guessed 6 correctly"

We want to store the tester block as an instance variable. So what we do is add a parameter at the very end with an ampersand, and what Ruby does is take the block and convert it to a Proc, which you can pass around as an object. And when you want to use it, you send it the #call method.

Now if you think about this for a second or two, you’ll realize that almost every Proc you ever create works this way: We pass a block to a method, and the method turns it into a proc by having a parameter with an &. Let’s try writing another one:


def our_proc(&proc)
    proc
end

double = our_proc { |n| n * 2}
double.call(7)
    => 14

Not much to it, is there? When you want a Proc, you can create one by calling a method with a block and using &parameter to convert the block to a Proc. There is no other way to convert a block to a proc because the only place blocks exist is in method invocations.

converting procs to blocks

Okay, we know how to make a Proc out of a block. What about going the other way? What if we want to make a block out of a Proc?

Let’s reopen our Guesser class:


class Guesser
    def three_guesses
        guesses = Array.new(3) { range.begin + rand(range.end - range.begin) }
        if guesses.any?(&tester)
            "Yay, #{guesses.join(', ')} includes a correct guess"
        else
            "Boo hoo, my guesses of #{guesses.join(', ')} were wrong"
        end
    end
end

bar = Guesser.new(1..10) { |x| x == 3 }
bar.three_guesses
    => "Yay, 5, 9, 3 includes a correct guess"

What just happened?

For starters, we made an array with three guesses in it. That line of code includes a block, but let’s ignore that as being irrelevant to this particular discussion. The next part is what we’re after:

We then want to call Enumerable#any? to ask the array if any of its members are the correct guess. Now #any? expects a block. But we don’t have a block, we have a Proc. So now we do the reverse of what we did when we wanted to convert a block to a Proc: instead of a method having an extra parameter with an ampersand, we pass a parameter to the method and apply the ampersand to the parameter we are passing.

So “&tester” says to Ruby: “Take this object and pass it to a method as a block.” The #any? method gets a block, it has no idea we are doing any Proc to block conversion shenanigans. We can prove that:


def did_you_pass_me_a_block?
    if block_given?
        yield
    else
        "NO BLOCK"
    end
end

did_you_pass_me_a_block?
    => "NO BLOCK"
did_you_pass_me_a_block? { 'I passed you a block' }
    => "I passed you a block"
proc = Proc.new { 'I passed you a proc' }
did_you_pass_me_a_block?(&proc)
    => "I passed you a proc"

As you can see, our methods don’t really know whether they get a block or a proc passed as a block. They just yield and all is well. (And yes, you can convert a block to a proc and then the method can convert it right back into another proc.)

to_proc shakur

Which leads us to the final piece of the puzzle. How does Ruby convert whatever you pass with “&” into a block? The answer is that if it is not already a Proc, it tries to convert the object to a Proc by calling the object’s #to_proc method, and from there it converts the Proc into a block.

So you can do fun stuff like convert strings into blocks by defining a method that converts a string to a Proc:


(1..5).map &'*2'
    => [2, 4, 6, 8, 10]

Note that this conversion only happens when you try to pass a string as a block to a method with “&.” It is not correct to say that “&” converts an object to a Proc, the #to_proc method does that. It is correct to say that when passing an object to a method, the “&” unary operator tries to convert the object to a block, using the Object’s #to_proc method if need be.

We’ll close with an example, where we decide that a Guesser can be converted to a Proc:


class Guesser
    def to_proc
        Proc.new { |guess|
            "Yay, I guessed #{guess} correctly" if tester.call(guess)
        }
    end
end

foo = Guesser.new(1..10) { |n| n == 8 }
&foo
    => syntax error, unexpected tAMPER
foo.to_proc
    => #<Proc:0x0008cc08@-:26>
(1..100).map(foo).compact.first
    => ArgumentError: wrong number of arguments (1 for 0)
(1..100).map(&foo).compact.first
    => "Yay, I guessed 8 correctly"

So that’s it: When you want to convert a block to a Proc, define an extra parameter for your method and preface it with an ampersand. When the method is called, the parameter will hold a proc.

When you want to convert any object to a block, pass it to a method expecting a block and preface it with an ampersand. Ruby will call the #to_proc method if need be, then convert the Proc into a block.

because it’s friday

I really hate writing anything too serious on a Friday. So if you’re eyes have glazed over and you’ve marked this to be read later, here’s a little diversion for you. I just picked up The Magic Garden of George B. And Other Logic Puzzles by my favourite nonfiction author, Raymond Smullyan.

Right off the bat, the first paragraphs of the preface introduce a neat puzzle:

Here is a remarkable problem: Imagine a garden of magic flowers that can change color from day to day. On any one day, a flower is either blue the entire day or red the entire day, but can change from one day to another. Given any flower A and any flower B, there is a flower C that is red on all and only those days on which A and B are both blue. Also, we are given that for any distinct flowers A and B, there is at least one day on which A and B are of different colors. Now suppose that the number of flowers is somewhere between 200 and 500. How many flowers are in the garden?Amazingly enough, the problem actually has a unique solution! Doesn’t this surprise you?

Have fun!

A bit of history and an interesting conjecture

noreply@blogger.com (Reginald Braithwaite) — Wed, 25 Jun 2008 02:05:00 +0000

We Smalltalkers used to think the advantages of our language were so significant that it would take over the world. We had a huge productivity advantage over C coders. Then C++ came along and gave C coders just enough to let them improve their productivity and their ability to write larger more complex systems. It still wasn’t as good as Smalltalk, but it was better than C, and much more accessible to most programmers than Smalltalk.

C++ eventually sucked up all the oxygen and Smalltalk is now only a language for hobbyists and the occasional programming god. I think this is the most likely threat to the Rails surplus, that C# or Scala or something can do a good enough job that people can double their productivity with far less of a change in mindset or tools, and eventually no one will care about the ten times (or whatever) productivity of Rails.

“Good enough is good enough.”

—Josh Susser

Interview questions I have never been asked, Episode I

noreply@blogger.com (Reginald Braithwaite) — Tue, 24 Jun 2008 14:45:00 +0000

Compare and contrast:


some_array.any? { |n| n > 100 }

And:


!!some_array.detect { |n| n > 100 }

Which do you think is easier to read?

Are classes and modules easier to read and understand when they have lots and lots of specific methods like #any? or are they easier to read and understand with a small number of axiomatic methods that can be composed and combined like #fold and #unfold? When you design a module or class, do you write lots of convenience methods in advance? Or do you refactor code by writing convenience methods when you find yourself repeating the same code?

The Ruby Way is the perfect second Ruby book for serious programmers. The Ruby Way contains more than four hundred examples explaining how to do everything from distribute Ruby with Rinda to functional programming techniques.

If you do refactor code to eliminate duplication, is there an amount of duplication that is too small to matter, like "!!"? Or is there an underlying principle of documenting intent that you wish to make explicit?

Is:


do_something() if some_array.detect { |n| n > 100 }

Misleading because it doesn’t actually use the element it detects? Or is it a reasonable idiom to test for an element’s existence without using a specific method like #any or #nil?

Are applications easier to read and understand when they make use of lots and lots of specific methods like #any or are they easier to read and understand when they compose and combine a smaller number of axiomatic methods so that you aren’t constantly looking things up?

Do you think applications should have large or small vocabularies?

This example can be easily translated to the language du jour, the underlying principle applies to programming in general

Macros, Hygiene, and Call By Name in Ruby

noreply@blogger.com (Reginald Braithwaite) — Mon, 23 Jun 2008 01:25:00 +0000

Never send a macro to do a function’s job.

Sound advice, however just because functions (or methods) are better than macros for the things they both can do, that doesn’t mean functions can do everything macros can do. Let’s look at andand for a moment. When you write:


foo().andand.bar(blitz())

Using the andand gem, Ruby treats this something like:


temp1 = foo()
temp2 = temp1.andand
temp3 = blitz()
temp2.bar(temp3)

As it happens, if you call nil.andand.bar(blitz()), it will return nil. But it will still evaluate blitz() before returning nil. What I would expect from something named andand is that if foo() is nil, Ruby will never evaluate blitz(). Something like:


temp1 = foo()
if temp1.nil?
    nil
else
    temp2 = blitz()
    temp1.bar(temp2)
end

What we want is that when we pass blitz() to andand, it is not evaluated unless the andand function uses it. The trouble is, you cannot write an andand method in Ruby that delivers these semantics.

Let’s hand wave over the difference between methods and functions for a moment and just look at calling functions. We’ll consider writing “our_and,” a function that emulates the short-circuit evaluation behaviour of Ruby’s “&&” and “and” operators. Ruby (and most other languages in use these days) uses call-by-value when it passes parameters to functions. In other words, when you write:


our_and(foo(), blitz())

Ruby turns that into something that looks like this:


var temp1 = foo()
var temp2 = blitz()
our_and(temp1, temp2)

It doesn’t matter if the function our_and uses blitz() internally or not, it is evaluated before our_and is called and its value is passed to our_and. Whereas our “if” statement in the previous example does not evaluate “blitz()” unless “foo().andand” is not nil.

Well, well, well. The inescapable conclusion is that there are some sequences of expressions in Ruby that cannot be represented as functions or methods. That’s right, functions and methods can’t do everything that Ruby code can do.

Macros and code rewriting can do an awful lot. The implementation of andand in the rewrite gem does rewrite code. When you write:


with(andand) do
    # ...
    foo().andand.bar(blitz())
    # ...
end

Rewrite rewrites your code in place to look something like:


# ...
lambda do |__121414053598468__|
    if __121414053598468__.nil?
        nil
    else
        __121414053598468__.bar(blitz())
    end
end.call(foo)
#...

And for that reason when you write foo().andand.bar(blitz()) using the rewrite gem instead of the andand gem, blitz() is not evaluated if foo() is nil. Big difference!! So it looks like one way to get around call-by-value is to rewrite your Ruby code. Excellent. Or is it?

What’s wrong with rewrite

Right now, the rewrite gem supports writing sexp processors. These are objects that encapsulate a way of transforming sexps. For example, here is the code that transforms expressions like “foo().andand.bar(blitz()):”


  def process_call(exp)
    exp.shift
    receiver_sexp = exp.first
    if matches_andand_invocation(receiver_sexp)
      exp.shift
      mono_parameter = Rewrite.gensym()
      s(:call, 
        s(:iter, 
          s(:fcall, :lambda), 
          s(:dasgn_curr, mono_parameter), 
          s(:if, 
            s(:call, s(:dvar, mono_parameter), :nil?), 
            s(:nil), 
            begin
              s(:call, 
                s(:dvar, mono_parameter), 
                *(exp.map { |inner| process_inner_expr inner })
              )
            ensure
              exp.clear
            end
          )
        ), 
        :call, 
        s(:array, 
          process_inner_expr(receiver_sexp[1])
        )
      )
    else
      begin
        s(:call,
          *(exp.map { |inner| process_inner_expr inner })
        )
      ensure
        exp.clear
      end
    end
  end

And that’s just a third of andand: There is another method that handles expressions like “foo().andand { |x| x.bar(blitz()) }” and a third that handles “foo().andand(&bar_proc).” Brutal.

Now, rewriting code has many other uses. One on my wish list is a rewriter that transforms expressions like: “foo.select { |x| … }.map { |y| … }.inject { |z| … }” into one big inject as an optimization. So I’m not ready to throw rewrite in the trash can just yet. But there’s no way I want to be writing all that out by hand every time I want to implement a function but work around call-by-value semantics.

What about macros?

Why can’t I write:


def_macro our_and(x,y)
    ((temp = x) ? (y) : (temp))
end

…And have it automatically expand my code such that when I write:


# ...
foo = our_and(bar(), blitz())
# ...

The macro expander rewrites it as:


# ...
foo = ((temp = bar() ? blitz() : temp)
# ...

Wouldn’t that work? Maybe. Then again, maybe not.

The problem given above—working around call-by-value—is just one small problem. A macro implementation would solve that problem, but there’s an awful lot of overhead required to make the implementation work, and whatever you do ends up being an incredibly leaky abstraction.

Take our example above. What happens if we have our own variable named temp? Does it get clobbered by expanding our_and? Or do we rename temp? Or do some automagic jigger-pokery with scopes?

Getting macros right is very tricky. I don’t personally plan to try my hand at implementing macros until I’m an expert on the subject of variable capture and can hold forth on the design trade-offs inherent in different schemes for implementing hygienic macros. But that’s just me.

Perhaps there are other ways to solve it without diving into a full-blown macro facility?

Lambdas and blocks

Indeed there are other ways. Ruby already has one-and-a-half of them: blocks and lambdas. Using blocks and lambdas, you can control evaluation precisely. The andand gem actually does support short-circuit semantics using a block. When you write:


nil.andand { |x| x.foo(blitz()) }

It does not evaluate blitz(). This alternate way of using andand supports the semantics we want by explicitly placing the code that should not be eagerly evaluated in a block. Given patience and a taste for squiggly braces, you can create non-standard evaluation without resorting to macros.

We said at the beginning that the reason we cannot use functions and methods to represent everything we can write in code is because Ruby uses call-by-value to pass parameters to functions. One way to work around that is this: instead of passing the value of each expression to a function, we can pass the expression itself, wrapped up in its own lambda.

Then, when the function needs the value, it can call the lambda. This technique has a name: it is called thunking.

We could implement our_and as follows:


our_and = lambda { |x,y|
    if temp = x.call
        y.call
    else
        temp
    end
end

Then when we call it, we could wrap our parameters in lambdas:


our_and.call(
    lambda { a() },
    lambda { b() }
)

Verify for yourself that this produces the behaviour we want, without the worry of our local variables messing things up for the calling code. Let’s go further: we can implement functions with a variable number of arguments using an enumeration of thunks. For example, we could write:


def try_these(*clauses)
    clauses.each { |clause| return clause.call rescue nil }
    nil
end

And call our function like this:


try_these(
    lambda { http_util.fetch(url, :login_as => :anonymous) },
    lambda { http_util.fetch(url, :login_as => ['user', 'password']) },
    lambda { default_value() }
)

We have just implemented the Try.these function from the Prototype Javascript library.

This technique gets us almost all of what we want for this common case of wanting to work around call-by-value semantics. As you can surmise from the fact that it has a name, it is not some newfangled shiny toy idea, it goes back to ALGOL 60, where it was known as call-by-name. (PHP has something called “Call By Name,” but it has a lot more in common with C++ references than it does with ALGOL parameter passing.)

The application of call-by-name as a substitute for full-blown macros isn’t novel either. Joel Klein pointed out that Call by need is a poor man’s macro. Another suggestion along similar lines is to rethink macros in Arc.

thunks: ugly name, ugly code

Our thunking approach solves a lot of our problems, but the implementation severely protrudes into the interface! We could argue that since our call-by-name functions have different behaviour than ordinary functions or methods, they ought to have different syntax.

That’s a reasonable point of view, and that’s exactly how languages like Smalltalk work: everything that involves delaying evaluation in some way uses blocks, even the if statements, which are methods that take blocks as arguments. So in Smalltalk, everything is consistent.

Ruby, OTOH, is not consistent. Operators like “&” and “|” are actually methods with call-by-value semantics, while operators like “&&” and “||” are special forms with call-by-value semantics. Likewise if you only need to delay one expression you can use a block, but if you need to delay two or more, you need at least one lambda. So another reasonable point of view is that we should follow Ruby’s philosophy of making the common case easy to use and not become reductionists trying to build everything out of five axiomatic forms.

So we have one approach—rewriting—that is crazy-hard to write but produces nicely readable code. And we have another approach—thunking—that is easy to write but produces unsightly boilerplate.

Maybe what we want is a rewriter, but we want an easier way to write rewriters for this simple case?

Called by name

Here’s how we could define and use a call-by-name function called “our_and”:


with (
    called_by_name(:our_and) { |x,y|
        if temp = x
            y
        else
            temp
        end
    }
) do
    # ...
    foo = our_and(bar(), blitz()) # method-like syntactic sugar
    # ...
end

What we just did is manufacture a rewriter without any sexps. Instead of getting rid of sexps, we’re treating them like assembler and using a declarative language to write the assembler for us. Our rewriter dutifully rewrites our code to look something like:


our_and = lambda { |x,y|
    if temp = x.call
        y.call
    else
        temp
    end
end
# ...
foo = our_and.call(
    lambda { bar() },
    lambda { blitz() }
)
# ...

We can define a rewriter for functions with splatted parameters too:


with(
    called_by_name(:try_these) { |*clauses|
        clauses.each { |clause| return clause rescue nil }
        nil
    }
) do
    # ...
    try_these(
        http_util.fetch(url, :login_as => :anonymous),
        http_util.fetch(url, :login_as => ['user', 'password']),
        default_value()
    )
    # ...
end

Becomes something like:


try_these = lambda { |*clauses|
    clauses.each { |clause| return clause.call rescue nil }
    nil
}
# ...
try_these.call(
    lambda { http_util.fetch(url, :login_as => :anonymous) },
    lambda { http_util.fetch(url, :login_as => ['user', 'password']) },
    lambda { default_value() }
)
# ...

It goes, boys!

—Lynn Hill after becoming the first person of either sex to climb The Nose of El Capitan, all free.

As of now, the rewrite gem supports called_by_name. You can write your own functions with call-by-name semantics using called_by_name just as you see here. As is standard with the rewrite gem, only the code in the do… end block is affected by your change.

call-by-name, in summary

To summarize, with the rewrite gem you can write functions that have call-by-name semantics without wrestling sexps into submission or encumbering your code with a lot of superfluous lambdas and calls:


with(
    called_by_name(:try_these) { |*clauses|
        clauses.each { |clause| return clause rescue nil }
        nil
    },
    called_by_name(:our_and) { |x,y|
        if temp = x
            y
        else
            temp
        end
    }
) do
    # ...
    try_these(
        http_util.fetch(url, :login_as => :anonymous),
        http_util.fetch(url, :login_as => ['user', 'password']),
        default_value()
    )
    # ...
    foo = our_and(bar(), blitz())
    # ...
end

This is a win when you don’t want your code encumbered with more lambdas than business logic. It may be a matter of taste, but part of what I like about Ruby having a special case for blocks is that they act as a huge hint that an expression is temporary: a block after #map suggests we are only using that expression in one place. Whereas when I see “Proc.new” or “lambda,” I expect that the expression will be passed around and used elsewhere.

Functions with call-by-name semantics communicate the same thing as blocks: the expressions are to be consumed by the function. When I see a lambda being passed to a function, I automatically expect it to be saved and possibly used elsewhere. For that reason, I prefer call-by-name semantics when an expression is not meant to be persisted beyond the function invocation.

Now, called_by_name is not a replacement for macros. There are lots of things macros can do that called_by_name cannot do (not to mention that there are lots of things code rewriting can do that macros cannot do). But just as Ruby’s blocks are a deliberate attempt to make a common case for anonymous functions easy to write, called_by_name makes a common case for macros easy to write and safe from variable capture problems.

Of course, called_by_name does so with lots of anonymous functions, and that is a much more expensive implementation than using a hygienic macro to rewrite code inline. But it feels like a move in an interesting direction: if it is a win to sometimes meta-program Ruby’s syntax with DSLs, it ought to also be a win to sometimes meta-program Ruby’s semantics with call-by-name functions.

afterword

So… Is this merely a way to replicate things that are already built into Ruby but do them fifty times slower?

I don’t know how to answer that question. When I heard Matz talk about Ruby at LL1, I didn’t catch the part of his speech where he described how to use metaprogramming to build a really neat web development framework. When you first see a new tool, you naturally start by applying it to problems you already know how to solve with your existing tools in the same way you have always solved such problems.

Only later, after this tool becomes perfectly natural to you, do you start to think of entirely new ways to use the tool. I’m not there yet, but my experience tells me that it’s always a win to have more freedom, to have fewer things you can’s do with a language.

If just one person—maybe it’s me, maybe it’s somebody else—leans forward one day and sees a new way of solving a problem with call-by-name semantics, I’ll consider working on this feature time well spent.

It probably won’t be something trivial like replicating short-circuit boolean operators. But it will be interesting, and I’m looking forward to finding out what it is.

A little something about dumbing code down to the lowest common denominator

noreply@blogger.com (Reginald Braithwaite) — Sun, 22 Jun 2008 11:42:00 +0000

When the productive have to ask permission from the unproductive in order to produce, then you may know your culture is doomed.

—Ayn Rand, via Matt Mower

update: Chalain nails it, especially when he writes “… she's right about those two little words: ask permission. And not about dumbing down code.” Although the examples he gives are nothing like the examples that crossed my mind.

I have been happiest when I have worked with people much, much smarter than I am, people who forced me to raise my game. I recall that when I worked on JProbe Threadalyzer, the three key people on my team were Steve Rosenberg, Christian Jaekl, and John MacMillan. Steve was my manager, and he forced me to raise my game as a team lead. Christian was our domain expert, he had written his Masters Thesis on analyzing the behaviour of concurrent software, and he forced me to immerse myself deeply in concurrency just to understand what he was designing. And John was a C++ Wizard. I recall him being extremely patient with me a I got up to speed writing production C++ code.

All three were patient with my shortcomings, but our culture was to help me be more productive, to help me understand their ideas. Not to hold them back so that I could stay in my comfort zone.

Not going dark

noreply@blogger.com (Reginald Braithwaite) — Wed, 18 Jun 2008 03:30:00 +0000

Rubyforge is now hosting an “initial pre-release of a preview of an alpha of an undocumented proof-of-concept” of the rewrite gem. More on this presently, but the important (and really the only thing of interest) about rewrite is that when you use rewrite to write:

Person.find_by_last_name("Braithwaite").andand.first_name

...You are not opening the Object, Person or Nil classes to add an andand method, nor are you creating a weird temporary BlankSlate object. Rewrite does its thing by rewriting Ruby code: it performs syntactic metaprogramming, much as Lisp macros rewrite Lisp code.

what problem does rewrite solve?

Recall that when you use the “standard” implementation of things like andand or try, you are openly modifying core classes like Object.

Therefore, you are reaching out and touching every line of code in your project. You probably aren’t breaking everything, but even if the chance of introducing a bug by adopting something like “try” is infinitesimal for each source code file in your project, the chance grows greater and greater as your application grows.

The problem is that you are introducing a change on Object, and everything depends on Object. This is very different than introducing a change in your code. In that case, only the other bits of code that directly depend on your code are at risk.

Also, imagine if you introduce try and are careful not to break anything. Now somebody else wakes up one day and decides they need a method that works like Prototype’s Try.these. They call it “try.” They just broke your code, dude! Not only are you making everything dependant upon your version of try, but your code is dependent upon everyone else not breaking try as well. It’s a train-wreck waiting to happen.

Rewrite restricts things like andand or try to your code and your code alone. Sure, if you introduce a bug in your code, you may break things that directly depend on your code. But if you introduce “try” using rewrite instead of modifying Object, you will not reach out across your project and break something entirely unrelated that happens to have defined its own version of try in a completely different way.

how does it work?

Rewrite takes your code, converts it to an sexp with Parse Tree, then rewrites the sexp using one or more rewriters you specify. Finally, it converts the sexp back to Ruby with Ruby2Ruby and evals it. It does this when the code is first read, not every time it is invoked, so we mitigate the “do not use andand in a tight loop” problem.

For example, rewrite converts this:


emails.find_by_email(email).try(:destroy)

Into:


lambda { |receiver, method|
   receiver.send(method) if receiver.respond_to? method
 }.call(emails.find_by_email(email), :destroy)

And this:


 numbers.andand.inject(base_sum()) { |total, number| total + number }

Into:


 lambda { |__1234567890__|
   if __1234567890__.nil?
     nil
   else
     __1234567890__.inject(base_sum()) { |total, number| total + number }
   end
 }.call(numbers)

Note that with the examples, the names “andand” and “try” completely go away. If someone else defines a try method elsewhere, it will not affect your code because your code never executes a method called try.

the most important open problem to solve in this area

At the moment that is a huge PITA when creating new rewriters. I don’t think that it's OK that a language makes it harder for a library creator than for an application developer, so I am working on making it easy to write things like andand or try.

puzzles for language weenies

Note that the andand example above uses gemsym for its parameter while the try example does not. Why? What could break if it used a name like “receiver?” If you can figure out the salient difference between these two example rewrites, you can probably explain why rewriting try produces exactly the same semantics as the original open classes implementation, but rewriting andand produces a subtle change in semantics.

Today's public key is { :n => 46, :e => 5 }

noreply@blogger.com (Reginald Braithwaite) — Sat, 14 Jun 2008 11:15:00 +0000


def rsa_encrypt(message, public_key)
  n = public_key[:n]
  e = public_key[:e]
  n_arr = message.downcase.scan(/[a-z]| /).map { |character|
    "abcdefghijklmnopqrstuvwxyz ".index(character)
  }
  n_arr.map { |m|
    (m ** e).modulo n
  }
end

def rsa_decrypt(ciphertext, private_key)
  n = private_key[:n]
  d = private_key[:d]
  n_arr = ciphertext.map { |c| 
    (c ** d).modulo n
  }
  n_arr.map { |n| "abcdefghijklmnopqrstuvwxyz "[n,1] }.join('')
end

And the secret message is: [11, 38, 13, 0, 24, 36, 16, 26, 36, 18, 24, 36, 1, 16, 21, 11, 17, 13, 0, 24]

Bitchiness also sucks

noreply@blogger.com (Reginald Braithwaite) — Mon, 09 Jun 2008 11:06:00 +0000

“This framework I never use in a programming language I also never use is designed in a way that I don’t like, so I’m going to raise a ruckus even though I’m never going to use it and there are other frameworks that meet my design standards better” just seems like a dumb thing to write a bitchy blog post about.

—Chuck Hoffman, activerecord sucks, but it doesn't matter

Chuck, thank you for saying what needs to be said. Bitchy blog posts purport to describe shortcomings, but are they anything more than thinly veiled chest thumping from an author who feels he is the smartest person in the room?

p.s. Question: Are bitchy blog posts about bitchy blog posts (like this one!) still bitchy blog posts?

Skirting the edge of topicality: Five days left to register for RubyFringe

noreply@blogger.com (Reginald Braithwaite) — Thu, 05 Jun 2008 16:59:00 +0000

Pete Forde wrote to say that there are only 5 days left to register for RubyFringe, and currently fewer than 20 tickets left.

Coïncidentally, I have submitted my title and abstract:

Ruby.rewrite(Ruby)

An introduction to writing Ruby that reads, writes, and rewrites Ruby. In an extremely short period of time we will extend the Ruby language to include new conditional expressions, add new forms of evaluation such as call-by-name and call-by-need, and if time permits we’ll define new recursive combinators.

In other words we’ll practice the truest form of constructive criticism: Instead of complaining about missing language features, we’ll implement them.

Luckily for you, the other speakers are people with actual talent and a knack for entertaining people. What are you waiting for? Register!

"Lets" organize our Ruby Code

noreply@blogger.com (Reginald Braithwaite) — Fri, 30 May 2008 19:02:00 +0000

This post is for folks interested in the Invocation Construction Kit for Ruby. There is a new invocation, Ick::Syntax::Lets:


lets(
    :person => Person.find(:first, ...),
    :place  => City.select { ... },
    :thing  => %w(ever loving blue eyed)
) {
    "#{person.name} lives in #{place} where he is known as the '#{thing.join(' ')} thing.'"
}

Quite simply, #lets provides you with a way of making block-local variables. Ick already includes #let, which does almost exactly the same thing. However, #let is limited to just one variable:


let(Person.find(:first, ...)) { |person| ... }

Expanding #let to multiple variables with its existing syntax just doesn’t scale properly:


let(
    Person.find(:first, ...),
    City.select { ... },
    %w(ever loving blue eyed)
) { |person, place, thing|
    "#{person.name} lives in #{place} where he is known as the '#{thing.join(' ')} thing.'"
}

Thus, the #lets syntax uses a hash so that the variable names are close to the expressions denoting their values.

That’s all. If you are interested in why anyone would need #lets, read on…

Scope Creep

Recently there has been a lot of interest in exploring the extremes of OO style. One of the tenets of such a style is to work with small classes and short methods. Why do you suppose we want to do that?

Well, the underlying principle is to chunk things into small, workable pieces with clear purposes. This principle abounds everywhere: posts just like this are broken up into sections with headings and the prose is subdivided into paragraphs. So I quite agree with the principle behind small classes and short methods.

__________. You just knew that a word like “However” would follow a paragraph like that, didn’t you?

The principle of subdivision is terrific. But are classes and methods the only mechanisms we have for subdividing code? In some languages, that may be true. In other languages, that is not true.

For example, since version 1.1 Java has permitted classes to be nested inside of other classes, calling them “inner classes.” This is a very powerful technique of organizing code: If class A is the only class that ever uses class B, why should class B live in its own file, visible to every other class? Placing class B inside of class A is a big win: it is immediately clear that A is the only user of B, and when looking at the code in your IDE you do not see class B promoted to equal standing with A: it is clearly subordinate to A.

Of course, placing class B inside of class A makes A a bigger class. Is that wrong? Perhaps it is at some times, but it’s a big win at others. Remember we said B is subordinate to A? An inner class can be made private, explicitly telling the compiler and other programmers that it is limited in scope.

Limiting scope is a very powerful organizing technique in code. We can debate how important it is that the compiler enforce scope, however given the importance of writing code for humans to read and understand, I’m personally in favour of any technique that sends a strong signal to your fellow programmers explaining the intended structure and organization of the code.

Small Methods

There are techniques for organizing methods into classes or modules, and techniques for organizing classes and modules into larger classes and modules. But what about individual lines of code? Are methods really the only mechanism we have for organization at the lowest level?

I think not. Even within methods, there are certain practices that logically chunk your code into small, workable pieces with clear purposes. For example, Algol, Pascal, Modula, and many other languages support nested functions or procedures:


procedure print(var j: integer);

  function next(k: integer): integer;
  begin
    next := k + 1
  end;

begin
  writeln('The total is: ', j);
  j := next(j)
end;

It is clear that #next is used solely by #print. Since most OO languages do not permit nested procedures, the programmer is left to choose between making #nest a private method or carving #print out and making it a strategy class. Making #next a private method does keep it from prying eyes outside of the class, however it implies that all methods of the class may want to use it, which is clearly not the case.

If we are allowed to nest objects, we can fake nested procedures with objects. Ruby’s Proc class makes it easy:


def print(j)
    next = proc { |k|
        k + 1
    }
    p "the total is: #{j}"
    j = next.call(j)
end

(There is a fairly major difference in what these two snippets of code do thanks to the difference between call-by-value and call-by-reference, but let’s wave our hands furiously and stick to the point about nesting functions.)

Containing Variables

In imperative languages, one of our biggest headaches is managing mutable local variables. If you only need one in one particular place, it is helpful to have a way of nesting the variable definition. In Java:


for(int i=1; i<11; i++) {
    StringBuffer j = new StringBuffer();
    // ...
}

And in Perl (thanks Chromatic):


{
  my $person = Person->find( first => ... );
  my $place = City->select( sub { ... } );
  my $thing = [qw( ever loving blue eyed )];

  return $person->name() . " lives in $place where he is known as the "
    . join( ' ', @$thing ) . ' thing.';
}

The variables i and j are limited in scope to the body of the for loop. This called block scoping, and I personally love it. Block scoping permits us to make small, self-contained blocks of code and use them inline. If we need to move them or change them, we know which things are limited in scope to the block and which reach outside of the block and thus might be affected by our changes.

In Javascript we can fake block scoping using procedures, but it’s syntactically noisy. Can we do the same thing in Ruby? Yes, and that’s exactly what #lets does:


lets(
    :person => Person.find(:first, ...),
    :place  => City.select { ... },
    :thing  => %w(ever loving blue eyed)
) {
    "#{person.name} lives in #{place} where he is known as the '#{thing.join(' ')} thing.'"
}

This is a block with three block-local variables in it, just like you might find in Java. It breaks a complicated expression up into smaller pieces (“How to find the person,” “How to find the place,” “What kind of thing we have,” and finally “How to describe it all as a string”). And those pieces are all grouped together so you know they are not used elsewhere.

A Hack, wrapped in a Workaround, inside a Kludge

Unlike the other elements of Ick, #lets actually rewrites your ruby code. The code example above is actually rewritten as:


proc { |__121217088531733__|
  lambda do |person, place, thing|
    "#{person.name} lives in #{place} where he is known as the '#{thing.join(" ")} thing.'"
  end.call(
    __121217088531733__[:person], 
    __121217088531733__[:place], 
    __121217088531733__[:thing]
  )
}.call(
    :person => Person.find(:first, ...),
    :place  => City.select { ... },
    :thing  => %w(ever loving blue eyed)
)

The rewritten code is then evaluated. At the moment, this happens every time you call #lets, so it is expensive. Even by Ruby standards. Hopefully, there will be a future version of Ick::Syntax that only rewrites by need or perhaps just once when the code is first read.

So, #lets takes the syntactic noise of using procs to create block structure and hides it from us. How?

Well, my first cut at it used Ruby2Ruby and regular expressions. Then I was inspired by Aanand Prasad’s Haskell-style monad do-notation for Ruby to work directly with s-expressions. Although the code is technically longer, it’s actually much, much better. For meta-syntactic programming, you need to work with the abstract syntax tree.

And with ParseTree and Ruby2Ruby, you have all the tools you need to write code that writes code, with the teeny-weeny proviso that what you want to do is translate your Ruby into Lisp, manipulate the Lisp, and then translate it back into Ruby:


def rewrite_sexp(names_to_values, sexp)
  mono_parameter = :"__#{Time.now.to_i}#{rand(100000)}__"
  # the next four assignments are exactly why we want #lets:
  sorted_symbols = (names_to_values || {}).keys.map(&:to_s).sort.map(&:to_sym)
  parameters = if sorted_symbols.size == 1
    s(:dasgn_curr, sorted_symbols.first)
  else
    s(:masgn, s(:array, *sorted_symbols.map { |sym| s(:dasgn_curr, sym) }) )
  end
  values = s(:array,
    *sorted_symbols.map { |sym|
      s(:call, s(:dvar, mono_parameter), :[], s(:array, s(:lit, sym)))
    }
  )
  body = sexp.last

  s(:defn, :__anonymous__, 
    s(:bmethod, 
      s(:dasgn_curr, mono_parameter), 
      s(:call, 
        s(:iter, 
          s(:fcall, :lambda), 
          parameters,
          body
        ), 
        :call, 
        values
      )
    )
  )
end

You know what to do:

sudo gem install ick

Cheers, and thanks very much to Ryan Davis, Eric Hodel, Matt Mower, and Aanand Prasad.

A Limitless Resource

noreply@blogger.com (Reginald Braithwaite) — Thu, 29 May 2008 15:44:00 +0000

Spam is the only truly limitless resource on the web and the ability to resist the endless waves of spam may well well be the single most important requirement of future distributed computing architectures…

Anybody who proposes a web architecture that doesn’t provide a means to handle spam is just engaging in wishful thinking.

—Those Who Live by the Spam…