A Brief History of Dangerous Ideas (Brief because (a) I really don’t know that much history, and (b) there’s really only one idea here with a few examples, and once you get the point, we’re done and can move to the outro.)

Astronomy and Celestial Mechanics were dangerous ideas because they undermined the most powerful organization of their day, The Church. That’s why people have been banned, tortured, and burned at the stake for talking about these ideas.

Crossbows were a dangerous idea because they allowed an untrained peasant to kill a knight. Longbows were not a dangerous idea, because only trained archers can kill a knight with a longbow, and the nobility were the only people who could compel peasants to practise yeomanry.

Cryptography is not a dangerous idea, because we really don’t know if any of our algorithms and protocols are resistant to the NSA. This was hi-lighted when researchers “discovered” differential cryptanalysis. When they looked at the DES algorithm IBM has been promoting since the 1976, they found that it had been specifically tuned to resist differential cryptanalysis. IBM ‘fessed up: the US government had told them to tune things that way without explaining why, leading us to conclude they had known about this attack for decades before it became public knowledge. Today, we have no idea whether what we think is strong is actually strong or whether it has vulnerabilities and back doors governments can exploit.


Zed and Reg at Rubyfringe exchanging dangerous ideas, photo by Libin Pan
Miles Davis was a walking, talking, trumpet-playing dangerous idea. Not because he reinvented Jazz five different times (Dear Steve: Apple II, Macintosh, iPod/iPhone, Pixar. One more for the tie, two for the record.) Miles was an egocentric, venal man who worked the system, not undermined the system. But he was still dangerous because he got white people directly interested in black music. There was no Elvis or Vanilla Ice or anyone else between his music and the mainstream audience. For a government intent on keeping America’s two dominant cultures divided through fear, anything uniting them was a threat.

People say Miles’ legacy is his music. To me, Miles’ lasting legacy is people like me, people with one parent from each culture who grew up dancing to the same music together. People who, incidentally, do not vote for governments that take a divide and conquer approach to culture.

And on to tech. Microcomputers were not a dangerous idea. But personal computers were dangerous. It took decades for IT departments to regain control over people bringing their own computers to work. They can thank Microsoft for helping them get back into the driver’s seat.

(This, incidentally, is why I really dislike Microsoft’s policies: it has nothing to do with their lack of taste, it has to do with their mission to make the computer on my desk belong to my IT department or the record label or the movie studio or—I suspect—the government.)

Web applications are dangerous. Never mind the fact that they make desktop applications obsolete. The people who built desktop applications just go and get jobs writing web applications. Same people, different shit. But as Giles Bowkett pointed out, web applications just might make venture capital obsolete! When you don’t need hundreds of programmers and distribution channels and all the other friction-managing elements of a company that ships old-school software, you need a lot less money to start a business.

And on to media. You know that the web is busy putting newspapers out of business. My wife and I watched YouTube last Saturday Night. I’m not talking about the advertising business: I think we would have been happy to watch ads to watch our Mitch Hedberg and Billy Connoly comedy clips. But the web lets us choose what we want to watch, when we want to watch it. The network can’t put their up-and-coming show on right after their hit to give it a boost. The new show has to compete on its own merits. That puts users in control, and that’s dangerous.

Joel Spolsky said a similar thing about pricing all music at 99 cents a track: it means the labels can’t kill an artist by sticking their CD in the $3.99 crapola bin. Users choose what they want to listen to. That’s dangerous, again because users are in control.

Okay, that’s enough. Dangerous ideas are the ones that subvert the existing hierarchy of control, not just the ones that shuffle people around in the same old chairs. Apple Macintosh with a GUI replacing a PC with a command line? Not dangerous. Apple Macintosh with a Laserwriter and Aldus Pagemaker allowing someone to launch a magazine in their basement that competes with a company employing dozens of layout artists? That’s dangerous, and that’s interesting.

Dangerous equals subversive equals interesting.

outro

Deep breath. Okay, the next thing is not particularly dangerous for the world at large, but it is for me. I am retiring from blogging and retiring from hacking on Ruby. Maybe I’ll un-retire one day. I don’t know, the future is not set.

Miles Davis wasn’t afraid to move on when the time was right, even if what he was doing seemed to make people happy. Respecting his legacy means seeking what he sought.

Remember how I said that microcomputers were not dangerous, but personal computers were? Right now, I would say this: Ruby is not dangerous, but Rails is dangerous and Merb is dangerous and Sinatra is dangerous. Rewriting for Ruby is interesting. I believe it is useful. But is it dangerous? No.

Likewise, I can say with a clear conscience that while writing is gratifying, and trying to write out and explain ideas has helped me understand things, the writing I’ve been doing is not dangerous. It doesn’t subvert.

So while hacking away on Ruby the language and blogging about software development is gratifying and useful, they are not dangerous activities. They are microcomputers, but they are not personal computers.

I am going on vacation from August 2nd through 10th. During that time I plan to do absolutely no thinking about computers. For what it’s worth, I will be engaging in activities with faux danger: wreck diving and sport climbing. (p.s. these are not solitary pursuits: if you want to try some of the world’s greatest wreck diving and sport climbing, get in touch).

When I return, I will give things some serious thought and hopefully, discover a way I can help make our world a more dangerous place.

Thank you ever so much for your support and interest and feedback. Especial thanks to my fellow bloggers like Joel, Joey, Giles, Damien, Obie and so many others who exchanged ideas with me and kept the debate alive. And Reddit? And Hacker News? You rock, you are the future. I can’t wait to see how your communities and technologies evolve.

Warmest regards,


Reginald Braithwaite  ¶ 10:04 AM 55 comments

  That not is not the not I meant During my presentation at RubyFringe, I put this slide up while talking about “Adverbs:”



Judging by the fun we had discussing this point later, I really needed a better example or a new idea. But for thse who are a little curious about where I was trying to go with this…

more on why not is not the same as not

I saw the #not method somewhere, and the point was that the author felt it read better than writing “!blitz.blank?” As someone pointed out, Ruby also gives us a not keyword, so you could write “not blitz.blank?” if you prefer the word “not” to the punctuation “!”

So the argument is that:

blitz.not.blank? == !blitz.blank? # and also == (not blitz.blank?)

But there’s another point we’re missing comparing the word to the punctuation. Let’s switch to English and you’ll see what I mean. The sentence “It is not the case that blitz is blank’ does not mean the same thing as “blitz has the property of not being blank.”

Whaaaa?

The key here is to think about where the parentheses go when we parse the two ways to express ourselves. Imagine “blank?” and “not” are functions instead of methods or keyword. Then two forms would look like this:

not(blank?)(blitz) # == blitz.not.blank?
not(blank?(blitz)) # == not blitz.blank?

Very different. The first expression passes the blank? function to the not function. The result is another function, the inverse of blank? itself, and that’s what we use. The second form is our old not keyword, where we pass blitz to out blank function, which presumably returns true or false, and we pass the result to our not function which inverses the boolean.

So in one case we are taking the inverse of a function, in the other we are taking the inverse of a boolean. This is what I was feeling out with my digression into wondering why Ruby reads from right to left. We have ways of applying functions to functions, but we don’t have easy ways of writing methods that alter other methods directly. And we certainly don’t have an easy way of sending a message that invokes such a method on an arbitrary object.

I guess I wonder whether we could ever write this blitz.not(blank?) to express this idea, or perhaps blitz.(not blank?), or perhaps some other syntax. But the point would be that we want the inverse of a method, not the inverse of a boolean.

speaking of jazz, so what?

It seems very abstract when we look at something like #not.blank?, because the result of an inverted #blank? method would usually be identical to the result of inverting its result.

But let’s consider the case where blitz is actually a proxy for a value in a SQL database column. So you might have something like “my_model.blitz.not.blank?” Fair enough?

Now if you know SQL, I think you will agree that there are times when the following is true:

  (model1.blitz != model2.blitz) != (not model1.blitz == model2.blitz)

Although it looks like x.not.eql?(y) should always have the same semantics as not x.eql?(y), SQL has special semantics for testing NULLs. Specifically, NULL is neither = nor <> any other value. So if you carefully map your proxies to the underlying SQL behaviour, you want the above to have this behaviour.

(For those who are blessed with a lack of SQL in their lives, NULL is a special value that is not the same thing as a Ruby nil. For example NULL = NULL is false in SQL. So if we map SQL values to Ruby objects, the following is the case: NULL != anything is false, NULL == anything is also false, therefore not NULL == anything is true, and therefore (NULL != anything) != (not NULL == anything).

If this digression into SQL NULLs now makes you regard your current ORM framework with deep suspicion, congratulations. They all suck, the trick to choosing an ORM is finding one that sucks in ways that aren’t fatal to your project.)

Essentially, Ruby already has a not adverb as a special case for #not==. #!= is there because sometimes (not foo == bar) is not equal to (foo != bar). However, like many things in Ruby this is a one-off thing. There isn’t a bang-method construct, #== and #!= are two completely arbitrary symbols that we happen to associate with equality and its inverse.

But maybe if we had a way to derive #!= from #== using a not adverb, maybe if we could write #== and then write a #not for the method, we would be able to do interesting things like replicate SQL’s trivalent logic more accurately. We would know that foo.not < 5 is not the same thing as not foo < 5, because if foo is a SQL null, foo < 5 is false, as is foo.not < 5. And neither throws an exception!

So getting back to the #not method as presented, I don’t know if it’s necessary, and if it does nothing more than replicated the exact semantics of the bang punctuation or not keyword it is clearly a matter of taste.

But I do feel that there is a use case for “adverbs,” for a way to talk about modifying methods themselves. Unfortunately, right now the mechanisms for working directly with methods are somewhat arcane and thus we don’t naturally consider them. So discussions like this feel very fringe and “out there.”

But at least now I feel like I have properly explained the not I meant, and not the not I didn’t mean.  ¶ 5:00 AM 2 comments

  "L" is not a code smell During my presentation at RubyFringe, I shared a question that has been swirling around in my brain for a while: Are IDE features really language smells?

I don’t think it’s an original thought. If nothing else, it’s a corollary to what I believe to be true about many of the GoF design patterns: Many of them are workarounds, ways to Greenspun missing language features. Now, this is probably not the case for all IDE features, and in truth it may be that there are some features which could be implemented in either the language or the IDE, but the IDE may be the best place to put them.

But there is a fairly large class of IDE features that strike me as language workarounds. One of them is definitely the ability to spit out a lot of boilerplate. If you need a lot of code written, you ought to be able to get your programming language to do it for you, not your IDE.

There is room for people to disagree about this. There are some who feel (Strawman alert!) that programs consisting of large numbers of simple elements are easier to understand than programs consisting of a small number of highly abstract elements. Those folks feel an IDE gives you the ease of writing a program quickly plus the ease of reading that same program quickly. They feel that abstractions make the program easier to write but harder to read.

I happen to disagree with this, and if you have been reading this weblog for more than a couple of days you have already read why my experience leads me down a different path. Although in deference to my colleagues with different views, I offer this quote:

All problems can be solved by adding another layer of abstraction, except the problem of having too many layers of abstraction.

Anyone who has dealt with an hammer factory will agree.

So back to “L.”

Two speakers before me, Giles Bowkett gave his excellent Archaeopteryx—um—presentation. I hesitated over that word, because I could just as easily say performance. Performances are terrific entertainment, but they sometimes obscure the message behind them. I want to say outright that while this is true of many other subjects, I felt it worked for Giles because the subject of his presentation was software development as a how rather than a what, and for Giles the “what” is performance.

(Giles got a standing “O,” and many people might be tempted to rush out and make their presentations just as stimulating (400+ in-your-face slides punctuated with loud, driving drum and bass). Be sure that your material matches your presentation style! If not, people may walk away saying “Wow, amazing, but what exactly did she say?” I think it worked for Giles and that’s quite an accomplishment.)

Now really, back to “L.”

Giles is one of the people using closures in Ruby. Meaning, he is passing functions around and storing functions in objects. I am not going to try to say exactly what Archaeopteryx does, so I will describe this style of programming using an imaginary companion program that creates walking bass lines. I will call it Troody.

Let’s simplify things greatly and say that Troody will only ever play in perfect 4-4 time and further that Troody only ever play one of the eight notes in a particular chord’s standard scale. The probability of playing each of those notes on any one “beta” could be represented as an array with eight elements, like this: [.35, .05, .1, .05, .25, .05, .1, .05 ]. You can imagine passing arrays like this around in Troody.

For example, we can pass this array to an object that actually plucks the strings: Plucker.new.start_plucking([ 0.35, 0.05, 0.1, 0.05, 0.25, 0.05, 0.1, 0.05 ]).

Let’s try writing a naïve Troody Plucker:

class Plucker
    def start_plucking(probs)
        while (self.tune.playing)
            if (Metronone.on_the_beat)
                r = rand
                cumulative_probs = probs.inject([]) { |cum, element| 
                    cum + [ cum[-1] && (cum[-1] + element) || element ] 
                }
                notes_to_cumulative_probs = (1..8).zip(cumulative_probs)
                note_to_play = notes_to_cumulative_probs.detect { |note, prob| prob >= r }
                self.pluck(note_to_play)
            end
        end
    end
end

You pass it a set of probabilities, it produces bass notes. But stop, that’s so procedural. Let’s learn from a flying creature, let’s learn from Archaeopteryx. Instead of passing arrays, let’s pass lambdas, like this: lambda { [.35, .05, .1, .05, .25, .05, .1, .05 ] }. Now whenever Troody needs the probability of something, we call the function with .call or Ruby’s [] alternative syntax. So now we write Plucker.new.start_plucking(lambda { [.35, .05, .1, .05, .25, .05, .1, .05 ] })

Our new Plucker code is the same as the old, except we write:

cumulative_probs = probs.call.inject([]) { |cum, element| 
    cum + [ cum[-1] && (cum[-1] + element) || element ] 
}

We now call the probs lambda when we need a note. That’s it, we’ve added a .call call. What does that get us? Well, here’s one thing: If we want the probability to change over time, our function can do that, and we don’t have to rewrite our start_plucking method to handle the idea.

For example, here’s a probability lambda that usually plays the same way but from time to time decides it ought to play pedal notes (refactoring to OO is an optional exercise):

probs = lambda { |bars_of_pedal, beat|
    lambda {
        if beat == 0
            if bars_of_pedal == 0
                bars_of_pedal = 1 if rand < .05
            elsif bars_of_pedal == 5
                if rand < .25
                    bars_of_pedal = 0 
                else
                    bars_of_pedal += 1
                end
            elsif bars_of_pedal == 9
                if rand < .5
                    bars_of_pedal = 0 
                else
                    bars_of_pedal += 1
                end
            elsif bars_of_pedal == 13
                bars_of_pedal = 0
            end
        end
        beat = (beat + 1) % 4
        if bars_of_pedal == 0
            [ 0.35, 0.05, 0.1, 0.05, 0.25, 0.05, 0.1, 0.05 ]
        else
            [ 1.0, 0, 0, 0, 0, 0, 0, 0 ]
        end
    }

}.call(0, 0)

Thanks to the way we’ve separated the probabilities from the plucking, we do not need to subclass Plucker to try a different playing style in Troody.

As Giles pointed out, this is the Strategy Pattern. We are making different kinds of pluckers by encapsulating the logic of what to pluck in something we pass to a plucker. Archaeopteryx appears to do this everywhere. There are lambdas paramaterized by lambdas, lambdas that return lambdas…

This creates a problem. Imagine a programing language where all the keywords are in upper case: IF foo THEN bar ELSE bizzat. Try reading such a program aloud, and you end up shouting the punctuation but speaking the words. This is wrong! We should be shouting the words and whispering the punctuation!

And the problem with Ruby’s lambdas is that if you use a lot of them the word lambda really starts to stand out. So Giles fixed this by aliasing it to L: and using [] instead of .call():

alias :L :lambda

L{ |a| a + a}[5].
    => 10

Much nicer, and as Giles pointed out, this is an example of Ruby’s strength. If you have a program that rarely uses lambdas, you probably want lambdas to stand out when you use them, so you don’t alias lambdas to L and you use the call method, not the square brackets. But if you use lambdas a lot, it’s a win to abbreviate things.

Okay, we’re talking about “L.” Good.

Now in my talk, I said that abbreviating lambda to L was a code smell. I was wrong! Giles, my bad!!

What I actually think is that needing to abbreviate lambda to L is a language smell. Very different. If you show me a Java program and you show me Strategy Pattern, I shouldn’t say it’s a code smell. I should say too bad for you that you need all that boilerplate when Ruby lets you do that with the word lambda and a pair of curly braces.

So now to “L:” Giles, if lambdas are integral to Archaeopteryx, if they are so woven into the fabric of what Archaeopteryx does that you want the keyword “lambda” to fade away, I honestly think this is a place where the language could help you.

For example, what if Ruby had call-by-name semantics? You could write:

Plucker.new.start_plucking([ 0.35, 0.05, 0.1, 0.05, 0.25, 0.05, 0.1, 0.05 ])

# or...

bars_of_pedal, beat = 0, 0
Plucker.new.start_plucking(
    if beat == 0
        if bars_of_pedal == 0
            bars_of_pedal = 1 if rand < .05
        elsif bars_of_pedal == 5
            if rand < .25
                bars_of_pedal = 0 
            else
                bars_of_pedal += 1
            end
        elsif bars_of_pedal == 9
            if rand < .5
                bars_of_pedal = 0 
            else
                bars_of_pedal += 1
            end
        elsif bars_of_pedal == 13
            bars_of_pedal = 0
        end
    end
    beat = (beat + 1) % 4
    if bars_of_pedal == 0
        [ 0.35, 0.05, 0.1, 0.05, 0.25, 0.05, 0.1, 0.05 ]
    else
        [ 1.0, 0, 0, 0, 0, 0, 0, 0 ]
    end
)

And you would get the same behaviour as if you were using lambdas.

That’s it, that’s what I should have said on stage: any time you are working around your language—whether in your IDE, or by modifying open classes, or by abbreviating things—that’s a place where we should step back, where we should ask if our language is missing something.

The answer may very well be “no.” But we ought to at least ask the question.  ¶ 12:43 PM 6 comments

  Raganwald on the Fringe of a Nervous Breakdown Thanks to Joel, I now have a format for Sunday’s presentation at RubyFringe: Twenty slides, each on the screen for exactly twenty seconds.  ¶ 11:40 AM 2 comments

  What does the word "Quality" mean? The phrase “Quality Code” came up in a conversation with some colleagues. Just so we’re clear on what I mean when I use the word “Quality” with respect to code or architecture or software design:

Quality is the antonym of “defective,” with the addition of time. Meaning, it has few known defects in its current state, and we have high confidence that we will not discover defects in its current state over time, and additionally that we will not create further defects in it as we add to or alter its functionality in the normal course of maintaining this software.

That’s a lot of blather suggesting that quality in a code base is all about making code that works as expected. Some code is designed to be extremely flexible or malleable to make it easier to add functionality in the future. This is an admirable property of a code base, however I do not think of that as affecting a code base’s quality except for the fact that a more flexible code base does sometimes allow adding functionality with a lower probability of introducing new bugs or regressions.

I am not attempting to sum up all of the desirable properties of a code base. For example, in some business circumstances, increasing velocity of development at the expense of quality is a good trade-off, such as when putting together a demo that wins a large piece of business or when creating prototypes that users can use to hone a design.

But when I talk about code having very high quality I mean that it has few defects and that I expect it to continue to have few defects over time. Code that is extremely brittle and highly coupled is not of high quality because I expect that any attempt to make changes will introduce regressions.

Thus, the word “Quality” is not a subjective, aesthetic judgment. It is an objective measure of one axis of value to its owners.  ¶ 4:13 PM 7 comments

  Ruby.rewrite(Ruby) in Ten... Nine... Eight... Seven Days... Checking the RubyFringe schedule, I see I must somehow keep the audience from yawning for thirty minutes between 2:10 and 2:40 on Sunday afternoon. I have the unenviable task of appearing between two incredibly smart and hardworking people, Damien Katz and Chris Wanstrath.

Upon reflection, I think I’d better open my talk with this quote from Horse Feathers:

I’ve got to stay here, but there’s no reason why you folks shouldn’t go out into the lobby until this thing to blows over.

See you at RubyFringe!  ¶ 2:45 PM 3 comments

  My analyst warned me, but metaprogramming was so beautiful I got another analyst

Try to imagine a world where every programmer you know is a wannabe language designer, bent on molding the language to their whims. When I close my eyes and imagine it, I have a vision of the apocalypse, a perfect, pitch-black storm of utterly incomprehensible, pathologically difficult to debug code.

Whereas I have a vision of a world not too much different than the one we live in today. A few people invent ridiculous things, incomprehensible things. Most people carry on doing what they’ve always done. And a few people invent new, worthwhile things that move us forward.

In my world, a few people inventing great things more than makes up for a few other people inventing ridiculous things. If we get Rails out of the deal, isn’t it worth giving Ick a bemused smile, tousling the creator’s hair and encouraging him to “keep trying?”

Jeff goes on to put some smart people on a pedestal, prostrating before their greatness. I admire them as well, but I don’t confer a priesthood upon them. Once upon a time, reading and writing was not something plebeians did. Even the bible was a closed book, you only heard what was recited in Church.

Does anyone doubt that democratizing the written word—through education and especially Gutenberg’s invention—has been a force for good?


I imagine that if we traveled back in time we would find monks bewailing what would happen if anyone was allowed to write. People—the monks would stammer—people might write doggerel! Pornography!! Graffiti!!!

But of course, they will also write beautiful things. And useful things. Like theories of Astronomy that violate what the priests “knew” to be true. And that is the point: if you leave things up to the priesthood, all you get is stuff that is neatly aligned with what they already know. You cannot make progress by subjecting new ideas to scrutiny in a committee, even a smart and well-educated committee. Maybe especially not a smart and well-educated committee.

If it were up to educated people, do you think there would be personal computers today?

Language design in programming is the same thing. Yes, it creates an opportunity for misunderstanding. Yes, you can make an amazing amount of trouble for yourself and for others. But to put it under lock and key in the Church where only the priest are permitted to mention it in hushed tones… this is wrong.

The message is clear: You and I are worker ants. We do not think. We do not question our tools. We simply use them as we are directed by those in control. I find that far more horrifying than just about anything else I can imagine coming out of giving people more freedom.

Come on. Really. Let’s get ahold of ourselves. As a wonderful movie’s last line went:

“You have to have a little faith in people.”  ¶ 9:45 AM 20 comments

  I've seen things you people wouldn't believe Pete Forde of Unspace emailed to say that the very first time he tried andand, he discovered a bug:

nil
  => nil
nil.present?
  => false
nil.andand.present?
  => true

Reg: Embarrassing.

Pete: No sir. Not embarrassing, because no one’s ever going to find out they’re down here. Because you’re going to spot them, and you’re going to air them out.

#present? is defined in Edge Rails as:

class Object
    def present?
        !blank?
    end
end

#blank? is also defined in Rails in Object, String, and NilClass. Hmmm. It didn’t take long to figure out what was happening. Internally, andand uses a BlankSlate class to implement proxies for returning nil or for forwarding a method.

module AndAnd

    class BlankSlate
        instance_methods.each { |m| undef_method m unless m =~ /^__/ }
        def initialize(me)
          @me = me
        end
    end

    class MockReturningMe < BlankSlate
        def method_missing(*args)
            @me
        end
    end

end

When you write nil.andand.present? What happens is this: nil.andand returns MockReturningMe.new(nil), and that object returns nil whenever you send it a method it doesn’t understand. Since it inherits from BlankSlate, we don’t expect it to understand #present?, so we expect it to return nil.

(Before you ask, nil.nil? returns true, but nil.andand.nil? returns nil. That’s intentional, because nil && nil.nil? returns nil, not true.)

So back to the bug. The BlankSlate wipes all of its instance methods out when it is first initialized. But what happens if the #present? method is added to Object after andand initializes its BlankSlate? Ta da! The MockReturningMe class does handle the method, so it never gets to use its method_missing magic.

And as it happens, a MockReturningMe object is not nil, therefore it is not blank, therefore it is present and #present? returns true.

If, that is, #present? is added to Object after AndAnd::BlankSlate is created. If #present? is added to Object before AndAnd::BlankSlate, you get entirely different behaviour. WTF‽

I’m stating facts here, not rendering judgement: This is another example of multiple metaprogramming whatsis doohickeys all gleefully re-plumbing the same core classes and stepping on each other’s toes. Even when they aren’t redefining the same methods.

Reg: It seems you feel our work is not a benefit to the public.

Avdi: Replicants are like any other machine. They’re either a benefit or a hazard. If they’re a benefit, it’s not my problem.

Update: Coderrr pointed out that the version of BlankSlate included in Rails (amongst other gems) fixes the problem by hooking the Object class. There are some subtleties in how this works or does not work, especially when several different pieces of code are all hooking the same events. After discussion with Nathan Weizenbaum, I decided that I certainly didn’t want andand being that invasive, but on the other hand if you have already decided you can live with a BlankSlate class…

So version 1.3.1 works as follows: If you already have a BlankSlate class defined, such as if you have installed Rails and used BlankSlate, or if you explicitly require BlankSlate before you require andand, andand will make use of the existing class and whatever mechanism that class uses to avoid this problem.

If you don’t have a BlankSlate class defined but you do have a BasicObject class (such as from Ruby Facets), andand will use that instead. And if you have neither a BlankSlate nor a BasicObject, andand will roll its own fairly uninvasive BlankSlate that wipes instance methods when it is instantiated (this is a performance hit compared to wiping them when they are created, however no hooks are required).

Pete is now happy. But this really encourages me to redouble my efforts on rewrite. Opening core classes to add certain kinds of functionality is very cool. But I believe it is unsustainable.

Methods like #present?, #andand, and even #nil? really don’t seem to work as polymorphic methods on objects. They need to be deeper. For example, you can define your own class where #nil? return true:

class Nullo
    def nil?
        true
    end
end

But guess what? This makes little sense in Ruby because your Nullo class is still truthy. (And no, you cannot write class Nullo < NilClass.)

I’ve been kvetching a little lately about Ruby not being turtles all the way down. I guess I’m doing that here as well: it is very hard to write things that extend or modify Ruby’s semantics consistently and safely, which i why I’m looking at rewriting.

With rewrite, for example, both #present? and #andand can be expressed as rewrite rules rather than as methods. (#blank? is a little bit more complicated, since you may want the luxury of writing a #blank? method for your own classes.) When you build semantic abstractions our of rewrite rules instead of methods in core classes, you benefit from restricted scope and you benefit from being able to work directly with Ruby’s existing semantics.  ¶ 10:47 PM 10 comments

  Off topic: Antagonyms An antagonym (also called a contranym) is a single word that has meanings that contradict each other.

For example, “Reg Braithwaite buckled his belt and lifted, hoping he wouldn’t buckle under the strain.” The word “buckle” has two entirely opposite meanings.

Antagonyms aren’t nearly as important to the world as finding a way to make our avocation our vocation, but all the same the very idea brings a pleasant smile to my face, and I hope you’re amused as well.

(via Hacker News)  ¶ 12:30 PM 9 comments

  Separating the concern of "what to do" from "how to do it quickly" Over the week-end, I put together a rudimentary rewrite-by-example feature for the rewrite gem (please be patient, I won’t be adding this feature to the gem until it has gone through a few more design iterations). The first thing I tried to do with it was simulating unhygienic macros:

Unhygienic.from {

    __to_receiver.andand.__to_message(__splat_parameters)

}.to {

    lambda { |andand_temp|
      andand_temp.__to_message(__splat_parameters) if andand_temp
    }.call(__to_receiver)

}

What this code produces is a sexp-rewriter that does a global search-and-replace. It looks for code like this:

Person.find(:first, ...).andand.friends(true)

And replaces it inline with:

lambda { |andand_temp|
    andand_temp.friends(true) if andand_temp
}.call(Person.find(:first, ...))

Declarative rewriting by example is a darn sight better than hand-written sexp manipulation:

def process_call(exp)
  exp.shift
  receiver_sexp = exp.first
  if matches_andand_invocation(receiver_sexp)
    exp.shift
    mono_parameter = Rewrite.gensym()
    s(:call, 
      s(:iter, 
        s(:fcall, :lambda), 
        s(:dasgn_curr, mono_parameter), 
        s(:if, 
          s(:call, s(:dvar, mono_parameter), :nil?), 
          s(:nil), 
          begin
            s(:call, 
              s(:dvar, mono_parameter), 
              *(exp.map { |inner| process_inner_expr inner })
            )
          ensure
            exp.clear
          end
        )
      ), 
      :call, 
      s(:array, 
        process_inner_expr(receiver_sexp[1])
      )
    )
  else
    begin
      s(:call,
        *(exp.map { |inner| process_inner_expr inner })
      )
    ensure
      exp.clear
    end
  end
end

But back to rewriting by example:

Unhygienic.from {

    __to_receiver.andand.__to_message(__splat_parameters)

}.to {

    lambda { |andand_temp|
      andand_temp.__to_message(__splat_parameters) if andand_temp
    }.call(__to_receiver)

}

As you can deduce, it is using some magic symbols. In the “from” part of the definition, __to_receiver and __to_message mean “Match something here and name the result receiver and message respectively,” while __splat_parameters means “Match a list of things here and name the result parameters.”

In the “to” part of the definitions, those magic symbols insert whatever was matched in the from. This is a crude approximation of how regular expressions capture things with () and insert them with $1..$n, only there is no way to capture any arbitrary sub-pattern. You can use any names you want, as long as they have the magic prefixes __to_ or __splat_. (Magic symbols are a blight upon all that is right and good with code, suggestions for a better way to express capturing by name gratefully solicited!)

The example above is very much like a Lisp macro. In Lisp, everything is an sexp, so of course macro invocations are sexps just as function calls are sexps. In Ruby, some things look like function calls, some things look like method invocations. In this particular case, “Person.find(:first, …).andand.friends(true)” looks like a method invocation, but it actually behaves like a macro invocation.

Although it looks like a method, the rewriter version of #andand is not polymorphic. You can’t override #andand in any classes, just as you can’t override other syntactic constructions like “&&” or “!”. This bothers OO purists, however I am a member of the OO radical fringe who take purity to another level, a level where overriding functionality is not allowed to violate Liskov Equivalence.

(I personally do not care for the idea that something like #andand sometimes means one thing and sometimes means another, just as many people would completely freak out if !foo didn’t always mean “not foo.”)

beyond macros

Rewriting goes beyond adding new functions and verbs (I personally consider #andand to be an adverb). Basically, what we have here is an extremely weak version of XSLT for Ruby code. Match this, turn it into that. IMO, XSLT transformations is a much better analogy than macro expansion. Rewriters can match and replace fairly arbitrary expressions, not just implement things that look like function calls and method calls.

Consider this hypothetical example:

Unhygienic.from {

    __to_receiver.select { |__to_x| 
        __to_select_body 
    }.inject(__to_seed) { |__to_acc, __to_y| 
        __to_inject_body
    }

}.to {

    __to_receiver.inject(__to_seed) { |__to_acc, __to_x|
        if __to_select_body
            __to_y = __to_x
            __to_inject_body
        else
            __to_acc
        end
    }

}

This would transformation code like this:

heads_of_state = locations.select { |any_loc| 
    zips_in_this_state.include? any_loc.zip_code
}.inject(0) { |heads, loc_in_state|
    heads + loc_in_state.head_count
}

Into this:

heads_of_state = locations.inject({}) { |heads, any_loc|
    if zips_in_this_state.include? any_loc.zip_code
        loc_in_state = any_loc
        heads + loc_in_state.head_count
    else
        heads
    end
}

This is an example of an optimization.

For a large class of expressions chaining a select and an inject (the ones that don’t rely on side effects), this transformation retains the original semantics while rewriting the code to only traverse the collection once and to eliminate the creation of an intermediate collection.

Of course, compilers do this kind of thing all the time for many types of optimizations, so it’s tempting to wait for someone else to write a sufficiently smart compiler that can figure these things out. There are two troubles with waiting for someone else to do it. First, we might be that someone—maybe we’re the one who ought to Just Do It, and waiting for someone else won’t work because nobody else is going to do it.

Second, many problems like this are intractable in the general case. It’s hard (in the mathematical sense) to know when select {…. }.inject { … } can be transformed like this in an imperative language without accidentally stepping on some hidden side-effect. But just because it’s hard in the general case doesn’t mean it isn’t easy in the specific case. For example, you might be the sort of person who never knowingly relies on side effects in select and inject expressions.

So you could use this optimization, while a compiler-level optimization would be a disaster: even if 99.9% of the code out there wouldn’t break, the programmers behind the 0.1% of the broken programs would be furiously blogging about how Ruby wasn’t ready for the Enterprise.

optimizing your code

The joy and the pain of optimizing your code is that you don’t need rewrite to perform that optimization. The joy is that if you discover chaining select and inject is a performance hog somewhere in your code, you simply rewrite the code yourself.

The pain is that the code is no longer in the form you originally decided best represents its intent. In the trivial example I gave above, a rewritten version looks reasonable, especially if you rewrite it with #each in an imperative style:

heads_of_state = 0
locations.each { |any_loc|
    if zips_in_this_state.include? any_loc.zip_code
        heads_of_state += any_loc.head_count
    end
}

Now I’m not going to say that this is necessarily more or less readable than:

heads_of_state = locations.select { |any_loc| 
    zips_in_this_state.include? any_loc.zip_code
}.inject(0) { |heads, loc_in_state|
    heads + loc_in_state.head_count
}

Some people actively dislike using #select and #inject, so they might feel the #each version is better. For once, let’s talk about something other than a bike shed. Let’s focus on the fact that foo.select {…}.inject {…} says one thing: “Filter this collection using this predicate, and then fold the result as follows.” Whereas foo.each {…} says “Iterate over this collection doing the following thing with each element.”

If you wrote this as a #select/#inject pair, you might have a good reason for doing so. Perhaps most of your program is written in a functional style. Perhaps you like to signal that the there are no side effects in those snippets of code and your team share this understanding of how selects and injects are written.

Granting that you believe that #select and #inject do a better job of communicating the code’s intent to your fellow team members, it’s a win to optimize the code (in the compiler or using a rewriter) behind the scenes rather than rewrite it yourself. The code retains its semantics and the form you have decided best expresses its intent, while using less memory and running faster.

separation of concerns

What we have just done with our trivial example is separated two concerns: The concerns of how to best express an algorithm and the concern of how to best implement an algorithm. If for whatever reason—furious hand-waving to avoid arguing how to write loops—we believe that the best way to express a certain algorithm for readability is not the best way to express a certain algorithm for performance, we have two separate concerns: How to write readable code and how to write performant fast code.

Doesn’t it make sense to separate those concerns? So that the code explaining what the algorithm is supposed to do is in one place and the code expressing how to make such things go fast is in another?

This is pure speculation here, but I am conjecturing that being able to rewrite arbitrary snippets of code could be used like a compiler optimization directive. When debugging, you don’t rewrite the code. But when things have stabilized and you need to tweak performance, instead of rewriting the code, you use rewriters to do it for you, separating the concern of “what to do” from the concern of “how to do it quickly.”  ¶ 7:45 AM 7 comments

  Sneak Peek: Unhygienic Rewriting Ruby by Example

def test_unhygienic_andand
  andand = Unhygienic.from {
    __to_receiver.andand.__to_message(__splat_parameters)
  }.to {
    lambda { |andand_temp|
      andand_temp.__to_message(__splat_parameters) if andand_temp
    }.call(__to_receiver)
  }
  assert_equal(
    'Hello' + ' World', 
    with(andand) do
        'Hello'.andand + ' World'
    end
  )
  assert_nil(
    with(andand) do
        nil.andand + ' World'
    end
  )
end

Let’s take a closer look:

Rewrite::With.expand(andand) do
  class Funky
    def hello_world
      'Hello'.andand + ' World'
    end
  end
end

produces:

class Funky
  def hello_world
    lambda { |andand_temp| 
        (andand_temp + " World") if andand_temp 
    }.call("Hello")
  end
end

As you can probably deduce, it’s unhygienic: Variables like andand_temp shadow variables declared elsewhere. This is a problem if you fail to choose a sufficiently obfuscated name or—more likely—nest constructions like #andand.

Both the Classically Metaprogrammed and the Sexp-Rewriting versions of #andand avoid this problem.

Next step: Better hygiene.  ¶ 9:59 PM 0 comments

  Process Theatre You know how Bruce Schneier uses the term Security Theatre to describe measures designed to make us feel safer but not actually safer? I am going to start using the term Process Theatre. I trust you can grasp the meaning immediately.  ¶ 4:00 PM 1 comments

  A blatant exercise in pimping an affiliate link to Art of the Metaobject Protocol

The Art of the Metaobject Protocol is the best book written in computing in ten years.

The bottom line: Languages like Java give you an object protocol: There is one way to do things with objects and classes and interfaces, period. Anything else, like adding generics or annotations, must be done outside of the language, it must be done by the creators of the language.

Languages like Common Lisp and Smalltalk give you a meta object protocol: You can decide for yourself how to do things with objects, classes, interfaces, generic functions, whatever you want. You don’t need to wait for a committee to try something different.

People who believe that software is best created in a Soviet-era Bureaucracy celebrating process and mediocrity—all the while spouting propaganda about celebrating the common worker and the primacy of the proletariat, of course—fear and loathe this idea. They will tell you that standardization trumps innovation, and that the only thing worse than metaprogramming within a standard object system is metaprogramming the system itself.

But we saw how valuing process over people played out, and our minds are actually open to new ideas, even when those ideas are forty years old.

  ¶ 7:42 PM 5 comments