The Optimistic View

I think it’s funny that aspects were invented a while back to shield the masses from the full horror of class level meta-programming, yet with Rails the concepts are becoming commonplace and no one is flinching (yet). It's a measure of progress.

There is a movement advocating throwing our hands in the air and banning all forms of “advanced” programming idioms in the hope that our code bases will be somehow more accessible. Everyone has horror stories about incompetent and downright dangerous programmers who somehow keep their jobs, and there’s an argument that code bases must cater to them.

But maybe, just maybe, most programmers rise to the occasion when challenged. Maybe instead of catering to the bottom 10th percentile we can be open-minded about what the 50th to 75th percentile are capable of achieving, much less the top quarter.

Maybe Rails and Ocaml and Lisp and Factor and Haskell aren’t really advanced or scary or obtuse. Maybe they are a little unfamiliar at first, but no more unfamiliar than SQL or Regular Expressions or XML or JSON or Javascript. Maybe we should put these tools in people’s hands and let them show us what they can accomplish.

all programmers are equal, but some are more equal than others

There’s always someone out there claiming that closures and expressive typing and meta-programming and all of these other “academic” or “hacker culture” tools are elitist, as if they are restricted to the privileged few that speak cryptically of folds and unfolds being duals in category theory, or who argue whether homoiconicity is a requirement for a self-interpreter to be metacircular.

But maybe that’s a prejudice foisted upon us. Personal computers taught us that anyone can operate a computer. Spreadsheets taught us that anyone can program. Maybe we should open our minds to the possibility that although not everyone does program in this manner today, they are capable of it if we stop telling them that they aren’t.

And when we say “Anybody can program,” we say that without reservation or disclaimer. We don’t mean anybody can program provided they are banned from using certain langauges, or anybody can program provided we hide the details of relational databases behind ORMs, or anybody can program provided they are not required to learn any idioms beyond those authorized by the local High Priest. We mean anybody can program, period, without being lashed to a heavy yoke and driven through the fields under the whip.

Maybe—just maybe—the person advocating the non-mainstream tool isn’t the elitist. Maybe they are the egalitarian, the one who believes everyone can program if given the chance. What if the person saying “No, no, that doesn’t scale to a team of one hundred hum-drum monkeys” is the real elitist? Maybe the elitists are the ones who are convinced that other, lesser programmers cannot be trusted to program.

I wrote a parser combinator library to support simple parsing and pattern matching tasks, and I used closures to handle constraints. Unfortunately, both ideas were shot down. Here are a few reasons my tech lead gave to me: Closures are just too hard and too complicated; Nobody will understand it. Nobody will understand that functions can be created, passed around, and destroyed like objects.

Maybe all programmers are not created equally. Maybe some over come early disadvantage to make themselves great. Maybe some squander their gift. Maybe it’s just a matter of sweat and toil and dedication. We really don’t know very much about our brains at all.

But what I personally know is this: I have never met someone who desperately wanted to be great but failed to be at least decent. I have never met someone who applies themselves to the obvious means of self-improvement such as learning a new language every year but is still unfit to create a home page in HTML.

Not everyone is motivated to go against the grain, to study subjects that the vendors say you do not need to know, to gain skills that recruiters say are not in demand, to go against the technical leads in their organization who shoot their ideas down.

Why should they be motivated to swim against the current? Why are the incentives in business programming working against people fulfilling their potential? Are we surprised it is a self-fulfilling prophesy?

Maybe we should just stop assuming that any of this is unusual or special or exceptional. Maybe we should just start assuming it’s normal and everyday and part of the professional toolkit and just see what happens.

Maybe we should just try stuff.

We might get a very pleasant surprise.

---

This post was inspired by Think Big.  ¶ 8:26 PM 14 comments

  The Vast and Endless Sea

If you want to build a ship, don’t drum up the men to gather wood, divide the work and give orders. Instead, teach them to yearn for the vast and endless sea.

This is all I am trying to accomplish with my blog. When I recount stories from my experience, when I giggle with delight at a new discovery, when I describe a different way to solve problems… All I am trying to do is share my own yearnings to sail over the horizon and see what there is to discover.

And de Saint-Exupery puts it best when he describes the sea as vast and endless. It is truly about the journey, not the destination. It is fascinating precisely because there is no answer, no best way, no way to decide “we have arrived, we are done.” I know with certainty that when I end my journey, when I go to my rest, there will still be unsolved problems of software development.

For me, software development is an activity carried out in our minds. Our “sea” is the human brain. All of our tools and processes are ways of engineering thought and communication. The software we build in turn is a way of guiding thought and communication.

I do not say that there is no Engineering in software development. But I can say with confidence that there is room for considering it a form of Applied Psychology, or perhaps a specialized branch of Cognitive Science. And what greater journey is there than the exploration of how we think, of how we solve problems, of how we communicate and coördinate our activities?

Certainly there are practical considerations. Ships do need wood. Sailors must work with a purpose. Software must get built, and it must validate phone numbers or ensure that transactions conform to ACID semantics.

But yet, the allure of our work is the sea and the mysteries it contains. Thank you for joining me on the voyage.  ¶ 12:22 PM 9 comments

  Fun with Symbol#to_proc

If you want a golden rule that will fit everything, this is it: Have nothing in your houses that you do not know to be useful or believe to be beautiful.

Most people love Symbol#to_proc sugar in Ruby:

%w[dsf fgdg fg].map(&:capitalize)
    → ["Dsf", "Fgdg", "Fg"]

And Googling around, all of the examples I could find were for unary methods like this. But what about binary methods like :merge? If Symbol#to_proc supported binary methods, you could do things like:

(1..5).inject &:+
    → 15

Or:

[{ :foo => 1 }, { :bar => 2 }, { :blitz => 3 }].inject &:merge
    → {:foo=>1, :bar=>2, :blitz=>3}

That would be brilliant.¹ And the implementation can’t be that hard in a language like Ruby, can it? Let’s have a look at the source code (I don’t have the Ruby 1.9 source, but I do have Symbol.rb from the Rails Core Extensions):

class Symbol
  # Turns the symbol into a simple proc, which is especially useful for enumerations. Examples:
  #
  #   # The same as people.collect { |p| p.name }
  #   people.collect(&:name)
  #
  #   # The same as people.select { |p| p.manager? }.collect { |p| p.salary }
  #   people.select(&:manager?).collect(&:salary)
  def to_proc
    Proc.new { |*args| args.shift.__send__(self, *args) }
  end
end

Would you look at that, Symbol#to_proc already handles methods with arguments. And indeed, the examples above work right out of the box in Rails. I expect them to work in Ruby 1.9 as well.

You still can’t do some of String#to_proc’s other point-free tricks like collection.map(&'*2'), but quite frankly this is a really nice feature that deserves more publicity. So… don’t forget that Symbol#to_proc handles your unary and binary methods.

Cheers!

1. I know what you’re thinking, but try to remember that the other meaning of the word “brilliant” is shiny and sparkling.

p.s. You’re wondering where I get the time to think up stuff like this? Actually, I’m lucky enough use stuff like this for a living. I needed to fold a collection of hashes together, and my first thought was that I’d like to write .inject(&:merge) to do it. This is the principle of least astonishment in action…

Labels: ruby

  ¶ 8:52 PM 3 comments

  Can expressive languages make code more accessible to the masses?

IT is no-longer a hackers paradise populated only by people with a background in Computer Science, it is a discipline for the masses and as such the IT technologies and standards that we adopt should recognise that stopping the majority doing something stupid is the goal.

An interesting read. The characterization of “dynamic languages == hackers doing programming as art” does not match my experience talking to people who use dynamic languages in commercial projects, so I’ll respond to that one point.

This quote shouldn’t be taken as the entire premise of his post: please read the whole thing and decide for yourself what he is and isn’t saying. And my response here isn’t really an argument with Mr. Jones, it’s more of a riff inspired by it. I’m going to talk about different ways to achieve a similar goal. Perhaps that is disagreement. Or perhaps it is violent agreement.

doing no wrong

One of the many things people do with languages like Python and Ruby is try to solve the exact problem he mentions, stopping someone from accidentally doing something wrong. Obviously, not with static typing. So how else can this be done?

Let’s start with Python. What, do you suppose, is with the enforced indentation. Do you think it might have something to do with standardizing code style to make it readable? Which in turn lowers the bar for someone to come in and understand code and therefore fix it more easily and without accidentally breaking everything? Although this has nothing to do with Python’s “dynamic” nature, it has everything to do with the language’s design goals.

And I am very sorry that type inference is not a common feature in “popular” statically typed languages. One of the reasons people like dynamic languages is making the boilerplate go away. And agreed, the baby goes out with the bathwater. But it’s more than just saving yourself typing, especially for someone like myself who spends more time thinking about what I’m going to write than typing it. Even if it takes me twice as long to type my code, I don’t become half as productive, not by a long shot.

The point is also readability, and readability directly drives application support. Code that is easier to read in the aggregate is easier to maintain. And dynamic languages can be used to make code very easy to read.

Put in this light, the “dynamic” nature of certain languages is not the goal, but rather a means to an end, or even an unhappy side-effect of the goal, which is making code more expressive and therefore readable.

For example Regular Expressions could be considered an artful way to play “golf,” to reduce code from pages of loops to perform searches down to a few characters with some cryptic special features—[^a-z]* and so forth—but they could also be seen as a way to reduce errors by abstracting something that is error-prone and difficult for “the majority” to get right the first time, much less maintain when someone else has re-invented the text search wheel in their own idiosyncratic way.

Likewise, Ruby on Rails—which is a framework and includes several DSLs—could be DHH futzing about writing poetry in Ruby. Or it could be a way of stopping the majority from doing stupid things like getting relational databases wrong. reducing relations to conventional table and columns names plus readable “declarations” like “has_many :ideas” also serves to make it easier for the non-expert to write and maintain code.

This is not to say that bondage and discipline doesn’t have its place. Or that you can’t achieve all of the same goals with Java and stop someone from accidentally calling .to_s on an object which actually responds to .toString. But I am definitely saying that writing code for others to maintain is both a laudable goal and one of the things that the current crop of dynamic languages can do well.

In the end, my suggestion to everyone is simply this: Programming is Art—to a programmer who is also an artist. And Programming is Engineering—to a programmer who is also an engineer. And finally, Programming is “A Means to Organize and Mold the Work of The Majority”—to a programmer who seeks to organize and mold the work of the majority.

In the hands of a programmer seeking these goals, each language has its own advantages and disadvantages, no doubt. And I will not argue with suggestions that a language with veryLongCamelCaseConventions using WidgetTemplateFactoryMethodFactoryFacadeFactoryVisitors is the preferred choice for certain projects in certain environments.

But I am pointing out that if you begin with the same ends in mind, such as writing code to be maintained by others, you can use radically different tools in radically different ways, yet wind up reaching the same objective.

---

p.s. A good read from Steve Jones: What is it with vendors’ view on time?  ¶ 9:38 PM 6 comments

  How I've been spending my time off work lately The weather being fine, and in the flagrant pursuit of happiness, I’ve managed to get a few holes in this week (using Ruby of course). Here’s my scorecard:

 palindrome =  (+(+[] | +[-:_] | ((-'[0]' >> -:ends) & (-'[1..-2]' >> -:p) 
               & (-'[-1]' >> -:ends) & -:_)) >> (-:'{}' >> -:p)).call
(palindrome =~ []).should_not be_nil
(palindrome =~ [:fish]).should_not be_nil
(palindrome =~ [:fish, :dog]).should be_nil
(palindrome =~ [:fish, :dog, :fish]).should_not be_nil
(palindrome =~ [:fish, :fish, :fish, :fish]).should_not be_nil
(palindrome =~ [:fish, :fish, :fish, :fish, :fish]).should_not be_nil
(palindrome =~ [:fish, :dog, :dog, :fish]).should_not be_nil
(palindrome =~ [:fish, :dog, :cat, :dog, :fish]).should_not be_nil
(palindrome =~ [:fish, :dog, :cat, :fish]).should be_nil

Perhaps Guido has a point, indentation matters:

palindrome = (
    +(                            # we'll start by creating a pattern
        +[]    |                  # match the empty list, or:
        +[-:_] |                  # match a list containing anything
                                  # (-:_ matches any one entity), or:
        (                         # alternative three:
            (-'[0]' >> -:ends)  & # extract the first element and
                                  # bind it to 'ends'
            (-'[1..-2]' >> -:p) & # and everything from the second to the
                                  # second-last will be matched against a
                                  # pattern :p
            (-'[-1]' >> -:ends) & # and the last element must match whatever
                                  # was bound to 'ends' above.
                                  # it's equally effective to include
                                  # -'_[0].eql?(_[1])', same thing.
            -:_                   # if all that matches, match the entire array
                                  # so that the result is the entire array, not
                                  # whatever part of the array we were just
                                  # matching
        )                         # therein ends the pattern
    ) >> (-:'{}' >> -:p)          # pass the pattern to a closure-maker (-:'{}')
                                  # and then bind it to 'p' so that the pattern
                                  # is now recursive: this pattern matches an
                                  # empty list, a list with one element, or
                                  # a list with the first and last elements eql?
                                  # to each other and whatever is in-between them
                                  # matching the palindrome pattern bound to 'p'.
).call

Hmmm. Still doesn’t make any sense. I guess home-brew programs are like babies: they appear to be delightful and impart a sense of wonder in the mysteries of the universe to their creators, but are noisy and obstreperous to everyone else.

f = (
    +(
      (-'==0' >> -'0') | 
      (-'==1' >> -'1') |
      (-'n>1' >> -'n * f[n-1]') ) >> (-:'{}' >> -:f)).call
f[0].should == 0
f[1].should == 1
f[2].should == 2
f[3].should == 6
f[4].should == 24
f[5].should == 120

Labels: ruby

  ¶ 3:42 PM 4 comments

  What if powerful languages and idioms only work for small teams? What indeed? What if programming conventions and closures and meta-programming and expressive type systems and annotations and all of the other tools that give us the power to build powerful abstractions actually don’t scale to larger teams?

Two ranchers, a Canadian and an American, meet at a Cattle Convention. The American boasts “My ranch is so big, I can git’ in my truck and drive for a day in any direction, and I’m still on ma’ propuh-ty.”

The Canadian nods in sympathy. “I used to have a truck just like that.”

So what?

Perhaps we have somehow confused the cart with the horse. Teams exist to deliver working software. So, shouldn’t we choose the tools and processes proven to produce working software and build the team around them, rather than choosing the team and then dumbing the tools down to the level of whomever we hired last Tuesday?

All of our experience in the last sixty years has suggested that productivity drops off a cliff as team size increases. So, if you want more code from a larger team, you have to invest heavily in ways of extracting value out of unproductive people in an unproductive environment.

But what about the other direction? If everything we know about teams suggests that smaller teams have the greatest productivity, isn’t it a valid choice to invest heavily in tools and processes that are specifically tuned to maximizing the productivity of small teams?

“Slick Willie” Sutton is famous for stating the obvious. When asked why he robbed banks, he is alleged to have replied, “Because that’s where the money is.” If we know that smaller teams are more productive, why do we try to scale teams out in personnel rather than up in productivity? If we know that bug per line of code remains amazingly constant, why do we try to scale code out in verbosity rather than up in abstraction?

Some people obtain results they consider acceptable from very large teams, using B&D tools and processes. But there is another option, to go “where the money is,” to find ways to enlarge the productivity sweet spot rather than live without it.

---

Elsewhere:

I argue the flip side of the coin: Can expressive languages make code more accessible to the masses?

And two great perspectives from other bloggers: Why Small Software Teams Grow Large And Other Software Development Conundrums and Create software, not code.  ¶ 9:59 PM 19 comments

  For a moment, I thought I was reading _why the lucky stiff

Think of it this way. Items in an array are dogs. Arrays are people
holding leashes. Anyone can attach a leash to a dog. So I (tempArr) can
have a leash on Fido, and so can you (one). If you let go of your leash
(one.clear), Fido is still Fido; you just don't have a leash on him. But
if you cut off one Fido's legs (modify one[0]), that leg on my Fido
(tempArr[0]) is also cut off, because they are the same Fido.

  ¶ 10:50 AM 1 comments

  I brag about being open-minded, so... …I really ought to have no objection to proving it by linking to something that disagrees with my words and paints me as narrow-minded and describes me as ignoring some people who are obviously fairly bright, and…

I’m sure you get the idea. There’re good points in this post. But I have a clear conflict of interest: it characterizes me in a way I find unflattering, yet it discusses issues I find very interesting and well worth saying “hmmm…”

All static typing—and unit testing, for that matter—do is help ensure the program does when I mean. That’s all. But that’s important, because the more time we the programmers have to spend making sure the program does what we mean, the less time we have to spend on figuring out what it is we want the computer to do. What means we’re more likely to get to the “does what we want (and what we mean)” nirvana, and more likely to get there faster.

  ¶ 12:38 PM 4 comments

  Off Topic: Please excuse my reddened eyes There’s a nasty thing going around the weblogs. I think the disease made the jump from online forums. And the forums caught it when it mutated from something infecting UseNet. Raganwald was just exposed to the pathogen, but hopefully twenty-four hours of rest will set things right.

The symptoms are this: Someone, we’ll call them Person X, has been involved in heated debates and flame-fests over something-or-other, we’ll call it Topic Y. They are at the stage of taking it very personally. With every exposure to debates over Topic Y, they become more and more emotional. They also slide from seeing the world in colours to shades of grey to stark contrast between black and white. People are either with them or against them, no middle ground, no non-combatants.

Then you say or write something touching on Topic Y that fails to establish yourself clearly and directly as agreeing fully with Person X.

Person X flies into a rage. They read your words, combing over them for “inferences” and “suggestions.” They don’t read your words, they read what they think you meant, assuming that you are in fact one of Person X’s sworn enemies. They don’t read your remarks about Topic Y as parenthetical, or even uninformed. No, they read them as a deliberate attack on their views, and by extension, as an attack on Person X.

In fact, the less direct your remarks are, the less you openly disagree with them, the more they read your actions as being sly or manipulative. They see Topic Y as a battlefield, and everyone on the battlefield as combatants with an agenda.

And they attack you with everything they’ve got. Or rather, they attack whatever fiction they constructed to make you into the enemy. Even though, quite honestly, you have no axe to grind and no interest in sparring with them.

It’s unfortunate, because it is so divisive. These attacks come from a place of people trying to hurt other people, to insult them, to make them feel bad about themselves. There is no interest in furthering our profession and sharing our knowledge.

This saddens me greatly. I feel sorrow when a complete stranger brings unhappiness and vituperation from somewhere else and deposits it here.

And like another thing that saddened me, I have no suggestions. I can’t even really say it is a problem, other than that it makes me feel sad when it happens.

And now, dear reader, I will set this down by the river. Thank you for listening.  ¶ 8:15 PM 8 comments

  The best programming tip you'll read today! Twenty-five plus years of experience. More than three million people using my software. More than fifty million dollars of created value. And it comes down to this:

In Ruby, never type “rescue” when you really meant to type “ensure.”

No need to say it, I know, I know. You’re welcome.  ¶ 1:19 PM 5 comments

  The best question asked this week: What, exactly, are you trying to prove?

To prove the correctness of code, you need to know exactly what correct behavior is for the code, i.e. a complete and unambiguous specification for what the code should do. So tell me dear reader, when was the last time you received an unambiguous fully detailed specification of an application?

If I ever received such a thing, I would simply execute that sucker, because the only unambiguous complete spec for what an application does is code. Even then, you have to ask, how do you prove that the specification does what the customers want?

Proving that the software meets the rigorous specification written by the customer before they had seen the software work, before they had tried it, has been of secondary importance to me in my career. The overarching concern has been the transformation of the requirements from what the customer guessed they might need, based on superficial desires and buzzwords, into what the customer discovered was actually important, over the lifetime of the project.

No, no, no! You don’t know what you want!

Hey, I am not blaming customers. They don’t know what they want because nobody really knows what the optimum finished software looks like before development begins. Not the Customer, not the Developers, and not even the “Architects,” “Product Managers,” “Business Analysts” or anybody else who think they are a requirements expert.

Software development is a process of discovery, for the customer, for management, and for development. As information is acquired—through inquiry, inspection, or experience gained from misadventure—our understanding of what we are trying to accomplish is refined. In that respect, every tool, be it a programming language, a development process, or a testing practice, must be judged for its contribution to the transformation of both the requirements and the software over time.

I am not saying that strong typing or provability, iterative development or BDD, or anything else is necessarily better or worse in this respect. I am just pointing out that the surface, buzzwords for these things—fewer bugs, provability, test coverage, separation of concerns—are not the goal, they are ways and means of achieving the goal, which is the discovery of the true requirements and the delivery of working software which meets the true requirements over the project’s complete span.  ¶ 1:34 PM 15 comments

  Until now, Microsoft could sell code better than anyone

Until now, Microsoft could sell code better than anyone, but it seems the company would rather sell services: software as a service, ads, search engine results—you name it.

This is like the local storefront that opens as a knife-sharpening business and is soon selling junk jewellery, moose heads, toaster repair, and cheap chocolate. In the meantime, the knife-sharpening business goes by the wayside.

This is what has happened to Microsoft, and Vista is the result.

  ¶ 9:44 AM 8 comments

  Diversity Here’s a picture snarfed from Tim Bray’s excellent blog, ongoing. It’s a bunch of people playing a game at a recent RubyConf:



Notice the diversity? Me neither. And you know I’m not talking about the paucity of .NET aficionados in the room.

Of course, there’s a long-standing and futile argument we can re-hash here. Something-something enrolment in CS programs at University. Something-something birds of a feather. Something-something everyone over a certain age is taking care of their children, not traipsing off to conferences on weekends.

I don’t really have a suggestion here. I can’t say with certainty that there is a problem here. But all the same… I confess that this picture saddens me.  ¶ 2:30 PM 48 comments

  Programming conventions as signals Some time ago, I spent a great deal of my waking and sleeping hours thinking about Contract Bridge. I especially thought about bidding systems. Bidding in bridge is a very hard problem: you are trying to coöperatively seek a maximal payoff contract using an incredibly limited vocabulary that is a scarce resource: the pool of legal bids shrinks with each bid. And that’s without considering the fact that two opponents are competing for the same resource to try to seek your minimal payoff and frustrate your attempts to communicate.

Anyhow.

Given the limited information available, it is critical in bridge to use every possible bidding sequence productively. Therefore, if there are two ways to say the same thing, bidding systems are designed such that there is a clear understanding that one of the two ways means something subtly different than the other. Although each bid means exactly the same literal thing—Five Diamonds always means eleven tricks with Diamonds as trumps—players can draw elaborate inferences of what a player bidding Five Diamonds holds based on the sequence of bids leading up to that moment.

If you and your partner play that a 1NT opening shows a balanced hand with 15-17 high card points, what do you make of it when your partner opens One Diamond and rebids No Trumps later? A balanced hand, of course, but you know she doesn’t have 15-17 high card points, because she would have opened 1NT if she did. And if you play five card majors, you know she doesn’t have five or more Hearts or Spades.

So for programming languages, I believe the same thing. When there is more than one way to do it, don’t randomly choose which way to do it based on the phase of the moon. Don’t straightjacket yourselves by appointing some martinet to decide which way to do it on each project. Instead, use the different idioms as signals, as ways to provide additional information to programmers.

Signaling with syntax

Consider blocks in Ruby. Some people use do … end when the block needs multiple lines and { … } when it fits on one line. A popular (AFAIK) and superior idea is to use do..end and {…} to disambiguate between blocks that are executed chiefly for their side effects and blocks that are executed for their return values:

foo.map { |x|
    # I care about the result
}

foo.each do |x|
    # I care about the side effects
end

The computer doesn’t care, of course, but it signals an extra piece of information, the fact that you are chiefly interested in side effects in one case and chiefly interested in the result in another case.

In Java, the final keyword is a gold mine of signals, especially for variable declarations. If you mandate that every parameter and every local should be declared final unless you plan to modify it, this is like having a bidding system where a 1NT opening means 15-17 points and a balanced hand. The moment you see a variable that isn’t final, you know that the code is expected to mutate it, just as bridge players know that a rebid of 1NT means that their partner doesn’t have 15-17 points and a balanced hand.

TIMTOWTDI

If you look at any language where there is more than one way to do it, you will trip over opportunities like this to use the code itself to communicate. In Ruby, if you want to sort an array of something, there are two ways to indicate the sort order. First, you can define the Boat Operator (<=>) for the values being sorted. Second, you can provide a block to the :sort method telling it how to order the values.

Which is better?

My preference is that when a value has a natural, default sort order, it’s best to define the Boat Operator for it (and brave all of the challenges inherent in using methods for object comparisons). When you see a sort without a block, you know that the values are being sorted in a natural, obvious order.

Then, if you see a sort with a block, you are immediately alerted to the fact that this sort is an exception to whatever obvious, natural order exists for that object. The computer doesn’t care, but it makes the code that extra bit easier to read: it makes the exceptional cases… exceptional.

class Value
  def <=> other
    # ...
  end
end

values.sort # uses the <=> operator: I want the natural order

values.sort { |a,b| ... } # eschews the boat: this block MUST sort by something unusual

Multi-paradigm programming

There is a special case of the idea that “When there is more than one way to do it, each way should be used differently.” Some languages, such as Ruby, are deliberately multi-paradigm: not only are there several ways to do things, there are several entirely different philosophies the language supports. In Ruby”s case, for example, you find the usual assortment of imperative structured programming suspects like for, while, and until. But you also find some of the more obvious functional programming suspects, like lambdas as first-class values and Enumerable’s collection methods :map, :select, and :detect.

I think the idea that doing the same thing in different ways signals different intents scales up to the paradigms you use. If you have a list of phone numbers (say for people you'd like to call), and you wish to remove certain numbers from it (say a do-not-call list), in Ruby I think you should always use some kind of functional approach like combining :reject and :include or :detect. If you write this out using for loops or :each, it signals that there are side effects going on.

By making functional things look functional, you clearly signal that you are performing a calculation strictly for the result. Using imperative syntax like for and while should be reserved for the times you need to mutate variables or cause other side effects. Naturally, you may need to break this so-called rule from time to time for performance purposes. But you’re an adult, you know that just because a rule needs to be broken here and there for pragmatic reasons doesn’t mean you toss it out and write all of your code using whatever keyword is laying about at the time.

Likewise, OO is a great paradigm—when you are modelling entities in the real world or when you need long-term persistent state. Languages like Ruby force you to use objects behind the scenes, but you code shouldn’t look OO unless you are trying to signal your colleagues that there are entities involved. For this reason, you should use Proc.new when you want objects and lambda when you want functions.

The benefits of traveling far while you build a life at home

People sometimes ask why they should bother to learn other languages or other programming styles, when they do not have an opportunity to use those languages at their 9-5. I believe that if you embrace the idea of multi-paradigm programming, these other languages can teach you useful techniques such as using :unfold to model iteration over data structures.

Of course, people will ask why you are using a Haskell idiom in Ruby, or a Lisp idiom in Java. One of the answers is that it may be more expressive. Another answer could be that by making functional things look functional, by making OO things look OO, by making distributed things look distributed, and so on, by borrowing paradigms from languages where those ideas are fully exploited, you can make your Ruby and Java code signal its intent more clearly.

Say what you mean and mean what you say

In the end, this is a really simple idea: when you have several different ways to do something, your first choice should be the way that signals what you are trying to accomplish in a natural and obvious way. That may mean borrowing an idiom from a language that expresses that idea more succintly or more directly.

And when you eschew the natural and obvious idiom, it should be to signal that you have a different intent, that your exception to the standard idiom reflects the code’s exceptional nature.  ¶ 5:38 PM 16 comments

  Whoa A self-interpreter. For Javascript.

Whoa.  ¶ 10:20 AM 2 comments

  Thank you I don’t know your name. Or your age. Or what you look like. You are young, your whole life is ahead of you. Or it would have been ahead of you, but your life was cut short. I don’t know exactly how. A bullet, most likely. Perhaps an explosion. Maybe you were killed instantly. But quite possibly, it took far too long, in pain and agony and mud and squalor, with no decent medical attention to save you or to ease the discomfort. You may have died alone.

You are not famous. Nobody but your friends and family missed you. There was the telegram, the sorrow, the hurt that was repeated day after day in your hometown. So many like you shared your fate, our country was numbed to it. But although time heals our wounds, we do not need to forget them.

And I will always remember you. I will always honour what you did, what you faced, and the finality of your sacrifice.

I am the product of a cross-social marriage, between a Black woman and a White man. My great-grandfather came to this country from Barbados, the descendant of slaves. He worked himself into a position of responsibility and sent his children to school. One of them sat in our parliament and practised law. A Jewish man loaned him the money he used to build two houses for his family. Today, my colleagues, clients, and the people who use my work are European, African, South-East Asian, Russian, and every other culture or country you can name. They practice every religion you can think of. Alongside each other. We travel across borders, we are children of a small village in today’s world.

It might not have been that way, had you not heeded the call when it came. I have no words to describe how much my life owes to your bitterly short life. I could never repay you for what you have done, for what it means to my family before me and to my children after me.

On this day, more than any other day, you are in my thoughts and my heart.

Thank you.  ¶ 11:24 AM 5 comments

  Really useful anamorphisms in Ruby Really simple anamorphisms in Ruby introduced a very simple unfold. Its chief characteristics were that it generated an Array from a value of some sort, and it did so by applying an incrementor block to its seed recursively until it generated nil. For example:

10.class.unfold(&:superclass)
    => [Fixnum, Integer, Numeric, Object]

A very simple modification allows us to separate the two blocks with a :while or :to pseudo-keyword, and to add a :map keyword for transforming the state into the desired result. Thus, this really simple unfold:

1.unfold(&'_+1 unless _==10').map(&'**2')
    => [1, 4, 9, 16, 25, 36, 49, 64, 81, 100]

Can also be expressed as:

1.unfold(:to => '==10', :map => '**2', &'_+1')
    => [1, 4, 9, 16, 25, 36, 49, 64, 81, 100]

The latter form is helpful once unfolds become larger and more complex than these simple one-liners.

(There is another style of writing :unfold, using method chaining and lazy evaluation to eliminate lambda keywords, but we will save that for another time: it is a great examination of syntax but does not change :unfold’s fundamental behaviour.)

Let’s turn it up a notch

These trivial examples are not particularly compelling. Unfold is touted as the complement to :inject. So you would expect :unfold to be as useful as :inject. And :inject is very, very useful—you “reduce” lists of things to values all the time.

But how often do you need to turn a value into a list? How often do you need to turn ‘10’ into ‘[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]’? And if you do, what’s wrong with using (1..10).map(&'**2')?

Remember that :unfold can be applied to objects with a lot more information to them. The thing that had me stuck when I first saw :unfold was thinking of it as the opposite of :inject. Or at least, the opposite of how I used :inject. I tended to use :inject in a way that reduced information. For example:

[7, 6, 10, 3, 9, 4, 8, 5, 2, 1].inject(&'+')

This gives us the sum of the numbers from one to ten, as it happens. It also gives us a value that is considerably simpler than the list we used to generate the number. Information is lost when we use :inject to “reduce” a list to a very simple value. So my first reaction to :unfold was to think of ways to use :unfold on very simple values, like numerics.

But :unfold doesn’t have to work with simple values. It can work with arbitraily complex data structures. Consider:

def zip(*lists)
  lists.unfold(
      :while => '.first', 
      :map => '.map(&".first")',
      &'_.reject(&".length < 2").map(&"[1..-1]")')
end

Zip is a function that takes two (or more, but let’s just say two for now) lists, and produces a list of pairs of items. So:

zip([:a, :b, :c], [1, 2, 3])
    => [[:a, 1], [:b, 2], [:c, 3]]

How does :unfold do it? First, of course, it makes a single list of lists. It then performs an unfold on this single data structure. The incrementor successively reduces each sublist by removing the first items. So the output of the successive incrementor operations is:

[
    [[:a, :b, :c], [1, 2, 3]],
    [[b, :c],      [2, 3]]
    [[:c],         [3]]
]

The :map then extracts the first items from each sublist and presents them as a list:

[
    [:a,           1], 
    [:b,           2], 
    [:c,           3]
]

Neat. But why do we care about zip? Well, if you’ll notice, we already have a bunch of really useful things we can do with lists, like :map, :select, :reject, :detect, and so on. What would you do if you had two lists and needed to do something with each pair in the list, like… A list of first names and surnames that need to be catenated together?

zip(first_names, surnames).map(&'"#{_[0]} #{_[1]}"')

Zip is useful when we have a bunch of parallel lists and there’s something we want to do with each tuple from the lists.

Generalized iteration

We recognize this “pattern,” it’s one of the most powerful in programming. Zip was one algorithm, a way of iterating over several lists simultaneously. The other algorithm was "#{_[0]} #{_[1]}", a recipe for what to do with the successive tuples of values.

The powerful idea was to separate the mechanics of turning a data structure into a linear series of values—iterating—from what we want to actually do with each value. (In OO-style programming, we would define a method for lists of lists that returns an iterator over the tuples of values. Same thing, proving that how you do it is not as important as understanding why you do it.)

Unfold has other uses, but this one alone is worth the trouble to understand the pattern even if you aren’t rushing to implement this exact unfold method: Converting a single data structure to a list is one way to implement iteration: for any data structure, you can use unfold to define a linear iteration. You can then use :each or :map or :inject just as our parents before us would have used DO or FOR loops.

Consider this (inelegant, but I’m writing this rather late at night) unfold:

[[1, 2, 3], [4, 5, 6, [7, 8]], 9, 10].unfold(
  :while => '.first',
  :map => lambda { |first|
    first = first.first while first.kind_of?(Array)
    first
  }
) { |state|
  state = state.first + state[1..-1] while state.first.kind_of?(Array)
  state[1..-1]
}
    => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

This is the same idea, only we convert a tree into a list representing a depth-first search of a simple tree. You may recognize it as Array’s :flatten method. Once again, it’s really a way of iterating over the elements of a tree. So one way to think of this :unfold is that it is an iterator over a tree’s leaves:

def flatten(arr)
  arr.unfold(
    :while => '.first',
    :map => lambda { |first|
      first = first.first while first.kind_of?(Array)
      first
    }
  ) { |state|
    state = state.first + state[1..-1] while state.first.kind_of?(Array)
    state[1..-1]
  }
end

But I already know how to write Zip and Flatten methods, honest I do.

Zip and Flatten are relatively common, that’s why :flatten and :zip can both be found in Ruby’s standard Array class. And if there’s a data structure that needs regular unfolding, you ought to weigh the advantages and disadvantages of writing an :unfold for it or using more humdrum ways of writing an iterator.

However, what do you do when you only need to unfold something once? For example, perhaps you have code that obtains some data in JSON format, and having used a library to parse the JSON into a one-off list or hash, you want to iterate through it.

With unfold, you can write your one-time, specific iterator right in place. This is no different than using blocks and lambdas in Ruby for one-off functions that really don’t need the cermony and weight of being implemented as methods.

When you want to iterate through something, and you want to separate the mechanism for iterating through the data from what you do with the data, :unfold should be in your tool box.

Unfold and the bio-sciences. Not really.

I like to think of :unfold like unfolding a protein molecule. When you stare at a data structure, it’s dense, opaque. But you supply an unfold algorithm, and what looked like a messy ball of twine unravels into a long filament made up of simple elements. You can then operate on the simple elements, without getting what you want to do en-snarled in how you iterate over the data structure.

So there you have it. Unfold can be really useful if we see it as a standardized way to write iterators for data structures.

---

Update: The under-appreciated unfold.

Labels: lispy, popular, ruby

  ¶ 10:46 PM 13 comments

  Really simple anamorphisms in Ruby Anamorphisms are functions that map from some object to a more complex structure containing the type of the object. They are the dual—a fancy word for complement—of Catamorphisms, functions that map from some complex structure down to a simpler object.

In simpler terms, a Catamorphism is a function like inject: In Ruby, Inject takes a collection and produces something simpler. It is also called fold or reduce in other languages. Inject can do something like produce the sum of a list:

(1..5).inject &'+' => 15

Unfold does the reverse: it takes a single value and turns it into a collection. Now, a proper unfold can be configured with a seed value, a transformation, a stopping predicate, maybe a distinction between states and output values, and even the type of structure you want to create. A proper unfold would even work with lazy lists if it didn’t have a stopping condition.

But sometimes you want something really simple. The state and the output value could be the same thing, eliminating a transformation from state to output. The stopping predicate could simple, it could stop when it reaches nil. And it could always return Arrays. So you could use such a simple unfold whenever you have a seed value and some sort of function (expressed as a block) that returns nil when it has no more values.

For example:

10.unfold { |n| n-1 unless n == 1 }.inspect => [10, 9, 8, 7, 6, 5, 4, 3, 2, 1]
10.class.unfold(&:superclass).inspect => [Fixnum, Integer, Numeric, Object]

Hey, what happens if you combine really simple anamorphisms with really simple catamorphisms?

5.unfold(&'_-1 unless _==1').inject(&'*') => 120

Here is the code unfold.rb. (If overloading core classes with too much responsibility is not to your taste, it is trivial to express unfold as a function taking a seed and a block as arguments).

Now that I’ve whet your appetite, here’re Really useful anamorphisms in Ruby. And for another implementation, everything you ever wanted to know about implementing a true unfold in Ruby.

Enjoy!

(Code courtesy of Mobile Commons. Thanks!)


Here’s an interesting email from Hugh Sasse:

I was reading Code Complete 2 last night and Steve McConnell says that he wouldn’t hire a programmer who wrote a recursive factorial function. [I’m not sure that would be such a crime in languages like Lua with proper tail recursion, but still. (I’m probably wrong, now I’ve written that! “Open mouth, insert foot, echo internationally” as they used to say on Fidonet :-))] I thought “that seems a bit harsh”, especially since in real life they’d get one from a library most of the time, anyway.

So, as I have often run out of stack on Ruby, I’ve tried to rewrite unfold as an iterative function. It seems to be working. The script as modified blows up at 5000 on my system with the recursive version, but the iterative version succeeds at 5000.

Thanks for this blog entry. There’s something REALLY nice about this idea, which I can’t put my finger on. Might be something to do with loops terminating when they should, and The Pragmatic Programmers’ article about “cook until done”

http://www.pragprog.com/articles/cook-until-done

Unfold feels like a “do until finished” loop.

class Object
  # As above, but iterative, rather than recursive.
  def unfold2 &block
    result = [self]
    x = block.call(self)
    while not x.nil?
      result.push x
      x = block.call(x)
    end
    return result
  end
end

Labels: lispy, popular, ruby

  ¶ 5:34 PM 0 comments

  Off topic: Advice Most of you can shrug your shoulders at the following piece of advice. If, for some reason, it strikes you as novel, please pay close attention. And after thinking things through, if it doesn’t seem to be a nearly infallible rule, a dictum to follow 99% of the time, please, please, please talk to your closest friends and family, people who care about your well-being, before you deliberately go against what I’m about to say:

Do not publish the content of private—as in one-on-one—conversations, meetings, IMs, or emails—without consent in advance from the other party involved. It will mark you forever as someone who cannot be trusted.

You may think “Oh I don’t care if I make that person angry, I don’t like them.” However, you are not just burning your bridge with that person, you are wearing a sign around your neck warning everyone else who meets you that they run the serious risk of having their laundry—dirty or clean—aired out in your memoirs, at your local pub, or worst of all on your blog.

Do not publish the content of private conversations, meetings, IMs, or emails without consent in advance from the other party involved. It will mark you forever as someone who cannot be trusted.

I saw a variation of this malfeasance recently. Giles Bowkett wrote something that—for some reason—inflamed and outraged people. I will say immediately that although I didn’t agree with the literal premise of his post, I found it interesting, and hardly on the scale of an insult against my family. I did not rush to my keyboard to accuse him of heresies and poll my readership for the most appropriate wood to use for his immolation.

However, some people jumped into the flame storm with both feet. Fine. One such person had worked at the same company as Giles for some period of time, and saw fit to air his opinion of Giles’ talents on a public site, describing workplace incidents in the process. Were I the critic’s employer, I would ask that person to step into the conference room, bearing with them their employment contract and a match. They simply cannot be trusted.

People discuss their work. They criticize their colleagues. But the rule is that you do not “name names” from the office. At least, you do not name names until your career is at an end and you are dictating your tell-all memoirs to a ghost-writer. Should you choose to name names before then, do not be surprised if you find yourself prematurely retired. Even if everybody at the company shared your opinion, they will now worry that theirs will be the next name plastered over the Internet.

Notice that this is very different from writing a comment or blog post saying, “raganwald is an idiot, here is why.” Fire away, especially if you are criticizing publicly available information: So-and-so wrote this blog post, such-and-such a company released this terrible product.

But when you criticize colleagues—current or former—you run the risk of disclosing confidential information about your employer, about other colleagues, about clients… It is very difficult to discuss the workplace in a public forum without betraying a confidence of some sort.

I am not preaching from a pristine pulpit. I have, in my life, broken trusts. I regret these expressions of weak character to this day. And I don’t mean the mealy-mouth regretting the consequences, I mean I regret my own weaknesses deeply and with a pain that time does not lessen.

Consider this questionable choice on my part: The Interviewee’s Perspective. Now I did not name names. But all the same, if you were considering working with me or interviewing me, would you hesitate for a fraction of a second after reading this? Even if you sympathize with my plight as an interviewee, would you worry that somehow what transpires between us will end up on my weblog?

I was going to remove The Interviewee’s Perspective, however I am leaving it up as a warning to myself and as an example of what not to do.

Do not make this kind of mistake in your own life. No matter what your walk of life, rich, poor, popular, unpopular, smart, so-so: being trustworthy is a quality of character that everyone can and should cultivate. Do not discuss private conversations. Do not air workplace laundry.

Do not squander the trust of others.  ¶ 9:30 AM 7 comments

  Are you in? Right now we are in the middle of the biggest land rush in computer history. For those of you who are too busy getting stuff done to keep score, there was a big rush to create computer companies. IBM won the sweetest and biggest estate on that one. Then we watched wave after wave of companies bringing out smaller computers: Digital, Apple, IBM again. But the next big rush was software, and Microsoft was the big winner that time around.

If all of network TV vanished from existence tomorrow morning would it make a difference? I mean come on. Deal or No Deal? Retards guessing about briefcases? Or dancing with the fucking stars? How did Tom Bergeron ever get on TV and why is he still there?

One of the interesting things to do in a land rush is to figure out whose land is getting taken. We’ve watched eBay, Google, and Craigslist eating Newsprint’s lunch. We’ve watched Apple eat Sony’s lunch, and now it’s going after the Recording Industry, Hollywood and the TV Networks. Simultaneously.

Sure, there are a few exceptions. The Simpsons. Thirty Rock. The Office. But even the good stuff is forced to be somewhat retarded by the strictures of the network system. Ever seen the original Office, the one made in England? Way better than the one on NBC. Let’s face it. Network TV blows. The system blows. The business model blows. The consumer experience blows. But worst of all the content blows. What’s more, the system is set up in such a way that it pretty much requires the content to blow.

Apple having an outside chance—or any, any chance at all—of pulling this off is only possible because those three industries are all equally bloated, ignorant, and wilfully spiteful to both the artists and the customers who pay for everything. That, and demographics matter: the heavy purchasing demographic in all three industries is the first generation to grow up with personal computing devices, portable electronics, and cell phones as the default. The executives running these business still think of this stuff as “new technology.”

Meanwhile all the good stuff is happening off the network grid. There’s this huge pool of young smart funny talent who want nothing to do with networks and are just rolling their own. Right now they’re not getting paid much because the bulk of the frigtarded audience just sits there in front of the network boob tube watching moronic former boy band members trying to do ballroom dancing. It’s just inertia. The viewers do what they’ve always done.

You don’t need to bet on or against Apple to be a big winner. Who was the big winner in the SF Gold Rush? Hint: you may be sitting on his name right now. Levi Strauss sold jeans to miners, he thrived on the ecosystem. There are opportunities for everyone to get a piece of the pie. And there will be pie, a lot of it. A lot of people are going to have a lot of fun, make a lot of money, and literally change our culture.

But that’s changing. The networks know it. The frigtards will get old and die and the people who are young kids today are not even going to pay attention to the networks.

Let’s face it, Western Culture is pretty-much defined by the consumption of media. After we stuff ourselves with food we don’t need, squeeze into Hummers and SUVs we don’t need, then head down to The Mall to enjoy the sterile atmosphere of a pretend-main street in a pretend-small town, where we eat some more, what do we do? Watch TV, lots of it. Go to the movies, lots of them. And listen to music, lots of it.

Somebody is going to change everything we know about television, movies, and music. It could be you.

Are you in?  ¶ 12:57 PM 13 comments

  Simplicity and Complexity

Simple things should be simple. Complex things should be possible.

To make something generic is to make the simple things complex.

Simplicity does not precede complexity, but follows it.

  ¶ 10:31 AM 5 comments