Blogging is Hard, Let's Mine Larry Wall for Quotes All quotes are from Programming is Hard, Let’s Go Scripting… by Larry Wall, Perl’s creator:

what is scripting?

When we call something a scripting language, we’re primarily making a linguistic and cultural judgement, not a technical judgement.

For a given dimension X, different scripting languages make different choices, set the knob at different locations.

• You can’t even think about X!
• There’s only one way to do X!
• There’s more than one way to do X!
• There are too many ways to do X!

programming languages

I started out as a BASIC programmer. Some people would say that I’m permanently damaged. Some people are undoubtedly right.

I had a college buddy I did pair programming with. We took a compiler writing class together and studied all that fancy stuff from the dragon book. Then of course the professor announced we would be implementing our own language, called PL/0. After thinking about it a while, we announced that we were going to do our project in BASIC. The professor looked at us like were insane.

Nobody else in the class was using BASIC. And you know what? Nobody else in the class finished their compiler either. We not only finished but added I/O extensions, and called it PL 0.5. That’s rapid prototyping.

Is LISP a candidate for a scripting language? While you can certainly write things rapidly in it, I cannot in good conscience call LISP a scripting language. By policy, LISP has never really catered to mere mortals.

And, of course, mere mortals have never really forgiven LISP for not catering to them.

I’ve always admired Tcl’s delegational model of semantics. But it fell into the same trap as LISP by expecting everyone to use the One True Syntax. Speaking of the One True Syntax: I don’t really know much about Python. I only stole its object system for Perl 5. I have since repented.

PHP takes the worse-is-better approach to dazzling new depths

postmodernism

I read [the word "text"] from a postmodern perspective. Of course, the term Postmodern is itself context-sensitive. Some folks think Postmodernism means little more than the Empowerment of the Vulgar. Some folks think the same about Perl. But I take Postmodernism to mean that a Text, whether spoken or written, is an act of communication requiring intelligence on both ends, and sometimes in the middle too. I don’t want to talk to a stupid computer language. I want my computer language to understand the strings I type.

Human languages therefore differ not so much in what you can say but in what you must say. In English, you are forced to differentiate singular from plural. In Japanese, you don’t have to distinguish singular from plural, but you do have to pick a specific level of politeness, taking into account not only your degree of respect for the person you’re talking to, but also your degree of respect for the person or thing you’re talking about.

So languages differ in what you’re forced to say. Obviously, if your language forces you to say something, you can’t be concise in that particular dimension using your language. Which brings us back to scripting.

I like it when most of the actual words are those chosen by the programmer to represent the problem at hand. I don’t like to see words used for mere syntax. Such syntactic functors merely obscure the real words. That’s one thing I learned when I switched from Pascal to C. Braces for blocks. It’s just right visually.

Actually, there are languages that do it even worse than COBOL. I remember one Pascal variant that required your keywords to be capitalized so that they would stand out. No, no, no, no, no! You don’t want your functors to stand out. It’s shouting the wrong words: IF! foo THEN! bar ELSE! baz END! END! END! END!

worse is better

Classes in Java are closed, which is one of the reasons Java can run pretty fast. In contrast, Ruby’s classes are open, which means you can add new things to them at any time. Keeping that option open is perhaps one of the reasons Ruby runs so slow. But that flexibility is also why Ruby has Rails.

Multiple dispatch is like democracy. It’s the worst way to do late binding, except for all the others.

Among the generalists, the conventional wisdom is that the worse-is-better approach is more adaptive. Personally, I get a little tired of the argument: My worse-is-better is better than your worse-is-better because I’m better at being worser! Is it really true that the worse-is-better approach always wins?

With Perl 6 we’re trying to sneak one better-is-better cycle in there and hope to come out ahead before reverting to the tried and true worse-is-better approach. Whether that works, only time will tell.

  ¶ 2:56 PM 4 comments

  At Last! Researchers explain why there are so many weblogs about programming and programming languages

The Dunning-Kruger effect is the phenomenon wherein people who have little knowledge think that they know more than others who have much more knowledge.

  ¶ 2:17 PM 14 comments

  Why Rubinius Matters to Ruby's Future I am a long-time fan of self-hosted languages. In that post I listed the reasons I thought that a language should be mostly or entirely written in itself. Here’s another reason writing a language in itself is important: If a language’s core libraries and frameworks are written in that language, it is possible for every programmer to improve on them.

Ruby’s core libraries are written in C. Here’s the source for Ruby’s collect method:

/*
 *  call-seq:
 *     array.collect {|item| block }  -> anarray
 *     array.map     {|item| block }  -> anarray
 * 

 *  Invokes block once for each element of self. Creates a 
 *  new array containing the values returned by the block.
 *  See also Enumerable#collect.
 * 

 *     a = [ “a”, “b”, “c”, “d” ]
 *     a.collect {|x| x + “!” }   #=> [“a!”, “b!”, “c!”, “d!”]
 *     a                          #=> [“a”, “b”, “c”, “d”]
 */
static VALUE
rb_ary_collect(ary)
    VALUE ary;
{
    long i;
    VALUE collect;

    if (!rb_block_given_p()) {
    return rb_ary_new4(RARRAY(ary)->len, RARRAY(ary)->ptr);
    }

    collect = rb_ary_new2(RARRAY(ary)->len);
    for (i = 0; i < RARRAY(ary)->len; i++) {
    rb_ary_push(collect, rb_yield(RARRAY(ary)->ptr[i]));
    }
    return collect;
}

Perhaps you like working with Haskell-style fold and unfold rather than the Smalltalk-style collect, select, and detect. No problem, you can hack your own in Ruby, like this:

class Object
  def unfold options = {}, &incrementor
    return [] unless options[:while].nil? || options[:while].to_proc.call(self)
    transformed = options[:map] && options[:map].to_proc[self] || self
    return [transformed] if options[:to] && options[:to].to_proc.call(self)
    incrementor.call(self).unfold(options, &incrementor).unshift(transformed)
  end
end

One hitch: your fold and unfold are hundreds of times slower than Ruby’s built-in-C methods and classes. It reminds me of Newton programming. Apple gave us a really cool language—NewtonScript—for writing applications. Except, the built-in applications were written in C, and the C compiler was only for Apple engineers.

The good news about Ruby is that you can write your own classes in C if you want to. But that is a significant barrier to entry for many programmers, shrinking the available pool of programmers who will enhance the language.

Having core libraries in C is a great choice for implementing a language that is to be used for other things like building web applications. But it is not a great choice for a language that is to be used to build other languages. And “building other languages” is exactly what Bottom-Up Programming or Meta-Linguistic Abstractions are all about. In other words, writing the core libraries in C is not a great choice for a language where programmers write their own abstractions.

Now for many things, the speed penalty of writing your own abstractions in Ruby is negligible. But not everything. So there is always going to be this class of things—and I think collection manipulation is one of those things—where you need to be able to write stuff that is as good as what comes out-of-the-box.

If new stuff is an order of magnitude slower, you might be able to use it for non-critical things, but your chances of persuading anyone else to use it are very low. Which means that the language as a whole progresses slowly because real progress can only happen in areas where performance doesn’t matter. Like database-bound web applications.

Having an implementation where the built-in stuff is on the same footing as your stuff opens up the doors for actual progress. It forces the language itself to be Good Enough, and it makes it possible for every Ruby programmer to improve the language.

what we can learn from java, whoops smalltalk

Java has this incredibly powerful and popular IDE, Eclipse. It is so powerful that many people feel it is impossible to write production Java code without it. Why is it so powerful?

One of the major reasons it is both powerful and popular is the availability of plug ins. It seems to support the language, UML diagrams, source code control, and everything up to (I’m pretty sure) time tracking for client billing. Naturally, the plug ins are written in Java, just like the almost all of the built-in functionality.

Clara Creative can write her own plug in and it won’t be a second-class citizen. And since it is a tool for Java programmers, Clara Creative already knows how to write plug ins, she doesn’t need to drop into another language.

Wow, that is neat. And it does help explain why Eclipse has so many plug ins, and why they are popular: there is no low-level language barrier, and they all on an equal footing with each other.

This shouldn’t surprise you. Eclipse evolved from IBM VisualAge Micro, which was written by Smalltalk programmers in Smalltalk. And of course, Smalltalk is a language where almost everything is written in Smalltalk itself. Smalltalk programmers expect to be able to extend the language and environment without penalty.

In the end, the choice of whether to implement core features in C or Ruby will always be difficult. The temptation to optimize for speed will always be strong, especially when the language is fighting for mind share. But extensibility and variety is also a win, and Ruby fights with its libraries and features as much as with its performance.

Perhaps we won’t all be using fold and unfold instead of collect, select, and detect. But if we aren’t, it ought to be because we prefer the originals, not because the replacements are crippled in comparison, or because the kind of person who likes inventing new tools prefers to write them in Ruby instead of in C.

---

I’m looking forward to hearing more from the Rubinius team. I really think they hold the key to the future. Thanks, Ezra, for your comment.

Labels: ruby

  ¶ 5:05 PM 7 comments

  Something in Ruby that bit me at work today The methods :class and :dup are a bit of a leaky abstraction when used in combination with an object’s Singleton Class:

  class Marx
    def jokes (laughs)
      laughs * 2
    end
  end

  groucho = Marx.new
  chico = groucho.dup

  puts "#{groucho.jokes(10)} == #{chico.jokes(10)}"

  def groucho.jokes (laughs)
    laughs / 2
  end

  harpo = groucho.dup

  puts "#{groucho.jokes(10)} != #{harpo.jokes(10)}"
  puts "...but #{harpo.jokes(10)} == #{chico.jokes(10)}"
  puts "because Groucho is still considered a #{groucho.class}"

  ¶ 2:16 PM 6 comments

  If you cannot change your language, change your language

You want to strive to make sure every single line of code has some value or meaning to the programmer. Programming languages are for humans, not machines. If you have code that looks like it doesn’t do anything useful, is hard to read, or seems tedious, then chances are good that Python has some language feature that will let you remove it.

Generalizing this, I would say:

You want to strive to make sure every single line of code has some value or meaning to the programmer. Programming languages are for humans, not machines. If you have code that looks like it doesn’t do anything useful, is hard to read, or seems tedious, then introduce an abstraction that will let you remove it.

If your language’s mechanisms for abstracting away accidental complexity are so laborious that you cannot remove the useless, the hard to read, and the tedious from your programs without introducing code that is even more useless, harder to read, and more tedious to your framework, then change languages.

  ¶ 4:10 PM 14 comments

  Groucho Marx on Programming Flame Wars

Years ago, I tried to top everybody, but I don’t anymore. I realized it was killing conversation. When you’re always trying for a topper you aren’t really listening. It ruins communication.

  ¶ 7:12 AM 3 comments

  Paul W. Homer on The Nature of Simple

Back in [my VMS] days, we could place as many C functions as we wanted into individual files. One developer, while working, came up with the philosophy of sticking one, and only one function into each file. This, he figured was the ‘simplest’ way to handle the issue. A nice side effect was you could list out all of the files in a directory and because the file name is the same as the function name, this produced a catalog of all of the available functions. And so off he went.

At first this worked well. In a small or even medium system, this type of ‘simplified’ approach can work, but it truly is simplified with respect to so few variables that we need to understand how quickly it goes wrong. At some point, the tide turned, probably when the system had passed a hundred or so files, but by the time it got to 300 it was getting really ugly…

Although many of the functions were related, the order in the ‘dir’ was not. Thus similar functions were hard to place together. All of this was originally for a common library, but as it got larger it became hard to actually know what was in the library. It became less and less likely that the other coders were using the routines in common. Thus the ‘mechanics’ of accessing the source became the barrier to prevent people from utilizing the source, which was rapidly diminishing the usefulness and work already put into it…

By now, most developers would easily suggest just collapsing the 300 files into 12 that were ordered by related functions. With each file containing between 10 and 30 functions—named for the general type of functions in the file, such as date routine —navigating the directory and paging through a few files until you find the right function is easy. 12 files vs. 300 is a huge difference.

Clearly the solution of combining the functions together into a smaller number of files is fairly obvious, but the original programmer refused to see it. He had “zoomed” himself into believing that one-function-per-file was the ‘simplest’ answer. He couldn’t break out of that perspective. In fact, he stuck with that awkward arrangement right up to the end, even when he was clearly having doubts. He just couldn’t bring himself to ‘complicate’ the code.

Just one thought to ponder: sometimes what is simple at one scale is not simple at another. You can use this argument both ways when thinking about idioms and programming paradigms.

You could argue that things like functional programming techniques don’t scale to larger teams. You could also argue that sticking to the lowest-common-denominator patterns works fine for small projects, but you need to employ higher levels of abstraction if you are going to keep a large application with many maintainers flexible and cost-effective.

Hmmm.  ¶ 11:45 PM 4 comments

  Pouring water back into the flask

The University of Flatland uses a novel two-part practical test to determine whether Comp. Sci. undergraduates should be steered into practical programming or abstract theory streams. In the first part, the students are given a beaker, a bunsen burner, a stand, and a flask of water. They are told to boil water. Naturally, they fill the beaker, place it on the stand, light the burner, and are queued up to perform the second test.

In the second part of the test, they are presented with a beaker of water sitting on a stand over a Bunsen burner, and an empty flask. Most of the students light the burner and are led away to begin their studies in programming. But a precious few disassemble the apparatus and pour the water back into the flask, reducing it to a problem they have already solved. They are led away to begin the long road to their Ph.D. in Lisp, Recursion, and Category Theory.

Like most the jokes I retell, this is not particularly funny. But let’s talk about “Design Patterns” and then come back to it. Design patterns allow developers communicate their intentions to each other with a common vocabulary.

This makes sense. If I create a Flyweight and I want to describe it to another developer, having a word for it, along with a common understanding of what problem a flyweight solves, streamlines our communication. Design patterns in that light are jargon, a sub-language used by specialists to discuss their speciality.

If it stopped right there, you would have the design patterns invented by Christopher Alexander and articulated in the incredible book A Pattern Language.

But it doesn’t stop there. In certain programming cultures, people consider Design Patterns to be a core set of practices that must be used to build software. It isn’t a case of when you need to solve a problem best addressed by a design pattern, then use the pattern and refer to it by name. It’s a case of always use design patterns. If your solution doesn’t naturally conform to patterns, refactor it to patterns.

Is this madness? Yes… And no…

Consider Scheme. Everything in Scheme is built out of just five primitive “special forms.” It is positively Forth-like in its economy of power. So it is clearly possible to build very powerful programs out of five primitives. So why not build programs out of thirty-five patterns?

While we digest that, back to the joke. Scheme programmers are clearly the impractical theoreticians, aren’t they? Reducing everything to their five special forms that they have already solved and what-not. While the practical programmers look for the simple, direct way to boil water.

When you take a problem with a straightforward solution and make it more complex by re-expressing it as a combination of patterns, you are pouring the water back into the flask.

So what do we make of the “everything should be one of these thirty-five standard design patterns” argument? I make of it the same thing that I make of the joke. It is clearly possible. But when you take a problem with a straightforward solution outside of the core patterns in one book and make it more complex by re-expressing it as a combination of patterns, you are pouring the water back into the flask.

The argument that “everything should be one of these thirty-five standard design patterns” is an argument that fits the theoretician, not the pragmatist. It is motivated by a desire for building very complicated things out of very simple parts. That is possible. But it is a fallacy to believe that simplifying the constituent parts simplifies the software. Like boiling water, you can make it more complex when you place the pattern ahead of the solution.

Of course there are arguments ad nauseam about standardization and readability. But I have this strong suspicion that at the core of it, the motivation is a belief that the world should be reduced to a simple set of easy-to-understand things that can be combined and recombined into complex solutions.

And Scheme programmers? You may have noticed that although Scheme programs are built out of the five special forms, Scheme programmers do not write everything in the five forms: they use abstractions like continuations, macros, and functions to write expressive and powerful programs.

If you re-wrote a complex Scheme program in the five primitives, would it really be easier to understand because one programmer could describe it to another using just five words in their common vocabulary?

---

This is a follow-up to Newly Discovered Design Pattern: “Code Well.”

addendum

Scheme’s five special forms: The ones I was thinking of are define, lambda, if, quote, and set!. And I’m not convinced you need define. This is from memory, and furthermore just because you can build everything from a few primitives doesn’t mean that that’s how the implementation works. Smalltalk took this aggressive approach to building Smalltalk in Smalltalk, but I am not immersed in Scheme, I do not know what current implementations actually do.

If you do a little Googling, you will find that people often refer to constructs like let as special forms, because they are not function calls. However, they are not primitive special forms because you can build let out of lambda and function calls.

Given the primitive special forms, you also need a library with functions like equals, car, and cdr defined. Between the primitive forms and the primitive library, you can define eval metacircularly and define-syntax for all of the other special forms. From there, you can build a modern Scheme in Scheme.

(There may be some other good choices for building “a Scheme.” You may want to consider call/cc a primitive special form. If you don’t, you have to rewrite function calls (probably using CPS), and this means you are no longer running your Scheme programs in your primitive Scheme, but rather in your evaluator.)

And furthermore…

Keith Braithwaite’s critique.  ¶ 10:37 AM 12 comments

  Newly Discovered Design Pattern: "Code Well."

When I was in college, one of the jobs I had was a TA for an intro programming class. For one of their projects, I was asked to whip up a kind of web browser “shell” in Java. The basic idea was to make a browser that would be highly extendable by students, while freeing them from worrying about the overall rendering framework.

Now, the first language that I learned was Smalltalk, which has historically been relatively design-pattern-free due to blocks and whatnot, and I had only learned Java as an afterthought while in college, so I coded in a slightly unorthodox way, making use of anonymous inner classes (i.e., shitty lambdas), reflection, and the like. I ended up with an extremely loosely coupled design that was extremely easy to extend; just unorthodox.

When I gave it to the prof, his first reaction, on reading the code, was…utter bafflement. He and another TA actually went over what I wrote in an attempt to find and catalog the GoF patterns that I’d used when coding the application. Their conclusion after a fairly thorough review was that my main pattern was, “Code well.”

That year, the class didn’t actually end up using any of the code I wrote, but a year or two after I quit being a TA for the course, they did. I took a look at what became of what I wrote. Needless to say, it now made extraordinarily heavy use of patterns everywhere—and had a massively larger code footprint as a result—but, after all, it’s important to show the young peons how to use patterns. (Euf…)

So, yes, I have to agree that dogmatic use of design patterns hurts, and that a new language won’t automatically eliminate that.

But I’d like to believe that a language that doesn’t take ten lines of code to invoke a method by reflection, or five lines to write what in any decent language would be a one-line lambda, would at least help.

The follow-up: Pouring water back into the flask.  ¶ 5:41 PM 8 comments

  Golf is a good program spoiled A reader mentioned that “It’s not about lines of code.” And he was right: Brevity alone is an unreliable way to judge a program. The problem is—quite simply—the existence of Golf. In any Turing Equivalent language, it is possible to construct programs that are very small and very difficult to understand.

Brevity is not our goal, but it is a side-effect of what happens when we write software well. These are my thoughts about the relationship between program size and readability. It’s my take on how you can look at one short program and dismiss it as Golf, and look at another and praise it as being succinct and expressive.

what really matters

So what really matters? What is it about well-written software that makes it succinct and elegant? Let’s look at an absurdly trivial snippet of code and talk about five different ways we could express its intent:

a = b + c * 2
d = e + f * 2
g = h + j * 2
k = m + n * 2
p = q + r * 2

Hmm. Thirty-five symbols. What about this:

def addtimestwo(x,y)
    x + y * 2
end
# ...
a = addtimestwo(b,c)
d = addtimestwo(e,f)
g = addtimestwo(e,f)
k = addtimestwo(m,n)
p = addtimestwo(q,r)

Twenty-eight symbols. Hey, we can also do this:

def addtimestwo(x,y)
    x + y * 2
end
# ...
a, d, g, k, p = addtimestwo(b,c), addtimestwo(e,f), addtimestwo(e,f), addtimestwo(m,n), addtimestwo(q,r)

The same twenty-eight symbols. That leads us to:

a, d, g, k, p =  *[[b,c],[e,f],[h,j],[m,n],[q,r]].map { |x,y| x + y * 2 }

Twenty-five symbols. Which reminds me:

a, d, g, k, p =  *[[b,e,h,m,q],[c,f,j,n,r]].zip.map { |x,y| x + y * 2 }

Twenty-six symbols, but fewer arrays. Comparing them, is the last one less readable than the first? Less “intuitive”?

Let’s get that word out of the way immediately. “Intuitive” is not some wonderful principle that guides us to creating great software. It means no more and no less than “familiar.” There’s value in doing the same things the same ways, but only when it’s the right thing to do in the first place. Learn your tools.

So which of these trivial examples is better? And what does that tell us about program size? Let’s look at the examples and what they tell us. The first example tells us that there is no relationship between the various expressions, none. The second, where we have ‘abstracted out’ the addtimestwo method, tells us that it is no coïncidence that the five expressions used the same formula: they are the same thing.

def addtimestwo(x,y)
    x + y * 2
end
a = addtimestwo(b,c)
d = addtimestwo(e,f)
# ...

If that calculation was a piece of business logic, we might want to ensure that if you change one, the others all change as well. In the second example, they are on separate lines, suggesting that although the calculation is shared, the individual assignments are still decoupled from one another. It suggests to me that you might want to remove one, or move it elsewhere.

In the third example, we have moved them all onto the same line.

a, d, g, k, p = addtimestwo(b,c), addtimestwo(e,f), addtimestwo(e,f), addtimestwo(m,n), addtimestwo(q,r)

In some languages, we might have moved all of the assignments into their own block. By grouping them, we have indicated very strongly that the five assignments belong together. In an imperative language like Ruby, this signals that there is something ‘atomic’ about this, and that you don’t want anything else to happen until all five assignments have been mode.

a, d, g, k, p =  *[[b,c],[e,f],[h,j],[m,n],[q,r]].map { |x,y| x + y * 2 }
a, d, g, k, p =  *[[b,e,h,m,q],[c,f,j,n,r]].zip.map { |x,y| x + y * 2 }

The fourth and fifth examples are special. The fourth example says that the inputs to these calculations is already arranged in a particular structure. The fifth example says that they are arranged in a different structure, but we will use a particular unfold—zip—to get them into the form we want to do the calculating.

detour ahead

This is exactly where expressive languages shine, and simultaneously it is exactly where you can go badly wrong in a quest to achieve size for size’s sake. This is a trivial example, but please extrapolate the principle upwards to lambdas, methods, objects, hierarchies, modules, frameworks, and applications.

The fourth and fifth examples are shorter than the first and second, but they are much worse if what they imply—a relationship between the inputs to the calculations—is contrived solely to shorten the code.

On the other hand, if that relationship actually does exist, then they are far, far superior to the other examples. If there is a relationship there, the most important thing, the clearest thing, is to express the relationship, to make the code we write clearly communicate the semantics of the data it manipulates.

no free lunch

Let’s look at example two again:

def addtimestwo(x,y)
    x + y * 2
end
a = addtimestwo(b,c)
d = addtimestwo(e,f)
# ...

Many people feel that regardless of the expressive power of their language, if it supports procedure calls, they’re good to go: they can abstract things into procedures, and bundle procedures into libraries, and with good design and decoupling and muttering yaddda yadda yadda while waving their hands just right, the result will be manageable, readable, abstracted code.

Ok, I’m pulling their legs. It isn’t wrong, it’s absolutely correct that you can get an enormous win out of abstracting procedures, just like example two. But although that is good, having other ways to organize code is better. Example two adds some overhead: you have to read addtimestwo and control-click the label to jump to the implementation (alert to IDE lovers: I do this kind of thing in Ruby as well, please don’t assume that the rest of us are scrabbling with Notepad.exe).

This abstraction adds overhead. “Abstracting” the common operation has made it more difficult to read, not less difficult to read. People for who consider meta-programming some sort of Black Magic often make this exact point: The mechanism for removing duplication adds complexity itself. One view is that the overall effect is only a win if the complexity added is small compared to the duplication removed.

Another view, the one I am promoting here, is that it isn’t about removing symbols, it’s about communicating something about the underlying relationships. Even if your language is so crufty that example two is longer than example one, it’s the right thing to do if the calculations are the same for a deep reason, if you are trying to communicate that they will always be the same.

That being said, my experience is that when a relationship exists, the code that clearly expresses it usually winds up being shorter than code that does not express it.

relevance in the age of blub

Programming languages teach you not to want what they cannot provide.

I said usually. And here is the pitch for more expressive languages. Languages make expressing certain things easier than others. Each language has its strengths and weaknesses in these areas. Have you noticed how many blog posts have been written explaining Monads?

The problem with languages is that if you are diligent, you will carefully organize your code to express all of the relationships that your language supports. So in Pascal, you factor out all of the common functions and procedures. And when you are done, you have maximally “compressed” your code. Not in the sense of creating an unreadable mess, but you have made your code as small and as elegant as you think is possible and prudent.

But there are two problems. Pick one:

• You do know other languages, and you recognize the other relationships that Pascal does not express well. So you Greenspun OO into your code base, and you’re hacking away at first-class functions. Now you can express these other relationships, but your code base is extremely idiosyncratic, and in many cases the code is longer and more convoluted because you have to do weird backflips to make the language try to pretend to be Java or Factor.


• You don’t know any other languages, so you think that you have done your best. You have no idea why other programmers are mumbling about Objects, or First-Class Functions, these weird things merely make the code an unreadable mess. You don’t see the underlying relationships those techniques express because you are a Pascal programmer.

Either way, Pascal places limits on how well you can express the underlying relationships between things in your programs. The difference between the two situations is the difference between blissfully moving dirt because “that’s what we do, we move dirt,” and swearing under your breath all day as you fight a losing battle against dirt taking over your construction site.

An “X programmer”, for any value of X, is a weak player. You have to cross-train to be a decent athlete these days. Programmers need to be fluent in multiple languages with fundamentally different “character” before they can make truly informed design decisions.

Let’s look at an oft-quoted example from a fictional language. We’ll call the language Blub:

float frubbish = 0.0;
for (int i = 0; i < foo.length; ++i) {
    for (int j = 0; j < bar.length; ++j) {
        for (int k = 0; k < bash.length; ++k) {
            frubbish += some_function(foo[i], bar[k], bash[j]); # updated
        }
    }
}

This is so wrong in so many ways, but let’s start with the obvious bug. If are a Junior Blub Programmer, you believe that this algorithm expresses the underlying relationship properly, so you just fix the bug. If you are a Senior Blub Programmer, you don’t just fix the bug, but you add meaningful variable names so that wrong code looks wrong:

float frubbish = 0.0;
for (int index_foo = 0; index_foo < foo.length; ++index_foo) {
    for (int index_bar = 0; index_bar < bar.length; ++index_bar) {
        for (int index_bash = 0; index_bash < bash.length; ++index_bash) {
            frubbish += some_function(foo[index_foo], bar[index_bar], bash[index_bash]);
        }
    }
}

Ah, but if you are a Blub Architect, you realize that asking programmers to use meaningful variable names to keep out of trouble won’t work. Why, if that’s all it took, we could use dynamically typed languages like Javascript! No, we must put the compiler to work so that it is impossible to make this mistake. In fact, that’s exactly how Ada works: There is a special type for “the index of a foo,” and it is different from the type for “the index of a bar,” so the buggy code wouldn’t even have compiled. You can do this with C++ as well, and let’s pretend you can do it with Blub:

float frubbish = 0.0;
for (IndexFoo index_foo = new IndexFoo(0); index_foo < foo.length; ++index_foo) {
    for (IndexBar index_bar = new IndexBar(0); index_bar < bar.length; ++index_bar) {
        for (IndexBash index_bash = new IndexBash(0); index_bash < bash.length; ++index_bash) {
            frubbish += some_function(foo[index_foo], bar[index_bar], bash[index_bash]);
        }
    }
}

The Junior, the Senior, and the Architect are all thinking within the limitations of Blub. They still want to write the same program, they don’t see that the index variable, the test for the end of the loop, the nesting of loops, and incrementing an index have nothing to do with the basic relationship you are trying to express.

Instead of asking themselves why they have index variables, (or better still, why they have nested loops), the Blub programmers are asking themselves how to minimize the problems index variables create.

People tend to view code bases much the way construction workers view dirt: they want great big machines that can move the dirt this way and that. There’s conservation of dirt at work: you can’t compress dirt, not much, so their solution set consists of various ways of shoveling the dirt around…

Index variables are dirt, and being a better Blub programmer is an exercise in learning more sophisticated ways to handle dirt. The Java Programmer has been liberated from this kind of thinking. She writes:

float frubbish = 0.0;
for (float each_foo: foo) {
    for (float each_bar: bar) {
        for (float each_bash: bash) {
            frubbish += some_function(each_foo, each_bar, each_bash);
        }
    }
}

She is not concerned with ways to prevent mistakes with index variables, because Java gives her a tool to make them go away. While the Blub programmers are arguing about how to manage dirt, she is writing code that expresses the relationship she wishes to communicate.

Her code is shorter, yes. And much better. And the reason it is much better, the desirable property of her code, is that it communicates the actual thing going on and does not communicate a bunch of other things that are irrelevant.

Those extra moving parts, the yellow code, the accidental complexity makes the code harder to read. It adds opportunities for errors. It makes it harder to maintain. And all because it contains dirt, it contains things that do not express some fundamental relationship in the data.

Shorter code is more readable when it is shorter by dint of expressing the underlying relationship, without irrelevant details.

Alas, while everyone agrees with this statement, the tyranny of programming in Blub is that the Blub programmer does not know that his code contain irrelevant details: he thinks his code does express all of the necessary ideas in his algorithm, he considers shorter code “cryptic.”

thank god, his typewriter ribbon is running dry

Now we can see that although each language provides abstraction mechanisms, and lets you build new abstractions with the mechanism, new kinds of abstractions give us new ways to express relationships. These things can be abused, of course, but nothing can save you from this: If you don’t let your Architect play with Domain-Specific Languages, what is to stop them from configuring everything in your application with XML?

The goal is readable code that expresses the underlying relationships. When we are programming in our personal Blub, the only relationships we see are the ones our language affords. Thus, we are always tempted to think that are programs already are as readable as the underlying problem domain allows, and that efforts to make them shorter are circus tricks that make the code less readable.

And the only way to break out of Blub is to cross-train, to really learn new languages. Otherwise, how will you know if your code could be shorter in a good way?

Shorter code is better if it expresses underlying relationships, if it is shorter because it does not contain irrelevant cruft, if it does not contain workarounds and hacks to deal with limitations in the programming language. Of course, shorter code is not always better. If it is “shorter by coïncidence,” by exploiting language tricks that do not reflect underlying relationships in your programs, that is bad.

Golf is a good program spoiled.  ¶ 12:31 PM 17 comments

  A growing sense of doom washed over me as I read Steve's latest post As a few people have noticed, this went out over RSS but was not on my site. The reason is that I “pulled” it. I started out thinking I would include some choice quotes from Steve’s essay and leave it at that, but… At some point I got enthusiastic and decided I wanted to say something about the culture of moving dirt (I think I have run out of things to say about Java the language, thank goodness). However, reality in the form of interesting work at work popped up, so I decided to put the essay off.

Bad luck, the post was already out on RSS. So here it is. I would like to rewrite it fully, just not right now. Sorry for letting it out onto RSS in such a half-baked form. I realize your time is valuable, and I think you deserve better from me.

Update: Golf is a good program spoiled.

---

I happen to hold a hard-won minority opinion about code bases. In particular I believe, quite staunchly I might add, that the worst thing that can happen to a code base is size…

People in the industry are very excited about various ideas that nominally help you deal with large code bases, such as IDEs that can manipulate code as “algebraic structures”, and search indexes, and so on. These people tend to view code bases much the way construction workers view dirt: they want great big machines that can move the dirt this way and that. There’s conservation of dirt at work: you can’t compress dirt, not much, so their solution set consists of various ways of shovelling the dirt around…

It’s just a mountain of dirt, and you just need big tools to move it around. The tools are exciting but the dirt is not…

The problem with Refactoring as applied to languages like Java, and this is really quite central to my thesis today, is that Refactoring makes the code base larger. I’d estimate that fewer than 5% of the standard refactorings supported by IDEs today make the code smaller. Refactoring is like cleaning your closet without being allowed to throw anything away. If you get a bigger closet, and put everything into nice labelled boxes, then your closet will unquestionably be more organized. But programmers tend to overlook the fact that spring cleaning works best when you’re willing to throw away stuff you don’t need…

Design Patterns was a mid-1990s book that provided twenty-three fancy new boxes for organizing your closet, plus an extensibility mechanism for defining new types of boxes. It was really great for those of us who were trying to organize jam-packed closets with almost no boxes, bags, shelves or drawers. All we had to do was remodel our houses to make the closets four times bigger, and suddenly we could make them as clean as a Nordstrom merchandise rack.

A design pattern isn’t a feature. A Factory isn’t a feature, nor is a Delegate nor a Proxy nor a Bridge. They “enable” features in a very loose sense, by providing nice boxes to hold the features in. But boxes and bags and shelves take space. And design patterns – at least most of the patterns in the “Gang of Four” book – make code bases get bigger. Tragically, the only GoF pattern that can help code get smaller (Interpreter) is utterly ignored by programmers who otherwise have the names of Design Patterns tattooed on their various body parts…

If you begin with the assumption that you need to shrink your code base, you will eventually be forced to conclude that you cannot continue to use Java. Conversely, if you begin with the assumption that you must use Java, then you will eventually be forced to conclude that you will have millions of lines of code…

You should take anything a “Java programmer” tells you with a hefty grain of salt, because an “X programmer”, for any value of X, is a weak player. You have to cross-train to be a decent athlete these days. Programmers need to be fluent in multiple languages with fundamentally different “character” before they can make truly informed design decisions…

Java is like a variant of the game of Tetris in which none of the pieces can fill gaps created by the other pieces, so all you can do is pile them up endlessly.

Imagine that you have a tool that lets you manage huge Tetris screens that are hundreds of stories high. In this scenario, stacking the pieces isn’t a problem, so there’s no need to be able to eliminate pieces. This is the cultural problem: they don’t realize they’re not actually playing the right game anymore…

Java-style IDEs intrinsically create a circular problem. The circularity stems from the nature of programming languages: the “game piece” shapes are determined by the language’s static type system. Java’s game pieces don’t permit code elimination because Java’s static type system doesn’t have any compression facilities – no macros, no lambdas, no declarative data structures, no templates, nothing that would permit the removal of the copy-and-paste duplication patterns that Java programmers think of as “inevitable boilerplate”, but which are in fact easily factored out in dynamic languages.

Dynamic features make it more difficult for IDEs to work their static code-base-management magic. IDEs don’t work as well with dynamic code features, so IDEs are responsible for encouraging the use of languages that require… IDEs. Ouch…

I have read many essays taking the minority position and explaining what is wrong with the “Java Programmer” mindset (as opposed to the “Programmer who happens to use Java mindset”), but Steve’s essay laid the cultural problems bare for me. He explains the technical issues well, but where this essay really shines is explaining the cultural issues and how they work with the technical issues to create a vicious cycle.

Of course I recommend reading the original. But may I add, please do not get sucked into arguing whether Design Patterns are good, or whether IDE refactorings really work, or any of the other technical points that are so much fun to rehash for the millionth time.

Instead, consider the cultural forces at work. Cultural problems cannot be solved with technology. If you are an advocate for change, ask yourself what sort of cultural change is needed, not what sort of technical problems need to be solved.

And oh yes… What do you think of his central thesis?  ¶ 11:31 AM 12 comments

  Yard Sale at Raganwald's House I’ve taken a weed whacker to the article links on the weblog’s sidebar (Heh, only an author would think that removing seven links is anything but a minor adjustment! But still, I’m fond of them all, it is difficult to let them go).

Here are the links I have removed:

1. Ockham’s razor as it applies to the big rewrite


2. Icebreakers


3. Hiring a Juggler


4. Are you thinking of working for a start up?


5. Three blog posts I'd love to read (and one that I wouldn't)


6. What Haskell teaches us about writing Enterprise-scale software I, II


7. Programming conventions as signals

Sniff. I’ll miss them!

While I’m wallowing in grief, I might as well mention a few of my favourites that haven’t quite made it into the starting rotation. You know, sixth man and all that:

1. Zen in the Art of Rewriting


2. Rails-style creators in Java, or, how I learned to stop worrying and love anonymous inner classes


3. What does Barbara Liskov have to say about Equality in Java?


4. Why are local variables bad?

Well, it’s time to stick a price tag on them and carry them out to the lawn. Good bye, dear articles. Best of luck wherever you end up…  ¶ 9:00 AM 2 comments

  Trial-and-error with a feedback cycle

Don’t EVER make the mistake that you can design something better than what you get from ruthless massively parallel trial-and-error with a feedback cycle.

That's giving your intelligence much too much credit.

trial and error

A common mistake when trying to be “Agile” (or trying to appear to be Agile without actually being Agile) is to shorten milestones while still maintaining the idea that we have a plan for every milestone done up at the beginning of the project and we consider changing the plan to be a management failure.

The whole point of trial and error is to make our plan, but to accept the inevitability that our plan will not survive contact with reality. I was working on one project, and the first two-week cycle went badly sideways. The project manager complained that she hates missing milestones early in a project, it foreshadows all sorts of problems. She wanted to impose overtime to hit the date.

I urged restraint. If we were wrong about our velocity, or wrong about some of our technical predictions, or wrong about some of the requirements, it would be far better to adjust our expectations. The two-week cycles were a bonus because they were providing the feedback we needed to adjust our plans.

If you ignore the evidence, if you try to twist the appearance of reality to suit your plan rather than twist your plan to suit reality, the feedback is wasted.

feedback

Feedback is so critical, that even if you do everything else right, and even if you think you are gathering and responding to feedback, you can still fail. A recent flurry of blog posts discussed The Waterfall Trap. This is when you build software incrementally instead of iteratively.

Increments are components of the finished software that do not provide value in and of themselves, like a database and its associated ORM doo-hicky. When you build an increment, what feedback do you have? Feedback about how well you are executing against your plan.

This is like deciding to march through the jungle and measuring how many steps you take per day. Are you going in the right direction? Who knows, but you know you how fast you are going, and that is worth something, isn’t it?

I can tell you exactly what it is worth. At some meeting later on, when the project has been judged an expensive failure, you can prove it was a well-managed failure under your watch. Congratulations.

For feedback to really work, you must have feedback about how well the finished software delivers value to its users. The only way to do that is to deliver it to those users and get their feedback. Thus, the other approach: to build the software iteratively. This means to build things that are valuable in and of themselves, so users can judge how well they work.

productization

My Golden Hammer for building software iteratively is to divide it into separate products. We sit down and look at the pile of functionality on the plan and ask, “Which of these pieces could be useful things in and of themselves?”

On one project, there was a complex bit of loan eligibility reasoning that needed to be built into a database application. We decided that although the finished application would have the reasoning built into certain input screens, the reasoning could be a separate product. We imagined this thing sitting in its own screen alongside what their existing process.

There would be some double entry involved, but we knew that if we built this as a stand-alone product, it could be used in production and it could provide value. So we did. And sure enough, we got a lot of feedback from the users. We didn't end up deploying it in production, they wanted to wait for the rest of the pieces to be built, but they could play with it, they could see how it worked, they could give us real feedback.

Did I mention they decided against deploying it in production until the rest of the project was ready? That was the most critical piece of feedback yet. It told us it wasn’t as important as we thought, and we were able to reprioritize around other things.

That kind of feedback is only possible when you get feedback about the value you are providing. And that is exactly the feedback Linus is talking about: the ruthless feedback of users who don’t care about your plans or your years of experience or your blog or whether you work for a billion-dollar company or whether you use Java or JRuby, or whether your servers are MSFT or Linux.

The feedback you get from people who just want to get stuff done.  ¶ 11:45 AM 5 comments

  The Grinch Who Stole the Ruby Jobs A reader was kind enough to point out that making statements about jobs being advertised could be misconstrued as a statement about the policies of the companies doing the hiring.

I do not speak for any hiring company, and I’m sure that hiring companies do not particularly want me articulating their policies for them.

Therefore, my policy hereforward is that when I wish to describe a job, I will provide links to open positions or name companies hiring. If I quote something, it is partially or fully information provided by the company and it will be indicated as such in quote marks.

I know, what a bother. Sheesh. But there you have it, that seems to be the way the world works. Roll with it and let’s move on.

---

mdlogix, a rapidly growing medical research software company is seeking Ruby on Rails developers for our downtown Toronto office.

We're looking for passionate developers who love Rails and want to make the world a better place by supporting clinical research!

For more information, visit http://www.mdlogix.com/aboutUs/jobSoftwareEngineerToronto.html or speak to us at the next Rails pub night!

Hewitt Associates is looking for a “P/A with Ruby on Rails experience,” possibly in Toronto (No link: they use an HR application for job listings that supports nice searches but mightily resists bookmarking).

GigPark is a Toronto-based web startup that's growing fast
Built with Ruby on Rails and hosted on EC2
We develop iteratively and release weekly
We care about quality - our code is well-designed and tested
We work hard, but smart. We believe less code is better and simple UI design is key We're not religious about technology. We believe in using the right tool for the job

http://www.gigpark.com/page/jobs

Mobile Commons is looking for “junior and senior hackers”.

Labels: ruby

  ¶ 2:41 PM 1 comments

  IT's time to stop blaming management

Enterprise systems do produce real, significant cost-savings by imposing standards and metrics on what was once chaos. But the result is inevitably software that’s inflexible and doesn’t grow well with the business. Enterprise software never ages well, it never evolves fast enough and it is always vulnerable to disruption — as you might expect of any system that adopts such a command-and-control, monolithic position with regard to its users.

I’ll repeat one bit for emphasis: …The result is inevitably software that’s inflexible and doesn’t grow well with the business. Enterprise software never ages well, it never evolves fast enough and it is always vulnerable to disruption — as you might expect of any system that adopts such a command-and-control, monolithic position with regard to its users.

Now let me ask: Should we expect this to apply to finished Enterprise Software Applications and not to the code produced inside Enterprises? I think it applies decisively to Enterprise Code Bases, and this isn’t something we can blame on Pointy-Haired Bosses.

When an “Enterprise Architect” drives a Platform, Framework, Tool, and Language “Strategy” that is all about Command and Control over the users of same platforms, frameworks, tools, and languages, you get exactly the same result. And who uses these things? Developers. The moment your driving motivation is controlling developers and restricting what they do, you go down the exact same road and end up in the exact same place as when your driving motivation is command and control over users.

While some consider the slow rate of change of enterprise software to be a feature and not a bug this sort of logic isn’t very persuasive anymore and won’t fly far in the age of the web. The the web has clearly demonstrated that it’s possible to build a huge, stable, decentralized platform that still allow for all kinds of innovation without imposing undue restrictions on everybody and everything.

(Ironically the key ingredient may turn out to be trust, of all things, which is something any enterprises—being notoriously paranoid—are forever lacking.)

Software built with tools and policies that fight developers at every step is frighteningly difficult to change, it resists evolution tooth and nail. The argument that it has a lower Total Cost of Ownership in the long run is not just wrong, it turns out to be badly wrong: software developed by people in straight jackets is more expensive to maintain, not less expensive.¹

Centralization is a bug… Anything in the system that requires central authority, that’s something that holds you back.

I am not saying that Anarchy is a viable alternative to Bondage and Discipline. Lots of businesses fail from having too little structure, and software is no different than business in that respect. However, we see plenty of businesses that thrive by pushing decisions downwards, flattening their management structure, by motivating people to do the right thing, by making a commitment to quality staffing, and—yes, it’s true—by choosing to right-size their teams and resist the temptation to grow for growth’s sake.

There is no golden hammer, no silver bullet. It is hard for businesses to thrive and succeed in an environment of change, and it is hard for IT departments as well. But we have to wake up and smell the coffee. We can only blame “management” for so much, and then we have to turn our attention to our own practises, to the mistakes we make that mirror the mistakes they make.

---

1. Oh yeah? Where’s your study to prove it? Good question. I pulled that statement right out of my assets, to tell you the truth. Pure opinion based on the Appeal to Authority (20+ years of experience) and Confirmation Bias (successful projects) fallacies. I was going to pull that line, but I find it outrageous that tool vendors and their sycophants promote TCO as a benefit of their approaches without the slightest credible evidence backing it up.
   
   Honestly, can anyone produce a study that shows business software written in Popular Language X has a lower TCO than business software written in Lisp? Or Python? Or ML? Or Ruby? Anyone? Anyone? The only things I can find when I look are obvious Blowhard Job studies by consulting groups funded by vendors.
   
   Does anyone really know?
   
   I have decided that I no longer accept such claims at face value. Not even my own.

  ¶ 2:15 PM 2 comments

  I can change any time I want to, I just don't want to. Does that sound like anything you’ve heard before? Perhaps from a smoker? Or someone who does not exercise?

Or maybe someone who says they could learn new things, but they are too busy right now keeping on top of what they already know? Or the converse? Someone who is too busy playing with new things to buckle down and work hard with what they already know?

Change is brutally hard. Nobody finds it easy.  ¶ 9:36 AM 2 comments

  Have nothing in your house that you do not know to be useful, or believe to be beautiful The title is a quote by William Morris, a man with a deep interest in design and its relationship to our place in society. I think it applies to ideas as well as artefacts, and I must warn you that the following ideas fall under “beautiful” :-)

The Halting Problem

The halting problem is a decision problem about properties of computer programs on a fixed Turing-complete model of computation. The question is, given a program and an input to the program, whether the program will eventually halt when run with that input. In this abstract framework, there are no resource limitations of memory or time on the program’s execution; it can take arbitrarily long, and use arbitrarily much storage space, before halting. The question is simply whether the given program will ever halt on a particular input.

The reason the halting problem is famous is because it is undecidable, which means there is no computable function that correctly determines which programs halt and which ones do not.

(There seems to be some confusion about what this says. It says that we cannot construct one machine or function or program that can examine any combination of machine or program and input and then correctly decide whether the program halts or does not halt.)

1 + 1 = Cool

We absolutely know that the Halting Problem is true (meaning there is no function that correctly answers whether every program halts or does not halt). However, the math seems a bit gnarly if you were sleeping through your computability courses. And furthermore, just because we know that there must exist programs for which we do not know whether they will halt for at least one input, it’s always a lot more convincing to actually look at such a program.

The Wikipedia article provides a couple of examples based on unsolved problems in mathematics, the search of an odd perfect number and the search for the largest pair of twin primes. The idea is this: if you can write a program that uses brute force to search for a counter-example to an unsolved problem in mathematics, you have one of three things:

1. You run your program, it finds a counter-example, and you win a Nobel Prize in Mathematics (not really!) for disproving the unsolved problem, or:


2. You run your program, it goes on running forever, and you can’t prove that it will ever stop. But it might. If you could prove it would stop than you can also disprove the unsolved problem, so again you win a Nobel Prize in Mathematics.


3. You don’t need to run your program, you know darn well it will never stop, and you can prove it will never stop. Now you have proved the unsolved problem, so again you win a Nobel Prize in Mathematics.

Of course, alternative four is to accept the idea that there are very simple programs that might run forever but we have no way of knowing if they ever stop.

If you go through any of the proofs of the Halting Problem, you already know this. But sometimes, an example is worth a thousand words.

Goldbach’s Conjecture

Goldbach’s conjecture is one of the oldest unsolved problems in number theory and in all of mathematics. It states: Every even integer greater than 2 can be written as the sum of two primes. In other words, the Goldbach conjecture states that every even number greater than or equal to four is a Goldbach number, a number that can be expressed as the sum of two primes.

To review, by unsolved problem, we mean that we haven’t proved it is true and we also haven’t proved it is false. So here’s what we do: Let’s write a program that checks every even integer from 4 up. If it is the sum of two primes, we try the next higher even number. If it is not the sum of two primes, we stop and output the number.

require 'mathn'

@prime_generator = Prime.new
@primes = [@prime_generator.succ] # start with [2]

# given a number, answer a list with at least all 
# of the primes <= that number. may include
# larger primes.
#
# primes_at_least_up_to(12) -> [2,3,5,7,11,13] 
# and possibly more primes
# 
def primes_at_least_up_to(n)
  while @primes[-1] < n do 
    @primes << @prime_generator.succ
  end
  @primes
end

def sum_of_two_primes?(n)
  !!(primes_at_least_up_to(n).detect { |p1| 
      primes_at_least_up_to(n).include?(n-p1) 
    })
end

n = 4

while sum_of_two_primes?(n)
  n +=2
end

puts n

Does this program halt? If you can answer this question, you solve a huge problem in Number Theory. Now again, this does not prove the Halting Problem. For one thing, we might solve the Goldbach Conjecture by proving or disproving it, and then we could tell you whether this program halts.

Another possibility is that we will prove that the Goldbach Conjecture is undecidable. In that case, we will have a proof that we cannot prove whether this program halts. This has been done with other conjectures. For example, we have proven that we can neither prove nor disprove Cantor’s Continuum Hypothesis: it appears that within Number Theory, there is no way to know whether there exist infinities that are both larger than א-null and smaller than א-one.

This program does demonstrate that at a deep level, programs are math. You can draw an equivalence between certain programs and certain statements in mathematics. We can say that certain properties of Program P (like whether it halts) are equivalent to certain properties of Mathematical Statement S (like whether every even number is the sum of two primes).

That connection leads us to Incompleteness. If we know that there must be mathematical statements that are undecidable (like statements that are true but cannot be proved), we know that there are undecidable properties of computer programs. And if we know that there are undecidable properties of computer programs (like whether they halt), we know that there are undecidable statements in mathematics.

I find that deeply, deeply interesting.

---

This was provoked by an example from the Qi language manual. It came at a good time: after writing about various aspects of our industry and its limitations, I needed to be reminded how how much I enjoy thinking about programs.

Which brings me back around to the title. Computer Science is a big field. Programming is a big field. Working for a living developing software is a big field. I believe you have to pick and choose, you can’t learn everything, you can’t try to pick up every new thing that comes along.

But I believe that if you follow Morris’ dictum, you will be effective and also deeply engaged. Do not settle for mediocre ideas. Accept only those things that are useful, beautiful, or both. Do not settle for just beautiful or just useful. It is surprising how often useful things help you build beauty, and likewise how often beautiful things turn out to be useful in a different context.  ¶ 2:43 PM 15 comments

  Utility Belt Utility Belt is a grab-bag of tricks, tools, techniques, trifles, and toys for IRB, including convenience methods, language patches, and useful extensions. It also includes a couple command-line widgets. Its primary inspirations were an awesome gem called Wirble and a blog post by Amy Hoy called “Secrets Of The Rails Console Ninjas”.

INSTALL

sudo gem install utility_belt

FEATURES

• Interactively edit IRB code in your preferred text editor


• Read from and write to OS X clipboard


• Post your code to Pastie with one command (OS X only)


• Kick-ass Unix-style history buffer


• Write command history to file or vi


• Grep classes and methods for strings


• Verbosity controls for regular IRB and Rails console


• Finder shortcuts for Rails console


• Upload shortcut for Amazon S3


• Command-line Amazon S3 upload script


• Command-line Google shortcut (OS X only)


• Auto-indentation


• _ special variable (like Unix shell var !!)


• Extremely basic themes for Wirble syntax coloring


• Pascal/JavaScript-style “with” statement


• String#to_proc


• Grammatically-correct is_an? method - no more “is_a? Array” statements


• One-character exit command

---

Who can resist publicizing a grab-bag gem that just happens to contain some code you snarfed from someone else? Not me! It’s like one of those vanity books of poetry: your poem is included provided you buy twenty copies for your friends and family…

Labels: ruby

  ¶ 7:51 PM 4 comments

  Something's Fishy (updated) Part I: Continuums and False Dichotomies

On another of my posts, there were the following pair of comments:

There are many people who do get into our field purely for money. The sad thing is that they’re allowed to. There are not enough sufficiently hard courses that weed out the less dedicated ones available on the university/college level. Same could be said of the HR people who are not able to distinguish wheat from chaff…

I don’t see why that’s necessarily a sad thing…The fact is that the majority of programming tasks aren’t actually that difficult. They don’t require a superstar in order to get done…

I agree that many programming tasks are not difficult. I am not convinced we have found a good way of predicting which tasks are difficult and which are not, which require a cut and a paste and which require some careful thought to avoid breaking something else or adding cruft to code.

I don’t think we do a very good job of dividing up work on a project such that the people with the least experience are protected from damaging the code. But that is just my opinion. I may be wrong.

I am much more confident when I say that we have to be very careful when we set the bar for the minimum standard needed to ship software. The choice between (a) employing people who aren’t particularly dedicated and would fail anything remotely resembling challenging courses in computer science and, (b) hiring “superstars” is a false dichotomy. There is such a thing as someone who is competent and professional enough to pass tough courses. Someone who won’t get confused if you ask them to keep abreast of developments in their field.

Guillaume seems to agree.

In many other fields there are tough courses. MBA programs are often touted as separating the wheat from the chaff. Yet we don’t assume that every MBA is a superstar in business. We do assume they have a baseline of competency. Likewise Chartered Accountants do not tolerate people taking a lukewarm attitude towards their field. But nobody would describe every accountant as a financial wizard.

There’s a continuum of talent, and we can raise the bar above the gutter without leaping from mediocre to world-class in a single bound.

In any business situation, it’s prudent to be wary of people who put their own selfish interests ahead of the business. So it’s fine to avoid people who want to learn to use the latest bauble on your dime. But again, let’s not establish another false dichotomy: people who are curious about programming and eager to learn are not automatically disinterested in shipping solid code using conservative strategies.

The most important thing is to establish the baseline competency required for the job and find the people who meet it. We don’t lower the bar to match whomever is available when we are looking, and we don’t assume that our choices are always binary (Superstar vs. Mediocre, Passionate vs. Pragmatic, Java vs. Ruby).

Part II: Something is still fishy

There is another way we can misinterpret Guillaume’s comment. Many programming tasks are not difficult. True. But that does not mean that all programming tasks are not difficult. Furthermore, I have experienced a paradox: sometimes when a good person works on an easy task, they see things in the easy task that an average person would miss.

An average person might copy and paste code all day. A good person might refactor to eliminate duplication. Of course, that means the good person can do far more of the easy work than the average person. But more importantly, the code base is forever improved, its friction has been lowered.

The existence of easy tasks in programming can lead us to believe that a programming is a relatively unskilled trade. When we examine the pieces on their own, we might believe so. My question is this:

If anyone can do it, Why do we as an industry stink at it? Although each piece seems easy, the whole of completing a successful project on time and budget with acceptable quality and good long-term cost of ownership is elusive.

Now I know that Joel Spolsky and everyone else at Microsoft will tell you that software is ridiculously high quality compared to something-or-other. And with auto-widget-building wizards and AST-bending IDEs and memory management, today’s programmers ought to be ridiculously productive.

But yet…

I admit this is anecdotal, but I’m hearing a different story from my ex-colleagues and ex-clients out there: projects are late, bug lists are long and getting longer, and a lot of companies are scrutinizing their IT ROI and getting very Scrooge-like about starting new projects in-house.

They just aren’t happy with the results they’re getting.

Now maybe this has absolutely nothing to do with programmers. Maybe it’s all about requirements and waterfall and managing expectations. Maybe it’s all about looking at writing code and realizing that getting ’er done today with a few minor warts is more important than getting ’er done tomorrow with no bugs. (Or worse, tomorrow’s code is even buggier because it features an abstraction of a framework plugged into a component organized in a service architecture, and there’re so many moving parts that nothing works properly.)

Or maybe this has nothing to do with the problem-solving, hold a complicated thing in your head, visualize all the ways it can go wrong so you can program defensively part of programming. I buy that. I buy that maybe we have to be better at writing programs for people to read. I buy that we have to be better at testing our code. I buy that we have to be better at all sorts of software development skills like analyzing requirements.

It could be that programmers on the whole have all of the skills needed, and that our problem is we just haven’t found the right formula for harnessing their power, for getting them all to march in lock step towards project success.

Maybe we programmers are doing a fine job and there’s no need to be better at it. But somehow something isn’t right, and throwing more people with fewer skills at the problem doesn’t seem to be working.  ¶ 4:05 PM 16 comments

  Amortized Costs, DOS, and Magicicadas Jeff Atwood has just posted an interesting article, Hashtables, Pigeonholes, and Birthdays. Hash Tables are a very interesting subject, you can take almost any aspect of them and follow it down interesting trails.

For example:

Performance: Quantifying the performance of a data structure like a Hash Table is interesting. There are measures like Amortized Cost as well as Worst Case Cost. This subject is one of the core themes of the book Purely Functional Data Structures. (You can also download the author’s thesis on the same subject) I recommend it whether you have an interest in “purely functional” programming or not: “purely functional” data structures provide all of the imperative and side-effect oriented operations you know and love—like insertions and removals—while also providing the foundations for highly concurrent environments like threads with shared memory.

Security: One-way functions are strongly related to hash functions. They are the foundation for many of today’s security and secrecy protocols. For example, fingerprinting is used to allow mirror sites to host downloads while giving the public comfort that the downloads have not been compromised with Malware. However, if the one-way functions underlying them can be cracked—as MD5 was recently—miscreants can substitute bad files for good at will.

Even when security does not appear to be a factor, compromising a hash function can have a security impact. As one person mentioned in one of Jeff’s comments, if the hash function for a programming language is public, attackers can deliberately engineer worst-case performance to create a Denial Of Service attack. Here’s a deliberately trivialized example:

Consider a public-facing site with free user registration. When you log in, the site performs a look up on the user name. Now obviously there’s a database somewhere. But perhaps the programmers put a cache in front of it. You could imagine such a cache being useful if cookies are used to keep someone logged in. If the cache uses a hash table, a vandal can create lots and lots of user names that hash into the same bucket. Anyone with a user name in the same bucket will face a long wait as the system performs a serial look up within the bucket.

Whimsey: One of the comments mentioned that the number of buckets is usually a prime number. Did you know that insects figured this strategy out on their own?

---

Editorial/Love Fest:

And now for what I really wanted to say.

People have pointed out that Jeff often seems to be summarizing well-known computer science principles. A Reader’s Digest for programmers, as it were. The suggestion seems to be that this is not useful, since anybody with a degree or even a passing interest in programming ought to know these things. My own feeling is that Jeff’s blog is far more useful than my own.

Coding Horror is published out in the real world. It is written for real people.

This weblog operates firmly in Lake Wobegon. It’s true! The very fact that you are reading about programming means that you are exceptional for reading it and I am exceptional for writing it. I am not going to start a flame war and say you and I are better programmers. Who really knows? But I am very confident when I say that we are not representative of the typical programmer.

The typical programmer does not know what a closure is and does not care. If they are added to Java he or she will not use them any ways. The typical programmer may have obtained their degree, but they were far more interested in memorizing whatever was needed to get a B on the examination then they were in the subject matter.

They may have a certification, but they do not own and have not read books outside of their immediate requirements, so Knuth is foreign to them. While they may talk of refactoring, they have never heard of Martin Fowler or Joshua Kerievsky.

When Jeff writes a Digg-able post about multiple monitors, or how to optimize memory on a Windows PC, he get readers that would never subscribe to a blog if they were told it was about programming. And I am talking about programmers. Try to wrap your head around the idea of programmers who do not find programming blogs interesting (in fact, they don’t find programming interesting at the moment).

If they are then exposed to something insightful and interesting about hash tables… Many will read it and forget it, but a few will be interested enough to delve deeper into the whys behind the whats.

I am all for that. It is remarkably easy to turn and preach to the choir. Evangelizing to the bored and uninterested… This is a frustrating and difficult job. I wish I had the zeal to reach out across the divide and bring people over to where programming is deeply fulfilling and fascinating. I’m glad Jeff is doing the job he’s doing and wish his blog every success.  ¶ 10:36 PM 19 comments

  Is software about grades or ideas? Joel Spolsky yesterday:

The geeks want to solve the problem automatically, using software. They propose things like unit tests, test driven development, automated testing, dynamic logic and other ways to “prove” that a program is bug-free.

The suits aren’t really aware of the problem. They couldn’t care less if the software is buggy, as long as people are buying it.

And today:

One of the reasons Schank hated undergrads so much was that they were obsessed with grades. He wanted to talk about whether computers could think and all undergrads wanted to talk about was why their paper got a B instead of an A.

---

Do I really have to expand this into an essay? How about this joke:

A fellow is in Vegas to do some climbing at Red Rocks. He goes into a bar, it’s quiet, he leans over to the bartender. “Hey!” he calls out, “Wanna hear a joke about programmers?”

The bartender frowns. “I’m a programmer,” the bartender says flatly. He points at a big guy playing pool in the corner, the guy looks like he twists re-bar into Christmas ornaments in his spare time. “Schank over there, he’s a computer science prof. And over there, Lois and Maude?” Our guy looks, there’re two women the size of tanks with the jar-head haircuts to match. “Lois and Maude are programmers, too.”

The bartender leans forward menacingly. “Are you sure you want to tell us your little joke?”

Our man finishes his drink, slaps down the money, and eases out off his stool. “No, best not to bother,” he says. The bartender nods. Our guy speaks again.

“I hate it when I have to repeat the punch line.”

  ¶ 12:32 PM 11 comments

  Thus Spake Joel

What you’ll see is that the hard-core geeks tend to give up on all kinds of useful measures of quality, and basically they get left with the only one they can prove mechanically, which is, does the program behave according to specification. And so we get a very narrow, geeky definition of quality: how closely does the program correspond to the spec. Does it produce the defined outputs given the defined inputs.

The problem, here, is very fundamental. In order to mechanically prove that a program corresponds to some spec, the spec itself needs to be extremely detailed. In fact the spec has to define everything about the program, otherwise, nothing can be proven automatically and mechanically. Now, if the spec does define everything about how the program is going to behave, then, lo and behold, it contains all the information necessary to generate the program! And now certain geeks go off to a very dark place where they start thinking about automatically compiling specs into programs, and they start to think that they’ve just invented a way to program computers without programming.

Now, this is the software engineering equivalent of a perpetual motion machine. It’s one of those things that crackpots keep trying to do, no matter how much you tell them it could never work. If the spec defines precisely what a program will do, with enough detail that it can be used to generate the program itself, this just begs the question: how do you write the spec? Such a complete spec is just as hard to write as the underlying computer program, because just as many details have to be answered by spec writer as the programmer. To use terminology from information theory: the spec needs just as many bits of Shannon entropy as the computer program itself would have. Each bit of entropy is a decision taken by the spec-writer or the programmer.

So, the bottom line is that if there really were a mechanical way to prove things about the correctness of a program, all you’d be able to prove is whether that program is identical to some other program that must contain the same amount of entropy as the first program, otherwise some of the behaviors are going to be undefined, and thus unproven. So now the spec writing is just as hard as writing a program, and all you’ve done is moved one problem from over here to over there, and accomplished nothing whatsoever.

  ¶ 3:25 PM 17 comments

  Does your employer's wilful ignorance of software development principles piss you off? Cut them a little slack, they aren’t out to get you:

Any sufficiently advanced incompetence is indistinguishable from malice.

If you’re still seething, you ought to know that Mobile Commons is hiring. We have