raganwald
Saturday, April 21, 2007
  We don't need no stinkin' Scientific Method
Update I:

This post has been removed.

Originally, it was a critique of Does Functional Programming Really Matter?. However, since I wrote the original post, the authors have added a disclaimer—this is not a real attempt at a scientific paper—it's just a class project. Speaking for myself, I didn't believe the conclusions of this paper even when I wrote it, and doubly so now—to their document.

Under the circumstances, there is nothing left to dispute and I have removed my words.

I’ve been asked why I didn’t leave the critique on line so that people can “see what the fuss was all about.” Here’s the thing: now that the document has that disclaimer front and centre, the most interesting things to talk about have to do with Universities and grades and intellectual rigor and a bunch of other things that are not about programming.

Since this is a programming blog... this was an easy decision. If the original document was not written in earnest, it’s no longer of interest to me.


Update II:

All right, all right, I give. Here’s what I originally wrote, more or less. Remember, when I wrote this there was no disclaimer on the original article, so I was under the impression that the authors actually believed what they were saying (imagine that), and that the authors were writing a paper to academic standards.

It turns out that’s not the case, so now I’m the one making disclaimers. The following represented my view given the information available at the time. Now that I have new information, my belief is that the paper is a NOOP: It says, in effect, “the following is not true: X, Y, Z.” Such a statement is neither supportable nor refutable.

Anyhoo, here’s what I still have of the original post:



We don’t need no stinkin’ Scientific Method

Here’s a paper worth reading: “Does Functional Programming Really Matter? Two Approaches to Comparing Functional and Object-Oriented Programming.”

The authors show how the important ideas in Why Functional Programming Matters can be reproduced in Java programs.

Let me summarize the paper:

Java is Turing Complete. Therefore, all of the ideas in FP can be reproduced in Java. We show this by Greenspunning the important ideas in WhyFP. In essence, we show how to write a poorly specified, buggy implementation, of half of Haskell in Java.

We discover that Java is a cumbersome language for implementing functional programming. Instead of stopping right there, we show our bias by saying that writing programs in an object-oriented language is superior to a purely functional approach, but excuse our problems by saying that Java is at fault, not the premise of embedding FP in OOP.

We do not give examples of embedding FP in other OOP languages that might have radically different models, such as Ruby, SmallTalk, Self, Dylan, Ocaml, or CLOS, leaving the reader to perform their own experimentation. But we claim that a mixed approach is obviously superior to a pure FP approach.


Despite the fact that the document is formulated in a pseudo-academic style, I personally conclude this is junk science. I don’t argue with the authors’ conclusions: I enjoy writing mixed programs in Ruby, and on this very blog I showed how to emulate lazy evaluation using objects.

I do think it’s valuable to show techniques like this. I think it reaches out to the Blub community and shows them that you can expand your horizons without having to throw away everything you’ve learned about your current programming language.

If it gets even one person to read WhyFP for themselves, it’s a net win to share these thoughts with the world.

But conducting research with a bias, performing an experiment that disproves your conjecture, then hand waving the experiment’s results away is not science. It’s just an opinion, nothing more, no different than the stuff I write here in raganwald.

The value of the authors’ academic experience ought to be to teach them how to do basic scientific research.

Where is the study conducted on carefully balanced groups, some who complete a task in pure idiomatic Java, some who complete the same task in Haskell, and some who mix techniques?

Where are the tables showing comparative program lengths, running times, average number of bugs needing correction, or other hard metrics comparing programs to each other?

This “paper” should not be confused with science.
 

Comments on “We don't need no stinkin' Scientific Method:
You're being far, far too friendly. That paper was nothing more than ignorance.
 
Didn't Guy Steele already make the same point as this paper, but much more eloquently with this koan http://people.csail.mit.edu/gregs/ll1-discuss-archive-html/msg03277.html ?
 
I am learning erlang and this WhyFP paper gives greater context. Thanks!
 
It was a letter from Anton van Straaten, no Guy Steele.
 
you need to get off your high horse.

i am one of the authors. when i posted it online i made it very clear that i didn't even believe the conclusions of the paper, and that i only wrote it because that's what the prof wanted. (yeah, we were essentially asked to prove a pre-specified conclusion. i suppose we could have concluded the opposite, but i don't think we'd have made an A.)

it was a fucking class project, it wasn't science.

i personally wouldn't use java if you paid me to. in fact i spend a lot of time trying to convince other people to switch to functional programming. if you read my blog you'd see i rave about it all the time.
 
when i posted it online i made it very clear that i didn't even believe the conclusions of the paper, and that i only wrote it because that's what the prof wanted.

How unfortunate, then, that I read the paper without the disclaimer that although your wrote it and affixed your name to it, it doesn't reflect your actual beliefs.

But your beliefs are not what I criticized here. I mistakenly criticized the paper thinking it was a paper, not a class submission.

I suppose if it is not a "real" paper then it need not be held to the same standard as the actual paper your class project criticized.

I will annotate my post accordingly.
 
when i posted it online i made it very clear that i didn't even believe the conclusions of the paper, and that i only wrote it because that's what the prof wanted.

Arvind, I, too, can't see the disclaimer. Given that the 'net never forgets I urge you to make the disclaimer very, very obvious indeed or else this thing will likely become huge embarrassment to you in future.

But more alarming is this:

(yeah, we were essentially asked to prove a pre-specified conclusion. i suppose we could have concluded the opposite, but i don't think we'd have made an A.)


Unless the class in question was a class in rhetoric then this is a totally outrageous thing for a professor to require of their students.
 
I suppose if it is not a "real" paper then it need not be held to the same standard as the actual paper your class project criticized.

You bet. It needn't be held to any standard except the grade it made :)

I've already said I don't program in Java. Also, it is NOT SCIENCE. Would you stop beating a dead horse please.


Unless the class in question was a class in rhetoric then this is a totally outrageous thing for a professor to require of their students.

Yeah, yeah, yeah, in theory. In practice this sort of thing happens all the time. The prof in question had some research work along the same lines, and it was clear what he wanted. Four groups, or about half the class, picked the same project to work on, probably because there was no doubt about what was expected. And guess what -- all four reached the same conclusion :)

Given that the 'net never forgets I urge you to make the disclaimer very, very obvious indeed or else this thing will likely become huge embarrassment to you in future.

Oh, I seriously doubt that :) I'm not in this line of research (I do crypto/security.. or try to), and besides this doesn't even come close to some of my other accomplishments in the realm of saying stupid things online.
 
keithb: nevertheless, i have added a disclaimer on the page. thanks for the suggestion.
 
Arvind:

I regret that my critique, even after modification, troubles you.

That being said, I suspect that we will both have forgotten all about it by next Tuesday at the latest.

Cheers...
 
Arvind, I'm confused. You contend that you should be judged more leniently because instead of putting your name to an honest if unpopular opinion, you sold your spine for a passing grade? Is your integrity really worth that much less than your GPA?
 




<< Home
Reg Braithwaite


Recent Writing
Homoiconic Technical Writing / raganwald.posterous.com

Books
What I‘ve Learned From Failure / Kestrels, Quirky Birds, and Hopeless Egocentricity

Share
rewrite_rails / andand / unfold.rb / string_to_proc.rb / dsl_and_let.rb / comprehension.rb / lazy_lists.rb

Beauty
IS-STRICTLY-EQUIVALENT-TO-A / Spaghetti-Western Coding / Golf is a good program spoiled / Programming conventions as signals / Not all functions should be object methods

The Not So Big Software Design / Writing programs for people to read / Why Why Functional Programming Matters Matters / But Y would I want to do a thing like this?

Work
The single most important thing you must do to improve your programming career / The Naïve Approach to Hiring People / No Disrespect / Take control of your interview / Three tips for getting a job through a recruiter / My favourite interview question

Management
Exception Handling in Software Development / What if powerful languages and idioms only work for small teams? / Bricks / Which theory fits the evidence? / Still failing, still learning / What I’ve learned from failure

Notation
The unary ampersand in Ruby / (1..100).inject(&:+) / The challenge of teaching yourself a programming language / The significance of the meta-circular interpreter / Block-Structured Javascript / Haskell, Ruby and Infinity / Closures and Higher-Order Functions

Opinion
Why Apple is more expensive than Amazon / Why we are the biggest obstacles to our own growth / Is software the documentation of business process mistakes? / We have lost control of the apparatus / What I’ve Learned From Sales I, II, III

Whimsey
The Narcissism of Small Code Differences / Billy Martin’s Technique for Managing his Manager / Three stories about The Tao / Programming Language Stories / Why You Need a Degree to Work For BigCo

History
06/04 / 07/04 / 08/04 / 09/04 / 10/04 / 11/04 / 12/04 / 01/05 / 02/05 / 03/05 / 04/05 / 06/05 / 07/05 / 08/05 / 09/05 / 10/05 / 11/05 / 01/06 / 02/06 / 03/06 / 04/06 / 05/06 / 06/06 / 07/06 / 08/06 / 09/06 / 10/06 / 11/06 / 12/06 / 01/07 / 02/07 / 03/07 / 04/07 / 05/07 / 06/07 / 07/07 / 08/07 / 09/07 / 10/07 / 11/07 / 12/07 / 01/08 / 02/08 / 03/08 / 04/08 / 05/08 / 06/08 / 07/08 /