(This is a snapshot of my old weblog. New posts and selected republished essays can be found at raganwald.com.)
Tuesday, May 02, 2006
How to make programming hard for yourself
I posed a simple question: why do we have trouble admitting that programming is hard?
This generated a tempest in a teapot of comments. Most focused on the red herring of whether programming is an innate talent. That's actually irrelevant to the fundamental question.
Is programming actually hard?
The evidence is in that software development as an engineering practice is very hard. One person on reddit commented that he had worked for three successive game companies that shipped working software on time. Other than that comment, my own experience, the experience of everyone I've ever talked to, and the industry folklore is that software projects are almost always late, short of promised functionality, buggy, or all three.
Now I admit that lots of people think that this is a management problem. Their view is that the actual programming is quite tractable, it's just that getting people to work on the right things at the right time is the hard part of software development. This argument takes the same world view that the Egyptian engineers had about building pyramids: the labour is so easy that ignorant slaves can do it.
My view is that this is true for a certain class of programs. Most of these can be found in current business contexts. Quite honestly, most business applications are what I would call wide and shallow. There's a lot of work to be done, but little of it requires deep thinking. The challenges are around managing requirements and user expectations. These are hard problems, but they are not in the same class as, say, figuring out how to implement a fast join on two distributed hash tables.
I think that easy programming problems drive a lot of decisions about people and tools. If you perceive that a problem isn't very hard, why hire the 'cream' of the programming crop to solve it? And if you eschew the cream for a plain glass of 1% milk, won't you insist that the program be written in Blub instead of Python, Ruby, or Lisp?
This becomes a feedback loop. The 'B' players become hiring managers one day and they hire 'C' players. The 'C' players are asked to design applications, and they design things that they know how to build. This second effect is a huge lever pushing programming problems downwards in business.
Programs are written to solve problems posed by people, be they programmers, program managers, business analysts, or what-have-you. The difficulty of writing the programs is driven by the difficulty of the problems. The difficulty of the problems is limited by the imagination of the people posing the problems.
If your feedback loop is pushing difficulty downwards, the only problems you will have are the ones that can be solved by anyone who can pick up a Learn Blub in Twenty-One Days book.
Is this a problem?
Quite honestly, no. If what you want is a steady paycheque, and you can sleep at night knowing that your job might be outsourced one day, there's nothing wrong with chasing the easy work. It's no different than playing saxophone in a dance band at a wedding. Charlie Parker did it, you can do it.
But just because almost everybody does easy work doesn't mean there isn't hard work to be done. Hard problems are their own reward for some people, myself included (in my case I mean "hard for me," I know more than a few people that consider my hard problems to be recreational diversions). And some hard problems have lucrative consequences if you have a taste for starting companies.
Why it's hard to find hard problems
It's hard to find problems that are so challenging that you don't know how to solve them. It's especially hard if you also want a problem that can actually be solved. And it's very, very hard to find a hard problem that makes business sense, even in a startup. Most good startups solved the right problem rather than the hardest problem.
Very few people have the luxury of writing requirements for a program that they have no idea how to write, much less the imagination to see the results. Jump in your time machine and zoom back to 1980. Start asking people do sketch a word processor design on a napkin. How many people specify bit-mapped graphics and mouse-driven selection?
I compared programming to music and athletics. I suggested that programming is hard and the reason we don't see that is because we aren't competitive enough. That may be true, but now I sense a deeper challenge: in programming, we tend to pose questions for which we already know the answer.
In music, most people's reach far exceeds their grasp. It's quite easy to find pieces of music that are difficult or impossible for someone to play but nevertheless pleasing to that same person's ear.
If you are a wanna-be bassist and you listen to Jaco Pastorius' harmonics on "Portrait of Tracy," it is immediately obvious to you that you could be better and that being better is worthwhile. I leave it to your imagination to draw the same parallel to sports and athletics.
But in programming... we have a nasty habit of being very satisfied with the tools we already know. After all, they are good enough for all of the programs we have ever written. And, surprise surprise, they just might be good enough for all of the programs we will ever want to write.
How to make life hard for yourself
Most of us need outside forces to shake us up. Many bassists were shocked in 1976 when they heard Portrait of Tracy. Harmonics were used as accents here and there, but nobody dared to play an entire piece with harmonics as the main technique. This shook things up, and now playing harmonics is considered a basic skill on electric bass (although it's still fiendishly hard to play Portrait of Tracy!)
For all of my professed admiration of Ruby on rails, I personally don't think that easier and more productive CRUD application writing will shake things up. I personally care very much about writing applications in a tenth of the time, but using Rails is like listening to Jaco Pastorius. The real learning experience comes when you try to duplicate the feat.
If you want Rails to help you improve as a programmer, you need to look at the source code and figure out how to use Ruby to write Domain Specific Languages. Next, you'll have to write a few DSLs to become immersed in how programming changes when you can write DSLs. Now you're ready to look for a problem that is barely possible if you can solve it with a DSL.
Before you knew how to write your own languages, you probably couldn't design a program that couldn't be written without a DSL. Now you can.
This general template works surprisingly well: Learning things that force you to solve existing problems in new ways will help you find hard problems that can only be solved with the aid of your new techniques.
So when you're done learning how to write DSLs just like Rails, go ahead and write a web application using continuations as the basic metaphor. And don't forget to look under the hood and figure out how the framework makes continuations scale in a web context.
Come to think of it, figure out how to write continuation-based Rails applications that live on a Google-scaled grid and you might be headed towards a place where the inhabitants have no fear of hard.
I'll see you there.
Update: I found an early take of Jaco playing Portrait of Tracy (MP3). Nice.
I posed a simple question: why do we have trouble admitting that programming is hard?
This generated a tempest in a teapot of comments. Most focused on the red herring of whether programming is an innate talent. That's actually irrelevant to the fundamental question.
Is programming actually hard?
The evidence is in that software development as an engineering practice is very hard. One person on reddit commented that he had worked for three successive game companies that shipped working software on time. Other than that comment, my own experience, the experience of everyone I've ever talked to, and the industry folklore is that software projects are almost always late, short of promised functionality, buggy, or all three.
Now I admit that lots of people think that this is a management problem. Their view is that the actual programming is quite tractable, it's just that getting people to work on the right things at the right time is the hard part of software development. This argument takes the same world view that the Egyptian engineers had about building pyramids: the labour is so easy that ignorant slaves can do it.
My view is that this is true for a certain class of programs. Most of these can be found in current business contexts. Quite honestly, most business applications are what I would call wide and shallow. There's a lot of work to be done, but little of it requires deep thinking. The challenges are around managing requirements and user expectations. These are hard problems, but they are not in the same class as, say, figuring out how to implement a fast join on two distributed hash tables.
I think that easy programming problems drive a lot of decisions about people and tools. If you perceive that a problem isn't very hard, why hire the 'cream' of the programming crop to solve it? And if you eschew the cream for a plain glass of 1% milk, won't you insist that the program be written in Blub instead of Python, Ruby, or Lisp?
This becomes a feedback loop. The 'B' players become hiring managers one day and they hire 'C' players. The 'C' players are asked to design applications, and they design things that they know how to build. This second effect is a huge lever pushing programming problems downwards in business.
Programs are written to solve problems posed by people, be they programmers, program managers, business analysts, or what-have-you. The difficulty of writing the programs is driven by the difficulty of the problems. The difficulty of the problems is limited by the imagination of the people posing the problems.
If your feedback loop is pushing difficulty downwards, the only problems you will have are the ones that can be solved by anyone who can pick up a Learn Blub in Twenty-One Days book.
Is this a problem?
Quite honestly, no. If what you want is a steady paycheque, and you can sleep at night knowing that your job might be outsourced one day, there's nothing wrong with chasing the easy work. It's no different than playing saxophone in a dance band at a wedding. Charlie Parker did it, you can do it.
But just because almost everybody does easy work doesn't mean there isn't hard work to be done. Hard problems are their own reward for some people, myself included (in my case I mean "hard for me," I know more than a few people that consider my hard problems to be recreational diversions). And some hard problems have lucrative consequences if you have a taste for starting companies.
Why it's hard to find hard problems
It's hard to find problems that are so challenging that you don't know how to solve them. It's especially hard if you also want a problem that can actually be solved. And it's very, very hard to find a hard problem that makes business sense, even in a startup. Most good startups solved the right problem rather than the hardest problem.
Very few people have the luxury of writing requirements for a program that they have no idea how to write, much less the imagination to see the results. Jump in your time machine and zoom back to 1980. Start asking people do sketch a word processor design on a napkin. How many people specify bit-mapped graphics and mouse-driven selection?
I compared programming to music and athletics. I suggested that programming is hard and the reason we don't see that is because we aren't competitive enough. That may be true, but now I sense a deeper challenge: in programming, we tend to pose questions for which we already know the answer.
In music, most people's reach far exceeds their grasp. It's quite easy to find pieces of music that are difficult or impossible for someone to play but nevertheless pleasing to that same person's ear.
If you are a wanna-be bassist and you listen to Jaco Pastorius' harmonics on "Portrait of Tracy," it is immediately obvious to you that you could be better and that being better is worthwhile. I leave it to your imagination to draw the same parallel to sports and athletics.
But in programming... we have a nasty habit of being very satisfied with the tools we already know. After all, they are good enough for all of the programs we have ever written. And, surprise surprise, they just might be good enough for all of the programs we will ever want to write.
How to make life hard for yourself
Most of us need outside forces to shake us up. Many bassists were shocked in 1976 when they heard Portrait of Tracy. Harmonics were used as accents here and there, but nobody dared to play an entire piece with harmonics as the main technique. This shook things up, and now playing harmonics is considered a basic skill on electric bass (although it's still fiendishly hard to play Portrait of Tracy!)
For all of my professed admiration of Ruby on rails, I personally don't think that easier and more productive CRUD application writing will shake things up. I personally care very much about writing applications in a tenth of the time, but using Rails is like listening to Jaco Pastorius. The real learning experience comes when you try to duplicate the feat.
If you want Rails to help you improve as a programmer, you need to look at the source code and figure out how to use Ruby to write Domain Specific Languages. Next, you'll have to write a few DSLs to become immersed in how programming changes when you can write DSLs. Now you're ready to look for a problem that is barely possible if you can solve it with a DSL.
Before you knew how to write your own languages, you probably couldn't design a program that couldn't be written without a DSL. Now you can.
This general template works surprisingly well: Learning things that force you to solve existing problems in new ways will help you find hard problems that can only be solved with the aid of your new techniques.
So when you're done learning how to write DSLs just like Rails, go ahead and write a web application using continuations as the basic metaphor. And don't forget to look under the hood and figure out how the framework makes continuations scale in a web context.
Come to think of it, figure out how to write continuation-based Rails applications that live on a Google-scaled grid and you might be headed towards a place where the inhabitants have no fear of hard.
I'll see you there.
Update: I found an early take of Jaco playing Portrait of Tracy (MP3). Nice.
Labels: passion
Comments on “How to make programming hard for yourself”:
<< Home
Hmmm, interesting point you make. Personally I think that finding a problem hard is a matter of perspective. What I might find hard, you may not and vise versa.
IMHO Programming is not hard unless you are not geared into thinking in the 'languages' parlance. This was not more clear to me then when my wife picked up one of my Java learning books a few years back and said "this looks easy..." which brought us into a discussion of what a pointer was and how in the world could something exist but be NULL which wasn't 0 (zero) and her little head exploded. Programming like most things in life is a learning experience and you are either interested in learning and expanding - Solving the hard problems - or you are interested in sitting on what you know and doing the very least you can to get by.
I suppose you might say that programming and solving the hard problems is like comparing the worlds do-ers and the non-doers. Those who strive and those who are content. Programming itself is not 'hard' on its own, it is hard to do RIGHT or correctly because like the human language there are so many ways of accomplishing the same task. Solving the HARD problems well is the hard part of programming.
IMHO Programming is not hard unless you are not geared into thinking in the 'languages' parlance. This was not more clear to me then when my wife picked up one of my Java learning books a few years back and said "this looks easy..." which brought us into a discussion of what a pointer was and how in the world could something exist but be NULL which wasn't 0 (zero) and her little head exploded. Programming like most things in life is a learning experience and you are either interested in learning and expanding - Solving the hard problems - or you are interested in sitting on what you know and doing the very least you can to get by.
I suppose you might say that programming and solving the hard problems is like comparing the worlds do-ers and the non-doers. Those who strive and those who are content. Programming itself is not 'hard' on its own, it is hard to do RIGHT or correctly because like the human language there are so many ways of accomplishing the same task. Solving the HARD problems well is the hard part of programming.
# posted by 
: 4:35 PM
: 4:35 PM
Great post, very insightful and it makes me wanna go out and think about finding harder problems to test against.
Keep writing.
Keep writing.
# posted by : 9:08 PM
: 9:08 PM
Thought-provoking post.
To me, much of professional programming is not "intellectually hard" but it is a skilled craft and thus can't be done by anyone.
A lot of the difficulty comes not so much from intellectual hardness as from discipline, attention to detail, team dynamics, etc. This is why I think there are a lot of productivity gains to be had by improving processes - for example, using test-driven development and having coding standards and code reviews. Also, choosing the right tool for the job is a huge win. I think too many programmers look at languages as a "religious" issue, preferring one over all others and defending it with great zeal. I like to think of them as tools in our toolbox and knowing the strengths of each one means that you can choose the right tool for the job, which can often make things way easier.
To me, much of professional programming is not "intellectually hard" but it is a skilled craft and thus can't be done by anyone.
A lot of the difficulty comes not so much from intellectual hardness as from discipline, attention to detail, team dynamics, etc. This is why I think there are a lot of productivity gains to be had by improving processes - for example, using test-driven development and having coding standards and code reviews. Also, choosing the right tool for the job is a huge win. I think too many programmers look at languages as a "religious" issue, preferring one over all others and defending it with great zeal. I like to think of them as tools in our toolbox and knowing the strengths of each one means that you can choose the right tool for the job, which can often make things way easier.
First, I enjoy your blog.
In general I agree with your main point, which seems to be, seek out difficult/new things, which expands both your experience and your skill set.
I also agree that most business applications are wide but shallow, although I would say that I think getting them done right often can be hard, and maybe even hard in terms of the "there are important lessons to learn here" hard, rather than hard in terms of 1000 uninteresting details which need to be done correctly.
Still, I think except in the case of truly difficult algorithmic problems, a good process and craftsmanship skills are more important than raw talent.
I also think you're a bit harsh on companies that companies that want "Blub" programmers. The right tool for the task is obviously very important. However, if programmers at the company are unfamiliar with "the right tool" it's not. Of course, that makes it that much more important that a company supports and encourages employees to keep their skills up, something which is unfortunately rare. And also costs.
So what am I trying to say? Mostly agreement, I guess -- becoming a better programmer does involve stretching your skills, your languages, your processes, and your brain.
Ok, maybe not so easy.
jbstjohn
In general I agree with your main point, which seems to be, seek out difficult/new things, which expands both your experience and your skill set.
I also agree that most business applications are wide but shallow, although I would say that I think getting them done right often can be hard, and maybe even hard in terms of the "there are important lessons to learn here" hard, rather than hard in terms of 1000 uninteresting details which need to be done correctly.
Still, I think except in the case of truly difficult algorithmic problems, a good process and craftsmanship skills are more important than raw talent.
I also think you're a bit harsh on companies that companies that want "Blub" programmers. The right tool for the task is obviously very important. However, if programmers at the company are unfamiliar with "the right tool" it's not. Of course, that makes it that much more important that a company supports and encourages employees to keep their skills up, something which is unfortunately rare. And also costs.
So what am I trying to say? Mostly agreement, I guess -- becoming a better programmer does involve stretching your skills, your languages, your processes, and your brain.
Ok, maybe not so easy.
jbstjohn
# posted by : 9:35 AM
: 9:35 AM
Reg, you have described the basic difference between Art and Manufacturing. Manufacturing is about a process, you have a problem, you do these things to investigate it, and then you do these other things to solve it. Process can be learned and taught. You can go to school for development process.
Art is about creativity. There are techniques to fleshing out the art piece, that may be similar to a Manufaturing process, but there is no process to get the goal. For example, to make a wood carving you may use the widdling (sp?) technique, but that doesn't tell you how to make a sculpture of your pet dog out of wood. The thing about creativity is that it can't be taught but it can be learned. At least I think it can.
But this has nothing to do with programming and that's where I disagree with pretty much the entire post. Engineering and programming isn't hard. Creativity is hard. Therefore, doing creative things with software is also hard. Programming is really the same as widdling, just a technique to help you carve your sculpture. It's you that have to come up with the sculpture.
Art is about creativity. There are techniques to fleshing out the art piece, that may be similar to a Manufaturing process, but there is no process to get the goal. For example, to make a wood carving you may use the widdling (sp?) technique, but that doesn't tell you how to make a sculpture of your pet dog out of wood. The thing about creativity is that it can't be taught but it can be learned. At least I think it can.
But this has nothing to do with programming and that's where I disagree with pretty much the entire post. Engineering and programming isn't hard. Creativity is hard. Therefore, doing creative things with software is also hard. Programming is really the same as widdling, just a technique to help you carve your sculpture. It's you that have to come up with the sculpture.
Gp:
Ummm... No. What I've described is manufacturing. Most companies use existing techniques in well understood ways and achieve the same results as anyone else.
But every once in a while someone poses a hard question, like "can we reduce our costs even further" or "can we customize every piece" or what-have-you, and they are forced to find new ways to do things.
My thesis is that they go to the edges of what is believed to be achievable and seek out the hard problems in manufacturing.
As a result, manufacturing, like software development, has a rich history of innovation and progress, including Ford's assembly line, Japan's Just-In-Time manufacturing, and now "Lean Manufacturing" and "Theory of Constraints."
Finding new ways to solve existing problems is certainly creative, but it belongs in the manufacturing world as much as in the art world.
And now for an ad hominem attack:
Hey! You're a freakin' Product Manager! Your JOB is to get creative and pose hard problems for your development team. Get back to work, and don't let me catch you posting to blogs until I hear your coders complaining that the software you've designed can't be delivered using existing development techniques in the timeframe you've specified :-)
Ummm... No. What I've described is manufacturing. Most companies use existing techniques in well understood ways and achieve the same results as anyone else.
But every once in a while someone poses a hard question, like "can we reduce our costs even further" or "can we customize every piece" or what-have-you, and they are forced to find new ways to do things.
My thesis is that they go to the edges of what is believed to be achievable and seek out the hard problems in manufacturing.
As a result, manufacturing, like software development, has a rich history of innovation and progress, including Ford's assembly line, Japan's Just-In-Time manufacturing, and now "Lean Manufacturing" and "Theory of Constraints."
Finding new ways to solve existing problems is certainly creative, but it belongs in the manufacturing world as much as in the art world.
And now for an ad hominem attack:
Hey! You're a freakin' Product Manager! Your JOB is to get creative and pose hard problems for your development team. Get back to work, and don't let me catch you posting to blogs until I hear your coders complaining that the software you've designed can't be delivered using existing development techniques in the timeframe you've specified :-)
I think that programming is hard on two different meters.
On the one hand, you have the typing part of programming, which is annoying and frustrating without discipline, and more or less the *easy* part of programming.
The hard part of programming usually doesn't involve any code, but thinking inside of and about systems, and then coming up with ways to build those new systems efficiently inside of the system that you're working on.
Of course, you can't really do the former without skill in the latter.
On the one hand, you have the typing part of programming, which is annoying and frustrating without discipline, and more or less the *easy* part of programming.
The hard part of programming usually doesn't involve any code, but thinking inside of and about systems, and then coming up with ways to build those new systems efficiently inside of the system that you're working on.
Of course, you can't really do the former without skill in the latter.
in many of those wide but shallow business application challenges, there are, somewhere out in the middle of the pond, uncharted depths which may even be entry-ways to complex caverns. If I'm accustomed to wearing waders to fish the pond , I'll just work my way around the deepest spots and never know what wonderful species inhabit the depths, and I can still be happy with the day's catch, if my main objective is to feed my family. There's not a return on investment in scuba equipment, unless my thrill is in new discoveries. If we hire our programmers to be competent productive fishermen, then we'll keep getting the same kinds of fish using the same tools.
# posted by : 6:42 PM
: 6:42 PM
Another great post.
I used to work in games where I was writing code to see if it was viable to use an inverse kinematics system on next-gen consoles.
Now I write business software.
Although the pay is far, far better, I'm bored rigid as the code is completely trivial by comparison.
If you want to make programming challenging (and fun), pick a hard problem in the first place.
For me, a hard problem is implementing a maths algorithm, for others it may be optimising and SQL join.
I used to work in games where I was writing code to see if it was viable to use an inverse kinematics system on next-gen consoles.
Now I write business software.
Although the pay is far, far better, I'm bored rigid as the code is completely trivial by comparison.
If you want to make programming challenging (and fun), pick a hard problem in the first place.
For me, a hard problem is implementing a maths algorithm, for others it may be optimising and SQL join.
# posted by : 6:06 AM
: 6:06 AM<< Home
