(This is a snapshot of my old weblog. New posts and selected republished essays can be found at raganwald.com.)
Thursday, November 22, 2007
What if powerful languages and idioms only work for small teams?
What indeed? What if programming conventions and closures and meta-programming and expressive type systems and annotations and all of the other tools that give us the power to build powerful abstractions actually don’t scale to larger teams?
So what?
Perhaps we have somehow confused the cart with the horse. Teams exist to deliver working software. So, shouldn’t we choose the tools and processes proven to produce working software and build the team around them, rather than choosing the team and then dumbing the tools down to the level of whomever we hired last Tuesday?
The Seasoned Schemer is devoted to the myriad uses of first class functions. This book is approachable and a delight to read, but the ideas are provocative and when you close the back cover you will be able to compose programs from functions in powerful new ways.
All of our experience in the last sixty years has suggested that productivity drops off a cliff as team size increases. So, if you want more code from a larger team, you have to invest heavily in ways of extracting value out of unproductive people in an unproductive environment.
But what about the other direction? If everything we know about teams suggests that smaller teams have the greatest productivity, isn’t it a valid choice to invest heavily in tools and processes that are specifically tuned to maximizing the productivity of small teams?
“Slick Willie” Sutton is famous for stating the obvious. When asked why he robbed banks, he is alleged to have replied, “Because that’s where the money is.” If we know that smaller teams are more productive, why do we try to scale teams out in personnel rather than up in productivity? If we know that bug per line of code remains amazingly constant, why do we try to scale code out in verbosity rather than up in abstraction?
Some people obtain results they consider acceptable from very large teams, using B&D tools and processes. But there is another option, to go “where the money is,” to find ways to enlarge the productivity sweet spot rather than live without it.
Elsewhere:
I argue the flip side of the coin: Can expressive languages make code more accessible to the masses?
And two great perspectives from other bloggers: Why Small Software Teams Grow Large And Other Software Development Conundrums and Create software, not code.
What indeed? What if programming conventions and closures and meta-programming and expressive type systems and annotations and all of the other tools that give us the power to build powerful abstractions actually don’t scale to larger teams?
Two ranchers, a Canadian and an American, meet at a Cattle Convention. The American boasts “My ranch is so big, I can git’ in my truck and drive for a day in any direction, and I’m still on ma’ propuh-ty.”
The Canadian nods in sympathy. “I used to have a truck just like that.”
So what?
Perhaps we have somehow confused the cart with the horse. Teams exist to deliver working software. So, shouldn’t we choose the tools and processes proven to produce working software and build the team around them, rather than choosing the team and then dumbing the tools down to the level of whomever we hired last Tuesday?
The Seasoned Schemer
is devoted to the myriad uses of first class functions. This book is approachable and a delight to read, but the ideas are provocative and when you close the back cover you will be able to compose programs from functions in powerful new ways.But what about the other direction? If everything we know about teams suggests that smaller teams have the greatest productivity, isn’t it a valid choice to invest heavily in tools and processes that are specifically tuned to maximizing the productivity of small teams?
“Slick Willie” Sutton is famous for stating the obvious. When asked why he robbed banks, he is alleged to have replied, “Because that’s where the money is.” If we know that smaller teams are more productive, why do we try to scale teams out in personnel rather than up in productivity? If we know that bug per line of code remains amazingly constant, why do we try to scale code out in verbosity rather than up in abstraction?
Some people obtain results they consider acceptable from very large teams, using B&D tools and processes. But there is another option, to go “where the money is,” to find ways to enlarge the productivity sweet spot rather than live without it.
Elsewhere:
I argue the flip side of the coin: Can expressive languages make code more accessible to the masses?
And two great perspectives from other bloggers: Why Small Software Teams Grow Large And Other Software Development Conundrums and Create software, not code.
Comments on “What if powerful languages and idioms only work for small teams?”:
<< Home
The problem with large teams is that you generally end up with one of two choices: either you code for the lowest common denominator or you end up with a lot of programmers who don't understand the code base. Both are bad news.
This is the exact reason I don't want to work at a large software company. I don't want to end up at SAP or CGI or any of the large software farms around here because with their turnaround rate (CGI is often compared to McDonalds around here) they can't afford to have programmers who don't understand the code base.
So the final question is: how do you produce quality software if you're not allowed to do anything complex given that, like you said, the larger the code base, the more bugs are lurking. The answer most large companies have is layers of process. Layers of review. One senior programmer not doing much himself, spending the majority of his time supervising junior programmers.
I don't want to be a junior programmer and I want to be a senior programmer even less. Give me a small team that prides itself on giving high quality results quickly over all that any day. The only way you get that is by never compromising on your hires and by never being lazy.
This is the exact reason I don't want to work at a large software company. I don't want to end up at SAP or CGI or any of the large software farms around here because with their turnaround rate (CGI is often compared to McDonalds around here) they can't afford to have programmers who don't understand the code base.
So the final question is: how do you produce quality software if you're not allowed to do anything complex given that, like you said, the larger the code base, the more bugs are lurking. The answer most large companies have is layers of process. Layers of review. One senior programmer not doing much himself, spending the majority of his time supervising junior programmers.
I don't want to be a junior programmer and I want to be a senior programmer even less. Give me a small team that prides itself on giving high quality results quickly over all that any day. The only way you get that is by never compromising on your hires and by never being lazy.
There are some pieces of software that I could not see being developed by a small team. I'm talking about large scale software platforms like relational databases, operating systems, language platforms and office suites. There is no way you can develop software this sophisticated without a large team. I mean there may be only a few technical leads that are making all the creative design decisions. But you still need a large team of programmers to get the actual work done, not to mention testers, translators, artists and marketing just to name a few.
Now don't get me wrong, I'm a huge fan of abstraction and writing clean well documented code. In fact I think its exactly these practices that allow you to scale to a large team. Abstraction make code more modular and susceptible to chage. Readable code means someone else can figure out the code quickly and get productive maintaining it.
The trick is to not let any of the monkey coders make any decisions, and to review all the code they write.
Now don't get me wrong, I'm a huge fan of abstraction and writing clean well documented code. In fact I think its exactly these practices that allow you to scale to a large team. Abstraction make code more modular and susceptible to chage. Readable code means someone else can figure out the code quickly and get productive maintaining it.
The trick is to not let any of the monkey coders make any decisions, and to review all the code they write.
What about a large number of projects?
My team is small, but we easily have 5-10 independent projects per person. Most of them are fairly small, which means we only touch a given project every 4-6 months.
When this leads to questions about what the programming style du jour was when it was last opened.
What I need is not a way to heavily customize my language on a per project basis. I need a way to make completely unrelated projects 'feel' the same so I can quickly switch between them as the need arises.
I see that happening by having a lot of built-in power, not from having the ability to add power ala Lisp.
Question. Is Ruby on Rails popular because it is a dynamic language? Or is it popular because Rails provides a consistent platform that can be used across projects?
My team is small, but we easily have 5-10 independent projects per person. Most of them are fairly small, which means we only touch a given project every 4-6 months.
When this leads to questions about what the programming style du jour was when it was last opened.
What I need is not a way to heavily customize my language on a per project basis. I need a way to make completely unrelated projects 'feel' the same so I can quickly switch between them as the need arises.
I see that happening by having a lot of built-in power, not from having the ability to add power ala Lisp.
Question. Is Ruby on Rails popular because it is a dynamic language? Or is it popular because Rails provides a consistent platform that can be used across projects?
"There are some pieces of software that I could not see being developed by a small team. I'm talking about large scale software platforms like relational databases, operating systems, language platforms and office suites. There is no way you can develop software this sophisticated without a large team."
I disagree. Those are being already written by a hobbyist or a small group of them. I believe applications don't need to be "sophisticated and large scale", or in other words: bloated to hell. Actually, that may be a result of the said monkey coders. I think that for each "there's no way you can develop" there are one or more hackers that can do just that and practically overnight.
Besides, there's nothing inherently difficult in creating a huge piece of software. All you need to do is to create lots of code and if you're smart, you'll have the computer create the code for you. The difficult part is the core algorithms that really run each software, and those have to be written by the good programmers anyway. Like, for a database, an elegant transaction engine, scalable indexing backend, etc.
"The trick is to not let any of the monkey coders make any decisions, and to review all the code they write."
I believe the point in the article was to drop the monkey coders alltogether and not waste time reviewing their code, but instead let the good programmers use powerful tools and languages to write everything by themselves. I agree with that. Monkey programmers eat too much productivity, if you're serious about delivering and doing it well.
I disagree. Those are being already written by a hobbyist or a small group of them. I believe applications don't need to be "sophisticated and large scale", or in other words: bloated to hell. Actually, that may be a result of the said monkey coders. I think that for each "there's no way you can develop" there are one or more hackers that can do just that and practically overnight.
Besides, there's nothing inherently difficult in creating a huge piece of software. All you need to do is to create lots of code and if you're smart, you'll have the computer create the code for you. The difficult part is the core algorithms that really run each software, and those have to be written by the good programmers anyway. Like, for a database, an elegant transaction engine, scalable indexing backend, etc.
"The trick is to not let any of the monkey coders make any decisions, and to review all the code they write."
I believe the point in the article was to drop the monkey coders alltogether and not waste time reviewing their code, but instead let the good programmers use powerful tools and languages to write everything by themselves. I agree with that. Monkey programmers eat too much productivity, if you're serious about delivering and doing it well.
# posted by 
: 3:05 AM
: 3:05 AM
I've been thinking this over, and I'm starting to believe that what you are talking about is only wishful thinking.
Almost all successful software projects started small. But the longer they are used, the larger they grow. Eventually the change requests start coming in faster than the small team can think about them.
Note that I said "think about them". Not test them, not code them, not even design them. Literally faster than you can even look at the request and say whether or not it is a bad idea.
Really coding is such a small fraction of my day that even infinitely fast tools won't reduce my typical work day by more than roughly 20%.
Software either fails, stagnates or grows into a big ball of mud. If and when it grows to that size, the scalability of your tools matter a lot. (Though to be honest, it will inevitably grow bigger than any toolset can comfortably handle.)
"It isn't a large project until it has been in production for 5 years."
Almost all successful software projects started small. But the longer they are used, the larger they grow. Eventually the change requests start coming in faster than the small team can think about them.
Note that I said "think about them". Not test them, not code them, not even design them. Literally faster than you can even look at the request and say whether or not it is a bad idea.
Really coding is such a small fraction of my day that even infinitely fast tools won't reduce my typical work day by more than roughly 20%.
Software either fails, stagnates or grows into a big ball of mud. If and when it grows to that size, the scalability of your tools matter a lot. (Though to be honest, it will inevitably grow bigger than any toolset can comfortably handle.)
"It isn't a large project until it has been in production for 5 years."
Carmack is famous for saying that a team of 3 focused and creative people can accomplish almost anything.
If a team of 3 could really do anything, why do we have open source? Open source is the ultimate expression of large team development.
The problem is that small teams of the brightest people armed with the sharpest of tools are unable to bring to market certain classes of projects fast enough to retain relevancy. We are then faced with the problem of scaling up personnel despite how unproductive they will inevitably become. The Mythical Man-month contains a lucid paragraph on this.
So, at the risk of stating the obvious: Choose your team and your tools depending on the situation.
So, at the risk of stating the obvious: Choose your team and your tools depending on the situation.
Large teams produce crappy software regardless of tools.
""There are some pieces of software that I could not see being developed by a small team. I'm talking about large scale software platforms like relational databases, operating systems, language platforms and office suites. There is no way you can develop software this sophisticated without a large team."
There are great examples of small teams producing all of this stuff. Linux got written initially by a small team as did Unix. Delphi was written by a very small team. Quattro Pro was written by my very small team.
Put large teams on these things and progress slows until its hardly noticeable. Office 2007 was written by a large team as are all Microsoft's commercial products. That should be enough to tell you not to go there.
More on my blog about small teams:
http://smoothspan.wordpress.com/2007/10/17/to-build-better-software-you-need-fewer-people/
Cheers!
BW
""There are some pieces of software that I could not see being developed by a small team. I'm talking about large scale software platforms like relational databases, operating systems, language platforms and office suites. There is no way you can develop software this sophisticated without a large team."
There are great examples of small teams producing all of this stuff. Linux got written initially by a small team as did Unix. Delphi was written by a very small team. Quattro Pro was written by my very small team.
Put large teams on these things and progress slows until its hardly noticeable. Office 2007 was written by a large team as are all Microsoft's commercial products. That should be enough to tell you not to go there.
More on my blog about small teams:
http://smoothspan.wordpress.com/2007/10/17/to-build-better-software-you-need-fewer-people/
Cheers!
BW
There's always the EA-Maxis development model.
Have a small, skilled development team spend a long time producing a rock-solid core product, then turn over customizations and add-ins to bulky useless fast teams and sell them as 'expansion packs'.
Have a small, skilled development team spend a long time producing a rock-solid core product, then turn over customizations and add-ins to bulky useless fast teams and sell them as 'expansion packs'.
"shouldn't we choose the tools and processes proven to produce working software"
What does this mean? You're likely not going to get anyone to agree on the answer to this question.
It is less about the tools than it is the individuals comprising a team. A professional driver in a VW Beetle will outrun your mom in a Porsche every single time. Take five guys and have them use a bondage and discipline language like Java or Ruby, and they will still produce something solid.
What does this mean? You're likely not going to get anyone to agree on the answer to this question.
It is less about the tools than it is the individuals comprising a team. A professional driver in a VW Beetle will outrun your mom in a Porsche every single time. Take five guys and have them use a bondage and discipline language like Java or Ruby, and they will still produce something solid.
# posted by : 1:53 PM
: 1:53 PM
"There are great examples of small teams producing all of this stuff. Linux got written initially by a small team as did Unix. Delphi was written by a very small team. Quattro Pro was written by my very small team."
Linux isn't written by a small team any more.
Delphi and Quattro Pro are essentially dead products.
So back to my earlier point, successful projects invariably out-grow their original small teams.
Linux isn't written by a small team any more.
Delphi and Quattro Pro are essentially dead products.
So back to my earlier point, successful projects invariably out-grow their original small teams.
> Delphi and Quattro Pro are essentially dead products.
Delphi lives on in Visual Studio.net. And I think a lot of its success was precisely that a small team had to build a platform that leverages generic concepts to get somewhere. The community then used that lever to great effect. The Visual Studio.net core has the same feel, but the product admittedly shows a lot more larger-team polish and cruft.
Delphi lives on in Visual Studio.net. And I think a lot of its success was precisely that a small team had to build a platform that leverages generic concepts to get somewhere. The community then used that lever to great effect. The Visual Studio.net core has the same feel, but the product admittedly shows a lot more larger-team polish and cruft.
"Take five guys and have them use a bondage and discipline language like Java or Ruby"
Since when is Ruby a bondage and discipline language?
Small teams produce the software that eventually becomes larger software because small teams are capable of producing software that is a platform, on top of which other programmers can add. Larger teams have more trouble doing that because of the meetings, politics, and the general lowering of the average ability of the programmers involved (1 superstar vs. a mixed-bag).
Since when is Ruby a bondage and discipline language?
Small teams produce the software that eventually becomes larger software because small teams are capable of producing software that is a platform, on top of which other programmers can add. Larger teams have more trouble doing that because of the meetings, politics, and the general lowering of the average ability of the programmers involved (1 superstar vs. a mixed-bag).
Ok, I've been convinced that "sophisticated" software can be created by a small team. In fact I should know this because I'm on a small team and we are doing some pretty cutting edge stuff.
However, I work for a large software company and I see some huge teams that are consistently producing high quality products (obviously I don't work for Microsoft). I think the trick here is that the software is composed of several loosely coupled components and each component is developed by a smaller team. So the overall team is very large but it is actually a collection of small teams.
There is where abstraction comes into play. With the correct abstractions you can break software down into loosely coupled components, then have a small team focus on each component.
Lets say an open source project is started by just a few developers. If it ends up being an interesting project it might attract the interest of many other developers who want to work on it. Are you going to tell those developers to buzz off and that you don't want their help because you think that only a small team can be effective? No, that would be stupid, what you do is break the software down into components and have the developers form small teams around each component.
I don't believe that having a large team automatically means that the software will be crappy and bloated. There's nothing wrong with having a lot of developers on a project. It just needs to be handled correctly.
However, I work for a large software company and I see some huge teams that are consistently producing high quality products (obviously I don't work for Microsoft). I think the trick here is that the software is composed of several loosely coupled components and each component is developed by a smaller team. So the overall team is very large but it is actually a collection of small teams.
There is where abstraction comes into play. With the correct abstractions you can break software down into loosely coupled components, then have a small team focus on each component.
Lets say an open source project is started by just a few developers. If it ends up being an interesting project it might attract the interest of many other developers who want to work on it. Are you going to tell those developers to buzz off and that you don't want their help because you think that only a small team can be effective? No, that would be stupid, what you do is break the software down into components and have the developers form small teams around each component.
I don't believe that having a large team automatically means that the software will be crappy and bloated. There's nothing wrong with having a lot of developers on a project. It just needs to be handled correctly.
This topic strikes me of an agile theory - or at the very least based in some agile thinking. Powerful languages may actually work better in smaller teams. But doesn't this drive at what can be accomplished by the smaller team.
So in a smaller team assuming there is a decent programmer among them, lots of learning and knowledge transfer can go on. Something that is harder to do in larger teams.
In smaller teams lots of communication can happen easier. Larger teams can accomplish this but its a good bit harder.
Smaller teams can be born from a larger single team provided that sub division is possible. A page right out of the scrum handbook I imagine.
Since I fully believe in using the right tool for the job no matter the size of the team, I do not personally think that powerful languages are for small teams only. I do however believe that it would be harder in a really large 'SINGLE' team to make really good effective use of features and functionality of those languages in very large teams. In my mind, teams are 8-10 people always... need a larger group, form another team of 8-10 and figure out the communication links.
So in a smaller team assuming there is a decent programmer among them, lots of learning and knowledge transfer can go on. Something that is harder to do in larger teams.
In smaller teams lots of communication can happen easier. Larger teams can accomplish this but its a good bit harder.
Smaller teams can be born from a larger single team provided that sub division is possible. A page right out of the scrum handbook I imagine.
Since I fully believe in using the right tool for the job no matter the size of the team, I do not personally think that powerful languages are for small teams only. I do however believe that it would be harder in a really large 'SINGLE' team to make really good effective use of features and functionality of those languages in very large teams. In my mind, teams are 8-10 people always... need a larger group, form another team of 8-10 and figure out the communication links.
Saying "Teams exist to deliver working software." is as untrue as saying trucks exist to drive goods around.
Some teams exist to deliver working software. Others exist simply because there is a budget for them.
Assuming all teams exist to do good is not doing anything to fix any problems that may exist.
The most important thing to start with is to look at the people you're dealing with and how they handle responsibility. If something seems broken at this level there is not point in ignoring it and continuing fixes other things.
Some teams exist to deliver working software. Others exist simply because there is a budget for them.
Assuming all teams exist to do good is not doing anything to fix any problems that may exist.
The most important thing to start with is to look at the people you're dealing with and how they handle responsibility. If something seems broken at this level there is not point in ignoring it and continuing fixes other things.
# posted by : 3:36 AM
: 3:36 AM
Great back and forth. I went into some more of this on my blog:
http://smoothspan.wordpress.com/2007/11/27/why-small-software-teams-grow-large-and-other-software-development-conundrums/
Best,
BW
http://smoothspan.wordpress.com/2007/11/27/why-small-software-teams-grow-large-and-other-software-development-conundrums/
Best,
BW
Software quite often outlives the small, talented team that gives birth to it and gets inherited by a larger and less talented team doing maintenance and incremental development.
<< Home
