(This is a snapshot of my old weblog. New posts and selected republished essays can be found at raganwald.com.)
Saturday, February 16, 2008
The Stupidity of Crowds, Part I: The Wisdom of Crowds
In The Wisdom of Crowds
, James Surowiecki describes how a large crowd of people can be smarter than any one of the crowd’s members, including its experts.
The dynamics of crowd expertise are vital to the collaborative mechanisms that drive social networking, search engine rankings, and everybody’s favourite whipping boy, social news sites.
If you aren’t familiar with the notion, there are a couple of ideas that you may find counter-intuitive. For one thing, we are conditioned to believe that experts are smarter than the hoi-polloi. And if we want even more cowbell^H^H^H^H^H^H^H expertise, we impanel a committee of experts to pool their expertise.
In Part II, we will look at how crowds go wrong, including why committees are stupider than both their individual experts and the crowd at large. We’ll find out why you should never Ask Reddit for feedback about your software or business idea, and for kicks we’ll revisit the Game of Technology Survivor.
The Wisdom of Crowds
If you want to extract the actual wisdom of a crowd, you need to aggregate each member’s independent sincere opinion.
First, you need each member’s opinion: if your mechanism for sampling the crowd’s wisdom is biased, you will not get a good result. You may think you are getting a representative sample, or that you are only getting the smartest people in the crowd, but you will actually just get a biased sample that is wrong. The “expert’s committee” is flawed because it’s a committee, but also because it is made up of experts.
Take project management, for example. It’s pretty easy to identify experts. So-and-so writes a blog about shipping software. So-and-so has actually shipped a lot of software. So-and-so has a piece of paper certifying expertise in project management. So-and-so is actually managing a project of interest.
And if we are looking at a particular project, if we have experts running the project, all they do all day long is think about what will be ready by when. So you would think that we could get the most accurate information about what will be ready by when if we ask the experts. And that’s how we constantly get software development projects wrong.
An expert is better than the average member of the crowd. Perhaps even better than any other member of the crowd. But that is not the same thing as saying that the expert is better than the entire crowd. There are a number of reasons for this. One of the most important reasons the crowd can outperform its most expert member is that information is not perfectly disseminated. If the expert knew everything, they might be better than the crowd. But they don’t know everything, and neither does any other member of the crowd. You only get the complete picture when you find a way to poll the entire crowd, not just the ones you think are the experts.
This is absolutely the case with project management: that’s why most of our processes for project management revolve around communication, getting individuals to communicate what they know about the project to the experts managing the project. The first key to obtaining the wisdom of the crowd is to harness the wisdom of the entire crowd.
You also need each member’s independent opinion, as we discussed above. If you gather everyone in a room and ask them to vote, you get different results than if you ask them privately. You especially get different results than if each person jealously tries to hide what they think from each other: you get better results when each team member has no idea what the other members think. This sounds antithetical to team spirit and effectiveness: the whole point of communication is for team to tell each other what they know. But if team members are communicating in ‘real time with high bandwidth,’ you aren’t going to get independent opinions, even if you poll them privately.
For that reason, you often have to go beyond a team if you want to know how the team is going. You can’t get independent opinions by asking the team itself.
Finally, you need sincere opinions. This is almost impossible to achieve by asking people. As Greg House says, “Everybody lies.” My experience with software development is that people lie and they don’t even know they are lying! People say what they think the boss wants to hear, or what will get the meeting finished so they can get back to playing SimCity. They especially will say whatever they think is socially acceptable to say, even if they are filling out an anonymous poll.
Before continuing, please give me your feedback:
Do you see how the results are hidden from you before asking you to answer? That is Statistics 101: If you tell people how other people have voted, you influence the results. This effect is so important, it is often the only thing being debated in a democratic election: which person is “most likely to win.” That’s because the crowd tends to vote for whomever they perceive is going to win any ways, so each candidate struggles mightily to persuade everyone that everyone else is going to vote for them.
As I write this, Hillary Rodham Clinton is claiming that although Barak Obama polls well, that he will lose the presidential election if nominated by the Democrats. Her argument is that although Americans want to be seen as voting for the best person, in actuality they won’t vote for an African-American. Whether true or not, this argument illustrates that people will say what they think they ought to say, even in anonymous privacy.
Extracting the wisdom
Right now, all of the challenge of extracting the wisdom of a crowd goes into finding a way to get the entire crowd to give independent and sincere opinions. And all of the stupidity of crowds arises from mechanisms that bugle one or more of these three points.
Please return for Part II, where we will look at various mechanisms for extracting the expertise of a crowd and see why biased samples, lack of independence, or incentives to lie all create a crowd that is stupider than its members, not smarter.
In The Wisdom of Crowds
, James Surowiecki describes how a large crowd of people can be smarter than any one of the crowd’s members, including its experts.The dynamics of crowd expertise are vital to the collaborative mechanisms that drive social networking, search engine rankings, and everybody’s favourite whipping boy, social news sites.
If you aren’t familiar with the notion, there are a couple of ideas that you may find counter-intuitive. For one thing, we are conditioned to believe that experts are smarter than the hoi-polloi. And if we want even more cowbell^H^H^H^H^H^H^H expertise, we impanel a committee of experts to pool their expertise.
In Part II, we will look at how crowds go wrong, including why committees are stupider than both their individual experts and the crowd at large. We’ll find out why you should never Ask Reddit for feedback about your software or business idea, and for kicks we’ll revisit the Game of Technology Survivor.
The Wisdom of Crowds
If you want to extract the actual wisdom of a crowd, you need to aggregate each member’s independent sincere opinion.
First, you need each member’s opinion: if your mechanism for sampling the crowd’s wisdom is biased, you will not get a good result. You may think you are getting a representative sample, or that you are only getting the smartest people in the crowd, but you will actually just get a biased sample that is wrong. The “expert’s committee” is flawed because it’s a committee, but also because it is made up of experts.
Take project management, for example. It’s pretty easy to identify experts. So-and-so writes a blog about shipping software. So-and-so has actually shipped a lot of software. So-and-so has a piece of paper certifying expertise in project management. So-and-so is actually managing a project of interest.
And if we are looking at a particular project, if we have experts running the project, all they do all day long is think about what will be ready by when. So you would think that we could get the most accurate information about what will be ready by when if we ask the experts. And that’s how we constantly get software development projects wrong.
An expert is better than the average member of the crowd. Perhaps even better than any other member of the crowd. But that is not the same thing as saying that the expert is better than the entire crowd. There are a number of reasons for this. One of the most important reasons the crowd can outperform its most expert member is that information is not perfectly disseminated. If the expert knew everything, they might be better than the crowd. But they don’t know everything, and neither does any other member of the crowd. You only get the complete picture when you find a way to poll the entire crowd, not just the ones you think are the experts.
This is absolutely the case with project management: that’s why most of our processes for project management revolve around communication, getting individuals to communicate what they know about the project to the experts managing the project. The first key to obtaining the wisdom of the crowd is to harness the wisdom of the entire crowd.
You also need each member’s independent opinion, as we discussed above. If you gather everyone in a room and ask them to vote, you get different results than if you ask them privately. You especially get different results than if each person jealously tries to hide what they think from each other: you get better results when each team member has no idea what the other members think. This sounds antithetical to team spirit and effectiveness: the whole point of communication is for team to tell each other what they know. But if team members are communicating in ‘real time with high bandwidth,’ you aren’t going to get independent opinions, even if you poll them privately.
For that reason, you often have to go beyond a team if you want to know how the team is going. You can’t get independent opinions by asking the team itself.
Finally, you need sincere opinions. This is almost impossible to achieve by asking people. As Greg House says, “Everybody lies.” My experience with software development is that people lie and they don’t even know they are lying! People say what they think the boss wants to hear, or what will get the meeting finished so they can get back to playing SimCity. They especially will say whatever they think is socially acceptable to say, even if they are filling out an anonymous poll.
Before continuing, please give me your feedback:
Do you see how the results are hidden from you before asking you to answer? That is Statistics 101: If you tell people how other people have voted, you influence the results. This effect is so important, it is often the only thing being debated in a democratic election: which person is “most likely to win.” That’s because the crowd tends to vote for whomever they perceive is going to win any ways, so each candidate struggles mightily to persuade everyone that everyone else is going to vote for them.
As I write this, Hillary Rodham Clinton is claiming that although Barak Obama polls well, that he will lose the presidential election if nominated by the Democrats. Her argument is that although Americans want to be seen as voting for the best person, in actuality they won’t vote for an African-American. Whether true or not, this argument illustrates that people will say what they think they ought to say, even in anonymous privacy.
Extracting the wisdom
Right now, all of the challenge of extracting the wisdom of a crowd goes into finding a way to get the entire crowd to give independent and sincere opinions. And all of the stupidity of crowds arises from mechanisms that bugle one or more of these three points.
Please return for Part II, where we will look at various mechanisms for extracting the expertise of a crowd and see why biased samples, lack of independence, or incentives to lie all create a crowd that is stupider than its members, not smarter.
Comments on “The Stupidity of Crowds, Part I: The Wisdom of Crowds”:
<< Home
Someone will still have to pose the question to be decided, organize the vote, and aggregate the results. A management expert, probably?
Wisdom of Crowds is a bad name, statistically it's rarely better than individual expert opinion. The better one is Aggregated Wisdom of Individually Polled Crowds.
But talking about wisdom of crowds vs experts is treating the symptom. The problem comes from the algorithm you pick to reach consensus.
The main algorithms are:
1. Trusted source. That would be your expert, proven person, benevolent dictator, etc.
2. Collect and aggregate results without bias.
3. Reiterate until the variance of opinions is less than significant. That would be your typical design by committee.
Mostly what you would see out there in the field is strong bias towards either #1 or #3.
Any good examples of #2 in software development?
But talking about wisdom of crowds vs experts is treating the symptom. The problem comes from the algorithm you pick to reach consensus.
The main algorithms are:
1. Trusted source. That would be your expert, proven person, benevolent dictator, etc.
2. Collect and aggregate results without bias.
3. Reiterate until the variance of opinions is less than significant. That would be your typical design by committee.
Mostly what you would see out there in the field is strong bias towards either #1 or #3.
Any good examples of #2 in software development?
Wisdom of Crowds is a bad name, statistically it's rarely better than individual expert opinion.
That's not what I'm reading, although I am not studying the original sources.
Any good examples of #2 in software development?
I planned on mentioning Tech lessons learned from the wisdom of crowds in Part II.
That's not what I'm reading, although I am not studying the original sources.
Any good examples of #2 in software development?
I planned on mentioning Tech lessons learned from the wisdom of crowds in Part II.
Start here: http://www.snopes.com/
Pick something at random, and tell that story to random people.
Discount anyone who believes a false story or disputes a true on as wrose than expert.
Discount anyone who disputes a false story or believes a true one on flawed logic (ask for explanation) as worse than expert.
Do dot discount anyone for running a search to find out if the story is true or false. What we care for are opinions, not the way people reach them.
Collect, analyze and there's your proof.
(Note that expert here does not denote self-appointed, X years of experience, highly promoted, or McKinsey consulting. Expert is someone with "a high degree of skill in or knowledge of a certain subject." So in this trial, the people behind snopes.com can act as indisputible experts)
Pick something at random, and tell that story to random people.
Discount anyone who believes a false story or disputes a true on as wrose than expert.
Discount anyone who disputes a false story or believes a true one on flawed logic (ask for explanation) as worse than expert.
Do dot discount anyone for running a search to find out if the story is true or false. What we care for are opinions, not the way people reach them.
Collect, analyze and there's your proof.
(Note that expert here does not denote self-appointed, X years of experience, highly promoted, or McKinsey consulting. Expert is someone with "a high degree of skill in or knowledge of a certain subject." So in this trial, the people behind snopes.com can act as indisputible experts)
very interesting thoughts.. Your thought on how the environment affects the opinion is very interesting....I would be interested in knowing how you think that works in the setting of say wikipedia...would that then mean that the first person to pen her thoughts there influences the subsequent thoughts on that topic?
You can read some of my thoughts on this at: http://indradhanush-laal.blogspot.com/2008/01/weblogs-and-wisdom-of-crowds.html
You can read some of my thoughts on this at: http://indradhanush-laal.blogspot.com/2008/01/weblogs-and-wisdom-of-crowds.html
<< Home
