The Stupidity of Crowds, Part I: The Wisdom of Crowds
In The Wisdom of Crowds
, James Surowiecki describes how a large crowd of people can be smarter than any one of the crowd’s members, including its experts.
The dynamics of crowd expertise are vital to the collaborative mechanisms that drive social networking, search engine rankings, and everybody’s favourite whipping boy, social news sites
If you aren’t familiar with the notion, there are a couple of ideas that you may find counter-intuitive. For one thing, we are conditioned to believe that experts are smarter than the hoi-polloi. And if we want even more cowbell^H^H^H^H^H^H^H expertise, we impanel a committee of experts to pool their expertise.
In Part II, we will look at how crowds go wrong, including why committees are stupider than both their individual experts and the crowd at large. We’ll find out why you should never Ask Reddit for feedback about your software or business idea, and for kicks we’ll revisit the Game of Technology Survivor
.The Wisdom of Crowds
If you want to extract the actual wisdom of a crowd, you need to aggregate each member’s independent sincere opinion.
First, you need each member’s
opinion: if your mechanism for sampling the crowd’s wisdom is biased, you will not get a good result. You may think you are getting a representative sample, or that you are only getting the smartest people in the crowd, but you will actually just get a biased sample that is wrong. The “expert’s committee” is flawed because it’s a committee, but also because it is made up of experts.
Take project management, for example. It’s pretty easy to identify experts. So-and-so writes a blog about shipping software. So-and-so has actually shipped a lot of software. So-and-so has a piece of paper certifying expertise in project management. So-and-so is actually managing a project of interest.
And if we are looking at a particular project, if we have experts running the project, all they do all day long is think about what will be ready by when. So you would think that we could get the most accurate information about what will be ready by when if we ask the experts. And that’s how we constantly get software development projects wrong.
An expert is better than the average
member of the crowd. Perhaps even better than any other member of the crowd. But that is not the same thing as saying that the expert is better than the entire crowd. There are a number of reasons for this. One of the most important reasons the crowd can outperform its most expert member is that information is not perfectly disseminated. If the expert knew everything, they might be better than the crowd. But they don’t know everything, and neither does any other member of the crowd. You only get the complete picture when you find a way to poll the entire crowd, not just the ones you think are the experts.
This is absolutely the case with project management: that’s why most of our processes for project management revolve around communication, getting individuals to communicate what they know about the project to the experts managing the project. The first key to obtaining the wisdom of the crowd is to harness the wisdom of the entire crowd.
You also need each member’s independent
opinion, as we discussed above. If you gather everyone in a room and ask them to vote, you get different results than if you ask them privately. You especially get different results than if each person jealously tries to hide what they think from each other: you get better results when each team member has no idea what the other members think. This sounds antithetical to team spirit and effectiveness: the whole point of communication is for team to tell each other what they know. But if team members are communicating in ‘real time with high bandwidth,’ you aren’t going to get independent opinions, even if you poll them privately.
For that reason, you often have to go beyond a team if you want to know how the team is going. You can’t get independent opinions by asking the team itself.
Finally, you need sincere
opinions. This is almost impossible to achieve by asking people. As Greg House says, “Everybody lies.” My experience with software development is that people lie and they don’t even know they are lying! People say what they think the boss wants to hear, or what will get the meeting finished so they can get back to playing SimCity
. They especially will say whatever they think is socially acceptable to say
, even if they are filling out an anonymous poll.
Before continuing, please give me your feedback:
Do you see how the results are hidden from you before asking you to answer? That is Statistics 101: If you tell people how other people have voted, you influence the results. This effect is so important, it is often the only
thing being debated in a democratic election: which person is “most likely to win.” That’s because the crowd tends to vote for whomever they perceive is going to win any ways, so each candidate struggles mightily to persuade everyone that everyone else is going to vote for them.
As I write this, Hillary Rodham Clinton is claiming that although Barak Obama polls well, that he will lose the presidential election if nominated by the Democrats. Her argument is that although Americans want to be seen as voting for the best person, in actuality they won’t vote for an African-American. Whether true or not, this argument illustrates that people will say what they think they ought to say, even in anonymous privacy.Extracting the wisdom
Right now, all of the challenge of extracting the wisdom of a crowd goes into finding a way to get the entire crowd to give independent and sincere opinions. And all of the stupidity of crowds arises from mechanisms that bugle one or more of these three points.
Please return for Part II, where we will look at various mechanisms for extracting the expertise of a crowd and see why biased samples, lack of independence, or incentives to lie all create a crowd that is stupider than its members, not smarter.