David East - Sampling bias, FDR, and The State of JS

See David East - Sampling bias, FDR, and The State of JS on davidea.st

Some time ago I read The Data Detective/How To Make The World Add Up which provided suggestion on how to look at data. One rule was to “ask who’s missing”, which suggests looking to see what data is included and excluded. An example he gives is of the publication Literary Digest who put out a massive poll to see who would win the 1936 US election between FDR and Alfred Landon. The poll with 2.4 million responses suggested that Landon would win, but he didn’t. The discrepancy is down to how the magazines readers (wealthier), differed from the general population (struggling with the Great Depression).

This came up recently as the state of JS/CSS surveys are being conducted. One big issue with these are that advertising a survey online, Twitter no less, is likely to be biased to whatever seems trendy there rather than a reflection of what’s actually used in the wild where most people are dark matter developers so to speak.

David does a good job outlining this. Not sure how you’d fix it other than to get some random rural developers to fill it. You can measure actual usage like the Web Almanac does, but it’s hard to tell what’s actually used given all the ways a framework can be compiled. Also, just because jQuery is on a site, doesn’t mean it’s used.

We should probably hire some statisticians to help with this.

His conclusion:

The point is that I don’t believe you can make a valid inference from the survey’s results. Nor should you care to.

I’m not even sure if that’s the goal of the survey. The survey does not advertise itself as scientific. So don’t get mad at the team, don’t get mad at the results. Click around the graphs. Enjoy the beautiful charts. Respect and admire the hard work. Dive into the data and see what looks interesting. But please, don’t use this as cannon fodder for framework wars.