Surveys and questionnaires are fairly common today, and I believe most people must have filled out a few before. When I fill out questionnaires, I often cannot help thinking if the designer has tried to fill out his/her own questionnaire (eating your own dog food), because some questions are essentially very difficult to answer, although they look easy.
Earlier last month, I took the 2018 Stack Overflow Developer Survey, and I found a few really difficult questions, such as the one that asked me to rank 10 aspects of a job opportunity in order of importance:
You may think 10 is a small number. Indeed it is. I can count to 10 without using my toes. The problem is that when you compare n aspects with each other, you will end up with doing n * (n + 1) / 2 comparisons. In this case, you need to do 55 comparisons to accurately answer this question. How long do you think it will need to make 55 decisions? The time is probably not trivial. Also consider that this is only one of the many questions in the survey.
So what would a busy developer do? For me, I just dragged the most important aspect to the top, and left the rest of them untouched. I could imagine some other people would do the same. If any Stack Overflow employee (hey Julia) is interested, it will be great to verify it next year: randomize the 10 options for every developer who is going to fill out the questionnaire, and check if only a very small number of options have been moved when the survey is done. It could be verified this year, but since everyone filled out the same questionnaire, there is a tiny probability that most people actually agreed the default order was the order they believed to be appropriate. Without randomization, it is hard to exclude this possibility.
BTW, I’m not really sure how I should interpret the results of such surveys when they are published, because they are based on convenience sampling. For example, developers of one programming language could be more responsive than those of another language, so it could be potentially biased to infer that one language is more popular than the other.
Donate
As a freelancer (currently working as a contractor) and a dad of three kids, I truly appreciate your donation to support my writing and open-source software development! Your contribution helps me cope with financial uncertainty better, so I can spend more time on producing high-quality content and software. You can make a donation through methods below.
-
Venmo:
@yihui_xie, or Zelle:[email protected] -
Paypal
-
If you have a Paypal account, you can follow the link https://paypal.me/YihuiXie or find me on Paypal via my email
[email protected]. Please choose the payment type as “Family and Friends” (instead of “Goods and Services”) to avoid extra fees. -
If you don’t have Paypal, you may donate through this link via your debit or credit card. Paypal will charge a fee on my side.
-
-
Other ways:
WeChat Pay (微信支付：谢益辉) Alipay (支付宝：谢益辉) 
When sending money, please be sure to add a note “gift” or “donation” if possible, so it won’t be treated as my taxable income but a genuine gift. Needless to say, donation is completely voluntary and I appreciate any amount you can give.
Please feel free to email me if you prefer a different way to give. Thank you very much!
I’ll give back a significant portion of the donations to the open-source community and charities. For the record, I received about $30,000 in total (before tax) in 2024-25, and gave back about $15,000 (after tax).