Option Order

[SachaG]

I'm not 100% sure what makes the most sense when it comes to the order in which we display predefined options for a question.

Arbitrary Order

Good: Lets us add some meaning to the presentation, for example put the most popular items first.
Bad: Risks reinforcing existing biases in the data.

Random Order

Good: Safest way to avoid favoring the items that come up first and biasing the data.
Bad: Makes it harder to scan long lists of items to find the ones you want.

Alphabetical Order

Good: Easiest to scan when looking for specific items.
Bad: May give unfair advantage to items which appear first in the list, especially for longer lists where items lower down start out hidden behind a "show more…" button.

Right now I'm leaning towards making alphabetical order the default, because I think increasing the ease of finding items outweighs the potential bias towards earlier items in the list. Thoughts?

[michaelquiapos]

Hi @SachaG,

Here's what I know, when displaying predefined options for a protocol question, it is generally best to order them in a logical and meaningful way, it will be easier for users to understand and respond to the question, which will ultimately improve the quality of the survey data collected. And for the researcher to tag and outline a specific sequence of actions (Use Case / User Scenario) needed to achieve the goal, as well as alternative sequences.

This could be alphabetical, chronological, or ordered based on their importance or relevance to the question being asked.

1. If the survey question is asking about the most important factors when choosing a new car, the predefined options could be ordered from most to least important. For example:
   • Safety features
   • Fuel efficiency
   • Price
   • Brand reputation
   • Style
2. If the survey question is asking about the most important factors when choosing a new job, the predefined options could be ordered from most to least important. For example:
   • Salary
   • Company culture
   • Career growth opportunities
   • Work-life balance
   • Job security
3. If the survey question is asking about the most important skills for a CSS Developer, the predefined options could be ordered from most to least important.
   • Proficiency in CSS (including CSS3 and CSS frameworks)
   • Knowledge of HTML and JavaScript
   • Understanding of responsive design
   • Familiarity with design tools (e.g. Photoshop, Sketch)
   • Understanding of cross-browser compatibility issues

In this example, the predefined options are ordered based on their importance or relevance to the skills required. It is also important to consider the user's perspective and ensure that the order makes sense to them.

[shhaine]

Arbitrary Order

Good: Lets us add some meaning to the presentation, for example put the most popular items first. Bad: Risks reinforcing existing biases in the data.

This ordering is not arbitrary since it has clear basis, which is popularity. Based from the response of @michaelquiapos, I think that we use frequency from the last year's survey as metric of importance if we are to apply this. However, a much more recent result is a better basis. This eliminates the tendency to produce far from expected results when respondents tend to select the first options presented.

Random Order

Good: Safest way to avoid favoring the items that come up first and biasing the data. Bad: Makes it harder to scan long lists of items to find the ones you want.

I do not recommend this one. Unless we vary the order for each respondent, we do not address the primacy effect which you described as 'favoring the items that come up first'.

Alphabetical Order

Good: Easiest to scan when looking for specific items. Bad: May give unfair advantage to items which appear first in the list, especially for longer lists where items lower down start out hidden behind a "show more…" button.

You are right in saying that this is easy to scan but again does not address the primacy effect.

I think that, regardless of the arrangement that we use, the "show more…" button makes it even more common for primacy effect to occur. I have two thoughts on this: 1) Reducing the number of options to the most popular ones 2) Changing the format so that all the options are displayed right away (no need to click on a button or scroll further, maybe into matrix style?)

[SachaG]

Btw just a note, the random order is different for each respondent.