Qualitative tree testing helps reveal why users choose the wrong categories and which labels they find confusing; quantitative tree testing measures time spent finding content and success rates, and supports comparisons of navigation structures. Qualitative testing can inform early revisions with a small number of participants but is not suitable for producing statistics such as success rates. Quantitative testing requires larger samples and generally offers limited insight into the reasons behind users’ behavior. Both test only a simplified category hierarchy, so their results do not directly represent users’ performance on a complete website.
Qualitative testing: understand choices before revising categories and labels
Early in a project, before layouts, content, or interactions have been designed, tree testing can quickly assess a proposed information architecture. Qualitative tests are often moderated, allowing the moderator to follow up on behavior worth exploring by asking how participants interpret labels or why they avoid a particular category. These insights can inform navigation improvements.
In his article, Page Laubheimer describes how a pilot for a quantitative tree test found that some users avoided a category because its name was too broad, leading them to worry that it contained too much content. He notes that this pattern did not emerge in the quantitative results because task order was randomized. This was an observation from that particular study, not evidence that broad labels necessarily deter all users.
Quantitative testing: establish benchmarks and compare alternatives, but numbers do not explain everything
To assess existing navigation or compare alternatives and subsequent redesigns, quantitative tree testing can establish benchmarks by measuring task success rates and time spent finding content. However, knowing that users perform poorly on a task does not explain why they struggle.
- Larger samples are needed: Observations from a qualitative test with few participants should not be treated as statistical results suitable for comparing alternatives.
- Differences need testing: When comparing task metrics across tree structures, assess whether the differences are statistically significant rather than drawing conclusions from numerical differences alone.
Run a moderated pilot to catch problems early
A small qualitative study before quantitative testing can check that task wording is clear and does not lead participants, while also exploring which labels are confusing and why participants make particular choices. The inability to ask immediate follow-up questions is a limitation of unmoderated testing, not of all quantitative testing.
In remote unmoderated testing, click logs cannot fully capture participants’ comments and the context of their behavior during tasks. After the test, participants can be shown a list of labels and asked to select those they found confusing, then answer open-ended questions. This can fill some information gaps but cannot replace follow-up questions tailored to individual participants.
A shared limitation: testing the hierarchy, not the whole website
Tree testing deliberately excludes search, secondary navigation, and visual and content cues to test the navigation structure in isolation. Success rates are generally lower than in standard quantitative usability testing, but whether the complete website design improves users’ performance still depends on the quality of features such as search and menus.
For example, a mega menu may display several subcategories at once, unlike exploring a tree one level at a time. When analyzing such websites, it can help to focus on whether users choose the correct top-level category rather than looking only at final success rates. This can mitigate some limitations, but it does not make tree testing equivalent to testing a complete website.
Qualitative and quantitative tree testing: perspectives from across the web
When comparing two tree structures, is a numerical difference enough to decide which is better?
Numbers alone are not enough to draw a conclusion. When comparing metrics such as success rates, completion times, or paths taken for the same tasks, differences also need to be tested for statistical significance. Most tree-testing software does not perform this test automatically, so it must be calculated separately. Achieving sufficient precision in the comparison also means considering the cost of a larger sample.
Can tree testing replace standard qualitative usability testing?
Not entirely. Tree testing focuses on category structures and labels, making it useful for checking whether content is easy to find within a hierarchy. Standard qualitative usability testing can also provide navigation insights while covering visual design, layout, interactions, and content. If the question concerns how these design elements affect users’ ability to find content, a separate usability test of the complete design is advisable.
Does a testing tool’s requirement to select a bottom-level item affect how results should be interpreted?
It can. Tree-testing software typically allows participants to submit only bottom-level items as answers, but intermediate categories on a real website may have their own pages containing useful content. Having to continue down the hierarchy in a test therefore does not necessarily reflect what users would do on the actual website. When interpreting results, check whether the tool’s restrictions on answer selection match the website’s design.
References
- Card Sorting vs. Tree Testingwww.nngroup.com · 2024-02-23T18:00:00+0000 — Discusses: Can tree testing replace standard qualitative usability testing?
- Tree Testing: Evaluate Menu Labels and Categorieswww.nngroup.com · 2023-08-06T18:00:00+0000 — Discusses: Can tree testing replace standard qualitative usability testing?; Does a testing tool’s requirement to select a bottom-level item affect how results should be interpreted?
- Tree Testing Part 2: Interpreting the Resultswww.nngroup.com · 2024-01-19T17:00:00+0000 — Discusses: When comparing two tree structures, is a numerical difference enough to decide which is better?


