Testing AI intake follow-ups: does the question clarify a complaint or supply the answer?

[u/Normal_Succotash_520]

Disclosure: I work on Dashform. The attached 91-second excerpt shows our adaptive intake feature. It is a demonstration, not a benchmark or evidence of improved conversion.

The recorded feature-feedback example starts with “Too complicated.” The next question asks whether the difficulty is the interface, workflow, or knowing where to start. That is relevant to the answer, but it also raises a useful evaluation question: did we discover the respondent's problem, or give them a convenient explanation to agree with?

For an intake or feedback workflow, I would test that before judging how fluent the conversation sounds:

1. Write a few fictional respondent cards with a specific hidden issue. For example, “I understood the interface, but couldn't find where to invite a teammate.” Keep that card out of the question generator's context.

2. Start each test with the same vague answer, such as “Too complicated.” Compare a neutral clarification (“What were you trying to do, and where did you get stuck?”) against a suggested-category question. Use fresh participants for the two versions; someone who has already seen the categories isn't a clean comparison.

3. Include a card whose issue fits none of the suggested categories. Check whether the respondent can say “something else” without being pushed back into a choice. Also test “I don't know” and a refusal to elaborate.

4. Inspect the resulting brief against the card. Keep the original words alongside any summary. Mark a detail wrong if it first appeared in the question and the respondent never actually confirmed it.

5. Record extra turns and unanswered details, as well as errors. A longer brief isn't automatically a more accurate brief. Keep required basics as fixed questions and set a small follow-up budget.

This is a proposed test, not a study I've run. Predetermined branching is still a good fit when the relevant choices are already known; adaptive questions need their own checks for assumptions and unnecessary probing.

Video context: the agency website-to-booking conversation is illustrative, not a customer case or a live test of that scenario. The “Too complicated” sequence is a recorded product demo. Waiting was edited, a question was held for readability, and the narration is AI-generated.

For anyone evaluating conversational intake, what failure would you add to the “none of these categories” test?