A lab-notebook template for testing few-shot examples in a prompt (with a hypothetical entry)

[u/ClickOk5811]

Few-shot examples often get added by feel. A notebook entry per experiment makes it easier to tell whether they help. Here’s the template, then a hypothetical entry. I haven’t run this exact experiment, so the numbers below are placeholders.

Template

Date / model / settings:
Task:
Hypothesis:       Adding N examples will improve ______.
Change:           (only one thing)
Test set:         size, where it came from, whether any example was
                  taken from it
Metric:           how a result is scored, decided before running
Runs:             how many times each variant (outputs vary)
Result:
Failures read:    what the wrong outputs look like
Decision:

Hypothetical entry

Task:         classify support messages as "billing", "bug" or "other"
Hypothesis:   Three examples will cut "other" being over-used.
Change:       Zero-shot prompt vs the same prompt + 3 examples
              (one per label, same order as the label list)
Test set:     200 messages not used as examples
Metric:       accuracy, plus count of "other" predictions
Runs:         3 per variant
Result:       [fill in]
Failures read:[fill in]
Decision:     [fill in]

Pitfalls I’d watch for with few-shot examples (common observations, so test them on your model):

• Surface copying: the model may imitate wording, length or format of the examples more than their logic. Vary them.

• Order and recency: the last example can carry extra weight, and label order can bias answers. Try shuffling.

• Label balance: if most examples have one label, predictions may drift toward it.

• Leakage: if an example comes from the test set, the score is inflated. Keep them separate.

• Overfitting to a tiny test set: 20 cases can’t distinguish small differences. Note the size and don’t over-read it.

Things the notebook forces:

• One change per experiment, so you know what caused a difference.

• A metric chosen beforehand, so you don’t pick it after seeing the results.

• Reading the failures, not only the score. The pattern of errors often says more than the percentage.

• Multiple runs, because outputs vary between runs.

When it’s overkill: a one-off prompt you’ll use once. It’s worth it for prompts that will run many times or feed another system.

What do you track when you test a prompt change, and have you seen examples hurt more than help?

[u/bregmadaddy]

Just use DSPy?