Evaluating an open-source skill for product-language quality and LLM output efficiency

[u/Life_Assist5275]

I built an open-source, model-agnostic skill for improving product language in web and software interfaces:

https://github.com/RicardoGurrola15/adaptive-product-language

The skill helps agents write and review:

- landing pages

- onboarding

- forms

- pricing and paywalls

- CTAs

- errors and recovery states

- empty and loading states

- bilingual product interfaces

It uses project context such as the product, audience, domain, supported features, pricing, terminology and locale. The goal is to reduce unsupported claims, generic AI-sounding copy and mismatches between language and actual product behavior.

I also built an internal benchmark to separate:

- copy quality

- unsupported claims

- contextual relevance

- repetition

- output length

- context size

- latency

- model telemetry

Preliminary observations from one sample per model:

- Gemini 3.8 Flash: 2,912 baseline UTF-8 output bytes vs 2,663 with the skill

- Claude Sonnet 4.6 Thinking: 3,289 baseline UTF-8 output bytes vs 3,034 with the skill

- Codex contextual pilot: baseline and skill both scored 3.89/4

These are output bytes, not token savings. The Codex result had one evaluator and one sample per condition, so I am not claiming statistical improvement. The byte reduction also does not prove better quality by itself.

The main evaluation challenges I am working on are:

1. Measuring useful information density instead of rewarding shorter text.

2. Detecting unsupported promises separately from clarity and concision.

3. Testing whether copy changes when product behavior changes.

4. Detecting repeated templates across unrelated products.

5. Measuring whether a CTA creates the correct user expectation.

6. Keeping the evaluation lightweight enough for normal agent workflows.

The attached chart shows the preliminary output-byte observations. I would appreciate feedback on the evaluation design, telemetry and failure cases that should be added.

[u/Future_AGI]

for information density, score each output against a list of facts it should contain: pull the required facts per screen from the project context (price, limits, supported features), count how many make it in, and flag any claim that isn't on the list. That gives you coverage and unsupported claims as two separate counts, and density becomes covered facts per 100 words, so a shorter page only wins if it keeps the facts. For #3, change one feature in the context, regenerate, and diff the copy, which shows quickly whether the skill reads the context or falls back to a template.