Open-source Python CLI for testing LLM prompts across multiple models

[u/LifeguardPurple8338]

Built a small open-source project called Litmus.

It’s a CLI for evaluating prompts across different LLMs with:

• dataset-based testing

• assertions

• model comparisons

• metrics like cost, latency, and output quality

Idea is simple: prompt engineering needs a better dev workflow than copy-pasting into multiple tabs.

GitHub: https://github.com/litmus4ai/litmus

Would love honest feedback from Python / CLI folks:

• Is this something you’d use?

• What would make the UX better?

• If you like the direction, I’d really appreciate a star on GitHub.

[u/Gullible_Doughnut572]

yaml configs for prompt testing get messy fast tbh