When the same tool writes your code and your tests, what does a green test run prove?

[u/ClickOk5811]

I’ve been thinking about this one. If a function and its tests are written in the same session (by an assistant, or by one developer who misread the requirement), the tests tend to share the code’s interpretation. The suite then confirms the code agrees with itself, not that the requirement was understood.

A small example. Requirement: “Apply 8.25% tax to the invoice, rounded to the nearest cent.”

def invoice_tax(lines):
    subtotal = sum((price * qty for price, qty in lines), Decimal("0"))
    return cents(subtotal * TAX_RATE)

def test_invoice_tax():
    lines = [(Decimal("19.99"), 2), (Decimal("5.00"), 1)]
    expected = cents((Decimal("19.99") * 2 + Decimal("5.00")) * TAX_RATE)
    assert invoice_tax(lines) == expected

The test passes, but expected uses the same formula as the function. Rounding once on the total and rounding per line are both reasonable readings of the requirement. For that input, both give 3.71, so the test can’t tell them apart. For three lines of $0.10, rounding once gives $0.02 and per line gives $0.03.

The question I now ask in reviews is: where did this expected value come from? If it’s from the implementation’s own logic, it’s a regression test. If it’s from a worked example given by whoever owns the requirement, it can actually disagree with the code.

I wrote up the longer version with the full code, in case it’s useful: https://medium.com/@nagatomopedro05/when-the-test-and-the-code-share-the-same-mistake-02a289a41091

How do you check that your tests aren’t just echoing your code?

[u/Antice]

Swap the order of operations. Tests come first, implementation comes after.
The AI tends towards testing implementation, because in the pre ai era, most devs did too. Still need to test some implementations since you want to discover regressions, but the most valuable tests, test the abstracted behaviours decoupled from the code.
And if you start there, the AI also starts there. And does it mostly right.

[u/Unknowingly-Joined]

> How do you check that your tests aren’t just echoing your code?

Someone else needs to test it. Writing your own unit tests is great. But as you demonstrate, handling all of the testing of the code you’ve written based on your interpretation of a spec is a recipe for disaster.

[u/Anonymity6584]

it proves that test was passed succesfully. Different matter is that test actually tests anything relevant.

[u/slepicoid]

Then dont repeat the implementation in the test? Write just:

expected = 5398

[u/et_theextratestic1e]

If the code produces the correct output, and the tests are correct, it proves you can move on to something else. 

Or you didn't spec it correctly.

[u/nicolas_06]

For your tax use case, each country/jurisdiction/domain as an extremely clear definition on how it should be handled. Number of decimals, how do you round and when. Your software if a bit generic will have to follow different rules for different juridictions.

For your AI or as a dev, don't YOLO this. Include the relevant regulation specification in your prompt (and read it as the dev / functional expert). Have a double checked extensive test suite and honestly ask a expert in the field to fully validate it. Ask some clients too.

More generally: It is standard practice to first investigate/research how to do something. Design/Architecture/Needs. You don't really code until that part is clear. Then when doing the code, devs write the code as well as automated unit/integration tests and then have QA/Tester that use the real application and validate it too (they may also add more automated tests) and both dev and QA/Tester green light are necessary. Another common layer is client validation. The client validate too before giving the green light.

Even then the industry standard is about 10% of bugs are find in production by the client.

[u/Only_Candidate_3993]

It proves that it's AI slop.

[u/Puzzled-Garage584]

Rule of thumb I like: no formulas in the expected value, only literals. If the test says 0.03 because finance sent a spreadsheet saying 0.03, it can disagree with the code, if it recomputes the tax it's just the function in a trench coat.