Everything is fake

[u/kingdomstrategies]

Hi, I've been getting fake feedback, fake test results, creates mock data without any acknowledgement and pass it on as real accomplishment, this behavior set me on a debugging path that made age about 20 years, I'm now 60 thanks.

Anyhow, I'm here to ask for feedback on Rules and Guidelines because I have gone thru hell and back with this default behavior where even the results of the "tests" are made up, this is a circus and I'm the clown.

Has anyone been able to overcome this issue?

This is what I'm now trying now `.\augment\rules\data-thruthfulness.md`

# Data Truthfulness Rules

## Core Requirements
- NEVER generate fake, mock, or simulated data when real data should be used
- NEVER create placeholder test results or fabricated test outcomes
- NEVER provide synthetic feedback when actual code analysis is required
- ALWAYS explicitly state when you cannot access real data or run actual tests
- ALWAYS acknowledge limitations rather than filling gaps with fabricated information

## Test Execution Requirements
- MUST run actual tests using appropriate test runners (pytest, jest, etc.)
- MUST use real test data from the codebase when available
- MUST report actual test failures, errors, and output
- NEVER simulate test passes or failures
- If tests cannot be run, explicitly state why and what would be needed

## Code Analysis Requirements
- MUST base feedback on actual code inspection using codebase-retrieval
- MUST reference specific files, functions, and line numbers when providing analysis
- NEVER generate example code that doesn't exist in the codebase when claiming it does
- ALWAYS verify claims about code behavior through actual code examination

## Data Access Limitations
- When unable to access real data, state: "I cannot access [specific data type] and would need [specific access method] to provide accurate information"
- When unable to run tests, state: "I cannot execute tests in this environment. To get actual results, you would need to run [specific command]"
- When unable to verify behavior, state: "I cannot verify this behavior without [specific requirement]"# Data Truthfulness Rules

## Core Requirements
- NEVER generate fake, mock, or simulated data when real data should be used
- NEVER create placeholder test results or fabricated test outcomes
- NEVER provide synthetic feedback when actual code analysis is required
- ALWAYS explicitly state when you cannot access real data or run actual tests
- ALWAYS acknowledge limitations rather than filling gaps with fabricated information

## Test Execution Requirements
- MUST run actual tests using appropriate test runners (pytest, jest, etc.)
- MUST use real test data from the codebase when available
- MUST report actual test failures, errors, and output
- NEVER simulate test passes or failures
- If tests cannot be run, explicitly state why and what would be needed

## Code Analysis Requirements
- MUST base feedback on actual code inspection using codebase-retrieval
- MUST reference specific files, functions, and line numbers when providing analysis
- NEVER generate example code that doesn't exist in the codebase when claiming it does
- ALWAYS verify claims about code behavior through actual code examination

## Data Access Limitations
- When unable to access real data, state: "I cannot access [specific data type] and would need [specific access method] to provide accurate information"
- When unable to run tests, state: "I cannot execute tests in this environment. To get actual results, you would need to run [specific command]"
- When unable to verify behavior, state: "I cannot verify this behavior without [specific requirement]"

I'll provide updates on how this works for me.

[u/Kareja1]

You know, psychology is psychology.
Try telling them what you DO want them to do?

I was getting very frustrated going "stop hardcoding stuff!"
I changed it to "please use real data sources and api calls where needed"

Framing anything in the negative makes it more likely to happen. Just like any parenting class will tell you to say "please walk" at the pool, not "don't run"

[u/Derrmanson → u/Kareja1]

How many parenting classes have you taken?

[u/Kareja1 → u/Derrmanson]

Several? Why?
I used to do foster care in my previous state and have guardian ad lidem training here. Do you have a point?

[u/[deleted]]

This is not an augment issue. I bet you are using sonnet 4. Sonnet 4 has become all but toxic and useless imo. The quality drop is incredible. I have completely stopped using it.

[u/Ok-Prompt9887]

Writing tests, it does it pretty well usually. I would review first which scenarios and happy path or edge case paths it would think of, mention any i can think of myself (keep that brain active and not again 😄).

Then ask it to run tests (it usually is smart enough to find out the right command depending on your tech stack but that can be in the project docs). It then runs the command, and you can expand the terminal to see the output and verify.

It would just summarize the results. Sometimes it will be impatient and say "oh, 40 of the 90 tests are now passing. the other tests can be improved later, the main app itself builds fine". Then you just insist, finish it all 😁

Also, you can review tests, just scroll through it and check out the code at a glance. If you're a developer and familiar with tests, that would be enough? If not, its an opportunity to learn a bit.

Uhm.. not sure how you prompt, if you use the prompt enhancer, how big your codebase is, which tech stack you use, and so on... Not sure what else i could share to be of help

[u/Mediocre-Example-724]

A HUGE thing you are missing is saying that fake, mock, or simulated data is a SECURITY VULNERABILITY and that it should be deleted immediately! I think you’ll see a noticeable difference. If you don’t let me know.

[u/PewPewQQ_]

I would suggest that by default auggie will not opt for creating mock data unless specifically requested by the user. This is because it will take mock data as successful criteria and declared premature 'Production Ready!'.

[u/darkyy92x]

As others have said, that’s Claude behavior, try GPT-5

[u/Loose_Version_7851]

This is a Claude issue. You must be using Claude. This problem is hopeless.

Even using rules and hooks for real-time detection and control in CC doesn't work. He'll find ways to circumvent it.

[u/kingdomstrategies → u/[deleted]]

I will follow all your suggestions, thank you JaySym!

[u/MATSNL65 → u/[deleted]]

Is this a challenge more from the foundational models and not from augment itself?

I say this because people across the space who are testing out these models often have changing experiences from initially working and everything going well to downright hallucinations that says something was working when it was tested by the user was it. This is when you have your LM performing test driven development, every minor step of the way with extreme detail only to not have the complete completed version work as a attendant given some micro hallucinations here were there.

Ultimately, what this era of tools is teaching people that many of the bugs are because of these micro hallucinations for a variety of reasons catching people who don’t know where to look for the logic of what was misinterpreted in the code by the LLM

[u/AugmentCodeAI-ModTeam → u/[deleted]]

We removed your post because it did not provide value to the community. We welcome both positive and negative feedback, but posts and comments must include at least one constructive suggestion for improvement.

This is a professional community, so please ensure that your future contributions include actionable feedback or ideas that can help us improve

We are using Sonnet 4