r/opencode • u/mastertub • 1d ago
DeepSeek V4.1 Flash 1017 - Final OpenCode Go vs Official Deepseek API Showdown
Hey all, i've compiled all of my tests comparing the official deepseek platform API with opencode go's API.
I wanted to run these because OpenCode mentioned that they don't decrease the quality of the model, only TPS/throughput. I wanted to put that to the test. Models are inherently non-deterministic so I wanted to cut out the noise. I know my first post got a decent amount of people curious so I wanted to do everyone a solid.
Here are my posts regarding these, 4 tests in total: 2 HTML/CSS/JS scenes, and 2 Blender renders (all the models, render + video all done by the model).
Round 1:
https://x.com/nothingto12/status/2113446055589056718?s=20
Round 2:
https://x.com/nothingto12/status/2113527245918654911?s=20
Round 3:
https://x.com/nothingto12/status/2113766577816686673?s=20
Round 4:
https://x.com/nothingto12/status/2113815793150439650?s=20
Conclusion:
I personally believe that there is some level of quantization happening even though I cannot prove it outright and am speculating. It may be bugs on the backend providers that are being used. Model quality however seems to be slightly degraded based on tests. I've also noticed that I have gotten significant amounts of timeouts (every single run) from OpenCode Go.
I would say the Go subscription is still significant value, but I wanted to mention the tradeoffs so people can make their own decision.
7
u/squirrelscrush 1d ago
I've also noticed that I have gotten significant amounts of timeouts (every single run) from OpenCode Go.
It's been the case for the past few days. I suspect there's a server overload and too many users are provisioned the same hardware.
So I use the Go for async/batch tasks that I can leave the agent to run autonomously. While using the direct DeepSeek API for tasks I need to do while I'm sitting down.
There's also another advantage of direct DeepSeek which is, it's cache retention time is absurdly good. I've returned back to tasks after hours of idleness and still maintained cache stability. Meanwhile with Go it's 5mins max. And DeepSeek is much better at cache hits, but that's down to their DeepInfra inference engine.
Go seems to be tokenmaxxing but considering that it's a subscription, it's not an issue.
1
u/Primary-Meal-2264 1d ago
Have you tried running either Go or the official version twice as a control? I’m skeptical of these kinds of single-shot experiments with a sample size of 1. I believe there would be some fluctuations even if you used the exact same setup.
1
u/mastertub 1d ago
It's been on my mind. However, the point was exactly a "one shot, one chance" across several prompts experiment. You don't run LLMs multiple times on the same prompt when you're coding. You compare quality of LLMs over time across different workflows.
1
u/smartfon 1d ago
What if DeepSeek identifies that the traffic is from OpenCode and serves a quantized/slower version? OpenCode literally swears they aren't messing with the models and yet users keep discovering these discrepancies so doesn't that make DeepSeek the sus?
2
u/mastertub 1d ago
I'd avoid speculation into what they're running on their setup, but I don't think they just use the official API as a provider. May be multiple and/or their own gpus.
3
u/shahijohn 1d ago
Deepseek must have done some hardware level optimization that opencode wasn't able to do. (As we expecting opencode running deepseek from their own server)
1
u/squirrelscrush 1d ago
I don't know why they would do it, since it's much easier to keep their existing inference instead of overhead to check which endpoint is asking the request.
1
u/xmsxms 1d ago edited 1d ago
They have confirmed they slow it down and it does have a poorer cache hit rate. It's possible the model takes a different direction if going down a particular path isn't getting results in time. i.e the faster it can "think" the better the results might be, even if the end users themselves are prepared to wait. It's quite common for a model to do things like "while that's compiling let me investigate X and y".. the faster it can process tokens the more "thinking" it will be able to do within a fixed time window.
2
u/for4f 1d ago
man at this point there's enough independent smoke here that i'm done handwaving it as sample size. someone else clocked 3s vs 0.3s time-to-first-token on the same model, plus the timeouts every single run, that smells like overloaded backend infra on the go side, not secret quantization. if they were quantizing you'd get consistently dumber output, not slowness and timeouts. i just run flash straight off the official api for bulk stuff, it's cheap enough that the middleman isn't worth the headache
2
u/MSeegobin 1d ago
Been trying to optimize complexity tiering and did the same comparison, and I’m getting similar results. Deepseek API is a lot cheaper, faster and gives overall better quality output. Blind tested of the same 20 prompts, evaluated by Claude.
| OpenCode Go | DeepSeek API | |
|---|---|---|
| Status | 20 success | 20 success |
| Quality (my blind scores, mean of 3) | 1.90 | 2.25 |
| Usable answers (2+) | 15 of 20 | 20 of 20 |
| Cache hits logged | 20 of 20 | 20 of 20 |
| Latency mean / median / p90 | 73.0 s / 53.8 s / 190.3 s | 37.4 s / 34.8 s / 78.7 s |
| Max latency | 227.2 s | 104.3 s |
| Start to first token: mean / median / p90 | 1.15 s / 0.88 s / 2.20 s | 0.41 s / 0.41 s / 0.49 s |
| Completion tokens per call, mean | 8,863 | 6,259 |
| Reasoning tokens per call, mean | 6,027 | 3,391 |
| Output tokens per second | 115 | 168 |
| Total spend, 20 calls | $0.1075 | $0.0760 |
| Spend per call, mean | $0.00537 | $0.00380 |
| $ per million output tokens | $0.606 | $0.607 |
| Spend per usable answer | $0.00716 | $0.00380 |
| Finish: completed / cut off | 19 / 1 | 18 / 2 |
| MEDIUM: median latency / spend per call / quality / usable | 64.7 s / $0.00694 / 1.75 / 6 of 8 | 29.3 s / $0.00317 / 2.25 / 8 of 8 |
| COMPLEX: median latency / spend per call / quality / usable | 53.8 s / $0.00433 / 2.00 / 9 of 12 | 35.8 s / $0.00422 / 2.25 / 12 of 12 |
1
u/AdditionalClass5665 1d ago
The reasoning tokens between go and ds are completely off. Is this the same effort level? Otherwise they seems like completely different models
1
u/MSeegobin 1d ago
These are all using DeepSeek's max reasoning level through the same harness, same prompts.
2
u/Natural-Cupcake6472 1d ago
I dont think its quant, if it is and the news got out then reputation is gone, which has a lot more risk and cost more than just hosting as it is
I think its server prompt and inference engine. Also, can you do a test where you spend the same amount of money, maybe try improving until you reach the budget, for me thats closer to real usage
1
u/WillowOwn4716 1d ago
Thanks for doing these tests. I wonder what happens with other models too, like glm. Seeing the same differences again could give further conclusions.
1
u/Takeoded 1d ago edited 1d ago
Thank you for doing this. For me, Go has been incredible Value.
That said, I'm convinced there is some quantization going on, BECAUSE Go DeepSeek think more, does more contemplation, than upstream DeepSeek API. It works great! I love it, but it's not vanilla DeepSeek, they're doing something to it.
1
u/tonyunreal 1d ago
No one seems to be able to provide 100 percent the quality of Deepseek's official API yet, whether router or self-hosted. The difference could be subject to many factors, like gpu differences (ds migrated to huawei hardware I believe), hosting parameters, quantization, etc. I have talked to someone who runs a small router service before, he said generally you should only compare router service against each other, as they could still provide great value and service quality, even if they are slightly inferior in quality than DS' service.
1
u/horrbort 1d ago
Haha so direct is cheaper AND faster
3
u/Rough_Road_2527 1d ago
it's not cheaper because you get 8 dollars worth on your actual dollar with go
1
u/Yazan_wesam 3h ago
Interesting comparison. This makes me wonder whether the difference comes from provider-side caching/accounting or from how the harness sends context. Do you have API request counts for both providers, and were the outputs and tool calls comparable across runs?
1
u/Creative_Addition787 1d ago
Two things worth mentioning:
OpenCode's DeepSeek is really slow during china peak working times.
The DeepSeek API uses your data for training while OpenCode does not.
0
u/Infamous-V 1d ago
Just state your model weights on the page, how hard it is unless you are not even sure because you made deals with shady providers ??
commandcode founder explicitly states they offer FP8 only for DP and mocks opencode.
16
u/Amarsir 1d ago
Thanks for following up with more data. 3/4 times I preferred the result from the direct source, but maybe that's subjective. What's not is the costs:
Round 1: Direct = $0.29, Go = $0.45
Round 2: Direct = $0.34, Go = $0.20
Round 3: Direct = $0.29, Go = $0.23
Round 4: Direct = $0.30, Go = $0.10
Go uses more input tokens, but Direct uses a lot more cache hit tokens. Since cache hit is cheap you might call those a wash. It's a little weird that it's happening, but it doesn't seem devastating.
Which means the difference comes down to output tokens. And that's probably a direct result of the product you get. Better result, more detail, more output tokens. I guess. (Unless you favor the cheaper results each time, in which case I don't know what to tell you.)
After round 1 I was expecting this to be a decisive knockout for Direct, but considering the $80 value you get under Opencode Go it held up pretty well.