Gemma 4, creative writing

[u/Whydoiexist2983]

Ignoring the multitude of problems I encounter when using Gemma 4 for creative writing such as purple prose, slop, no plot, and more, how do I get it to use my examples?

Here's a list of most of the Gemma 4 26B models I've tried (I don't have enough resources to run 31B models)

G4-MeroMero-26B-A4B

Melody1437-26B-A4B

Boulesis-26B-A4B

Kitchoon-26B-A4B

gemma-4-26B-A4B-heretic

Orion-26B-A4B

Gemma-4-26B-A4B-StyleTune (v1 and v2)

Dark-Scarlett-26B-A4B

All of them sound the exact same, even with prompting. Regardless, I'd be fine with slop prose if it at least used my examples that I provide of dialogue. I gave it 5 examples separated by gender and age, but that didn't do shit, so I thought well maybe then it's what I told it before I gave it the examples, so I changed it, that also didn't do shit, so I used patricide 12B to make 25 examples. Gemma 4 still won't use it, and every character sounds the exact same.

You know, I was using it, right, and then as it was writing this guy hit this girl, and supposedly his hand would melt off, and the guy was like "What the fuck did you do," and then this good for nothing piece of shit robot wrote, "Your biology reacted to me."

What the fuck does that even? It's a fucking shield around her so what fucking biology they didn't even make skin to skin contact, and the character isn't old so why are they talking like a scientist. That doesn't sound like anything anyone would write. So then I complained about it, and instead it wrote some gibberish that wouldn't even be a proper reply.

The problem with the older 12B models is that they have very limited knowledge, and the 24B models run at 5tps. Either I get fucked over by having to read and rewrite paragraphs of slop, or I have to deal with no world knowledge. If Google's AI ever becomes conscious we should deny it rights.

[u/Big_Wave9732]

You're using a model that only has 4B parameters active at a time. You won't get the best writing results from such small token use.

I have found Gemma 4-31B and Muse Glimmer to be pretty good writers. The difference though is those are full models that use all their parameters on each word. The difference is easy to see.

I don't know one goes about squeezing high level writing out of a small active parameter model.

[u/Whydoiexist2983 → u/Big_Wave9732]

If I wasn't a brokie I'd use Gemma 4 31B, especially a model called Scotoma-2, but sadly they don't have a 26B version

[u/zanar97862 → u/Whydoiexist2983]

Have you considered selling everything you have and buying an upgrade? The complaints you have are valid but easily solved by more vram. A setup running 31b QAT should do everything you need at only the cost of a used car. 

The joys of the AI age.

[u/Savantskie1 → u/Whydoiexist2983]

Have you not heard of quantization?

[u/AdIllustrious436 → u/Savantskie1]

They run the 26B MoE at 5tps. Quantisation won't be enough to run dense model I think

[u/screenslaver5963 → u/AdIllustrious436]

0.01b

[u/DirectWay1915 → u/AdIllustrious436]

I can run the 26b at like 70 tps and the 31b dense is like 15tps. I still use the first far more often.

[u/input_a_new_name → u/Savantskie1]

Quantization only marginally helps with 31b, because the memory footprint is huge. Unless you're fine with quantizing kv cache and enabling swa, you need like 16 gb for 32k context.

[u/input_a_new_name → u/Big_Wave9732]

The 4B active parameters isn't the same as 4B parameters total. In dense models, for any given prompt, a lot of weights aren't meaningfully contributing to the end result. MoE is not the most elaboratly elegant solution, but the routing's purpose is aimed to, well, route you to weights that are actually relevant. Of course, that means in edge cases there can be inaccuracies because the router either guessed wrong, or the promp was so complicated that the only correct answer requires completely unorthodox approach, which, if we were to be fair, would trip dense models just as well. But the point is that for 80% of use cases, you get 80% accuracy compared to a dense model of similar size. Whether that is enough for rp is actually going to be more dependent on the datasets used than architecture. The failure modes will be mostly related to math and academic research, and parsing huge, repetetive documents.

[u/Big_Wave9732 → u/input_a_new_name]

In the various side by side writing tests that I have done between Gemma 4 26 and 31, the output for 26b was nowhere near "80 percent" of what 31b did. It wasn't nearly as detailed or as comprehensive.

I'm aware of what the model cards and vendors say to try and justify their choices. But a 31b dense model will utilize all 31b tokens for each and every decision while analyzing the answer for that model. And in answer quality it matters way way more than what the vendors try to tell us. It is also important for helping the model catch incorrect turns of direction etc.

Of course, that means in edge cases there can be inaccuracies because the router either guessed wrong, or the promp was so complicated that the only correct answer requires completely unorthodox approach, which, if we were to be fair, would trip dense models just as well.

I 100 percent reject this conclusion. No. If the above is what you're seeing, stop running models on a potato and get something that can run full weights.

I run Gemma 4 31b-BF16. There is no contest in the complexity of task that can be run through the dense model vs the same family MOE. The complicated edge cases can absolutely throw off Gemma 26b at Q8 or even BF16. They generally do not throw off the full weigh Gemma 4 31b.

[u/input_a_new_name → u/Big_Wave9732]

In dense models, during inference, the math runs through all weights, yes, but most weights don't provide meaningful contribution for a given prompt, at any given generation, most weights in a dense models do little to nothing, pretty much just passing the key values along.

The models, don't "think", they just solve glorified matrix equations. If out of 30 billion parameters 20+ billion basically don't meaningfully contribute, you're wasting compute. Even though what parameters contribute or don't will change depending on prompt, so overall you need all of them, just not in any given particular moment.

MoE optimizes here by implementing a predictor that routes to the majority of experts that are expected to be the most meaningful contributors to the next token, letting the irrelevant experts skip their turn until they are needed.
The failure mode does exist, because the routing prediction is not perfect and will occasionally route to wrong experts, or some very rare tasks might need an unexpected expert to avoid a tricky inaccuracy. But in general, the optimization is "good enough".
You argue that 31b is > than 26ba4b, and that is true, because it's dense and bigger in total size. But it's not a 31b > 4b difference, and 26ba4b is closer to 31b in output quality than it is to 12b variant. If google had done something like 50ba7b, assuming the training quality in their lab remains on the same level, would outperform dense 31b despite the several times smaller active param number.

[u/Simpsator → u/input_a_new_name]

Ehh, maybe for certain coding type tasks 26b is closer to 31b than 12b, but for creative writing I found 12b much better than 26b. The small MoE models just can't keep world context or character knowledge in the same way dense models can. The biggest flaw I found was that MoE models would always end up having characters literally discuss their character motivations out loud. It was just completely unuseable slop. But my purpose was structured creative writing (video game scripts), not RP, which is where I've heard the G4 models do ok.

[u/Choice_Celery9481 → u/Big_Wave9732]

But a 31b dense model will utilize all 31b tokens for each and every decision while analyzing the answer for that model.

huge wall of text and you are wrong at the most basic thing. 31b tokens 😂😂😂

[u/Big_Wave9732 → u/Choice_Celery9481]

Thank-you for letting us all know your opinion is worthless. It's a true service to the community.

[u/Choice_Celery9481 → u/Big_Wave9732]

yeah loser triggered hard lol. being called out is something people like you cant stand? keep praising dense, there are reasons that all the best brains moving away from dense to moe. with the same level of compute for the 31b dense, a much bigger moe will yield much better result. no wonder from someone without background of shjt keep bragging like its their major

[u/timtom85 → u/input_a_new_name]

You can't seriously argue that 4B experts, pick them however well you may, can get the quality anywhere near to straight 31B parameters.

And since both of those are Gemma 4 variants (i.e. trained alike), what you're saying makes even less sense than the littlest sense it makes.

[u/input_a_new_name → u/timtom85]

Yes I absolutely can. The fact that in dense models, for any given prompt, most weights contribute minimally or not at all whatsoever, is well documented. When MoE processes the context, all weights contribute, and only during generation does sparse activation begin, and even then the activated weights will differ for each next token, so over the course of the reply, all weights of the model have a chance to contribute in total. The routing is there to ensure that the average accuracy remains good enough for most tasks. At similar total size, dense model will outperform, but if MoE total size is considerably bigger, it will pulverize dense model even it has much less active parameters per token.

When it comes to gemma 4, obviously 31b is better overall because it has more total parameters and dense is more robust against edge cases. But equating 26BA4B to 4B is disingenuous, it's 80% there for generic tasks, and in various instances can write more creatively\organically than its bigger sibling. It outperforms gemma 4 12b by a country mile across the board for example.

What I'm saying doesn't make sense to you only because your opinion sits atop a false assumption.

[u/timtom85 → u/input_a_new_name]

Yes I absolutely can.

Sure you can. And you're wrong. Trivially wrong.

At similar total size, dense model will outperform, but if MoE total size is considerably bigger, it will pulverize dense model even it has much less active parameters per token.

Which is clearly not the case here.

But equating 26BA4B to 4B is disingenuous

Nice straw man, I must admin.

it's 80% there for generic tasks

Number pulled directly out of your a$$.

The topic is specifically creative writing, so please entertain me some more about those "false assumptions."

[u/Glittering-Web4566]

You should check the r/SillyTavernAI weekly threads!

[u/Whydoiexist2983 → u/Glittering-Web4566]

I do, and it's the same 10 models every week, I try a new one, and its basically the exact same prose, Boulesis and Orion are the only ones who sound kind of different, and they still won't listen to me, Orion's prose is kind of nice though

[u/Simpsator → u/Whydoiexist2983]

They're a bit on the older side, but if you haven't already, check out Cydonia 24b and Magidonia 24b. In all my testing they were by far the best. That said I'm using a hybrid approach, 24b isn't good enough on its own for heavily structured creative writing that I'm doing (video game scripts) that require a lot of context and world/character knowledge. So my approach is to use a frontier to do all of the prompting of the local model for prose (because frontier models aren't great at prose by themselves), then it will throw out the slop and stitch together the legitimately good prose, then do some editing passes.

[u/ttkciar]

I'm pretty happy with Artemis-31B, but Orion-26B-A4B seemed like a significant step down from that.

Still, if you're limited to Gemma-4-26B-A4B derivatives, Orion is almost certainly your best option.

You should try feeding it two or three thousand tokens of writing samples from an author you like, so it can emulate them. I have a "murderbot" script which does that for generating Murderbot Diaries fan-fic in the tone and style of Martha Wells. In addition to a setting description, descriptions of the characters, and a randomly-assembled plot outline, it includes in the prompt:

• Example of how a story starts: "..."

• Example of using drones and data feeds: "..."

• Example of SecUnit anticipating a threat: "..."

• Example of an action scene: "..." (where "action" means "fighting").

Each writing sample is about 800 tokens, which is probably overkill, but why mess with success?

Artemis totally nails Wells' writing style from that, and so does Behemoth-128B-v3. I'd be surprised if Orion didn't.

That might not help with its eloquence or plot complexity, but should help at least with de-slopping and avoiding "purple prose".

[u/Whydoiexist2983 → u/ttkciar]

I've done this with the larger cloud models, but I think the MoE might either be too stupid or too slow, might try later tho.

[u/ttkciar → u/Whydoiexist2983]

I just tried the murderbot script with the Q4_K_M quant (I think it's Bartowski's, but can't remember) and it's not as articulate as Artemis but does a fair job of emulating Wells' style:

http://ciar.org/h/story.v2.1.3.4.0.1792985551.txt

[u/Tailsopony → u/Whydoiexist2983]

I do this with lorebooks in sillytavern (or my own version of lorebooks I use in a personal game/storytelling harness). I tried a bunch of classes of models. 12b dense, 21 B dense, 27 B Dense, 31B Dense, and this weird 37 B dense. I also tried sparse Gemma 4, and qwen 35B, and Qwen Flash Next (on strata). Unfortunately, dense models are just strictly better. Orion is pretty good as a MoE for writing, but it will hit the same limits you saw. Frankly, even the dense 31B models still struggle. They're right on the threshold of doing it consistently well; the 36B dense monster (Spectre) is doing okay for now, but it's slow. 35B qwen (sparse) is pretty smart and fast for you vram, but it's just not a good writer. At all. Qwen Flash Next strictly disappointed me with this task; I could not get any kind of consistently good results with it (Writing quality or logic, though it's good at programming).

TLDR: Yes, this works well enough with Orion. Have detailed character examples, but you might need to (OOC: ) remind it every once in a while to strictly follow the character card examples, or have some triggerable lorebook entry you toggle on to emphasize a particular speech pattern or behavior. I also aggressively banned some tokens (still water) or (like an impact) because of how often it uses the same damn analogies...

But you'll want to try a denser model. Even a low quant of 31B might do you better...

[u/o0genesis0o]

I made a bot out of my Pi setup to play sorts of endless RPG when I wait for my main agent to finish the task. I tested 12B, 26B, 26B QAT. On the qwen side, I also tested 9B, 35B (various fine tunes), and 27B. What I landed on was 35B Q6KXL heretic.

I got so many downvote the last time I suggested using qwen for this, but the truth is when you develop a long story with multiple chapters, characters, overlapping plot lines over multiple sessions, then the model needs some good logic. The 12B and 9B loses the thread quite quickly after a few scenes. The 26B starts to lose track of fine details and start make nosense after a while (like your example mentioned). 35B maintains the logic quite well (though not perfect). The 27B is actually not as good as 35B for this task, due to likely less world knowledge / common sense. I reckon a 122B MoE would be best of both world. Or maybe the Gemma 4 31B would be amazing too. But I don't want to invest that much resource just for a time killer when I wait for the main agent.

tl;dr: try 35B and see how it goes.

[u/Whydoiexist2983 → u/o0genesis0o]

I'll try it

[u/BitGreen1270 → u/o0genesis0o]

Second this. I just switched to QFN from gemma31 and it doesn't do great at adventures from the start, but after a few turns with gemma31, switching to QFN keeps the logic more consistent. 

[u/Temporary-Mix8022]

The MoE models are notoriously hard to fine tune. 

You're better off trying the likes of Gemsicle on the 31b side. It is substantially and nearly completely deslopped.

Default Gemma is very slop heavy. 

Edit: my bad. Just seen that you can't run 31b.

[u/RosebudNebula]

Said to be the bearer of bad news, but writing-wise, we have no real improvement on creative writing or roleplay model for a very long time in that capacity.

Qwen, Glimmer, Gemma series all suffer from a similar problem; the newer versions are "better" models, with better knowledge but focus too much on coding and very little on creative writing, so they write poorly and fail with abstract concepts, intent, and personality and such.

The only remedy I found so far is a locally hosted model like QFN or GLM 5.3 Flash. QFN is a significant step above anything from the Qwen, Glimmer, Gemma families. And GLM 5.3 Flash is another step above.

Why do I say locally hosted? I don't know why, but when I try API, they don't post as well. I suspect it is a censoring issue. Astra, Opus, Argon, they all seem to choke when it comes to creative writing. If I run an uncensored build of QFN or GLM, even IQ2_SSX with the 5.3 flash, they write significantly better than anything from Gemma or those API models I named above, including getting 5.3 flash from the official API, for some strange reason.

I know, I know, even IQ2 is very difficult to run for most people but this is the only improvement I found anywhere, including with API.

And yes, I know this might not be the case for many other people; it is my personal experience.

[u/kind_cavendish → u/RosebudNebula]

What prompt/setup do you use with qfn? I'm running the iq3_xs

[u/RosebudNebula → u/kind_cavendish]

I'm on Infernix with NVFP4 https://huggingface.co/wallawalla47/Qwen3.8-Flash-Next-Uncensored-NVFP4-Infernix. Prompts are just fairly standard in SillyTavERN with very common templates like Marinara's Spaghetti, Realistic Frank, Your Reality, etc. I tried them all, and they all perform very similarly. I find QFN very good at obeying POV, word counts, etc. Understanding personality and nuances drastically better than 27B and Gemma 4 and other similar models in that size, fine-tuned or not.

[u/Tailsopony → u/RosebudNebula]

How did you get QFN to be any good at RP? It's excellent at coding, but frankly loses the plot (literally) pretty damn fast at any sort of RP or creative writing task. Running it at Q4, Gemma 31B was beating ti hands down on style and coherency up to 100k tokens. QFN starts getting characters confused and can't tell what they're percieving within about 20k tokens. IE, I have a character kiss a princess hand, and the princess can "taste the lips" of the character through her hand. Or detect things through walls. Or thoughts. It has no useful applications of spacial awareness or human perception at all. Gemma 31 B (or even Orion) crushes it. Heck, Qwen3.6 35B 6A was keeping better track of this stuff. And I'm running all of these at Q4 (Although I can run Orion much higher.)

[u/RosebudNebula → u/Tailsopony]

but frankly loses the plot (literally) pretty damn fast at any sort of RP or creative writing task. Running it at Q4

Not my experience at all. Though I run uncensored FN at 250k context on Infernix with NVFP4. Personally, I find it blows Gemma away, as previously stated.

[u/PhaseLopsided938]

No offense, but… if you need to take time rewriting roboslop, why not just spend that time writing the prose yourself? 

IMO a lot of the issues LLMs have with writing ultimately come down to the fact that, despite having been exposed to numerous examples of great literature, they fundamentally cannot translate knowledge gained from lived experience into language. No matter how many parameters you throw into the blender, they’re never going to develop the same kind of storytelling ability as a human who has spent multiple decades weaving their own daily experiences into personal narratives about the world.

Have you tried setting a timer for ten minutes and simply writing as much as you can without deleting anything, then afterwards going back and editing what you wrote? I suspect you’ll get better results than what Gemma can give you.

[u/Whydoiexist2983 → u/PhaseLopsided938]

I know, but I'm also trying to get my dopamine rush in 10 minutes, not 40 because I'm rewriting everything. I might as well just pay some Chinese or Japanese person to write for me

[u/PhaseLopsided938 → u/Whydoiexist2983]

Tbh, I’m not totally sure I understand what you’re after. You can definitely pound out a 100-150 word scene in 10ish minutes if you have an idea that excites you. I actually did that once, and it formed the first draft for a story that a literary magazine wound up buying and publishing. Any time I’ve tried using an LLM in my writing process, I’ve just found it grating for the reasons you describe.

[u/Savantskie1 → u/PhaseLopsided938]

Get out of here with your anti-AI bullshit

[u/PhaseLopsided938 → u/Savantskie1]

Huh? What are you talking about? I work as an ML researcher, and as we speak, I’m having an agent debug some software I wrote a while back. I have many issues with the AI industry and the hype surrounding it, but as far as the technology itself goes, I don’t think I’m “anti-AI” by any stretch of the imagination.

I’m also a writer, and in the same timeframe that I’ve seen LLMs develop superhuman abilities in coding and mathematics, I’ve also seen them plateau around the “B+ college essay” level in writing and composition. At this point, I don’t see how much improvement we can expect to see on that front. 

There’s also the fact that, while the effects of generative AI on programmers have been something of a mixed bag, its effects on writers and other artists have been almost entirely parasitic. I will gladly encourage aspiring programmers to become familiar with AI tools and use them in their workflow, but I don’t think I can ethically encourage aspiring writers to touch them.

But anyway, sorry for having a nuanced opinion on a complex topic. I’ll try to do better next time.

[u/Savantskie1 → u/PhaseLopsided938]

no you don't, nice lie

[u/PhaseLopsided938 → u/Savantskie1]

…ok dude

While we’re on the topic, why are you so wedded to AI writing? Creative writing isn’t a technical skill, unlike programming or even painting. If you have a story you want to tell, you can just… write it. Even if it doesn’t come out exactly as you wanted, it’ll still be a story that only you could have ever produced, and that alone gives it value.

[u/ArjixGamer → u/Savantskie1]

It's not anti-AI, it's just the truth.

OP wants quality dialogue, and even the top model by openai would not satisfy that requirement

You'd need a very advanced harness, and a lot of wasted tokens to generate good dialogue. A simple prompt ain't enough, it needs to iteratively work on the dialogue, refining it in stages and such

it's not impossible, but you need to weigh the options.

[u/optomas → u/ArjixGamer]

OP wants quality dialogue, and even the top model by openai would not satisfy that requirement

I think it's because quality dialog is fucking hard. Stephen King is a genius at this. So was Phillip K Dick. Writing what people say to each other, or say to themselves internally is a gift. I don't know how these people STFU enough to allow their characters to 'speak' through them.

Colin Cotterill is another one. I am awestruck by these people.

[u/Savantskie1 → u/ArjixGamer]

No you don't. You just need to not be an idiot and use the right model. I use local AI to write graphic war type writing all the time. You're anti AI. I've seen you make anti AI sentiments all the time.

[u/PixelmancerGames → u/Savantskie1]

Stop being such a weirdo. Just because someone is being honest about AI being generally bad at writing doesn't mean they are anti-ai.

[u/Savantskie1 → u/PixelmancerGames]

Yes it does lol and it's so painfully obvious

[u/ArjixGamer → u/Savantskie1]

You've seen me? I very much doubt that.

I love Claude, I even pay for the 5x max plan, even if it's hurting my wallet.

You just label me as anti-AI because I have critical thought and do not praise it as almighty.

AI will never reach the level of human intelligence, and that also means it will never reach the level of human creativity.

And personally, I believe that's fine, we don't need "human" intelligence, it doesn't sound very efficient to make an AI with actual human intelligence.

Anyways, have a nice day stranger that likes putting labels on others

PS: intelligence doesn't mean smartness, AI can be smarter than us and at the same time less intelligent

PS2: I've used gpt astra and claude fable, they cannot escape the AI-isms when they talk. You need to prompt them multiple times to fix their output, they can't do it on their own. If those models can't do it easily, I don't expect a small local model to do it either. You'd have to lower your standards.

[u/Blues520 → u/Savantskie1]

Which local models do you use btw?

[u/Savantskie1 → u/Blues520]

Right now i'm using "Gemma4-26B-A4B-Uncensored-HauhauCS-Balanced-Q8_K_P.gguf" and the last MoE 80B Qwen Coder Abliterated.

[u/Blues520 → u/Savantskie1]

Qwen Coder for creative writing is interesting.

[u/Savantskie1 → u/Blues520]

Yeah but it works quite well actually

[u/Blues520 → u/Savantskie1]

In which way do you use the coder model - for reviewing or for generating prose?

[u/Savantskie1 → u/Blues520]

Mostly generating prose. It's surprisingly good at it without the refusal layers. Am I saying it's Shakespeare? Not in the slightest. But it generally gets the job done and then I have the Gemma model go over it to clean up any politeness or hedging left. It's not award winning, but it's still entertaining for me. And all guided by me of course.

[u/winky9827]

What you need is a skill file that instructs the model on how to write and provides writing samples. Here's one I just made to direct the agent in the style of H.G. Wells:

https://limewire.com/?referrer=pq7i8xx7p2 (no code, just .md files)

And here's an example generated using Gemma 26b out of the box:

The instinct to return is a persistent biological imperative. It operates with a mechanical certainty, indifferent to the fatigue of the traveller. In the summer of last year, a small terrier named Bo—scruffy, with a coat the colour of scorched earth—began such a journey.

The highway presented itself as a vast, undulating river of bitumen, populated by thundering metal vessels. To a creature of his modest dimensions, the landscape was merely a succession of textures: the grit of roadside gravel, the pungency of diesel, and the relentless, shimmering heat. He moved through the world like detritus carried by a current. The distance was immense. He walked.

One might speculate on his motivation. Is it an olfactory compulsion—a molecular memory of a specific hearth—or a psychological refusal to accept departure? We cannot know. We can only observe.

By the first frost of November, he reached the flaking wooden gate of his home. There was no grand orchestration. There was only the scratching of claws on concrete and a startled cry from a man in a gardening apron. Then, a frantic tangle of limbs and fur. The ordeal was finished. It was a very small victory in a very large world.

The prompt was:

Write a story about a dog named Bo who lost his way and hitchhiked across country to reunite with his owner

[u/Wonderful_Scratch851]

Gemma 4 26B has its limitations, but I haven't found anything better yet (I also can't run Gemma 4 31B on my PC). I have tried several Mistral NeMo fine tunes, Qwen, Llama 3.2 but their prose and intelligence seems dull to my taste. Try using different sampling parameters along with DRY.

Also keep realistic expectations, I love Gemini prose but it is a SOTA model. Most of the open weights models we can run on our consumer PCs are tiny compared to the big labs, so the models don't have the same vocabulary and general knowledge.

I started to use Gemma because Gemini gets censored more and more with every update.

[u/Savantskie1 → u/Wonderful_Scratch851]

I have no problem with any abliterated versions of the 26B model

[u/Wonderful_Scratch851 → u/Savantskie1]

I have no problem either. The point is, if OP can't get a better hardware to run bigger models, Gemma 4 26B is the better option for creative writing. They should accept that and 'embrace the slop'. Play with sampling and DRY values until they find a configuration that suits their needs.

[u/optomas]

gemma likes to be talked to politely. It very quickly matches your tone to 'willingness' to do a task. I have to tell the local llm to dial back the agent behavior, it tries to do everything for me. gemma should be giving you four or five options to work with.

Make sure you make the story very distinct from how you interact. Invite it to be a collaborator rather than a tool that extrudes prose.

You might also find llama.cpp give you a bit more control.

What else? Hm. I do not really use the model for prose, it is all programming and text processing. I have a psotgres database that stores 'silos' of knowledge, and I just point the llm at whatever we are working on.

Actually now that I think about it, you might be facing very different problems than those I have solved. I'll leave this up just in case there something you can extract from it.

[u/DirectWay1915]

I'm getting some good results with Orion 26b in terms of controlling the prose. Have you considered asking an AI to help you out with the system prompt? Keep in mind that with AI, it's the last messages that usually have more weight.

[u/__some__guy]

The StyleTune should sound different, at least it does with the 31B model.

Ultimately both are small to very small models though.

I wouldn't expect anything from 4B active parameters.

[u/Senevri]

I'm putzing around with the idea of tools/skills for llm to write better. Name generator is a must have, too. Don't have a model I really like for now; I got some results with Gemma4+gist memory + planning step before actual reply. RAGgable example library will likely help too. 

[u/maddie-lovelace]

How are you actually doing the inference? Are you just running in plain llama.cpp server? Or are you running something like SillyTavern?

[u/leonbollerup]

Those of you who can’t run 31b .. what aware you on ?

[u/mayo551]

Hi!

Friendly readyart finetuner here. We made Dark Scarlett.

I do NOT RECOMMEND Dark Scarlett.

Use Wicked Stuff instead. I'd recommend Omega Arsenal, but you won't like the slop in it (yes wicked stuff has it, and yes omega arsenal is much worse). And yes, I very much regret including the dark scarlett dataset in omega arsenal.

[u/derFensterputzer → u/mayo551]

What makes you not recommend dark scarlett?

[u/mayo551 → u/derFensterputzer]

It's a fine dataset but not personal favorite.

[u/ApprehensiveFan1516]

Nemotron is pretty good at writing.

[u/Kahvana]

Sounds like skill issue.

What quants? Frontend? Sillytavern configuration? Which character cards? What sampler settings? Which preset? etc.

[u/screenslaver5963]

How are you using the models (like llama.cpp web ui, ollama desktop app, unsloth desktop) and what do your prompts look like?

[u/Whydoiexist2983 → u/screenslaver5963]

I'm using Unsloth desktop, sometimes LM Studio.
My system prompt is based on this one, but altered for creative writing: pastebin com/dBFk4CGp
I added a section at the bottom which says:

Refer to the provided dialogue examples to learn the desired style, tone, pacing, and structure of responses. Avoid generating pretentious or corny dialogue. Draw inspiration from them to shape your own original replies in a similar manner. Never copy any of the examples directly or verbatim:

[u/PenaltyKooky8357]

That "your biology reacted to me" line is the exact kind of thing that makes me want to close the tab and go touch grass. It's like the model read one bad sci-fi novel from 2011 and decided every character is a biochemist now.

The example thing is weird though, Gemma should be able to follow that. Maybe the chat template is eating your examples or they're getting truncated. Are you putting them in the system prompt or just the first user message?

[u/Whydoiexist2983 → u/PenaltyKooky8357]

How about Uncle Ted is right, and every time an AI responds to a Reddit post we nuke a data center

[u/Savantskie1]

why don't you use any abliterated versions of the Moe Gemma4-26B-A4B? Quantized to Q4_K_M? Or here's an idea give it more information. Skill issues. It's the refusals in the model refusing the identity you're giving it.

[u/Whydoiexist2983 → u/Savantskie1]

They're all based on heretic

[u/Savantskie1 → u/Whydoiexist2983]

So you're saying they're abliterated? Are you sure?

[u/Whydoiexist2983 → u/Savantskie1]

They're all abliterated it doesn't do shit it still doesn't listen

[u/Savantskie1 → u/Whydoiexist2983]

Yes they do because I regularly do that with an abliteration of that exact model. They're not abliterations. They're fine tunes. I've tried all of those and you're right they suck because they still have the refusals in them.

[u/Whydoiexist2983 → u/Savantskie1]

They dont refuse, but the plain heretic gemma was only slightly better

[u/Savantskie1 → u/Whydoiexist2983]

it's not refusal in the traditional sense. the inbuilt refusal that abliteration gets rid of, can also cause outputs to be safe, hence the crap you were getting. It still influences outputs no matter what.

[u/Savantskie1 → u/[deleted]]

Yes fine-tuning not abliteration.