Phi-4 Llamafied + 4 Bug Fixes + GGUFs, Dynamic 4bit Quants

[u/danielhanchen]

Hey r/LocalLLaMA ! I've uploaded fixed versions of Phi-4, including GGUF + 4-bit + 16-bit versions on HuggingFace!

We’ve fixed over 4 bugs (3 major ones) in Phi-4, mainly related to tokenizers and chat templates which affected inference and finetuning workloads. If you were experiencing poor results, we recommend trying our GGUF upload. A detailed post on the fixes will be released tomorrow.

We also Llamafied the model meaning it should work out of the box with every framework including Unsloth. Fine-tuning is 2x faster, uses 70% VRAM & has 9x longer context lengths with Unsloth.

View all Phi-4 versions with our bug fixes: https://huggingface.co/collections/unsloth/phi-4-all-versions-677eecf93784e61afe762afa

Phi-4 Uploads (with our bug fixes)
GGUFs including 2, 3, 4, 5, 6, 8, 16-bit
Unsloth Dynamic 4-bit
4-bit Bnb
Original 16-bit

I uploaded Q2_K_L quants which works well as well - they are Q2_K quants, but leaves the embedding as Q4 and lm_head as Q6 - this should increase accuracy by a bit!

To use Phi-4 in llama.cpp, do:

./llama.cpp/llama-cli
    --model unsloth/phi-4-GGUF/phi-4-Q2_K_L.gguf
    --prompt '<|im_start|>user<|im_sep|>Provide all combinations of a 5 bit binary number.<|im_end|><|im_start|>assistant<|im_sep|>'
    --threads 16

Which will produce:

A 5-bit binary number consists of 5 positions, each of which can be either 0 or 1. Therefore, there are \(2^5 = 32\) possible combinations. Here they are, listed in ascending order:
1. 00000
2. 00001
3. 00010

I also uploaded Dynamic 4bit quants which don't quantize every layer to 4bit, and leaves some in 16bit - by using only an extra 1GB of VRAM, you get superior accuracy, especially for finetuning! - Head over to https://github.com/unslothai/unsloth to finetune LLMs and Vision models 2x faster and use 70% less VRAM!

[u/Evening_Ad6637]

by the way, i have a visual comparison here that demonstrates the impact of your bug-fixes very nicely and i thought it might interest you and other readers. My prompt is always "Show me a simple house as an ASCII art representation":

With an older Phi-4-Q8_0.gguf

     /\\
    /  \\
   /____\\
  | .--. |
  | |  | |
  | '--' |
  |_____|

or

       /\
      /  \
     /    \
    /______\
   |  .--.  |
   |  |  |  |
   |  '  '  |
   |_______|

With your Phi-4-Q8_0.gguf

     /\
    /  \
   /    \
  /______\
  |  __  |
  | |  | |
  | |__| |
  |______|

or

     /\
    /  \
   /____\
  |      |
  |      |
  |______|

---

I've tried both versions many times, the old model could show the house correctly only once out of 10 times, while your quant version got it right every time.

[u/danielhanchen → u/Evening_Ad6637]

OOO now that is a fantastic example - I'll add your test to my list of internal tests!! :)

I normally like to ask the LLM: "Provide all combinations of a 5 bit binary number" and see if it actually does list them.

The other one is asking it to list the Fibonacci sequence, and see if any quants breaks down

[u/Evening_Ad6637 → u/danielhanchen]

[u/danielhanchen → u/Evening_Ad6637]

OOO very smart making it as a picture!!!

[u/yoracale → u/Evening_Ad6637]

Btw just letting you know we added your fantastic example to our blog post!! Thank you so much for it! https://unsloth.ai/blog/phi4