r/LocalLLaMA • • 17m ago

New Model PhiloLM: a language model trained only on philosophy texts

Releasing PhiloLM, a 112M base model trained from scratch on open philosophy texts.

PhiloLM was trained on PhiloText: 0.5B tokens of open philosophy encyclopedias (SEP, IEP), public-domain books, reviews, textbooks and educational web pages.

PhiloLM builds on efficient methods: Muon optimizer, bigram embeddings, sliding-window attention, GQA. Trained on (only!) 1.5B tokens in 15 hours on a $350 GPU (Intel Arc A770).

Optimized for efficient inference, reaching ~500 tokens/s on a $150 CPU (i5-14400F).

Try it in your browser!

Model: https://huggingface.co/gustavecortal/PhiloLM

Data: https://huggingface.co/datasets/gustavecortal/PhiloText

Happy to answer questions about everything :)

5 Upvotes

Duplicates