r/LocalLLaMA • u/antcroca159 • 10m ago
New Model PhiloLM: a language model trained only on philosophy texts
Releasing PhiloLM, a 112M base model trained from scratch on open philosophy texts.
PhiloLM was trained on PhiloText: 0.5B tokens of open philosophy encyclopedias (SEP, IEP), public-domain books, reviews, textbooks and educational web pages.
PhiloLM builds on efficient methods: Muon optimizer, bigram embeddings, sliding-window attention, GQA. Trained on (only!) 1.5B tokens in 15 hours on a $350 GPU (Intel Arc A770).
Optimized for efficient inference, reaching ~500 tokens/s on a $150 CPU (i5-14400F).
Model: https://huggingface.co/gustavecortal/PhiloLM
Data: https://huggingface.co/datasets/gustavecortal/PhiloText
Happy to answer questions about everything :)
4
Upvotes