Today Oscowl AI launches Wiola, our first Small Language Model (SLM) built entirely from first principles. Wiola represents a deliberate departure from off-the-shelf transformer variants and large-scale model recipes; instead, we designed every component with clarity of assumptions, efficiency, and practical deployment in mind. The research paper is available now, and the model will be open-sourced on our website on Monday. Read the paper: https://lnkd.in/dXJfmy2w. Visit the project page: https://lnkd.in/dycaEERm. Why build Wiola from first principles? Reexamining assumptions: Modern LLM progress often stacks scale on top of similar architectural patterns. We asked: which design choices are essential, and which are inherited baggage? Wiola reconsiders tokenization, attention mechanics, optimization, and inference trade-offs from ground-up principles. Practical SLM needs: Not every application requires massive models. Edge deployment, privacy-preserving on-device inference, and cos...
Most language models today are built on the same underlying blueprint. While scaling has improved performance, the core architecture itself has remained largely unchanged. Wiola is an effort to rethink that foundation from first principles. Instead of introducing incremental improvements, Wiola redesigns the internal structure of a language model. The focus is not only on performance, but on how information is represented, processed, and preserved across layers. At the positional level, Wiola introduces a new encoding mechanism that goes beyond linear token representations. It captures structure across multiple scales, allowing the model to better understand both local and long-range relationships. This is complemented by a cross-layer attention mechanism that allows deeper layers to access compressed summaries from earlier layers, preventing useful information from being diluted as the model gets deeper. Efficiency is achieved through adaptive token processing, in which semantic...