Urdu Language Model, from scratch

Overview

A small Urdu language model trained from scratch: the full stack, from raw public text to a decoder-only transformer, built for a low-resource language that off-the-shelf tokenizers and models handle poorly.

Most projects that train a small GPT reach for English data and a pretrained tokenizer. Urdu (Perso-Arabic script, right-to-left, rich morphology) exposes the parts those projects skip: script normalization, a tokenizer that does not shred the language into bytes, script-aware language-ID filtering, and an honest token budget.

Everything here is written from scratch in PyTorch, with no modeling code from the transformers library.

Key features

• A streaming, resumable data pipeline that downloads and cleans public Urdu text into tokenized shards.
• A 32k byte-level BPE tokenizer trained on the cleaned corpus, with a measured compression comparison against GPT-2 on Urdu.
• A decoder-only transformer with RMSNorm, rotary position embeddings, SwiGLU, tied embeddings, and grouped-query-capable attention, in four presets (13.0M / 35.7M / 123.7M / 336.4M).
• A training loop with AMP, gradient accumulation, cosine LR with warmup, full resume, and single-GPU and DDP paths.
• Evaluation: held-out perplexity, a script-mix diagnostic, and a 95-item few-shot cloze harness.

Results

• v1 corpus: 3,695,487 raw units filtered and deduplicated to 2,515,082 documents (438.5M chars), 111.7M train tokens.
• v2 corpus: added CC-100 Urdu plus a newer Wikipedia dump, deduplicated to 13.3M documents. Two tokenized outputs: 862.4M-token main mix (Wikipedia upsampled 3x) and a 70.8M-token Wikipedia-only anneal set, about 7.7x more unique training tokens than v1.
• Tokenizer: on an 806,495-character held-out Urdu sample, the 32k Urdu BPE uses 4.078 chars per token; GPT-2 spends 5.23x more tokens on the same text.
• Tiny proof run (13M preset, CPU, 500 steps, 4.10M tokens): train loss 9.74 to 5.42, held-out perplexity 248.8, and 98.2% of generated characters stay in Arabic script.
• Base run v1 (124M preset, A100, 858 steps, 449.8M tokens): loss 7.35 to 3.51, ~75 minutes, about $1.90. Held-out perplexity 38.02, script mix 99.43%, cloze 34.9%.
• Base run v2 (124M preset, A100, 4,768 steps, 2.5B tokens): loss 9.63 to ~2.74, real sustained throughput ~105,000 tokens/second, 6h 39m, about $13.06. Held-out perplexity 28.93, script mix 99.75%, cloze 39.5% (2-shot, chance is 33.3%).
• 13 tests passing (5 model, 8 pipeline), including a causal-masking property test and per-preset parameter counts.

Reading the base model honestly

Script mix confirms the model reliably writes in Urdu's actual script, near-perfect in v2 at 99.75%. The v2 corpus expansion produced a real, meaningful perplexity improvement (38.02 to 28.93), consistent with roughly 7.7x more unique tokens and a token-per-parameter ratio much closer to Chinchilla-optimal. Cloze accuracy moved from 34.9% to 39.5%, a 2-item difference on a 43-item eval set whose standard error is roughly 7 points, so this specific number is inside the noise band rather than a proven gain.

Why cloze barely moved despite a much bigger corpus: research on language model fact storage (Roberts et al., 2020) finds that how many discrete facts a model can retain scales mainly with parameter count, not training-text volume. A 124M-parameter model has a real capacity ceiling for factual recall regardless of how much prose it reads. More text raised fluency; it did not raise how many facts the model can store.

The eval set itself was also too small to trust: 45 items carries a standard error of roughly 7 points, wide enough to make a 2-item swing look like progress. It has since been expanded to 95 items (standard error roughly 5.2 points), and the v2 model's real score on the larger set is 45.2% (42/93, 2-shot), the new baseline for future comparisons. Two pieces are built and tested but not yet run: a 336M `medium` preset, and a Wikidata-derived declarative-fact training source (verified live against the public SPARQL endpoint, 890 real triples fetched, 865 surviving corpus-wide dedup), both aimed directly at the fact-storage-capacity ceiling above. A real, honestly-caveated cost estimate for that run is in the README: roughly $40-45 and 29 GPU-pod hours, not yet spent.

Three real v2 generations, shown as evidence rather than cherry-picked: a prompt about Pakistan drifts into religious-history fragments, grammatically fluent Urdu but topically disconnected, a different instance of the same drift seen in v1; a prompt about the Urdu language stays on-topic this time with coherent sentences about Urdu literature, an improvement over v1's repetition-loop failure on the same prompt; a prompt about today's weather produces fluent, topically relevant sentences about winter weather. The honest read: v2 is more fluent and holds topic longer in two of three samples, but topical coherence is still not reliable prompt-following, and factual recall did not move meaningfully, exactly what the fact-capacity research predicts for a corpus-only change at this model size.

Tech stack

• Python
• PyTorch
• Byte-level BPE
• numpy
• datasketch (MinHash LSH)
• pytest

Challenges

• Near-dedup was two hours slow with per-shingle MinHash updates (about 250 docs/s); rewriting the signature into two vectorized numpy operations took it to about 2,400 docs/s, a roughly 10x speedup, with the same duplicates caught.
• The tiny preset is embedding-bound: at a 32k vocabulary the tied embedding table alone is 8.2M parameters, so a genuinely tiny transformer still lands around 13M, reported honestly rather than shrinking the vocab.
• Language ID without a heavyweight model: because Urdu shares its script only with other Perso-Arabic languages absent at volume here, a script-range ratio catches the real contaminant (embedded English) directly.

What I learned

• Vectorizing the hot loop matters more than picking a fast library: the same LSH went from unusable to fine by moving per-element work into two numpy calls per document.
• With a real 32k subword vocabulary, a 10M-parameter model is a fiction below a certain size because the embedding table dominates.
• SwiGLU at an 8/3 hidden ratio matches a 4x GELU block's parameter count, so a SwiGLU model needs more depth to reach a target size.
• A resumable, token-count-scheduled loop up front is the difference between the GPU run being one command and babysitting step counts.