balochi-poet-classifier
Balochi Poet Classifier Team
Given two lines of Balochi poetry, can a computer tell which of three poets wrote them? Balochi has almost no language technology built for it, so even assembling the 1,670 hand-checked verses mattered; three students shared that work. Salman built the part that cleans the verses and teaches the computer to tell the poets apart; that step needs a faster machine than he had, so no success rate is claimed.
Salman's part of a three-person course project: the classification pipeline (preprocessing, training, evaluation, inference). Preprocessing runs end to end on the real dataset and the training and evaluation code is verified to execute, but full fine-tuning of the ~278M-parameter model was not completed (CPU was about 28 seconds per step). No accuracy number is claimed.
Overview
Authorship attribution for Balochi poetry: given a single couplet, predict which of three poets wrote it, by fine-tuning xlm-roberta-base for 3-way sequence classification. Balochi is a low-resource language with almost no prior computational work, which is the point.
The approach shows that a multilingual transformer can be fine-tuned for author attribution with under 2,000 labeled verses. Built as a university Deep Learning course project by a team of three; this repository is Salman's part, the classification pipeline. Dataset collection was a team effort.
Dataset
- Data.csv: 1,670 couplets in Perso-Arabic script. Adam Haqqani 860 (51.5%), M.H. Khalil 670 (40.1%), Mir Ahmed Ali Mir 140 (8.4%).
- Verses are short: 3 to 58 characters, mean 33.6. No rows are dropped by cleaning.
- Provenance: compiled from the Balochi Academy's digital archive and published poetry books, extracted with OCR and PdfPlumber, then manually verified.
Approach
- Fine-tune xlm-roberta-base, chosen because its SentencePiece vocabulary already covers Perso-Arabic characters, so Balochi tokenizes into subwords instead of unknown tokens.
- Class imbalance (8.4% minority class) is handled with sklearn balanced class weights (computed 0.647 / 0.832 / 3.973) fed into a custom WeightedTrainer that overrides compute_loss with weighted cross-entropy.
- Stratified 70/15/15 split, seed 42: train 1,168, val 251, test 251. Batch 16, lr 2e-5, max 10 epochs, early stopping on weighted F1.
Results
No accuracy is reported. The pipeline runs end to end on the real dataset, the training and evaluation code is verified to execute (a few steps on CPU with a tiny stand-in model under transformers 4.57.6), and the honest next step is a GPU run on a free Colab T4, which should finish under an hour.
Tech stack
Challenges
- CPU training is not viable for this model: each optimizer step took roughly 28 seconds, about 6 hours per epoch, so the repo stays CPU-runnable for preprocessing and quick checks and points training at Colab.
- Heavy class imbalance: a model on raw cross-entropy would ignore the 8.4% class and still look fine on plain accuracy, so weighted loss and weighted-F1 monitoring were used.
- A transformers API deprecation broke the trainer (a renamed argument, a deprecated tokenizer parameter, and an extra compute_loss argument); the fix was verified by running a few real steps against a tiny random model.
What I learned
- Wall-clock cost per step is the number that decides whether a plan is real; measuring one step early (28s) would have saved the whole first attempt.
- Class weights change what the loss optimizes, not what accuracy reports, so the metric you monitor matters as much as the loss.
- Pinned dependency ranges do not protect you from library API drift; the only way to catch it was to actually execute the trainer.
- Stratified splitting keeps class ratios stable but does nothing about leakage between couplets from the same poem.