earnings-regression
Earnings Regression
Does more schooling, or more years of work, mean a higher hourly wage? This fits a trend line through a public survey of American young people: move two sliders, schooling and experience, and it predicts a wage. The answer: those two facts barely predict anything. Tested on people it had never seen, it scores -0.008, slightly worse than guessing the average wage for everyone, reported rather than hidden.
Solo work by Salman Adnan.
Overview
A multivariable linear regression that predicts hourly earnings from years of schooling and years of work experience, using a 2,000-row subset of the NLSY97 (National Longitudinal Survey of Youth 1997-2011), served through an interactive Plotly/Flask dashboard as well as a plain headless script.
This is a compact, end-to-end regression exercise: load survey data, clean it, explore it, fit a simple model then a multivariable model, check the residuals, and use the model to make a concrete prediction. The dashboard on top turns the static output into something you can actually interact with: move a slider, see a real prediction, and look at the whole schooling/experience plane at once instead of a single point estimate.
Key features
- Loads and cleans the NLSY97 subset (drops rows with missing values and duplicates), then fits a simple regression (earnings on schooling alone) and a multivariable regression (earnings on schooling and experience), reporting R-squared on both a training split and a held-out 20% test split.
- A live prediction panel: two sliders (schooling, experience) that call a
/predictAPI endpoint on every move and display the real model's output, not a hardcoded formula. - A Plotly scatter of actual vs. predicted earnings on the held-out test split, with a y = x reference line for perfect prediction.
- A 3D Plotly surface of predicted earnings across the entire schooling/experience plane, with the real training rows plotted on top of it as points.
- A residuals plot (predicted minus actual) against predicted earnings, plus a headless mode (
main.py) that still prints the same exploration output and writes the same two static PNGs it always did.
Verification
The prediction endpoint was hit directly and returned a real model output: {"experience":5.0,"predicted_earnings":20.29,"schooling":16.0} for 16 years of schooling and 5 years of experience. The multivariable model's held-out test R-squared came out at -0.008, meaning on this split it performs marginally worse than predicting the mean training earnings for everyone. That's reported as-is rather than hidden: with only two features and 492 usable rows after dropping missing values, the model isn't capturing much of what actually drives earnings.
Tech stack
A challenge worth noting
The source script's comments claimed to fit a model on schooling alone, but the code actually fit schooling and experience together in one step, skipping the simple model entirely. Fixing that meant actually implementing both: a simple model on schooling alone and a multivariable model on both features, so the comparison the comments describe exists in the real output. Keeping two entry points (a headless script and the Flask dashboard) from silently drifting apart was solved by pulling all fitting logic into one shared model.py, so both call the same fit_models() and are structurally guaranteed to use the same split and coefficients rather than two copies that happen to agree.