The information ceiling: why an AVM built on public sales data tops out at 12% MdAPE

Background

I run a side project called RealPing: it takes Taiwan’s public real-estate transaction batches (the LVR dataset) and cleans them into a “usable floor area” per-ping price — parking stripped out, common areas stripped out. Taiwan’s official unit price divides by a denominator that includes common areas and parking, so the headline number runs roughly 45% below the real cost of usable space. That “common-area-stripped” layer is itself something nobody else publishes.

Once the cleaning layer was solid, the obvious next move was an AVM — an automated valuation model that takes a property’s features and returns an estimated unit price. Zillow’s Zestimate, Taiwan’s House+, all the same idea. I built one, measured it, and landed on a conclusion far more interesting than “I hit X% accuracy 🎉” — this data can’t estimate accurately, and I can prove why.

This is that negative result.

How it was built

The target is log(usable unit price). The model is a gradient-boosted tree (sklearn’s HistGradientBoostingRegressor by default, auto-switching to lightgbm when it’s importable). Every feature comes from columns already in the data — no external sources:

• Categorical: city, building type, primary use, elevator (y/n), total floors
• Numeric: building age, bedrooms / living rooms / bathrooms, common-area ratio, usable ping, transaction year-month

The one subtlety is the district. There are ~370 of them nationwide — well past sklearn’s native-category limit — and the district is the single strongest signal in house prices. So I target-encode it: replace each district with the median log price of that district in the training set. This is where time leakage loves to creep in, so the encoding table is computed on the training set only, serialized into the artifact, and unseen districts fall back to city median, then national median.

Validation is always an out-of-time split: hold out the most recent quarter as the test set, simulating “use past sales to estimate future sales” — never a random split. A random split lets the model peek at the future and inflates the score.

Results

107k live records, out-of-time (train 105,579 → test 2026 Q1, 1,885 rows). I added features in stages to watch each one’s marginal lift:

model                       MdAPE    PE10     PE20     MAPE     bias
─────────────────────────   ─────    ─────    ─────    ─────    ─────
district baseline           13.2%    41.8%    66.4%    29.8%    +0.8%
+ street-level location     12.4%    42.1%    67.5%    27.9%    +1.0%
+ floor                     12.7%    42.3%    68.5%    27.5%    +1.0%
      metrics as features are added (more features lower down)
    

Across Taiwan’s six metros, MdAPE lands at 10–13%, no single city collapses, bias is near zero. Sounds fine — until you put it next to the industry benchmark: mature comps-based AVMs run PE10 around 60–70%, MdAPE 5–8%. My PE10 is 42% — meaning out of every ten estimates, only four land within ±10% of the real sale.

That gap isn’t a bug. It’s the whole point of this post.

Two negative results

Most of my effort went into pushing that number down. Two intuitive routes, two walls.

One: refining location barely helped. Going from “district” to “street-level” (parsing the road name from the address, with hierarchical smoothed encoding) moved MdAPE only from 13.2% to 12.4%. More telling: it was completely insensitive to the smoothing strength — I swept the smoothing coefficient from 3 to 40 and MdAPE just wandered between 12.4–12.7%. If signal were being smoothed away, that knob should respond. It didn’t, which means within-district location variation was never the main residual in the first place.

Two: adding floor was a no-op. I parsed unit floor, total floors, and relative floor from the address and fed them in. PE10 / MdAPE all moved inside the noise band (MdAPE even ticked up slightly). Floor isn’t the main residual either.

”What about external data?” — that won’t save it either

Every time I get here, someone says: just geocode it, join POI data, pull in external sources. I checked, adversarially (deliberately hunting for counter-evidence), and the conclusion is: for this problem, no external-data route pays off.

• Geocoding is fully feasible and useless here: I measured ~100% of addresses carry a complete street number on the input side, and the official address basemap resolves ~100% to building level on the output side (even rural and outer-island place-name addresses). But since within-district location variation isn’t the main residual, pinning coordinates to lat/long adds <1pp. Demoted to backlog.
• POI / amenity features add ≈0 once the model already has location: this is supported in the literature — once a model knows neighboring sale prices, adding convenience-store density or transit distance barely lifts anything. My location signal is already saturated.
• What would actually help is interior photos / condition — but the only source rich enough is 591, and scraping its public listings to build a database is a legal no-go zone in Taiwan (Fair Trade Act Article 25, with an FTC enforcement precedent). The legally safe sources are useless; the useful source is illegal.
• Even repeat-sales is out: addresses with a floor suffix are unit-unique, but there are only 1.04 sales per address on average, and just 2.9% of addresses have ≥2 sales. Same-unit repeat transactions are too sparse to carry the method.

So I stopped

The model is built, tested, documented. Another version of me would now bolt on MLflow, wire up a model registry, ship it to serving, schedule automated retraining — an entire MLOps machine around this model.

I didn’t. Because that machine’s core assumption is “the model keeps getting better, so it needs monitoring, retraining, iteration” — and I’d just proven it doesn’t get better. Retraining can’t fix a problem whose bottleneck is missing columns, not a stale model. Building infrastructure for a problem you don’t have is engineering busywork.

Takeaways