fix(cian): persist detail rich fields (description) + stamp detail_enriched_at #1718

Merged

lekss361 merged 1 commit from fix/cian-detail-rich-fields into main

2026-06-17 19:42:59 +00:00

lekss361 commented

2026-06-17 19:33:12 +00:00

Owner

Summary

Fixes cian detail enrichment silently looping with no persistence. Root cause in cian_detail.py::save_detail_enrichment:

detail_enriched_at never stamped (primary). The cian save UPDATE omitted detail_enriched_at = NOW() (avito_detail.py:581 has it). The sweep selects WHERE detail_enriched_at IS NULL (scrape_pipeline.py:1553/1935), enriches, bumps the detail_enriched counter (~50/run — looked "healthy"), but never stamps → same rows re-selected every run, column stays NULL for ALL cian rows forever. Prod: 0/19461 cian rows stamped.
description never extracted/saved — DetailEnrichment had no description field; parser only read offer.description for repair_state inference. Prod: cian description = 0 filled.

(ceiling/kitchen already in the save UPDATE; low fill was a side-effect of the limping enrich loop. ceiling_height_m=0 in prod is expected — that's avito's column; cian uses ceiling_height.)

Changes (cian_detail.py only)

DetailEnrichment += description.
Parser: description = offer.get("description") or None.
save_detail_enrichment UPDATE: + description = COALESCE(:descr, description) and detail_enriched_at = NOW().
priceChanges logic (#1657) untouched; no save signature change.

Test plan

pytest -k cian → 222 passed; ruff clean
parser extracts description; save passes descr; save SQL stamps detail_enriched_at
Post-deploy smoke: after a cian sweep, SELECT count(*) FILTER(WHERE detail_enriched_at IS NOT NULL), count(*) FILTER(WHERE description IS NOT NULL) FROM listings WHERE source='cian' both climb from 0.

Existing cian rows self-heal (still match detail_enriched_at IS NULL) — no backfill needed.

Refs #1657

## Summary Fixes cian detail enrichment silently looping with no persistence. Root cause in `cian_detail.py::save_detail_enrichment`: 1. **`detail_enriched_at` never stamped** (primary). The cian save UPDATE omitted `detail_enriched_at = NOW()` (avito_detail.py:581 has it). The sweep selects `WHERE detail_enriched_at IS NULL` (`scrape_pipeline.py:1553/1935`), enriches, bumps the `detail_enriched` counter (~50/run — looked "healthy"), but never stamps → same rows re-selected every run, column stays NULL for ALL cian rows forever. Prod: 0/19461 cian rows stamped. 2. **`description` never extracted/saved** — `DetailEnrichment` had no `description` field; parser only read `offer.description` for repair_state inference. Prod: cian description = 0 filled. (ceiling/kitchen already in the save UPDATE; low fill was a side-effect of the limping enrich loop. `ceiling_height_m`=0 in prod is expected — that's avito's column; cian uses `ceiling_height`.) ## Changes (cian_detail.py only) - `DetailEnrichment` += `description`. - Parser: `description = offer.get("description") or None`. - `save_detail_enrichment` UPDATE: `+ description = COALESCE(:descr, description)` and **`detail_enriched_at = NOW()`**. - priceChanges logic (#1657) untouched; no save signature change. ## Test plan - [x] `pytest -k cian` → 222 passed; ruff clean - [x] parser extracts description; save passes descr; save SQL stamps detail_enriched_at - [ ] Post-deploy smoke: after a cian sweep, `SELECT count(*) FILTER(WHERE detail_enriched_at IS NOT NULL), count(*) FILTER(WHERE description IS NOT NULL) FROM listings WHERE source='cian'` both climb from 0. Existing cian rows self-heal (still match `detail_enriched_at IS NULL`) — no backfill needed. Refs #1657

lekss361 added 1 commit 2026-06-17 19:33:12 +00:00