The data-freshness monitor classified by run RECENCY only, so the domrf_kn
FLATS loader running status=done but extracting 0 flats for ~5 weeks went
undetected — and the kn source watched objects_count (healthy ~1548), not
flats_count (the broken =0 metric).
Add an opt-in zero-output check: an otherwise-fresh run-ledger source (recent
success, would-be fresh by age) that produced 0 work-rows in the 7d window is
downgraded to status="failed" (so scrape_freshness_check alerts), with an
additive "reason". Guards: alert_on_zero_output flag, run-ledger only
(timestamp_col is None), status=="ok" (age-stale/failed already covered), and
upd_7d==0 (SUM of the source's own work_col over done-runs).
Registry: new kn_flats source (kn_scrape_runs, work_col=flats_count, critical,
flag on) — watches the column that was broken; existing kn (objects_count)
unchanged. Flag also enabled on objective (rows_lots, critical). nspd/nspd_geo/
cadastre left unflagged (legitimate-0 / data-table).
JSON additive only (new nullable "reason" key; endpoint is dict[str,Any], no
frontend consumer / no codegen needed). 4 new tests (downgrade, no-false-
positive, age-precedence, registry). code-reviewer APPROVE.
Would have caught #1945 within ~8-14d instead of 5 weeks.
_fetch_anchor_comps Tier A runs on EVERY /estimate (flag
estimate_same_building_anchor_enabled defaults True, not overridden in prod).
Its address predicate wraps address in lower(translate(address,'ёЁ','ее')) for
both the LIKE '%street%' and the ~house-number regex — non-sargable, and the
existing listings_address_trgm_idx is on the RAW address, so the planner
seq-scanned all ~66.7k listings (167ms, 15801 buffers) every estimate.
Add a GIN trigram expression index on lower(translate(address,'ёЁ','ее'))
matching the predicate exactly (gin_trgm_ops serves both LIKE and the regex).
Prod EXPLAIN (BEGIN/ROLLBACK, "Хохрякова 48"): 167ms -> 1.77ms (~94x), buffers
15801 -> 97. Independently dry-run-verified: Bitmap Index Scan, 0.66ms.
Result-identical (pure access-path; count/sum/min/max unchanged). No code
change (expression matches the predicate). ~8MB, sub-second build.
Other estimate-path queries confirmed already optimal (Tier W/H geom+rooms
BitmapAnd, Tier S house_id_fk) — no other index warranted.
tradein-mvp/backend/data/sql/137 (correct tradein migrations path).
domrf_kn_objects is a snapshot dimension (UNIQUE (obj_id, snapshot_date), ~8
snapshots/obj_id). _SUPPLY_BATCH_SQL joined flats to ALL object-snapshot rows
(no o.snapshot_date filter), counting each flat ~8.5x → supply_units_in_radius
inflated ~8.5x, sold_pct_of_supply deflated ~8.5x, is_oversold under-fired
(all user-facing, best_layouts.py:571-611; sold_pct=deals/supply is a raw
ratio so no canceling).
Fix: dedup objects to one row per obj_id (latest-snapshot coords) via
DISTINCT ON in an objects-first MATERIALIZED CTE, then join domrf_kn_flats via
idx_kn_flats_obj. units now = one count per flat (prod cross-check at radius
1.5km: units == count(*) == count(DISTINCT f.id) == 9612 for 65 objects;
correction factor 8.56x at 1.5km, 9.13x at 1.0km). This also aligns the supply
denominator with the deals numerator (_COMPETITORS_IN_RADIUS_SQL already uses
DISTINCT ON latest snapshot).
Perf bonus: objects-first avoids the parallel seq scan of the ~376k-row flats
snapshot. radius 1.5km / snapshot 2026-05-17: 240ms/~28k buffers/6712 disk
reads -> 49ms/1554 buffers/0 disk reads (~5x).
Tests: add SQL-text fan-out guard (DISTINCT ON + MATERIALIZED, no bare
flats->objects join); update stale EXPLAIN mirror in test_phantom_columns.
USER-FACING: best-layouts supply/sold_pct/is_oversold/sell-out-months shift
~8.5x toward correct (frontend BestLayoutsBlock only; ТЗ recommendation + PDF
unchanged — they derive from sum_deals, not supply). Deep-reviewed (APPROVE).
The objective_corpus_room_month index migration (#1942) was committed to
backend/data/sql/171_*, but the main deploy runner applies migrations from
repo-root data/sql/*.sql (deploy.yml:280) and the path trigger is data/sql/**.
So the file never ran — prod still seq-scans (verified post-deploy: index
absent, query 265ms). Move it to data/sql/171_* (alongside 170) so it deploys.
No SQL change; the index itself was dry-run-verified on prod (BEGIN/ROLLBACK).
_get_ekb_median() (velocity.py:706) runs on EVERY POST /parcels/{cad}/analyze
(the hottest endpoint) and seq-scanned the whole objective_corpus_room_month
(~95MB, ~12159 buffers, 144ms) — its predicates (report_month >= now-6mo AND
deals_total_count > 0) had no usable index (the 5 existing report_month indexes
aren't partial on deals_total_count; a bare range matches 27% of rows, so the
planner correctly chose Seq Scan).
Add partial b-tree (report_month) WHERE deals_total_count > 0 (~280kB, 8.9%
selectivity). Prod EXPLAIN (BEGIN/ROLLBACK): 144ms→38ms (~3.8x), buffers
12281→3136 (-74%); planner uses it naturally (Index/Bitmap scan). Independently
dry-run-verified: Index Only Scan, 2747 buffers.
Write cost negligible (objective_corpus_room_month written only by weekly ETL,
not request-path). Idempotent (IF NOT EXISTS); plain CREATE INDEX (not
CONCURRENTLY, can't run in the migration's BEGIN/COMMIT) — sub-second build,
SHARE lock blocks only the weekly ETL writer, not analyze readers.
Found via pg_stat_user_tables seq-scan audit + database-expert EXPLAIN analysis.
App-level logs (logging.getLogger("app.*")) had no StreamHandler, so their
INFO was lost — `docker logs gendesign-backend-1` showed only uvicorn loggers.
Useful lines (e.g. "OSRM road-distance applied", RBAC, analyze) were invisible
when debugging on the VPS (only Sentry/GlitchTip received them as breadcrumbs).
main.py now attaches one StreamHandler to the root "app" logger at
APP_LOG_LEVEL (default INFO), idempotent (marker guard), propagate=True so the
Sentry LoggingIntegration on root still gets breadcrumbs/events and there's no
stdout double (root has no stdout handler). API process only — celery worker
(celery_app.py) untouched, since its loaders log per-sync and would be noisy.
Flood-safe on the API: the analyze hot path logs INFO per-request (parcels.py
has 3 logger.info), not per-row.
Adds tests/test_app_logging.py (handler present, level INFO, propagate intact).
Refs #1926
Found by adversarial valuation audit (2 confirmed, bot-safe).
FIX A (#5): both radius comp queries (Tier H ~3990, Tier W ~4135) ended with
a bare ORDER BY relevance_score; on score ties Postgres returned rows in
undefined order, so the same /analyze could pick different comps across runs.
Append deterministic tiebreaker: relevance_score ASC, distance_m ASC,
scraped_at DESC NULLS LAST, id ASC (id = listings PK → total order). Added id
to each base CTE; outer projection unchanged (no leak downstream).
FIX B (#2): _filter_outliers keeps rows with price_per_m2 IS NULL, but the
median is built from prices_ppm2 (drops them) while n_analogs counted all of
listings_clean — overstating contributing comps ("Найдено N аналогов"
misleading; all-price-less -> median=0 but n_analogs>0). Count n_analogs from
prices_ppm2 in the radius path. #698 anchor overwrite + #691 zero-analog->low
guard unaffected; listings_clean itself not filtered.
Adds tests/test_estimator_n_analogs_priced.py (verified to fail on old code).
Audit also flagged velocity fan-out (false-positive: 0 duplicate domrf_obj_id
on prod) and >/>= disclosure tweaks (cosmetic) — deliberately not changed.
Refs #1871
First XHR after goto(origin) intermittently hit "NetworkError when
attempting to fetch resource" (page net stack/anti-bot not ready),
healed only by the outer re-navigation retry (~30-45s/house in the
house_imv_backfill, #562).
_fetch_json_once now:
- settles FETCH_JSON_SETTLE_MS (default 1200, was hardcoded 500) — fewer
first-fails;
- wraps the in-page fetch() in a JS retry loop (FETCH_JSON_INPAGE_RETRIES
default 1, FETCH_JSON_RETRY_DELAY_MS default 800) that retries ONLY on
network throw, never on HTTP status (4xx/5xx short-circuit, caller
decides). An in-page retry costs ~retryDelayMs vs the ~30-45s outer
re-navigation. Last error re-thrown — outer crash-retry contract intact.
/fetch (SERP) path untouched. +2 tests (settle ms, retry params).
Refs #1917, #562
#1928 fixed scrape_runs.total_seen/new_count for sweeps via _column_counts
but only regression-tested the yandex mark_done path. Add equivalents for
the CitySweepCounters (avito/cian) sweep and the mark_failed path — both
explicitly named in the #1926 audit — so the all-sweeps + failure-path
guarantee is locked in.
Refs #1926
Migration 130 only filled house_id_fk for listings carrying
house_source/house_ext_id. Prod (2026-06-27): 1884 active NULL-FK rows
remain, 0 with house_source/ext_id (130 re-run = no-op). They were
matched via the per-listing-identity path the realtime mirror uses when
no house catalog id exists: match_or_create_house(ext_source=source,
ext_id=source_id) -> Tier-1 house_sources lookup (confidence 1.0).
136 replays that exact lookup offline: UPDATE listings SET house_id_fk
= house_sources.house_id WHERE (ext_source=source, ext_id=source_id) and
house_id_fk IS NULL. Faithful to base.py:704-750; idempotent (NULL
guard); deterministic (no fuzzy/geo); collision-safe (catalog ids live
under distinct ext_source like 'cian_newbuilding'). Dry-run BEGIN/ROLLBACK
fills 4608 NULL-FK rows (744 active), restoring estimator Tier-S
"same-building" grouping. The remaining ~1140 active need re-scrape/
re-match (scraper coverage — #1781 Лёха).
Refs #1781
Walk-relevant POIs (school/shop/park/kindergarten/pharmacy/stops) now route
via a FOOT OSRM graph (osrm-walk service), car-relevant POIs (mall/hospital +
unknown) keep the DRIVING graph. Validation showed driving overstated
pedestrian-proximity distance — median 1.6-2.9x straight-line (#39).
- config: osrm_walk_local_url + osrm_walk_categories (frozenset, 9 walk cats)
- osrm_client_local: base_url override on get_road_distances_m (default unchanged)
- _apply_osrm_road_distances: split POIs by category, per-group OSRM call with
independent graceful fallback (one server down -> its group keeps straight-line),
in-place write-back by original index; never raises; flag-OFF byte-identical
- docker-compose: osrm-walk service (foot graph, internal, mem_limit 1.5g)
- build_osrm.sh: CAR=0 gate for foot-only refresh (doesn't touch live car graph)
- tests: per-category split, per-group fallback, asymmetric intra-group write-back
Still flag-gated (use_osrm_distances OFF) — enabling is a product decision.
Refs #39
Добавляет тест test_n_analogs_reflects_deduped_count: 3 дубля одного объекта
(один source_id) → 1 кандидат (freshest scraped_at); 3 разных объекта → все 3.
Верифицирует ключевое требование #1871 P2: n_analogs отражает дедупнутое число,
а не raw кол-во строк в listings с дублями (prod: 797 yandex + 93 cian дублей
раздували n_analogs и искажали медиану/перцентили).
Добавляет settings-флаг estimate_confidence_floor_no_analogs (дефолт True),
гейтящий вызов _enforce_zero_analog_low перед сборкой AggregatedEstimate.
При n_analogs==0 и confidence!='low' форсит 'low' + добавляет caveat в
explanation («без сопоставимых аналогов рядом»), предотвращая показ выдуманного
«высокого» доверия когда оценка построена только на внешних оценщиках
(yandex_valuation/cian_valuation) без реальных рыночных аналогов.
Migration 133 failed on prod (offer_price_history UNIQUE (listing_id,change_time) + multi-loser collisions). Drop data-child re-points, rely on FK ON DELETE CASCADE; keep only merged_into. Dry-run on prod: 9840 groups, 9889 losers deleted, clean. Refs #1773
