lekss361/gendesign

Fork 0

feat(tradein): yandex city sweep — EKB-center rooms×price combos (replaces 5-anchor shallow) #1658

Merged

lekss361 merged 1 commit from feat/yandex-center-rooms-price-sweep into main

2026-06-16 19:57:38 +00:00

lekss361 commented

2026-06-16 19:43:35 +00:00

Owner

Summary

Replaces the narrow nightly yandex sweep (5 geographic anchors × 2 pages, ~50 lots/run) with an EKB-center rooms×price exhaustive sweep.

Problem

yandex_city_sweep iterated EKB_ANCHORS (5 points, 1.5km radius each) × pages_per_anchor=2 — geographically narrow + shallow → only ~48 of 3969 active yandex listings refreshed within 14 days.

Fix

run_yandex_city_sweep now calls the existing YandexRealtyScraper.fetch_around_multi_room in combos mode on the EKB center (56.8400, 60.6050): all 5 room types × DEFAULT_PRICE_RANGES (6 buckets) = 30 combos, which bypasses the SERP ~575-card cap via price/room segmentation. Covers the whole city from one center instead of 5 small circles.

Center anchor (56.8400, 60.6050); radius_m default raised 1500 → 25000 (city-scale; Yandex SERP is path/city-scoped so geo-bbox is secondary).
EKB_ANCHORS unchanged — avito/cian sweeps still use the 5-anchor path. Only yandex changed.
CitySweepCounters fields unchanged (anchors_total=1 for the center run). trigger_yandex_city_sweep_run unchanged (pages_per_anchor → max_pages).
Migration 119_yandex_city_sweep_center_combos.sql: updates yandex schedule default_params (radius_m=25000, pages_per_anchor=3); rooms/price use code defaults (ROOM_PATH.keys() + DEFAULT_PRICE_RANGES).

Test plan

Full gate: 1836 passed, 2 deselected (added test_prod_mode_uses_center_anchor_with_combos + back-compat test for explicit anchors)
ruff clean
Post-deploy smoke: trigger via next_run=now → expect lots_fetched ≫ ~50 (30 combos)

Note: new code can't be smoked on prod until deployed — verified statically + via unit tests.

Refs #759

## Summary Replaces the narrow nightly yandex sweep (5 geographic anchors × 2 pages, ~50 lots/run) with an **EKB-center rooms×price exhaustive** sweep. ### Problem `yandex_city_sweep` iterated `EKB_ANCHORS` (5 points, 1.5km radius each) × `pages_per_anchor=2` — geographically narrow + shallow → only ~48 of 3969 active yandex listings refreshed within 14 days. ### Fix `run_yandex_city_sweep` now calls the existing `YandexRealtyScraper.fetch_around_multi_room` in **combos mode** on the **EKB center (56.8400, 60.6050)**: all 5 room types × `DEFAULT_PRICE_RANGES` (6 buckets) = 30 combos, which bypasses the SERP ~575-card cap via price/room segmentation. Covers the whole city from one center instead of 5 small circles. - Center anchor `(56.8400, 60.6050)`; `radius_m` default raised `1500 → 25000` (city-scale; Yandex SERP is path/city-scoped so geo-bbox is secondary). - `EKB_ANCHORS` unchanged — avito/cian sweeps still use the 5-anchor path. Only yandex changed. - `CitySweepCounters` fields unchanged (`anchors_total=1` for the center run). `trigger_yandex_city_sweep_run` unchanged (`pages_per_anchor` → `max_pages`). - Migration `119_yandex_city_sweep_center_combos.sql`: updates yandex schedule `default_params` (`radius_m=25000, pages_per_anchor=3`); rooms/price use code defaults (`ROOM_PATH.keys()` + `DEFAULT_PRICE_RANGES`). ## Test plan - [x] Full gate: **1836 passed, 2 deselected** (added `test_prod_mode_uses_center_anchor_with_combos` + back-compat test for explicit anchors) - [x] ruff clean - [ ] Post-deploy smoke: trigger via `next_run=now` → expect lots_fetched ≫ ~50 (30 combos) Note: new code can't be smoked on prod until deployed — verified statically + via unit tests. Refs #759

lekss361 added 1 commit 2026-06-16 19:43:35 +00:00

feat(tradein): yandex city sweep — center combos replaces 5-anchor shallow

CI / changes (push) Successful in 8s

Details

CI / openapi-codegen-check (push) Has been skipped

Details

CI / changes (pull_request) Successful in 6s

Details

CI / backend-tests (pull_request) Has been skipped

Details

CI / backend-tests (push) Has been skipped

Details

CI / frontend-tests (push) Has been skipped

Details

CI / frontend-tests (pull_request) Has been skipped

Details

CI / openapi-codegen-check (pull_request) Has been skipped

Details

e551eda289

Switch run_yandex_city_sweep from iterating 5 geographic anchors x 2 shallow
pages to a single EKB center anchor (56.8400, 60.6050) with exhaustive
rooms x price-range combos (5 room types x 6 price buckets = 30 combos),
radius_m=25000 to cover the whole city.

- scrape_pipeline.py: new params rooms_list/price_ranges; anchors=None (prod)
  always uses EKB center; fetch_around_multi_room called in combos mode
- 119_yandex_city_sweep_center_combos.sql: updates default_params
  (radius_m: 1500 -> 25000, pages_per_anchor: 2 -> 3)
- tests: 2 new tests assert center coords + combos wiring (prod mode and
  explicit-anchors backward-compat); 1836 suite green

lekss361 added the

scope/backend

scrapers

labels 2026-06-16 19:43:47 +00:00

lekss361 commented

2026-06-16 19:57:36 +00:00

Author

Owner

Deep review verdict: APPROVE.

EKB_ANCHORS untouched — avito + cian sweeps still iterate the 5 anchors; only run_yandex_city_sweep changed.
Combos wiring correct: center (56.8400, 60.6050), rooms_list -> all 5 ROOM_PATH keys, price_ranges -> 6 DEFAULT_PRICE_RANGES = 30 combos; pages_per_anchor -> max_pages.
radius_m=25000 safe: YandexRealtyScraper.fetch_around explicitly ignores lat/lon/radius_m (path-based vtorichka URL), so 25km does not break the URL builder.
Back-compat: anchors param preserved for tests; scheduler trigger_yandex_city_sweep_run + admin caller remain signature-compatible (new kwargs default to None -> full sets).
Counters/scaffolding preserved: anchors_total=1, abort/heartbeat/mark_done/address-enrich intact.
Migration 119 free (main at 118), idempotent UPDATE on yandex_city_sweep schedule.
Regression test test_prod_mode_uses_center_anchor_with_combos asserts prod center path + combos sets.
Gate: ruff clean on PR files (2 pre-existing B011 errors in tests/services/test_cadastral_reverse.py are unrelated to this PR); full pytest 1836 passed, 2 deselected.

Merging squash.

Deep review verdict: APPROVE. - EKB_ANCHORS untouched — avito + cian sweeps still iterate the 5 anchors; only run_yandex_city_sweep changed. - Combos wiring correct: center (56.8400, 60.6050), rooms_list -> all 5 ROOM_PATH keys, price_ranges -> 6 DEFAULT_PRICE_RANGES = 30 combos; pages_per_anchor -> max_pages. - radius_m=25000 safe: YandexRealtyScraper.fetch_around explicitly ignores lat/lon/radius_m (path-based vtorichka URL), so 25km does not break the URL builder. - Back-compat: anchors param preserved for tests; scheduler trigger_yandex_city_sweep_run + admin caller remain signature-compatible (new kwargs default to None -> full sets). - Counters/scaffolding preserved: anchors_total=1, abort/heartbeat/mark_done/address-enrich intact. - Migration 119 free (main at 118), idempotent UPDATE on yandex_city_sweep schedule. - Regression test test_prod_mode_uses_center_anchor_with_combos asserts prod center path + combos sets. - Gate: ruff clean on PR files (2 pre-existing B011 errors in tests/services/test_cadastral_reverse.py are unrelated to this PR); full pytest 1836 passed, 2 deselected. Merging squash.

lekss361 merged commit e7901bc1e8 into main

2026-06-16 19:57:38 +00:00

lekss361 deleted branch feat/yandex-center-rooms-price-sweep

2026-06-16 19:57:38 +00:00

lekss361 referenced this pull request from a commit

2026-06-16 19:57:39 +00:00

feat(tradein): yandex city sweep — EKB-center rooms×price combos (replaces 5-anchor shallow) (#1658)

lekss361 commented

2026-06-16 20:10:48 +00:00

Author

Owner

Post-deploy smoke (run #103, prod, new combos code) — REGRESSION FOUND.

The new center-combos sweep hits the per-anchor watchdog ANCHOR_TIMEOUT_SEC=240s. In the old model each of 5 anchors had its own 240s budget; the new model wraps ALL 30 combos (5 rooms x 6 price_ranges) + address-enrich in a SINGLE 240s asyncio.wait_for (scrape_pipeline.py:957). 30 combos x ~30s each (9s inter-request delay + page fetch) cannot finish in 240s.

Result run #103: status=done, lots_fetched=0, errors_count=1, 'anchor #1/1 timed out after 240s — skipping'. The partial seen-dict is discarded on timeout (fetch_around_multi_room never returns), so 0 lots saved. Compare run #102 (old code, same trigger): lots_fetched=23.

Proof the combos logic itself works: log showed 'rooms=1 price=[None-5000000]: 20 cards' right before the watchdog killed it. Studio price buckets legitimately returned 0.

Why offline tests passed: they mock fetch_around_multi_room to return instantly, bypassing the watchdog entirely. The timing interaction is only observable on real network.

FOLLOW-UP NEEDED (new PR): either (a) raise/parametrize ANCHOR_TIMEOUT_SEC for the yandex combos path (needs ~30*30=900s+), (b) make fetch_around_multi_room return partial seen-dict on a soft deadline, or (c) wrap each combo (not the whole sweep) in its own watchdog. Prod schedule restored (window 2-5, next_run 2026-06-17 02:00); the nightly run will produce ~0 lots until this is fixed.

Deploy note: deploy-tradein selectively skips scraper recreate on backend-only changes (#1182), so the scraper ran STALE code on first trigger (run #102); I force-recreated the scraper service to get the new code live for this smoke.

Post-deploy smoke (run #103, prod, new combos code) — REGRESSION FOUND. The new center-combos sweep hits the per-anchor watchdog ANCHOR_TIMEOUT_SEC=240s. In the old model each of 5 anchors had its own 240s budget; the new model wraps ALL 30 combos (5 rooms x 6 price_ranges) + address-enrich in a SINGLE 240s asyncio.wait_for (scrape_pipeline.py:957). 30 combos x ~30s each (9s inter-request delay + page fetch) cannot finish in 240s. Result run #103: status=done, lots_fetched=0, errors_count=1, 'anchor #1/1 timed out after 240s — skipping'. The partial seen-dict is discarded on timeout (fetch_around_multi_room never returns), so 0 lots saved. Compare run #102 (old code, same trigger): lots_fetched=23. Proof the combos logic itself works: log showed 'rooms=1 price=[None-5000000]: 20 cards' right before the watchdog killed it. Studio price buckets legitimately returned 0. Why offline tests passed: they mock fetch_around_multi_room to return instantly, bypassing the watchdog entirely. The timing interaction is only observable on real network. FOLLOW-UP NEEDED (new PR): either (a) raise/parametrize ANCHOR_TIMEOUT_SEC for the yandex combos path (needs ~30*30=900s+), (b) make fetch_around_multi_room return partial seen-dict on a soft deadline, or (c) wrap each combo (not the whole sweep) in its own watchdog. Prod schedule restored (window 2-5, next_run 2026-06-17 02:00); the nightly run will produce ~0 lots until this is fixed. Deploy note: deploy-tradein selectively skips scraper recreate on backend-only changes (#1182), so the scraper ran STALE code on first trigger (run #102); I force-recreated the scraper service to get the new code live for this smoke.