lekss361/gendesign

Fork 0

fix(tradein): yandex combos watchdog timeout (P0) + deploy scraper filter gap #1659

Merged

bot-reviewer merged 1 commit from fix/yandex-combos-watchdog-and-deploy-filter into main

2026-06-16 20:22:28 +00:00

lekss361 commented

2026-06-16 20:19:19 +00:00

Owner

Summary

Fixes a P0 regression introduced by #1658 + the deploy gap that hid it.

A — combos watchdog timeout (P0)

#1658 made run_yandex_city_sweep do 30 rooms×price combos in a single "center anchor", but the per-anchor watchdog asyncio.wait_for(..., timeout=ANCHOR_TIMEOUT_SEC=240) was sized for ONE quick geographic anchor. 30 combos × 3 pages × ~9s delay ≈ 18-30 min ≫ 240s → timed out mid-sweep, partial results discarded → lots_fetched=0 (prod-confirmed, worse than the old ~23-50).

Fix: compute the watchdog timeout from the combos workload —
max(240, num_combos × max_pages × (request_delay + 12) + 300) ≈ 2190s (~37min) for the default 30-combo config (fits the 02:00-05:00 window). Explicit-anchor mode (tests) keeps 240s/anchor.

B — deploy scraper filter gap

deploy-tradein.yml's scraper path filter covered scrapers/**, scheduler.py, tasks/** but not scrape_pipeline.py — even though the scheduler (tradein-scraper container) runs run_*_city_sweep from it. So #1658's deploy didn't recreate the scraper → scheduler ran stale code until manual recreate. Added scrape_pipeline.py to the filter. Self-applying: this PR touches both scrape_pipeline.py and the filter, so its own deploy recreates the scraper.

Test plan

Full gate: 1844 passed, 2 deselected (added: combos-timeout-is-large + explicit-anchor-stays-240 tests)
ruff clean
Post-deploy smoke: scraper recreates; yandex_city_sweep run → lots_fetched ≫ 0 (the regression proof)

Refs #1658, #759

## Summary Fixes a P0 regression introduced by #1658 + the deploy gap that hid it. ### A — combos watchdog timeout (P0) #1658 made `run_yandex_city_sweep` do 30 rooms×price combos in a single "center anchor", but the per-anchor watchdog `asyncio.wait_for(..., timeout=ANCHOR_TIMEOUT_SEC=240)` was sized for ONE quick geographic anchor. 30 combos × 3 pages × ~9s delay ≈ 18-30 min ≫ 240s → timed out mid-sweep, partial results discarded → **lots_fetched=0** (prod-confirmed, worse than the old ~23-50). Fix: compute the watchdog timeout from the combos workload — `max(240, num_combos × max_pages × (request_delay + 12) + 300)` ≈ **2190s (~37min)** for the default 30-combo config (fits the 02:00-05:00 window). Explicit-anchor mode (tests) keeps 240s/anchor. ### B — deploy scraper filter gap `deploy-tradein.yml`'s `scraper` path filter covered `scrapers/**`, `scheduler.py`, `tasks/**` but **not `scrape_pipeline.py`** — even though the scheduler (tradein-scraper container) runs `run_*_city_sweep` from it. So #1658's deploy didn't recreate the scraper → scheduler ran stale code until manual recreate. Added `scrape_pipeline.py` to the filter. **Self-applying**: this PR touches both `scrape_pipeline.py` and the filter, so its own deploy recreates the scraper. ## Test plan - [x] Full gate: **1844 passed, 2 deselected** (added: combos-timeout-is-large + explicit-anchor-stays-240 tests) - [x] ruff clean - [ ] Post-deploy smoke: scraper recreates; yandex_city_sweep run → lots_fetched ≫ 0 (the regression proof) Refs #1658, #759

lekss361 added 1 commit 2026-06-16 20:19:19 +00:00

fix(tradein): yandex combos watchdog timeout P0 + deploy filter gap

CI / frontend-tests (push) Has been skipped

Details

CI / changes (push) Successful in 7s

Details

CI / backend-tests (push) Has been skipped

Details

CI / openapi-codegen-check (push) Has been skipped

Details

CI / changes (pull_request) Successful in 6s

Details

CI / backend-tests (pull_request) Has been skipped

Details

CI / frontend-tests (pull_request) Has been skipped

Details

CI / openapi-codegen-check (pull_request) Has been skipped

Details

ed4662e37b

Fix A: scale asyncio.wait_for timeout in run_yandex_city_sweep for
combos/center mode. Old 240s budget was sized for one geo anchor;
with 30 combos × 3 pages × ~21s each = ~1890s+ of work, sweep
timed out at 240s producing lots_fetched=0. Now computes:
  sweep_timeout = max(240, num_combos * max_pages * (delay + 12) + 300)
For default 30×3×(9+12)+300 ≈ 2190s. Explicit-anchor mode keeps 240s.

Fix B: add scrape_pipeline.py to the `scraper` path filter in
deploy-tradein.yml so changes to sweep logic trigger scraper container
recreate. Previously only scrapers/** and scheduler.py were covered,
meaning #1658 deploy didn't recreate the scraper container.

Tests: 2 new unit tests for timeout computation (pure arithmetic, no
sleep) + explicit-anchor mode regression (wait_for called with 240s).
Full suite: 1844 passed, 2 deselected.

lekss361 added the

labels 2026-06-16 20:19:26 +00:00

bot-reviewer approved these changes 2026-06-16 20:22:21 +00:00

bot-reviewer left a comment

Collaborator

Deep review APPROVE (sha ed4662e3).

Gate: ruff clean on changed files (2 pre-existing B011 in test_cadastral_reverse.py are on main, not touched here); pytest 1844 passed / 2 deselected / 0 fail; both new watchdog tests pass.

Correctness: combos-aware watchdog budget (~1560s prod 2-page, ~2190s 3-page) replaces the fixed 240s that killed the #1658 sweep at lots_fetched=0. Explicit-anchor/legacy path keeps ANCHOR_TIMEOUT_SEC=240. Inner per-request curl subprocess is independently bounded (wait_for timeout+10) so no infinite hang; budget fits the 02:00-05:00 window with wide margin.

Deploy: scrape_pipeline.py added to the scraper change-filter -> SCRAPER_CHANGED=true -> up -d --no-deps scraper, so the tradein-scraper container gets the new sweep code this deploy (fixes the stale-code gotcha). Self-applying for this PR.

Minor (non-blocking): timeout budget uses the param request_delay_sec (default 9.0) while the scraper's actual per-page sleep is get_scraper_delay('yandex') from DB; if an operator raises the DB delay well above 9s the budget could under-estimate. Strictly better than the prior hardcoded 240s.

Deep review APPROVE (sha ed4662e3). Gate: ruff clean on changed files (2 pre-existing B011 in test_cadastral_reverse.py are on main, not touched here); pytest 1844 passed / 2 deselected / 0 fail; both new watchdog tests pass. Correctness: combos-aware watchdog budget (~1560s prod 2-page, ~2190s 3-page) replaces the fixed 240s that killed the #1658 sweep at lots_fetched=0. Explicit-anchor/legacy path keeps ANCHOR_TIMEOUT_SEC=240. Inner per-request curl subprocess is independently bounded (wait_for timeout+10) so no infinite hang; budget fits the 02:00-05:00 window with wide margin. Deploy: scrape_pipeline.py added to the scraper change-filter -> SCRAPER_CHANGED=true -> `up -d --no-deps scraper`, so the tradein-scraper container gets the new sweep code this deploy (fixes the stale-code gotcha). Self-applying for this PR. Minor (non-blocking): timeout budget uses the param request_delay_sec (default 9.0) while the scraper's actual per-page sleep is get_scraper_delay('yandex') from DB; if an operator raises the DB delay well above 9s the budget could under-estimate. Strictly better than the prior hardcoded 240s.