feat(tradein-scripts): scrapers + probes + Phase C improvements #551

Merged

lekss361 merged 1 commit from feat/tradein-phase-c-script-improvements into main

2026-05-24 19:58:39 +00:00

lekss361 commented

2026-05-24 19:50:03 +00:00

Owner

Что добавлено

Коммитим накопившиеся за день scrapers/probes — оперативные tools для локальной диагностики и residential-IP sweeps (когда VPS Beget IP заблокирован Avito/Yandex).

Sweep scripts (residential IP)

local-sweep-ekb.py — full ЕКБ Avito sweep через localhost
local-sweep-ekb-cian.py — Cian ЕКБ vtorichka sweep
local-sweep-ekb-yandex.py — Yandex Недвижимость ЕКБ sweep
avito-playwright-sweep.py — Playwright-based Avito sweep, обходит Qrator JS challenge
backfill-yandex-addresses-ekb.py — backfill listings.address (street + house number) для active EKB yandex с short addresses

Probes (diagnostics)

probe-avito-1anchor.py — 1 anchor × 1 page minimal dry-run (тест на bans)
probe-db-counts.py — Avito listings count + recent insert rate
probe-db-leak-full.py — региональная утечка в Avito listings (3 угла: URL slug, address, geo)
probe-db-region-leak.py / probe-db-url-leak.py — фокусированные leak checks
probe-migration-028.py / probe-migration-status.py — состояние SQL migrations 028..054
probe-parser-region-filter.py — verify URL-filter работает в parser
inspect-houses-html.py / inspect-search-html.py — Avito DOM analysis (после смены rendering)
test-playwright.py — Playwright smoke test

Phase C script (`backfill-house-imv.py`) updates

avito-raw.log — pipe app.services.scrapers.avito_imv logs в файл (env AVITO_LOG_PATH override). Сохраняет полную историю Avito API чисел для диагностики контракт-changes.
rooms=0 → 1 clamp — Avito IMV API rejects studio (rooms=0) with HTTP 400. Маппим к 1-room.

Context

Эти tools использовались параллельно с roadmap из [[Decision_TradeIn_DataQuality_8PR_Roadmap]] + Phase C house-IMV backfill — большинство из них рождались как ad-hoc helpers, но многие пригодятся для будущей диагностики.

Notes

*.cookies.json не комитятся (gitignored)
*.bak frontend файлы не комитятся (artifacts от ручного редактирования)

## Что добавлено Коммитим накопившиеся за день scrapers/probes — оперативные tools для локальной диагностики и residential-IP sweeps (когда VPS Beget IP заблокирован Avito/Yandex). ### Sweep scripts (residential IP) - `local-sweep-ekb.py` — full ЕКБ Avito sweep через localhost - `local-sweep-ekb-cian.py` — Cian ЕКБ vtorichka sweep - `local-sweep-ekb-yandex.py` — Yandex Недвижимость ЕКБ sweep - `avito-playwright-sweep.py` — Playwright-based Avito sweep, обходит Qrator JS challenge - `backfill-yandex-addresses-ekb.py` — backfill `listings.address` (street + house number) для active EKB yandex с short addresses ### Probes (diagnostics) - `probe-avito-1anchor.py` — 1 anchor × 1 page minimal dry-run (тест на bans) - `probe-db-counts.py` — Avito listings count + recent insert rate - `probe-db-leak-full.py` — региональная утечка в Avito listings (3 угла: URL slug, address, geo) - `probe-db-region-leak.py` / `probe-db-url-leak.py` — фокусированные leak checks - `probe-migration-028.py` / `probe-migration-status.py` — состояние SQL migrations 028..054 - `probe-parser-region-filter.py` — verify URL-filter работает в parser - `inspect-houses-html.py` / `inspect-search-html.py` — Avito DOM analysis (после смены rendering) - `test-playwright.py` — Playwright smoke test ### Phase C script (`backfill-house-imv.py`) updates - **avito-raw.log** — pipe `app.services.scrapers.avito_imv` logs в файл (env `AVITO_LOG_PATH` override). Сохраняет полную историю Avito API чисел для диагностики контракт-changes. - **rooms=0 → 1 clamp** — Avito IMV API rejects studio (rooms=0) with HTTP 400. Маппим к 1-room. ## Context Эти tools использовались параллельно с roadmap из `[[Decision_TradeIn_DataQuality_8PR_Roadmap]]` + Phase C house-IMV backfill — большинство из них рождались как ad-hoc helpers, но многие пригодятся для будущей диагностики. ## Notes - *.cookies.json не комитятся (gitignored) - *.bak frontend файлы не комитятся (artifacts от ручного редактирования)

lekss361 added 1 commit 2026-05-24 19:50:04 +00:00

feat(tradein-scripts): scrapers, probes, Phase C improvements 70e45ce5e9

New scripts:
- avito-playwright-sweep.py — Playwright-based Avito sweep (bypasses Qrator)
- backfill-yandex-addresses-ekb.py — fill street-only listings.address с house number
- local-sweep-ekb-{cian,yandex,}.py — full ЕКБ sweep through residential IP
- probe-* — quick diagnostics (region leak, migration status, parser filter, db counts)
- inspect-{houses,search}-html.py — Avito DOM structure analysis
- test-playwright.py — minimal Playwright smoke

Phase C script updates (backfill-house-imv.py):
- Pipe avito_imv module logs to avito-raw.log file (env AVITO_LOG_PATH override)
- Clamp rooms=0 (studio) → 1 — Avito IMV API rejects rooms=0 with HTTP 400

lekss361 commented

2026-05-24 19:58:30 +00:00

Author

Owner

Deep Code Review — PR #551 verdict: APPROVE

Summary

Status: APPROVE
Files: 17 (P2 ops scripts × 17; 1 modified, 16 new)
Lines: +2496/-1 · SHA: 70e45ce5e9
Scope: ad-hoc tradein operator scripts (sweeps + probes + Phase C log/clamp improvements)

Security (clean)

No real credentials in diff — all DSN strings use <PASS> placeholder in docstrings
DB DSN sourced from os.environ["DATABASE_URL"] everywhere (psycopg adapter swap for plain postgresql://)
avito-playwright-sweep.py:main masks password before logging (lines ~317-330)
.cian-cookies.json gitignored; no cookie payloads committed

Correctness / idempotency

All probe-db-*.py are read-only — only SELECT, zero UPDATE/INSERT/DELETE
backfill-yandex-addresses-ekb.py has proper --dry-run + --limit + skip when address unchanged
probe-migration-status.py:78 calls conn.rollback() on per-check error — one failing check cannot poison subsequent ones
Sweep scripts use canonical save_listings() (per-page commit) — crash mid-sweep doesn't lose prior pages
backfill-house-imv.py studio clamp max(rooms_raw, 1) is safe: SQL filter has WHERE rooms IS NOT NULL + early-return at row["rooms"] is None (line 154); no NoneType.int risk

Error handling per script (isolation)

Each script self-contained: own argparse, own SessionLocal(), own try/finally for DB sessions
except Exception as e: # noqa: BLE001 + continue pattern across all sweeps — one failing page/anchor never aborts whole run
Captcha-detected raises RuntimeError("captcha") caught in phase_combos, breaks combo loop cleanly
One failing script cannot affect siblings (separate Python processes per CLI invocation)

Phase C `backfill-house-imv.py` review

New file-handler attaching to app.services.scrapers.avito_imv logger: idempotent because CLI is one-shot. Caveat: if someone imports this script as a module, handler gets re-attached → multi-write to avito-raw.log. Not exercised in production path.
AVITO_LOG_PATH env override = good ergonomics
Studio→1 clamp documented inline with cause (HTTP 400 from Avito IMV)

Pattern consistency with prior sweep PRs (#533/#534/#535/#539)

Same _BACKEND_DIR / sys.path.insert(0, ...) + weasyprint stub idiom as merged backfill scripts
Same from app.services.scrapers.* import ... pattern (post-stub import)
Same argparse + asyncio.run(main()) shape
Same residential-IP delay defaults (15s sweep / 30s anchor / 3s probe)

Low-severity findings (non-blocking, do not require fix here)

avito-playwright-sweep.py:60-62 injects _anchor_lat / _anchor_lon / _anchor_radius_m on the AvitoScraper instance, but AvitoScraper._parse_html does not consume those fields (only n1.py reads them). Dead code, no harm — likely legacy from earlier iteration. Safe to drop in a follow-up cleanup.
inspect-houses-html.py:55 and inspect-search-html.py:31 write to hard-coded /tmp/... paths — Linux/WSL only; will fail on Windows. These are local-only inspect scripts, mixed-OS dev tradeoff is fine.
.gitignore covers .cian-cookies.json and *.cookies.json literal, but .avito-cookies.json and .yandex-cookies.json (filenames already referenced in this PR) match neither pattern. Followup: extend gitignore to tradein-mvp/scripts/.*-cookies.json so anyone running a git add . after generating local cookies can't accidentally commit them.
inspect-houses-html.py does inline import re mid-function (line ~83). Cosmetic.

Cross-file impact (blast radius: zero on prod)

Scripts live only in tradein-mvp/scripts/, not imported by backend app, not referenced by any cron/Celery/CI workflow
No DB schema changes, no API changes, no migrations
Only changed runtime artifact: backfill-house-imv.py (operator-run CLI), with strictly additive behavior

Positive

Excellent inline rationale (why-local, why-clamp, why-fallback parser) — makes scripts maintainable for future operators
probe-migration-status.py is a clean diagnostic primitive worth keeping as deploy reference
local-sweep-ekb.py exponential backoff on 429 with retry-same-page (lines ~263-281) is a meaningful improvement over continue-only patterns

Recommended next steps (optional, separate PR)

Extend .gitignore with tradein-mvp/scripts/.*-cookies.json glob
Drop the dead _anchor_* injection in avito-playwright-sweep.py
Vault: add operator script catalog under tradein/runbooks/ referencing each script's purpose and usage flags

Verdict

Approve for merge. Ad-hoc operator scripts, zero blast radius on production. Security clean, idempotency correct, error handling isolated.

## Deep Code Review — PR #551 verdict: APPROVE ### Summary - Status: APPROVE - Files: 17 (P2 ops scripts × 17; 1 modified, 16 new) - Lines: +2496/-1 · SHA: 70e45ce5e9d7919154b2acd796084209d0f9b00c - Scope: ad-hoc tradein operator scripts (sweeps + probes + Phase C log/clamp improvements) ### Security (clean) - No real credentials in diff — all DSN strings use `<PASS>` placeholder in docstrings - DB DSN sourced from `os.environ["DATABASE_URL"]` everywhere (psycopg adapter swap for plain `postgresql://`) - `avito-playwright-sweep.py:main` masks password before logging (lines ~317-330) - `.cian-cookies.json` gitignored; no cookie payloads committed ### Correctness / idempotency - All `probe-db-*.py` are read-only — only `SELECT`, zero `UPDATE/INSERT/DELETE` - `backfill-yandex-addresses-ekb.py` has proper `--dry-run` + `--limit` + skip when address unchanged - `probe-migration-status.py:78` calls `conn.rollback()` on per-check error — one failing check cannot poison subsequent ones - Sweep scripts use canonical `save_listings()` (per-page commit) — crash mid-sweep doesn't lose prior pages - `backfill-house-imv.py` studio clamp `max(rooms_raw, 1)` is safe: SQL filter has `WHERE rooms IS NOT NULL` + early-return at `row["rooms"] is None` (line 154); no NoneType.int risk ### Error handling per script (isolation) - Each script self-contained: own argparse, own `SessionLocal()`, own try/finally for DB sessions - `except Exception as e: # noqa: BLE001` + `continue` pattern across all sweeps — one failing page/anchor never aborts whole run - Captcha-detected raises `RuntimeError("captcha")` caught in `phase_combos`, breaks combo loop cleanly - One failing script cannot affect siblings (separate Python processes per CLI invocation) ### Phase C `backfill-house-imv.py` review - New file-handler attaching to `app.services.scrapers.avito_imv` logger: idempotent because CLI is one-shot. **Caveat**: if someone `import`s this script as a module, handler gets re-attached → multi-write to avito-raw.log. Not exercised in production path. - `AVITO_LOG_PATH` env override = good ergonomics - Studio→1 clamp documented inline with cause (HTTP 400 from Avito IMV) ### Pattern consistency with prior sweep PRs (#533/#534/#535/#539) - Same `_BACKEND_DIR / sys.path.insert(0, ...)` + weasyprint stub idiom as merged backfill scripts - Same `from app.services.scrapers.* import ...` pattern (post-stub import) - Same `argparse + asyncio.run(main())` shape - Same residential-IP delay defaults (15s sweep / 30s anchor / 3s probe) ### Low-severity findings (non-blocking, do not require fix here) - `avito-playwright-sweep.py:60-62` injects `_anchor_lat / _anchor_lon / _anchor_radius_m` on the `AvitoScraper` instance, but `AvitoScraper._parse_html` does not consume those fields (only `n1.py` reads them). Dead code, no harm — likely legacy from earlier iteration. Safe to drop in a follow-up cleanup. - `inspect-houses-html.py:55` and `inspect-search-html.py:31` write to hard-coded `/tmp/...` paths — Linux/WSL only; will fail on Windows. These are local-only inspect scripts, mixed-OS dev tradeoff is fine. - `.gitignore` covers `.cian-cookies.json` and `*.cookies.json` literal, but `.avito-cookies.json` and `.yandex-cookies.json` (filenames already referenced in this PR) match neither pattern. Followup: extend gitignore to `tradein-mvp/scripts/.*-cookies.json` so anyone running a `git add .` after generating local cookies can't accidentally commit them. - `inspect-houses-html.py` does inline `import re` mid-function (line ~83). Cosmetic. ### Cross-file impact (blast radius: zero on prod) - Scripts live only in `tradein-mvp/scripts/`, not imported by backend app, not referenced by any cron/Celery/CI workflow - No DB schema changes, no API changes, no migrations - Only changed runtime artifact: `backfill-house-imv.py` (operator-run CLI), with strictly additive behavior ### Positive - Excellent inline rationale (why-local, why-clamp, why-fallback parser) — makes scripts maintainable for future operators - `probe-migration-status.py` is a clean diagnostic primitive worth keeping as deploy reference - `local-sweep-ekb.py` exponential backoff on 429 with retry-same-page (lines ~263-281) is a meaningful improvement over `continue`-only patterns ### Recommended next steps (optional, separate PR) 1. Extend `.gitignore` with `tradein-mvp/scripts/.*-cookies.json` glob 2. Drop the dead `_anchor_*` injection in `avito-playwright-sweep.py` 3. Vault: add operator script catalog under `tradein/runbooks/` referencing each script's purpose and usage flags ### Verdict Approve for merge. Ad-hoc operator scripts, zero blast radius on production. Security clean, idempotency correct, error handling isolated.