lekss361/gendesign
1
0
Fork 0
Code Issues 57 Pull requests Projects Releases Packages Wiki Activity Actions

feat(tradein): city sweep — auto ЕКБ pipeline (anchors × pages + enrichment + cancel) #477

Merged
lekss361 merged 1 commit from feat/tradein-avito-city-sweep into main 2026-05-23 14:38:08 +00:00
Conversation 1 Commits 1 Files changed 7 +796 -30
lekss361 commented 2026-05-23 14:35:00 +00:00
Owner
Copy link

Summary

Auto full city sweep для AvitoScraper v2: iterate 5 anchors ЕКБ × N pages → save listings → enrich houses + detail top-N. Async background execution (FastAPI BackgroundTasks), cancel mid-flight через scrape_runs.status='cancelled', configurable delay, daily cron с random 0-3h delay для anti-pattern detection.

LOC: ~+500 (5 files modified/new + 1 migration + 1 cron + 1 test file).

Changes

Backend

  • avito.py: fetch_around(pages=N, delay_override_sec=N) — pagination 1..N (backward compat default=1)
  • scrape_pipeline.py:
    • EKB_ANCHORS const (5 точек, matches cron-scrape.sh)
    • CitySweepCounters dataclass (anchors_done/total + per-step aggregates + errors_count)
    • run_avito_city_sweep(db, *, run_id, pages_per_anchor, detail_top_n, request_delay_sec, ...):
      • cooperative cancel check (scrape_runs.status) перед каждым anchor
      • per-anchor heartbeat + counters refresh
      • per-anchor errors logged без блокировки run
  • NEW scrape_runs.py: utility module (create_run, update_heartbeat, mark_done/failed/cancelled, is_cancelled, list_recent)
  • admin.py — 3 NEW endpoints:
    • POST /admin/scrape/avito-city-sweep — start (returns run_id, BackgroundTasks)
    • GET /admin/scrape/avito-city-sweep/runs?limit=N — list recent (для UI polling)
    • POST /admin/scrape/avito-city-sweep/{run_id}/cancel — cooperative cancel
  • NEW migration 051_scrape_runs_extend.sql: 015 не имел params, counters, error, finished_at + cancelled в status enum + дефолт для run_type. Migration добавляет.
  • NEW tests/test_city_sweep.py — 16 offline tests pass

Cron

  • NEW deploy/avito-city-sweep.sh — wraps curl к admin endpoint с random delay 0-3h
  • Crontab entry (manual setup на VPS): 0 2 * * * /opt/gendesign/tradein-mvp/deploy/avito-city-sweep.sh → реальный запуск 02:00-05:00 UTC случайно каждый день

Verify

  • ✅ ruff clean
  • ✅ pytest test_city_sweep.py 16/16 pass
  • ✅ pytest test_api_avito_stage4a.py 5/5 pass (no regression)

Deviations

  1. Миграция 051 вместо 031 (031 уже занят в схеме)
  2. run_type в 015 был NOT NULL без default → 051 добавляет DEFAULT 'city_sweep' чтобы create_run не падал
  3. Pre-existing E501 в ScrapeRequest (admin.py) попутно исправлены — комментарии перенесены
  4. SessionLocal (sync) для background task — соответствует pattern всего codebase

Auth

Все 3 endpoints без require_admin (consistent с PR #474 — Caddy basic_auth достаточен).

Test plan

  • ruff + pytest
  • Deploy → migration 051 auto-apply через GHA → verify scrape_runs enum has 'cancelled' + new columns
  • POST /admin/scrape/avito-city-sweep {pages_per_anchor: 1, detail_top_n: 5} → verify run_id returned + background task progresses через /runs polling
  • During run → POST /cancel/{run_id} → verify status → cancelled в течение следующего anchor
  • Crontab setup на prod VPS (out of PR scope, manual deploy step)

Frontend contract

Frontend PR (parallel) wires /scrapers/avito UI к этим 3 endpoints с polling каждые 5s.

Refs

  • Plan: AvitoScraper_v2 Stage 4d (auto pipeline, поверх Stage 2e per-anchor wrapper)
  • scrape_runs schema base: Stage 1 migration 015
## Summary Auto **full city sweep** для AvitoScraper v2: iterate 5 anchors ЕКБ × N pages → save listings → enrich houses + detail top-N. Async background execution (FastAPI BackgroundTasks), cancel mid-flight через `scrape_runs.status='cancelled'`, configurable delay, daily cron с **random 0-3h delay** для anti-pattern detection. LOC: ~+500 (5 files modified/new + 1 migration + 1 cron + 1 test file). ## Changes ### Backend - **`avito.py`**: `fetch_around(pages=N, delay_override_sec=N)` — pagination 1..N (backward compat default=1) - **`scrape_pipeline.py`**: - `EKB_ANCHORS` const (5 точек, matches `cron-scrape.sh`) - `CitySweepCounters` dataclass (anchors_done/total + per-step aggregates + errors_count) - `run_avito_city_sweep(db, *, run_id, pages_per_anchor, detail_top_n, request_delay_sec, ...)`: - cooperative cancel check (scrape_runs.status) перед каждым anchor - per-anchor heartbeat + counters refresh - per-anchor errors logged без блокировки run - **NEW `scrape_runs.py`**: utility module (create_run, update_heartbeat, mark_done/failed/cancelled, is_cancelled, list_recent) - **`admin.py`** — 3 NEW endpoints: - `POST /admin/scrape/avito-city-sweep` — start (returns run_id, BackgroundTasks) - `GET /admin/scrape/avito-city-sweep/runs?limit=N` — list recent (для UI polling) - `POST /admin/scrape/avito-city-sweep/{run_id}/cancel` — cooperative cancel - **NEW migration `051_scrape_runs_extend.sql`**: 015 не имел `params`, `counters`, `error`, `finished_at` + `cancelled` в status enum + дефолт для `run_type`. Migration добавляет. - **NEW `tests/test_city_sweep.py`** — 16 offline tests pass ### Cron - **NEW `deploy/avito-city-sweep.sh`** — wraps `curl` к admin endpoint с random delay 0-3h - Crontab entry (manual setup на VPS): `0 2 * * * /opt/gendesign/tradein-mvp/deploy/avito-city-sweep.sh` → реальный запуск 02:00-05:00 UTC случайно каждый день ## Verify - ✅ ruff clean - ✅ pytest `test_city_sweep.py` 16/16 pass - ✅ pytest `test_api_avito_stage4a.py` 5/5 pass (no regression) ## Deviations 1. Миграция `051` вместо `031` (031 уже занят в схеме) 2. `run_type` в 015 был `NOT NULL` без default → 051 добавляет `DEFAULT 'city_sweep'` чтобы `create_run` не падал 3. Pre-existing E501 в `ScrapeRequest` (admin.py) попутно исправлены — комментарии перенесены 4. `SessionLocal` (sync) для background task — соответствует pattern всего codebase ## Auth Все 3 endpoints **без require_admin** (consistent с PR #474 — Caddy basic_auth достаточен). ## Test plan - [x] ruff + pytest - [ ] Deploy → migration 051 auto-apply через GHA → verify `scrape_runs` enum has 'cancelled' + new columns - [ ] POST `/admin/scrape/avito-city-sweep {pages_per_anchor: 1, detail_top_n: 5}` → verify run_id returned + background task progresses через `/runs` polling - [ ] During run → POST `/cancel/{run_id}` → verify status → `cancelled` в течение следующего anchor - [ ] Crontab setup на prod VPS (out of PR scope, manual deploy step) ## Frontend contract Frontend PR (parallel) wires `/scrapers/avito` UI к этим 3 endpoints с polling каждые 5s. ## Refs - Plan: AvitoScraper_v2 Stage 4d (auto pipeline, поверх Stage 2e per-anchor wrapper) - scrape_runs schema base: Stage 1 migration 015
lekss361 added 1 commit 2026-05-23 14:35:01 +00:00 
- avito.py: fetch_around(pages=N, delay_override_sec) — pagination 1..N (backward compat default=1)
- scrape_pipeline.py:
  * EKB_ANCHORS const (5 точек ЕКБ)
  * run_avito_pipeline: pages + request_delay_sec params (backward compat defaults)
  * CitySweepCounters dataclass с anchors_done/total + aggregate counters
  * run_avito_city_sweep(db, run_id, *, pages_per_anchor, detail_top_n, request_delay_sec)
  * Cooperative cancel: checks scrape_runs.status каждый anchor
  * Per-anchor heartbeat + counters refresh
- scrape_runs.py NEW: utility module (create_run/update_heartbeat/mark_done/failed/cancelled/list_recent/is_cancelled)
- admin.py: 3 NEW endpoints:
  * POST /admin/scrape/avito-city-sweep — start (returns run_id, FastAPI BackgroundTasks)
  * GET  /admin/scrape/avito-city-sweep/runs?limit=N — recent runs list
  * POST /admin/scrape/avito-city-sweep/{run_id}/cancel — cooperative cancel
- 051_scrape_runs_extend.sql: ADD COLUMN params/counters/error/finished_at + cancelled в status enum
- avito-city-sweep.sh: cron wrapper с random 0-3h delay (anti-pattern-detection)
- tests/test_city_sweep.py: 16 offline smoke tests (counters, scrape_runs utils, endpoints)

Crontab entry: 0 2 * * * /opt/.../avito-city-sweep.sh (runs 02:00-05:00 UTC daily)
lekss361 merged commit 2914ff2ef2 into main 2026-05-23 14:38:08 +00:00
lekss361 commented 2026-05-23 14:38:43 +00:00
Author
Owner
Copy link

Merged via deep-code-reviewer — verdict APPROVE.

Key safety findings

Migration 051 (CRITICAL focus)

  • status хранится как TEXT + CHECK constraint (не PostgreSQL ENUM type) — концерн ALTER TYPE ADD VALUE inside transaction N/A. DROP/ADD CHECK работает в BEGIN/COMMIT ✅
  • ADD COLUMN IF NOT EXISTS для всех 4 столбцов — идемпотентно ✅
  • ALTER COLUMN run_type SET DEFAULT 'city_sweep' — affects future inserts only, existing rows untouched ✅
  • Маленькая таблица, нет lock/rewrite concerns ✅

Cooperative cancel race (CRITICAL focus)

  • Race scenario: cancel arrives после последнего anchor но до mark_done →
    mark_cancelled succeeds (WHERE status='running') → status=cancelled →
    mark_done UPDATE с WHERE status='running' matches 0 rows → cancelled status persists ✅
  • Cancel priority guaranteed by AND status='running' guard в mark_done/mark_failed.

Backward compat

  • fetch_around(pages=1) default — existing callers (run_avito_pipeline from non-sweep paths) не affected ✅

Contract stability для PR #478

Все 3 endpoints stable:

  • POST /admin/scrape/avito-city-sweep → {run_id, status, pages_per_anchor, detail_top_n}
  • GET /admin/scrape/avito-city-sweep/runs?limit=N → [{run_id, source, status, params, counters, error, started_at, finished_at, heartbeat_at}] (ISO strings или null)
  • POST /admin/scrape/avito-city-sweep/{run_id}/cancel → {ok, run_id, cancelled}

Frontend PR #478 готов к merge после backend deploy confirmed live (миграция 051 applied + endpoint smoke).

Follow-ups (отдельным PR, не блокирующее)

  1. Zombie detection: FastAPI BackgroundTasks теряются при backend restart → run zависает в status='running' навсегда. Добавить cron job: UPDATE scrape_runs SET status='zombie' WHERE status='running' AND heartbeat_at < NOW() - INTERVAL '2 hours' AND source='avito_city_sweep'. Index scrape_runs_zombies_idx уже есть (015).
  2. Celery migration (long-term): для production-grade reliability перевести на Celery task с retry / acks_late. Acceptable trade-off для pilot Stage 4d.
  3. Within-sweep anchor order randomization: сейчас 5 anchors hit sequentially в fixed order → IP ban risk mitigated random 0-3h delay daily, но в пределах sweep порядок предсказуем. Future: random.shuffle(_anchors).

Post-merge verify checklist

  • deploy-tradein.yml triggers (paths backend + deploy + data/sql)
  • Migration 051 apply без error (CHECK constraint DROP+ADD)
  • \d+ scrape_runs → есть params jsonb, counters jsonb, error text, finished_at timestamptz + CHECK включает 'cancelled'
  • POST /admin/scrape/avito-city-sweep {pages_per_anchor:1, detail_top_n:5} → run_id returned, через 60s /runs показывает progress
  • Cancel mid-flight → status='cancelled' в течение 1-2 минут (next anchor boundary)
Merged via deep-code-reviewer — verdict **APPROVE**. ## Key safety findings **Migration 051 (CRITICAL focus)** - `status` хранится как **TEXT + CHECK constraint** (не PostgreSQL ENUM type) — концерн `ALTER TYPE ADD VALUE` inside transaction **N/A**. DROP/ADD CHECK работает в BEGIN/COMMIT ✅ - `ADD COLUMN IF NOT EXISTS` для всех 4 столбцов — идемпотентно ✅ - `ALTER COLUMN run_type SET DEFAULT 'city_sweep'` — affects future inserts only, existing rows untouched ✅ - Маленькая таблица, нет lock/rewrite concerns ✅ **Cooperative cancel race (CRITICAL focus)** - Race scenario: cancel arrives после последнего anchor но до `mark_done` → `mark_cancelled` succeeds (`WHERE status='running'`) → status=cancelled → `mark_done` UPDATE с `WHERE status='running'` matches 0 rows → cancelled status persists ✅ - Cancel priority guaranteed by `AND status='running'` guard в mark_done/mark_failed. **Backward compat** - `fetch_around(pages=1)` default — existing callers (`run_avito_pipeline` from non-sweep paths) не affected ✅ ## Contract stability для PR #478 Все 3 endpoints **stable**: - `POST /admin/scrape/avito-city-sweep` → `{run_id, status, pages_per_anchor, detail_top_n}` - `GET /admin/scrape/avito-city-sweep/runs?limit=N` → `[{run_id, source, status, params, counters, error, started_at, finished_at, heartbeat_at}]` (ISO strings или null) - `POST /admin/scrape/avito-city-sweep/{run_id}/cancel` → `{ok, run_id, cancelled}` Frontend PR #478 готов к merge **после backend deploy confirmed live** (миграция 051 applied + endpoint smoke). ## Follow-ups (отдельным PR, не блокирующее) 1. **Zombie detection**: FastAPI BackgroundTasks теряются при backend restart → run zависает в `status='running'` навсегда. Добавить cron job: `UPDATE scrape_runs SET status='zombie' WHERE status='running' AND heartbeat_at < NOW() - INTERVAL '2 hours' AND source='avito_city_sweep'`. Index `scrape_runs_zombies_idx` уже есть (015). 2. **Celery migration** (long-term): для production-grade reliability перевести на Celery task с retry / acks_late. Acceptable trade-off для pilot Stage 4d. 3. **Within-sweep anchor order randomization**: сейчас 5 anchors hit sequentially в fixed order → IP ban risk mitigated random 0-3h delay daily, но в пределах sweep порядок предсказуем. Future: `random.shuffle(_anchors)`. ## Post-merge verify checklist - [ ] `deploy-tradein.yml` triggers (paths backend + deploy + data/sql) - [ ] Migration 051 apply без error (CHECK constraint DROP+ADD) - [ ] `\d+ scrape_runs` → есть `params jsonb`, `counters jsonb`, `error text`, `finished_at timestamptz` + CHECK включает `'cancelled'` - [ ] `POST /admin/scrape/avito-city-sweep {pages_per_anchor:1, detail_top_n:5}` → run_id returned, через 60s `/runs` показывает progress - [ ] Cancel mid-flight → status='cancelled' в течение 1-2 минут (next anchor boundary)
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
admin

Admin UI / internal tools / pipeline tracker

  
analytics

Аналитика данных / matching

  
auth

Authentication / authorization / multi-tenancy

  
automation

Автоматизация / cron / scheduler

  
bug

Something isn't working

  
business

Коммерческая воронка / пилоты / pipeline / реестр ПО

  
chore

Operational housekeeping

  
ci

Forgejo Actions / build pipeline

  
compliance

152-ФЗ / реестр ПО / legal / TOS

  
data

Data sources, parsers, ETL

  
data-moat

Уникальные данные — sales tracker, CRM aggregation

  
docs

README / runbooks / onboarding

  
duplicate

This issue or pull request already exists

  
dx

Developer experience / build speed

  
enhancement

New feature or request

  
Fable 5 ревью

Находки полного мульти-агентного ревью (Claude Fable 5, 2026-06-09)

  
feedback/max

Feedback from Max (Senior partner)

  
generative

Generative Design — concept generation engine

  
GG-форсайт

Новое ТЗ от Ген Геныча (июнь 2026): Site Finder v2 — ИИ-прогноз продуктовой стратегии девелопера. Актуальный приоритет.

  
needs-discussion

Pending team sync / decision

  
needs-human

Human review/decision required

  
observability

Monitoring / metrics / logs / Sentry / Uptime

  
pause-bots

Kill-switch — all autonomous bots pause

  
performance

Latency / throughput / DB query tuning

  
priority/p0

Critical / production-blocker

  
priority/p1

High priority

  
priority/p2

Normal priority

  
priority/p3

Низкий приоритет (follow-up, tech-debt)

  
scope/backend

Backend Python — FastAPI, Celery, scrapers

  
scope/db

DB migrations, indexes, SQL

  
scope/devops

docker-compose, Caddy, workflows

  
scope/frontend

Frontend — Next.js, React, TypeScript

  
scope/qa

Tests, smoke, Playwright

  
scrapers

Скрапперы (Avito/Cian/Yandex/Rosreestr)

  
security

Security audit / secrets / vulnerabilities

  
site-finder

Site Finder scoring quality & data layer

  
stage/1

Stage 1: Generative MVP (нед 2-7)

  
stage/2

Stage 2: Site Finder MVP (нед 8-12)

  
status/blocked

Blocked — needs human intervention

  
status/done

Closed, completed

  
status/needs-analysis

Raw/fuzzy task filed by human — auto-analyst подхватывает: code-archeology + декомпозиция в actionable scope/* + status/ready (или split на под-issues)

  
status/needs-fix

Reviewer/QA вернул worker на доработку — worker сам чинит свой PR (fixup), НЕ требует human (в отличие от status/blocked)

  
status/qa

PR merged, awaiting smoke test

  
status/ready

Ready for pickup by worker bot

  
status/review

PR open, awaiting code review

  
status/wip

Work in progress (worker claimed)

  
tech-debt

Технический долг

  
tradein

Trade-In MVP проект

  
ux

User experience / responsive / conversion

  
week ревью 1

Week review 1 — находки multi-agent аудита проекта (2026-06-14)

  
wontfix

This will not be worked on

  
вторичка

Trade-In Estimator — проект оценки вторичного жилья

  
ИРД

Исходно-разрешительная документация — слой ограничений участка для analyze-таблицы

No labels
admin
analytics
auth
automation
bug
business
chore
ci
compliance
data
data-moat
docs
duplicate
dx
enhancement
Fable 5 ревью
feedback/max
generative
GG-форсайт
needs-discussion
needs-human
observability
pause-bots
performance
priority/p0
priority/p1
priority/p2
priority/p3
scope/backend
scope/db
scope/devops
scope/frontend
scope/qa
scrapers
security
site-finder
stage/1
stage/2
status/blocked
status/done
status/needs-analysis
status/needs-fix
status/qa
status/ready
status/review
status/wip
tech-debt
tradein
ux
week ревью 1
wontfix
вторичка
ИРД
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: lekss361/gendesign#477
No description provided.
Delete branch "feat/tradein-avito-city-sweep"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?