Commit graph

4 commits

Author SHA1 Message Date
lekss361
36df5d92d2 fix(tradein/avito): #487 BLOCK — mark_banned + defensive rollback (Option B)
deep-code-reviewer P0: mark_rate_limited() writes status='rate_limited' but
SQL CHECK constraint (015 + 051) only allows banned/done/failed/etc.
First blocking event would CheckViolation -> cascade through mark_failed ->
runs stuck in 'running' -> zombie cleanup. Frontend never shows badge.

Option B fix (no migration): rename mark_rate_limited -> mark_banned, use
existing 'banned' status (migration 015 doc'd as 'Avito вернул 403/captcha' -
exactly this scenario).

Plus:
- mark_failed + mark_banned now defensive db.rollback() at start (cascade-safe)
- All mark_* now RETURNING id + log warning when no rows updated (M3)
- Callers updated: scrape_pipeline.run_avito_city_sweep + test names

Refs PR #487 review (deep-code-reviewer, 2026-05-23).
2026-05-23 23:01:49 +03:00
lekss361
ee87575052 fix(tradein/avito): anti-bot hardening — sleep + abort + shared session
Production был заблокирован 23.05 17:48-17:55 MSK из-за tight loop без delay
в detail enrichment (~20 req/sec к Avito). Все 3 подряд runs получили 403.

Bundled fix:
1. AvitoBlockedError / AvitoRateLimitedError (новый avito_exceptions.py)
2. fetch_around, fetch_detail, fetch_house_catalog — raise на HTTP 403/429
3. scrape_pipeline: shared AsyncSession через весь sweep (один TLS handshake)
4. asyncio.sleep с +-20% random jitter между detail requests (default 7s)
5. Early abort: после 3 consecutive blocks — abort sweep + propagate
6. scrape_runs.mark_rate_limited (status=rate_limited vs failed/done)
7. detail_top_n default 20->10 в CitySweepStartRequest
8. logger.error вместо warning для 403/429 -> GlitchTip alert

Confirmed root cause: prod logs показали 403 в 40-60ms intervals
(blast pattern без delay между fetch_detail calls).
2026-05-23 23:00:23 +03:00
2914ff2ef2 feat(tradein): city sweep — auto ЕКБ pipeline (#477)
All checks were successful
Deploy Trade-In / build-frontend (push) Successful in 23s
Deploy Trade-In / deploy (push) Successful in 22s
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-backend (push) Successful in 38s
Stage 4d: full city sweep ЕКБ (5 anchors × N pages + enrichment + cooperative cancel).

- avito.py fetch_around(pages=N) — pagination backward compat default=1
- scrape_pipeline.run_avito_city_sweep — anchors loop + heartbeat + cancel check
- scrape_runs.py utility (create/heartbeat/done/failed/cancelled/list_recent)
- admin.py 3 endpoints: start (BackgroundTasks) / runs / cancel
- migration 051 ADD COLUMN IF NOT EXISTS + DROP/ADD CHECK constraint (text+CHECK, не enum — safe в transaction)
- deploy/avito-city-sweep.sh с random delay 0-3h (anti-pattern)
- 16 offline tests

Deep review approved: migration idempotent, cancel priority correct (mark_done guarded by status='running'), no SQL injection, psycopg v3 compliant.
2026-05-23 14:38:07 +00:00
eb4764f75a feat(tradein): scrape_pipeline.py — Avito full orchestrator (search → houses → detail) (#451)
All checks were successful
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-backend (push) Successful in 56s
Deploy Trade-In / deploy (push) Successful in 28s
2026-05-23 12:55:13 +00:00