42 commits

Author	SHA1	Message	Date
lekss361	32d1818fd4	fix(scrapers/avito): unify _unix_to_date timezone across serp/houses/imv (#2129)	2026-07-02 14:03:11 +00:00
lekss361	69845ddcee	fix(tradein/avito-detail): живой прогрев сессии + батч-на-сессию — снимает 403 на curl-backfill (#1551)	2026-06-28 22:34:38 +00:00
bot-backend	a50af40ad6	fix(tradein/avito): reap dead 404 listings in detail backfill (don't trip block breaker) ... avito_detail_backfill (mode=browser) aborted prod run 458 with attempted=5 enriched=0 blocked=5: the lat-null pending queue is dominated by DEAD listings (404 — that's why they are stale coord-holes). In browser mode a dead listing's 404 page ("Ошибка 404. Страница не найдена") has no item-view, so _is_detail_soft_block returned True → AvitoBlockedError → backfill counted it as `blocked` and the consecutive-block breaker aborted the run before reaching live listings. - New AvitoListingGoneError (subclass of common AvitoError, NOT of AvitoBlockedError/AvitoRateLimitedError) so the backfill block-trap does not catch it. - New _is_detail_not_found() detector + browser-mode branch in fetch_detail, checked BEFORE firewall/soft-block (keyed on the 404 title so it does not swallow the generic soft-block deflect). Curl path unchanged (404 → 302/ non-200 → ValueError → failed, as before). - Backfill: new `gone` counter; dedicated except AvitoListingGoneError branch that is neutral to the consecutive-block breaker and marks the listing is_active=FALSE (SAVEPOINT) so it leaves the snapshot scope. Tests: unit for _is_detail_not_found (404 True; soft-block/item-view/empty False), browser-mode fetch_detail raising AvitoListingGoneError (not Blocked), and backfill gone-row marks inactive without aborting the breaker.	2026-06-28 14:05:31 +03:00
lekss361	ca4637e5c8	fix(tradein/avito): detect HTTP-200 soft-block → rotate + coords-priority backfill ordering (#1950, #1967) ... Soft-block (HTTP 200 generic витрина без item-view) теперь детектится как блок → существующая 403/firewall reconnect-машинерия → ротация exit-IP вместо silent failed (blocked=0, enriched→0 каскад). + snapshot ORDER BY (lat IS NULL) DESC для приоритизации coord-holes (#1967, in-scope 321). Tests: test_avito_detail_soft_block.py. Full tradein suite 2591 passed.	2026-06-28 08:06:12 +00:00
bot-backend	43b392b809	feat(tradein/domclick): local-runner + ingest tooling → main (split from parked Layer-B) ... Выделяет МЕРДЖАБЕЛЬНУЮ часть DomClick-тулинга из запаркованного PR #2005 (BLOCKED #2000 — прод Layer-B скрейпер через прокси режет QRATOR-403): - scripts/domclick_local_runner.py — ручной offline-сборщик с домашнего IP: SERP-enumerate (фронт, BFF-JSON режет QRATOR) + card __SSR_STATE__ + offer-card v3 (AVM price_prediction + sold_similar). --enum-cache (enumerate 1 раз → файл с sentinel → следующий запуск грузит кэш и пропускает enumerate), wait_for_selector (SERP-карточки догидрируются), resume по id. Self-contained. - scripts/ingest_domclick_jsonl.py — JSONL → trade-in БД (save_listings + house-match в SAVEPOINT + save_detail_enrichment, идемпотентно, --limit/--dry-run). - app/services/scrapers/domclick_detail.py + domclick_exceptions.py — card-SSR parse + save_detail_enrichment (нужны ингесту). Импортятся ТОЛЬКО ингестом (operator-run); прод-runtime их не дёргает (dormant в образе). - tests/scrapers/test_domclick_detail.py + обновлённый scripts/README.md. НЕ включает прод Layer-B (tasks/domclick_detail_backfill.py, scheduler-расписание, data/sql/139_*) — остаётся запаркован на PR #2005 до чистого прокси (#2000). БД-схема не меняется (detail-колонки уже в проде, миграции нет). Live-validated: раннер качает ЕКБ-вторичку прямо сейчас (600+ карт, 0 блоков), ингест залил их в прод-БД (house_matched, идемпотентно).	2026-06-28 10:26:52 +03:00
bot-backend	54a5d13c48	feat(tradein/domclick): enrich Layer A SERP parser — description, repair-inference, minhash, seller ... Capture extra bff-search fields in _map_item (verified live): - description → ScrapedLot.description + infer_repair_state_from_text → repair_state (DomClick search has no renovation-enum; text inference like cian #622) - sha1 description_minhash (cross-source dedup) - agency_name from seller (defensive str|dict) - raw_payload: is_sber_collateral, has_discount, discount_value, duplicates_offer_count, seller Additive only; existing Layer A behavior unchanged. Deep card detail (renovation-enum, wallType, full price-history, owners_count) is card-only = QRATOR-blocked on prod proxy → separate effort (needs dedicated residential IP).	2026-06-27 16:35:25 +03:00
bot-backend	52fc920507	fix(tradein/domclick): rewrite dead scraper to BFF JSON API + honest status (#1846, #1968) ... Old HTML scraper dead: /search?city_id retired in DomClick redesign + VPS IP QRATOR-banned (prod domclick_city_sweep reported done/0 every run). New path: - GET bff-search-web.domclick.ru/api/offers/v1 + count/v1 via BrowserFetcher(source="domclick") -> generic provider -> shared mobile proxy -> bypasses QRATOR (prod-verified live: 20 items returned, no block). - Sweeps 6 room buckets (st/1/2/3/4/5+); recursive binary price-split for oversized buckets (rooms=2 ~2215 > offset cap 2000); count/v1 sizes buckets. - Maps SERP JSON -> ScrapedLot with lat/lon at ~100% (closes avito 54% NULL-lat gap); per-card EKB geo-guard. - Honest status (#1968): QRATOR block OR fetch errors with 0 lots -> mark_failed, not mark_done. Parse failures are per-page resilient (break bucket, keep partial lots) so a transient bad page never aborts the whole sweep. - Schedule shipped DORMANT (enabled=false, pages=100) via 138_domclick_bff_rewrite_schedule.sql; operator enables after prod smoke. 42 scraper tests (price-split boundaries, parse-error resilience, honest status); full backend suite green; ruff clean. Layer B (detail renovation/wallType/ priceHistory) deferred to #1846 follow-up.	2026-06-27 13:57:07 +03:00
bot-backend	e03c810306	feat(tradein-imv): route IMV backfill через BrowserFetcher.fetch_json за dormant-флагом (#915 Stage 3) (#1916)	2026-06-26 09:54:28 +00:00
bot-backend	d479edeb3f	feat(tradein-browser): additive POST /fetch-json + BrowserFetcher.fetch_json (#915 Stage 2) (#1915)	2026-06-26 09:30:35 +00:00
bot-backend	cd91d5e279	fix(scrapers): avito pipeline — новостройки house graceful (без housePage), AvitoBlockedError per-item не крашит прогон (#1820)	2026-06-20 09:53:46 +03:00
bot-backend	37fc50ad08	fix(scrapers): avito_detail_backfill — снапшот ЕКБ+active, detail browser-only без curl-fallback на бан (#1814)	2026-06-19 21:58:35 +03:00
bot-backend	b1fb0c5711	feat(scrapers): cian SERP на BrowserFetcher (camoufox), снят code-side rotate (#1806)	2026-06-19 20:41:44 +03:00
bot-backend	d65a88f704	fix(avito): reconnect-retry on detail 429 under backconnect proxy ... Detail backfill (run 217) hit 5 consecutive HTTP 429s and aborted at 19 enriched because the detail fetcher only had a same-session 429 short-retry (no reconnect). 429s are persistent: keep-alive reuses one backconnect exit IP across sequential detail fetches -> avito rate-limits it. Mirror the merged SERP 429 fix (#1769): widen short-retry 4->8 and add up to 3 ephemeral-session reconnects (fresh exit IP) on 429 exhaustion before raising. Caller session never touched (as with 403 path).	2026-06-18 23:29:47 +03:00
bot-backend	bee44ffd3d	fix(avito): tolerate 429 per-bucket instead of banning whole sweep ... Run 216 recovered 403s (merged fix) but still banned on a 429 at page 36 (deep pagination, backconnect conn-limit). Widen the 429 retry budget + one reconnect on exhaustion in _fetch_serp_html, and make fetch_all_secondary skip a blocked bucket and continue rather than aborting the run; only N consecutive blocked buckets (real hard ban) re-raises. Preserves hard-ban detection.	2026-06-18 23:03:28 +03:00
bot-backend	c993243bc7	fix(avito): reconnect-retry on detail 403/firewall under backconnect proxy ... fetch_detail raised on the first 403/429, aborting the detail backfill (run 191 enriched only 48 then zombied) -> avito detail coverage stuck at 14% of active. Mirror the merged SERP fix (#1765): on 403/firewall under the backconnect proxy, rebuild an ephemeral session (fresh exit IP) and retry up to N before raising; short-retry 429. Never close/rebuild the caller-provided shared session. Also add the missing HTTP-200 firewall-interstitial check on the curl detail path.	2026-06-18 22:12:03 +03:00
bot-backend	bdedac3012	feat(avito): incremental SERP pagination with date early-stop (param-gated, default off) ... Adds since=date mode to fetch_all_secondary: per room x seed-bracket, paginate sequentially newest-first and stop once cards fall below the watermark, skipping deep pagination (and the 403 clusters it triggers). since=None keeps the exact exhaustive bisection walk. No scheduler/schedule change here — wiring follows in a separate PR.	2026-06-18 22:01:46 +03:00
bot-backend	44e316c514	fix(avito): gate 403 reconnect-retry on proxy presence only, not rotate_url absence ... PR #1764's reconnect-retry was dead code in prod: the gate excluded configs with avito_proxy_rotate_url set, but prod has BOTH the backconnect SCRAPER_PROXY_URL (what curl_cffi egresses through) AND a leftover changeip URL for the separate auv/browser proxy. With max_rotations=0 neither path fired -> a single SERP 403 still aborted the sweep (run 212). The curl session always uses scraper_proxy_url, so session-rebuild rotates the exit IP regardless of changeip config.	2026-06-18 21:31:20 +03:00
bot-backend	9dc3be56e3	fix(avito): reconnect-retry on SERP 403/firewall under backconnect proxy ... Under a backconnect rotating proxy (no changeip URL) a single HTTP 403 or firewall interstitial aborted the entire avito_full_load sweep via AvitoBlockedError -> mark_banned. Each new connection through backconnect gets a fresh exit IP, so retry by rebuilding the curl_cffi session up to _AVITO_403_MAX_RETRIES times before giving up. Legacy single-IP changeip rotation and browser-mode paths unchanged. Mirrors existing 429 short-retry.	2026-06-18 20:41:32 +03:00
bot-backend	6454aa312d	fix(tradein): avito _extract_total_count returns 0 on no-results page (stop empty-bracket split waste + fallback-listing scrape)	2026-06-18 20:01:41 +03:00
bot-backend	2c6d0e4c71	feat(tradein): avito curl_cffi 429 short-retry (backconnect conn-limit transient, no rotate/ban)	2026-06-18 19:24:05 +03:00
bot-backend	ffecc6962d	perf(tradein): shared EKB seed price-brackets across avito/cian/yandex (city-dict ready for multi-city)	2026-06-18 17:55:55 +03:00
bot-backend	3f85fa59ff	perf(tradein): avito seed price-brackets with open top bucket (≤2k blocks, no price ceiling)	2026-06-18 17:29:31 +03:00
bot-backend	cbeeb9c414	fix(tradein): avito full-load probe resilience — browser 500/timeout splits/skips instead of crashing run	2026-06-18 16:03:39 +03:00
bot-backend	6c132d7f0a	fix(tradein): avito sweep page-level retry — transient page error no longer aborts run	2026-06-18 14:29:01 +03:00
bot-backend	1ddbf867f8	feat(tradein): avito exhaustive full-load (room×price bisection) — mirror cian	2026-06-18 13:55:18 +03:00
bot-backend	b9eb478c8f	feat(browser): per-source proxy pool behind FEATURE_BROWSER_POOL_ENABLED (Phase 1) ... Route each scraper source (avito/cian/yandex/domclick) to its own camoufox browser+proxy so they no longer wedge each other through a single global egress. Feature-flagged (FEATURE_BROWSER_POOL_ENABLED, default OFF): with the flag off the /fetch and /login paths are byte-for-byte the existing single-browser behavior. When on, /fetch routes by body["source"] to a per-proxy browser via BROWSER_PROXY_MAP (BROWSER_PROXY_AVITO/CIAN/YANDEX/DOMCLICK with legacy fallbacks), each guarded by its own lazy-launched lock. /login stays single-browser in Phase 1. BrowserFetcher gains a source arg (default avito) and sends it in the /fetch body; all scraper callsites pass their source. No docker-compose/.env.runtime changes (Phase 2, owner-gated).	2026-06-18 09:01:50 +03:00
bot-backend	17c494a063	fix(scrapers): cian _walk_price_range stop silently truncating price-range tail (#1393)	2026-06-17 20:39:16 +03:00
lekss361	7a88964ef5	fix(avito): scale per-anchor timeout for detail-enrich + durable SERP counters (#1661)	2026-06-17 06:07:50 +00:00
bot-backend	567549c3aa	feat(tradein): cian full-load secondary-only — drop novostroyki at parse (#938) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 21:35:32 +00:00
bot-backend	846c5014c6	feat(tradein): checkpoint/resume for cian full-load — survive restart (#930) (#932) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 21:11:48 +00:00
bot-backend	5e94d558b5	perf(tradein): cian full-load incremental save + concurrent pagination (#929) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 20:27:27 +00:00
bot-backend	0883f6d0f1	feat(tradein): exhaustive Cian EKB secondary load — room×price partitioning (#926) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 19:54:19 +00:00
bot-backend	6044a1fe3c	fix(browser): local camoufox + HTTP wrapper, drop broken playwright WS server (#905) ... The playwright WS-server path (camoufox.server.launch_server) is incompatible with playwright >=1.45: camoufox 0.4.11's launchServer.js requires playwright/driver/package/lib/browserServerImpl.js, which no longer exists in any playwright 1.45-1.60 → tradein-browser crash-looped with MODULE_NOT_FOUND ('Server process terminated unexpectedly'). Verified by running the built image. Pivot (no playwright protocol between containers): - browser/server.py: aiohttp service holding a local AsyncCamoufox (headless, os/locale/geoip/humanize, proxy from SCRAPER_PROXY_URL). POST /fetch {url} → new_page/goto/content/close → {html}; GET /health. Page-recycle every N + crash-recovery (relaunch + 1 retry), serialized via asyncio.Lock. - browser/Dockerfile: +aiohttp (camoufox fetch-as-root/#908 block untouched). - backend BrowserFetcher: playwright.connect → httpx POST to the browser service (one retry on transport/HTTP error). All playwright imports dropped. - config: browser_ws_endpoint → browser_http_endpoint (http://tradein-browser:3000). Verified by a REAL local image build + run: image builds, container stays Up (Restarts=0, no crash-loop), GET /health=200, POST /fetch launches camoufox and drives Firefox (only NS_ERROR_UNKNOWN_HOST = no DNS egress in the local build env, proving the full HTTP→camoufox→Firefox chain; real fetch validates on prod with network+proxy). 7 unit tests + ruff clean. Dormant: scraper_fetch_mode stays curl_cffi. Refs #905, #883, #884	2026-05-31 18:25:00 +03:00
lekss361	d6479c13b9	Merge pull request 'feat(browser): split camoufox into a separate tradein-browser container (Phase 0.5, #905)' (#907) from feat/884-browser-separate-container into main ... Reviewed-on: #907	2026-05-31 14:44:00 +00:00
bot-backend	5fb367295a	feat(browser): split camoufox into tradein-browser container, connect-mode (#905) ... Phase 0.5 — revise #884's in-backend camoufox into a dedicated browser service. camoufox.server.launch_server exposes a Playwright WS endpoint (fingerprint/geoip/ proxy configured server-side); the backend connects remotely via playwright, so it no longer ships Firefox. - NEW tradein-mvp/browser/{server.py,Dockerfile}: camoufox WS server on :3000; Firefox fetch under app-user + ENV HOME (carries the #899 cache-path fix) - BrowserFetcher rewritten launch-in-process → playwright.firefox.connect(ws_endpoint); recycle (close+reconnect every N) + crash-recovery (TargetClosed → reconnect + 1 retry) - backend Dockerfile slimmed: drop 5 Firefox libs + camoufox fetch layer - pyproject: remove camoufox[geoip] from backend, promote playwright to main deps - docker-compose.prod.yml: add browser service (mem_limit 2.5g moves here), remove mem_limit from backend, backend depends_on browser (service_started) - config: browser_ws_endpoint default ws://tradein-browser:3000/playwright - tests/scrapers/test_browser_fetcher.py rewritten for connect-mode (6 pass) Dormant: scraper_fetch_mode stays curl_cffi → no prod behavior change. launch_server signature verified (**kwargs→launch_options, port passes through). Refs #905, #883, #884, #899	2026-05-31 17:24:35 +03:00
bot-backend	4c0338ce1d	fix(tradein): domklik source name, scraper delay, DataDome block detect (#796 fixup)	2026-05-31 17:21:30 +03:00
bot-backend	3c5e9ad368	feat(tradein): DomClick вторичка scraper via BrowserFetcher (#796)	2026-05-31 16:58:12 +03:00
bot-backend	e03f4e4534	feat(tradein): BrowserFetcher camoufox wrapper + Docker/config infra (Phase 0, #884) ... Epic #883 foundation. Provider-agnostic BrowserFetcher (camoufox headless Firefox): async ctx mgr + page-recycle + crash-recovery + proxy. camoufox/playwright lazy imports (won't break app import). scraper_fetch_mode defaults curl_cffi -> DORMANT (no parser/estimator/avito/cian change). config: 4 browser_* settings. Dockerfile: +5 Firefox libs + camoufox fetch layer. compose: mem_limit 2.5g. (uv.lock gitignored -> re-resolved at build.) 9 mocked tests pass, ruff clean. WARN: Docker build + container fetch UNVERIFIED (no local Docker); ~960MB image; merge auto-deploys to prod. MERGE-GATE post-demo + CI-build-verify. Refs #884, #883	2026-05-31 15:41:56 +03:00
bot-backend	950d0f60de	fix(tradein): sweep shared-session (SERP+detail) через mobile proxy (#623) (#805) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-30 18:07:37 +00:00
bot-backend	96027357b5	feat(tradein): Avito mobile proxy egress + firewall detect + MSK date (#623, #726) (#788) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-30 16:52:35 +00:00
bot-backend	489666c735	fix(tradein): avito content-block (HTTP-200) не маскируется под end-of-pagination (Refs #754) (#779) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-30 16:28:23 +00:00
lekss361	222389cba9	fix(tradein/avito): anti-bot hardening — sleep + abort + shared session (#487)	2026-05-23 20:08:40 +00:00