Commit graph

6 commits

Author SHA1 Message Date
bot-backend
f627b0fa4d feat(tradein/domclick): production Layer B detail-backfill orchestrator (#2000)
All checks were successful
CI Trade-In / changes (pull_request) Successful in 8s
CI / changes (pull_request) Successful in 8s
CI Trade-In / frontend-checks (pull_request) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
CI Trade-In / backend-tests (pull_request) Successful in 53s
CI / frontend-tests (pull_request) Has been skipped
Ports the cookie-injection + organic SERP-origin session->cookies->fetch_detail
wiring (PR #2430, PR #2433), previously only reachable via the manual debug
endpoint, into a scheduled orchestrator mirroring avito_detail_backfill.py's
shape: snapshot-once-then-loop, budget guard, SIGTERM-drain, jittered delay,
heartbeat every 25 attempts.

DomClickBlockedError (QRATOR challenge / browser-fetch failure) increments a
consecutive-block counter and aborts via mark_done (not mark_failed) past
max_consecutive_blocks -- a block is an expected operational outcome for a
QRATOR reputation-based anti-bot, not a task failure. DomClickParseError
(__SSR_STATE__ schema drift) is a plain failed++, neutral to the breaker.
Unlike Avito there is no curl fallback (one BrowserFetcher per run) and no
IP-rotation recovery (single dedicated residential proxy).

Migration 175 seeds the schedule with enabled=false -- activation is a
deliberate manual step after smoke-testing against a live cookie session.
default_params are more conservative than Avito's (batch_size=200,
request_delay_sec=12, max_consecutive_blocks=3) since a burned QRATOR
reputation/session is costlier to recover than an Avito IP-block.

Also fixes a stale docstring in scraper_kit's domclick/detail.py that
incorrectly named DataDome as the anti-bot mechanism -- it's QRATOR
(exit-IP reputation-based), per the extensive live verification in #2000.
2026-07-04 22:40:40 +03:00
5fa505b809 fix(tradein/scrapers): подключить geoportal coords backfill в scheduler (#1967) (#2346)
All checks were successful
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / build-backend (push) Successful in 52s
Deploy Trade-In / changes (push) Successful in 9s
Deploy Trade-In / test (push) Successful in 1m30s
Deploy Trade-In / deploy (push) Successful in 1m48s
2026-07-03 23:25:47 +00:00
bot-backend
90731da537 feat(tradein): location-coef MVP через FDW-мост к OSM POI Птицы (#2045)
All checks were successful
CI Trade-In / changes (pull_request) Successful in 7s
CI / changes (pull_request) Successful in 8s
CI Trade-In / frontend-checks (pull_request) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
CI Trade-In / backend-tests (pull_request) Successful in 1m36s
Финальный PR issue #2045 (BE-3): GET /api/v1/trade-in/location-coef для
LocationDrawer. FDW foreign table -> локальное зеркало osm_poi_ekb_local
(TRUNCATE+INSERT, тот же паттерн что cad_buildings_local/cadastral_geo_match,
избегает ~1.16s/row FDW round-trip) -> straight-line POI-скоринг, портированный
из Site Finder poi_score.py::compute_poi_weighted_top7 (CATEGORY_WEIGHTS as-is,
радиус 1200м для квартир вместо Ptica 2000м для участков). score->coef -
новая MVP-эвристика (0.95..1.05, не откалибрована на реальных дельтах).

Graceful fallback (не 500, не фабрикуем факторы): пустая/не отрефрешенная
osm_poi_ekb_local или отсутствие lat/lon у оценки -> coef=1.0, factors=[],
geo_source="unavailable".

Scheduler: source=osm_poi_ekb_refresh, daily, зарегистрирован и в боевом
dispatch (scheduler.py), и в kit-registry (product_handlers.py) - иначе
test_kit_registry_completeness падает на ship-dark инварианте (#2192).

Frontend wiring (mappers.ts/LocationDrawer.tsx) - вне scope, отдельная задача
после проверки endpoint'а curl'ом на деплое.
2026-07-03 23:44:55 +03:00
ac19c4f7b3 feat(tradein/monitor): алерт на staleness данных deals по max(deal_date) (#2212) (#2226)
All checks were successful
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / changes (push) Successful in 8s
Deploy Trade-In / test (push) Successful in 1m35s
Deploy Trade-In / build-backend (push) Successful in 56s
Deploy Trade-In / deploy (push) Successful in 1m2s
2026-07-02 20:50:37 +00:00
f5b0076e6f fix(tradein): deactivate_stale для domklik/n1 + честная freshness по scraped_at (#2204) (#2221)
All checks were successful
Deploy Trade-In / changes (push) Successful in 9s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 1m30s
Deploy Trade-In / build-backend (push) Successful in 58s
Deploy Trade-In / deploy (push) Successful in 2m57s
2026-07-02 20:10:34 +00:00
bcdec5ebd4 feat(rewire): scheduler_main → kit-scheduler behind USE_KIT_SCHEDULER flag, ship-dark (#2192)
All checks were successful
Deploy Trade-In / deploy (push) Successful in 58s
Deploy Trade-In / changes (push) Successful in 12s
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / build-frontend (push) Successful in 37s
Deploy Trade-In / test (push) Successful in 1m38s
Deploy Trade-In / build-backend (push) Successful in 50s
2026-07-02 18:41:12 +00:00