Финальный PR issue #2045 (BE-3): GET /api/v1/trade-in/location-coef для
LocationDrawer. FDW foreign table -> локальное зеркало osm_poi_ekb_local
(TRUNCATE+INSERT, тот же паттерн что cad_buildings_local/cadastral_geo_match,
избегает ~1.16s/row FDW round-trip) -> straight-line POI-скоринг, портированный
из Site Finder poi_score.py::compute_poi_weighted_top7 (CATEGORY_WEIGHTS as-is,
радиус 1200м для квартир вместо Ptica 2000м для участков). score->coef -
новая MVP-эвристика (0.95..1.05, не откалибрована на реальных дельтах).
Graceful fallback (не 500, не фабрикуем факторы): пустая/не отрефрешенная
osm_poi_ekb_local или отсутствие lat/lon у оценки -> coef=1.0, factors=[],
geo_source="unavailable".
Scheduler: source=osm_poi_ekb_refresh, daily, зарегистрирован и в боевом
dispatch (scheduler.py), и в kit-registry (product_handlers.py) - иначе
test_kit_registry_completeness падает на ship-dark инварианте (#2192).
Frontend wiring (mappers.ts/LocationDrawer.tsx) - вне scope, отдельная задача
после проверки endpoint'а curl'ом на деплое.
Деплой (docker recreate tradein-scraper) шлёт SIGTERM с stop_grace_period=120s
(Phase 1). Раньше scheduler_main отвечал hard task.cancel() → бегущий scrape-unit
получал CancelledError посреди браузер-карточки → карточка гибла без COMMIT.
Теперь SIGTERM/SIGINT лишь выставляют кооперативный флаг; tick-loop и длинные
задачи опрашивают его на СВОИХ существующих between-unit checkpoint'ах,
докоммичивают текущий unit и выходят сами.
- app/core/shutdown.py: новый standalone-модуль (asyncio.Event + request_shutdown
/ shutdown_requested / wait_for_shutdown), без app-зависимостей → нет циклов.
- scheduler_main._run: SIGTERM → request_shutdown() вместо task.cancel(); ждём
добровольного drain'а, safety-net wait_for(100s < docker grace) с fallback на
hard-cancel для некооперирующей задачи.
- scheduler_loop: проверка флага в начале тика и после каждого dispatch — не
reap/claim новые run'ы во время shutdown, выходим из loop'а.
- avito_detail_backfill: break на границе карточки рядом с budget-guard; snapshot
pending-query идемпотентен → следующий run сам резюмит остаток.
- import_rosreestr_dkp: расширен is_cancelled checkpoint — drain делает mark_done
(partial), не mark_cancelled; user-cancel семантика без изменений.
Без SIGTERM поведение идентично прежнему. reap_zombies не трогает drained-run'ы
(они mark_done, не 'running').
Tests: tests/core/test_shutdown.py, scheduler_main drain+timeout-fallback,
avito_detail_backfill partial-drain. 26 + 29 scheduler passed.
avito_detail_backfill (mode=browser) aborted prod run 458 with
attempted=5 enriched=0 blocked=5: the lat-null pending queue is dominated
by DEAD listings (404 — that's why they are stale coord-holes). In browser
mode a dead listing's 404 page ("Ошибка 404. Страница не найдена") has no
item-view, so _is_detail_soft_block returned True → AvitoBlockedError →
backfill counted it as `blocked` and the consecutive-block breaker aborted
the run before reaching live listings.
- New AvitoListingGoneError (subclass of common AvitoError, NOT of
AvitoBlockedError/AvitoRateLimitedError) so the backfill block-trap
does not catch it.
- New _is_detail_not_found() detector + browser-mode branch in fetch_detail,
checked BEFORE firewall/soft-block (keyed on the 404 title so it does not
swallow the generic soft-block deflect). Curl path unchanged (404 → 302/
non-200 → ValueError → failed, as before).
- Backfill: new `gone` counter; dedicated except AvitoListingGoneError branch
that is neutral to the consecutive-block breaker and marks the listing
is_active=FALSE (SAVEPOINT) so it leaves the snapshot scope.
Tests: unit for _is_detail_not_found (404 True; soft-block/item-view/empty
False), browser-mode fetch_detail raising AvitoListingGoneError (not Blocked),
and backfill gone-row marks inactive without aborting the breaker.
Soft-block (HTTP 200 generic витрина без item-view) теперь детектится как блок → существующая 403/firewall reconnect-машинерия → ротация exit-IP вместо silent failed (blocked=0, enriched→0 каскад). + snapshot ORDER BY (lat IS NULL) DESC для приоритизации coord-holes (#1967, in-scope 321). Tests: test_avito_detail_soft_block.py. Full tradein suite 2591 passed.
The chunked matcher paged the candidate set with OFFSET over the very predicate
it mutates (building_cadastral_number IS NULL). Matched rows drop out, so OFFSET
(reset to 0 OR advanced) skips un-processed rows: once >= batch_size unmatched
listings (beyond threshold) accumulate at the low-id front, the chunk matches 0
and the loop breaks early, leaving higher-id matchable listings unfilled.
Counterexample (batch=3, matchable ids 1,4,7,10): matches 1 and 4 then breaks,
missing 7 and 10. Unit tests used single-chunk fixtures so never caught it.
The full UPDATE is ~5s for ~41k listings, so replace the loop with ONE set-based
LATERAL KNN UPDATE over all candidates — correct and fast. batch_size kept in the
signature for schedule-param compat (now unused).
Backfill listings.building_cadastral_number (was 0%) from the nearest
cadastral building. gendesign_cad_buildings is a postgres_fdw foreign table
with no geom — a per-listing FDW nearest query is ~1.16s/row (~13h for 43k
listings). Instead materialize the FDW once into a LOCAL cad_buildings_local
table (Point geom + GIST), then run a fast local KNN nearest-neighbour join.
Perf: the distance gate uses geometry-space ST_DWithin(geom, point, deg) (GIST
index, no geography cast) + geom <-> KNN order + ST_DistanceSphere metric
recheck on the single nearest row. A geography-cast ST_DWithin in the WHERE
defeated the index (58s+, full update never finished); the geometry-gate design
does the whole ~41.9k-listing UPDATE in ~5.4s (refresh+match ~7s end-to-end),
matching 16027 listings at 50m across 2229 distinct buildings.
The match is GEO-NEAREST (approximate): a street-level-geocoded listing matches
the nearest building within threshold_m (default 50m), not necessarily its exact
cadastral building. Exact cadastral + parcel-containment deferred (cad_parcels
FDW not exposed). Threshold is logged.
- 124: cad_buildings_local table (empty) + GIST. Migration does not read the FDW
(deploy-independent); populated by the refresh job.
- 125: scrape_schedules seed source=cadastral_geo_match, enabled, next_run_at
tomorrow 09:00 UTC (after geocode_missing).
- tasks/cadastral_geo_match.py: refresh_cad_buildings_local (bulk FDW scan),
match_listings_to_buildings (chunked LATERAL KNN UPDATE), run_cadastral_geo_match
run-lifecycle wrapper.
- scheduler: trigger_cadastral_geo_match_run + dispatch (sync DB-only, executor).
Route each scraper source (avito/cian/yandex/domclick) to its own camoufox
browser+proxy so they no longer wedge each other through a single global egress.
Feature-flagged (FEATURE_BROWSER_POOL_ENABLED, default OFF): with the flag off the
/fetch and /login paths are byte-for-byte the existing single-browser behavior.
When on, /fetch routes by body["source"] to a per-proxy browser via BROWSER_PROXY_MAP
(BROWSER_PROXY_AVITO/CIAN/YANDEX/DOMCLICK with legacy fallbacks), each guarded by its
own lazy-launched lock. /login stays single-browser in Phase 1.
BrowserFetcher gains a source arg (default avito) and sends it in the /fetch body;
all scraper callsites pass their source. No docker-compose/.env.runtime changes
(Phase 2, owner-gated).
- scheduler.py: исправлен stale-комментарий yandex_detail_backfill
(был: «YandexDetailScraper httpx, no proxy layer»;
теперь: fetch via curl_cffi chrome120 + scraper_proxy_url,
parse via YandexDetailScraper.parse)
- yandex_detail_backfill.py: outer except — сообщение лога изменено на
«save/iteration error for listing_id=%d» чтобы отличать save-ошибку
от fetch/parse-None путей (run_id убран — он менее важен в этом scope)
- New task app/tasks/avito_detail_backfill.py with run_avito_detail_backfill()
* Single snapshot SELECT at start (guarantees termination)
* Same proxy/AsyncSession path as scrape_pipeline.py step 5
* Budget guard (budget_sec), consecutive block abort (mark_done not mark_failed)
* rotate_ip() on every AvitoBlockedError; rollback on generic Exception (#1368)
* start = time.monotonic() initialized before try so except can reference it
- Scheduler wiring: trigger_avito_detail_backfill_run() + elif in scheduler_loop()
- Migration 112: scrape_schedules INSERT window 09-12 UTC, batch_size=800,
budget_sec=3600, request_delay_sec=6, max_consecutive_blocks=5
- 7 unit tests (no pytest-mock, unittest.mock only): all 7 passing,
full CI suite 1809 passed
Add geocode_tried_at = NOW() to the success UPDATE SET clause so
successfully-geocoded listings record their attempt time, matching
the per-ID admin endpoint contract and the column spec.
Регистрирует отдельный in-app scheduler-источник для cian_newbuilding
enrichment-backfill (houses_price_dynamics / house_reliability_checks /
house_reviews), который раньше бежал только инлайн в Cian full-sweep.
- run_newbuilding_enrich() wrapper (heartbeat → backfill → mark_done/failed),
делегирует backfill_newbuilding_enrichment (#972, уже в main).
- trigger_newbuilding_enrich_run() + dispatch-ветка в scheduler_loop
(зеркалит trigger_sber_index_pull_run): _claim_run guard, async task,
zombie-reap наследуется; idempotency наследуется (skip уже обогащённых,
per-house SAVEPOINT, bounded per-fire limit дренирует backlog 306 домов).
- seed-миграция 103: source='newbuilding_enrich', окно 00:00-01:00 UTC
(03:00-04:00 МСК, до 01:00+ UTC sweep-блока), limit=25, next_run_at на завтра.
Засеяно enabled=FALSE (dormant): на prod весь external-HTTP scraping намеренно
на паузе. Расписание+триггер готовы, но не стартуют до намеренного возобновления
(UPDATE ... SET enabled=true). Не активирую одиночный источник при общей паузе.
Closes#973
The legacy resolve_cian_zhk_url hit /zhk/<id>/ which now 404s, leaving the
318 geo-matched cian houses (house_sources.ext_id set, cian_zhk_url NULL)
unfetchable -> the newbuilding-enrichment backfill matched 0 houses.
Add resolve_cian_zhk_url_via_search(nb_id): fetch the cat.php newbuilding SERP
and extract the canonical zhk-<slug>.cian.ru url, ANCHORED on the
<h1 data-name="Title"> header anchor (NOT naive first-match — the SERP carries
promo/recommendation zhk-* links before the title that would otherwise resolve
the wrong ЖК and silently corrupt enrichment). Validated against the real
2.81MB prod SERP + an adversarial poisoned-recommendation test.
Wire into newbuilding_enrich_backfill: broaden selection to "has ext_id OR
cian_zhk_url", resolve+persist the url under a SAVEPOINT before enriching,
rate-limited + resumable + idempotent. Keep the old resolver (deprecated).
Bounded prod proof (5 houses, direct): 5/5 urls resolved+persisted, 3/5 fully
enriched (+18 price_dynamics, +3 reliability); the 2 misses were direct-mode
anti-bot on the 2nd fetch. Full 318-run gated on the cian mobile proxy
(mproxy.site) being restored. code-reviewer APPROVE (SQL/idempotency) +
resolver hardened against wrong-ЖК. 19 tests green.
Refs #972.
950-E1: app/tasks/newbuilding_enrich_backfill.py selects houses linked to
ext_source='cian_newbuilding' with a fetchable cian_zhk_url and runs the existing
fetch_newbuilding + save_newbuilding_enrichment over each, populating
houses_price_dynamics + house_reliability_checks (+ house_reviews when present).
- Idempotent + resumable: skips already-enriched houses; force= re-run dup-safe
(price_dynamics UPSERT via dim_key, reliability dedup guard with id-DESC
tiebreaker, reviews ON CONFLICT (source,ext_review_id)).
- Anti-bot aware: get_scraper_delay('cian') + jitter, SAVEPOINT-per-house
(one captcha/failure logs + continues, never aborts the batch).
- limit= for bounded proof runs; sizing counters (total/fetchable/pending).
Fix cian_newbuilding._extract_transport_rate: cian transportAccessibilityRate
drifted to a nested dict and broke save_newbuilding_enrichment's CAST bind on
LIVE data (cannot adapt type 'dict') — coerce dict/float/bad -> int|None
(bool excluded before int). Fixes the live enrichment crash, not just backfill.
house_reviews stays ~0 for now: cian reviews are in a separate XHR, not the ЖК
initialState — extending fetch_newbuilding for them is a documented follow-up.
Tested: 8 new unit tests; isolated-DB proof-run landed 6 price_dynamics + 1
reliability row, idempotency proven across re-runs. Refs #972.
Закрывает последнюю часть #769 (A1/A2/B/C/D в #798/#804). Finding #17:
bare-city адреса геокодились в city-центроид и сохранялись как точные lat/lon →
mislocated листинги участвовали в radius ST_DWithin-аналогах.
- 089: ALTER TABLE listings ADD COLUMN IF NOT EXISTS geo_precision text (idempot).
- ScrapedLot.geo_precision + проброс в save_listings.
- geocode_missing.py: geo_precision='city' через существующий _geocode_is_coarse.
- estimator _fetch_analogs Tier H+W: + AND (geo_precision IS DISTINCT FROM 'city').
NULL проходит (консервативно, сдвига до backfill нет). Pricing не тронут.
7 тестов; 270 passed, ruff clean.
