Adds since=date mode to fetch_all_secondary: per room x seed-bracket, paginate
sequentially newest-first and stop once cards fall below the watermark, skipping
deep pagination (and the 403 clusters it triggers). since=None keeps the exact
exhaustive bisection walk. No scheduler/schedule change here — wiring follows
in a separate PR.
PR #1764's reconnect-retry was dead code in prod: the gate excluded configs
with avito_proxy_rotate_url set, but prod has BOTH the backconnect SCRAPER_PROXY_URL
(what curl_cffi egresses through) AND a leftover changeip URL for the separate
auv/browser proxy. With max_rotations=0 neither path fired -> a single SERP 403
still aborted the sweep (run 212). The curl session always uses scraper_proxy_url,
so session-rebuild rotates the exit IP regardless of changeip config.
Under a backconnect rotating proxy (no changeip URL) a single HTTP 403 or
firewall interstitial aborted the entire avito_full_load sweep via
AvitoBlockedError -> mark_banned. Each new connection through backconnect
gets a fresh exit IP, so retry by rebuilding the curl_cffi session up to
_AVITO_403_MAX_RETRIES times before giving up. Legacy single-IP changeip
rotation and browser-mode paths unchanged. Mirrors existing 429 short-retry.
Подключает prod-проверенный run_cian_full_load (exhaustive региональный сбор
Cian ЕКБ вторички, room×price партиционирование, incremental on_bucket save) в
in-app scheduler как recurring-источник cian_full_load (окно 20-22 UTC — отделено
от cian_city_sweep/history_backfill на 2-5 чтобы не конкурировать на shared
kf-прокси).
run_cian_city_sweep теперь NB-only (newbuilding_only=True default): SERP-фаза
сохраняет только novostroyki-лоты, вторичку отбрасывает — ею авторитетно владеет
full_load. Убирает дубль secondary SERP-сейва. DETAIL/HOUSES-фазы не затронуты.
The chunked matcher paged the candidate set with OFFSET over the very predicate
it mutates (building_cadastral_number IS NULL). Matched rows drop out, so OFFSET
(reset to 0 OR advanced) skips un-processed rows: once >= batch_size unmatched
listings (beyond threshold) accumulate at the low-id front, the chunk matches 0
and the loop breaks early, leaving higher-id matchable listings unfilled.
Counterexample (batch=3, matchable ids 1,4,7,10): matches 1 and 4 then breaks,
missing 7 and 10. Unit tests used single-chunk fixtures so never caught it.
The full UPDATE is ~5s for ~41k listings, so replace the loop with ONE set-based
LATERAL KNN UPDATE over all candidates — correct and fast. batch_size kept in the
signature for schedule-param compat (now unused).
Backfill listings.building_cadastral_number (was 0%) from the nearest
cadastral building. gendesign_cad_buildings is a postgres_fdw foreign table
with no geom — a per-listing FDW nearest query is ~1.16s/row (~13h for 43k
listings). Instead materialize the FDW once into a LOCAL cad_buildings_local
table (Point geom + GIST), then run a fast local KNN nearest-neighbour join.
Perf: the distance gate uses geometry-space ST_DWithin(geom, point, deg) (GIST
index, no geography cast) + geom <-> KNN order + ST_DistanceSphere metric
recheck on the single nearest row. A geography-cast ST_DWithin in the WHERE
defeated the index (58s+, full update never finished); the geometry-gate design
does the whole ~41.9k-listing UPDATE in ~5.4s (refresh+match ~7s end-to-end),
matching 16027 listings at 50m across 2229 distinct buildings.
The match is GEO-NEAREST (approximate): a street-level-geocoded listing matches
the nearest building within threshold_m (default 50m), not necessarily its exact
cadastral building. Exact cadastral + parcel-containment deferred (cad_parcels
FDW not exposed). Threshold is logged.
- 124: cad_buildings_local table (empty) + GIST. Migration does not read the FDW
(deploy-independent); populated by the refresh job.
- 125: scrape_schedules seed source=cadastral_geo_match, enabled, next_run_at
tomorrow 09:00 UTC (after geocode_missing).
- tasks/cadastral_geo_match.py: refresh_cad_buildings_local (bulk FDW scan),
match_listings_to_buildings (chunked LATERAL KNN UPDATE), run_cadastral_geo_match
run-lifecycle wrapper.
- scheduler: trigger_cadastral_geo_match_run + dispatch (sync DB-only, executor).
Route each scraper source (avito/cian/yandex/domclick) to its own camoufox
browser+proxy so they no longer wedge each other through a single global egress.
Feature-flagged (FEATURE_BROWSER_POOL_ENABLED, default OFF): with the flag off the
/fetch and /login paths are byte-for-byte the existing single-browser behavior.
When on, /fetch routes by body["source"] to a per-proxy browser via BROWSER_PROXY_MAP
(BROWSER_PROXY_AVITO/CIAN/YANDEX/DOMCLICK with legacy fallbacks), each guarded by its
own lazy-launched lock. /login stays single-browser in Phase 1.
BrowserFetcher gains a source arg (default avito) and sends it in the /fetch body;
all scraper callsites pass their source. No docker-compose/.env.runtime changes
(Phase 2, owner-gated).
Gate-API nests Yandex's per-offer valuation under predictions.predictedPrice,
not predictedPrice.predictedPrice (that key does not exist). _predicted_prices
read the wrong outer key, so predicted_price_rub/_min/_max were None for every
offer (prod: pp=0 across all 6189 yandex rows). Verified live against the
gate-API 2026-06-18.
The rich-field test fixture used the same wrong wrapper, so the test passed
vacuously. Updated _ENTITY_RICH_OWNER to the real predictions.predictedPrice
shape — with the old code the assertions now fail, confirming non-vacuity.
Replace the system-curl SOCKS5 subprocess transport with the shared
BrowserFetcher (camoufox). camoufox's real-browser fingerprint is not
tarpitted by yandex and does not burn the mobile-proxy IP; raw system-curl
got the proxy IP flagged.
One BrowserFetcher is opened per scraper session in __aenter__ and reused
across all page fetches. _http_get now fetches via the browser, extracts the
gate-API JSON from camoufox's HTML <pre> wrapper, and returns it as the body
of a _CurlResponse (status 200 on success, 0 on fetch/extraction failure) so
all existing callers and the tarpit retry/rotate logic keep working unchanged.
URL building, _parse_gate_json, pagination, combos and field mapping are
untouched. Removed the dead curl subprocess constants/method.
097_index_hygiene.sql de-partialized 5 listings indexes on the false premise
that is_active=100% true. 104_index_hygiene_geom_dedup.sql acknowledged the
premise is stale (49% active on prod, 2026-06-13) but only cleaned up the
redundant geom duplicate, leaving 3 composite indexes full.
Restore WHERE is_active=true on the composite hot-path indexes:
- listings_active_filter_idx (rooms, price_rub, area_m2, scraped_at DESC)
- listings_house_price_idx (house_id_fk, price_rub)
- listings_rooms_area_idx (rooms, area_m2)
Excluded: listings_geom_active_idx (dropped by 104, listings_geom_idx covers it),
listings_scraped_desc_idx (not in issue #1398 suggested fix scope).
CONCURRENTLY not used — migration runner wraps in BEGIN/COMMIT (see 117 note).
Replace absolute min/max with robust percentiles so a single outlier ДКП
deal cannot shift the soft-bound corridor boundaries. Small samples fall
back gracefully via linear interpolation (n=3: P10≈index 0.2, P90≈index 2.8).
