The new geoportal house-match tier runs first in geocode(); this legacy
cadastral test must mock it to None so it still exercises the cadastral
forward path (otherwise the MagicMock db row leaks through the new tier).
The Cian Valuation enrichment was the only ungated external call in
estimate_quality() — its internal curl timeout (~25s) blocks the whole
/estimate request on every cache-miss, intermittently pushing latency
past the gateway/patience window (perceived as 'сервис не считает').
Wrap it in _with_budget() (mirrors the existing Yandex-valuation guard):
on timeout it degrades to None — the same graceful path as a network
error — instead of stalling. New setting estimate_cian_valuation_timeout_s
(default 8.0s, env ESTIMATE_CIAN_VALUATION_TIMEOUT_S).
Visible warn-banner trailing sentence no longer names the brand.
Code comments in the file docstring updated to be brand-neutral.
No logic or behaviour changed.
Tier A (same building) dropped cian listings tagged listing_segment='novostroyki'
even in delivered buildings where that tag marks owner resales/переуступки
(sale_type=free). Gated relaxation of the #1186 guard, Tier A ONLY: include
novostroyki rows iff the same house has secondary (vtorichka/NULL) listings AND
secondary_count >= primary_count — secondary-dominant signal of a delivered house.
Prevents a lone vtorichka + clustered developer-priced novostroyki from anchoring
on застройщик prices after MAD-clip drops the rare secondary (~27% overvaluation).
Pure-primary / primary-dominated houses keep the #1186 guard. Behind
estimate_sb_tier_a_allow_primary_if_secondary_present (default True).
Dedup duplicate active rows by (source, source_id) primary key (codebase canon,
namespaced id/url tags, source_url fallback) — collapses rows whose source_url
differs only by trailing slash / query-param (e.g. cian source_id=330047129 ×2).
Tier C / radius / asking_to_sold_ratio paths untouched. 84 unit tests pass.
Avito leaks addresses with house number glued to district marker without
separator («…Савкова, 29р-н Академический»), breaking house-number extraction
and same-house matching (41 unmatched avito listings on one street).
_clean_address now inserts ", " between a house-number token (digits +
optional Cyrillic letter + optional /N|сN|кN|корпус|стр) and a following
marker (р-н, мкр, район, г., пос, снт, …).
- avito.py: idempotent _deglue_house_marker step (CSS/tail-strip kept)
- tests: de-glue positive/negative/idempotency (24 passed)
- data/sql/124_deglue_avito_addresses.sql: idempotent backfill for stored rows
normalize.py НЕ тронут (общий fingerprint матчинга, #1534).
Закрывает avito-часть #1773 (cian-без-номера + дедуп — отдельно, зона cian).
Detail backfill (run 217) hit 5 consecutive HTTP 429s and aborted at 19 enriched
because the detail fetcher only had a same-session 429 short-retry (no reconnect).
429s are persistent: keep-alive reuses one backconnect exit IP across sequential
detail fetches -> avito rate-limits it. Mirror the merged SERP 429 fix (#1769):
widen short-retry 4->8 and add up to 3 ephemeral-session reconnects (fresh exit IP)
on 429 exhaustion before raising. Caller session never touched (as with 403 path).
scrape_schedules had no interval column -> every source ran daily. Add
interval_days from default_params (default 1, backward-compatible) so
compute_next_run_at can schedule N days ahead. Set avito_full_load_exhaustive
to interval_days=7 (weekly full pass for last_seen/delisting + silent price
edits); avito_full_load stays daily incremental.
run_avito_full_load gains incremental_days param -> passes since= to the
(merged) incremental SERP engine. avito_full_load schedule flipped to
incremental_days=2 (shallow, date early-stop -> avoids deep-pagination 429
bans). New avito_full_load_exhaustive source runs the full walk weekly to
refresh last_seen (10-day delisting TTL) and catch silent price edits.
Run 216 recovered 403s (merged fix) but still banned on a 429 at page 36
(deep pagination, backconnect conn-limit). Widen the 429 retry budget + one
reconnect on exhaustion in _fetch_serp_html, and make fetch_all_secondary skip
a blocked bucket and continue rather than aborting the run; only N consecutive
blocked buckets (real hard ban) re-raises. Preserves hard-ban detection.
fetch_detail raised on the first 403/429, aborting the detail backfill (run 191
enriched only 48 then zombied) -> avito detail coverage stuck at 14% of active.
Mirror the merged SERP fix (#1765): on 403/firewall under the backconnect proxy,
rebuild an ephemeral session (fresh exit IP) and retry up to N before raising;
short-retry 429. Never close/rebuild the caller-provided shared session. Also
add the missing HTTP-200 firewall-interstitial check on the curl detail path.
Adds listings.card_hash (SHA-256 of volatile SERP-card fields) computed and
stored on every scrape. Skips the redundant per-day listings_snapshots write
when a listing's card is unchanged. listings.last_seen_at/is_active still
bumped on every sighting (delisting TTL). Lays groundwork for detail-refresh
gating (C2).
Adds since=date mode to fetch_all_secondary: per room x seed-bracket, paginate
sequentially newest-first and stop once cards fall below the watermark, skipping
deep pagination (and the 403 clusters it triggers). since=None keeps the exact
exhaustive bisection walk. No scheduler/schedule change here — wiring follows
in a separate PR.
PR #1764's reconnect-retry was dead code in prod: the gate excluded configs
with avito_proxy_rotate_url set, but prod has BOTH the backconnect SCRAPER_PROXY_URL
(what curl_cffi egresses through) AND a leftover changeip URL for the separate
auv/browser proxy. With max_rotations=0 neither path fired -> a single SERP 403
still aborted the sweep (run 212). The curl session always uses scraper_proxy_url,
so session-rebuild rotates the exit IP regardless of changeip config.
Under a backconnect rotating proxy (no changeip URL) a single HTTP 403 or
firewall interstitial aborted the entire avito_full_load sweep via
AvitoBlockedError -> mark_banned. Each new connection through backconnect
gets a fresh exit IP, so retry by rebuilding the curl_cffi session up to
_AVITO_403_MAX_RETRIES times before giving up. Legacy single-IP changeip
rotation and browser-mode paths unchanged. Mirrors existing 429 short-retry.
Подключает prod-проверенный run_cian_full_load (exhaustive региональный сбор
Cian ЕКБ вторички, room×price партиционирование, incremental on_bucket save) в
in-app scheduler как recurring-источник cian_full_load (окно 20-22 UTC — отделено
от cian_city_sweep/history_backfill на 2-5 чтобы не конкурировать на shared
kf-прокси).
run_cian_city_sweep теперь NB-only (newbuilding_only=True default): SERP-фаза
сохраняет только novostroyki-лоты, вторичку отбрасывает — ею авторитетно владеет
full_load. Убирает дубль secondary SERP-сейва. DETAIL/HOUSES-фазы не затронуты.
The chunked matcher paged the candidate set with OFFSET over the very predicate
it mutates (building_cadastral_number IS NULL). Matched rows drop out, so OFFSET
(reset to 0 OR advanced) skips un-processed rows: once >= batch_size unmatched
listings (beyond threshold) accumulate at the low-id front, the chunk matches 0
and the loop breaks early, leaving higher-id matchable listings unfilled.
Counterexample (batch=3, matchable ids 1,4,7,10): matches 1 and 4 then breaks,
missing 7 and 10. Unit tests used single-chunk fixtures so never caught it.
The full UPDATE is ~5s for ~41k listings, so replace the loop with ONE set-based
LATERAL KNN UPDATE over all candidates — correct and fast. batch_size kept in the
signature for schedule-param compat (now unused).
Backfill listings.building_cadastral_number (was 0%) from the nearest
cadastral building. gendesign_cad_buildings is a postgres_fdw foreign table
with no geom — a per-listing FDW nearest query is ~1.16s/row (~13h for 43k
listings). Instead materialize the FDW once into a LOCAL cad_buildings_local
table (Point geom + GIST), then run a fast local KNN nearest-neighbour join.
Perf: the distance gate uses geometry-space ST_DWithin(geom, point, deg) (GIST
index, no geography cast) + geom <-> KNN order + ST_DistanceSphere metric
recheck on the single nearest row. A geography-cast ST_DWithin in the WHERE
defeated the index (58s+, full update never finished); the geometry-gate design
does the whole ~41.9k-listing UPDATE in ~5.4s (refresh+match ~7s end-to-end),
matching 16027 listings at 50m across 2229 distinct buildings.
The match is GEO-NEAREST (approximate): a street-level-geocoded listing matches
the nearest building within threshold_m (default 50m), not necessarily its exact
cadastral building. Exact cadastral + parcel-containment deferred (cad_parcels
FDW not exposed). Threshold is logged.
- 124: cad_buildings_local table (empty) + GIST. Migration does not read the FDW
(deploy-independent); populated by the refresh job.
- 125: scrape_schedules seed source=cadastral_geo_match, enabled, next_run_at
tomorrow 09:00 UTC (after geocode_missing).
- tasks/cadastral_geo_match.py: refresh_cad_buildings_local (bulk FDW scan),
match_listings_to_buildings (chunked LATERAL KNN UPDATE), run_cadastral_geo_match
run-lifecycle wrapper.
- scheduler: trigger_cadastral_geo_match_run + dispatch (sync DB-only, executor).
Route each scraper source (avito/cian/yandex/domclick) to its own camoufox
browser+proxy so they no longer wedge each other through a single global egress.
Feature-flagged (FEATURE_BROWSER_POOL_ENABLED, default OFF): with the flag off the
/fetch and /login paths are byte-for-byte the existing single-browser behavior.
When on, /fetch routes by body["source"] to a per-proxy browser via BROWSER_PROXY_MAP
(BROWSER_PROXY_AVITO/CIAN/YANDEX/DOMCLICK with legacy fallbacks), each guarded by its
own lazy-launched lock. /login stays single-browser in Phase 1.
BrowserFetcher gains a source arg (default avito) and sends it in the /fetch body;
all scraper callsites pass their source. No docker-compose/.env.runtime changes
(Phase 2, owner-gated).
Gate-API nests Yandex's per-offer valuation under predictions.predictedPrice,
not predictedPrice.predictedPrice (that key does not exist). _predicted_prices
read the wrong outer key, so predicted_price_rub/_min/_max were None for every
offer (prod: pp=0 across all 6189 yandex rows). Verified live against the
gate-API 2026-06-18.
The rich-field test fixture used the same wrong wrapper, so the test passed
vacuously. Updated _ENTITY_RICH_OWNER to the real predictions.predictedPrice
shape — with the old code the assertions now fail, confirming non-vacuity.
