468 commits

Author	SHA1	Message	Date
bot-backend	9dc3be56e3	fix(avito): reconnect-retry on SERP 403/firewall under backconnect proxy ... Under a backconnect rotating proxy (no changeip URL) a single HTTP 403 or firewall interstitial aborted the entire avito_full_load sweep via AvitoBlockedError -> mark_banned. Each new connection through backconnect gets a fresh exit IP, so retry by rebuilding the curl_cffi session up to _AVITO_403_MAX_RETRIES times before giving up. Legacy single-IP changeip rotation and browser-mode paths unchanged. Mirrors existing 429 short-retry.	2026-06-18 20:41:32 +03:00
bot-backend	6454aa312d	fix(tradein): avito _extract_total_count returns 0 on no-results page (stop empty-bracket split waste + fallback-listing scrape)	2026-06-18 20:01:41 +03:00
bot-backend	2c6d0e4c71	feat(tradein): avito curl_cffi 429 short-retry (backconnect conn-limit transient, no rotate/ban)	2026-06-18 19:24:05 +03:00
bot-backend	1533baad97	perf(browser): block image/media/font resources in camoufox fetch (~10x smaller pages, faster via proxy)	2026-06-18 18:16:53 +03:00
bot-backend	ffecc6962d	perf(tradein): shared EKB seed price-brackets across avito/cian/yandex (city-dict ready for multi-city)	2026-06-18 17:55:55 +03:00
bot-backend	3f85fa59ff	perf(tradein): avito seed price-brackets with open top bucket (≤2k blocks, no price ceiling)	2026-06-18 17:29:31 +03:00
bot-backend	5c37fcbfad	fix(tradein): praktika brand only under praktika account — drop kopylov mapping (brand leak)	2026-06-18 16:29:18 +03:00
bot-backend	cbeeb9c414	fix(tradein): avito full-load probe resilience — browser 500/timeout splits/skips instead of crashing run	2026-06-18 16:03:39 +03:00
lekss361	493142b990	Merge pull request 'feat(tradein): populate offer_price_history for yandex from gate price.previous/trend' (#1754) from feat/yandex-price-history into main	2026-06-18 11:37:05 +00:00
bot-backend	6c132d7f0a	fix(tradein): avito sweep page-level retry — transient page error no longer aborts run	2026-06-18 14:29:01 +03:00
bot-backend	6c82528db5	feat(tradein): populate offer_price_history for yandex from gate price.previous/trend	2026-06-18 14:23:23 +03:00
bot-backend	1ddbf867f8	feat(tradein): avito exhaustive full-load (room×price bisection) — mirror cian	2026-06-18 13:55:18 +03:00
bot-backend	158a24c25b	feat(tradein): schedule cian_full_load (exhaustive secondary) + anchor sweep NB-only ... Подключает prod-проверенный run_cian_full_load (exhaustive региональный сбор Cian ЕКБ вторички, room×price партиционирование, incremental on_bucket save) в in-app scheduler как recurring-источник cian_full_load (окно 20-22 UTC — отделено от cian_city_sweep/history_backfill на 2-5 чтобы не конкурировать на shared kf-прокси). run_cian_city_sweep теперь NB-only (newbuilding_only=True default): SERP-фаза сохраняет только novostroyki-лоты, вторичку отбрасывает — ею авторитетно владеет full_load. Убирает дубль secondary SERP-сейва. DETAIL/HOUSES-фазы не затронуты.	2026-06-18 13:25:23 +03:00
lekss361	48bedb5f65	Merge branch 'main' into feat/cadastral-geo-match	2026-06-18 08:44:06 +00:00
bot-backend	f9769571a3	fix(cadastral): single-statement match UPDATE — OFFSET chunk skipped rows ... The chunked matcher paged the candidate set with OFFSET over the very predicate it mutates (building_cadastral_number IS NULL). Matched rows drop out, so OFFSET (reset to 0 OR advanced) skips un-processed rows: once >= batch_size unmatched listings (beyond threshold) accumulate at the low-id front, the chunk matches 0 and the loop breaks early, leaving higher-id matchable listings unfilled. Counterexample (batch=3, matchable ids 1,4,7,10): matches 1 and 4 then breaks, missing 7 and 10. Unit tests used single-chunk fixtures so never caught it. The full UPDATE is ~5s for ~41k listings, so replace the loop with ONE set-based LATERAL KNN UPDATE over all candidates — correct and fast. batch_size kept in the signature for schedule-param compat (now unused).	2026-06-18 10:28:19 +03:00
bot-backend	edd187b47f	feat(cadastral): geo-nearest building matcher via local cad mirror ... Backfill listings.building_cadastral_number (was 0%) from the nearest cadastral building. gendesign_cad_buildings is a postgres_fdw foreign table with no geom — a per-listing FDW nearest query is ~1.16s/row (~13h for 43k listings). Instead materialize the FDW once into a LOCAL cad_buildings_local table (Point geom + GIST), then run a fast local KNN nearest-neighbour join. Perf: the distance gate uses geometry-space ST_DWithin(geom, point, deg) (GIST index, no geography cast) + geom <-> KNN order + ST_DistanceSphere metric recheck on the single nearest row. A geography-cast ST_DWithin in the WHERE defeated the index (58s+, full update never finished); the geometry-gate design does the whole ~41.9k-listing UPDATE in ~5.4s (refresh+match ~7s end-to-end), matching 16027 listings at 50m across 2229 distinct buildings. The match is GEO-NEAREST (approximate): a street-level-geocoded listing matches the nearest building within threshold_m (default 50m), not necessarily its exact cadastral building. Exact cadastral + parcel-containment deferred (cad_parcels FDW not exposed). Threshold is logged. - 124: cad_buildings_local table (empty) + GIST. Migration does not read the FDW (deploy-independent); populated by the refresh job. - 125: scrape_schedules seed source=cadastral_geo_match, enabled, next_run_at tomorrow 09:00 UTC (after geocode_missing). - tasks/cadastral_geo_match.py: refresh_cad_buildings_local (bulk FDW scan), match_listings_to_buildings (chunked LATERAL KNN UPDATE), run_cadastral_geo_match run-lifecycle wrapper. - scheduler: trigger_cadastral_geo_match_run + dispatch (sync DB-only, executor).	2026-06-18 10:18:54 +03:00
bot-backend	1a658a65a6	feat(avito): dedicated newbuilding (novostroyka) citywide sweep	2026-06-18 09:45:54 +03:00
bot-backend	b9eb478c8f	feat(browser): per-source proxy pool behind FEATURE_BROWSER_POOL_ENABLED (Phase 1) ... Route each scraper source (avito/cian/yandex/domclick) to its own camoufox browser+proxy so they no longer wedge each other through a single global egress. Feature-flagged (FEATURE_BROWSER_POOL_ENABLED, default OFF): with the flag off the /fetch and /login paths are byte-for-byte the existing single-browser behavior. When on, /fetch routes by body["source"] to a per-proxy browser via BROWSER_PROXY_MAP (BROWSER_PROXY_AVITO/CIAN/YANDEX/DOMCLICK with legacy fallbacks), each guarded by its own lazy-launched lock. /login stays single-browser in Phase 1. BrowserFetcher gains a source arg (default avito) and sends it in the /fetch body; all scraper callsites pass their source. No docker-compose/.env.runtime changes (Phase 2, owner-gated).	2026-06-18 09:01:50 +03:00
lekss361	635851a71c	Merge pull request 'fix(yandex): read predicted price from entity.predictions.predictedPrice (#1725)' (#1734) from fix/yandex-predicted-price-path-1725 into main ... Reviewed-on: #1734	2026-06-18 05:12:25 +00:00
bot-backend	41f2402b25	fix(yandex): read predicted price from entity.predictions.predictedPrice (#1725) ... Gate-API nests Yandex's per-offer valuation under predictions.predictedPrice, not predictedPrice.predictedPrice (that key does not exist). _predicted_prices read the wrong outer key, so predicted_price_rub/_min/_max were None for every offer (prod: pp=0 across all 6189 yandex rows). Verified live against the gate-API 2026-06-18. The rich-field test fixture used the same wrong wrapper, so the test passed vacuously. Updated _ENTITY_RICH_OWNER to the real predictions.predictedPrice shape — with the old code the assertions now fail, confirming non-vacuity.	2026-06-18 08:03:05 +03:00
lekss361	8f9c11a45f	Merge pull request 'feat(browser): concurrent page pool (BROWSER_CONCURRENCY, default 4) + raise mem_limit to 5g' (#1733) from feat/browser-concurrent-pool into main	2026-06-17 21:07:31 +00:00
bot-backend	a6717705b9	fix(browser): reset page_counter on recycle + close login page in finally (#796 review)	2026-06-18 00:06:50 +03:00
bot-backend	820fd819bb	feat(browser): concurrent page pool (BROWSER_CONCURRENCY, default 4) + raise mem_limit to 5g	2026-06-17 23:55:11 +03:00
bot-backend	73fa739e0f	fix(backfill): drop nonexistent kadastr_num column from listings SELECT	2026-06-17 23:47:30 +03:00
bot-backend	687c9e5e9d	fix(avito): proactive IP rotate at sweep start + raise max_rotations 2->4 (Datadome ban recovery)	2026-06-17 23:35:24 +03:00
lekss361	84902984e3	Merge pull request 'feat(domclick): wire DomClick citywide sweep into pipeline+scheduler (seeded disabled)' (#1730) from feat/activate-domclick-sweep into main	2026-06-17 20:29:43 +00:00
lekss361	07eed4b55e	Merge pull request 'feat(avito): capture newbuilding ЖК id/url from SERP card' (#1729) from feat/avito-newbuilding-capture into main	2026-06-17 20:29:39 +00:00
bot-backend	2ed3ae81bc	feat(domclick): wire DomClick citywide sweep into pipeline+scheduler (seeded disabled)	2026-06-17 23:21:49 +03:00
bot-backend	dd59ef0516	feat(avito): capture newbuilding ЖК id/url from SERP card; persist newbuilding_id/url	2026-06-17 23:15:35 +03:00
bot-backend	e63b21eccb	fix(scrapers): write resolved house_id to listings.house_id_fk in realtime link hook	2026-06-17 23:07:22 +03:00
lekss361	97fce43484	Merge pull request 'feat(yandex): map rich entity fields (predictedPrice, seller, listing_date, trend, metro)' (#1725) from feat/yandex-rich-fields into main ... Reviewed-on: #1725	2026-06-17 20:03:25 +00:00
lekss361	df83e0256b	Merge pull request 'fix(tradein): correct SberIndex hedonic/asking dataset filter dims + 404 warning (#902)' (#1724) from fix/sber-dataset-paths-902 into main	2026-06-17 19:58:48 +00:00
bot-backend	1fe670ea6a	feat(yandex): map rich entity fields (predictedPrice, seller, listing_date, trend, metro)	2026-06-17 22:58:16 +03:00
bot-backend	5ca987fcd3	fix(tradein): correct SberIndex hedonic/asking dataset-paths + 404 warning (#902)	2026-06-17 22:50:52 +03:00
lekss361	f25e89767b	Merge pull request 'fix(matching): conflict_resolution picks freshest (last_seen_at) value, deterministic (#1539)' (#1722) from fix/conflict-resolution-freshness-1539 into main	2026-06-17 19:49:51 +00:00
bot-backend	f30f6acbf7	fix(matching): conflict_resolution picks freshest (last_seen_at) value, deterministic (#1539)	2026-06-17 22:43:59 +03:00
lekss361	45d8e9e4a5	Merge pull request 'fix(cian): persist detail rich fields (description) + stamp detail_enriched_at' (#1718) from fix/cian-detail-rich-fields into main	2026-06-17 19:42:58 +00:00
lekss361	5b8ee994d5	Merge pull request 'feat(tradein): segment=vtorichka|novostroyki|all on listings search (#1188)' (#1710) from feat/listings-segment-param-1188 into main	2026-06-17 19:34:50 +00:00
lekss361	931a57ac59	Merge pull request 'feat(yandex): ingest novostroyki via gate-API newFlat=YES' (#1715) from feat/yandex-novostroyki-newflat into main	2026-06-17 19:34:32 +00:00
bot-backend	fe805ea528	fix(cian): persist detail rich fields (description/kitchen/ceiling) + stamp detail_enriched_at	2026-06-17 22:32:18 +03:00
bot-backend	22a7a3f086	fix(avito): classify novostroyki by item-development-name marker; map переуступка sale_type	2026-06-17 22:24:13 +03:00
bot-backend	bc4c6852a1	feat(yandex): ingest novostroyki via gate-API newFlat=YES	2026-06-17 22:22:17 +03:00
bot-backend	b64334d7bd	feat(tradein): segment=vtorichka|novostroyki|all query-param on listings (#1188)	2026-06-17 21:43:01 +03:00
lekss361	16990af11c	Merge pull request 'docs(estimator): DKP corridor described as P10/P90 not min/max in schema+TS (#1520 follow-up)' (#1708) from fix/dkp-corridor-doc-p10p90-1695 into main	2026-06-17 18:42:10 +00:00
bot-backend	f1c5056397	docs(estimator): DKP corridor described as P10/P90 not min/max in schema+TS (#1520 follow-up)	2026-06-17 21:22:49 +03:00
bot-backend	a8d7be68ad	feat(yandex): route gate-API fetch through camoufox BrowserFetcher ... Replace the system-curl SOCKS5 subprocess transport with the shared BrowserFetcher (camoufox). camoufox's real-browser fingerprint is not tarpitted by yandex and does not burn the mobile-proxy IP; raw system-curl got the proxy IP flagged. One BrowserFetcher is opened per scraper session in __aenter__ and reused across all page fetches. _http_get now fetches via the browser, extracts the gate-API JSON from camoufox's HTML <pre> wrapper, and returns it as the body of a _CurlResponse (status 200 on success, 0 on fetch/extraction failure) so all existing callers and the tarpit retry/rotate logic keep working unchanged. URL building, _parse_gate_json, pagination, combos and field mapping are untouched. Removed the dead curl subprocess constants/method.	2026-06-17 21:16:48 +03:00
lekss361	262adbf762	Merge pull request 'fix(estimator): DKP corridor uses P10/P90 not min/max (#1520)' (#1695) from fix/dkp-corridor-percentile-1520 into main	2026-06-17 18:05:39 +00:00
lekss361	1fffb7a0a7	Merge pull request 'fix(sql): restore partial WHERE is_active predicate on listings indexes (#1398)' (#1693) from fix/listings-partial-indexes-1398 into main	2026-06-17 18:05:30 +00:00
lekss361	5c44233ed6	Merge pull request 'fix(scrapers): cian _walk_price_range stop silently truncating tail (#1393)' (#1689) from fix/cian-paginate-tail-1393 into main	2026-06-17 17:55:36 +00:00
bot-backend	403020ad56	fix(sql): restore partial predicate on 5 listings indexes (#1398) ... 097_index_hygiene.sql de-partialized 5 listings indexes on the false premise that is_active=100% true. 104_index_hygiene_geom_dedup.sql acknowledged the premise is stale (49% active on prod, 2026-06-13) but only cleaned up the redundant geom duplicate, leaving 3 composite indexes full. Restore WHERE is_active=true on the composite hot-path indexes: - listings_active_filter_idx (rooms, price_rub, area_m2, scraped_at DESC) - listings_house_price_idx (house_id_fk, price_rub) - listings_rooms_area_idx (rooms, area_m2) Excluded: listings_geom_active_idx (dropped by 104, listings_geom_idx covers it), listings_scraped_desc_idx (not in issue #1398 suggested fix scope). CONCURRENTLY not used — migration runner wraps in BEGIN/COMMIT (see 117 note).	2026-06-17 20:50:51 +03:00