257 commits

Author	SHA1	Message	Date
bot-backend	cbeeb9c414	fix(tradein): avito full-load probe resilience — browser 500/timeout splits/skips instead of crashing run	2026-06-18 16:03:39 +03:00
lekss361	493142b990	Merge pull request 'feat(tradein): populate offer_price_history for yandex from gate price.previous/trend' (#1754) from feat/yandex-price-history into main	2026-06-18 11:37:05 +00:00
bot-backend	6c132d7f0a	fix(tradein): avito sweep page-level retry — transient page error no longer aborts run	2026-06-18 14:29:01 +03:00
bot-backend	6c82528db5	feat(tradein): populate offer_price_history for yandex from gate price.previous/trend	2026-06-18 14:23:23 +03:00
bot-backend	1ddbf867f8	feat(tradein): avito exhaustive full-load (room×price bisection) — mirror cian	2026-06-18 13:55:18 +03:00
bot-backend	158a24c25b	feat(tradein): schedule cian_full_load (exhaustive secondary) + anchor sweep NB-only ... Подключает prod-проверенный run_cian_full_load (exhaustive региональный сбор Cian ЕКБ вторички, room×price партиционирование, incremental on_bucket save) в in-app scheduler как recurring-источник cian_full_load (окно 20-22 UTC — отделено от cian_city_sweep/history_backfill на 2-5 чтобы не конкурировать на shared kf-прокси). run_cian_city_sweep теперь NB-only (newbuilding_only=True default): SERP-фаза сохраняет только novostroyki-лоты, вторичку отбрасывает — ею авторитетно владеет full_load. Убирает дубль secondary SERP-сейва. DETAIL/HOUSES-фазы не затронуты.	2026-06-18 13:25:23 +03:00
lekss361	48bedb5f65	Merge branch 'main' into feat/cadastral-geo-match	2026-06-18 08:44:06 +00:00
bot-backend	f9769571a3	fix(cadastral): single-statement match UPDATE — OFFSET chunk skipped rows ... The chunked matcher paged the candidate set with OFFSET over the very predicate it mutates (building_cadastral_number IS NULL). Matched rows drop out, so OFFSET (reset to 0 OR advanced) skips un-processed rows: once >= batch_size unmatched listings (beyond threshold) accumulate at the low-id front, the chunk matches 0 and the loop breaks early, leaving higher-id matchable listings unfilled. Counterexample (batch=3, matchable ids 1,4,7,10): matches 1 and 4 then breaks, missing 7 and 10. Unit tests used single-chunk fixtures so never caught it. The full UPDATE is ~5s for ~41k listings, so replace the loop with ONE set-based LATERAL KNN UPDATE over all candidates — correct and fast. batch_size kept in the signature for schedule-param compat (now unused).	2026-06-18 10:28:19 +03:00
bot-backend	edd187b47f	feat(cadastral): geo-nearest building matcher via local cad mirror ... Backfill listings.building_cadastral_number (was 0%) from the nearest cadastral building. gendesign_cad_buildings is a postgres_fdw foreign table with no geom — a per-listing FDW nearest query is ~1.16s/row (~13h for 43k listings). Instead materialize the FDW once into a LOCAL cad_buildings_local table (Point geom + GIST), then run a fast local KNN nearest-neighbour join. Perf: the distance gate uses geometry-space ST_DWithin(geom, point, deg) (GIST index, no geography cast) + geom <-> KNN order + ST_DistanceSphere metric recheck on the single nearest row. A geography-cast ST_DWithin in the WHERE defeated the index (58s+, full update never finished); the geometry-gate design does the whole ~41.9k-listing UPDATE in ~5.4s (refresh+match ~7s end-to-end), matching 16027 listings at 50m across 2229 distinct buildings. The match is GEO-NEAREST (approximate): a street-level-geocoded listing matches the nearest building within threshold_m (default 50m), not necessarily its exact cadastral building. Exact cadastral + parcel-containment deferred (cad_parcels FDW not exposed). Threshold is logged. - 124: cad_buildings_local table (empty) + GIST. Migration does not read the FDW (deploy-independent); populated by the refresh job. - 125: scrape_schedules seed source=cadastral_geo_match, enabled, next_run_at tomorrow 09:00 UTC (after geocode_missing). - tasks/cadastral_geo_match.py: refresh_cad_buildings_local (bulk FDW scan), match_listings_to_buildings (chunked LATERAL KNN UPDATE), run_cadastral_geo_match run-lifecycle wrapper. - scheduler: trigger_cadastral_geo_match_run + dispatch (sync DB-only, executor).	2026-06-18 10:18:54 +03:00
bot-backend	1a658a65a6	feat(avito): dedicated newbuilding (novostroyka) citywide sweep	2026-06-18 09:45:54 +03:00
bot-backend	b9eb478c8f	feat(browser): per-source proxy pool behind FEATURE_BROWSER_POOL_ENABLED (Phase 1) ... Route each scraper source (avito/cian/yandex/domclick) to its own camoufox browser+proxy so they no longer wedge each other through a single global egress. Feature-flagged (FEATURE_BROWSER_POOL_ENABLED, default OFF): with the flag off the /fetch and /login paths are byte-for-byte the existing single-browser behavior. When on, /fetch routes by body["source"] to a per-proxy browser via BROWSER_PROXY_MAP (BROWSER_PROXY_AVITO/CIAN/YANDEX/DOMCLICK with legacy fallbacks), each guarded by its own lazy-launched lock. /login stays single-browser in Phase 1. BrowserFetcher gains a source arg (default avito) and sends it in the /fetch body; all scraper callsites pass their source. No docker-compose/.env.runtime changes (Phase 2, owner-gated).	2026-06-18 09:01:50 +03:00
lekss361	635851a71c	Merge pull request 'fix(yandex): read predicted price from entity.predictions.predictedPrice (#1725)' (#1734) from fix/yandex-predicted-price-path-1725 into main ... Reviewed-on: #1734	2026-06-18 05:12:25 +00:00
bot-backend	41f2402b25	fix(yandex): read predicted price from entity.predictions.predictedPrice (#1725) ... Gate-API nests Yandex's per-offer valuation under predictions.predictedPrice, not predictedPrice.predictedPrice (that key does not exist). _predicted_prices read the wrong outer key, so predicted_price_rub/_min/_max were None for every offer (prod: pp=0 across all 6189 yandex rows). Verified live against the gate-API 2026-06-18. The rich-field test fixture used the same wrong wrapper, so the test passed vacuously. Updated _ENTITY_RICH_OWNER to the real predictions.predictedPrice shape — with the old code the assertions now fail, confirming non-vacuity.	2026-06-18 08:03:05 +03:00
bot-backend	73fa739e0f	fix(backfill): drop nonexistent kadastr_num column from listings SELECT	2026-06-17 23:47:30 +03:00
bot-backend	687c9e5e9d	fix(avito): proactive IP rotate at sweep start + raise max_rotations 2->4 (Datadome ban recovery)	2026-06-17 23:35:24 +03:00
lekss361	84902984e3	Merge pull request 'feat(domclick): wire DomClick citywide sweep into pipeline+scheduler (seeded disabled)' (#1730) from feat/activate-domclick-sweep into main	2026-06-17 20:29:43 +00:00
lekss361	07eed4b55e	Merge pull request 'feat(avito): capture newbuilding ЖК id/url from SERP card' (#1729) from feat/avito-newbuilding-capture into main	2026-06-17 20:29:39 +00:00
bot-backend	2ed3ae81bc	feat(domclick): wire DomClick citywide sweep into pipeline+scheduler (seeded disabled)	2026-06-17 23:21:49 +03:00
bot-backend	dd59ef0516	feat(avito): capture newbuilding ЖК id/url from SERP card; persist newbuilding_id/url	2026-06-17 23:15:35 +03:00
bot-backend	e63b21eccb	fix(scrapers): write resolved house_id to listings.house_id_fk in realtime link hook	2026-06-17 23:07:22 +03:00
lekss361	97fce43484	Merge pull request 'feat(yandex): map rich entity fields (predictedPrice, seller, listing_date, trend, metro)' (#1725) from feat/yandex-rich-fields into main ... Reviewed-on: #1725	2026-06-17 20:03:25 +00:00
lekss361	df83e0256b	Merge pull request 'fix(tradein): correct SberIndex hedonic/asking dataset filter dims + 404 warning (#902)' (#1724) from fix/sber-dataset-paths-902 into main	2026-06-17 19:58:48 +00:00
bot-backend	1fe670ea6a	feat(yandex): map rich entity fields (predictedPrice, seller, listing_date, trend, metro)	2026-06-17 22:58:16 +03:00
bot-backend	5ca987fcd3	fix(tradein): correct SberIndex hedonic/asking dataset-paths + 404 warning (#902)	2026-06-17 22:50:52 +03:00
lekss361	f25e89767b	Merge pull request 'fix(matching): conflict_resolution picks freshest (last_seen_at) value, deterministic (#1539)' (#1722) from fix/conflict-resolution-freshness-1539 into main	2026-06-17 19:49:51 +00:00
bot-backend	f30f6acbf7	fix(matching): conflict_resolution picks freshest (last_seen_at) value, deterministic (#1539)	2026-06-17 22:43:59 +03:00
lekss361	45d8e9e4a5	Merge pull request 'fix(cian): persist detail rich fields (description) + stamp detail_enriched_at' (#1718) from fix/cian-detail-rich-fields into main	2026-06-17 19:42:58 +00:00
lekss361	5b8ee994d5	Merge pull request 'feat(tradein): segment=vtorichka|novostroyki|all on listings search (#1188)' (#1710) from feat/listings-segment-param-1188 into main	2026-06-17 19:34:50 +00:00
lekss361	931a57ac59	Merge pull request 'feat(yandex): ingest novostroyki via gate-API newFlat=YES' (#1715) from feat/yandex-novostroyki-newflat into main	2026-06-17 19:34:32 +00:00
bot-backend	fe805ea528	fix(cian): persist detail rich fields (description/kitchen/ceiling) + stamp detail_enriched_at	2026-06-17 22:32:18 +03:00
bot-backend	22a7a3f086	fix(avito): classify novostroyki by item-development-name marker; map переуступка sale_type	2026-06-17 22:24:13 +03:00
bot-backend	bc4c6852a1	feat(yandex): ingest novostroyki via gate-API newFlat=YES	2026-06-17 22:22:17 +03:00
bot-backend	b64334d7bd	feat(tradein): segment=vtorichka|novostroyki|all query-param on listings (#1188)	2026-06-17 21:43:01 +03:00
bot-backend	a8d7be68ad	feat(yandex): route gate-API fetch through camoufox BrowserFetcher ... Replace the system-curl SOCKS5 subprocess transport with the shared BrowserFetcher (camoufox). camoufox's real-browser fingerprint is not tarpitted by yandex and does not burn the mobile-proxy IP; raw system-curl got the proxy IP flagged. One BrowserFetcher is opened per scraper session in __aenter__ and reused across all page fetches. _http_get now fetches via the browser, extracts the gate-API JSON from camoufox's HTML <pre> wrapper, and returns it as the body of a _CurlResponse (status 200 on success, 0 on fetch/extraction failure) so all existing callers and the tarpit retry/rotate logic keep working unchanged. URL building, _parse_gate_json, pagination, combos and field mapping are untouched. Removed the dead curl subprocess constants/method.	2026-06-17 21:16:48 +03:00
lekss361	262adbf762	Merge pull request 'fix(estimator): DKP corridor uses P10/P90 not min/max (#1520)' (#1695) from fix/dkp-corridor-percentile-1520 into main	2026-06-17 18:05:39 +00:00
bot-backend	6eda4a6d0a	fix(estimator): DKP corridor uses P10/P90 not min/max (#1520) ... Replace absolute min/max with robust percentiles so a single outlier ДКП deal cannot shift the soft-bound corridor boundaries. Small samples fall back gracefully via linear interpolation (n=3: P10≈index 0.2, P90≈index 2.8).	2026-06-17 20:50:15 +03:00
bot-backend	17c494a063	fix(scrapers): cian _walk_price_range stop silently truncating price-range tail (#1393)	2026-06-17 20:39:16 +03:00
lekss361	f8779c0812	feat(yandex): switch SERP scraping to JSON gate-API (bypasses bandwidth tarpit) (#1665)	2026-06-17 08:20:46 +00:00
bot-backend	b19c4a0b8c	fix(tradein): yandex fetch_around rotate-on-tarpit + retry (combos sweep resilience) ... Add _YANDEX_TARPIT_MAX_RETRIES=2 constant and rewrite fetch_around with a loop: on status_code==0 (curl exit 28, bandwidth tarpit) or captcha detected, call _rotate_ip() + sleep(2s) and retry. Non-zero non-200 codes (404 etc.) return [] immediately without rotation. Mirrors the existing exhaustive-path pattern in _walk_price_range. 4 new unit tests cover: tarpit→rotate→success, captcha→rotate→success, exhausted retries→empty, 404→no-rotate.	2026-06-17 10:12:24 +03:00
lekss361	7a88964ef5	fix(avito): scale per-anchor timeout for detail-enrich + durable SERP counters (#1661)	2026-06-17 06:07:50 +00:00
lekss361	4811920d28	fix(tradein): yandex combos watchdog timeout (P0) + deploy scraper filter gap (#1659) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-16 20:22:27 +00:00
lekss361	e7901bc1e8	feat(tradein): yandex city sweep — EKB-center rooms×price combos (replaces 5-anchor shallow) (#1658)	2026-06-16 19:57:37 +00:00
lekss361	66433dfe1b	feat(cian): collect price history via BrowserFetcher (fixes silent no-op backfill) (#1657) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-16 19:47:16 +00:00
lekss361	91f9a95f93	feat(scrapers): segment-aware generic deactivate_stale for yandex/cian (#1558)	2026-06-16 16:09:19 +00:00
bot-backend	c07b5d7c5d	chore(tradein): деактивировать N1.ru scraper — источник переехал в Cian ... N1.ru перестал быть самостоятельным источником (переехал в Cian, подтверждено владельцем 2026-06). Активный скрап прекращён; 389 исторических listings в БД сохраняются без изменений. Убрано: - app/services/scrapers/n1.py (удалён) - run_n1_city_sweep + N1CitySweepCounters + N1_SWEEP_MAX_CONSECUTIVE_FAILURES из scrape_pipeline.py - trigger_n1_city_sweep_run + dispatch elif в scheduler.py - POST /scrape/n1 + GET /scrape/n1/runs endpoints из admin.py - N1Scraper из scrape_around dispatch dict и ScrapeRequest.multi_room_n1 - "n1": 5.0 из scraper_settings._DEFAULT_DELAY_BY_SOURCE - tests/test_n1_city_sweep.py, tests/test_n1_floor_parse.py (удалены) Добавлено: - data/sql/114_disable_n1_sweep.sql: UPDATE scrape_schedules SET enabled=false WHERE source='n1_city_sweep' (idempotent, строка сохраняется для audit-истории) Сохранено (историческое): - schemas/search.py Literal[..."n1"] — API-фильтр исторических listings - config.py scrape_allowed_hosts n1.ru / ekaterinburg.n1.ru - exporters/trade_in_pdf.py — цвет/лейбл N1 для отображения старых записей - geocoder.py и admin.py docstring N1-комментарии где не активный код 1802 passed, 2 deselected (test_search_cache_hit, test_cian_valuation.cache_hit)	2026-06-16 14:50:51 +03:00
bot-backend	b390c8334c	fix(tradein): cian detail-парсер не извлекал kitchen_area_m2 из offer.kitchenArea ... Добавлено поле kitchen_area_m2 в DetailEnrichment dataclass, извлечение через _parse_float(offer.get("kitchenArea")) в fetch_detail, и COALESCE(CAST(:ka AS double precision), kitchen_area_m2) в UPDATE listings внутри save_detail_enrichment. Имя поля kitchenArea — стандартное cian offer API (live-verify не делался, прокси недоступен в dev-окружении). Closes #1553	2026-06-16 14:37:49 +03:00
bot-backend	e2e0291a0a	fix(tradein): yandex detail backfill — заменить fetch_detail на curl_cffi+proxy ... YandexDetailScraper.fetch_detail использует plain httpx (BaseScraper._http_get) без прокси и без TLS-fingerprinting → на датацентровом IP Yandex отдаёт капчу / shell-HTML → parse всегда None → backfill 0% эффективен. Замена: curl_cffi AsyncSession(chrome120 + scraper_proxy_url), идентично yandex_address_backfill.py (доказано работающий путь на проде). HTTP 200 → YandexDetailScraper.parse(html) (чистая функция, без сети). parse→None считается fail (captcha wall); consecutive None → abort после max_consecutive_blocks. Тесты обновлены: мокается AsyncSession.get + YandexDetailScraper.parse (не fetch_detail). Добавлены кейсы: non-200 → abort, proxy=None → proxies=None.	2026-06-16 14:05:22 +03:00
bot-backend	03e43f12b5	feat(tradein): yandex detail backfill — save_detail_enrichment + nightly task (#1553) ... - yandex_detail.py: добавить save_detail_enrichment(db, listing_id, enrichment) → bool; UPDATE listings COALESCE по 17 полям (rooms/area_m2/floor/total_floors/address/ description/repair_state/publish_date/views_total_yandex/publish_date_relative/ agency_name/agency_founded_year/agency_objects_count/metro_stations/photo_urls/ newbuilding_url/newbuilding_id); metro/photos → jsonb; detail_enriched_at = NOW() - app/tasks/yandex_detail_backfill.py: новый task run_yandex_detail_backfill; snapshot SELECT source='yandex' AND detail_enriched_at IS NULL; YandexDetailScraper async context manager; consecutive None abort (max_consecutive_blocks=5); budget guard; run lifecycle update_heartbeat/mark_done/mark_failed - scheduler.py: trigger_yandex_detail_backfill_run + elif source=='yandex_detail_backfill' в scheduler_loop; window 12-15 UTC (после avito_detail 09-12 UTC) - migration 113: ADD COLUMN IF NOT EXISTS photo_urls jsonb, newbuilding_url text, newbuilding_id text; seed scrape_schedules yandex_detail_backfill enabled=true - tests: 10 тестов (8 backfill + 2 save_detail_enrichment); 1819 passed 0 failed Closes #1553	2026-06-16 13:58:20 +03:00
bot-backend	227e82dc01	feat(tradein): avito detail-enrichment nightly backfill (#1551) ... - New task app/tasks/avito_detail_backfill.py with run_avito_detail_backfill() * Single snapshot SELECT at start (guarantees termination) * Same proxy/AsyncSession path as scrape_pipeline.py step 5 * Budget guard (budget_sec), consecutive block abort (mark_done not mark_failed) * rotate_ip() on every AvitoBlockedError; rollback on generic Exception (#1368) * start = time.monotonic() initialized before try so except can reference it - Scheduler wiring: trigger_avito_detail_backfill_run() + elif in scheduler_loop() - Migration 112: scrape_schedules INSERT window 09-12 UTC, batch_size=800, budget_sec=3600, request_delay_sec=6, max_consecutive_blocks=5 - 7 unit tests (no pytest-mock, unittest.mock only): all 7 passing, full CI suite 1809 passed	2026-06-16 13:25:29 +03:00
lekss361	b968b811cb	Merge pull request 'fix(scrapers): geocode-missing listings nightly scheduler — listings geom coverage backfill (all sources)' (#1549) from fix/geocode-missing-listings-scheduler into main ... Reviewed-on: #1549	2026-06-16 08:32:22 +00:00