448 commits

Author	SHA1	Message	Date
lekss361	8f9c11a45f	Merge pull request 'feat(browser): concurrent page pool (BROWSER_CONCURRENCY, default 4) + raise mem_limit to 5g' (#1733) from feat/browser-concurrent-pool into main	2026-06-17 21:07:31 +00:00
bot-backend	a6717705b9	fix(browser): reset page_counter on recycle + close login page in finally (#796 review)	2026-06-18 00:06:50 +03:00
bot-backend	820fd819bb	feat(browser): concurrent page pool (BROWSER_CONCURRENCY, default 4) + raise mem_limit to 5g	2026-06-17 23:55:11 +03:00
bot-backend	73fa739e0f	fix(backfill): drop nonexistent kadastr_num column from listings SELECT	2026-06-17 23:47:30 +03:00
bot-backend	687c9e5e9d	fix(avito): proactive IP rotate at sweep start + raise max_rotations 2->4 (Datadome ban recovery)	2026-06-17 23:35:24 +03:00
lekss361	84902984e3	Merge pull request 'feat(domclick): wire DomClick citywide sweep into pipeline+scheduler (seeded disabled)' (#1730) from feat/activate-domclick-sweep into main	2026-06-17 20:29:43 +00:00
lekss361	07eed4b55e	Merge pull request 'feat(avito): capture newbuilding ЖК id/url from SERP card' (#1729) from feat/avito-newbuilding-capture into main	2026-06-17 20:29:39 +00:00
bot-backend	2ed3ae81bc	feat(domclick): wire DomClick citywide sweep into pipeline+scheduler (seeded disabled)	2026-06-17 23:21:49 +03:00
bot-backend	dd59ef0516	feat(avito): capture newbuilding ЖК id/url from SERP card; persist newbuilding_id/url	2026-06-17 23:15:35 +03:00
bot-backend	e63b21eccb	fix(scrapers): write resolved house_id to listings.house_id_fk in realtime link hook	2026-06-17 23:07:22 +03:00
lekss361	97fce43484	Merge pull request 'feat(yandex): map rich entity fields (predictedPrice, seller, listing_date, trend, metro)' (#1725) from feat/yandex-rich-fields into main ... Reviewed-on: #1725	2026-06-17 20:03:25 +00:00
lekss361	df83e0256b	Merge pull request 'fix(tradein): correct SberIndex hedonic/asking dataset filter dims + 404 warning (#902)' (#1724) from fix/sber-dataset-paths-902 into main	2026-06-17 19:58:48 +00:00
bot-backend	1fe670ea6a	feat(yandex): map rich entity fields (predictedPrice, seller, listing_date, trend, metro)	2026-06-17 22:58:16 +03:00
bot-backend	5ca987fcd3	fix(tradein): correct SberIndex hedonic/asking dataset-paths + 404 warning (#902)	2026-06-17 22:50:52 +03:00
lekss361	f25e89767b	Merge pull request 'fix(matching): conflict_resolution picks freshest (last_seen_at) value, deterministic (#1539)' (#1722) from fix/conflict-resolution-freshness-1539 into main	2026-06-17 19:49:51 +00:00
bot-backend	f30f6acbf7	fix(matching): conflict_resolution picks freshest (last_seen_at) value, deterministic (#1539)	2026-06-17 22:43:59 +03:00
lekss361	45d8e9e4a5	Merge pull request 'fix(cian): persist detail rich fields (description) + stamp detail_enriched_at' (#1718) from fix/cian-detail-rich-fields into main	2026-06-17 19:42:58 +00:00
lekss361	5b8ee994d5	Merge pull request 'feat(tradein): segment=vtorichka|novostroyki|all on listings search (#1188)' (#1710) from feat/listings-segment-param-1188 into main	2026-06-17 19:34:50 +00:00
lekss361	931a57ac59	Merge pull request 'feat(yandex): ingest novostroyki via gate-API newFlat=YES' (#1715) from feat/yandex-novostroyki-newflat into main	2026-06-17 19:34:32 +00:00
bot-backend	fe805ea528	fix(cian): persist detail rich fields (description/kitchen/ceiling) + stamp detail_enriched_at	2026-06-17 22:32:18 +03:00
bot-backend	22a7a3f086	fix(avito): classify novostroyki by item-development-name marker; map переуступка sale_type	2026-06-17 22:24:13 +03:00
bot-backend	bc4c6852a1	feat(yandex): ingest novostroyki via gate-API newFlat=YES	2026-06-17 22:22:17 +03:00
bot-backend	b64334d7bd	feat(tradein): segment=vtorichka|novostroyki|all query-param on listings (#1188)	2026-06-17 21:43:01 +03:00
lekss361	16990af11c	Merge pull request 'docs(estimator): DKP corridor described as P10/P90 not min/max in schema+TS (#1520 follow-up)' (#1708) from fix/dkp-corridor-doc-p10p90-1695 into main	2026-06-17 18:42:10 +00:00
bot-backend	f1c5056397	docs(estimator): DKP corridor described as P10/P90 not min/max in schema+TS (#1520 follow-up)	2026-06-17 21:22:49 +03:00
bot-backend	a8d7be68ad	feat(yandex): route gate-API fetch through camoufox BrowserFetcher ... Replace the system-curl SOCKS5 subprocess transport with the shared BrowserFetcher (camoufox). camoufox's real-browser fingerprint is not tarpitted by yandex and does not burn the mobile-proxy IP; raw system-curl got the proxy IP flagged. One BrowserFetcher is opened per scraper session in __aenter__ and reused across all page fetches. _http_get now fetches via the browser, extracts the gate-API JSON from camoufox's HTML <pre> wrapper, and returns it as the body of a _CurlResponse (status 200 on success, 0 on fetch/extraction failure) so all existing callers and the tarpit retry/rotate logic keep working unchanged. URL building, _parse_gate_json, pagination, combos and field mapping are untouched. Removed the dead curl subprocess constants/method.	2026-06-17 21:16:48 +03:00
lekss361	262adbf762	Merge pull request 'fix(estimator): DKP corridor uses P10/P90 not min/max (#1520)' (#1695) from fix/dkp-corridor-percentile-1520 into main	2026-06-17 18:05:39 +00:00
lekss361	1fffb7a0a7	Merge pull request 'fix(sql): restore partial WHERE is_active predicate on listings indexes (#1398)' (#1693) from fix/listings-partial-indexes-1398 into main	2026-06-17 18:05:30 +00:00
lekss361	5c44233ed6	Merge pull request 'fix(scrapers): cian _walk_price_range stop silently truncating tail (#1393)' (#1689) from fix/cian-paginate-tail-1393 into main	2026-06-17 17:55:36 +00:00
bot-backend	403020ad56	fix(sql): restore partial predicate on 5 listings indexes (#1398) ... 097_index_hygiene.sql de-partialized 5 listings indexes on the false premise that is_active=100% true. 104_index_hygiene_geom_dedup.sql acknowledged the premise is stale (49% active on prod, 2026-06-13) but only cleaned up the redundant geom duplicate, leaving 3 composite indexes full. Restore WHERE is_active=true on the composite hot-path indexes: - listings_active_filter_idx (rooms, price_rub, area_m2, scraped_at DESC) - listings_house_price_idx (house_id_fk, price_rub) - listings_rooms_area_idx (rooms, area_m2) Excluded: listings_geom_active_idx (dropped by 104, listings_geom_idx covers it), listings_scraped_desc_idx (not in issue #1398 suggested fix scope). CONCURRENTLY not used — migration runner wraps in BEGIN/COMMIT (see 117 note).	2026-06-17 20:50:51 +03:00
bot-backend	6eda4a6d0a	fix(estimator): DKP corridor uses P10/P90 not min/max (#1520) ... Replace absolute min/max with robust percentiles so a single outlier ДКП deal cannot shift the soft-bound corridor boundaries. Small samples fall back gracefully via linear interpolation (n=3: P10≈index 0.2, P90≈index 2.8).	2026-06-17 20:50:15 +03:00
bot-backend	17c494a063	fix(scrapers): cian _walk_price_range stop silently truncating price-range tail (#1393)	2026-06-17 20:39:16 +03:00
bot-backend	f8ea119d22	fix(tradein): trade-in PDF cover uses estimate.expires_at, not hardcoded +30d (#1530)	2026-06-17 20:27:57 +03:00
lekss361	f8779c0812	feat(yandex): switch SERP scraping to JSON gate-API (bypasses bandwidth tarpit) (#1665)	2026-06-17 08:20:46 +00:00
bot-backend	b19c4a0b8c	fix(tradein): yandex fetch_around rotate-on-tarpit + retry (combos sweep resilience) ... Add _YANDEX_TARPIT_MAX_RETRIES=2 constant and rewrite fetch_around with a loop: on status_code==0 (curl exit 28, bandwidth tarpit) or captcha detected, call _rotate_ip() + sleep(2s) and retry. Non-zero non-200 codes (404 etc.) return [] immediately without rotation. Mirrors the existing exhaustive-path pattern in _walk_price_range. 4 new unit tests cover: tarpit→rotate→success, captcha→rotate→success, exhausted retries→empty, 404→no-rotate.	2026-06-17 10:12:24 +03:00
lekss361	7a88964ef5	fix(avito): scale per-anchor timeout for detail-enrich + durable SERP counters (#1661)	2026-06-17 06:07:50 +00:00
lekss361	4811920d28	fix(tradein): yandex combos watchdog timeout (P0) + deploy scraper filter gap (#1659) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-16 20:22:27 +00:00
lekss361	e7901bc1e8	feat(tradein): yandex city sweep — EKB-center rooms×price combos (replaces 5-anchor shallow) (#1658)	2026-06-16 19:57:37 +00:00
lekss361	66433dfe1b	feat(cian): collect price history via BrowserFetcher (fixes silent no-op backfill) (#1657) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-16 19:47:16 +00:00
lekss361	b5829fa226	feat(scrapers): enable cian_city_sweep (proxy alive, sweep is cookieless) (#1568)	2026-06-16 17:08:47 +00:00
lekss361	e46867fa7f	feat(tradein/db): derived price-history trigger (avito/yandex) + last_seen_at deactivate index (#1559)	2026-06-16 16:48:58 +00:00
lekss361	91f9a95f93	feat(scrapers): segment-aware generic deactivate_stale for yandex/cian (#1558)	2026-06-16 16:09:19 +00:00
bot-backend	6b7391ba93	fix(db): deactivate pre-#753 ghost duplicate listings (167 rows) ... Before #753 dedup_hash included price, so each price change created a new active row. The canonical rows are now upserted correctly; ghost rows sit frozen with stale hashes. Deactivates only active non-canonical rows in groups that already have a canonical twin, excluding all-ghost groups. Dry-run on prod (2026-06-16): UPDATE 167 rows as expected (yandex ~164, cian ~1, n1 ~2, avito 0). Refs #753	2026-06-16 15:40:30 +03:00
bot-backend	c07b5d7c5d	chore(tradein): деактивировать N1.ru scraper — источник переехал в Cian ... N1.ru перестал быть самостоятельным источником (переехал в Cian, подтверждено владельцем 2026-06). Активный скрап прекращён; 389 исторических listings в БД сохраняются без изменений. Убрано: - app/services/scrapers/n1.py (удалён) - run_n1_city_sweep + N1CitySweepCounters + N1_SWEEP_MAX_CONSECUTIVE_FAILURES из scrape_pipeline.py - trigger_n1_city_sweep_run + dispatch elif в scheduler.py - POST /scrape/n1 + GET /scrape/n1/runs endpoints из admin.py - N1Scraper из scrape_around dispatch dict и ScrapeRequest.multi_room_n1 - "n1": 5.0 из scraper_settings._DEFAULT_DELAY_BY_SOURCE - tests/test_n1_city_sweep.py, tests/test_n1_floor_parse.py (удалены) Добавлено: - data/sql/114_disable_n1_sweep.sql: UPDATE scrape_schedules SET enabled=false WHERE source='n1_city_sweep' (idempotent, строка сохраняется для audit-истории) Сохранено (историческое): - schemas/search.py Literal[..."n1"] — API-фильтр исторических listings - config.py scrape_allowed_hosts n1.ru / ekaterinburg.n1.ru - exporters/trade_in_pdf.py — цвет/лейбл N1 для отображения старых записей - geocoder.py и admin.py docstring N1-комментарии где не активный код 1802 passed, 2 deselected (test_search_cache_hit, test_cian_valuation.cache_hit)	2026-06-16 14:50:51 +03:00
bot-backend	b390c8334c	fix(tradein): cian detail-парсер не извлекал kitchen_area_m2 из offer.kitchenArea ... Добавлено поле kitchen_area_m2 в DetailEnrichment dataclass, извлечение через _parse_float(offer.get("kitchenArea")) в fetch_detail, и COALESCE(CAST(:ka AS double precision), kitchen_area_m2) в UPDATE listings внутри save_detail_enrichment. Имя поля kitchenArea — стандартное cian offer API (live-verify не делался, прокси недоступен в dev-окружении). Closes #1553	2026-06-16 14:37:49 +03:00
bot-backend	fd4a0d643d	fix(tradein): review minors — scheduler comment + save-error log label ... - scheduler.py: исправлен stale-комментарий yandex_detail_backfill (был: «YandexDetailScraper httpx, no proxy layer»; теперь: fetch via curl_cffi chrome120 + scraper_proxy_url, parse via YandexDetailScraper.parse) - yandex_detail_backfill.py: outer except — сообщение лога изменено на «save/iteration error for listing_id=%d» чтобы отличать save-ошибку от fetch/parse-None путей (run_id убран — он менее важен в этом scope)	2026-06-16 14:14:07 +03:00
bot-backend	e2e0291a0a	fix(tradein): yandex detail backfill — заменить fetch_detail на curl_cffi+proxy ... YandexDetailScraper.fetch_detail использует plain httpx (BaseScraper._http_get) без прокси и без TLS-fingerprinting → на датацентровом IP Yandex отдаёт капчу / shell-HTML → parse всегда None → backfill 0% эффективен. Замена: curl_cffi AsyncSession(chrome120 + scraper_proxy_url), идентично yandex_address_backfill.py (доказано работающий путь на проде). HTTP 200 → YandexDetailScraper.parse(html) (чистая функция, без сети). parse→None считается fail (captcha wall); consecutive None → abort после max_consecutive_blocks. Тесты обновлены: мокается AsyncSession.get + YandexDetailScraper.parse (не fetch_detail). Добавлены кейсы: non-200 → abort, proxy=None → proxies=None.	2026-06-16 14:05:22 +03:00
bot-backend	03e43f12b5	feat(tradein): yandex detail backfill — save_detail_enrichment + nightly task (#1553) ... - yandex_detail.py: добавить save_detail_enrichment(db, listing_id, enrichment) → bool; UPDATE listings COALESCE по 17 полям (rooms/area_m2/floor/total_floors/address/ description/repair_state/publish_date/views_total_yandex/publish_date_relative/ agency_name/agency_founded_year/agency_objects_count/metro_stations/photo_urls/ newbuilding_url/newbuilding_id); metro/photos → jsonb; detail_enriched_at = NOW() - app/tasks/yandex_detail_backfill.py: новый task run_yandex_detail_backfill; snapshot SELECT source='yandex' AND detail_enriched_at IS NULL; YandexDetailScraper async context manager; consecutive None abort (max_consecutive_blocks=5); budget guard; run lifecycle update_heartbeat/mark_done/mark_failed - scheduler.py: trigger_yandex_detail_backfill_run + elif source=='yandex_detail_backfill' в scheduler_loop; window 12-15 UTC (после avito_detail 09-12 UTC) - migration 113: ADD COLUMN IF NOT EXISTS photo_urls jsonb, newbuilding_url text, newbuilding_id text; seed scrape_schedules yandex_detail_backfill enabled=true - tests: 10 тестов (8 backfill + 2 save_detail_enrichment); 1819 passed 0 failed Closes #1553	2026-06-16 13:58:20 +03:00
bot-backend	227e82dc01	feat(tradein): avito detail-enrichment nightly backfill (#1551) ... - New task app/tasks/avito_detail_backfill.py with run_avito_detail_backfill() * Single snapshot SELECT at start (guarantees termination) * Same proxy/AsyncSession path as scrape_pipeline.py step 5 * Budget guard (budget_sec), consecutive block abort (mark_done not mark_failed) * rotate_ip() on every AvitoBlockedError; rollback on generic Exception (#1368) * start = time.monotonic() initialized before try so except can reference it - Scheduler wiring: trigger_avito_detail_backfill_run() + elif in scheduler_loop() - Migration 112: scrape_schedules INSERT window 09-12 UTC, batch_size=800, budget_sec=3600, request_delay_sec=6, max_consecutive_blocks=5 - 7 unit tests (no pytest-mock, unittest.mock only): all 7 passing, full CI suite 1809 passed	2026-06-16 13:25:29 +03:00
lekss361	b968b811cb	Merge pull request 'fix(scrapers): geocode-missing listings nightly scheduler — listings geom coverage backfill (all sources)' (#1549) from fix/geocode-missing-listings-scheduler into main ... Reviewed-on: #1549	2026-06-16 08:32:22 +00:00