49 commits

Author	SHA1	Message	Date
Light1YT	312df8536a	feat(tradein): infer repair_state from listing description text (#622) (#632)	2026-05-28 15:37:34 +00:00
Light1YT	d842811390	refactor(tradein): normalize repair_state raw→enum at ingest (#621) (#629)	2026-05-28 15:18:11 +00:00
Light1YT	1731704ddc	feat(tradein): houses.cian_zhk_url column + backfill (#568) (#627)	2026-05-28 13:57:43 +00:00
Light1YT	ec26e5813f	fix(tradein): implement listings_snapshots + offer_price_history writers (#567) (#624)	2026-05-28 13:49:52 +00:00
Light1YT	81bf7e6def	fix(tradein-imv): harden Avito IMV anti-bot path + offline contract tests (#565) ... Add warm-up GET /evaluation/realty + production browser header set (Sec-Fetch, X-Requested-With, 25s timeout) mirroring avito.py, чтобы защитить 3-step IMV flow от server-IP анти-бот 403. Refactor step B/C парсинга в pure helpers (_parse_geo_position/_parse_price/_parse_placement_history/_parse_suggestions) + fixture-driven offline тесты из живых JSON-captures. Честный диагноз: контракт (geoFieldsHash, price-блок, categoryId=24) НЕ менялся — проверено живым capture'ом с этой машины (A→B→C, recommended 7.39M). Реальная причина #565 — скорее анти-бот по PROD-IP, не код → если #565 останется после deploy, настоящий фикс = ротация egress IP воркера (devops, отдельно). code-reviewer: ✅ APPROVE no critical — warm-up non-fatal, injected-session path unchanged, refactor preserves shapes, graceful-None сохранён, fixtures без PII. Tests: 12 new (27 total). Ruff clean. Closes #565	2026-05-28 16:29:05 +05:00
Light1YT	a3f45a6722	fix(tradein): decode Avito houses URL-encoded __preloadedState__ (#566) ... Avito switched the houses-catalog state blob from JSON.parse("<js-escaped>") to a URL-encoded string literal, wrapped the 10 widgets in placeholders[0].widgets, и re-nested developmentData под props.housePage. JS-unescape path silently failed → houses rows перестали наполняться (starve'ит #6 same-building + #569 deal centroids). Extract decode в pure _extract_preloaded_state (urllib.parse.unquote, JSON.parse fallback для старой формы), add _flatten_widgets, и сделать _parse_house_page tolerant к обеим developmentData/ratingPreview формам. Real-HTML fixture (Yumasheva-6 EKB) + 4 regression-теста. code-reviewer: ✅ APPROVE no critical (regex anchoring verified, decode safe — no eval/unicode_escape, backward-compat реальный, delisted-house envelope degrade'ит gracefully, fixture без PII/secrets). Tests: 22 passed. Closes #566	2026-05-28 15:58:19 +05:00
Light1YT	99a5a0f73b	fix(tradein): Yandex listing_date — keyword + absolute-no-year (#602)	2026-05-27 13:11:22 +00:00
Light1YT	f5c87201f1	fix(tradein): Avito listing_date extraction (3 формата) (#601)	2026-05-27 12:55:27 +00:00
Light1YT	7e24ccbf2c	feat(tradein): hook matching в save_listings — fill listing_sources (PR I) (#595)	2026-05-27 10:02:37 +00:00
Light1YT	aa70e9e155	fix(tradein): yandex_realty captures house number + parse_rub strips NBSP (#593)	2026-05-27 07:23:36 +00:00
lekss361	6bbb026cb4	fix(yandex-valuation): off-by-one cell index in _parse_row_cells (#552)	2026-05-24 19:57:17 +00:00
lekss361	f7a5c5d4f7	fix(yandex-valuation): curl_cffi chrome120 — Yandex gates SSR on Chrome TLS (#545)	2026-05-24 16:42:07 +00:00
lekss361	8a88ea6508	refactor(yandex-valuation): DOM-based parser via .OffersArchiveSearchOffers__row (#544)	2026-05-24 16:25:56 +00:00
lekss361	3ed58e3e7a	fix(yandex-valuation): relax area regex lookbehind (2-комн regression) (#542)	2026-05-24 16:02:06 +00:00
lekss361	490bc8ca5d	fix(yandex-valuation): tighten area regex + chronological date ordering (#541)	2026-05-24 15:37:48 +00:00
lekss361	9505471129	fix(yandex-valuation): cap area_m2 sanity (drop >10000 or ≤0) (#538)	2026-05-24 15:14:13 +00:00
lekss361	ee2262d605	fix(tradein-phase-c): house_type 'monolith'→'monolithic' + log 400 body (#537)	2026-05-24 15:05:49 +00:00
lekss361	7ef4e91e50	fix(yandex-valuation): parse removed_date, fix total_floors regex, normalize NBSP (#526)	2026-05-24 14:12:16 +00:00
lekss361	0f7d6c4996	fix(tradein-imv): adapt to Avito 3-step API contract (2026-05+) (#524)	2026-05-24 13:48:52 +00:00
lekss361	bad23095cf	fix(tradein/cian): sanitize valuation cache writes (047 UNIQUE + 026/044 cols) (#523)	2026-05-24 13:44:39 +00:00
lekss361	d4cbd4bb9f	feat(tradein-scrapers): parse listing_date from Avito + Yandex (#508)	2026-05-24 11:17:13 +00:00
lekss361	7f23a9b3f1	feat(tradein-imv): retry with cleaned address + categorize errors (#505)	2026-05-24 11:04:24 +00:00
lekss361	8fc309d9ad	fix(tradein-avito): strip Emotion CSS from listings.address (#502)	2026-05-24 10:52:50 +00:00
lekss361	222389cba9	fix(tradein/avito): anti-bot hardening — sleep + abort + shared session (#487)	2026-05-23 20:08:40 +00:00
lekss361	e79c9cec88	feat(tradein): global scraper delay setting (applies across all scrapers) (#485) ... Add global request delay floor + per-source delay management. - 054: seed 'global'=0 + per-source rows (avito=7, cian=5, n1=5, domrf=5, rosreestr=5) using ON CONFLICT DO NOTHING; reuses existing 053 table from #484 - scraper_settings.py: get_scraper_delay() = max(per_source, global); _GLOBAL_KEY, refactored _get_setting_cached(); preserved yandex umbrella alias logic - admin.py: GET /scraper-settings + PUT /scraper-settings/{source} (ge=0.0, le=60.0); cache invalidation on update; CAST(:d AS numeric) per psycopg v3 - avito/cian/n1 scrapers wired (yandex was wired by #484) - 18 unit tests pass (10 original updated for max() + 8 new global delay tests) Closes follow-up gap from #486 (Cian admin slider min=0 now accepted by API).	2026-05-23 15:43:28 +00:00
lekss361	b21d7c7c85	feat(tradein): scraper_settings live-config + Yandex admin trigger endpoints (#484)	2026-05-23 15:28:34 +00:00
lekss361	2914ff2ef2	feat(tradein): city sweep — auto ЕКБ pipeline (#477) ... Stage 4d: full city sweep ЕКБ (5 anchors × N pages + enrichment + cooperative cancel). - avito.py fetch_around(pages=N) — pagination backward compat default=1 - scrape_pipeline.run_avito_city_sweep — anchors loop + heartbeat + cancel check - scrape_runs.py utility (create/heartbeat/done/failed/cancelled/list_recent) - admin.py 3 endpoints: start (BackgroundTasks) / runs / cancel - migration 051 ADD COLUMN IF NOT EXISTS + DROP/ADD CHECK constraint (text+CHECK, не enum — safe в transaction) - deploy/avito-city-sweep.sh с random delay 0-3h (anti-pattern) - 16 offline tests Deep review approved: migration idempotent, cancel priority correct (mark_done guarded by status='running'), no SQL injection, psycopg v3 compliant.	2026-05-23 14:38:07 +00:00
lekss361	51650276b2	feat(tradein): yandex_newbuilding.py — ЖК landing parser (149 selectors + NLP) (#467)	2026-05-23 13:46:45 +00:00
lekss361	7056ecb23b	feat(tradein): yandex_valuation.py — anonymous house-history scraper (#468)	2026-05-23 13:46:20 +00:00
lekss361	3d90221fa0	feat(tradein): yandex_detail.py — Product JSON-LD + DOM detail parser (#466) ... Stage 4 of YandexRealtyScraper v1. YandexDetailScraper.fetch_detail(offer_url) extracts authoritative price from Product JSON-LD (offers.price exact int) + DOM sections (description, agent, stats, metro, photos 8 sizes, newbuilding link, NLP). 35 unit tests, ruff clean. Photo URLs sanitized downstream via _safe_url CDN allowlist (avatars.mds.yandex.net).	2026-05-23 13:45:09 +00:00
lekss361	b0a7235110	feat(tradein): cian_valuation.py — auth-required Calculator scraper + 24h cache (#465)	2026-05-23 13:43:34 +00:00
lekss361	29f94963a7	Merge pull request 'feat(tradein): yandex_realty.py — DOM SERP refactor (vtorichka, 23 cards/page)' (#462) from feat/tradein-yandex-serp-refactor into main	2026-05-23 13:30:36 +00:00
lekss361	d28bf4e66a	feat(tradein): cian_newbuilding.py — ЖК catalog scraper (chart + reliability + offers, Wave 4 Worker C) (#459) ... Stage 6 / Wave 4 Worker C of CianScraper v1. Cian.ru newbuilding (ЖК) catalog page scraper — extracts newbuilding metadata + 6 sister state containers (realtyValuation chart / reliability / reviews / offers / builders / housesByTurn). Main Stage 6 deliverable: 7-month price dynamics chart → houses_price_dynamics table (cross-source benchmark для Stage 9 estimator). Files: - app/services/scrapers/cian_newbuilding.py (NEW, 548 LOC) - tests/test_cian_newbuilding.py (NEW, 357 LOC, 20 tests) Reviewed: schema match (migrations 020/021/025/029), ON CONFLICT constraint name verified (houses_price_dynamics_dim_key), graceful chart extraction (empty/None/fallback shapes), atomic transaction. Known minor follow-ups (non-blocking): - management_companies UPSERT: ext_id=NULL → potential duplicates (Postgres NULL!=NULL); bounded, document in vault - offers items[] fast path: no dict-only filter / no room_count injection - houses_price_dynamics primary shape stores total price into price_per_sqm column with prices_type='price' discriminator — Stage 9 estimator must respect prices_type filter - house_reliability_checks no UNIQUE → unbounded growth on rescrape (deferred to future migration 030+) Verdict: APPROVE (deep-code-reviewer).	2026-05-23 13:23:11 +00:00
lekss361	2dd76ea5e8	feat(tradein): cian_session cookies management + admin endpoints (Wave 4 Worker A) (#457)	2026-05-23 13:22:00 +00:00
lekss361	ead2ac16c5	feat(tradein): yandex_realty.py — DOM SERP refactor (vtorichka, 23 cards/page) ... Replace dead JSON-state extraction (Yandex moved to chunked/ephemeral state post-hydration) with selectolax DOM parsing using helpers from PR #456. Changes: - New URL: /{city}/kupit/kvartira/vtorichniy-rynok/?page=N (path-based, no geo) - Selector: [data-test="OffersSerpItem"] x ~23 per page - Per-card extraction: offer_id (URL regex), title (rooms/area/floor regex), price/ppm2 (regex + parse_rub), bargain flag, publish_date (parse_ru_date), street address, photos (avatars.mds.yandex domain filter + size upgrade), newbuilding linkage (house_source='yandex_realty_nb' + house_ext_id) - ScrapedLot mapping: lat/lon=None (no coords on SERP — geocode-missing fills later), listing_segment="vtorichka" - Pagination: fetch_around_multi_room -> up to MAX_PAGES=3 pages, dedup by source_id. **kwargs swallow keeps old rooms/sort/pages callers compat. - Preserved: BaseScraper interface, name="yandex" (matches existing rows + avoids breaking dedup_hash; Wave 5 conflict resolution can alias if needed), request_delay_sec=5.0, sleep_between_requests, tenacity retry. Tests: 10 unit tests against hand-crafted fixture HTML (single card / studio / no-price / empty page / multi-card / URL builder / city override) — all pass. Ruff clean.	2026-05-23 16:21:47 +03:00
lekss361	962daf2d47	feat(tradein): yandex_helpers.py — JSON-LD + regex + date + NLP + money helpers (#456)	2026-05-23 13:14:11 +00:00
lekss361	1b3edff6a9	feat(tradein): extend base.py save_listings to persist Cian-specific columns (PR #450 follow-up) (#455)	2026-05-23 13:05:20 +00:00
lekss361	4f0319297e	feat(tradein): cian.py SERP refactor — state-based extraction (137 fields, no jitter) (#450)	2026-05-23 12:52:04 +00:00
lekss361	5258a8f023	feat(tradein): avito_houses.py — Houses Catalog parser (state + reviews + history + recs) (#449)	2026-05-23 12:37:24 +00:00
lekss361	9ba776f2ca	feat(tradein): cian_state_parser shared utility + ScrapedLot Cian fields (#447)	2026-05-23 12:35:01 +00:00
lekss361	7cdde30b13	feat(tradein): avito_detail.py — detail page parser (30+ fields incl. domoteka) (#443)	2026-05-23 12:17:15 +00:00
lekss361	6a2f734356	feat(tradein): avito_imv.py — Avito IMV evaluation API client (2 requests) (#444)	2026-05-23 12:16:47 +00:00
lekss361	47f4cc7d58	refactor(tradein): avito search — remove jitter (C-5) + mobile API dead code + parse houseLinks (#441)	2026-05-23 11:55:18 +00:00
Light1YT	1d788b27d0	fix(tradein): Avito-парсер тащит фото и реальный адрес из DOM-карточек ... DOM-путь (_dom_card_to_lot) — основной для Avito: mobile API отдаёт бан, формат __initialState__ устарел, так что почти все объявления идут через него. Но он не извлекал фото вообще, а адрес брал мёртвым селектором [class*="geo-address"] — поэтому объявления Avito оставались без фото (серый блок в Сводке) и с заглушкой «Екатеринбург (Avito)». Теперь: - фото — из <img itemprop="image"> карточки-слайдера (до 5 шт); - адрес — первый <p> внутри data-marker="item-location" (улица + дом). Селекторы проверены на живой выдаче Avito.	2026-05-22 13:49:09 +05:00
TradeIn Deploy	7cc8d8370d	feat(tradein): пагинация Циан/N1 3→8 страниц — плотность данных ... После перехода на реальные координаты стало видно что данных тонко. 8 страниц × 4 комнаты ≈ 900+ лотов на источник вместо ~300.	2026-05-21 14:34:29 +03:00
TradeIn Deploy	d2a740779d	fix(tradein): реальные координаты Cian/N1 через геокодинг (был баг 0 аналогов) ... Cian/N1 ставили anchor+jitter координаты — фейковые, вокруг центрального якоря. Для адресов вдали от центра ST_DWithin не находил аналоги → 0. Теперь Cian/N1 оставляют lat/lon=None, geocode-missing проставляет реальные координаты по адресу (geom пересчитывается триггером). cron вызывает geocode-missing после скрейпа. Адрес чистится от суффикса · р-н.	2026-05-21 13:14:33 +03:00
TradeIn Deploy	f7b820984b	feat(tradein): радиус 1000м + пагинация Циан/N1 ... DEFAULT_RADIUS_M 800→1000 (по встрече Птицы: локация до 800-1000м). Cian/N1 multi-room теперь обходят 3 страницы выдачи → ~2-3x больше лотов.	2026-05-21 11:59:16 +03:00
TradeIn Deploy	42a0c31f27	feat(tradein): N1.ru scraper + cron для 4 источников ... Добавлен 4-й источник N1.ru (ekaterinburg.n1.ru) — региональный портал, хорошо представлен на Урале. DOM-парсинг карточек data-test=offers-list-item через curl_cffi. Multi-room сегментация 1/2/3/4к. cron-scrape.sh переписан под прод: запрос к backend через docker exec (без публичного admin/scrape), все 4 источника (avito/cian/yandex/n1).	2026-05-21 11:44:32 +03:00
lekss361	02267d0306	feat: add tradein-mvp subproject (Trade-In Estimator под /trade-in) ... - FastAPI backend: PostGIS estimator + 3 scrapers (Avito/Cian/Yandex) - Next.js 15 frontend: tradein.html mockup design, basePath=/trade-in - WeasyPrint PDF (Брусника-style 4-page report) - Address autocomplete с typo-tolerance + 6 EKB presets - Изолированный docker stack gendesign-tradein (отдельная postgres БД) - Caddy inline routes: gendsgn.ru/trade-in/* и /trade-in/api/v1/* - Forgejo Actions: .forgejo/workflows/deploy-tradein.yml (shell-based GHCR login) - Триггер только по paths: tradein-mvp/** (не пересекается с deploy.yml) - Образы: ghcr.io/lekss361/gendesign-tradein-{backend,frontend}:latest Первый запуск на сервере (вручную, один раз): - создать /opt/gendesign/tradein-mvp/.env.runtime (postgres pwd, contact email) - docker network create gendesign_shared (если нет) - docker compose -p gendesign-tradein up -d - docker compose -p gendesign exec caddy caddy reload	2026-05-21 00:25:39 +03:00