26 commits

Author	SHA1	Message	Date
lekss361	0202720096	feat(tradein): in-app scheduler — UI-managed schedule (replaces SSH crontab) (#482)	2026-05-23 14:54:25 +00:00
lekss361	99dd403c45	fix(tradein): CIAN_REQUIRED_COOKIES — add real auth cookies (DMIR_AUTH + Cian session) (#480) ... Updated CIAN_REQUIRED_COOKIES filter в cian_session.py по реальному DevTools-дампу logged-in cian.ru сессии (2026-05-23). Добавлены critical auth cookies: DMIR_AUTH (httpOnly), _CIAN_GK, _yasc, _ym_visorc, uxfb_card_satisfaction. Legacy entries сохранены как backward-compat fallback. Regression test test_required_cookies_includes_real_auth гарантирует что все реальные cookies проходят filter. Fixes Stage 4 PR #457 (filter guessed без real probe — DMIR_AUTH был dropped → verify_session falsey).	2026-05-23 14:52:24 +00:00
lekss361	9643979d2c	feat(tradein): Phase 3.2 — /api/v1/search endpoint + Redis cache (#479) ... POST /api/v1/search reading listings_search_mv with ~20 cross-source filters, parameterized SQL (CAST(:x AS type), psycopg v3), whitelisted ORDER BY (Pydantic Literal), Redis 5min hot cache with graceful degradation (singleton pool via lru_cache). Verified vs data/sql/050_search_optimization.sql: matview column refs (total_area, lng, house_rating, sources[], has_avito/cian/yandex), SQL injection safety, cache failure swallow, router prefix. Deep-code-reviewer: APPROVE.	2026-05-23 14:44:22 +00:00
lekss361	2914ff2ef2	feat(tradein): city sweep — auto ЕКБ pipeline (#477) ... Stage 4d: full city sweep ЕКБ (5 anchors × N pages + enrichment + cooperative cancel). - avito.py fetch_around(pages=N) — pagination backward compat default=1 - scrape_pipeline.run_avito_city_sweep — anchors loop + heartbeat + cancel check - scrape_runs.py utility (create/heartbeat/done/failed/cancelled/list_recent) - admin.py 3 endpoints: start (BackgroundTasks) / runs / cancel - migration 051 ADD COLUMN IF NOT EXISTS + DROP/ADD CHECK constraint (text+CHECK, не enum — safe в transaction) - deploy/avito-city-sweep.sh с random delay 0-3h (anti-pattern) - 16 offline tests Deep review approved: migration idempotent, cancel priority correct (mark_done guarded by status='running'), no SQL injection, psycopg v3 compliant.	2026-05-23 14:38:07 +00:00
lekss361	37c4574d0a	feat(tradein): estimator — Cian Valuation as on-demand 7th source (Stage 9) (#476)	2026-05-23 14:29:52 +00:00
lekss361	b26e0ff7c3	feat(tradein): estimator — Yandex Valuation as on-demand 6th source (#473) ... Stage 8 of YandexRealtyScraper v1. Adds Yandex Valuation (anonymous, no cookies) as 6th on-demand source in estimate_quality. Mirrors Avito IMV pattern: 24h cache via external_valuations(source='yandex_valuation', cache_key) [UNIQUE 026:34], idempotent history persistence via house_placement_history(source, ext_item_id) [UNIQUE 017:50] with synthesized SHA256(address|date|area|floor|prices)[:32] ext_item_id. Graceful degradation on geocode/fetch/cache/save errors. house_id=NULL — matching deferred to Stage 8.x. 11 new tests + 4 IMV regression = 15 green.	2026-05-23 14:14:49 +00:00
lekss361	89c4a2842d	feat(tradein): cross-source matching service (3-tier: cadastr / fingerprint / geo / composite) (#470)	2026-05-23 14:12:16 +00:00
lekss361	7d649f9239	feat(tradein): conflict_resolution — register Yandex sources in priority dicts (#471) ... Stage 7/9 YandexRealtyScraper v1 (Wave 5). Integrates 4 Yandex scrapers into cross-source field-priority arbitration. - HOUSE_FIELD_PRIORITY: lat/lon → yandex_realty_nb; +year_built/house_type/house_class/rating_score/reviews_count Yandex sources - New house fields: text_reviews_count, corpus_count, total_area_ha, commission_year/month, developer_name, has_panorama, yandex_total_listings, has_lift, ceiling_height - LISTING_FIELD_PRIORITY: description/house_type +yandex_detail - New listing fields (all yandex_detail): agency_name/founded_year/objects_count, views_total_yandex, publish_date_relative, sale_type_text - 37 unit tests (7 pre-existing + 30 new), all pass; ruff clean - Resolver engine _resolve() unchanged (this PR creates the file initially) Refs: decisions/Cross_Source_Matching_Strategy.md sec 3.5+4.5, decisions/YandexRealtyScraper_v1_Implementation_Plan.md Stage 7.	2026-05-23 14:03:35 +00:00
lekss361	51650276b2	feat(tradein): yandex_newbuilding.py — ЖК landing parser (149 selectors + NLP) (#467)	2026-05-23 13:46:45 +00:00
lekss361	7056ecb23b	feat(tradein): yandex_valuation.py — anonymous house-history scraper (#468)	2026-05-23 13:46:20 +00:00
lekss361	3d90221fa0	feat(tradein): yandex_detail.py — Product JSON-LD + DOM detail parser (#466) ... Stage 4 of YandexRealtyScraper v1. YandexDetailScraper.fetch_detail(offer_url) extracts authoritative price from Product JSON-LD (offers.price exact int) + DOM sections (description, agent, stats, metro, photos 8 sizes, newbuilding link, NLP). 35 unit tests, ruff clean. Photo URLs sanitized downstream via _safe_url CDN allowlist (avatars.mds.yandex.net).	2026-05-23 13:45:09 +00:00
lekss361	b0a7235110	feat(tradein): cian_valuation.py — auth-required Calculator scraper + 24h cache (#465)	2026-05-23 13:43:34 +00:00
lekss361	b5d206bfa8	feat(tradein): Stage 4a — API endpoints for Avito enrichment + IMV benchmark UI (#460)	2026-05-23 13:34:38 +00:00
lekss361	29f94963a7	Merge pull request 'feat(tradein): yandex_realty.py — DOM SERP refactor (vtorichka, 23 cards/page)' (#462) from feat/tradein-yandex-serp-refactor into main	2026-05-23 13:30:36 +00:00
lekss361	d28bf4e66a	feat(tradein): cian_newbuilding.py — ЖК catalog scraper (chart + reliability + offers, Wave 4 Worker C) (#459) ... Stage 6 / Wave 4 Worker C of CianScraper v1. Cian.ru newbuilding (ЖК) catalog page scraper — extracts newbuilding metadata + 6 sister state containers (realtyValuation chart / reliability / reviews / offers / builders / housesByTurn). Main Stage 6 deliverable: 7-month price dynamics chart → houses_price_dynamics table (cross-source benchmark для Stage 9 estimator). Files: - app/services/scrapers/cian_newbuilding.py (NEW, 548 LOC) - tests/test_cian_newbuilding.py (NEW, 357 LOC, 20 tests) Reviewed: schema match (migrations 020/021/025/029), ON CONFLICT constraint name verified (houses_price_dynamics_dim_key), graceful chart extraction (empty/None/fallback shapes), atomic transaction. Known minor follow-ups (non-blocking): - management_companies UPSERT: ext_id=NULL → potential duplicates (Postgres NULL!=NULL); bounded, document in vault - offers items[] fast path: no dict-only filter / no room_count injection - houses_price_dynamics primary shape stores total price into price_per_sqm column with prices_type='price' discriminator — Stage 9 estimator must respect prices_type filter - house_reliability_checks no UNIQUE → unbounded growth on rescrape (deferred to future migration 030+) Verdict: APPROVE (deep-code-reviewer).	2026-05-23 13:23:11 +00:00
lekss361	2dd76ea5e8	feat(tradein): cian_session cookies management + admin endpoints (Wave 4 Worker A) (#457)	2026-05-23 13:22:00 +00:00
lekss361	ead2ac16c5	feat(tradein): yandex_realty.py — DOM SERP refactor (vtorichka, 23 cards/page) ... Replace dead JSON-state extraction (Yandex moved to chunked/ephemeral state post-hydration) with selectolax DOM parsing using helpers from PR #456. Changes: - New URL: /{city}/kupit/kvartira/vtorichniy-rynok/?page=N (path-based, no geo) - Selector: [data-test="OffersSerpItem"] x ~23 per page - Per-card extraction: offer_id (URL regex), title (rooms/area/floor regex), price/ppm2 (regex + parse_rub), bargain flag, publish_date (parse_ru_date), street address, photos (avatars.mds.yandex domain filter + size upgrade), newbuilding linkage (house_source='yandex_realty_nb' + house_ext_id) - ScrapedLot mapping: lat/lon=None (no coords on SERP — geocode-missing fills later), listing_segment="vtorichka" - Pagination: fetch_around_multi_room -> up to MAX_PAGES=3 pages, dedup by source_id. **kwargs swallow keeps old rooms/sort/pages callers compat. - Preserved: BaseScraper interface, name="yandex" (matches existing rows + avoids breaking dedup_hash; Wave 5 conflict resolution can alias if needed), request_delay_sec=5.0, sleep_between_requests, tenacity retry. Tests: 10 unit tests against hand-crafted fixture HTML (single card / studio / no-price / empty page / multi-card / URL builder / city override) — all pass. Ruff clean.	2026-05-23 16:21:47 +03:00
lekss361	962daf2d47	feat(tradein): yandex_helpers.py — JSON-LD + regex + date + NLP + money helpers (#456)	2026-05-23 13:14:11 +00:00
lekss361	5051ebde2e	feat(tradein): integrate Avito IMV as 5th evaluation source (on-demand cached) (#452)	2026-05-23 13:08:22 +00:00
lekss361	1b3edff6a9	feat(tradein): extend base.py save_listings to persist Cian-specific columns (PR #450 follow-up) (#455)	2026-05-23 13:05:20 +00:00
lekss361	eb4764f75a	feat(tradein): scrape_pipeline.py — Avito full orchestrator (search → houses → detail) (#451)	2026-05-23 12:55:13 +00:00
lekss361	4f0319297e	feat(tradein): cian.py SERP refactor — state-based extraction (137 fields, no jitter) (#450)	2026-05-23 12:52:04 +00:00
lekss361	5258a8f023	feat(tradein): avito_houses.py — Houses Catalog parser (state + reviews + history + recs) (#449)	2026-05-23 12:37:24 +00:00
lekss361	9ba776f2ca	feat(tradein): cian_state_parser shared utility + ScrapedLot Cian fields (#447)	2026-05-23 12:35:01 +00:00
lekss361	7cdde30b13	feat(tradein): avito_detail.py — detail page parser (30+ fields incl. domoteka) (#443)	2026-05-23 12:17:15 +00:00
lekss361	6a2f734356	feat(tradein): avito_imv.py — Avito IMV evaluation API client (2 requests) (#444)	2026-05-23 12:16:47 +00:00