81 commits

Author	SHA1	Message	Date
lekss361	3ce76b4beb	fix(tradein): floor/total_floors NULL — migration + PDF defaults ... Bot review #558 нашёл 🔴: trade_in_estimates.floor/total_floors имеют NOT NULL constraint в DDL (001_trade_in_estimates.sql) — INSERT в estimator.py:893 и _empty_estimate шлют payload.floor=None → psycopg.errors.NotNullViolation на user-submit без этажа. - new migration 065: DROP NOT NULL для обеих колонок (idempotent) - PDF exporter: fallback "—" вместо "0" при missing floor/total_floors - estimator.py:744 style: `is not None` для consistency с pre-call gates - integration test: estimate без floor не крашится Bot reviewer ⚠️ findings addressed: 🔴 NOT NULL DB violation — FIXED via migration 065 🟠 PDF "0 / 0" regression — FIXED via or "—" 🟡 style inconsistency — FIXED 🟢 test coverage gap — FIXED via test_estimator_floor_optional.py	2026-05-25 00:36:49 +03:00
lekss361	8c540ecec6	fix(tradein): extract_street_name handles Nominatim reverse format (#557)	2026-05-24 21:03:42 +00:00
lekss361	a63e459e25	feat(tradein): GET /street-deals — ДКП-сделки Росреестра по улице target адреса (#555)	2026-05-24 20:05:48 +00:00
lekss361	6bbb026cb4	fix(yandex-valuation): off-by-one cell index in _parse_row_cells (#552)	2026-05-24 19:57:17 +00:00
lekss361	596842df8d	fix(tradein): filter ДКП-only in rosreestr importer + re-enable deals (#549)	2026-05-24 19:42:28 +00:00
lekss361	f7a5c5d4f7	fix(yandex-valuation): curl_cffi chrome120 — Yandex gates SSR on Chrome TLS (#545)	2026-05-24 16:42:07 +00:00
lekss361	8a88ea6508	refactor(yandex-valuation): DOM-based parser via .OffersArchiveSearchOffers__row (#544)	2026-05-24 16:25:56 +00:00
lekss361	3ed58e3e7a	fix(yandex-valuation): relax area regex lookbehind (2-комн regression) (#542)	2026-05-24 16:02:06 +00:00
lekss361	490bc8ca5d	fix(yandex-valuation): tighten area regex + chronological date ordering (#541)	2026-05-24 15:37:48 +00:00
lekss361	9505471129	fix(yandex-valuation): cap area_m2 sanity (drop >10000 or ≤0) (#538)	2026-05-24 15:14:13 +00:00
lekss361	ee2262d605	fix(tradein-phase-c): house_type 'monolith'→'monolithic' + log 400 body (#537)	2026-05-24 15:05:49 +00:00
lekss361	08cf615ad7	feat(tradein/cian): surface valuation chart + rent + change-pct on /trade-in (#530)	2026-05-24 14:32:54 +00:00
lekss361	c8675ce6e2	feat(yandex-valuation): link history rows to houses via match_or_create_house (#531)	2026-05-24 14:28:30 +00:00
lekss361	c6ab419100	feat(tradein-sql): bootstrap houses + link 98.8% of listings (Phase A+B) (#527)	2026-05-24 14:17:32 +00:00
lekss361	7ef4e91e50	fix(yandex-valuation): parse removed_date, fix total_floors regex, normalize NBSP (#526)	2026-05-24 14:12:16 +00:00
lekss361	0f7d6c4996	fix(tradein-imv): adapt to Avito 3-step API contract (2026-05+) (#524)	2026-05-24 13:48:52 +00:00
lekss361	bad23095cf	fix(tradein/cian): sanitize valuation cache writes (047 UNIQUE + 026/044 cols) (#523)	2026-05-24 13:44:39 +00:00
lekss361	45ee0a2945	fix(tradein-estimator): extend cohort filter to Tier S + Tier H, fix late_soviet overlap (#522)	2026-05-24 13:14:59 +00:00
lekss361	7cc7fafeeb	feat(tradein-estimator): cohort filter + Tier 0 for analog selection quality (#519)	2026-05-24 13:04:58 +00:00
lekss361	a0d50bd282	feat(tradein-estimator): tiered house-match S→H→W (#507)	2026-05-24 12:41:12 +00:00
lekss361	6cda0cf8d3	fix(tradein-estimator): restore CAST in IS NOT NULL predicates (revert audit #9, prod fix) (#518)	2026-05-24 12:35:28 +00:00
lekss361	9402702f32	feat(tradein-estimator): unique-address-based confidence threshold (#513)	2026-05-24 11:52:00 +00:00
lekss361	0767fb7401	fix(tradein-photos): sanitize uploaded images via Pillow re-encode (#511)	2026-05-24 11:32:04 +00:00
lekss361	7d8c22ff39	fix(tradein-estimator): atomic IMV link, batched yandex history, drop redundant CAST (#509)	2026-05-24 11:18:12 +00:00
lekss361	d4cbd4bb9f	feat(tradein-scrapers): parse listing_date from Avito + Yandex (#508)	2026-05-24 11:17:13 +00:00
lekss361	7f23a9b3f1	feat(tradein-imv): retry with cleaned address + categorize errors (#505)	2026-05-24 11:04:24 +00:00
lekss361	6bfa686f16	feat(tradein-estimator): exclude rosreestr deals from output (#504)	2026-05-24 10:52:55 +00:00
lekss361	8fc309d9ad	fix(tradein-avito): strip Emotion CSS from listings.address (#502)	2026-05-24 10:52:50 +00:00
lekss361	1026165abb	fix(tradein-matching): pg_advisory_xact_lock to prevent duplicate house INSERTs (#501)	2026-05-24 10:52:10 +00:00
lekss361	94cf199276	feat(tradein): postgres_fdw live read of gendesign.cad_buildings (replaces snapshot) (#493) ... Replaces tradein.cad_buildings snapshot with live postgres_fdw foreign table reading gendesign.v_tradein_cad_buildings. Fixes /trade-in/api/v1/geocode/reverse 500 (Nominatim ban) and address_not_geocoded for cadastre addresses (e.g. Хохрякова 81). Security (deep-review fixes): - 100_tradein_fdw_role.sql: passwordless CREATE ROLE; password set by deploy.yml ALTER ROLE bootstrap reading GENDESIGN_FDW_PASSWORD from backend/.env.runtime (via psql :'pw' var → format %L — injection-safe). - core/fdw.py: regex whitelist [A-Za-z0-9_-]{32,256} on password, ValueError without echoing value, try/rollback on commit. - 060_postgres_fdw_extension.sql: connect_timeout='3' on FOREIGN SERVER + ALTER ADD/SET fallback. - geocoder.py: _cadastral_forward_sync / _cadastral_reverse_sync wrapped in asyncio.to_thread. - 100_*.sql: REVOKE ALL ON ALL TABLES/SEQUENCES/FUNCTIONS IN SCHEMA public; only GRANT SELECT on v_tradein_cad_buildings. - pg_user_mappings query handles PUBLIC mapping (usename IS NULL). Tests: 3 SQL-injection guards on ensure_fdw_user_mapping + rewritten cadastral suite.	2026-05-24 08:57:30 +00:00
lekss361	0efd2dca1d	feat(tradein): cadastral reverse geocode via main backend cad_buildings (#492)	2026-05-24 07:31:12 +00:00
lekss361	7874a5c97b	feat(tradein): source-quota in estimator analogs (fix Cian/Yandex starvation) (#491)	2026-05-23 20:59:58 +00:00
lekss361	222389cba9	fix(tradein/avito): anti-bot hardening — sleep + abort + shared session (#487)	2026-05-23 20:08:40 +00:00
lekss361	52716a46c6	fix(tradein): geocode backfill + remove Avito exclusion from estimator (#490)	2026-05-23 20:03:48 +00:00
lekss361	e79c9cec88	feat(tradein): global scraper delay setting (applies across all scrapers) (#485) ... Add global request delay floor + per-source delay management. - 054: seed 'global'=0 + per-source rows (avito=7, cian=5, n1=5, domrf=5, rosreestr=5) using ON CONFLICT DO NOTHING; reuses existing 053 table from #484 - scraper_settings.py: get_scraper_delay() = max(per_source, global); _GLOBAL_KEY, refactored _get_setting_cached(); preserved yandex umbrella alias logic - admin.py: GET /scraper-settings + PUT /scraper-settings/{source} (ge=0.0, le=60.0); cache invalidation on update; CAST(:d AS numeric) per psycopg v3 - avito/cian/n1 scrapers wired (yandex was wired by #484) - 18 unit tests pass (10 original updated for max() + 8 new global delay tests) Closes follow-up gap from #486 (Cian admin slider min=0 now accepted by API).	2026-05-23 15:43:28 +00:00
lekss361	b21d7c7c85	feat(tradein): scraper_settings live-config + Yandex admin trigger endpoints (#484)	2026-05-23 15:28:34 +00:00
lekss361	0202720096	feat(tradein): in-app scheduler — UI-managed schedule (replaces SSH crontab) (#482)	2026-05-23 14:54:25 +00:00
lekss361	99dd403c45	fix(tradein): CIAN_REQUIRED_COOKIES — add real auth cookies (DMIR_AUTH + Cian session) (#480) ... Updated CIAN_REQUIRED_COOKIES filter в cian_session.py по реальному DevTools-дампу logged-in cian.ru сессии (2026-05-23). Добавлены critical auth cookies: DMIR_AUTH (httpOnly), _CIAN_GK, _yasc, _ym_visorc, uxfb_card_satisfaction. Legacy entries сохранены как backward-compat fallback. Regression test test_required_cookies_includes_real_auth гарантирует что все реальные cookies проходят filter. Fixes Stage 4 PR #457 (filter guessed без real probe — DMIR_AUTH был dropped → verify_session falsey).	2026-05-23 14:52:24 +00:00
lekss361	9643979d2c	feat(tradein): Phase 3.2 — /api/v1/search endpoint + Redis cache (#479) ... POST /api/v1/search reading listings_search_mv with ~20 cross-source filters, parameterized SQL (CAST(:x AS type), psycopg v3), whitelisted ORDER BY (Pydantic Literal), Redis 5min hot cache with graceful degradation (singleton pool via lru_cache). Verified vs data/sql/050_search_optimization.sql: matview column refs (total_area, lng, house_rating, sources[], has_avito/cian/yandex), SQL injection safety, cache failure swallow, router prefix. Deep-code-reviewer: APPROVE.	2026-05-23 14:44:22 +00:00
lekss361	2914ff2ef2	feat(tradein): city sweep — auto ЕКБ pipeline (#477) ... Stage 4d: full city sweep ЕКБ (5 anchors × N pages + enrichment + cooperative cancel). - avito.py fetch_around(pages=N) — pagination backward compat default=1 - scrape_pipeline.run_avito_city_sweep — anchors loop + heartbeat + cancel check - scrape_runs.py utility (create/heartbeat/done/failed/cancelled/list_recent) - admin.py 3 endpoints: start (BackgroundTasks) / runs / cancel - migration 051 ADD COLUMN IF NOT EXISTS + DROP/ADD CHECK constraint (text+CHECK, не enum — safe в transaction) - deploy/avito-city-sweep.sh с random delay 0-3h (anti-pattern) - 16 offline tests Deep review approved: migration idempotent, cancel priority correct (mark_done guarded by status='running'), no SQL injection, psycopg v3 compliant.	2026-05-23 14:38:07 +00:00
lekss361	37c4574d0a	feat(tradein): estimator — Cian Valuation as on-demand 7th source (Stage 9) (#476)	2026-05-23 14:29:52 +00:00
lekss361	b26e0ff7c3	feat(tradein): estimator — Yandex Valuation as on-demand 6th source (#473) ... Stage 8 of YandexRealtyScraper v1. Adds Yandex Valuation (anonymous, no cookies) as 6th on-demand source in estimate_quality. Mirrors Avito IMV pattern: 24h cache via external_valuations(source='yandex_valuation', cache_key) [UNIQUE 026:34], idempotent history persistence via house_placement_history(source, ext_item_id) [UNIQUE 017:50] with synthesized SHA256(address|date|area|floor|prices)[:32] ext_item_id. Graceful degradation on geocode/fetch/cache/save errors. house_id=NULL — matching deferred to Stage 8.x. 11 new tests + 4 IMV regression = 15 green.	2026-05-23 14:14:49 +00:00
lekss361	89c4a2842d	feat(tradein): cross-source matching service (3-tier: cadastr / fingerprint / geo / composite) (#470)	2026-05-23 14:12:16 +00:00
lekss361	7d649f9239	feat(tradein): conflict_resolution — register Yandex sources in priority dicts (#471) ... Stage 7/9 YandexRealtyScraper v1 (Wave 5). Integrates 4 Yandex scrapers into cross-source field-priority arbitration. - HOUSE_FIELD_PRIORITY: lat/lon → yandex_realty_nb; +year_built/house_type/house_class/rating_score/reviews_count Yandex sources - New house fields: text_reviews_count, corpus_count, total_area_ha, commission_year/month, developer_name, has_panorama, yandex_total_listings, has_lift, ceiling_height - LISTING_FIELD_PRIORITY: description/house_type +yandex_detail - New listing fields (all yandex_detail): agency_name/founded_year/objects_count, views_total_yandex, publish_date_relative, sale_type_text - 37 unit tests (7 pre-existing + 30 new), all pass; ruff clean - Resolver engine _resolve() unchanged (this PR creates the file initially) Refs: decisions/Cross_Source_Matching_Strategy.md sec 3.5+4.5, decisions/YandexRealtyScraper_v1_Implementation_Plan.md Stage 7.	2026-05-23 14:03:35 +00:00
lekss361	51650276b2	feat(tradein): yandex_newbuilding.py — ЖК landing parser (149 selectors + NLP) (#467)	2026-05-23 13:46:45 +00:00
lekss361	7056ecb23b	feat(tradein): yandex_valuation.py — anonymous house-history scraper (#468)	2026-05-23 13:46:20 +00:00
lekss361	3d90221fa0	feat(tradein): yandex_detail.py — Product JSON-LD + DOM detail parser (#466) ... Stage 4 of YandexRealtyScraper v1. YandexDetailScraper.fetch_detail(offer_url) extracts authoritative price from Product JSON-LD (offers.price exact int) + DOM sections (description, agent, stats, metro, photos 8 sizes, newbuilding link, NLP). 35 unit tests, ruff clean. Photo URLs sanitized downstream via _safe_url CDN allowlist (avatars.mds.yandex.net).	2026-05-23 13:45:09 +00:00
lekss361	b0a7235110	feat(tradein): cian_valuation.py — auth-required Calculator scraper + 24h cache (#465)	2026-05-23 13:43:34 +00:00
lekss361	29f94963a7	Merge pull request 'feat(tradein): yandex_realty.py — DOM SERP refactor (vtorichka, 23 cards/page)' (#462) from feat/tradein-yandex-serp-refactor into main	2026-05-23 13:30:36 +00:00
lekss361	d28bf4e66a	feat(tradein): cian_newbuilding.py — ЖК catalog scraper (chart + reliability + offers, Wave 4 Worker C) (#459) ... Stage 6 / Wave 4 Worker C of CianScraper v1. Cian.ru newbuilding (ЖК) catalog page scraper — extracts newbuilding metadata + 6 sister state containers (realtyValuation chart / reliability / reviews / offers / builders / housesByTurn). Main Stage 6 deliverable: 7-month price dynamics chart → houses_price_dynamics table (cross-source benchmark для Stage 9 estimator). Files: - app/services/scrapers/cian_newbuilding.py (NEW, 548 LOC) - tests/test_cian_newbuilding.py (NEW, 357 LOC, 20 tests) Reviewed: schema match (migrations 020/021/025/029), ON CONFLICT constraint name verified (houses_price_dynamics_dim_key), graceful chart extraction (empty/None/fallback shapes), atomic transaction. Known minor follow-ups (non-blocking): - management_companies UPSERT: ext_id=NULL → potential duplicates (Postgres NULL!=NULL); bounded, document in vault - offers items[] fast path: no dict-only filter / no room_count injection - houses_price_dynamics primary shape stores total price into price_per_sqm column with prices_type='price' discriminator — Stage 9 estimator must respect prices_type filter - house_reliability_checks no UNIQUE → unbounded growth on rescrape (deferred to future migration 030+) Verdict: APPROVE (deep-code-reviewer).	2026-05-23 13:23:11 +00:00