102 commits

Author	SHA1	Message	Date
lekss361	3ce76b4beb	fix(tradein): floor/total_floors NULL — migration + PDF defaults ... Bot review #558 нашёл 🔴: trade_in_estimates.floor/total_floors имеют NOT NULL constraint в DDL (001_trade_in_estimates.sql) — INSERT в estimator.py:893 и _empty_estimate шлют payload.floor=None → psycopg.errors.NotNullViolation на user-submit без этажа. - new migration 065: DROP NOT NULL для обеих колонок (idempotent) - PDF exporter: fallback "—" вместо "0" при missing floor/total_floors - estimator.py:744 style: `is not None` для consistency с pre-call gates - integration test: estimate без floor не крашится Bot reviewer ⚠️ findings addressed: 🔴 NOT NULL DB violation — FIXED via migration 065 🟠 PDF "0 / 0" regression — FIXED via or "—" 🟡 style inconsistency — FIXED 🟢 test coverage gap — FIXED via test_estimator_floor_optional.py	2026-05-25 00:36:49 +03:00
lekss361	c60d394ff0	feat(tradein): этаж/этажность optional + best test presets по deal count ... UX: floor и total_floors теперь optional — все 6 мест в estimator.py уже имеют None-guards (Tier H/IMV/Cian валидируют наличие до вызова). При None — Tier H/IMV/Cian skip, оценка через Tier W (wide) с понижением confidence до medium/low. Critical guard: tradein.deals также не имеют жёстких NOT NULL на этих полях — schema-only change достаточен. Test presets: заменены на top-6 ЕКБ улиц по deal_count за 12 мес (rosreestr_deals ДКП квартиры): - Академика Ландау (686 сделок/год, Академический новостройки) - Малышева 84 (центр классика — оставлен) - Краснолесья (487 сделок/год, Академический эконом) - Космонавтов 50 (415 сделок/год, Уралмаш бюджет) - 8 Марта 80 (305 сделок/год, центр Ленинский) - Московская (426 сделок/год, центр премиум) Разброс: 4 района, размеры 35-75м², 1-3 комнаты, цены 120-185 К ₽/м². Все адреса дают medium+ confidence из-за объёма ДКП-сделок на улице.	2026-05-25 00:20:08 +03:00
lekss361	8c540ecec6	fix(tradein): extract_street_name handles Nominatim reverse format (#557)	2026-05-24 21:03:42 +00:00
lekss361	a63e459e25	feat(tradein): GET /street-deals — ДКП-сделки Росреестра по улице target адреса (#555)	2026-05-24 20:05:48 +00:00
lekss361	6bbb026cb4	fix(yandex-valuation): off-by-one cell index in _parse_row_cells (#552)	2026-05-24 19:57:17 +00:00
lekss361	0ac54d1de0	feat(tradein): sell-time sensitivity block (4 цены × медиана срока продажи) ... Backend: GET /api/v1/trade-in/estimate/{id}/sell-time-sensitivity - 4 фиксированных бакета по premium к медиане года: −5% / медиана (±3%) / +5% / +10% - Median exposure_days + p25/p75 per bucket - Filter outliers: last_price > start_price * 0.7 (отбрасываем подозрительно дешёвые) - Benchmark: median ₽/м² за последние 2 года Frontend: новый SellTimeSensitivity component (4-card grid) - color-coded buckets (green→blue→yellow→red) - median highlighted синей рамкой - показывает "~31 дн" + "обычно 31-61 дн" + count лотов - встроен в HouseAnalyticsSection между KpiRow и PriceHistoryChart Использует существующий house_placement_history (15k+ rows).	2026-05-24 22:43:32 +03:00
lekss361	596842df8d	fix(tradein): filter ДКП-only in rosreestr importer + re-enable deals (#549)	2026-05-24 19:42:28 +00:00
lekss361	967e7185ca	feat(tradein): cian price-change badges + split chart Avito/Yandex (#547)	2026-05-24 18:24:00 +00:00
lekss361	80850a591e	feat(tradein): house analytics section (chart + KPI + sold list) (#546)	2026-05-24 18:09:45 +00:00
lekss361	f7a5c5d4f7	fix(yandex-valuation): curl_cffi chrome120 — Yandex gates SSR on Chrome TLS (#545)	2026-05-24 16:42:07 +00:00
lekss361	8a88ea6508	refactor(yandex-valuation): DOM-based parser via .OffersArchiveSearchOffers__row (#544)	2026-05-24 16:25:56 +00:00
lekss361	3ed58e3e7a	fix(yandex-valuation): relax area regex lookbehind (2-комн regression) (#542)	2026-05-24 16:02:06 +00:00
lekss361	490bc8ca5d	fix(yandex-valuation): tighten area regex + chronological date ordering (#541)	2026-05-24 15:37:48 +00:00
lekss361	ba9cc3aef1	feat(tradein/cian): valuation backfill + runner script для prod (#539)	2026-05-24 15:16:45 +00:00
lekss361	9505471129	fix(yandex-valuation): cap area_m2 sanity (drop >10000 or ≤0) (#538)	2026-05-24 15:14:13 +00:00
lekss361	ee2262d605	fix(tradein-phase-c): house_type 'monolith'→'monolithic' + log 400 body (#537)	2026-05-24 15:05:49 +00:00
lekss361	bc594e3f79	feat(tradein/cian): batch backfill task для offer_price_history + houses_price_dynamics (#535)	2026-05-24 14:55:47 +00:00
lekss361	0d10eb400f	fix(tradein-api): PlacementHistoryEntry schema + None checks (PR #528 follow-up) (#532)	2026-05-24 14:33:53 +00:00
lekss361	08cf615ad7	feat(tradein/cian): surface valuation chart + rent + change-pct on /trade-in (#530)	2026-05-24 14:32:54 +00:00
lekss361	c8675ce6e2	feat(yandex-valuation): link history rows to houses via match_or_create_house (#531)	2026-05-24 14:28:30 +00:00
lekss361	7dc11e4339	feat(tradein-api): surface backfilled houses + placement history (#528)	2026-05-24 14:22:50 +00:00
lekss361	c6ab419100	feat(tradein-sql): bootstrap houses + link 98.8% of listings (Phase A+B) (#527)	2026-05-24 14:17:32 +00:00
lekss361	7ef4e91e50	fix(yandex-valuation): parse removed_date, fix total_floors regex, normalize NBSP (#526)	2026-05-24 14:12:16 +00:00
lekss361	591039493c	feat(tradein/cian): admin endpoints + UI for detail+newbuilding history scrape (#525)	2026-05-24 14:05:55 +00:00
lekss361	0f7d6c4996	fix(tradein-imv): adapt to Avito 3-step API contract (2026-05+) (#524)	2026-05-24 13:48:52 +00:00
lekss361	bad23095cf	fix(tradein/cian): sanitize valuation cache writes (047 UNIQUE + 026/044 cols) (#523)	2026-05-24 13:44:39 +00:00
lekss361	45ee0a2945	fix(tradein-estimator): extend cohort filter to Tier S + Tier H, fix late_soviet overlap (#522)	2026-05-24 13:14:59 +00:00
lekss361	d9c6b1bd01	chore(tradein): remove dead mock-fallback code from trade_in.py (#521)	2026-05-24 13:11:13 +00:00
lekss361	7cc7fafeeb	feat(tradein-estimator): cohort filter + Tier 0 for analog selection quality (#519)	2026-05-24 13:04:58 +00:00
lekss361	a0d50bd282	feat(tradein-estimator): tiered house-match S→H→W (#507)	2026-05-24 12:41:12 +00:00
lekss361	6cda0cf8d3	fix(tradein-estimator): restore CAST in IS NOT NULL predicates (revert audit #9, prod fix) (#518)	2026-05-24 12:35:28 +00:00
lekss361	9402702f32	feat(tradein-estimator): unique-address-based confidence threshold (#513)	2026-05-24 11:52:00 +00:00
lekss361	0767fb7401	fix(tradein-photos): sanitize uploaded images via Pillow re-encode (#511)	2026-05-24 11:32:04 +00:00
lekss361	7d8c22ff39	fix(tradein-estimator): atomic IMV link, batched yandex history, drop redundant CAST (#509)	2026-05-24 11:18:12 +00:00
lekss361	d4cbd4bb9f	feat(tradein-scrapers): parse listing_date from Avito + Yandex (#508)	2026-05-24 11:17:13 +00:00
lekss361	7f23a9b3f1	feat(tradein-imv): retry with cleaned address + categorize errors (#505)	2026-05-24 11:04:24 +00:00
lekss361	6bfa686f16	feat(tradein-estimator): exclude rosreestr deals from output (#504)	2026-05-24 10:52:55 +00:00
lekss361	8fc309d9ad	fix(tradein-avito): strip Emotion CSS from listings.address (#502)	2026-05-24 10:52:50 +00:00
lekss361	1026165abb	fix(tradein-matching): pg_advisory_xact_lock to prevent duplicate house INSERTs (#501)	2026-05-24 10:52:10 +00:00
lekss361	94cf199276	feat(tradein): postgres_fdw live read of gendesign.cad_buildings (replaces snapshot) (#493) ... Replaces tradein.cad_buildings snapshot with live postgres_fdw foreign table reading gendesign.v_tradein_cad_buildings. Fixes /trade-in/api/v1/geocode/reverse 500 (Nominatim ban) and address_not_geocoded for cadastre addresses (e.g. Хохрякова 81). Security (deep-review fixes): - 100_tradein_fdw_role.sql: passwordless CREATE ROLE; password set by deploy.yml ALTER ROLE bootstrap reading GENDESIGN_FDW_PASSWORD from backend/.env.runtime (via psql :'pw' var → format %L — injection-safe). - core/fdw.py: regex whitelist [A-Za-z0-9_-]{32,256} on password, ValueError without echoing value, try/rollback on commit. - 060_postgres_fdw_extension.sql: connect_timeout='3' on FOREIGN SERVER + ALTER ADD/SET fallback. - geocoder.py: _cadastral_forward_sync / _cadastral_reverse_sync wrapped in asyncio.to_thread. - 100_*.sql: REVOKE ALL ON ALL TABLES/SEQUENCES/FUNCTIONS IN SCHEMA public; only GRANT SELECT on v_tradein_cad_buildings. - pg_user_mappings query handles PUBLIC mapping (usename IS NULL). Tests: 3 SQL-injection guards on ensure_fdw_user_mapping + rewritten cadastral suite.	2026-05-24 08:57:30 +00:00
lekss361	0efd2dca1d	feat(tradein): cadastral reverse geocode via main backend cad_buildings (#492)	2026-05-24 07:31:12 +00:00
lekss361	7874a5c97b	feat(tradein): source-quota in estimator analogs (fix Cian/Yandex starvation) (#491)	2026-05-23 20:59:58 +00:00
lekss361	222389cba9	fix(tradein/avito): anti-bot hardening — sleep + abort + shared session (#487)	2026-05-23 20:08:40 +00:00
lekss361	52716a46c6	fix(tradein): geocode backfill + remove Avito exclusion from estimator (#490)	2026-05-23 20:03:48 +00:00
lekss361	e79c9cec88	feat(tradein): global scraper delay setting (applies across all scrapers) (#485) ... Add global request delay floor + per-source delay management. - 054: seed 'global'=0 + per-source rows (avito=7, cian=5, n1=5, domrf=5, rosreestr=5) using ON CONFLICT DO NOTHING; reuses existing 053 table from #484 - scraper_settings.py: get_scraper_delay() = max(per_source, global); _GLOBAL_KEY, refactored _get_setting_cached(); preserved yandex umbrella alias logic - admin.py: GET /scraper-settings + PUT /scraper-settings/{source} (ge=0.0, le=60.0); cache invalidation on update; CAST(:d AS numeric) per psycopg v3 - avito/cian/n1 scrapers wired (yandex was wired by #484) - 18 unit tests pass (10 original updated for max() + 8 new global delay tests) Closes follow-up gap from #486 (Cian admin slider min=0 now accepted by API).	2026-05-23 15:43:28 +00:00
lekss361	b21d7c7c85	feat(tradein): scraper_settings live-config + Yandex admin trigger endpoints (#484)	2026-05-23 15:28:34 +00:00
lekss361	0202720096	feat(tradein): in-app scheduler — UI-managed schedule (replaces SSH crontab) (#482)	2026-05-23 14:54:25 +00:00
lekss361	99dd403c45	fix(tradein): CIAN_REQUIRED_COOKIES — add real auth cookies (DMIR_AUTH + Cian session) (#480) ... Updated CIAN_REQUIRED_COOKIES filter в cian_session.py по реальному DevTools-дампу logged-in cian.ru сессии (2026-05-23). Добавлены critical auth cookies: DMIR_AUTH (httpOnly), _CIAN_GK, _yasc, _ym_visorc, uxfb_card_satisfaction. Legacy entries сохранены как backward-compat fallback. Regression test test_required_cookies_includes_real_auth гарантирует что все реальные cookies проходят filter. Fixes Stage 4 PR #457 (filter guessed без real probe — DMIR_AUTH был dropped → verify_session falsey).	2026-05-23 14:52:24 +00:00
lekss361	9643979d2c	feat(tradein): Phase 3.2 — /api/v1/search endpoint + Redis cache (#479) ... POST /api/v1/search reading listings_search_mv with ~20 cross-source filters, parameterized SQL (CAST(:x AS type), psycopg v3), whitelisted ORDER BY (Pydantic Literal), Redis 5min hot cache with graceful degradation (singleton pool via lru_cache). Verified vs data/sql/050_search_optimization.sql: matview column refs (total_area, lng, house_rating, sources[], has_avito/cian/yandex), SQL injection safety, cache failure swallow, router prefix. Deep-code-reviewer: APPROVE.	2026-05-23 14:44:22 +00:00
lekss361	2914ff2ef2	feat(tradein): city sweep — auto ЕКБ pipeline (#477) ... Stage 4d: full city sweep ЕКБ (5 anchors × N pages + enrichment + cooperative cancel). - avito.py fetch_around(pages=N) — pagination backward compat default=1 - scrape_pipeline.run_avito_city_sweep — anchors loop + heartbeat + cancel check - scrape_runs.py utility (create/heartbeat/done/failed/cancelled/list_recent) - admin.py 3 endpoints: start (BackgroundTasks) / runs / cancel - migration 051 ADD COLUMN IF NOT EXISTS + DROP/ADD CHECK constraint (text+CHECK, не enum — safe в transaction) - deploy/avito-city-sweep.sh с random delay 0-3h (anti-pattern) - 16 offline tests Deep review approved: migration idempotent, cancel priority correct (mark_done guarded by status='running'), no SQL injection, psycopg v3 compliant.	2026-05-23 14:38:07 +00:00