920 commits

Author	SHA1	Message	Date
lekss361	0d10eb400f	fix(tradein-api): PlacementHistoryEntry schema + None checks (PR #528 follow-up) (#532)	2026-05-24 14:33:53 +00:00
lekss361	08cf615ad7	feat(tradein/cian): surface valuation chart + rent + change-pct on /trade-in (#530)	2026-05-24 14:32:54 +00:00
lekss361	c8675ce6e2	feat(yandex-valuation): link history rows to houses via match_or_create_house (#531)	2026-05-24 14:28:30 +00:00
lekss361	7dc11e4339	feat(tradein-api): surface backfilled houses + placement history (#528)	2026-05-24 14:22:50 +00:00
lekss361	f2f9e5b79b	feat(tradein-ui): placement history card (#529)	2026-05-24 14:22:16 +00:00
lekss361	c6ab419100	feat(tradein-sql): bootstrap houses + link 98.8% of listings (Phase A+B) (#527)	2026-05-24 14:17:32 +00:00
lekss361	7ef4e91e50	fix(yandex-valuation): parse removed_date, fix total_floors regex, normalize NBSP (#526)	2026-05-24 14:12:16 +00:00
lekss361	591039493c	feat(tradein/cian): admin endpoints + UI for detail+newbuilding history scrape (#525)	2026-05-24 14:05:55 +00:00
lekss361	0f7d6c4996	fix(tradein-imv): adapt to Avito 3-step API contract (2026-05+) (#524)	2026-05-24 13:48:52 +00:00
lekss361	bad23095cf	fix(tradein/cian): sanitize valuation cache writes (047 UNIQUE + 026/044 cols) (#523)	2026-05-24 13:44:39 +00:00
lekss361	45ee0a2945	fix(tradein-estimator): extend cohort filter to Tier S + Tier H, fix late_soviet overlap (#522)	2026-05-24 13:14:59 +00:00
lekss361	d9c6b1bd01	chore(tradein): remove dead mock-fallback code from trade_in.py (#521)	2026-05-24 13:11:13 +00:00
lekss361	7cc7fafeeb	feat(tradein-estimator): cohort filter + Tier 0 for analog selection quality (#519)	2026-05-24 13:04:58 +00:00
lekss361	92d00ec874	feat(tradein-ui): mock-disclaimer + progressive enrichment + UX polish (#517)	2026-05-24 12:58:39 +00:00
lekss361	a0d50bd282	feat(tradein-estimator): tiered house-match S→H→W (#507)	2026-05-24 12:41:12 +00:00
lekss361	6cda0cf8d3	fix(tradein-estimator): restore CAST in IS NOT NULL predicates (revert audit #9, prod fix) (#518)	2026-05-24 12:35:28 +00:00
lekss361	e45ff47aa6	fix(tradein-sql): remove duplicate CREATE INDEX from 050 (regression from PR #500) (#515)	2026-05-24 12:16:27 +00:00
lekss361	3190bbec54	fix(tradein-frontend): MapPicker focus trap + Leaflet SRI, PhotoUpload size check (#514)	2026-05-24 12:01:17 +00:00
lekss361	9402702f32	feat(tradein-estimator): unique-address-based confidence threshold (#513)	2026-05-24 11:52:00 +00:00
lekss361	3665a61e48	fix(tradein-frontend): safeUrl validator, apiFetch dedup, ListingsCard a11y, error boundary (#512)	2026-05-24 11:47:38 +00:00
lekss361	0767fb7401	fix(tradein-photos): sanitize uploaded images via Pillow re-encode (#511)	2026-05-24 11:32:04 +00:00
lekss361	7d8c22ff39	fix(tradein-estimator): atomic IMV link, batched yandex history, drop redundant CAST (#509)	2026-05-24 11:18:12 +00:00
lekss361	d4cbd4bb9f	feat(tradein-scrapers): parse listing_date from Avito + Yandex (#508)	2026-05-24 11:17:13 +00:00
lekss361	7f23a9b3f1	feat(tradein-imv): retry with cleaned address + categorize errors (#505)	2026-05-24 11:04:24 +00:00
lekss361	6bfa686f16	feat(tradein-estimator): exclude rosreestr deals from output (#504)	2026-05-24 10:52:55 +00:00
lekss361	8fc309d9ad	fix(tradein-avito): strip Emotion CSS from listings.address (#502)	2026-05-24 10:52:50 +00:00
lekss361	1026165abb	fix(tradein-matching): pg_advisory_xact_lock to prevent duplicate house INSERTs (#501)	2026-05-24 10:52:10 +00:00
lekss361	02b48b8d04	fix(tradein-sql): rename dup 030, add trade_in_estimates geom, drop dup indexes (#500) ... Close 3 SQL schema findings from 2026-05-24 trade-in audit. - Rename 030_listings_alter_yandex.sql -> 033_*.sql (resolve dup prefix with 030_avito_imv_cache_key_unique.sql). - New 034_trade_in_estimates_geom.sql: PostGIS geom column + GIST index + backfill + BEFORE INSERT/UPDATE trigger. - New 035_drop_duplicate_indexes.sql: drop listing_sources_listing_idx2 (dup of 028) and houses_geom_idx2 (dup of 009). All idempotent. Deploy script uses ls|sort + no tracking table — rename is safe because original 030 already applied (IF NOT EXISTS no-op on re-run as 033).	2026-05-24 10:04:40 +00:00
lekss361	94cf199276	feat(tradein): postgres_fdw live read of gendesign.cad_buildings (replaces snapshot) (#493) ... Replaces tradein.cad_buildings snapshot with live postgres_fdw foreign table reading gendesign.v_tradein_cad_buildings. Fixes /trade-in/api/v1/geocode/reverse 500 (Nominatim ban) and address_not_geocoded for cadastre addresses (e.g. Хохрякова 81). Security (deep-review fixes): - 100_tradein_fdw_role.sql: passwordless CREATE ROLE; password set by deploy.yml ALTER ROLE bootstrap reading GENDESIGN_FDW_PASSWORD from backend/.env.runtime (via psql :'pw' var → format %L — injection-safe). - core/fdw.py: regex whitelist [A-Za-z0-9_-]{32,256} on password, ValueError without echoing value, try/rollback on commit. - 060_postgres_fdw_extension.sql: connect_timeout='3' on FOREIGN SERVER + ALTER ADD/SET fallback. - geocoder.py: _cadastral_forward_sync / _cadastral_reverse_sync wrapped in asyncio.to_thread. - 100_*.sql: REVOKE ALL ON ALL TABLES/SEQUENCES/FUNCTIONS IN SCHEMA public; only GRANT SELECT on v_tradein_cad_buildings. - pg_user_mappings query handles PUBLIC mapping (usename IS NULL). Tests: 3 SQL-injection guards on ensure_fdw_user_mapping + rewritten cadastral suite.	2026-05-24 08:57:30 +00:00
lekss361	0efd2dca1d	feat(tradein): cadastral reverse geocode via main backend cad_buildings (#492)	2026-05-24 07:31:12 +00:00
lekss361	7874a5c97b	feat(tradein): source-quota in estimator analogs (fix Cian/Yandex starvation) (#491)	2026-05-23 20:59:58 +00:00
lekss361	222389cba9	fix(tradein/avito): anti-bot hardening — sleep + abort + shared session (#487)	2026-05-23 20:08:40 +00:00
lekss361	726e774a7f	feat(tradein-ui): Yandex admin scraper page — mirror Avito + global delay control (#488) ... Replaces 5-line stub at /scrapers/yandex with full admin page mirroring Avito. 5 sections: global delay (umbrella yandex key), search around, detail snapshot, JK newbuilding, valuation. TanStack Query hooks + apiFetch, TS strict, reuses trade-in.css. Fixup: useScraperSettings unwraps {settings:[...]} response wrapper; PUT body includes required source field (matches backend Pydantic ScraperSettingPayload). useEffect race guarded with initializedRef so 30s refetch doesn't clobber user input mid-edit. Verdict: APPROVE — deep-code-reviewer (response shape unwrap + source field both applied).	2026-05-23 20:07:48 +00:00
lekss361	52716a46c6	fix(tradein): geocode backfill + remove Avito exclusion from estimator (#490)	2026-05-23 20:03:48 +00:00
lekss361	ffb0c67c84	fix(tradein-ui): double /trade-in/ prefix in Cian page Link (PR #486 follow-up) (#489)	2026-05-23 15:56:01 +00:00
lekss361	e79c9cec88	feat(tradein): global scraper delay setting (applies across all scrapers) (#485) ... Add global request delay floor + per-source delay management. - 054: seed 'global'=0 + per-source rows (avito=7, cian=5, n1=5, domrf=5, rosreestr=5) using ON CONFLICT DO NOTHING; reuses existing 053 table from #484 - scraper_settings.py: get_scraper_delay() = max(per_source, global); _GLOBAL_KEY, refactored _get_setting_cached(); preserved yandex umbrella alias logic - admin.py: GET /scraper-settings + PUT /scraper-settings/{source} (ge=0.0, le=60.0); cache invalidation on update; CAST(:d AS numeric) per psycopg v3 - avito/cian/n1 scrapers wired (yandex was wired by #484) - 18 unit tests pass (10 original updated for max() + 8 new global delay tests) Closes follow-up gap from #486 (Cian admin slider min=0 now accepted by API).	2026-05-23 15:43:28 +00:00
lekss361	ebfd29a0a6	feat(tradein-ui): Cian scraper admin page + global request delay control (#486)	2026-05-23 15:36:07 +00:00
lekss361	b21d7c7c85	feat(tradein): scraper_settings live-config + Yandex admin trigger endpoints (#484)	2026-05-23 15:28:34 +00:00
lekss361	65029cd1a7	feat(tradein-ui): Cian cookies upload admin page (#481)	2026-05-23 15:19:42 +00:00
lekss361	fd3cb4e6ed	feat(tradein): /scrapers/avito — 6-я секция Расписание (in-app scheduler UI) (#483)	2026-05-23 14:55:31 +00:00
lekss361	0202720096	feat(tradein): in-app scheduler — UI-managed schedule (replaces SSH crontab) (#482)	2026-05-23 14:54:25 +00:00
lekss361	99dd403c45	fix(tradein): CIAN_REQUIRED_COOKIES — add real auth cookies (DMIR_AUTH + Cian session) (#480) ... Updated CIAN_REQUIRED_COOKIES filter в cian_session.py по реальному DevTools-дампу logged-in cian.ru сессии (2026-05-23). Добавлены critical auth cookies: DMIR_AUTH (httpOnly), _CIAN_GK, _yasc, _ym_visorc, uxfb_card_satisfaction. Legacy entries сохранены как backward-compat fallback. Regression test test_required_cookies_includes_real_auth гарантирует что все реальные cookies проходят filter. Fixes Stage 4 PR #457 (filter guessed без real probe — DMIR_AUTH был dropped → verify_session falsey).	2026-05-23 14:52:24 +00:00
lekss361	9643979d2c	feat(tradein): Phase 3.2 — /api/v1/search endpoint + Redis cache (#479) ... POST /api/v1/search reading listings_search_mv with ~20 cross-source filters, parameterized SQL (CAST(:x AS type), psycopg v3), whitelisted ORDER BY (Pydantic Literal), Redis 5min hot cache with graceful degradation (singleton pool via lru_cache). Verified vs data/sql/050_search_optimization.sql: matview column refs (total_area, lng, house_rating, sources[], has_avito/cian/yandex), SQL injection safety, cache failure swallow, router prefix. Deep-code-reviewer: APPROVE.	2026-05-23 14:44:22 +00:00
lekss361	dd24bd8136	feat(tradein): /scrapers/avito — 5-я секция City sweep (auto + cancel + runs list) (#478)	2026-05-23 14:40:11 +00:00
lekss361	2914ff2ef2	feat(tradein): city sweep — auto ЕКБ pipeline (#477) ... Stage 4d: full city sweep ЕКБ (5 anchors × N pages + enrichment + cooperative cancel). - avito.py fetch_around(pages=N) — pagination backward compat default=1 - scrape_pipeline.run_avito_city_sweep — anchors loop + heartbeat + cancel check - scrape_runs.py utility (create/heartbeat/done/failed/cancelled/list_recent) - admin.py 3 endpoints: start (BackgroundTasks) / runs / cancel - migration 051 ADD COLUMN IF NOT EXISTS + DROP/ADD CHECK constraint (text+CHECK, не enum — safe в transaction) - deploy/avito-city-sweep.sh с random delay 0-3h (anti-pattern) - 16 offline tests Deep review approved: migration idempotent, cancel priority correct (mark_done guarded by status='running'), no SQL injection, psycopg v3 compliant.	2026-05-23 14:38:07 +00:00
lekss361	37c4574d0a	feat(tradein): estimator — Cian Valuation as on-demand 7th source (Stage 9) (#476)	2026-05-23 14:29:52 +00:00
lekss361	c0a0bfa413	feat(tradein): Phase 1.3 backfill scripts — fingerprints + house/listing_sources (#475)	2026-05-23 14:29:39 +00:00
lekss361	dbad7a626b	fix(tradein): убрать require_admin полностью — Caddy basic_auth достаточен (#474)	2026-05-23 14:21:01 +00:00
lekss361	b26e0ff7c3	feat(tradein): estimator — Yandex Valuation as on-demand 6th source (#473) ... Stage 8 of YandexRealtyScraper v1. Adds Yandex Valuation (anonymous, no cookies) as 6th on-demand source in estimate_quality. Mirrors Avito IMV pattern: 24h cache via external_valuations(source='yandex_valuation', cache_key) [UNIQUE 026:34], idempotent history persistence via house_placement_history(source, ext_item_id) [UNIQUE 017:50] with synthesized SHA256(address|date|area|floor|prices)[:32] ext_item_id. Graceful degradation on geocode/fetch/cache/save errors. house_id=NULL — matching deferred to Stage 8.x. 11 new tests + 4 IMV regression = 15 green.	2026-05-23 14:14:49 +00:00
lekss361	89c4a2842d	feat(tradein): cross-source matching service (3-tier: cadastr / fingerprint / geo / composite) (#470)	2026-05-23 14:12:16 +00:00