57 commits

Author	SHA1	Message	Date
bot-backend	5d6238e89a	fix(tradein): 095 drop v_data_quality before dropping sale_type_text (#731) ... Prod deploy 2026-05-31 13:50 failed on 095: 'cannot drop column sale_type_text of table listings because view v_data_quality depends on it'. The view (recreated in 094) has a CTE 'SELECT * FROM listings' which records a column-level dependency on EVERY listings column incl sale_type_text — so the bare DROP COLUMN aborts the whole migration (exit 1), blocking 095/096/097 from applying. Fix: DROP VIEW v_data_quality -> DROP COLUMN sale_type_text -> recreate the view with the identical 094 DDL (sale_type_text was never an output column, only pulled in implicitly via SELECT *; after the drop, * no longer includes it). Verified v_data_quality is the only object doing SELECT * FROM listings — listings_search_mv and v_price_divergence reference explicit columns, so no other dependency blocks. 095 never applied on prod (transaction rolled back atomically, not in _schema_migrations) → next deploy re-runs the corrected file. Editing the merged migration in place is the correct recovery (a new 098 would not help: the runner hits 095 first and keeps failing). Refs #731	2026-05-31 16:57:18 +03:00
bot-backend	5b4531f4cc	perf(tradein): index hygiene — drop dup/redundant + de-partial is_active (#730) (#897) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 13:29:11 +00:00
bot-backend	c2888ac52d	feat(tradein): scheduled monthly Rosreestr new-quarter poll → ingest alert (#888) (#894) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 13:13:55 +00:00
bot-backend	fa2bd20ac6	refactor(tradein): drop dead schema — sale_type_text column + audit_log (#731) (#893) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 13:13:43 +00:00
lekss361	8df536ca5f	Merge pull request 'refactor(tradein): unify kadastr_num → cadastral_number (#732)' (#891) from feat/732-cadastral-unify into main ... Reviewed-on: #891	2026-05-31 12:47:10 +00:00
bot-backend	de92c54540	feat(tradein): СберИндекс monthly city-level price index pull (#887, data-layer) (#890) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 12:44:56 +00:00
bot-backend	42de30cd5c	refactor(tradein): unify kadastr_num -> cadastral_number (#732) ... p3 naming-consistency (cadastral cols 0% filled). Migration 094_cadastral_unify.sql (idempotent DO-block IF EXISTS): drop+recreate listings_search_mv (MV) + v_data_quality (view), RENAME deals/house_metadata kadastr_num->cadastral_number, DROP listings.kadastr_num + listings_kadastr_idx. Code SQL-string renames in search_query/estimator/base (Lot dataclass field NOT renamed). 240 tests pass. WARN: destructive DDL auto-applies to prod on deploy -> MERGE-GATE post-demo. Reviewer: verify recreated MV/view DDL vs LIVE pg_get_viewdef before apply (reconstructed from migration history 050/046, not live capture). Refs #732	2026-05-31 15:41:52 +03:00
bot-backend	2dd2d8e57e	feat(tradein): schedule yandex address backfill task (#855) (#866) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 07:31:10 +00:00
bot-backend	42007e9b04	feat(tradein): nightly task to deactivate stale avito listings (#759) (#862) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 07:19:37 +00:00
bot-backend	b736d7b7e0	feat(estimator): exclude city-centroid listings from radius analogs (Refs #769 Part E) ... Закрывает последнюю часть #769 (A1/A2/B/C/D в #798/#804). Finding #17: bare-city адреса геокодились в city-центроид и сохранялись как точные lat/lon → mislocated листинги участвовали в radius ST_DWithin-аналогах. - 089: ALTER TABLE listings ADD COLUMN IF NOT EXISTS geo_precision text (idempot). - ScrapedLot.geo_precision + проброс в save_listings. - geocode_missing.py: geo_precision='city' через существующий _geocode_is_coarse. - estimator _fetch_analogs Tier H+W: + AND (geo_precision IS DISTINCT FROM 'city'). NULL проходит (консервативно, сдвига до backfill нет). Pricing не тронут. 7 тестов; 270 passed, ruff clean.	2026-05-30 21:22:21 +03:00
bot-backend	a9e4aecb54	fix(tradein): data-layer tail A2+B+C — trgm index, FDW options, matview refresh (Refs #769) (#804) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-30 17:40:54 +00:00
bot-backend	87a5de0cae	feat(db): quarter_price_index FDW foreign table + monthly refresh (Refs #762) (#797) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-30 17:16:28 +00:00
bot-backend	6762a532eb	feat(scrapers): wire N1.ru city sweep — pipeline + scheduler + admin endpoint (#575) (#745) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-30 15:43:38 +00:00
bot-backend	3b29936245	feat(tradein): white-label branded page for client «Практика» (#657) (#672) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-29 16:09:23 +00:00
bot-backend	47fdd6e4df	fix(tradein): scope GET /history to caller, close cross-pilot data leak (#656) (#670) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-29 15:59:03 +00:00
Light1YT	a060651328	feat(tradein): daily asking→sold ratio refresh (#648 S4) (#663)	2026-05-29 13:49:36 +00:00
Light1YT	35bd0238ef	feat(tradein): estimator additive expected_sold price (#648 S3) (#661)	2026-05-29 13:37:33 +00:00
Light1YT	c7b9565cd6	feat(tradein): asking_to_sold_ratios table + derivation (#648 S2) (#660)	2026-05-29 13:18:28 +00:00
Light1YT	e925a0ba6a	feat(tradein): per-source listing price history — schema + daily snapshot (#570) (#646)	2026-05-29 12:18:04 +00:00
Light1YT	7ec19b3536	feat(scheduler): dormant Yandex Realty city-sweep (#561) (#642)	2026-05-29 08:50:10 +00:00
Light1YT	b6cddcd9a8	fix(tradein): plain natural-key dedup_hash for rosreestr deals (#576) (#638)	2026-05-29 06:56:35 +00:00
lekss361	bd3e5c8216	feat(tradein): аккаунт praktika + лимит 15 оценок/мес, чистка PRINZIP в UI (#635)	2026-05-28 22:04:58 +00:00
Light1YT	312df8536a	feat(tradein): infer repair_state from listing description text (#622) (#632)	2026-05-28 15:37:34 +00:00
Light1YT	e72cf87e35	feat(tradein): автоматизация cian-backfill + rosreestr-import через scheduler (#560 #563) (#630)	2026-05-28 15:31:24 +00:00
Light1YT	d842811390	refactor(tradein): normalize repair_state raw→enum at ingest (#621) (#629)	2026-05-28 15:18:11 +00:00
Light1YT	1731704ddc	feat(tradein): houses.cian_zhk_url column + backfill (#568) (#627)	2026-05-28 13:57:43 +00:00
Light1YT	e332052d0e	feat(tradein): DaData /clean/address backfill для houses (PR Q3) ... Migration 070 — добавляет houses.house_fias_id + dadata_qc_geo / qc_house / enriched_at для tracking. Скрипт scripts/backfill_houses_dadata.py проходит по ~3.8k rows с NULL coords + ~8.6k с NULL cadnum daily 100/day (DaData demo quota → ~124 дня walltime для полного прохода). Per-row SAVEPOINT, resume-safe (фильтр dadata_enriched_at IS NULL), qc_geo IN (0,1) → overwrite coord-payload, иначе только tracking. --priority coords|cadnum|both, --dry-run, --limit N (default 100). 20 тестов покрывают happy path qc_geo=0, qc_geo=3 city-level skip, DaData None response, per-row exception isolation, --dry-run, --limit, --priority filter, и missing-creds fail-fast.	2026-05-27 19:43:07 +05:00
Light1YT	a7fa1a4ffa	feat(tradein): DaData on-demand enrichment в estimate flow (PR Q1) ... Добавляет app/services/dadata.py — async client для DaData /clean/address, обогащающий target адрес canonical-формой, kadastr_num, ФИАС, координатами и ближайшим метро. Migration 069 расширяет trade_in_estimates 6 nullable колонками; estimator.estimate_quality вызывает service после geocode и сохраняет результат в основной INSERT. AggregatedEstimate отдаёт canonical_address / house_cadnum / house_fias_id / metro_nearest наружу. Service graceful: пустые credentials / quota 429 / 5xx / network fail → None, estimator продолжает работу. Demo tier 100 req/день — хватит для low-traffic prod и тестов. ENV: DADATA_API_TOKEN, DADATA_API_SECRET. Tests: 20 unit-тестов через httpx.MockTransport, zero real DaData calls. Покрыто: happy path, отсутствие creds, короткий адрес, timeout/network, HTTP 429/401/403/500/502/503, пустой массив, malformed shape, partial payload, coerce строковых qc-кодов в int, проверка request body+headers. Закладывает базу для PR Q2 (frontend DaData Suggest, 10k/день free) и PR Q3 (backfill 339 NULL-coords houses) — обе используют тот же сервис.	2026-05-27 18:50:12 +05:00
Light1YT	577ddc0654	chore(tradein): drop spurious v_street_sales_vs_listings (#598)	2026-05-27 12:23:18 +00:00
Light1YT	39da0f0644	feat(tradein): sales-vs-listings endpoint (PR K, foundation #564) (#597)	2026-05-27 11:55:44 +00:00
Light1YT	02ad9a1f58	feat(tradein): Phase 1 of #582 — address mismatch audit infrastructure ... Stratified 200-house Yandex reverse-geocode audit comparing our DB's (address, lat/lon) pair against what Yandex.Карты returns for those coordinates. Лежит фундамент данных для Phases 2-5 (canonical = Yandex per user decision). What ships: - Migration 066: address_mismatch_audit table + FDW foreign table gendesign_ekb_districts_geom для stratified sampling. - Sampling SQL: 8 EKB districts × 25 houses = 200. - Driver scripts/audit_address_mismatch.py с двумя режимами: API (если YANDEX_GEOCODER_API_KEY есть) и Playwright (CAPTCHA-aware, персистентный контекст). Resumable по --batch. - Report SQL: p50/p75/p95 distance, top-20 outliers, per-district breakdown. - 19 unit-тестов: normalize, distance bind, API parser, resume idempotency, captcha → status=blocked. Devops note: после деплоя миграции — выдать tradein_fdw_reader SELECT на gendesign.public.ekb_districts_geom перед первым запуском audit. Refs #582 (Phase 1 of 5). Phase 3 backfill заблокирован до Yandex API key (#402).	2026-05-25 12:50:30 +05:00
lekss361	33c4c84467	feat(tradein): этаж/этажность optional + best test presets по deal count (#558)	2026-05-24 21:44:44 +00:00
lekss361	b0d377c984	feat(tradein): Phase C — Avito IMV per-house backfill (SQL + script) (#534)	2026-05-24 14:50:31 +00:00
lekss361	c6ab419100	feat(tradein-sql): bootstrap houses + link 98.8% of listings (Phase A+B) (#527)	2026-05-24 14:17:32 +00:00
lekss361	bad23095cf	fix(tradein/cian): sanitize valuation cache writes (047 UNIQUE + 026/044 cols) (#523)	2026-05-24 13:44:39 +00:00
lekss361	e45ff47aa6	fix(tradein-sql): remove duplicate CREATE INDEX from 050 (regression from PR #500) (#515)	2026-05-24 12:16:27 +00:00
lekss361	8fc309d9ad	fix(tradein-avito): strip Emotion CSS from listings.address (#502)	2026-05-24 10:52:50 +00:00
lekss361	02b48b8d04	fix(tradein-sql): rename dup 030, add trade_in_estimates geom, drop dup indexes (#500) ... Close 3 SQL schema findings from 2026-05-24 trade-in audit. - Rename 030_listings_alter_yandex.sql -> 033_*.sql (resolve dup prefix with 030_avito_imv_cache_key_unique.sql). - New 034_trade_in_estimates_geom.sql: PostGIS geom column + GIST index + backfill + BEFORE INSERT/UPDATE trigger. - New 035_drop_duplicate_indexes.sql: drop listing_sources_listing_idx2 (dup of 028) and houses_geom_idx2 (dup of 009). All idempotent. Deploy script uses ls|sort + no tracking table — rename is safe because original 030 already applied (IF NOT EXISTS no-op on re-run as 033).	2026-05-24 10:04:40 +00:00
lekss361	94cf199276	feat(tradein): postgres_fdw live read of gendesign.cad_buildings (replaces snapshot) (#493) ... Replaces tradein.cad_buildings snapshot with live postgres_fdw foreign table reading gendesign.v_tradein_cad_buildings. Fixes /trade-in/api/v1/geocode/reverse 500 (Nominatim ban) and address_not_geocoded for cadastre addresses (e.g. Хохрякова 81). Security (deep-review fixes): - 100_tradein_fdw_role.sql: passwordless CREATE ROLE; password set by deploy.yml ALTER ROLE bootstrap reading GENDESIGN_FDW_PASSWORD from backend/.env.runtime (via psql :'pw' var → format %L — injection-safe). - core/fdw.py: regex whitelist [A-Za-z0-9_-]{32,256} on password, ValueError without echoing value, try/rollback on commit. - 060_postgres_fdw_extension.sql: connect_timeout='3' on FOREIGN SERVER + ALTER ADD/SET fallback. - geocoder.py: _cadastral_forward_sync / _cadastral_reverse_sync wrapped in asyncio.to_thread. - 100_*.sql: REVOKE ALL ON ALL TABLES/SEQUENCES/FUNCTIONS IN SCHEMA public; only GRANT SELECT on v_tradein_cad_buildings. - pg_user_mappings query handles PUBLIC mapping (usename IS NULL). Tests: 3 SQL-injection guards on ensure_fdw_user_mapping + rewritten cadastral suite.	2026-05-24 08:57:30 +00:00
lekss361	e79c9cec88	feat(tradein): global scraper delay setting (applies across all scrapers) (#485) ... Add global request delay floor + per-source delay management. - 054: seed 'global'=0 + per-source rows (avito=7, cian=5, n1=5, domrf=5, rosreestr=5) using ON CONFLICT DO NOTHING; reuses existing 053 table from #484 - scraper_settings.py: get_scraper_delay() = max(per_source, global); _GLOBAL_KEY, refactored _get_setting_cached(); preserved yandex umbrella alias logic - admin.py: GET /scraper-settings + PUT /scraper-settings/{source} (ge=0.0, le=60.0); cache invalidation on update; CAST(:d AS numeric) per psycopg v3 - avito/cian/n1 scrapers wired (yandex was wired by #484) - 18 unit tests pass (10 original updated for max() + 8 new global delay tests) Closes follow-up gap from #486 (Cian admin slider min=0 now accepted by API).	2026-05-23 15:43:28 +00:00
lekss361	b21d7c7c85	feat(tradein): scraper_settings live-config + Yandex admin trigger endpoints (#484)	2026-05-23 15:28:34 +00:00
lekss361	0202720096	feat(tradein): in-app scheduler — UI-managed schedule (replaces SSH crontab) (#482)	2026-05-23 14:54:25 +00:00
lekss361	2914ff2ef2	feat(tradein): city sweep — auto ЕКБ pipeline (#477) ... Stage 4d: full city sweep ЕКБ (5 anchors × N pages + enrichment + cooperative cancel). - avito.py fetch_around(pages=N) — pagination backward compat default=1 - scrape_pipeline.run_avito_city_sweep — anchors loop + heartbeat + cancel check - scrape_runs.py utility (create/heartbeat/done/failed/cancelled/list_recent) - admin.py 3 endpoints: start (BackgroundTasks) / runs / cancel - migration 051 ADD COLUMN IF NOT EXISTS + DROP/ADD CHECK constraint (text+CHECK, не enum — safe в transaction) - deploy/avito-city-sweep.sh с random delay 0-3h (anti-pattern) - 16 offline tests Deep review approved: migration idempotent, cancel priority correct (mark_done guarded by status='running'), no SQL injection, psycopg v3 compliant.	2026-05-23 14:38:07 +00:00
lekss361	e46b7d778e	feat(tradein): search matview + indexes (Phase 3.1) (#469) ... Multi-Source Integration Phase 3.1. Foundation for /api/v1/search p95 < 100ms on 100K+ listings. SQL 050_search_optimization.sql: - CREATE EXTENSION IF NOT EXISTS pg_trgm - ALTER listings ADD COLUMN tsv tsvector GENERATED ALWAYS AS (...) STORED - 10 indexes (geom GIST, composite, gin_trgm, GIN tsv, partial kadastr, ...) - CREATE MATERIALIZED VIEW listings_search_mv (per-listing aggregated across sources) - 5 matview indexes (UNIQUE on listing_id for REFRESH CONCURRENTLY) Celery task tradein.refresh_search_matview: - REFRESH MATERIALIZED VIEW CONCURRENTLY listings_search_mv (psycopg v3) - Logged duration via logger.info - Graceful try/except ImportError fallback (no celery_app.py in tradein-mvp yet) - Beat schedule snippet in docstring for future wiring Fixups (deep-code-reviewer 2026-05-23): - h.ratings_count → h.reviews_count (alias house_ratings_count preserved) - Strip +psycopg dialect prefix from DSN for libpq compat Refs Master Plan sec 5.1 + 6.5 + 10.1.	2026-05-23 14:01:18 +00:00
lekss361	5e90797a9f	feat(tradein): SQL 040-046 — matching schema delta (Phase 1.1) (#464)	2026-05-23 13:30:22 +00:00
lekss361	0db71033b4	fix(tradein): 030 — UNIQUE constraint on avito_imv_evaluations.cache_key (unblocks PR #452) (#454) ... Hotfix unblocking PR #452 (Stage 3 IMV estimator integration). Adds 030_avito_imv_cache_key_unique.sql: creates UNIQUE INDEX imv_cache_key_uniq_idx on avito_imv_evaluations(cache_key), allowing save_imv_evaluation's ON CONFLICT (cache_key) DO UPDATE to resolve. Migration 018 only created a non-unique composite index, which would have caused runtime error on first IMV cache write. DELETE-USING dedup guards against historical duplicates (no-op on fresh table). BEGIN/COMMIT atomic; IF NOT EXISTS idempotent; CONCURRENTLY unnecessary (table empty). Lexicographic apply order with PR #453's 030_listings_alter_yandex.sql is safe — different tables, no functional interaction. Reviewed by deep-code-reviewer: APPROVE.	2026-05-23 13:04:21 +00:00
lekss361	f452b132c9	feat(tradein): SQL migrations 030-032 — Yandex schema (listings/houses/history) (#453)	2026-05-23 13:00:54 +00:00
lekss361	7fad898c7f	feat(tradein): SQL migration 029 — extend matching/valuation/dynamics tables (PR #445 follow-up) (#448)	2026-05-23 12:35:03 +00:00
lekss361	1ad2565af1	feat(tradein): SQL migrations 019-028 — Cian schema + matching + cookies ... - 019: ALTER listings (windows_view_type, wcs counts, ceiling_height, repair_type, views, living_area, bedrooms, balconies, loggias, description_minhash, cadastral_number, phones, bargain_allowed, sale_type, house_source/ext_id, metro_stations) - 020: ALTER houses (BTI: series_name, entrances, flat_count, is_emergency, heat/gas/overlap types; Cian: cian_internal_house_id, management_company_id, transport_accessibility_rate, advantages/banks/builders/houses_by_turn jsonb + GIN indexes) - 021: CREATE management_companies + FK to houses.management_company_id - 022: CREATE agents table + agent_id_fk on listings - 023: CREATE offer_price_history (per-listing price change timeline) - 024: CREATE houses_price_dynamics (monthly price/sqm series) - 025: CREATE house_reliability_checks (наш.дом.рф checks) - 026: CREATE external_valuations (unified Avito IMV + Cian Valuation cache) - 027: CREATE EXTENSION pgcrypto + cian_session_cookies (encrypted) - 028: CREATE house_sources, listing_sources, house_address_aliases + address_fingerprint/canonical/merged_into alters Refs: decisions/CianScraper_v1_Implementation_Plan.md Stage 1 decisions/Schema_Cian_SERP_Inventory.md sec 16, 20.3, 26 decisions/Cross_Source_Matching_Strategy.md sec 2	2026-05-23 15:14:49 +03:00
lekss361	a2147872ed	feat(tradein): SQL migrations 009-018 — houses + listings extension + IMV cache (#440)	2026-05-23 11:41:44 +00:00