558 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
lekss361	b26e0ff7c3	feat(tradein): estimator — Yandex Valuation as on-demand 6th source (#473) ... Stage 8 of YandexRealtyScraper v1. Adds Yandex Valuation (anonymous, no cookies) as 6th on-demand source in estimate_quality. Mirrors Avito IMV pattern: 24h cache via external_valuations(source='yandex_valuation', cache_key) [UNIQUE 026:34], idempotent history persistence via house_placement_history(source, ext_item_id) [UNIQUE 017:50] with synthesized SHA256(address|date|area|floor|prices)[:32] ext_item_id. Graceful degradation on geocode/fetch/cache/save errors. house_id=NULL — matching deferred to Stage 8.x. 11 new tests + 4 IMV regression = 15 green.	2026-05-23 14:14:49 +00:00
lekss361	89c4a2842d	feat(tradein): cross-source matching service (3-tier: cadastr / fingerprint / geo / composite) (#470)	2026-05-23 14:12:16 +00:00
lekss361	649034479f	feat(tradein): Stage 4c — scrapers admin pages (Avito real + Cian/Yandex stubs) (#472)	2026-05-23 14:05:56 +00:00
lekss361	7d649f9239	feat(tradein): conflict_resolution — register Yandex sources in priority dicts (#471) ... Stage 7/9 YandexRealtyScraper v1 (Wave 5). Integrates 4 Yandex scrapers into cross-source field-priority arbitration. - HOUSE_FIELD_PRIORITY: lat/lon → yandex_realty_nb; +year_built/house_type/house_class/rating_score/reviews_count Yandex sources - New house fields: text_reviews_count, corpus_count, total_area_ha, commission_year/month, developer_name, has_panorama, yandex_total_listings, has_lift, ceiling_height - LISTING_FIELD_PRIORITY: description/house_type +yandex_detail - New listing fields (all yandex_detail): agency_name/founded_year/objects_count, views_total_yandex, publish_date_relative, sale_type_text - 37 unit tests (7 pre-existing + 30 new), all pass; ruff clean - Resolver engine _resolve() unchanged (this PR creates the file initially) Refs: decisions/Cross_Source_Matching_Strategy.md sec 3.5+4.5, decisions/YandexRealtyScraper_v1_Implementation_Plan.md Stage 7.	2026-05-23 14:03:35 +00:00
lekss361	e46b7d778e	feat(tradein): search matview + indexes (Phase 3.1) (#469) ... Multi-Source Integration Phase 3.1. Foundation for /api/v1/search p95 < 100ms on 100K+ listings. SQL 050_search_optimization.sql: - CREATE EXTENSION IF NOT EXISTS pg_trgm - ALTER listings ADD COLUMN tsv tsvector GENERATED ALWAYS AS (...) STORED - 10 indexes (geom GIST, composite, gin_trgm, GIN tsv, partial kadastr, ...) - CREATE MATERIALIZED VIEW listings_search_mv (per-listing aggregated across sources) - 5 matview indexes (UNIQUE on listing_id for REFRESH CONCURRENTLY) Celery task tradein.refresh_search_matview: - REFRESH MATERIALIZED VIEW CONCURRENTLY listings_search_mv (psycopg v3) - Logged duration via logger.info - Graceful try/except ImportError fallback (no celery_app.py in tradein-mvp yet) - Beat schedule snippet in docstring for future wiring Fixups (deep-code-reviewer 2026-05-23): - h.ratings_count → h.reviews_count (alias house_ratings_count preserved) - Strip +psycopg dialect prefix from DSN for libpq compat Refs Master Plan sec 5.1 + 6.5 + 10.1.	2026-05-23 14:01:18 +00:00
lekss361	51650276b2	feat(tradein): yandex_newbuilding.py — ЖК landing parser (149 selectors + NLP) (#467)	2026-05-23 13:46:45 +00:00
lekss361	7056ecb23b	feat(tradein): yandex_valuation.py — anonymous house-history scraper (#468)	2026-05-23 13:46:20 +00:00
lekss361	3d90221fa0	feat(tradein): yandex_detail.py — Product JSON-LD + DOM detail parser (#466) ... Stage 4 of YandexRealtyScraper v1. YandexDetailScraper.fetch_detail(offer_url) extracts authoritative price from Product JSON-LD (offers.price exact int) + DOM sections (description, agent, stats, metro, photos 8 sizes, newbuilding link, NLP). 35 unit tests, ruff clean. Photo URLs sanitized downstream via _safe_url CDN allowlist (avatars.mds.yandex.net).	2026-05-23 13:45:09 +00:00
lekss361	b0a7235110	feat(tradein): cian_valuation.py — auth-required Calculator scraper + 24h cache (#465)	2026-05-23 13:43:34 +00:00
lekss361	b5d206bfa8	feat(tradein): Stage 4a — API endpoints for Avito enrichment + IMV benchmark UI (#460)	2026-05-23 13:34:38 +00:00
lekss361	29f94963a7	Merge pull request 'feat(tradein): yandex_realty.py — DOM SERP refactor (vtorichka, 23 cards/page)' (#462) from feat/tradein-yandex-serp-refactor into main	2026-05-23 13:30:36 +00:00
lekss361	5e90797a9f	feat(tradein): SQL 040-046 — matching schema delta (Phase 1.1) (#464)	2026-05-23 13:30:22 +00:00
lekss361	72d2b97ee0	feat(tradein): Stage 4b — IMV benchmark badge + house info card on estimate result (#461)	2026-05-23 13:28:23 +00:00
lekss361	d28bf4e66a	feat(tradein): cian_newbuilding.py — ЖК catalog scraper (chart + reliability + offers, Wave 4 Worker C) (#459) ... Stage 6 / Wave 4 Worker C of CianScraper v1. Cian.ru newbuilding (ЖК) catalog page scraper — extracts newbuilding metadata + 6 sister state containers (realtyValuation chart / reliability / reviews / offers / builders / housesByTurn). Main Stage 6 deliverable: 7-month price dynamics chart → houses_price_dynamics table (cross-source benchmark для Stage 9 estimator). Files: - app/services/scrapers/cian_newbuilding.py (NEW, 548 LOC) - tests/test_cian_newbuilding.py (NEW, 357 LOC, 20 tests) Reviewed: schema match (migrations 020/021/025/029), ON CONFLICT constraint name verified (houses_price_dynamics_dim_key), graceful chart extraction (empty/None/fallback shapes), atomic transaction. Known minor follow-ups (non-blocking): - management_companies UPSERT: ext_id=NULL → potential duplicates (Postgres NULL!=NULL); bounded, document in vault - offers items[] fast path: no dict-only filter / no room_count injection - houses_price_dynamics primary shape stores total price into price_per_sqm column with prices_type='price' discriminator — Stage 9 estimator must respect prices_type filter - house_reliability_checks no UNIQUE → unbounded growth on rescrape (deferred to future migration 030+) Verdict: APPROVE (deep-code-reviewer).	2026-05-23 13:23:11 +00:00
lekss361	2dd76ea5e8	feat(tradein): cian_session cookies management + admin endpoints (Wave 4 Worker A) (#457)	2026-05-23 13:22:00 +00:00
lekss361	ead2ac16c5	feat(tradein): yandex_realty.py — DOM SERP refactor (vtorichka, 23 cards/page) ... Replace dead JSON-state extraction (Yandex moved to chunked/ephemeral state post-hydration) with selectolax DOM parsing using helpers from PR #456. Changes: - New URL: /{city}/kupit/kvartira/vtorichniy-rynok/?page=N (path-based, no geo) - Selector: [data-test="OffersSerpItem"] x ~23 per page - Per-card extraction: offer_id (URL regex), title (rooms/area/floor regex), price/ppm2 (regex + parse_rub), bargain flag, publish_date (parse_ru_date), street address, photos (avatars.mds.yandex domain filter + size upgrade), newbuilding linkage (house_source='yandex_realty_nb' + house_ext_id) - ScrapedLot mapping: lat/lon=None (no coords on SERP — geocode-missing fills later), listing_segment="vtorichka" - Pagination: fetch_around_multi_room -> up to MAX_PAGES=3 pages, dedup by source_id. **kwargs swallow keeps old rooms/sort/pages callers compat. - Preserved: BaseScraper interface, name="yandex" (matches existing rows + avoids breaking dedup_hash; Wave 5 conflict resolution can alias if needed), request_delay_sec=5.0, sleep_between_requests, tenacity retry. Tests: 10 unit tests against hand-crafted fixture HTML (single card / studio / no-price / empty page / multi-card / URL builder / city override) — all pass. Ruff clean.	2026-05-23 16:21:47 +03:00
lekss361	962daf2d47	feat(tradein): yandex_helpers.py — JSON-LD + regex + date + NLP + money helpers (#456)	2026-05-23 13:14:11 +00:00
lekss361	5051ebde2e	feat(tradein): integrate Avito IMV as 5th evaluation source (on-demand cached) (#452)	2026-05-23 13:08:22 +00:00
lekss361	1b3edff6a9	feat(tradein): extend base.py save_listings to persist Cian-specific columns (PR #450 follow-up) (#455)	2026-05-23 13:05:20 +00:00
lekss361	0db71033b4	fix(tradein): 030 — UNIQUE constraint on avito_imv_evaluations.cache_key (unblocks PR #452) (#454) ... Hotfix unblocking PR #452 (Stage 3 IMV estimator integration). Adds 030_avito_imv_cache_key_unique.sql: creates UNIQUE INDEX imv_cache_key_uniq_idx on avito_imv_evaluations(cache_key), allowing save_imv_evaluation's ON CONFLICT (cache_key) DO UPDATE to resolve. Migration 018 only created a non-unique composite index, which would have caused runtime error on first IMV cache write. DELETE-USING dedup guards against historical duplicates (no-op on fresh table). BEGIN/COMMIT atomic; IF NOT EXISTS idempotent; CONCURRENTLY unnecessary (table empty). Lexicographic apply order with PR #453's 030_listings_alter_yandex.sql is safe — different tables, no functional interaction. Reviewed by deep-code-reviewer: APPROVE.	2026-05-23 13:04:21 +00:00
lekss361	f452b132c9	feat(tradein): SQL migrations 030-032 — Yandex schema (listings/houses/history) (#453)	2026-05-23 13:00:54 +00:00
lekss361	eb4764f75a	feat(tradein): scrape_pipeline.py — Avito full orchestrator (search → houses → detail) (#451)	2026-05-23 12:55:13 +00:00
lekss361	4f0319297e	feat(tradein): cian.py SERP refactor — state-based extraction (137 fields, no jitter) (#450)	2026-05-23 12:52:04 +00:00
lekss361	5258a8f023	feat(tradein): avito_houses.py — Houses Catalog parser (state + reviews + history + recs) (#449)	2026-05-23 12:37:24 +00:00
lekss361	7fad898c7f	feat(tradein): SQL migration 029 — extend matching/valuation/dynamics tables (PR #445 follow-up) (#448)	2026-05-23 12:35:03 +00:00
lekss361	9ba776f2ca	feat(tradein): cian_state_parser shared utility + ScrapedLot Cian fields (#447)	2026-05-23 12:35:01 +00:00
lekss361	23e6cbc773	feat(tradein): SQL migrations 019-028 — Cian schema + matching + cookies (#445) ... Stage 1 / Wave 1 of CianScraper v1 — schema foundation for 4 Cian sources (SERP/detail/newbuilding/valuation) + cross-source matching. 10 SQL migrations: - 019-020: ALTER listings/houses with Cian-specific cols - 021-022: management_companies, agents tables - 023-024: offer_price_history, houses_price_dynamics - 025-026: house_reliability_checks, external_valuations cache - 027: cian_session_cookies (pgcrypto encryption) - 028: matching tables (house_sources/listing_sources/house_address_aliases) All migrations are idempotent (BEGIN/COMMIT + IF NOT EXISTS). Reviewed by deep-code-reviewer.	2026-05-23 12:19:28 +00:00
lekss361	7cdde30b13	feat(tradein): avito_detail.py — detail page parser (30+ fields incl. domoteka) (#443)	2026-05-23 12:17:15 +00:00
lekss361	6a2f734356	feat(tradein): avito_imv.py — Avito IMV evaluation API client (2 requests) (#444)	2026-05-23 12:16:47 +00:00
lekss361	1ad2565af1	feat(tradein): SQL migrations 019-028 — Cian schema + matching + cookies ... - 019: ALTER listings (windows_view_type, wcs counts, ceiling_height, repair_type, views, living_area, bedrooms, balconies, loggias, description_minhash, cadastral_number, phones, bargain_allowed, sale_type, house_source/ext_id, metro_stations) - 020: ALTER houses (BTI: series_name, entrances, flat_count, is_emergency, heat/gas/overlap types; Cian: cian_internal_house_id, management_company_id, transport_accessibility_rate, advantages/banks/builders/houses_by_turn jsonb + GIN indexes) - 021: CREATE management_companies + FK to houses.management_company_id - 022: CREATE agents table + agent_id_fk on listings - 023: CREATE offer_price_history (per-listing price change timeline) - 024: CREATE houses_price_dynamics (monthly price/sqm series) - 025: CREATE house_reliability_checks (наш.дом.рф checks) - 026: CREATE external_valuations (unified Avito IMV + Cian Valuation cache) - 027: CREATE EXTENSION pgcrypto + cian_session_cookies (encrypted) - 028: CREATE house_sources, listing_sources, house_address_aliases + address_fingerprint/canonical/merged_into alters Refs: decisions/CianScraper_v1_Implementation_Plan.md Stage 1 decisions/Schema_Cian_SERP_Inventory.md sec 16, 20.3, 26 decisions/Cross_Source_Matching_Strategy.md sec 2	2026-05-23 15:14:49 +03:00
lekss361	a2953e97c4	refactor(security): убрать UI ввода admin token (backend не требует с PR #437) (#442)	2026-05-23 12:06:37 +00:00
lekss361	47f4cc7d58	refactor(tradein): avito search — remove jitter (C-5) + mobile API dead code + parse houseLinks (#441)	2026-05-23 11:55:18 +00:00
lekss361	a2147872ed	feat(tradein): SQL migrations 009-018 — houses + listings extension + IMV cache (#440)	2026-05-23 11:41:44 +00:00
lekss361	43a8ec426e	fix(ui): &lt;Link&gt; вместо &lt;a&gt; для internal nav routes (build error fix)	2026-05-23 11:31:59 +00:00
lekss361	65c77b96b6	feat(ui): добавить Trade-In / Концепция / Админ в landing nav (#438)	2026-05-23 11:11:21 +00:00
lekss361	b233bf91cc	refactor(security): убрать X-Admin-Token (Caddy basic_auth достаточен) (#437)	2026-05-23 10:41:22 +00:00
lekss361	0e294641a1	fix(security): log_credentials + auth_audit.log → username из 401 (#436)	2026-05-23 09:59:50 +00:00
lekss361	5e3ba351a0	fix(tradein): security hotfix — C-3 config default + C-4 empty estimate persist + C-6 PDF URL allowlist (#435)	2026-05-23 09:38:23 +00:00
lekss361	3295808b41	fix(security): caddy/** в deploy paths + username из 401 попытки (#434)	2026-05-23 09:25:41 +00:00
lekss361	de1652a6d8	feat(security): добавить юзера kopylov в basic_auth gate (#433)	2026-05-23 09:07:50 +00:00
lekss361	f4d43d1cfc	fix(security): forwarder PermissionError — убрать USER nobody + deploy automation (#432) ... Hotfix для GlitchTip Issue #71 (PermissionError при чтении /var/log/caddy/gendsgn.ru.log). Changes: - ops/glitchtip-auth-forwarder/Dockerfile — убран USER nobody, форвардер под root - ops/glitchtip-auth-forwarder/forwarder.py — throttle capture_exception (300s) - .forgejo/workflows/deploy.yml — paths-filter + build + force-recreate sidecar - docs/runbooks/basic_auth_management.md — Deploy section Security: caddy_logs mount read-only, нет network listener, нет shell, auth_forwarder_state dedicated volume. Attack surface минимален. После merge deploy.yml автоматически rebuild+recreate sidecar на VPS.	2026-05-23 09:04:00 +00:00
lekss361	03dc8e6735	feat(security): GlitchTip forwarder для basic_auth 401 events (Phase 2 PR #426) (#430)	2026-05-23 08:42:36 +00:00
lekss361	cc2d148967	feat(security): закрыть gendsgn.ru basic_auth gate (multi-user, 11 users) (#426) ... Pilot demo 28.05.2026: closed gendsgn.ru/* (включая trade-in/) basic_auth gate. /health и /preview/* остаются public через route { } wrapper (Caddy 2.x directive ordering fix). 11 users (admin + user1..user10), bcrypt cost=14, snippet в git (audit trail). Scripts: scripts/auth/{add,remove,list}_user.sh. Fixup PR #426 (route{} wrap + log rotation + base64 busybox-compat + runbook audit log fix).	2026-05-23 08:16:10 +00:00
lekss361	99c19f9fbb	Merge pull request 'fix(tradein): PDF — рабочая ссылка на отчёт + «Есть/Нет» для балкона' (#425) from fix/tradein-pdf-link-balcony into main	2026-05-22 11:01:50 +00:00
Light1YT	420057aa02	fix(tradein): PDF — рабочая ссылка на отчёт + «Есть/Нет» для балкона ... «Этот отчёт онлайн» и QR-код в PDF вели на gendsgn.ru/trade-in/trade-in?id=… — двойной /trade-in/: PUBLIC_URL на проде уже = https://gendsgn.ru/trade-in, а код добавлял /trade-in ещё раз → ссылка/QR давали 404. Плюс «Балкон / лоджия» печатался как «1»/«0» (сырое булево) вместо «Есть»/«Нет».	2026-05-22 16:01:12 +05:00
lekss361	7a9b183296	Merge pull request 'fix(tradein): восстановленная оценка показывает этаж/площадь, не 0/0' (#424) from fix/tradein-restore-estimate-params into main	2026-05-22 10:51:04 +00:00
Light1YT	9b2a54788b	fix(tradein): восстановленная оценка показывает этаж/площадь, не 0/0 ... AggregatedEstimate не нёс параметры целевой квартиры (площадь, этаж, комнаты, год, тип дома). HeroSummary брал их из формы-инпута — у свежей оценки работало, но при открытии оценки по ссылке (?id=, из Истории) формы нет → «Сводка» показывала «ЭТАЖ 0/0, Площадь 0 м²». Параметры теперь проходят сквозь весь стек: - AggregatedEstimate + TS-тип получили area_m2/rooms/floor/ total_floors/year_built/house_type/repair_state/has_balcony; - estimate_quality заполняет их из payload, get_estimate — из строки trade_in_estimates (колонки там уже есть); - HeroSummary читает из estimate с откатом на input. Заодно в TS-тип добавлен пропущенный est_days_on_market.	2026-05-22 15:50:19 +05:00
lekss361	7daa2dec55	Merge pull request 'fix(tradein): фото аналога в Сводке больше не скрыто плейсхолдером' (#423) from fix/tradein-hero-photo-visible into main	2026-05-22 10:35:44 +00:00
Light1YT	6041ac8464	fix(tradein): фото аналога в Сводке больше не скрыто плейсхолдером ... В .hero-photo псевдоэлементы ::before/::after рисуют декоративный скелет-плейсхолдер «фасада дома». Они position:absolute, а <img> с фото аналога — position:static, поэтому абсолютные псевдоэлементы рисовались ПОВЕРХ картинки: реальное фото загружалось (naturalWidth есть), но визуально всегда был серый плейсхолдер. Фото (z-index 1) и подпись .photo-meta (z-index 2) теперь над скелетом. Скелет остаётся виден, когда фото нет или оно не загрузилось (onError → display:none).	2026-05-22 15:35:01 +05:00
lekss361	07903b1c36	Merge pull request 'fix(tradein): бюджет по времени в geocode-missing — cron не таймаутит' (#422) from fix/tradein-geocode-time-budget into main	2026-05-22 09:37:42 +00:00