599 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
lekss361	5bd9c2f66d	fix(tradein-frontend): safeUrl validator, apiFetch dedup, ListingsCard a11y, error boundary ... Four security/UX fixes from 2026-05-24 audit: * lib/safeUrl.ts (new): validate external URLs, reject non-http(s) schemes (closes finding #3 — backend supplies lot.source_url verbatim from scraped sites; javascript:/data:/file: would have executed on click). * lib/api.ts: refactor apiFetch to delegate to apiFetchWithStatus — single read-body-once implementation, drops the body-stream-twice anti-pattern (finding #8). * components/trade-in/ListingsCard.tsx: drop tr.onClick (not keyboard accessible), keep the inline <a> as the only interactive element, gate it through safeUrl() (findings #7 + #3). * app/error.tsx (new): Next 15 App Router route-segment error boundary — prevents white-screen on component crash (finding #16). Scope-cut from original audit: * #17 (sessionId crypto.randomUUID) — already implemented in sessionId.ts; no change needed. * safeUrl wiring in DealsCard / IMVBenchmark — neither renders user-supplied URLs (DealsCard has no links, IMVBenchmark links a hardcoded domain).	2026-05-24 14:38:56 +03:00
lekss361	0767fb7401	fix(tradein-photos): sanitize uploaded images via Pillow re-encode (#511)	2026-05-24 11:32:04 +00:00
lekss361	f04359592f	Merge pull request 'fix(deploy): tradein_fdw_reader password — psql var interpolation in DO block' (#510) from fix/tradein-fdw-psql-var-in-do into main ... Reviewed-on: #510	2026-05-24 11:28:59 +00:00
lekss361	606d3fe7c0	Merge pull request 'chore(rules): drop blocked-scope list from git-pr.md' (#506) from chore/git-pr-rules-any-scope-v2 into main ... Reviewed-on: #506	2026-05-24 11:26:10 +00:00
lekss361	08a8cf51b3	fix(deploy): mute psql set_config stdout — prevent password leak in CI logs ... Review-bot PR #510 нашёл P0: `set_config(name,value,is_local)` возвращает установленное значение, psql печатает rows на stdout без `-q`/`-A`/`-t`, итог — фактический pw появляется в Forgejo Actions deploy logs (retained, visible всем с repo read access). Fix: оборачиваем оба `SELECT set_config(...)` в `\o /dev/null` / `\o` brackets. Output mutes только для этих строк — NOTICE-сообщения из DO block (idempotency signal 'tradein_fdw_reader password set') остаются видимыми. Также добавлен rollback hint в комментарии: НЕ revert файл (вернёт сломанный :'pw' внутри $$), а unset GENDESIGN_FDW_PASSWORD в .env.runtime на VPS.	2026-05-24 14:21:36 +03:00
lekss361	7d8c22ff39	fix(tradein-estimator): atomic IMV link, batched yandex history, drop redundant CAST (#509)	2026-05-24 11:18:12 +00:00
lekss361	d4cbd4bb9f	feat(tradein-scrapers): parse listing_date from Avito + Yandex (#508)	2026-05-24 11:17:13 +00:00
lekss361	35bf050e63	chore(rules): add decision date to self-extending guard (review #506 fixup) ... Reviewer's optional recommendation #3: добавить дату в hard exception для audit trail. Также расширил wording: 'предотвращает bot-loop где bot сам расширяет свои merge права' — делает intent явным для будущих reviewers.	2026-05-24 14:10:53 +03:00
lekss361	7f23a9b3f1	feat(tradein-imv): retry with cleaned address + categorize errors (#505)	2026-05-24 11:04:24 +00:00
lekss361	da7814810c	fix(deploy): tradein_fdw_reader password — psql var interpolation in DO block ... psql :'pw' substitution НЕ работает внутри dollar-quoted блока $$..$$ — это правило psql, не bug. Symptom: deploy 2026-05-24 после merge PR #503 упал с 'syntax error at or near :' on LINE 4 of set_tradein_fdw_password.sql, :'pw' дошёл до сервера как literal вместо интерполяции. Fix: передаём password в DO через session GUC (set_config) — psql интерполирует :'pw' ВНЕ dollar quote, внутри читаем через current_setting(). Очищаем GUC после выполнения (defense-in-depth).	2026-05-24 14:03:15 +03:00
lekss361	c9737d26c9	chore(rules): drop blocked-scope list from git-pr.md — align with any-scope policy ... Header line 10 + memory rule [Auto-merge any scope] (2026-05-16) заявили APPROVE → merge любой scope. Body всё ещё содержал устаревший blocked-list (data/sql, services, scrapers, и т.д.), из-за которого session-side classifier'ы ошибочно блокировали auto-merge approved PR (incident: PRs #501-504 ждали ручного merge). Заменил Auto-merge scope section на Auto-merge policy: - Любой scope OK при verdict=approve + SHA match - Hard exception: secrets в diff + self-extending guard (изменение самих правил всегда через human) Также убрал Scope guard и Blocked-scope cap из Polling loop section.	2026-05-24 13:58:01 +03:00
lekss361	099b8e4b59	fix(scrapers): DOM.РФ KN extras — bypass WAF + update renamed endpoints (#503)	2026-05-24 10:53:28 +00:00
lekss361	6bfa686f16	feat(tradein-estimator): exclude rosreestr deals from output (#504)	2026-05-24 10:52:55 +00:00
lekss361	8fc309d9ad	fix(tradein-avito): strip Emotion CSS from listings.address (#502)	2026-05-24 10:52:50 +00:00
lekss361	1026165abb	fix(tradein-matching): pg_advisory_xact_lock to prevent duplicate house INSERTs (#501)	2026-05-24 10:52:10 +00:00
lekss361	02b48b8d04	fix(tradein-sql): rename dup 030, add trade_in_estimates geom, drop dup indexes (#500) ... Close 3 SQL schema findings from 2026-05-24 trade-in audit. - Rename 030_listings_alter_yandex.sql -> 033_*.sql (resolve dup prefix with 030_avito_imv_cache_key_unique.sql). - New 034_trade_in_estimates_geom.sql: PostGIS geom column + GIST index + backfill + BEFORE INSERT/UPDATE trigger. - New 035_drop_duplicate_indexes.sql: drop listing_sources_listing_idx2 (dup of 028) and houses_geom_idx2 (dup of 009). All idempotent. Deploy script uses ls|sort + no tracking table — rename is safe because original 030 already applied (IF NOT EXISTS no-op on re-run as 033).	2026-05-24 10:04:40 +00:00
lekss361	ed49f6b953	fix(deploy): resolve FDW bootstrap race + extract bootstrap DO block to .sql file (#499)	2026-05-24 09:46:00 +00:00
lekss361	03ab1faa7c	chore(claude): aggressive worktree cleanup via Forgejo PR API (#498)	2026-05-24 09:34:48 +00:00
lekss361	998c8a17a8	chore(claude): agents view readiness — cleanup, .worktreeinclude, workflow (#497)	2026-05-24 09:20:26 +00:00
lekss361	643064ee0c	chore(claude-rules): promote 3 deferred feedback rules from memory (#496)	2026-05-24 09:01:27 +00:00
lekss361	94cf199276	feat(tradein): postgres_fdw live read of gendesign.cad_buildings (replaces snapshot) (#493) ... Replaces tradein.cad_buildings snapshot with live postgres_fdw foreign table reading gendesign.v_tradein_cad_buildings. Fixes /trade-in/api/v1/geocode/reverse 500 (Nominatim ban) and address_not_geocoded for cadastre addresses (e.g. Хохрякова 81). Security (deep-review fixes): - 100_tradein_fdw_role.sql: passwordless CREATE ROLE; password set by deploy.yml ALTER ROLE bootstrap reading GENDESIGN_FDW_PASSWORD from backend/.env.runtime (via psql :'pw' var → format %L — injection-safe). - core/fdw.py: regex whitelist [A-Za-z0-9_-]{32,256} on password, ValueError without echoing value, try/rollback on commit. - 060_postgres_fdw_extension.sql: connect_timeout='3' on FOREIGN SERVER + ALTER ADD/SET fallback. - geocoder.py: _cadastral_forward_sync / _cadastral_reverse_sync wrapped in asyncio.to_thread. - 100_*.sql: REVOKE ALL ON ALL TABLES/SEQUENCES/FUNCTIONS IN SCHEMA public; only GRANT SELECT on v_tradein_cad_buildings. - pg_user_mappings query handles PUBLIC mapping (usename IS NULL). Tests: 3 SQL-injection guards on ensure_fdw_user_mapping + rewritten cadastral suite.	2026-05-24 08:57:30 +00:00
lekss361	73bdff41a7	chore(claude-config): align with 2026 best practices (#495)	2026-05-24 08:48:42 +00:00
lekss361	0efd2dca1d	feat(tradein): cadastral reverse geocode via main backend cad_buildings (#492)	2026-05-24 07:31:12 +00:00
lekss361	7874a5c97b	feat(tradein): source-quota in estimator analogs (fix Cian/Yandex starvation) (#491)	2026-05-23 20:59:58 +00:00
lekss361	222389cba9	fix(tradein/avito): anti-bot hardening — sleep + abort + shared session (#487)	2026-05-23 20:08:40 +00:00
lekss361	726e774a7f	feat(tradein-ui): Yandex admin scraper page — mirror Avito + global delay control (#488) ... Replaces 5-line stub at /scrapers/yandex with full admin page mirroring Avito. 5 sections: global delay (umbrella yandex key), search around, detail snapshot, JK newbuilding, valuation. TanStack Query hooks + apiFetch, TS strict, reuses trade-in.css. Fixup: useScraperSettings unwraps {settings:[...]} response wrapper; PUT body includes required source field (matches backend Pydantic ScraperSettingPayload). useEffect race guarded with initializedRef so 30s refetch doesn't clobber user input mid-edit. Verdict: APPROVE — deep-code-reviewer (response shape unwrap + source field both applied).	2026-05-23 20:07:48 +00:00
lekss361	52716a46c6	fix(tradein): geocode backfill + remove Avito exclusion from estimator (#490)	2026-05-23 20:03:48 +00:00
lekss361	ffb0c67c84	fix(tradein-ui): double /trade-in/ prefix in Cian page Link (PR #486 follow-up) (#489)	2026-05-23 15:56:01 +00:00
lekss361	e79c9cec88	feat(tradein): global scraper delay setting (applies across all scrapers) (#485) ... Add global request delay floor + per-source delay management. - 054: seed 'global'=0 + per-source rows (avito=7, cian=5, n1=5, domrf=5, rosreestr=5) using ON CONFLICT DO NOTHING; reuses existing 053 table from #484 - scraper_settings.py: get_scraper_delay() = max(per_source, global); _GLOBAL_KEY, refactored _get_setting_cached(); preserved yandex umbrella alias logic - admin.py: GET /scraper-settings + PUT /scraper-settings/{source} (ge=0.0, le=60.0); cache invalidation on update; CAST(:d AS numeric) per psycopg v3 - avito/cian/n1 scrapers wired (yandex was wired by #484) - 18 unit tests pass (10 original updated for max() + 8 new global delay tests) Closes follow-up gap from #486 (Cian admin slider min=0 now accepted by API).	2026-05-23 15:43:28 +00:00
lekss361	ebfd29a0a6	feat(tradein-ui): Cian scraper admin page + global request delay control (#486)	2026-05-23 15:36:07 +00:00
lekss361	b21d7c7c85	feat(tradein): scraper_settings live-config + Yandex admin trigger endpoints (#484)	2026-05-23 15:28:34 +00:00
lekss361	65029cd1a7	feat(tradein-ui): Cian cookies upload admin page (#481)	2026-05-23 15:19:42 +00:00
lekss361	fd3cb4e6ed	feat(tradein): /scrapers/avito — 6-я секция Расписание (in-app scheduler UI) (#483)	2026-05-23 14:55:31 +00:00
lekss361	0202720096	feat(tradein): in-app scheduler — UI-managed schedule (replaces SSH crontab) (#482)	2026-05-23 14:54:25 +00:00
lekss361	99dd403c45	fix(tradein): CIAN_REQUIRED_COOKIES — add real auth cookies (DMIR_AUTH + Cian session) (#480) ... Updated CIAN_REQUIRED_COOKIES filter в cian_session.py по реальному DevTools-дампу logged-in cian.ru сессии (2026-05-23). Добавлены critical auth cookies: DMIR_AUTH (httpOnly), _CIAN_GK, _yasc, _ym_visorc, uxfb_card_satisfaction. Legacy entries сохранены как backward-compat fallback. Regression test test_required_cookies_includes_real_auth гарантирует что все реальные cookies проходят filter. Fixes Stage 4 PR #457 (filter guessed без real probe — DMIR_AUTH был dropped → verify_session falsey).	2026-05-23 14:52:24 +00:00
lekss361	9643979d2c	feat(tradein): Phase 3.2 — /api/v1/search endpoint + Redis cache (#479) ... POST /api/v1/search reading listings_search_mv with ~20 cross-source filters, parameterized SQL (CAST(:x AS type), psycopg v3), whitelisted ORDER BY (Pydantic Literal), Redis 5min hot cache with graceful degradation (singleton pool via lru_cache). Verified vs data/sql/050_search_optimization.sql: matview column refs (total_area, lng, house_rating, sources[], has_avito/cian/yandex), SQL injection safety, cache failure swallow, router prefix. Deep-code-reviewer: APPROVE.	2026-05-23 14:44:22 +00:00
lekss361	dd24bd8136	feat(tradein): /scrapers/avito — 5-я секция City sweep (auto + cancel + runs list) (#478)	2026-05-23 14:40:11 +00:00
lekss361	2914ff2ef2	feat(tradein): city sweep — auto ЕКБ pipeline (#477) ... Stage 4d: full city sweep ЕКБ (5 anchors × N pages + enrichment + cooperative cancel). - avito.py fetch_around(pages=N) — pagination backward compat default=1 - scrape_pipeline.run_avito_city_sweep — anchors loop + heartbeat + cancel check - scrape_runs.py utility (create/heartbeat/done/failed/cancelled/list_recent) - admin.py 3 endpoints: start (BackgroundTasks) / runs / cancel - migration 051 ADD COLUMN IF NOT EXISTS + DROP/ADD CHECK constraint (text+CHECK, не enum — safe в transaction) - deploy/avito-city-sweep.sh с random delay 0-3h (anti-pattern) - 16 offline tests Deep review approved: migration idempotent, cancel priority correct (mark_done guarded by status='running'), no SQL injection, psycopg v3 compliant.	2026-05-23 14:38:07 +00:00
lekss361	37c4574d0a	feat(tradein): estimator — Cian Valuation as on-demand 7th source (Stage 9) (#476)	2026-05-23 14:29:52 +00:00
lekss361	c0a0bfa413	feat(tradein): Phase 1.3 backfill scripts — fingerprints + house/listing_sources (#475)	2026-05-23 14:29:39 +00:00
lekss361	dbad7a626b	fix(tradein): убрать require_admin полностью — Caddy basic_auth достаточен (#474)	2026-05-23 14:21:01 +00:00
lekss361	b26e0ff7c3	feat(tradein): estimator — Yandex Valuation as on-demand 6th source (#473) ... Stage 8 of YandexRealtyScraper v1. Adds Yandex Valuation (anonymous, no cookies) as 6th on-demand source in estimate_quality. Mirrors Avito IMV pattern: 24h cache via external_valuations(source='yandex_valuation', cache_key) [UNIQUE 026:34], idempotent history persistence via house_placement_history(source, ext_item_id) [UNIQUE 017:50] with synthesized SHA256(address|date|area|floor|prices)[:32] ext_item_id. Graceful degradation on geocode/fetch/cache/save errors. house_id=NULL — matching deferred to Stage 8.x. 11 new tests + 4 IMV regression = 15 green.	2026-05-23 14:14:49 +00:00
lekss361	89c4a2842d	feat(tradein): cross-source matching service (3-tier: cadastr / fingerprint / geo / composite) (#470)	2026-05-23 14:12:16 +00:00
lekss361	649034479f	feat(tradein): Stage 4c — scrapers admin pages (Avito real + Cian/Yandex stubs) (#472)	2026-05-23 14:05:56 +00:00
lekss361	7d649f9239	feat(tradein): conflict_resolution — register Yandex sources in priority dicts (#471) ... Stage 7/9 YandexRealtyScraper v1 (Wave 5). Integrates 4 Yandex scrapers into cross-source field-priority arbitration. - HOUSE_FIELD_PRIORITY: lat/lon → yandex_realty_nb; +year_built/house_type/house_class/rating_score/reviews_count Yandex sources - New house fields: text_reviews_count, corpus_count, total_area_ha, commission_year/month, developer_name, has_panorama, yandex_total_listings, has_lift, ceiling_height - LISTING_FIELD_PRIORITY: description/house_type +yandex_detail - New listing fields (all yandex_detail): agency_name/founded_year/objects_count, views_total_yandex, publish_date_relative, sale_type_text - 37 unit tests (7 pre-existing + 30 new), all pass; ruff clean - Resolver engine _resolve() unchanged (this PR creates the file initially) Refs: decisions/Cross_Source_Matching_Strategy.md sec 3.5+4.5, decisions/YandexRealtyScraper_v1_Implementation_Plan.md Stage 7.	2026-05-23 14:03:35 +00:00
lekss361	e46b7d778e	feat(tradein): search matview + indexes (Phase 3.1) (#469) ... Multi-Source Integration Phase 3.1. Foundation for /api/v1/search p95 < 100ms on 100K+ listings. SQL 050_search_optimization.sql: - CREATE EXTENSION IF NOT EXISTS pg_trgm - ALTER listings ADD COLUMN tsv tsvector GENERATED ALWAYS AS (...) STORED - 10 indexes (geom GIST, composite, gin_trgm, GIN tsv, partial kadastr, ...) - CREATE MATERIALIZED VIEW listings_search_mv (per-listing aggregated across sources) - 5 matview indexes (UNIQUE on listing_id for REFRESH CONCURRENTLY) Celery task tradein.refresh_search_matview: - REFRESH MATERIALIZED VIEW CONCURRENTLY listings_search_mv (psycopg v3) - Logged duration via logger.info - Graceful try/except ImportError fallback (no celery_app.py in tradein-mvp yet) - Beat schedule snippet in docstring for future wiring Fixups (deep-code-reviewer 2026-05-23): - h.ratings_count → h.reviews_count (alias house_ratings_count preserved) - Strip +psycopg dialect prefix from DSN for libpq compat Refs Master Plan sec 5.1 + 6.5 + 10.1.	2026-05-23 14:01:18 +00:00
lekss361	51650276b2	feat(tradein): yandex_newbuilding.py — ЖК landing parser (149 selectors + NLP) (#467)	2026-05-23 13:46:45 +00:00
lekss361	7056ecb23b	feat(tradein): yandex_valuation.py — anonymous house-history scraper (#468)	2026-05-23 13:46:20 +00:00
lekss361	3d90221fa0	feat(tradein): yandex_detail.py — Product JSON-LD + DOM detail parser (#466) ... Stage 4 of YandexRealtyScraper v1. YandexDetailScraper.fetch_detail(offer_url) extracts authoritative price from Product JSON-LD (offers.price exact int) + DOM sections (description, agent, stats, metro, photos 8 sizes, newbuilding link, NLP). 35 unit tests, ruff clean. Photo URLs sanitized downstream via _safe_url CDN allowlist (avatars.mds.yandex.net).	2026-05-23 13:45:09 +00:00
lekss361	b0a7235110	feat(tradein): cian_valuation.py — auth-required Calculator scraper + 24h cache (#465)	2026-05-23 13:43:34 +00:00