613 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
lekss361	85f1f90410	fix(yandex-valuation): parse removed_date, fix total_floors regex, normalize NBSP ... - Add removed_date field to ValuationHistoryItem (extracts 2nd DD.MM.YYYY in chunk) - RE_FLOORS: \d+\s+этаж → \d+\s+этажей (plural-only to avoid matching item floors) - Normalize NBSP \xa0 → space in parse() before regex (fixes start_price_per_m2 / last_price_per_m2 = NULL bug) - Add ValuationHouseMeta.validate_match() for cross-checking against expected house Bugs observed on prod: 160 rows in house_placement_history with all removed_date=NULL, total_floors=NULL, *_price_per_m2=NULL.	2026-05-24 16:58:57 +03:00
lekss361	0f7d6c4996	fix(tradein-imv): adapt to Avito 3-step API contract (2026-05+) (#524)	2026-05-24 13:48:52 +00:00
lekss361	bad23095cf	fix(tradein/cian): sanitize valuation cache writes (047 UNIQUE + 026/044 cols) (#523)	2026-05-24 13:44:39 +00:00
lekss361	45ee0a2945	fix(tradein-estimator): extend cohort filter to Tier S + Tier H, fix late_soviet overlap (#522)	2026-05-24 13:14:59 +00:00
lekss361	44cb5b509d	fix(beat): distinguish DB-unreachable from intentional-empty schedule (#520)	2026-05-24 13:14:11 +00:00
lekss361	d9c6b1bd01	chore(tradein): remove dead mock-fallback code from trade_in.py (#521)	2026-05-24 13:11:13 +00:00
lekss361	7cc7fafeeb	feat(tradein-estimator): cohort filter + Tier 0 for analog selection quality (#519)	2026-05-24 13:04:58 +00:00
lekss361	92d00ec874	feat(tradein-ui): mock-disclaimer + progressive enrichment + UX polish (#517)	2026-05-24 12:58:39 +00:00
lekss361	a0d50bd282	feat(tradein-estimator): tiered house-match S→H→W (#507)	2026-05-24 12:41:12 +00:00
lekss361	6cda0cf8d3	fix(tradein-estimator): restore CAST in IS NOT NULL predicates (revert audit #9, prod fix) (#518)	2026-05-24 12:35:28 +00:00
lekss361	ac57b8e905	fix(scrapers): disable catalog-object SSR beat-cron — DOM.РФ WAF cooldown (#516)	2026-05-24 12:21:10 +00:00
lekss361	e45ff47aa6	fix(tradein-sql): remove duplicate CREATE INDEX from 050 (regression from PR #500) (#515)	2026-05-24 12:16:27 +00:00
lekss361	3190bbec54	fix(tradein-frontend): MapPicker focus trap + Leaflet SRI, PhotoUpload size check (#514)	2026-05-24 12:01:17 +00:00
lekss361	9402702f32	feat(tradein-estimator): unique-address-based confidence threshold (#513)	2026-05-24 11:52:00 +00:00
lekss361	3665a61e48	fix(tradein-frontend): safeUrl validator, apiFetch dedup, ListingsCard a11y, error boundary (#512)	2026-05-24 11:47:38 +00:00
lekss361	0767fb7401	fix(tradein-photos): sanitize uploaded images via Pillow re-encode (#511)	2026-05-24 11:32:04 +00:00
lekss361	f04359592f	Merge pull request 'fix(deploy): tradein_fdw_reader password — psql var interpolation in DO block' (#510) from fix/tradein-fdw-psql-var-in-do into main ... Reviewed-on: #510	2026-05-24 11:28:59 +00:00
lekss361	606d3fe7c0	Merge pull request 'chore(rules): drop blocked-scope list from git-pr.md' (#506) from chore/git-pr-rules-any-scope-v2 into main ... Reviewed-on: #506	2026-05-24 11:26:10 +00:00
lekss361	08a8cf51b3	fix(deploy): mute psql set_config stdout — prevent password leak in CI logs ... Review-bot PR #510 нашёл P0: `set_config(name,value,is_local)` возвращает установленное значение, psql печатает rows на stdout без `-q`/`-A`/`-t`, итог — фактический pw появляется в Forgejo Actions deploy logs (retained, visible всем с repo read access). Fix: оборачиваем оба `SELECT set_config(...)` в `\o /dev/null` / `\o` brackets. Output mutes только для этих строк — NOTICE-сообщения из DO block (idempotency signal 'tradein_fdw_reader password set') остаются видимыми. Также добавлен rollback hint в комментарии: НЕ revert файл (вернёт сломанный :'pw' внутри $$), а unset GENDESIGN_FDW_PASSWORD в .env.runtime на VPS.	2026-05-24 14:21:36 +03:00
lekss361	7d8c22ff39	fix(tradein-estimator): atomic IMV link, batched yandex history, drop redundant CAST (#509)	2026-05-24 11:18:12 +00:00
lekss361	d4cbd4bb9f	feat(tradein-scrapers): parse listing_date from Avito + Yandex (#508)	2026-05-24 11:17:13 +00:00
lekss361	35bf050e63	chore(rules): add decision date to self-extending guard (review #506 fixup) ... Reviewer's optional recommendation #3: добавить дату в hard exception для audit trail. Также расширил wording: 'предотвращает bot-loop где bot сам расширяет свои merge права' — делает intent явным для будущих reviewers.	2026-05-24 14:10:53 +03:00
lekss361	7f23a9b3f1	feat(tradein-imv): retry with cleaned address + categorize errors (#505)	2026-05-24 11:04:24 +00:00
lekss361	da7814810c	fix(deploy): tradein_fdw_reader password — psql var interpolation in DO block ... psql :'pw' substitution НЕ работает внутри dollar-quoted блока $$..$$ — это правило psql, не bug. Symptom: deploy 2026-05-24 после merge PR #503 упал с 'syntax error at or near :' on LINE 4 of set_tradein_fdw_password.sql, :'pw' дошёл до сервера как literal вместо интерполяции. Fix: передаём password в DO через session GUC (set_config) — psql интерполирует :'pw' ВНЕ dollar quote, внутри читаем через current_setting(). Очищаем GUC после выполнения (defense-in-depth).	2026-05-24 14:03:15 +03:00
lekss361	c9737d26c9	chore(rules): drop blocked-scope list from git-pr.md — align with any-scope policy ... Header line 10 + memory rule [Auto-merge any scope] (2026-05-16) заявили APPROVE → merge любой scope. Body всё ещё содержал устаревший blocked-list (data/sql, services, scrapers, и т.д.), из-за которого session-side classifier'ы ошибочно блокировали auto-merge approved PR (incident: PRs #501-504 ждали ручного merge). Заменил Auto-merge scope section на Auto-merge policy: - Любой scope OK при verdict=approve + SHA match - Hard exception: secrets в diff + self-extending guard (изменение самих правил всегда через human) Также убрал Scope guard и Blocked-scope cap из Polling loop section.	2026-05-24 13:58:01 +03:00
lekss361	099b8e4b59	fix(scrapers): DOM.РФ KN extras — bypass WAF + update renamed endpoints (#503)	2026-05-24 10:53:28 +00:00
lekss361	6bfa686f16	feat(tradein-estimator): exclude rosreestr deals from output (#504)	2026-05-24 10:52:55 +00:00
lekss361	8fc309d9ad	fix(tradein-avito): strip Emotion CSS from listings.address (#502)	2026-05-24 10:52:50 +00:00
lekss361	1026165abb	fix(tradein-matching): pg_advisory_xact_lock to prevent duplicate house INSERTs (#501)	2026-05-24 10:52:10 +00:00
lekss361	02b48b8d04	fix(tradein-sql): rename dup 030, add trade_in_estimates geom, drop dup indexes (#500) ... Close 3 SQL schema findings from 2026-05-24 trade-in audit. - Rename 030_listings_alter_yandex.sql -> 033_*.sql (resolve dup prefix with 030_avito_imv_cache_key_unique.sql). - New 034_trade_in_estimates_geom.sql: PostGIS geom column + GIST index + backfill + BEFORE INSERT/UPDATE trigger. - New 035_drop_duplicate_indexes.sql: drop listing_sources_listing_idx2 (dup of 028) and houses_geom_idx2 (dup of 009). All idempotent. Deploy script uses ls|sort + no tracking table — rename is safe because original 030 already applied (IF NOT EXISTS no-op on re-run as 033).	2026-05-24 10:04:40 +00:00
lekss361	ed49f6b953	fix(deploy): resolve FDW bootstrap race + extract bootstrap DO block to .sql file (#499)	2026-05-24 09:46:00 +00:00
lekss361	03ab1faa7c	chore(claude): aggressive worktree cleanup via Forgejo PR API (#498)	2026-05-24 09:34:48 +00:00
lekss361	998c8a17a8	chore(claude): agents view readiness — cleanup, .worktreeinclude, workflow (#497)	2026-05-24 09:20:26 +00:00
lekss361	643064ee0c	chore(claude-rules): promote 3 deferred feedback rules from memory (#496)	2026-05-24 09:01:27 +00:00
lekss361	94cf199276	feat(tradein): postgres_fdw live read of gendesign.cad_buildings (replaces snapshot) (#493) ... Replaces tradein.cad_buildings snapshot with live postgres_fdw foreign table reading gendesign.v_tradein_cad_buildings. Fixes /trade-in/api/v1/geocode/reverse 500 (Nominatim ban) and address_not_geocoded for cadastre addresses (e.g. Хохрякова 81). Security (deep-review fixes): - 100_tradein_fdw_role.sql: passwordless CREATE ROLE; password set by deploy.yml ALTER ROLE bootstrap reading GENDESIGN_FDW_PASSWORD from backend/.env.runtime (via psql :'pw' var → format %L — injection-safe). - core/fdw.py: regex whitelist [A-Za-z0-9_-]{32,256} on password, ValueError without echoing value, try/rollback on commit. - 060_postgres_fdw_extension.sql: connect_timeout='3' on FOREIGN SERVER + ALTER ADD/SET fallback. - geocoder.py: _cadastral_forward_sync / _cadastral_reverse_sync wrapped in asyncio.to_thread. - 100_*.sql: REVOKE ALL ON ALL TABLES/SEQUENCES/FUNCTIONS IN SCHEMA public; only GRANT SELECT on v_tradein_cad_buildings. - pg_user_mappings query handles PUBLIC mapping (usename IS NULL). Tests: 3 SQL-injection guards on ensure_fdw_user_mapping + rewritten cadastral suite.	2026-05-24 08:57:30 +00:00
lekss361	73bdff41a7	chore(claude-config): align with 2026 best practices (#495)	2026-05-24 08:48:42 +00:00
lekss361	0efd2dca1d	feat(tradein): cadastral reverse geocode via main backend cad_buildings (#492)	2026-05-24 07:31:12 +00:00
lekss361	7874a5c97b	feat(tradein): source-quota in estimator analogs (fix Cian/Yandex starvation) (#491)	2026-05-23 20:59:58 +00:00
lekss361	222389cba9	fix(tradein/avito): anti-bot hardening — sleep + abort + shared session (#487)	2026-05-23 20:08:40 +00:00
lekss361	726e774a7f	feat(tradein-ui): Yandex admin scraper page — mirror Avito + global delay control (#488) ... Replaces 5-line stub at /scrapers/yandex with full admin page mirroring Avito. 5 sections: global delay (umbrella yandex key), search around, detail snapshot, JK newbuilding, valuation. TanStack Query hooks + apiFetch, TS strict, reuses trade-in.css. Fixup: useScraperSettings unwraps {settings:[...]} response wrapper; PUT body includes required source field (matches backend Pydantic ScraperSettingPayload). useEffect race guarded with initializedRef so 30s refetch doesn't clobber user input mid-edit. Verdict: APPROVE — deep-code-reviewer (response shape unwrap + source field both applied).	2026-05-23 20:07:48 +00:00
lekss361	52716a46c6	fix(tradein): geocode backfill + remove Avito exclusion from estimator (#490)	2026-05-23 20:03:48 +00:00
lekss361	ffb0c67c84	fix(tradein-ui): double /trade-in/ prefix in Cian page Link (PR #486 follow-up) (#489)	2026-05-23 15:56:01 +00:00
lekss361	e79c9cec88	feat(tradein): global scraper delay setting (applies across all scrapers) (#485) ... Add global request delay floor + per-source delay management. - 054: seed 'global'=0 + per-source rows (avito=7, cian=5, n1=5, domrf=5, rosreestr=5) using ON CONFLICT DO NOTHING; reuses existing 053 table from #484 - scraper_settings.py: get_scraper_delay() = max(per_source, global); _GLOBAL_KEY, refactored _get_setting_cached(); preserved yandex umbrella alias logic - admin.py: GET /scraper-settings + PUT /scraper-settings/{source} (ge=0.0, le=60.0); cache invalidation on update; CAST(:d AS numeric) per psycopg v3 - avito/cian/n1 scrapers wired (yandex was wired by #484) - 18 unit tests pass (10 original updated for max() + 8 new global delay tests) Closes follow-up gap from #486 (Cian admin slider min=0 now accepted by API).	2026-05-23 15:43:28 +00:00
lekss361	ebfd29a0a6	feat(tradein-ui): Cian scraper admin page + global request delay control (#486)	2026-05-23 15:36:07 +00:00
lekss361	b21d7c7c85	feat(tradein): scraper_settings live-config + Yandex admin trigger endpoints (#484)	2026-05-23 15:28:34 +00:00
lekss361	65029cd1a7	feat(tradein-ui): Cian cookies upload admin page (#481)	2026-05-23 15:19:42 +00:00
lekss361	fd3cb4e6ed	feat(tradein): /scrapers/avito — 6-я секция Расписание (in-app scheduler UI) (#483)	2026-05-23 14:55:31 +00:00
lekss361	0202720096	feat(tradein): in-app scheduler — UI-managed schedule (replaces SSH crontab) (#482)	2026-05-23 14:54:25 +00:00
lekss361	99dd403c45	fix(tradein): CIAN_REQUIRED_COOKIES — add real auth cookies (DMIR_AUTH + Cian session) (#480) ... Updated CIAN_REQUIRED_COOKIES filter в cian_session.py по реальному DevTools-дампу logged-in cian.ru сессии (2026-05-23). Добавлены critical auth cookies: DMIR_AUTH (httpOnly), _CIAN_GK, _yasc, _ym_visorc, uxfb_card_satisfaction. Legacy entries сохранены как backward-compat fallback. Regression test test_required_cookies_includes_real_auth гарантирует что все реальные cookies проходят filter. Fixes Stage 4 PR #457 (filter guessed без real probe — DMIR_AUTH был dropped → verify_session falsey).	2026-05-23 14:52:24 +00:00
lekss361	9643979d2c	feat(tradein): Phase 3.2 — /api/v1/search endpoint + Redis cache (#479) ... POST /api/v1/search reading listings_search_mv with ~20 cross-source filters, parameterized SQL (CAST(:x AS type), psycopg v3), whitelisted ORDER BY (Pydantic Literal), Redis 5min hot cache with graceful degradation (singleton pool via lru_cache). Verified vs data/sql/050_search_optimization.sql: matview column refs (total_area, lng, house_rating, sources[], has_avito/cian/yandex), SQL injection safety, cache failure swallow, router prefix. Deep-code-reviewer: APPROVE.	2026-05-23 14:44:22 +00:00