A physical building with both Avito ext_house_id (3 listings) and
cian/yandex listings (5 lots, no ext_id) was getting TWO houses rows
(one avito, one derived). Step 4 then split the 8 listings across them.
Fix:
- Step 2 (derived insert): NOT IN (avito-addresses) — skip if already covered
- Step 4 (fuzzy link): COALESCE prefers avito-house at same addr over derived
Adds sanity check: WARN if any normalized address has multiple houses rows.
Backfill 18,428 listings against an empty houses table:
- INSERT Avito/CIAN-sourced houses (~562 unique via house_source+house_ext_id)
- INSERT derived houses for remaining listings (group by tradein_normalize_short_addr,
synthetic ext_house_id = sha256 truncated to 32 hex chars) → ~6102 derived
- UPDATE listings.house_id_fk by both paths (direct: 2997, fuzzy: 15200)
- Best-effort relink orphan house_placement_history via raw_payload.address (0 rows)
Syntax-validated live: 6664 houses, 18197/18428 listings linked (98.8%).
Idempotent (ON CONFLICT DO NOTHING, WHERE house_id_fk IS NULL).
Logs result counters via RAISE NOTICE.
Phase C (Avito IMV history scrape per house) deferred to separate Celery job.
Review-bot PR #510 нашёл P0: `set_config(name,value,is_local)` возвращает
установленное значение, psql печатает rows на stdout без `-q`/`-A`/`-t`,
итог — фактический pw появляется в Forgejo Actions deploy logs (retained,
visible всем с repo read access).
Fix: оборачиваем оба `SELECT set_config(...)` в `\o /dev/null` / `\o`
brackets. Output mutes только для этих строк — NOTICE-сообщения из DO block
(idempotency signal 'tradein_fdw_reader password set') остаются видимыми.
Также добавлен rollback hint в комментарии: НЕ revert файл (вернёт сломанный
:'pw' внутри $$), а unset GENDESIGN_FDW_PASSWORD в .env.runtime на VPS.
Reviewer's optional recommendation #3: добавить дату в hard
exception для audit trail. Также расширил wording: 'предотвращает
bot-loop где bot сам расширяет свои merge права' — делает intent
явным для будущих reviewers.
psql :'pw' substitution НЕ работает внутри dollar-quoted блока $$..$$ — это
правило psql, не bug. Symptom: deploy 2026-05-24 после merge PR #503 упал с
'syntax error at or near :' on LINE 4 of set_tradein_fdw_password.sql, :'pw'
дошёл до сервера как literal вместо интерполяции.
Fix: передаём password в DO через session GUC (set_config) — psql интерполирует
:'pw' ВНЕ dollar quote, внутри читаем через current_setting(). Очищаем GUC
после выполнения (defense-in-depth).
Header line 10 + memory rule [Auto-merge any scope] (2026-05-16)
заявили APPROVE → merge любой scope. Body всё ещё содержал
устаревший blocked-list (data/sql, services, scrapers, и т.д.),
из-за которого session-side classifier'ы ошибочно блокировали
auto-merge approved PR (incident: PRs #501-504 ждали ручного merge).
Заменил Auto-merge scope section на Auto-merge policy:
- Любой scope OK при verdict=approve + SHA match
- Hard exception: secrets в diff + self-extending guard (изменение
самих правил всегда через human)
Также убрал Scope guard и Blocked-scope cap из Polling loop section.
Close 3 SQL schema findings from 2026-05-24 trade-in audit.
- Rename 030_listings_alter_yandex.sql -> 033_*.sql (resolve dup prefix with 030_avito_imv_cache_key_unique.sql).
- New 034_trade_in_estimates_geom.sql: PostGIS geom column + GIST index + backfill + BEFORE INSERT/UPDATE trigger.
- New 035_drop_duplicate_indexes.sql: drop listing_sources_listing_idx2 (dup of 028) and houses_geom_idx2 (dup of 009).
All idempotent. Deploy script uses ls|sort + no tracking table — rename is safe because original 030 already applied (IF NOT EXISTS no-op on re-run as 033).
Replaces tradein.cad_buildings snapshot with live postgres_fdw foreign table reading gendesign.v_tradein_cad_buildings. Fixes /trade-in/api/v1/geocode/reverse 500 (Nominatim ban) and address_not_geocoded for cadastre addresses (e.g. Хохрякова 81).
Security (deep-review fixes):
- 100_tradein_fdw_role.sql: passwordless CREATE ROLE; password set by deploy.yml ALTER ROLE bootstrap reading GENDESIGN_FDW_PASSWORD from backend/.env.runtime (via psql :'pw' var → format %L — injection-safe).
- core/fdw.py: regex whitelist [A-Za-z0-9_-]{32,256} on password, ValueError without echoing value, try/rollback on commit.
- 060_postgres_fdw_extension.sql: connect_timeout='3' on FOREIGN SERVER + ALTER ADD/SET fallback.
- geocoder.py: _cadastral_forward_sync / _cadastral_reverse_sync wrapped in asyncio.to_thread.
- 100_*.sql: REVOKE ALL ON ALL TABLES/SEQUENCES/FUNCTIONS IN SCHEMA public; only GRANT SELECT on v_tradein_cad_buildings.
- pg_user_mappings query handles PUBLIC mapping (usename IS NULL).
Tests: 3 SQL-injection guards on ensure_fdw_user_mapping + rewritten cadastral suite.
Add global request delay floor + per-source delay management.
- 054: seed 'global'=0 + per-source rows (avito=7, cian=5, n1=5, domrf=5, rosreestr=5)
using ON CONFLICT DO NOTHING; reuses existing 053 table from #484
- scraper_settings.py: get_scraper_delay() = max(per_source, global); _GLOBAL_KEY,
refactored _get_setting_cached(); preserved yandex umbrella alias logic
- admin.py: GET /scraper-settings + PUT /scraper-settings/{source} (ge=0.0, le=60.0);
cache invalidation on update; CAST(:d AS numeric) per psycopg v3
- avito/cian/n1 scrapers wired (yandex was wired by #484)
- 18 unit tests pass (10 original updated for max() + 8 new global delay tests)
Closes follow-up gap from #486 (Cian admin slider min=0 now accepted by API).
Updated CIAN_REQUIRED_COOKIES filter в cian_session.py по реальному DevTools-дампу logged-in cian.ru сессии (2026-05-23). Добавлены critical auth cookies: DMIR_AUTH (httpOnly), _CIAN_GK, _yasc, _ym_visorc, uxfb_card_satisfaction. Legacy entries сохранены как backward-compat fallback. Regression test test_required_cookies_includes_real_auth гарантирует что все реальные cookies проходят filter.
Fixes Stage 4 PR #457 (filter guessed без real probe — DMIR_AUTH был dropped → verify_session falsey).
