20 commits

Author	SHA1	Message	Date
lekss361	698ef4f003	feat(tradein): postgres_fdw live read of gendesign.cad_buildings (replaces snapshot) ... Replaces tradein.cad_buildings manual snapshot (36k rows, frozen 2026-05-22) with live postgres_fdw foreign table reading gendesign.v_tradein_cad_buildings directly. Also reverts PR #492 HTTP-based cadastral.py — superseded by FDW. Architecture: - gendesign DB: new role tradein_fdw_reader + flat view v_tradein_cad_buildings (EKB-only slice of cad_buildings with lat/lon flattened from geom) - networks: gendesign-postgres added to gendesign_shared (alias); tradein-postgres added to gendesign_shared for FDW connect - tradein DB: postgres_fdw extension + FOREIGN TABLE gendesign_cad_buildings - tradein backend: startup hook creates/refreshes USER MAPPING with password from env GENDESIGN_FDW_PASSWORD (password rotation handled via restart) - geocoder.py: cadastral primary for forward + reverse + suggest; Yandex/Nominatim fallback. reverse_geocode wraps Nominatim in try/except — no more HTTPStatusError → 500. - house_metadata.py and trade_in.py admin stats switched to foreign table - DROP TABLE cad_buildings in tradein (legacy snapshot removed entirely) - import-cadastre.sh deleted (no manual sync needed) Fixes: - /trade-in/api/v1/geocode/reverse 500 (Nominatim ban → no fallback) - estimate confidence_explanation=address_not_geocoded for addresses in our cadastre (e.g. Хохрякова 81) that Nominatim doesn't return Deploy ordering: main migration 100_tradein_fdw_role.sql adds role+view first (strict deploy.yml). Tradein next deploy applies 060_postgres_fdw_extension.sql and 061_drop_legacy_cad_buildings.sql (idempotent, errors ignored). Tradein backend startup creates USER MAPPING when env var present. Verify post-deploy with: docker exec tradein-postgres psql -U tradein -d tradein -c \ "SELECT count(*) FROM gendesign_cad_buildings" # expect ~36k EKB buildings.	2026-05-24 11:13:44 +03:00
lekss361	0efd2dca1d	feat(tradein): cadastral reverse geocode via main backend cad_buildings (#492)	2026-05-24 07:31:12 +00:00
lekss361	bb7fbd5bc0	feat(sf-b4): GET /landing/stats — 5 KPI + paradox for landing page hero (#331)	2026-05-17 21:47:10 +00:00
lekss361	bf55cc9e56	merge: resolve main.py conflict (keep both pilot + users routers)	2026-05-18 00:08:48 +03:00
lekss361	8673290009	feat(sf-b2): GET /users/me/recent-parcels — stub endpoint (auth TODO) (#329)	2026-05-17 21:04:16 +00:00
lekss361	99e55c0bc2	feat(sf-b3): POST /pilot/request + pilot_requests table ... Добавляет endpoint для приёма заявок на пилот (lead-gen). INSERT в pilot_requests, response {id, created_at, status}. Telegram-уведомление — TODO (creds не настроены, #307 SF-B3). Migration: data/sql/118_pilot_requests.sql	2026-05-18 00:01:35 +03:00
lekss361	29dbbee2ac	feat(trade-in): TI-1 mock /trade-in/estimate endpoint + Pydantic + SQL migration (#316)	2026-05-17 16:45:06 +00:00
lekss361	485f489bd2	feat(#254): user_custom_pois — backend schema + CRUD + scoring integration ... - SQL migration 101_user_custom_pois.sql: table with GEOGRAPHY(POINT) + GIST index - Pydantic schemas: CustomPoiCreate / CustomPoiUpdate / CustomPoiOut - Service custom_pois.py: CRUD + get_overlaps_for_scoring (1km radius, psycopg v3 CAST) - API /api/v1/custom-pois: POST(201)/GET/PATCH/DELETE, X-Session-Id header auth - Scoring in analyze_parcel: custom POI block after OSM POI loop, decay weight * max(0, 1 - dist/1000), custom_poi_score_items in response - Tests: CRUD + validation (weight/lon/lat 422) + scoring integration (mock-based) - Vault: Schema_User_Custom_Pois.md + Endpoint_Parcel_Analyze v3.9 section	2026-05-17 09:34:33 +03:00
lekss361	b61f025673	feat(backend): sentry-sdk init для FastAPI + Celery (#204 backend) (#207)	2026-05-16 15:09:16 +00:00
lekss361	dbf31cc911	feat(etl): objective_complex_mapping backfill via pg_trgm (#203) (#214)	2026-05-16 12:47:09 +00:00
lekss361	5da66bbc93	feat(cadastre): bulk_harvest worker + grid-walker + admin API (#168 PR3/5) (#171) ... * feat(cadastre): bulk_harvest_quarter Celery task + grid-walker + saga state (#168 PR3/5) Add bulk cadastre harvest pipeline: - services/cadastre/bulk_harvest.py: async harvest_quarter() orchestrator (4 phases) + 7 upsert helpers (parcels/buildings/constructions/oncs/enks/zouit/quarter_stats) using CAST(:x AS jsonb) pattern, begin_nested() SAVEPOINT per grid-walk upsert - services/cadastre/grid_geometry.py: quarter_bbox_3857 (PostGIS ST_Extent) + generate_grid_click_points (15x15 = 225 sub-bbox grid) - workers/tasks/scrape_cadastre.py: bulk_harvest_quarter_task (acks_late=True, dont_autoretry_for NspdBulkWafError), enqueue_cadastre_harvest, cleanup_zombies - api/v1/admin_cadastre.py: 5 endpoints behind AdminTokenAuth — create/list/get/cancel/resume - Tests: 13 service unit + 8 API tests * fixup(cadastre): drop importorskip (PR2 merged) — imports now top-level (#168 PR3) * fixup(cadastre): tile_size unified param + register slow marker (#168 PR3) Blocker #1: rename tile_width/tile_height → tile_size in generate_grid_click_points. Callers in bulk_harvest.py and test_cadastre_bulk.py used tile_size; def had tile_width+tile_height → TypeError at runtime. Blocker #2 (10 failures in test_admin_cadastre.py) was side-effect of #1: TypeError at import chain (app.main → admin_cadastre → bulk_harvest) broke FastAPI app load → dependency_overrides AttributeError. Now resolved. Also register `slow` pytest marker in pyproject.toml to suppress PytestUnknownMarkWarning. * fixup(cadastre): use importlib.util.find_spec instead of import (#168 PR3) Root cause Blocker #2: `import app.workers.tasks.scrape_cadastre` at module top-level REBOUND `app` from FastAPI instance (line 14) to the Python package. All subsequent `app.dependency_overrides[get_db] = ...` failed with AttributeError because `app` was now the namespace module. Fix: probe module availability with importlib.util.find_spec — does not import or rebind names. FastAPI `app` instance stays intact. All 28 cadastre tests pass locally. --------- Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 13:31:32 +03:00
lekss361	7250aa6187	feat(site-finder): weight profiles endpoints + analyze integration (#114 sub-PR 3/4) (#138) ... Admin CRUD router + non-breaking analyze_parcel integration. 46 tests pass. Refs: #114 Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 00:40:06 +03:00
lekss361	2fee7739e8	feat(jobs): centralized job_settings API + DB-driven beat schedule ... - JobSetting ORM model (JSONB extra_config) + Pydantic schemas - GET/PUT /api/v1/admin/jobs/settings + /{job_type} (X-Admin-Token auth) - celery_app.py builds beat_schedule from job_settings DB (env fallback retained for safety on first deploy / DB unreachable) - nspd_geo task reads rate_ms from job_settings when per-job row has no override - enqueue/resume geo jobs route to queue_name from job_settings - Worker container: --queues=celery,scrape_kn,geo (one container, three named queues — kn sweep no longer blocks nspd_geo) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-11 15:38:13 +03:00
lekss361	25b73035a1	sprint1: nspd scraper industrialization, per-bucket elasticity, cadastre cross-check, sentry releases ... - NSPD-skraper переехал в backend/app/services/scrapers/nspd_kn.py + Celery task scrape_nspd_region (beat: 20-е февраля/мая/авг/нояб). Redis lock 3h, WAF auto-retry, heartbeat в nspd_scrape_runs. - Recommend_mix Tier 3: per-bucket elasticity через регрессию по «доминирующему bucket» каждого ЖК. Weighted-elasticity для inverse-mode. UI показывает разброс эластичностей и переключение regression/fallback. - Cadastre vs market cross-check: spatial-join cad_buildings → ekb_districts_geom; cadastre_vs_market_pct в scope, аномалии (>+50% / <-30%) подсвечены в UI. - Sentry release tracking (#4): IMAGE_TAG → backend/.env.runtime → sentry_sdk.init(release=...). Compose v2 env_file optional path. Schemas: 63_schema_nspd_runs.sql (cad_buildings + nspd_scrape_runs/log формализуют то, что уже жило в проде через 61_import_nspd_batch.py), 64_v_zk_rosreestr_velocity.sql (refresh с cad_buildings).	2026-04-30 21:51:19 +03:00
lekss361	b3c15bb8ea	fix ui lag	2026-04-27 20:26:55 +03:00
lekss361	5056438fcf	add anal4	2026-04-27 19:45:47 +03:00
lekss361	79070bfe99	add interactive parser of наш.дом.рф ЖК+flats with stealth Playwright ... - Discovered real endpoints via chrome-devtools: /сервисы/api/kn/object (offset/limit/place/objStatus) + /portal-kn/api/sales/portal/table. Server-side filter ignored — local filter by developer.companyGroup. - Pure Playwright in-context fetch (httpx blocked by ServicePipe TLS-fp). - Saved fingerprint state in data/playwright_state.json for server reuse. - Celery beat (configurable cron + jitter) + admin trigger endpoint + /admin/scrape UI page. - Schema migration: UNIQUE(id, snapshot_date) for versioned snapshots, kn_scrape_runs journal. - Smoke-tested on PRINZIP: 28 ЖК / 826 квартир as expected.	2026-04-27 17:58:40 +03:00
lekss361	8d3a0874ef	add interactive analytics dashboard for Sverdlovsk market and PRINZIP ... 3 pages (market, PRINZIP drilldown, developers leaderboard) on top of existing v_developer_full_metrics + domrf_realization views. ECharts on the frontend, FastAPI router /api/v1/analytics on the backend.	2026-04-27 16:55:30 +03:00
lekss361	d0909e6200	backend: ruff lint fixes ... - main.py: AsyncIterator from collections.abc (typing.AsyncIterator deprecated, UP035) - geometry.py: TEAP first in import (isort: ALL_CAPS before MixedCase, I001) - pyproject.toml: ignore RUF001/002/003 — Russian-language project legitimately uses Cyrillic lookalikes in comments/docstrings (МСК-66 etc). Verified: ruff check . — All checks passed! Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-26 10:53:50 +03:00
lekss361	b95559eac9	init	2026-04-25 13:45:19 +03:00