lekss361/gendesign
1
0
Fork 0
Code Issues 57 Pull requests Projects Releases Packages Wiki Activity Actions
233 commits 226 branches 0 tags 42 MiB
Commit graph

11 commits

Author SHA1 Message Date
lekss361
2f79f7c40a feat(backend): sentry-sdk init для FastAPI + Celery через GlitchTip (#204 backend) 
Подключает sentry-sdk ^2.18 к FastAPI и Celery-воркеру:
- Интеграции: StarletteIntegration, FastApiIntegration, CeleryIntegration,
  SqlalchemyIntegration, HttpxIntegration, LoggingIntegration
- SDK инициализируется до создания FastAPI app (module-level), а также
  отдельно в celery_app.py для Celery-процесса
- Без GLITCHTIP_DSN — SDK no-op, контейнер стартует без ошибок
- profiles_sample_rate=0.0 — GlitchTip профили не поддерживает
- GIT_SHA из ENV как release-тег; fallback "unknown"
 2026-05-16 13:39:17 +03:00
lekss361
5da66bbc93
feat(cadastre): bulk_harvest worker + grid-walker + admin API (#168 PR3/5) (#171) 
* feat(cadastre): bulk_harvest_quarter Celery task + grid-walker + saga state (#168 PR3/5)

Add bulk cadastre harvest pipeline:
- services/cadastre/bulk_harvest.py: async harvest_quarter() orchestrator (4 phases)
  + 7 upsert helpers (parcels/buildings/constructions/oncs/enks/zouit/quarter_stats)
  using CAST(:x AS jsonb) pattern, begin_nested() SAVEPOINT per grid-walk upsert
- services/cadastre/grid_geometry.py: quarter_bbox_3857 (PostGIS ST_Extent)
  + generate_grid_click_points (15x15 = 225 sub-bbox grid)
- workers/tasks/scrape_cadastre.py: bulk_harvest_quarter_task (acks_late=True,
  dont_autoretry_for NspdBulkWafError), enqueue_cadastre_harvest, cleanup_zombies
- api/v1/admin_cadastre.py: 5 endpoints behind AdminTokenAuth — create/list/get/cancel/resume
- Tests: 13 service unit + 8 API tests

* fixup(cadastre): drop importorskip (PR2 merged) — imports now top-level (#168 PR3)

* fixup(cadastre): tile_size unified param + register slow marker (#168 PR3)

Blocker #1: rename tile_width/tile_height → tile_size in generate_grid_click_points.
Callers in bulk_harvest.py and test_cadastre_bulk.py used tile_size; def had
tile_width+tile_height → TypeError at runtime.

Blocker #2 (10 failures in test_admin_cadastre.py) was side-effect of #1:
TypeError at import chain (app.main → admin_cadastre → bulk_harvest)
broke FastAPI app load → dependency_overrides AttributeError. Now resolved.

Also register `slow` pytest marker in pyproject.toml to suppress
PytestUnknownMarkWarning.

* fixup(cadastre): use importlib.util.find_spec instead of import (#168 PR3)

Root cause Blocker #2: `import app.workers.tasks.scrape_cadastre` at module
top-level REBOUND `app` from FastAPI instance (line 14) to the Python package.
All subsequent `app.dependency_overrides[get_db] = ...` failed with
AttributeError because `app` was now the namespace module.

Fix: probe module availability with importlib.util.find_spec — does not
import or rebind names. FastAPI `app` instance stays intact.

All 28 cadastre tests pass locally.

---------

Co-authored-by: lekss361 <claudestars@proton.me>
 2026-05-15 13:31:32 +03:00
lekss361
7250aa6187
feat(site-finder): weight profiles endpoints + analyze integration (#114 sub-PR 3/4) (#138) 
Admin CRUD router + non-breaking analyze_parcel integration. 46 tests pass.

Refs: #114

Co-authored-by: lekss361 <claudestars@proton.me>
 2026-05-15 00:40:06 +03:00
lekss361
2fee7739e8 feat(jobs): centralized job_settings API + DB-driven beat schedule 
- JobSetting ORM model (JSONB extra_config) + Pydantic schemas
- GET/PUT /api/v1/admin/jobs/settings + /{job_type} (X-Admin-Token auth)
- celery_app.py builds beat_schedule from job_settings DB (env fallback
  retained for safety on first deploy / DB unreachable)
- nspd_geo task reads rate_ms from job_settings when per-job row has
  no override
- enqueue/resume geo jobs route to queue_name from job_settings
- Worker container: --queues=celery,scrape_kn,geo (one container,
  three named queues — kn sweep no longer blocks nspd_geo)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-11 15:38:13 +03:00
lekss361
25b73035a1 sprint1: nspd scraper industrialization, per-bucket elasticity, cadastre cross-check, sentry releases 
- NSPD-skraper переехал в backend/app/services/scrapers/nspd_kn.py +
  Celery task scrape_nspd_region (beat: 20-е февраля/мая/авг/нояб).
  Redis lock 3h, WAF auto-retry, heartbeat в nspd_scrape_runs.
- Recommend_mix Tier 3: per-bucket elasticity через регрессию по
  «доминирующему bucket» каждого ЖК. Weighted-elasticity для inverse-mode.
  UI показывает разброс эластичностей и переключение regression/fallback.
- Cadastre vs market cross-check: spatial-join cad_buildings →
  ekb_districts_geom; cadastre_vs_market_pct в scope, аномалии
  (>+50% / <-30%) подсвечены в UI.
- Sentry release tracking (#4): IMAGE_TAG → backend/.env.runtime →
  sentry_sdk.init(release=...). Compose v2 env_file optional path.

Schemas: 63_schema_nspd_runs.sql (cad_buildings + nspd_scrape_runs/log
формализуют то, что уже жило в проде через 61_import_nspd_batch.py),
64_v_zk_rosreestr_velocity.sql (refresh с cad_buildings).
 2026-04-30 21:51:19 +03:00
lekss361
b3c15bb8ea fix ui lag 2026-04-27 20:26:55 +03:00
lekss361
5056438fcf add anal4 2026-04-27 19:45:47 +03:00
lekss361
79070bfe99 add interactive parser of наш.дом.рф ЖК+flats with stealth Playwright 
- Discovered real endpoints via chrome-devtools: /сервисы/api/kn/object
  (offset/limit/place/objStatus) + /portal-kn/api/sales/portal/table.
  Server-side filter ignored — local filter by developer.companyGroup.
- Pure Playwright in-context fetch (httpx blocked by ServicePipe TLS-fp).
- Saved fingerprint state in data/playwright_state.json for server reuse.
- Celery beat (configurable cron + jitter) + admin trigger endpoint
  + /admin/scrape UI page.
- Schema migration: UNIQUE(id, snapshot_date) for versioned snapshots,
  kn_scrape_runs journal.
- Smoke-tested on PRINZIP: 28 ЖК / 826 квартир as expected.
 2026-04-27 17:58:40 +03:00
lekss361
8d3a0874ef add interactive analytics dashboard for Sverdlovsk market and PRINZIP 
3 pages (market, PRINZIP drilldown, developers leaderboard) on top of
existing v_developer_full_metrics + domrf_realization views. ECharts on
the frontend, FastAPI router /api/v1/analytics on the backend.
 2026-04-27 16:55:30 +03:00
lekss361
d0909e6200 backend: ruff lint fixes 
- main.py: AsyncIterator from collections.abc (typing.AsyncIterator deprecated, UP035)
- geometry.py: TEAP first in import (isort: ALL_CAPS before MixedCase, I001)
- pyproject.toml: ignore RUF001/002/003 — Russian-language project legitimately
  uses Cyrillic lookalikes in comments/docstrings (МСК-66 etc).

Verified: ruff check . — All checks passed!

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-26 10:53:50 +03:00
lekss361
b95559eac9 init 2026-04-25 13:45:19 +03:00