Commit graph

2296 commits

Author SHA1 Message Date
lekss361
6d3fa8cbd2 add domrf extras parser (sale_graph + sales_agg + infra + photos) +
fix Docker build for playwright

- Regenerate uv.lock with playwright + tenacity (was missing → uv sync
  --frozen skipped them, hence playwright not in /app/.venv/bin/).
- Dockerfile: explicit Chromium runtime libs in apt (libnss3 + 14 more)
  + `playwright install chromium` BEFORE apt-lists cleanup.
- Schema 51_schema_kn_extras.sql: 5 new tables for time-series sales,
  current aggregates, POI infrastructure, photo metadata, failure log.
- Scraper: 4 new fetch helpers + upserts; photo binary download via
  Playwright APIRequest; failure-logging with full_url to kn_scrape_failures.
- CLI: --no-extras / --download-photos / --no-flats / --probe URL.
- Admin UI: extras checkboxes + failures table with copy-URL button.
- PRINZIP smoke: 28 objs, 5409 POI, 5150 photos metadata, sales_agg
  matches probe (Парк Победы 145/291 = 49%).
2026-04-27 18:35:46 +03:00
lekss361
621d56c064 fix 2026-04-27 18:12:57 +03:00
lekss361
1ae4c9522d fix 2026-04-27 18:04:00 +03:00
lekss361
79070bfe99 add interactive parser of наш.дом.рф ЖК+flats with stealth Playwright
- Discovered real endpoints via chrome-devtools: /сервисы/api/kn/object
  (offset/limit/place/objStatus) + /portal-kn/api/sales/portal/table.
  Server-side filter ignored — local filter by developer.companyGroup.
- Pure Playwright in-context fetch (httpx blocked by ServicePipe TLS-fp).
- Saved fingerprint state in data/playwright_state.json for server reuse.
- Celery beat (configurable cron + jitter) + admin trigger endpoint
  + /admin/scrape UI page.
- Schema migration: UNIQUE(id, snapshot_date) for versioned snapshots,
  kn_scrape_runs journal.
- Smoke-tested on PRINZIP: 28 ЖК / 826 квартир as expected.
2026-04-27 17:58:40 +03:00
lekss361
8d3a0874ef add interactive analytics dashboard for Sverdlovsk market and PRINZIP
3 pages (market, PRINZIP drilldown, developers leaderboard) on top of
existing v_developer_full_metrics + domrf_realization views. ECharts on
the frontend, FastAPI router /api/v1/analytics on the backend.
2026-04-27 16:55:30 +03:00
lekss361
16481868a6 add anal 2 2026-04-27 13:05:36 +03:00
lekss361
c13cbaef2e add data 2026-04-26 22:30:52 +03:00
lekss361
a2dcd9844a add link 2026-04-26 18:35:55 +03:00
lekss361
e84115a185 add nspd 2026-04-26 18:20:59 +03:00
lekss361
a5e5ad31e6 add rosreestr 2026-04-26 17:40:05 +03:00
lekss361
6c90c79859 anal.. 2026-04-26 17:14:44 +03:00
lekss361
a2728637ad anal 3 2026-04-26 16:04:11 +03:00
lekss361
0569ccdcf3 anal2 2026-04-26 15:30:02 +03:00
lekss361
ae8abf9f15 domrf api 2026-04-26 15:21:53 +03:00
lekss361
14a9b1415e add anal 2026-04-26 15:18:31 +03:00
lekss361
b132091fe7 docs+memory: HTTPS live, Apr26 catch-up
- README: status table reflects HTTPS-live state
- memory: +HTTPS_Live_Apr26 entity (the chain ESIA→DNS→cert,
  caddy_data wipe needed to clear staging-fallback retry queue)
2026-04-26 14:50:17 +03:00
lekss361
b7230f1bc8 deploy.yml: sync compose+Caddyfile from git, reload caddy after up 2026-04-26 14:26:57 +03:00
lekss361
1f9dde2bf3 backend: pin setuptools package to app/
CI build broke after adding `alembic/` and `db/` directories — flat-layout
auto-discovery saw three top-level dirs and refused to pick. Make it
explicit: only `app/` is a Python package, the rest are ops/migration data.
2026-04-26 13:59:54 +03:00
lekss361
a1633bf6db docs+memory: catch up — domain, Sentry, ops bundle, prod wipe Apr26
- README: status table updated to 2026-04-26 (gendsgn.ru pending DNS,
  Sentry live, healthchecks all green, multi-stage images deployed),
  added ops/ and backend/db/init/ to repo layout, "operational gotchas"
  section (restart vs --force-recreate; reset --hard clobbers manual edits).
- memory/memory-gendesign.jsonl: +5 entities (Domain_Setup_Apr26,
  Sentry_Setup_Apr26, OpsBundle_Apr26, Production_Wipe_Apr26, DockerOps_Apr26)
  + 8 relations linking them into the graph. 34 lines → 47.
2026-04-26 13:42:55 +03:00
lekss361
6c4ba1777a deploy.yml: sync compose+Caddyfile from git, reload caddy after up 2026-04-26 13:31:26 +03:00
lekss361
4f034bd86b ops: alembic baseline, pre-commit, TIGER cleanup, pg_dump scripts
- backend/alembic/* — alembic infra (env.py, script.py.mako). versions/ empty
  for now; first migration goes in Stage 2a when models are finalized.
- backend/Dockerfile: bake alembic.ini + alembic/ into the image so
  `docker compose exec backend alembic upgrade head` works in prod.
- backend/db/init/99_drop_unused_extensions.sql + bind-mount in both compose
  files: drops postgis-image's TIGER/topology/fuzzystrmatch on fresh volumes.
- .pre-commit-config.yaml + pre-commit in dev deps: ruff/prettier on commit
  to stop CI failures like UP035 from leaking out.
- ops/backup.sh, ops/restore.sh: pg_dump cron script with optional Selectel S3
  upload. 7-day local retention. Restore guard: requires RESTORE_CONFIRM=yes.
- Makefile: new targets `make migration NAME=...`, `make pre-commit-install`.
- backend/.env.example: SENTRY_DSN comment with sentry.io reference.
2026-04-26 13:08:51 +03:00
lekss361
68e95aa1ce ops: real domain in Caddyfile + ignore egg-info 2026-04-26 12:49:20 +03:00
lekss361
08a63f611d backend: fix mypy errors in geometry stub
- empty_buildings: dict → dict[str, Any] (mypy strict needs type args)
- drop unused `# type: ignore[arg-type]` on strategy= (mypy now narrows
  the loop variable to Literal[...] from a tuple of literals)
2026-04-26 11:02:22 +03:00
lekss361
d0909e6200 backend: ruff lint fixes
- main.py: AsyncIterator from collections.abc (typing.AsyncIterator deprecated, UP035)
- geometry.py: TEAP first in import (isort: ALL_CAPS before MixedCase, I001)
- pyproject.toml: ignore RUF001/002/003 — Russian-language project legitimately
  uses Cyrillic lookalikes in comments/docstrings (МСК-66 etc).

Verified: ruff check . — All checks passed!

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-26 10:53:50 +03:00
lekss361
e22aeb9b5a Docker: multi-stage backend, npm ci, healthcheck fix, same-origin API
- backend/Dockerfile: split builder/runner, runtime libs only, non-root app user, curl for healthcheck, --frozen via new uv.lock
- frontend/Dockerfile: npm ci instead of npm install (deterministic), USER node
- docker-compose.prod.yml: working backend healthcheck (curl now in image), redis healthcheck, drop dead NEXT_PUBLIC_API_BASE_URL env (Next.js bakes NEXT_PUBLIC_* at build time, runtime override is no-op)
- docker-compose.yml: same redis healthcheck, BACKEND_URL for next.config rewrites, drop uv from CMD (not in new image)
- Caddyfile: handle (not handle_path) so /api prefix is preserved into FastAPI router
- next.config.ts: rewrite /api/* and /health to BACKEND_URL in dev (no Caddy locally)
- frontend/src/lib/api.ts: empty default = same-origin relative URLs
- Makefile: drop uv run from migrate target
- Add backend/uv.lock and frontend/package-lock.json for reproducible builds

Verified: docker build succeeds for both, backend container starts, /health responds, curl healthcheck works inside image, container runs as non-root.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-26 10:47:35 +03:00
lekss361
c8b2cc8152 fix 2026-04-25 20:59:26 +03:00
lekss361
9d95b80b54 test 2026-04-25 20:33:13 +03:00
lekss361
4bd70cde06 Memory: catch up — hosting research, sales tracking, plan A, runbook, hardening 2026-04-25 20:29:07 +03:00
lekss361
09b9f18d3f Add empty frontend/public/ for Next.js standalone Docker build 2026-04-25 20:04:04 +03:00
lekss361
93c5f3f4a9 Fix frontend build: react-leaflet v5 + npm legacy-peer-deps 2026-04-25 20:00:55 +03:00
lekss361
21882182be ix Dockerfile: remove invalid conditional COPY 2026-04-25 19:56:19 +03:00
lekss361
b49cebb577 GHCR migration + env defaults + memory updates 2026-04-25 19:47:07 +03:00
lekss361
384d4d5489 Fix env defaults: use Docker network hostnames + parametrize compose 2026-04-25 19:36:10 +03:00
lekss361
18ade1e252 Memory: hosting research + sales tracking + plan A pivot + VPS runbook 2026-04-25 19:18:44 +03:00
1dcaff0ea2 Memory: ~ent MVP_Scope_AreaSelect_Apr25 (2026-04-25 16:17:12 +05) 2026-04-25 16:17:12 +05:00
ed54d148f9 Memory: ~ent MVP_Scope_AreaSelect_Apr25 (2026-04-25 16:16:11 +05) 2026-04-25 16:16:11 +05:00
917391f045 Memory: ~ent MVP_Scope_AreaSelect_Apr25 (2026-04-25 16:15:11 +05) 2026-04-25 16:15:11 +05:00
726942cd72 Memory: +rel MVP_Scope_AreaSelect_Apr25→NorthStar_DiscoveryMVP (informs), MVP_Scope_AreaSelect_Apr25→Project_GenDesign (refin... (2026-04-25 16:10:12 +05) 2026-04-25 16:10:12 +05:00
d61a5a5e82 Memory: +ent MVP_Scope_AreaSelect_Apr25 (2026-04-25 16:09:11 +05) 2026-04-25 16:09:11 +05:00
29e84e5df9 Memory: non-structural change (2026-04-25 16:01:17 +05) 2026-04-25 16:01:17 +05:00
1df2e74f2e Memory: non-structural change (2026-04-25 16:00:17 +05) 2026-04-25 16:00:17 +05:00
lekss361
b95559eac9 init 2026-04-25 13:45:19 +03:00
lekss361
d66086575a Track memory graph via hardlink to C:\mcp\.aim\memory-gendesign.jsonl 2026-04-25 08:48:59 +03:00
lekss361
f170ef4d35 Merge remote README, keep local 2026-04-25 08:45:06 +03:00
lekss361
1c47f781f0 Initial commit: Discovery MVP docs (v2.1 + v2.2) 2026-04-25 08:40:09 +03:00
lekss361
345695f75f
Initial commit 2026-04-25 08:37:05 +03:00