lekss361
|
522ecca5c4
|
ship #13-16: multi-target Dockerfile, per-object drill-in, CRM pipeline,
442-sweep instructions
#16 Dockerfile: runner (lean, 1.6 GB) + runner-with-chromium (2.9 GB).
Compose + deploy.yml pushes 2 GHCR images. Backend saves 1.3 GB.
#15 5 backend object endpoints + /analytics/objects/[id] page (gallery,
POI table, sale chart) + sparkline-driven PrinzipObjectsTable.
#14 prinzip_leads/deals + 2 MVs. Import script with PII-hashing &
--inspect-only mode. 3 funnel endpoints + Sankey/Monthly charts.
#13 README section with UI/CLI/SQL for 442-object sweep.
|
2026-04-27 19:32:37 +03:00 |
|
lekss361
|
6d3fa8cbd2
|
add domrf extras parser (sale_graph + sales_agg + infra + photos) +
fix Docker build for playwright
- Regenerate uv.lock with playwright + tenacity (was missing → uv sync
--frozen skipped them, hence playwright not in /app/.venv/bin/).
- Dockerfile: explicit Chromium runtime libs in apt (libnss3 + 14 more)
+ `playwright install chromium` BEFORE apt-lists cleanup.
- Schema 51_schema_kn_extras.sql: 5 new tables for time-series sales,
current aggregates, POI infrastructure, photo metadata, failure log.
- Scraper: 4 new fetch helpers + upserts; photo binary download via
Playwright APIRequest; failure-logging with full_url to kn_scrape_failures.
- CLI: --no-extras / --download-photos / --no-flats / --probe URL.
- Admin UI: extras checkboxes + failures table with copy-URL button.
- PRINZIP smoke: 28 objs, 5409 POI, 5150 photos metadata, sales_agg
matches probe (Парк Победы 145/291 = 49%).
|
2026-04-27 18:35:46 +03:00 |
|
lekss361
|
79070bfe99
|
add interactive parser of наш.дом.рф ЖК+flats with stealth Playwright
- Discovered real endpoints via chrome-devtools: /сервисы/api/kn/object
(offset/limit/place/objStatus) + /portal-kn/api/sales/portal/table.
Server-side filter ignored — local filter by developer.companyGroup.
- Pure Playwright in-context fetch (httpx blocked by ServicePipe TLS-fp).
- Saved fingerprint state in data/playwright_state.json for server reuse.
- Celery beat (configurable cron + jitter) + admin trigger endpoint
+ /admin/scrape UI page.
- Schema migration: UNIQUE(id, snapshot_date) for versioned snapshots,
kn_scrape_runs journal.
- Smoke-tested on PRINZIP: 28 ЖК / 826 квартир as expected.
|
2026-04-27 17:58:40 +03:00 |
|