- nspd_geo: add _save_parcel() for thematic_id=1 → cad_parcels_geom (UPSERT,
ST_Transform from Web Mercator); _persist_target now handles 1/2/5
- parcels.py: analyze endpoint geom lookup extended with cad_parcels_geom as
3rd fallback source (after cad_quarters_geom, cad_buildings); both SELECT
and WKT subqueries updated
- parcels.py: POI score_breakdown items now include lat/lon for map markers
- poi_loader: OSM_CATEGORIES expanded — college+university→school,
hypermarket→shop_supermarket; coverage +3 tag pairs
Two UX nits:
- Caveats badge had whiteSpace:nowrap on a 250-char data_caveat → overflowed
visibly outside the section in the screenshot. Drop nowrap, keep small
font + light grey background. Reads as a compact note.
- Headline KPIs (369 млн ₽ · 19.4 мес · темп 6.0 · ликв 97/100) are
computed by backend at price_factor=1.0, while live KPI cards below
recompute with the current slider position. When user moves price to
+40% the discrepancy is confusing. Append small grey '(базовая цена)'
marker after headline with a tooltip explaining baseline vs live.
The comparables block on /analytics/recommend was showing the same ЖК
multiple times (ЖК СТАРТ × 3, ЭХО ЛЕСА × 2 etc) because domrf_kn_objects
has ~3 historical snapshots per obj_id. The previous query joined all
rows and LIMIT 5 cherry-picked duplicates by flat_count.
Fix: wrap base in CTE `latest_obj` with `DISTINCT ON (obj_id) ORDER BY
obj_id, snapshot_date DESC` to pick only the latest snapshot per object
before sorting by flat_count.
Previously `docker image prune -af --filter until=72h` only removed
images older than 72h. With multiple deploys/day we accumulate many
fresh (<72h) tagged images — each gendesign-backend / worker / frontend
SHA-tag is 500MB-1GB, filling /var/lib/docker.
New logic: for each ghcr.io/lekss361/gendesign-* repo, list tags sorted
by Created DESC, skip :latest, keep top 2, remove the rest. Still runs
the standard prune after for dangling layers + build cache.
Root cause of "auto-resume never fired": the kn_scrape_runs resume section
hit `if not rows: return` (and similar `return` in except) before reaching
the nspd_geo_jobs resume section. Whenever there were no zombie kn-runs
(the normal case), the handler bailed out and geo jobs stayed forever
'running' with stale heartbeats — users had to manual cancel/resume after
every deploy.
Fix: don't return early. Initialize `ids = []`, only run UPDATE if rows
exist, drop the inner `return` from exception branch. The for-loop over
ids becomes a no-op when empty, and execution falls through to the geo
section. Same pattern as the breadcrumb above — fail soft, continue.
cleanup_zombies beat task (added in caa467f) stays as belt-and-suspenders
in case worker_ready signal ever misbehaves again.
worker_ready signal handler was NOT firing in our setup (verified via
DB breadcrumb after 3 deploys — zero rows of stage='worker_ready' in
nspd_geo_log). Root cause of unreliability unknown — possibly Celery
internals, possibly compose recreate timing. Either way, after every
redeploy users had to manually cancel/resume jobs to keep them moving.
Replace signal-based resume with periodic beat task:
- cleanup_zombies runs every minute (* * * * *)
- Finds nspd_geo_jobs in status running/paused with heartbeat >2 min stale
- Sets status='queued' + apply_async with queue=geo
- Idempotent — if no zombies, no-op
worker_ready handler kept (with FK-fix breadcrumb on NULL job_id) for
diagnostic purposes — if signal ever does fire, we'll have evidence.
The 'stale' badge in /admin/scrape/* tables was a UI-only label
computed as Date.now() - heartbeat_at > 60s (90s for geo). Heartbeats
naturally lag past 60s during WAF backoff (30s) or NSPD HTTP timeout
(15s) — every page render flashed running jobs as 'stale' even though
they were processing.
300s threshold matches the real auto-resume threshold in the worker
(jobs are zombies only after >5 min of silence). Less false-positive
churn in the UI.
Two related fixes:
1. worker_process_init handler disposes the SQLAlchemy engine in each
prefork child. Without this, child processes inherit open psycopg
sockets from the parent. First use in a child raises
ProgrammingError: can't change 'autocommit' now: connection in
transaction status INTRANS. This was killing 1 of every 5 parallel
geo jobs on cold start (job 13 in latest bulk run).
2. Add logger.info at start/end of worker_ready resume handler so we
can see in worker logs whether it actually fired and how many jobs
it resumed.
Auto-resume bug: kn and nspd_geo resume-on-worker_ready required
heartbeat >5min / >10min stale. After redeploy worker boots in
~1-2 min, so jobs killed seconds before deploy had fresh heartbeat
and were NEVER auto-resumed — required manual cancel + resume.
Fix: drop time threshold entirely. On worker_ready ANY 'running'
or 'paused' job is by definition a zombie (no worker exists yet),
safe to resume all of them.
Concurrency bump: 5 -> 8 prefork slots. Headroom for 5 geo jobs +
1 kn sweep + 2 objective tasks running simultaneously. Each slot
~150MB RSS -> ~1.2GB total, well within 6GB VPS RAM budget.
- ObjectBuilding type + useObjectBuildings hook
- Objects detail page: render Корпуса ЖК table (cad_num, floors, area,
purpose, name, address) when buildings_count > 0
- Developers page: drop hardcoded compareIds, use top-3 from
useTopDevelopers sorted by jk_count DESC (fallback to legacy static
array when API has no data yet)
Move JobSettings UI from /admin/jobs/settings into JobSettingsPanel component,
render it alongside new BulkGeoPanel on /admin/scrape/all. BulkGeoPanel has a
parallelism input (1-10, default 5) and POSTs /admin/scrape/geo/bulk. Remove
the standalone /admin/jobs/ route and the nav tab pointing to it.
- POST /api/v1/admin/scrape/geo/bulk — splits pending Sverdlovsk cad-nums
into N chunks (parallelism 1..10, default 5), creates N jobs and enqueues
each in queue=geo. source_kind='rosreestr_pending_chunk' for tracking.
- analytics_queries.complex_buildings(db, obj_id) — returns list of buildings
from cad_buildings (cad_num, floors, area, purpose, name, address, geom).
- object_detail: LEFT JOIN v_complex_buildings, adds buildings_count.
- top_developers: adds complexes_count via correlated subquery.
- GET /api/v1/analytics/object/{obj_id}/buildings → list[ComplexBuilding].
Add GET+PUT table for all job_types (scrape_kn, nspd_geo, objective_etl,
objective_sync). Per-row drafts, collapsible extra_config JSON editor,
toast notifications. Tab added to admin layout. Hand-rolled types until
codegen picks up new endpoints.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- JobSetting ORM model (JSONB extra_config) + Pydantic schemas
- GET/PUT /api/v1/admin/jobs/settings + /{job_type} (X-Admin-Token auth)
- celery_app.py builds beat_schedule from job_settings DB (env fallback
retained for safety on first deploy / DB unreachable)
- nspd_geo task reads rate_ms from job_settings when per-job row has
no override
- enqueue/resume geo jobs route to queue_name from job_settings
- Worker container: --queues=celery,scrape_kn,geo (one container,
three named queues — kn sweep no longer blocks nspd_geo)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add `changes` job (dorny/paths-filter@v3) to detect which parts of the
codebase changed. Build jobs now run only when their source tree is
touched; deploy runs as long as no build failed (skipped builds are OK).
Split IMAGE_TAG (latest) from SENTRY_RELEASE_VAL (git sha) so Sentry
release tracking still points to the exact commit while compose pulls
the :latest tag.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Split single build-and-deploy job into 4 jobs: build-backend,
build-worker, build-frontend run in parallel (all 3 independent),
deploy job has needs:[...] on all three. Removes ~2 min of sequential
build time on warm cache runs.
Also: scope GHA cache keys per image (backend-lean / worker-chromium /
frontend) so builds don't invalidate each other's layer cache. Replace
sleep 8 health-check with a 30-iteration polling loop (fails fast on
bad deploy, doesn't over-wait on fast ones). Add --mount=type=cache to
frontend npm ci for node_modules cache across builds.
_save_quarter использовал str(dict).replace("'", '"') как hack для GeoJSON
→ JSON. Это ломалось на Python booleans (True/False/None), которые в JSON
должны быть true/false/null. После dumps=False фикса rosreestr2coord
возвращает Python dict с булями в properties (is_actual=True) → INSERT
падал с psycopg.errors.InvalidTextRepresentation. _save_building уже
использовал json.dumps корректно.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three incremental bugs surfaced while wiring rosreestr2coord v5 into our
worker. All in backend/app/services/scrapers/nspd_lite.py.
1. `Area.__init__()` removed `delay` kwarg in v5 (was throttling). Drop
it from our call. Rate limit now happens in worker via
time.sleep(rate_ms/1000) after each fetch.
2. `Area(..., use_cache=True)` default writes media cache to ./tmp/
relative to CWD. In our docker image CWD=/app (owned by root), worker
runs as non-root → PermissionError on every target. Fix:
use_cache=False, media_path="/tmp/rosreestr2coord".
3. `Area.to_geojson_poly(dumps=True)` default returns JSON-serialized
STRING, not dict (changed in v5). Worker's _persist_target expects
dict with .get("properties") → AttributeError. Fix: dumps=False.
Vault entries:
- fixes/Bug_Rosreestr2coord_Delay_Arg_v5_Fixed.md
- fixes/Bug_Nspd_Geo_Tmp_Permission_Fixed.md
- fixes/Bug_Nspd_Geo_Str_Object_No_Get_Fixed.md
