Extract compute_freshness(db) from the /admin/scrape/freshness handler (endpoint unchanged) + daily beat task scrape_freshness_check that alerts via sentry_sdk.capture_message when a source is stale/failed (error on critical-failed, else warning). Registered in celery include + beat (09:00 MSK). Refs #73
financial_estimate (эпик #1881) фаерит только при разрезолвленном НСПД-регламенте
(max_far), а он резолвится ЛЕНИВО при анализе участка → zone_regulation_cache всего
33 зоны → у большинства участков financial_estimate=None («—» в кокпите). Backfill
проактивно кэширует ВСЕ террзоны ЕКБ (~100-120) → ЛЮБОй участок в известной зоне
сразу получает регламент и финмодель.
- backfill_ekb_zone_regulations (zone_regulation.py): WFS-перечисление террзон ЕКБ
(features_in_bbox 'territorial_zone') → dedup по urban_index (предпочёт фичу с
геометрией) → representative_point (внутри полигона, не centroid) → zone_regulation_at
→ upsert. ИДЕМПОТЕНТНО (cached skip перед fetch), per-zone try/except (один сбой не
рушит batch), rate_delay вежливость к геопорталу, limit для частичного прогона.
- Celery task backfill_zone_regulations + регистрация в celery_app include +
admin POST /api/v1/admin/scrape/zone-regulations/backfill (паттерн objective sync-our).
- Ленивый refresh_zone_regulations / get_or_fetch / upsert НЕ тронуты (additive путь).
- +9 тестов (dedup, idempotency cached-skip, per-zone error isolation, centroid-inside,
limit). mypy/ruff clean.
Прод-прогон backfill — отдельный шаг после deploy (как Objective sync), не на deploy.
Code-review поймал незарегистрированный task (был бы NotRegistered) — исправлено.
Refs #1881
Promote district to a first-class `location` entity (ТЗ §8.2), ADDITIVE — no
district->FK refactor. New `location` table keyed by district_name (joinable by
string), carrying 4 normalized [0,1] indices (NULL when no data, never 0):
- infra: _district_poi_score / _city_avg_poi_score (per-district POI aggregate)
- competition: market_metrics.overstock_index (available/stuck competing supply
— orthogonal to demand; NOT sell-through, which is market-heat correlated w/ demand)
- demand: market_metrics.unit_velocity (saturating /50)
- future_supply: future_supply_pressure.index (passthrough, already 0..1)
- data/sql/146_location.sql: idempotent table + UNIQUE(district_name) + range
CHECK + centroid GIST
- services/site_finder/locations.py: compute_location_indices (reuses forecast
per-district fns) + refresh_locations (SAVEPOINT per-row, CAST, ON CONFLICT)
- workers/tasks/location_refresh.py + beat (Mon 07:00 MSK, after supply-layers)
- api/v1/locations.py: read-only GET list + GET by name (analyst+admin via rbac,
frontend pilot-gated)
- tests: 34 (normalization 0..1/null, competition⊥demand orthogonality, idempotent
upsert, read API list/by-name/404)
Part of #948 (Part A insight shipped #1164).
build_site_finder_report (§22) takes ~30-180s → runs in a background Celery task,
not inline on the sync /analyze endpoint.
- repository: latest_run_for gains keyword-only schema_version (default None keeps
v_analysis_runs_latest behavior, backward-compat); when given, reads base
analysis_runs filtered by schema_version ORDER BY created_at DESC LIMIT 1 — fetches
the latest analyze-1.0 site-analysis run even when newer 1.0 (§22) rows exist on top
(index-served via 127's (cad_num, created_at DESC)).
- new workers/tasks/forecast.py::forecast_site_finder_report: reads latest analyze-1.0,
runs the §22 orchestrator, persists SiteFinderReport.as_dict() as a 1.0 run via
persist_analysis_run. Graceful: no base run / compute error → logger + return None
(worker not crashed). time_limit=900/soft=840 (no global limit). Registered in include.
Prod-confirmed: analyze-1.0 result carries the full analyze dict (competitors+district)
→ orchestrator input valid. Endpoint trigger (3b-ii) + §9.x untouched. 943 tests pass;
code-review APPROVE (contracts verified vs real as_dict(); status done→complete normalized,
no IntegrityError). Refs #994#961.
Root cause of "auto-resume never fired": the kn_scrape_runs resume section
hit `if not rows: return` (and similar `return` in except) before reaching
the nspd_geo_jobs resume section. Whenever there were no zombie kn-runs
(the normal case), the handler bailed out and geo jobs stayed forever
'running' with stale heartbeats — users had to manual cancel/resume after
every deploy.
Fix: don't return early. Initialize `ids = []`, only run UPDATE if rows
exist, drop the inner `return` from exception branch. The for-loop over
ids becomes a no-op when empty, and execution falls through to the geo
section. Same pattern as the breadcrumb above — fail soft, continue.
cleanup_zombies beat task (added in caa467f) stays as belt-and-suspenders
in case worker_ready signal ever misbehaves again.
worker_ready signal handler was NOT firing in our setup (verified via
DB breadcrumb after 3 deploys — zero rows of stage='worker_ready' in
nspd_geo_log). Root cause of unreliability unknown — possibly Celery
internals, possibly compose recreate timing. Either way, after every
redeploy users had to manually cancel/resume jobs to keep them moving.
Replace signal-based resume with periodic beat task:
- cleanup_zombies runs every minute (* * * * *)
- Finds nspd_geo_jobs in status running/paused with heartbeat >2 min stale
- Sets status='queued' + apply_async with queue=geo
- Idempotent — if no zombies, no-op
worker_ready handler kept (with FK-fix breadcrumb on NULL job_id) for
diagnostic purposes — if signal ever does fire, we'll have evidence.
Two related fixes:
1. worker_process_init handler disposes the SQLAlchemy engine in each
prefork child. Without this, child processes inherit open psycopg
sockets from the parent. First use in a child raises
ProgrammingError: can't change 'autocommit' now: connection in
transaction status INTRANS. This was killing 1 of every 5 parallel
geo jobs on cold start (job 13 in latest bulk run).
2. Add logger.info at start/end of worker_ready resume handler so we
can see in worker logs whether it actually fired and how many jobs
it resumed.
Auto-resume bug: kn and nspd_geo resume-on-worker_ready required
heartbeat >5min / >10min stale. After redeploy worker boots in
~1-2 min, so jobs killed seconds before deploy had fresh heartbeat
and were NEVER auto-resumed — required manual cancel + resume.
Fix: drop time threshold entirely. On worker_ready ANY 'running'
or 'paused' job is by definition a zombie (no worker exists yet),
safe to resume all of them.
Concurrency bump: 5 -> 8 prefork slots. Headroom for 5 geo jobs +
1 kn sweep + 2 objective tasks running simultaneously. Each slot
~150MB RSS -> ~1.2GB total, well within 6GB VPS RAM budget.
- JobSetting ORM model (JSONB extra_config) + Pydantic schemas
- GET/PUT /api/v1/admin/jobs/settings + /{job_type} (X-Admin-Token auth)
- celery_app.py builds beat_schedule from job_settings DB (env fallback
retained for safety on first deploy / DB unreachable)
- nspd_geo task reads rate_ms from job_settings when per-job row has
no override
- enqueue/resume geo jobs route to queue_name from job_settings
- Worker container: --queues=celery,scrape_kn,geo (one container,
three named queues — kn sweep no longer blocks nspd_geo)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ОТКРЫТИЕ: stdlib urllib проходит WAF nspd.gov.ru (TLS-fingerprint stdlib
отличается от mainstream HTTP-clients). Это позволяет полностью убрать
Playwright + Chromium для NSPD-задач.
* nspd_lite.py: urllib + ssl._create_unverified_context(). 4 публичных
fetcher'а (geoportal/quarter/parcel/building) + fetch_via_rosreestr2coord
fallback на community-lib
* schema 77: nspd_geo_jobs (журнал) + nspd_geo_targets (cad-номера со
статусом pending/done/failed). Resume-state в БД.
* tasks/nspd_geo.py: Celery task process_nspd_geo_job(job_id). Heartbeat
каждые 5 items, WAF backoff 30s × 2^N (max 8 → paused). UPSERT в
cad_quarters_geom / cad_buildings → идемпотентно.
* worker_ready hook: stale jobs (>10min без heartbeat) автоматически
re-enqueue после redeploy. Никаких потерь прогресса.
* 4 admin endpoint + UI /admin/scrape/geo с формой запуска (manual_list /
rosreestr_pending для авто-наполнения из ДДУ-кварталов), progress-bars
и cancel/resume кнопками per job.
* settings: use_nspd_lite=True, nspd_lite_rate_ms=600.
* dep: rosreestr2coord>=4.0.0 (community lib для fallback).
TODO следующих шагов:
- smoke-test на проде через SSH (validation RU-IP не банит urllib)
- feature toggle USE_NSPD_LITE в scrape_nspd.py для переключения с старого
Playwright-пути
- docker lean image (-300 MB после убирания Chromium)
- расширение SCRAPE_NSPD_DEFAULT_REGIONS на 66,74,72,59 (Челябинск/Тюмень/Пермь)
Objective API (api.objctv.ru) интегрирован как новый source of truth для
per-flat данных по новостройкам УрФО. Заменяет промежуточный Anton-SQLite
(legacy bootstrap-ETL остался как fallback в свёрнутом блоке UI).
Schema (data/sql/68_v2 — applied на проде, 6 таблиц):
- objective_lots (UPSERT по lot_id; 303 677 rows)
- objective_corpus_room_month (long-формат месяц×корпус×room_bucket; 19 738 rows)
- objective_lots_history (append-only weekly snapshots для elasticity)
- objective_complex_mapping (Objective ComplexName ↔ domrf_kn_objects.obj_id)
- objective_raw_reports (jsonb страховка на смену схемы API)
- objective_scrape_runs (журнал прогонов)
+ data/sql/72_objective_sync_config (single-row динамический конфиг)
Backend:
- services/scrapers/objective.py: ObjectiveClient — Bearer-токен (Redis +
in-memory fallback), retry на 401/429/5xx, Retry-After header support
- services/objective_etl.py: ETL SQLite Антона → PG (legacy)
- services/objective_sync_config.py: read/update single-row config
- workers/tasks/scrape_objective.py:
* sync_objective_group: 2 рабочих отчёта (corp_sum, lots_pf), inline-парсинг
* sync_all_groups: wrapper, перебирает группы из БД-конфига с
inter-group паузой; PATCH-merge explicit args > DB config
- workers/tasks/objective_etl.py: Celery task для legacy bootstrap
- workers/celery_app.py: beat читает cron из БД при старте (fallback на env)
- api/v1/admin_scrape.py: 5 новых endpoints для /objective/*
Frontend (frontend/src/app/admin/scrape/objective/page.tsx):
- PRIMARY blue блок «🌐 Наш sync» с input для override групп
- Collapsible «⚙️ Настройки» с формой (cron + 8 параметров) → PUT в БД
- Coverage-панель с PG counts + строка про SQLite Антона как legacy
- Collapsible «🛠 Bootstrap ETL» — legacy-инструмент
Beat schedule: вторник 06:00 МСК, ~10-15 мин на 4 группы (Свердл.обл +
Челябинск + Тюмень + Пермь = ~700K квартир УрФО). Расписание и параметры
меняются через админку без редеплоя (cron требует restart beat).
Эмпирические находки об API (probe 2026-05-10):
- 13 из 21 проверенных group_name доступны на тарифе (включая «Свердловская
область», «Челябинск», «Тюмень», «Пермь», но НЕ «Свердловская обл» —
формат имени строгий)
- ComplexName требует БЕЗ префикса «ЖК» и БЕЗ кавычек
- Поле «Банк» для всех 303k = NULL (тариф не отдаёт), но «Тип обременения»
работает (36% строк = ипотека) → ipoteka_share возможен, банковская
атрибуция — нет
Docker:
- bind-mount /opt/gendesign/site-finder:/data/anton-sqlite:ro в worker
GitHub backlog: добавлены #22-25 (recommend_mix v3 — 4 сабтаска по
улучшению алгоритма на основе фидбэка про POI / границы районов /
конкурентов / окно данных / success-driven mix).
Knowledge graph (memory/memory-gendesign.jsonl): обновлены entities
Objective_Integration_May07_2026, Schema_Objective_v2_May07,
Objective_API_Findings_May07, Module_Objective_Client + новый
Session_End_May07_2026.
TODO для прода: прописать OBJECTIVE_API_KEY=<key> в backend/.env +
docker compose restart worker beat.
В nspd_kn.py:get_pending_cads и admin_scrape.py:nspd_coverage regex-фильтры
quarter_cad_number !~ '^00:00:' и !~ ':0000000$' падали с psycopg.errors.
DataError — двоеточие в text() трактуется как ':param'. Заменил на NOT LIKE
с bindparams (быстрее regex и без issue с парсингом).
Эффект на проде:
- /api/v1/admin/scrape/nspd/coverage возвращал 500 → теперь 200
- Кнопка «Запустить sweep» больше не дизейбленa (pending=964 теперь видно)
- Сам scraper тоже бы валился на этом же баге при beat-запуске
