refactor(nspd-geo): unify on rosreestr2coord v5, drop legacy column

Объединяю несколько связанных изменений вокруг NSPD geo bulk-fetcher:

Adapt to rosreestr2coord v5 API:
- nspd_lite.fetch_via_rosreestr2coord: drop `delay` kwarg from Area()
  (removed upstream in v5); keep it in our function signature for
  backward-compat, comment why.
- nspd_geo worker: add explicit time.sleep(rate_ms/1000) after lib-branch
  fetch — in v4 the library throttled internally via delay, in v5
  rate limiting is the caller's job. Без этого получали Area.__init__()
  unexpected kwarg `delay` на каждом target.

Drop use_rosreestr2coord switch:
- Removed urllib-vs-lib choice everywhere. We always use community
  rosreestr2coord library — авторы регулярно обновляют WAF-tricks,
  наш urllib-fetcher (fetch_geoportal) уже неактуален.
- admin_scrape.py: Pydantic schema, INSERT, SELECT, API response
  cleaned of `use_rosreestr2coord`.
- nspd_geo.enqueue_geo_job: param dropped, INSERT shrunk.
- worker process loop: dropped `if use_lib:` branch + import of
  fetch_geoportal.
- frontend/geo/page.tsx: removed checkbox + GeoJob.use_rosreestr2coord
  field + POST body field.

DB column drop:
- data/sql/78_drop_use_rosreestr2coord.sql (NEW):
  DROP COLUMN nspd_geo_jobs.use_rosreestr2coord + CREATE OR REPLACE
  VIEW v_scrape_runs_unified (которая depended on the column).
- data/sql/77_nspd_geo_jobs.sql: cleaned historical DDL for fresh setups.
- Migration applied to prod (in-conversation via postgres MCP).

Frontend polish:
- Thematic ID changed from free-form number input to labeled select
  (1=parcel / 2=quarter / 4=admin / 5=building / 7=zone / 15=complex).
- Auto-sync thematic_id from Job kind on change (override possible).
- ScrapeLogsPanel: extended union type with "nspd_geo" + fixed
  /admin/scrape/geo to pass scraperType="nspd_geo" (was "nspd",
  filtering empty legacy nspd_scrape_log table; real logs live in
  nspd_geo_log via v_scrape_log_unified).

Verified: ruff ✓, tsc --noEmit ✓, migration ran (BEGIN..COMMIT clean).

Deploy order safe: prod column уже удалена → новый backend код, который
не INSERT'ит use_rosreestr2coord, совпадёт со схемой после deploy.

backend/app/api/v1/admin_scrape.py

@@ -618,7 +618,6 @@ class EnqueueGeoJobRequest(BaseModel):
         default=None, max_length=20, description="Для source=rosreestr_pending"
     )
     rate_ms: int = Field(default=600, ge=100, le=10000)
-    use_rosreestr2coord: bool = False
 
 
 @router.post("/geo/jobs")
@@ -681,7 +680,6 @@ def enqueue_geo_job(
         source_params={"region_codes": payload.region_codes, "thematic_id": payload.thematic_id},
         cad_nums_with_thematic=cad_with_thematic,
         rate_ms=payload.rate_ms,
-        use_rosreestr2coord=payload.use_rosreestr2coord,
         triggered_by="manual",
     )
     process_nspd_geo_job.apply_async(args=[job_id])
@@ -708,7 +706,7 @@ def list_geo_jobs(
                    started_at, finished_at, heartbeat_at,
                    targets_total, targets_done, targets_failed, targets_skipped,
                    requests_count, waf_blocked_count, error,
-                   rate_ms, use_rosreestr2coord, created_at
+                   rate_ms, created_at
               FROM nspd_geo_jobs
              ORDER BY created_at DESC
              LIMIT :lim
@@ -738,7 +736,6 @@ def list_geo_jobs(
             "waf_blocked_count": r["waf_blocked_count"],
             "error": r["error"],
             "rate_ms": r["rate_ms"],
-            "use_rosreestr2coord": r["use_rosreestr2coord"],
             "created_at": r["created_at"].isoformat() if r["created_at"] else None,
             "progress_pct": round(
                 100.0 * (r["targets_done"] or 0) / max(r["targets_total"] or 1, 1), 1

backend/app/services/scrapers/nspd_lite.py

@@ -145,16 +145,21 @@ def fetch_via_rosreestr2coord(
     cad_num: str,
     area_type: int = 1,
     timeout: int = 15,
-    delay: float = 1.0,
+    delay: float = 0.0,  # legacy-параметр, оставлен для совместимости signature
 ) -> dict[str, Any] | None:
-    """Тот же запрос но через community-библиотеку rosreestr2coord.
+    """Тот же запрос но через community-библиотеку rosreestr2coord (v5+).
 
     Преимущество: они поддерживают upstream обновления headers/WAF-tricks.
     Использовать когда наш fetch_* поломается из-за изменений WAF.
 
+    Параметр `delay` сохранён в signature для обратной совместимости с воркером,
+    но в rosreestr2coord v5 он убран — rate limiting между запросами теперь
+    делает наш воркер через `rate_ms` (см. nspd_geo.py).
+
     Returns:
         GeoJSON Feature или None если participant не найден.
     """
+    _ = delay  # silence unused — см. docstring выше
     try:
         from rosreestr2coord.parser import Area
     except ImportError as e:
@@ -164,7 +169,6 @@ def fetch_via_rosreestr2coord(
         code=cad_num,
         area_type=area_type,
         timeout=timeout,
-        delay=delay,
         with_log=False,
     )
     try:

backend/app/workers/tasks/nspd_geo.py

@@ -29,7 +29,6 @@ from app.core.db import SessionLocal
 from app.services.scrapers.nspd_lite import (
     NspdLiteError,
     NspdLiteWafError,
-    fetch_geoportal,
     fetch_via_rosreestr2coord,
 )
 from app.workers.celery_app import celery_app
@@ -228,7 +227,6 @@ def enqueue_geo_job(
     source_params: dict[str, Any],
     cad_nums_with_thematic: list[tuple[str, int]],
     rate_ms: int = 600,
-    use_rosreestr2coord: bool = False,
     triggered_by: str = "manual",
 ) -> int:
     """Создаёт job + targets и возвращает job_id. Не enqueue'ит task сам.
@@ -243,8 +241,8 @@ def enqueue_geo_job(
                 """
                 INSERT INTO nspd_geo_jobs (
                     name, job_kind, source_kind, source_params,
-                    rate_ms, use_rosreestr2coord, triggered_by, status, targets_total
-                ) VALUES (:n, :k, :sk, CAST(:sp AS jsonb), :r, :ur, :tb, 'queued', :tt)
+                    rate_ms, triggered_by, status, targets_total
+                ) VALUES (:n, :k, :sk, CAST(:sp AS jsonb), :r, :tb, 'queued', :tt)
                 RETURNING job_id
                 """
             ),
@@ -254,7 +252,6 @@ def enqueue_geo_job(
                 "sk": source_kind,
                 "sp": _json.dumps(source_params, ensure_ascii=False),
                 "r": rate_ms,
-                "ur": use_rosreestr2coord,
                 "tb": triggered_by,
                 "tt": len(cad_nums_with_thematic),
             },
@@ -311,7 +308,6 @@ def process_nspd_geo_job(self: Any, job_id: int) -> dict[str, Any]:
 
         _start_job(db, job_id)
         rate_ms = int(job["rate_ms"] or 600)
-        use_lib = bool(job["use_rosreestr2coord"])
 
         # Текущие счётчики (для resume — продолжаем с тех значений)
         done = int(job["targets_done"] or 0)
@@ -321,9 +317,7 @@ def process_nspd_geo_job(self: Any, job_id: int) -> dict[str, Any]:
         n_waf = int(job["waf_blocked_count"] or 0)
         consecutive_waf = 0
 
-        logger.info(
-            "process_nspd_geo_job start: job=%s use_lib=%s rate=%dms", job_id, use_lib, rate_ms
-        )
+        logger.info("process_nspd_geo_job start: job=%s rate=%dms", job_id, rate_ms)
 
         while True:
             target = (
@@ -349,15 +343,19 @@ def process_nspd_geo_job(self: Any, job_id: int) -> dict[str, Any]:
             cad = target["cad_num"]
 
             try:
-                if use_lib:
-                    payload = {
-                        "data": {
-                            "features": [fetch_via_rosreestr2coord(cad, area_type=tid, timeout=15)]
-                        }
+                # NSPD geo fetch теперь только через rosreestr2coord lib (v5+).
+                # Старая ветка с urllib-fetch (fetch_geoportal) удалена — community
+                # пакет нам надёжнее, авторы обновляют под WAF.
+                payload = {
+                    "data": {
+                        "features": [fetch_via_rosreestr2coord(cad, area_type=tid, timeout=15)]
                     }
-                    payload["data"]["features"] = [f for f in payload["data"]["features"] if f]
-                else:
-                    payload = fetch_geoportal(cad, thematic_id=tid, rate_ms=rate_ms)
+                }
+                payload["data"]["features"] = [f for f in payload["data"]["features"] if f]
+                # rosreestr2coord v5 убрал internal delay-параметр → rate limit
+                # делаем здесь, чтобы не словить WAF за высокую частоту.
+                if rate_ms > 0:
+                    time.sleep(rate_ms / 1000.0)
                 n_requests += 1
                 consecutive_waf = 0
 

data/sql/77_nspd_geo_jobs.sql

@@ -26,11 +26,9 @@ CREATE TABLE IF NOT EXISTS nspd_geo_jobs (
     -- Источник целей
     source_kind     TEXT NOT NULL,                 -- 'rosreestr_pending' | 'manual_list' | 'region_bbox'
     source_params   jsonb,                         -- что-то вроде {"region_codes":[66,74,72,59], "thematic_id":2}
-    -- Параметры fetch
+    -- Параметры fetch (всегда через rosreestr2coord lib v5+;
+    -- legacy колонка `use_rosreestr2coord` удалена миграцией 78_).
     rate_ms         INTEGER NOT NULL DEFAULT 600,
-    -- TRUE = community-lib rosreestr2coord (default с 2026-05-11),
-    -- FALSE = наш nspd_lite urllib-based (fallback).
-    use_rosreestr2coord BOOLEAN NOT NULL DEFAULT TRUE,
     -- Состояние
     status          TEXT NOT NULL DEFAULT 'queued', -- queued | running | done | failed | paused
     started_at      timestamptz,
@@ -134,7 +132,7 @@ SELECT 'nspd_geo', job_id, started_at, finished_at, heartbeat_at, status, error,
        requests_count, targets_done, targets_failed, targets_skipped,
        targets_done, targets_total,
        jsonb_build_object('job_kind', job_kind, 'source_kind', source_kind,
-                          'rate_ms', rate_ms, 'use_rosreestr2coord', use_rosreestr2coord,
+                          'rate_ms', rate_ms,
                           'waf_blocked_count', waf_blocked_count,
                           'source_params', source_params)
   FROM nspd_geo_jobs;

data/sql/78_drop_use_rosreestr2coord.sql

@@ -0,0 +1,58 @@
+-- Drop legacy `use_rosreestr2coord` column from nspd_geo_jobs.
+--
+-- Контекст:
+-- В первой итерации NSPD geo bulk-fetcher был выбор между нашим urllib-fetcher
+-- (fetch_geoportal) и community-библиотекой rosreestr2coord. На практике
+-- библиотека намного устойчивее к WAF — её авторы регулярно обновляют
+-- headers/tricks. Поэтому отказались от urllib-ветки и теперь ВСЕГДА используем
+-- rosreestr2coord. Колонка больше не несёт смысла.
+--
+-- Порядок: сначала пересоздаём v_scrape_runs_unified без зависимости от
+-- колонки, потом DROP COLUMN.
+
+BEGIN;
+
+-- Чистим возможную мусорную view с опечаткой (создавалась предыдущей попыткой
+-- миграции — без 's' в имени, ошибка).
+DROP VIEW IF EXISTS v_scrape_run_unified;
+
+-- Главная unified-view (см. 77_nspd_geo_jobs.sql) пересоздаётся без
+-- use_rosreestr2coord в JSONB-секции nspd_geo.
+CREATE OR REPLACE VIEW v_scrape_runs_unified AS
+SELECT 'kn' AS scraper_type, run_id, started_at, finished_at, heartbeat_at,
+       status, error, NULL::text AS triggered_by,
+       region_codes::text AS scope, requests_count,
+       objects_count AS items_ok, NULL::integer AS items_failed,
+       flats_count AS sub_items_ok,
+       progress_obj_index AS progress_idx, total_obj_count AS progress_total,
+       jsonb_build_object('developers', developer_ids, 'snapshot_date', snapshot_date,
+                          'resumed_from', resumed_from_run_id, 'params', params) AS extra
+  FROM kn_scrape_runs
+UNION ALL
+SELECT 'nspd', run_id, started_at, finished_at, heartbeat_at, status, error,
+       triggered_by, region_code::text, requests_count,
+       quarters_ok, quarters_failed, buildings_ok,
+       NULL, pending_count,
+       jsonb_build_object('waf_429_count', waf_429_count)
+  FROM nspd_scrape_runs
+UNION ALL
+SELECT 'objective', run_id, started_at, finished_at, heartbeat_at, status, error,
+       triggered_by, group_name, requests_count,
+       reports_ok, reports_failed, rows_lots,
+       NULL, NULL,
+       jsonb_build_object('rows_corpus_room', rows_corpus_room, 'rows_history', rows_history)
+  FROM objective_scrape_runs
+UNION ALL
+SELECT 'nspd_geo', job_id, started_at, finished_at, heartbeat_at, status, error,
+       triggered_by, COALESCE(name, job_kind),
+       requests_count, targets_done, targets_failed, targets_skipped,
+       targets_done, targets_total,
+       jsonb_build_object('job_kind', job_kind, 'source_kind', source_kind,
+                          'rate_ms', rate_ms,
+                          'waf_blocked_count', waf_blocked_count,
+                          'source_params', source_params)
+  FROM nspd_geo_jobs;
+
+ALTER TABLE nspd_geo_jobs DROP COLUMN IF EXISTS use_rosreestr2coord;
+
+COMMIT;

frontend/src/app/admin/scrape/geo/page.tsx

@@ -24,7 +24,6 @@ interface GeoJob {
   waf_blocked_count: number;
   error: string | null;
   rate_ms: number;
-  use_rosreestr2coord: boolean;
   created_at: string | null;
   progress_pct: number;
 }
@@ -71,10 +70,18 @@ export default function GeoScrapeAdminPage() {
     "quarters",
   );
   const [thematicId, setThematicId] = useState(2);
+
+  // Авто-синк thematic_id при смене job_kind. Пользователь может override через
+  // селект ниже (например quarters + thematic_id=15 для комплексов).
+  const handleJobKindChange = (kind: "quarters" | "parcels" | "buildings") => {
+    setJobKind(kind);
+    if (kind === "quarters") setThematicId(2);
+    else if (kind === "parcels") setThematicId(1);
+    else if (kind === "buildings") setThematicId(5);
+  };
   const [cadNumsText, setCadNumsText] = useState("");
   const [regionCodes, setRegionCodes] = useState("66,74,72,59");
   const [rateMs, setRateMs] = useState(600);
-  const [useLib, setUseLib] = useState(true);
   const [lastResp, setLastResp] = useState<EnqueueResp | null>(null);
   const queryClient = useQueryClient();
 
@@ -120,7 +127,6 @@ export default function GeoScrapeAdminPage() {
           thematic_id: thematicId,
           region_codes,
           rate_ms: rateMs,
-          use_rosreestr2coord: useLib,
         }),
       });
     },
@@ -220,27 +226,33 @@ export default function GeoScrapeAdminPage() {
               <select
                 value={jobKind}
                 onChange={(e) =>
-                  setJobKind(
+                  handleJobKindChange(
                     e.target.value as "quarters" | "parcels" | "buildings",
                   )
                 }
                 style={inputStyle}
               >
-                <option value="quarters">кадастровые кварталы (2)</option>
-                <option value="parcels">участки ЗУ (1)</option>
-                <option value="buildings">здания ОКС (5)</option>
+                <option value="quarters">кадастровые кварталы</option>
+                <option value="parcels">участки ЗУ</option>
+                <option value="buildings">здания ОКС</option>
               </select>
             </label>
             <label>
-              <span style={labelStyle}>Thematic ID</span>
-              <input
-                type="number"
+              <span style={labelStyle}>
+                Thematic ID — NSPD thematicSearchId
+              </span>
+              <select
                 value={thematicId}
-                onChange={(e) =>
-                  setThematicId(parseInt(e.target.value, 10) || 2)
-                }
+                onChange={(e) => setThematicId(parseInt(e.target.value, 10))}
                 style={inputStyle}
-              />
+              >
+                <option value={1}>1 — parcel (земельный участок)</option>
+                <option value={2}>2 — quarter (кадастровый квартал)</option>
+                <option value={4}>4 — admin (адм-территориальное)</option>
+                <option value={5}>5 — building (ОКС)</option>
+                <option value={7}>7 — zone (территориальная зона)</option>
+                <option value={15}>15 — complex (комплекс объектов)</option>
+              </select>
             </label>
           </div>
 
@@ -292,16 +304,6 @@ export default function GeoScrapeAdminPage() {
                 style={inputStyle}
               />
             </label>
-            <label style={{ display: "flex", alignItems: "flex-end", gap: 8 }}>
-              <input
-                type="checkbox"
-                checked={useLib}
-                onChange={(e) => setUseLib(e.target.checked)}
-              />
-              <span>
-                Использовать rosreestr2coord lib (вместо нашего urllib)
-              </span>
-            </label>
           </div>
 
           <div style={{ display: "flex", gap: 12, alignItems: "center" }}>
@@ -520,9 +522,9 @@ export default function GeoScrapeAdminPage() {
 
       <ScrapeLogsPanel
         token={token}
-        scraperType="nspd"
+        scraperType="nspd_geo"
         title="Журнал NSPD geo (per-target)"
-        subtitle="фильтр scraper_type=nspd · обновляется каждые 3 сек"
+        subtitle="из nspd_geo_log · обновляется каждые 3 сек"
         containerStyle={{ marginTop: 16 }}
       />
     </>

frontend/src/components/admin/ScrapeLogsPanel.tsx

@@ -19,8 +19,8 @@ interface LogRow {
 interface Props {
   /** Admin token (для X-Admin-Token header). */
   token: string;
-  /** scraper_type filter — `kn`, `nspd`, `objective`. Передаётся в `/admin/scrape/all/logs`. */
-  scraperType: "kn" | "nspd" | "objective";
+  /** scraper_type filter — kn / nspd / nspd_geo / objective. */
+  scraperType: "kn" | "nspd" | "nspd_geo" | "objective";
   /** Имя поля по которому фильтруем run в запросе (по умолчанию `run_id`). */
   runIdParam?: string;
   /** Лимит записей (default 200). */