3919a40c49
Объединяю несколько связанных изменений вокруг NSPD geo bulk-fetcher: Adapt to rosreestr2coord v5 API: - nspd_lite.fetch_via_rosreestr2coord: drop `delay` kwarg from Area() (removed upstream in v5); keep it in our function signature for backward-compat, comment why. - nspd_geo worker: add explicit time.sleep(rate_ms/1000) after lib-branch fetch — in v4 the library throttled internally via delay, in v5 rate limiting is the caller's job. Без этого получали Area.__init__() unexpected kwarg `delay` на каждом target. Drop use_rosreestr2coord switch: - Removed urllib-vs-lib choice everywhere. We always use community rosreestr2coord library — авторы регулярно обновляют WAF-tricks, наш urllib-fetcher (fetch_geoportal) уже неактуален. - admin_scrape.py: Pydantic schema, INSERT, SELECT, API response cleaned of `use_rosreestr2coord`. - nspd_geo.enqueue_geo_job: param dropped, INSERT shrunk. - worker process loop: dropped `if use_lib:` branch + import of fetch_geoportal. - frontend/geo/page.tsx: removed checkbox + GeoJob.use_rosreestr2coord field + POST body field. DB column drop: - data/sql/78_drop_use_rosreestr2coord.sql (NEW): DROP COLUMN nspd_geo_jobs.use_rosreestr2coord + CREATE OR REPLACE VIEW v_scrape_runs_unified (которая depended on the column). - data/sql/77_nspd_geo_jobs.sql: cleaned historical DDL for fresh setups. - Migration applied to prod (in-conversation via postgres MCP). Frontend polish: - Thematic ID changed from free-form number input to labeled select (1=parcel / 2=quarter / 4=admin / 5=building / 7=zone / 15=complex). - Auto-sync thematic_id from Job kind on change (override possible). - ScrapeLogsPanel: extended union type with "nspd_geo" + fixed /admin/scrape/geo to pass scraperType="nspd_geo" (was "nspd", filtering empty legacy nspd_scrape_log table; real logs live in nspd_geo_log via v_scrape_log_unified). Verified: ruff ✓, tsc --noEmit ✓, migration ran (BEGIN..COMMIT clean). Deploy order safe: prod column уже удалена → новый backend код, который не INSERT'ит use_rosreestr2coord, совпадёт со схемой после deploy.
58 lines
3 KiB
PL/PgSQL
58 lines
3 KiB
PL/PgSQL
-- Drop legacy `use_rosreestr2coord` column from nspd_geo_jobs.
|
||
--
|
||
-- Контекст:
|
||
-- В первой итерации NSPD geo bulk-fetcher был выбор между нашим urllib-fetcher
|
||
-- (fetch_geoportal) и community-библиотекой rosreestr2coord. На практике
|
||
-- библиотека намного устойчивее к WAF — её авторы регулярно обновляют
|
||
-- headers/tricks. Поэтому отказались от urllib-ветки и теперь ВСЕГДА используем
|
||
-- rosreestr2coord. Колонка больше не несёт смысла.
|
||
--
|
||
-- Порядок: сначала пересоздаём v_scrape_runs_unified без зависимости от
|
||
-- колонки, потом DROP COLUMN.
|
||
|
||
BEGIN;
|
||
|
||
-- Чистим возможную мусорную view с опечаткой (создавалась предыдущей попыткой
|
||
-- миграции — без 's' в имени, ошибка).
|
||
DROP VIEW IF EXISTS v_scrape_run_unified;
|
||
|
||
-- Главная unified-view (см. 77_nspd_geo_jobs.sql) пересоздаётся без
|
||
-- use_rosreestr2coord в JSONB-секции nspd_geo.
|
||
CREATE OR REPLACE VIEW v_scrape_runs_unified AS
|
||
SELECT 'kn' AS scraper_type, run_id, started_at, finished_at, heartbeat_at,
|
||
status, error, NULL::text AS triggered_by,
|
||
region_codes::text AS scope, requests_count,
|
||
objects_count AS items_ok, NULL::integer AS items_failed,
|
||
flats_count AS sub_items_ok,
|
||
progress_obj_index AS progress_idx, total_obj_count AS progress_total,
|
||
jsonb_build_object('developers', developer_ids, 'snapshot_date', snapshot_date,
|
||
'resumed_from', resumed_from_run_id, 'params', params) AS extra
|
||
FROM kn_scrape_runs
|
||
UNION ALL
|
||
SELECT 'nspd', run_id, started_at, finished_at, heartbeat_at, status, error,
|
||
triggered_by, region_code::text, requests_count,
|
||
quarters_ok, quarters_failed, buildings_ok,
|
||
NULL, pending_count,
|
||
jsonb_build_object('waf_429_count', waf_429_count)
|
||
FROM nspd_scrape_runs
|
||
UNION ALL
|
||
SELECT 'objective', run_id, started_at, finished_at, heartbeat_at, status, error,
|
||
triggered_by, group_name, requests_count,
|
||
reports_ok, reports_failed, rows_lots,
|
||
NULL, NULL,
|
||
jsonb_build_object('rows_corpus_room', rows_corpus_room, 'rows_history', rows_history)
|
||
FROM objective_scrape_runs
|
||
UNION ALL
|
||
SELECT 'nspd_geo', job_id, started_at, finished_at, heartbeat_at, status, error,
|
||
triggered_by, COALESCE(name, job_kind),
|
||
requests_count, targets_done, targets_failed, targets_skipped,
|
||
targets_done, targets_total,
|
||
jsonb_build_object('job_kind', job_kind, 'source_kind', source_kind,
|
||
'rate_ms', rate_ms,
|
||
'waf_blocked_count', waf_blocked_count,
|
||
'source_params', source_params)
|
||
FROM nspd_geo_jobs;
|
||
|
||
ALTER TABLE nspd_geo_jobs DROP COLUMN IF EXISTS use_rosreestr2coord;
|
||
|
||
COMMIT;
|