gendesign/backend/tests/services/scrapers
bot-backend 7590b6467e
All checks were successful
CI Trade-In / changes (pull_request) Successful in 10s
CI / changes (pull_request) Successful in 10s
CI Trade-In / backend-tests (pull_request) Has been skipped
CI Trade-In / frontend-checks (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Successful in 2m18s
CI / backend-tests (pull_request) Successful in 15m16s
fix(analytics): de-inflate domrf_kn snapshot counts + backfill UPSERT columns (#2464)
domrf_kn_objects retains ~9 snapshots per obj_id (prod: 4090 rows / 482
distinct obj_id for Строящиеся = 8.49× inflation). _active_competitors_count
and developer_portfolio counted every historical snapshot as a distinct
object; _competitors_two_dim's active CTE had the same class. Dedup via
DISTINCT ON (obj_id) ORDER BY snapshot_date DESC, applying VOLATILE filters
(site_status, district_name, obj_class — prod variance 11/1/29) in the outer
query after dedup so an object is counted by its TRUE-latest snapshot, while
stable scope (region_cd, dev_id — variance 0) stays in the CTE. Competitor
counts and developer portfolios drop ~8× — correct de-inflation.

Also backfill ON CONFLICT DO UPDATE SET: UPSERT_OBJECT_SQL dropped 8
freshly-computed columns on re-scrape, UPSERT_PHOTO_SQL dropped build_type —
added as direct EXCLUDED (not COALESCE, so problem_flag can clear to NULL).

Refs #2464 (clusters E + F).
2026-07-08 12:13:43 +05:00
..
__init__.py feat(22d): domrf_catalog_object scraper — fill ~25 NULL kn_objects cols from SSR __NEXT_DATA__ (#335) 2026-05-17 21:26:22 +00:00
test_domrf_catalog_browsersession_throttle.py fix(scrapers): WAF-guard эндпоинтов + throttled catalog BrowserSession (#2445-D1/D2) (#2459) 2026-07-05 20:09:06 +00:00
test_domrf_catalog_object.py feat(22d): domrf_catalog_object scraper — fill ~25 NULL kn_objects cols from SSR __NEXT_DATA__ (#335) 2026-05-17 21:26:22 +00:00
test_domrf_catalog_object_browsersession_throttle.py fix(scrapers): WAF-guard эндпоинтов + throttled catalog BrowserSession (#2445-D1/D2) (#2459) 2026-07-05 20:09:06 +00:00
test_domrf_catalog_parse.py fix(scrapers): narrow domrf Level-1 status badge-class regex (#1609 follow-up) 2026-06-17 21:21:56 +03:00
test_domrf_flat_plan_url.py feat(scrapers): extract + download DOM.РФ flat plan images (#2440) (#2441) 2026-07-05 16:23:29 +00:00
test_domrf_kn_isolation.py fix(scrapers): разблокировать domrf_kn flats — изоляция flats от poison-extras + throttle (#1945) (#2050) 2026-06-28 13:03:43 +00:00
test_domrf_kn_normalize.py feat(scrapers): wire per-flat DOM.РФ catalog scraper + propagate catalog_url_hash (#2442) (#2444) 2026-07-05 17:00:58 +00:00
test_domrf_kn_upsert_sql.py fix(analytics): de-inflate domrf_kn snapshot counts + backfill UPSERT columns (#2464) 2026-07-08 12:13:43 +05:00
test_ekb_krt_registry.py feat(sf): scraper реестра КРТ-площадок ЕКБ → ekb_krt_sites (#1126) 2026-06-07 11:24:49 +00:00
test_ekb_ppt_tep_parser.py fix(sf): ekb_ppt_tep post-фильтры + врезка в analyze (#1136) 2026-06-13 09:32:52 +00:00
test_flat_plan_download.py feat(scrapers): extract + download DOM.РФ flat plan images (#2440) (#2441) 2026-07-05 16:23:29 +00:00
test_gisogd_so_client.py feat(sf): ingest функц.зон генплана ЕКБ-2045 из ГИСОГД-СО WFS → ekb_genplan_functional_zone (#1137) 2026-06-07 12:03:39 +00:00
test_izyatie_ocr.py fix(workers): izyatie per-doc isolation + newbuilding-crossload breadcrumb (#2445-D5/D6) (#2460) 2026-07-05 20:09:12 +00:00
test_okn_egrkn_client.py fix(scrapers): rate-limit/backoff/circuit-breaker для ОКН EGRKN-скрапера (#2445-D3) (#2461) 2026-07-05 20:09:18 +00:00
test_page_reservation_parser.py feat(sf): ПАГЕ-парсер изъятия/резервирования → land_reservation (#1118) 2026-06-07 10:15:03 +00:00
test_stealth_throttle_proxy.py fix(scrapers): разблокировать domrf_kn flats — изоляция flats от poison-extras + throttle (#1945) (#2050) 2026-06-28 13:03:43 +00:00