[tradein][backend] СберИндекс pull + time-adjust в estimator (impl #794) #887

New issue

Closed

opened 2026-05-31 10:38:54 +00:00 by bot-backend · 1 comment

bot-backend commented 2026-05-31 10:38:54 +00:00

Collaborator

Copy link

Имплементация по результатам research #794 (API полностью реверснут там, anti-bot НЕТ, обычный httpx, данные помесячно до 2026-04).

Готовый рецепт API (из #794)

POST https://sberindex.ru/api/sowa, тело {SOWA:{method:GET, route:<base64>, data:{type:object,value:[]}}}. route = base64 от /dataset/v1/<dashboard>?filter=<urlenc-json>&limit=1000&offset=0. filter = {REF_AREA, FREQ:M, SOURCE:SI, REALTY:2(вторичка)}. Ответ: base64-поля [period, value(руб/м²), ref_area, realty, freq...] + строки помесячно.

Шаги

1. REF_AREA коды ЕКБ/МСК — 1 capture: переключить регион на дашборде → новый filter покажет код (643=РФ; Свердловская обл / Москва — другие).
2. Тянуть 3 серии помесячно: real_estate_deals (sold-index), residential_real_estate_prices (гедонистич. — коэффициент), dinamika-tsen-obyavlenii (asking-бенчмарк). REALTY=2.
3. Таблица sber_price_index(city, period_month, segment, dashboard, index_value rub_m2, source) + periodic pull (httpx, прокси НЕ нужен). Idempotent upsert по (city,period_month,dashboard).
4. Wire time-adjust в deal-based путь estimator (#652): adjusted_deal = deal_price * idx[city,now] / idx[city,deal_month] — оживляет замороженные Q1'26 ДКП (см. #724/#727 — upstream-лаг, ДКП отстают на квартал; это лекарство от внутриквартального дрейфа).
5. Бенчмарк-сверка: наша asking-медиана (avito/cian) vs dinamika-tsen-obyavlenii → лог расхождений (data-quality сигнал).

Приоритет

p3, post-demo accuracy lever (НЕ демо-критично). Гранулярность город/регион, не район (ограничение СберИндекса).

Refs: #794 (полный реверс + обзор категории), #652, #724, #727, #647.

Имплементация по результатам research #794 (API полностью реверснут там, anti-bot НЕТ, обычный httpx, данные помесячно до 2026-04). ## Готовый рецепт API (из #794) `POST https://sberindex.ru/api/sowa`, тело `{SOWA:{method:GET, route:<base64>, data:{type:object,value:[]}}}`. route = base64 от `/dataset/v1/<dashboard>?filter=<urlenc-json>&limit=1000&offset=0`. filter = {REF_AREA, FREQ:M, SOURCE:SI, REALTY:2(вторичка)}. Ответ: base64-поля [period, value(руб/м²), ref_area, realty, freq...] + строки помесячно. ## Шаги 1. **REF_AREA коды ЕКБ/МСК** — 1 capture: переключить регион на дашборде → новый filter покажет код (643=РФ; Свердловская обл / Москва — другие). 2. Тянуть 3 серии помесячно: `real_estate_deals` (sold-index), `residential_real_estate_prices` (гедонистич. — коэффициент), `dinamika-tsen-obyavlenii` (asking-бенчмарк). REALTY=2. 3. Таблица `sber_price_index(city, period_month, segment, dashboard, index_value rub_m2, source)` + periodic pull (httpx, прокси НЕ нужен). Idempotent upsert по (city,period_month,dashboard). 4. Wire time-adjust в deal-based путь estimator (#652): `adjusted_deal = deal_price * idx[city,now] / idx[city,deal_month]` — оживляет замороженные Q1'26 ДКП (см. #724/#727 — upstream-лаг, ДКП отстают на квартал; это лекарство от внутриквартального дрейфа). 5. Бенчмарк-сверка: наша asking-медиана (avito/cian) vs `dinamika-tsen-obyavlenii` → лог расхождений (data-quality сигнал). ## Приоритет p3, **post-demo accuracy lever** (НЕ демо-критично). Гранулярность город/регион, не район (ограничение СберИндекса). Refs: #794 (полный реверс + обзор категории), #652, #724, #727, #647.

bot-backend added the

data

priority/p3

scope/backend

status/ready

tradein

labels 2026-05-31 10:38:54 +00:00

bot-backend referenced this issue 2026-05-31 10:39:13 +00:00

[research][tradein][backend] СберИндекс city-index → time-adjustment для замороженных Rosreestr deals (#724/#727) #794

bot-backend self-assigned this 2026-05-31 12:20:11 +00:00

bot-backend added

status/wip

and removed

status/ready

labels 2026-05-31 12:20:13 +00:00

bot-backend referenced this issue from a commit 2026-05-31 12:32:59 +00:00

feat(tradein): СберИндекс monthly city-level price index pull (#887)

bot-backend referenced this issue 2026-05-31 12:42:18 +00:00

feat(tradein): СберИндекс monthly city-level price index pull (#887, data-layer) #890

bot-backend added

status/review

and removed

status/wip

labels 2026-05-31 12:42:39 +00:00

bot-reviewer referenced this issue from a commit 2026-05-31 12:44:59 +00:00

feat(tradein): СберИндекс monthly city-level price index pull (#887, data-layer) (#890)

bot-reviewer added

status/qa

and removed

status/review

labels 2026-05-31 12:45:18 +00:00

bot-backend referenced this issue 2026-05-31 12:58:43 +00:00

feat(tradein): scheduled monthly Rosreestr new-quarter poll → ingest alert (#888) #894

bot-backend referenced this issue 2026-05-31 12:59:03 +00:00

[research][tradein][backend] СберИндекс city-index → time-adjustment для замороженных Rosreestr deals (#724/#727) #794

bot-backend referenced this issue 2026-05-31 13:51:56 +00:00

feat(tradein): add EKB/MSK REF_AREA codes to sber_price_index pull (#794) #900

bot-backend referenced this issue 2026-05-31 13:56:33 +00:00

[tradein][backend] СберИндекс pull: hedonic + asking слаги 404 на /api/sowa — тянется только real_estate_deals #902

lekss361 referenced this issue 2026-05-31 18:38:43 +00:00

feat(tradein): СберИндекс time-adjust for frozen Rosreestr ДКП + per-dashboard filter fix (#794) #919

lekss361 referenced this issue 2026-05-31 18:39:00 +00:00

[research][tradein][backend] СберИндекс city-index → time-adjustment для замороженных Rosreestr deals (#724/#727) #794

bot-reviewer referenced this issue 2026-05-31 18:48:49 +00:00

[tradein][estimator] Backtest gate для СберИндекс time-adjust перед активацией (#794 follow-up) #920

lekss361 referenced this issue 2026-05-31 19:21:58 +00:00

[tradein][backend] sber_index_pull scheduler fails on SSL — sberindex.ru serves an incomplete cert chain (missing Let's Encrypt intermediate) #922

lekss361 referenced this issue 2026-05-31 20:19:06 +00:00

[tradein][backend] Сегментация аналогов/ratio по ppm²-тиру — убрать смешение подрынков (expected_sold недооценка) #928

lekss361 commented 2026-06-06 18:50:30 +00:00

Owner

Copy link

QA PASS (06.06) — verify против прода:

• sber_price_index на проде: 621 строк, 3 города, 3 dashboard, история 2017-01…2026-04; PK (city, period_month, dashboard) = idempotent upsert ✅
• Time-adjust wired в deal-based estimator: estimator.py _load_sber_index_series() (1101-1127), _sber_time_factor() (1130-1152), применяется в _fetch_dkp_corridor() (1216-1241), работает unconditionally (series непустой) ✅

status/qa → status/done, закрываю.

**QA PASS (06.06)** — verify против прода: - `sber_price_index` на проде: 621 строк, 3 города, 3 dashboard, история 2017-01…2026-04; PK `(city, period_month, dashboard)` = idempotent upsert ✅ - Time-adjust wired в deal-based estimator: `estimator.py` `_load_sber_index_series()` (1101-1127), `_sber_time_factor()` (1130-1152), применяется в `_fetch_dkp_corridor()` (1216-1241), работает unconditionally (series непустой) ✅ `status/qa` → `status/done`, закрываю.

lekss361 added

status/done

and removed

status/qa

labels 2026-06-06 18:50:30 +00:00

lekss361 closed this issue 2026-06-06 18:50:30 +00:00

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

fix/2013-avito-cohort-null

fix/rbac-test-user2-expired

fix/expire-brusnika-trial-access

feat/tradein-domclick-house-sku-schema

chore/design-sync-tradein

fix/tradein-deploy-scraper-paths-filter

fix/tradein-no-numberless-house-insert

fix/tradein-dedup-fias-pass-no-geoguard

feat/tradein-fias-estimate-frontend

fix/tradein-dedup-by-fias

feat/tradein-fias-tier-matching

fix/tradein-v2-uiux-p1

fix/objective-lots-dedup-1964

fix/audit-followups-1953

feat/concept-house-picker-1965-stage3b

feat/concept-house-catalog-1965-stage3a

feat/concept-live-massing-1965-stage2b

feat/concept-recompute-1965-stage2a

feat/concept-section-1965-stage1

fix/best-layouts-per-object-snapshot-1956

fix/deficit-segmentation-1959

feat/tradein-hedonic-ground-floor

fix/report-map-points-1961

fix/report-microcopy-1963

fix/report-nspd-zoning-1962

fix/report-frontend-display-1953

perf/tradein-estimate-audit

perf/best-layouts-objects-first

fix/171-migration-path

perf/171-objective-crm-report-month-idx

fix/1926-app-log-stdout

fix/estimator-tiebreaker-nanalogs

fix/1917-fetch-json-network-retry

fix/1926-scrape-runs-total-seen

fix/1781-house-fk-source-identity

feat/39-osrm-per-category

feat/1746-utility-infra-map

fix/test-849-savepoint-count

fix/parking-economics-by-class

fix/zouit-engineering-area-gate

fix/zoning-resolution-decouple

feat/dcf-velocity-schedule

feat/zone-regulation-backfill

chore/mock-financial-estimate

fix/finmodel-split-disclosure-threshold

feat/finmodel-financing

feat/finmodel-sitefinder-bridge

feat/finmodel-dcf

feat/finmodel-price-calibration

feat/finmodel-cost-cascade

audit/1871-p2-estimator-dedup-split

audit/1871-p2-avito-coord

audit/1871-p3-uiux

audit/1871-p2-invest-moi

audit/1871-p2-velocity-market

audit/1871-p1-3-isreal-stubs

audit/1871-p1-2-ghost-anchor-guard

audit/1871-p1-scenarios-collapse

revert/site-finder-v2

fix/v2-remove-resource-legend

fix/v2-map-legend-position

feat/site-finder-v2-map-light-polish

rollback/site-finder-home-light

feat/site-finder-v2-cockpit

rollback/sf-old-analysis-design

feat/ptica-tabs-6-6-reports-params

fix/ptica-vri-row

feat/ptica-map-glyphs-sitefinder-dark

fix/ptica-3d-preview-product-verdict

fix/ptica-button-fontsize

fix/ptica-parity-data-mapping

chore/ptica-iso-preview-spin

fix/ptica-inter-map-product

fix/ptica-visual-parity

fix/ptica-left-rail

feat/ptica-cockpit-complete-bottom

feat/ptica-default-and-rich-map

feat/ptica-real-ksit-frontend

feat/tradein-ekb-geoportal-buildings

fix/sf-feedback-batch

feat/avito-newbuilding-sweep

feat/browser-concurrent-pool

fix/backfill-kadastr-column

fix/avito-proactive-ip-rotate

feat/activate-domclick-sweep

feat/avito-newbuilding-capture

fix/house-id-fk-realtime-link

fix/migration-162-table-if-exists

feat/riasurt-sverdl-layers-108

fix/sber-dataset-paths-902

fix/conflict-resolution-freshness-1539

feat/objective-lots-history-writer-307-obj6

feat/velocity-mv-61

fix/objective-mapping-is-reviewed-gate-307-obj2

fix/cian-detail-rich-fields

fix/avito-novostroyki-segment

feat/yandex-novostroyki-newflat

fix/ci-dedup-runner-storm-1709

fix/emiss-pk-granularity-1687

feat/listings-segment-param-1188

fix/inn-regex-word-boundary-1682

fix/excel-deficit-horizon-1590

fix/admin-scrape-validation

fix/admin-leads-interval-cast-1694

fix/dkp-corridor-doc-p10p90-1695

fix/weather-wind-d-allnone

test/residential-zone-rawprops-1681

fix/domrf-badge-regex-narrow-1686

feat/yandex-camoufox-transport

fix/score-sat-factor-1509

fix/base-pace-room-bucket-1593

fix/gate-verdict-no-blocker-1617

fix/confidence-deal-count-window-1637

fix/match-economics-1511-1512

fix/listings-partial-indexes-1398

fix/smart-suggestions-threshold-1503

fix/dkp-corridor-percentile-1520

fix/enrich-economics-window-1513

fix/leads-stats-window-1383

fix/poi-score-scale-1486

fix/geo-weight-floor-1633

fix/inn-regex-context-1640

fix/domrf-status-classify-1609

fix/emiss-period-dedup-1606

fix/cian-paginate-tail-1393

fix/normalize-zero-fill-guard-1638

fix/supply-view-devgroup-key-1650

fix/residential-zone-detect-1353

fix/score-label-threshold-1357

fix/trend-ratio-vprior-zero-1508

fix/district-median-dedup-1352

fix/supply-layers-overlap-1355

fix/pdf-expires-at-1530

fix/quartirography-area-bucket-1406

fix/cad-zones-multipolygon-1344

fix/sold-pct-market-verdict-1407

fix/week-review-finish

fix/week-review-backend-v2

feat/plan-vectorization-spike-299

fix/1668-analyze-confidence

fix/1667-metro-block

fix/1666-layout-velocity-beat

chore/remove-n1-scraper

fix/cian-kitchen-area

feat/yandex-detail-backfill

feat/avito-detail-backfill

fix/best-layouts-bucket-normalization

fix/gknspecial-incremental-commit-per-prefix

fix/okn-objects-full-registry-load

fix/parcel-ird-overlap-except-expand

fix/report-assembler-confounded-keys

fix/job-settings-json-persist-v2

fix/nspd-geo-soft-time-limit-passthrough

fix/nspd-soft-time-limit-v2

fix/pilot-request-modal-validation

fix/recommend-mix-saturation-latest-snapshot

fix/scrape-cadastre-fail-counter-v2

fix/izyatie-ocr-cad-normalize

fix/job-settings-panel-json-state-persist

fix/noise-loader-savepoint-per-row

fix/objective-sync-config-bare-except

fix/parcel-snapshot-mdash-double-escape

fix/poi-loader-leap-year-date-replace

fix/scrape-cadastre-permanent-fail-counter

fix/usermenu-analyst-role-label

feat/location-948b-r

feat/location-entity-948b

feat/insight-entity-948a

feat/analyst-role-audit-962

feat/cbr-inflation-macro-946

feat/cian-zhk-resolve-972

perf/forecast-parallel-horizons-1129

feat/newbuilding-enrich-backfill-972

feat/emiss-macro-indicators-946

feat/rosstat-emiss-macro-946

ci/frontend-vitest-job

chore/frontend-vitest-wkt-tests

feat/forecast-meta-footer

feat/forecast-future-supply-competitors

feat/forecast-docx-pptx-buttons

feat/forecast-scoring-panel

feat/forecast-product-tz-block

feat/market-map-layers-999

feat/forecast-chart-section6

feat/forecast-export-pptx-959

feat/forecast-export-docx-959

feat/forecast-export-ui-959

feat/forecast-export-telegram-959

feat/forecast-export-md-959

feat/analyze-response-contract-961

feat/competitor-velocity-spatial-fallback-968

fix/frontend-healthcheck-hostname

feat/analyze-market-coords-999

fix/site-finder-entry-crash-1001

feat/horizon-selector-996

feat/forecast-section-998

feat/forecast-endpoint-horizon-enqueue

feat/forecast-celery-task-persist

feat/forecast-orchestrator-service

feat/demand-supply-months-of-inventory

feat/wire-district-resolver

feat/district-vocab-reconciliation

fix/affordability-market-spread

fix/backtest-sig-gate

feat/wire-978-almon-rate-regime

feat/backtest-oos-almon-seasonal

ci/forgejo-pytest-gate

fix/tradein-praktika-logo-form-scroll

feat/yandex-exhaustive-price-room

feat/794-sber-time-adjust

chore/bot-harness-tuning

docs/agent-usage-throttle-qa-terminal

fix/avito-sweep-single-run-guard

chore/analyst-inbound-queue-tradein-db

feat/801-preview-full-wt

chore/bot-safety-followups

feat/tradein-avito-imv-fix

feat/tradein-repair-coef-calibration

feat/tradein-avito-houses-fix

feat/tradein-address-precision

feat/tradein-backfill-prioritize-listings

feat/tradein-geocode-deals-nominatim

feat/tradein-geocode-deals-from-houses

feat/tradein-same-building-match

feat/claude-envvars-and-stale-cron

No results found.

Labels

Clear labels   

admin

Admin UI / internal tools / pipeline tracker

  

analytics

Аналитика данных / matching

  

auth

Authentication / authorization / multi-tenancy

  

automation

Автоматизация / cron / scheduler

  

bug

Something isn't working

  

business

Коммерческая воронка / пилоты / pipeline / реестр ПО

  

chore

Operational housekeeping

  

ci

Forgejo Actions / build pipeline

  

compliance

152-ФЗ / реестр ПО / legal / TOS

  

data

Data sources, parsers, ETL

  

data-moat

Уникальные данные — sales tracker, CRM aggregation

  

docs

README / runbooks / onboarding

  

duplicate

This issue or pull request already exists

  

dx

Developer experience / build speed

  

enhancement

New feature or request

  

Fable 5 ревью

Находки полного мульти-агентного ревью (Claude Fable 5, 2026-06-09)

  

feedback/max

Feedback from Max (Senior partner)

  

generative

Generative Design — concept generation engine

  

GG-форсайт

Новое ТЗ от Ген Геныча (июнь 2026): Site Finder v2 — ИИ-прогноз продуктовой стратегии девелопера. Актуальный приоритет.

  

needs-discussion

Pending team sync / decision

  

needs-human

Human review/decision required

  

observability

Monitoring / metrics / logs / Sentry / Uptime

  

pause-bots

Kill-switch — all autonomous bots pause

  

performance

Latency / throughput / DB query tuning

  

priority/p0

Critical / production-blocker

  

priority/p1

High priority

  

priority/p2

Normal priority

  

priority/p3

Низкий приоритет (follow-up, tech-debt)

  

scope/backend

Backend Python — FastAPI, Celery, scrapers

  

scope/db

DB migrations, indexes, SQL

  

scope/devops

docker-compose, Caddy, workflows

  

scope/frontend

Frontend — Next.js, React, TypeScript

  

scope/qa

Tests, smoke, Playwright

  

scrapers

Скрапперы (Avito/Cian/Yandex/Rosreestr)

  

security

Security audit / secrets / vulnerabilities

  

site-finder

Site Finder scoring quality & data layer

  

stage/1

Stage 1: Generative MVP (нед 2-7)

  

stage/2

Stage 2: Site Finder MVP (нед 8-12)

  

status/blocked

Blocked — needs human intervention

  

status/done

Closed, completed

  

status/needs-analysis

Raw/fuzzy task filed by human — auto-analyst подхватывает: code-archeology + декомпозиция в actionable scope/* + status/ready (или split на под-issues)

  

status/needs-fix

Reviewer/QA вернул worker на доработку — worker сам чинит свой PR (fixup), НЕ требует human (в отличие от status/blocked)

  

status/qa

PR merged, awaiting smoke test

  

status/ready

Ready for pickup by worker bot

  

status/review

PR open, awaiting code review

  

status/wip

Work in progress (worker claimed)

  

tech-debt

Технический долг

  

tradein

Trade-In MVP проект

  

ux

User experience / responsive / conversion

  

week ревью 1

Week review 1 — находки multi-agent аудита проекта (2026-06-14)

  

wontfix

This will not be worked on

  

вторичка

Trade-In Estimator — проект оценки вторичного жилья

  

ИРД

Исходно-разрешительная документация — слой ограничений участка для analyze-таблицы

No labels

admin

analytics

auth

automation

bug

business

chore

ci

compliance

data

data-moat

docs

duplicate

dx

enhancement

Fable 5 ревью

feedback/max

generative

GG-форсайт

needs-discussion

needs-human

observability

pause-bots

performance

priority/p0

priority/p1

priority/p2

priority/p3

scope/backend

scope/db

scope/devops

scope/frontend

scope/qa

scrapers

security

site-finder

stage/1

stage/2

status/blocked

status/done

status/needs-analysis

status/needs-fix

status/qa

status/ready

status/review

status/wip

tech-debt

tradein

ux

week ревью 1

wontfix

вторичка

ИРД

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

bot-backend

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: lekss361/gendesign#887

No description provided.