lekss361/gendesign

Fork 0

feat(tradein): yandex_newbuilding sweep-entrypoint + market-схема enrichment, seed dormant (#974) #1541

Merged

lekss361 merged 2 commits from feat/yandex-newbuilding-sweep into main

2026-06-15 15:44:28 +00:00

bot-backend commented

2026-06-15 15:29:42 +00:00

Collaborator

Summary

SQL 105: CREATE SCHEMA market + market.yandex_jk_enrichment (ext_id UNIQUE), houses.yandex_jk_slug, GRANT gendesign_reader
SQL 106: seed scrape_schedules yandex_newbuilding_sweep (enabled=false dormant, 02-05 UTC)
yandex_newbuilding.py: fetch_jk via BrowserFetcher + resolve_yandex_jk_slug (SERP siteId)
tasks/yandex_newbuilding_sweep.py: enrich_yandex_newbuilding_sweep (resolve slug -> fetch_jk -> UPSERT market.yandex_jk_enrichment, SAVEPOINT per house, dry_run, resumable)
admin.py: POST /scrape/yandex-newbuilding-sweep + GET .../runs
scheduler.py: trigger_yandex_newbuilding_sweep_run + dispatch elif

ADR

Новые скрейперы пишут в схему market в существующем tradein-postgres. market-слой читается gendesign_reader.

Test plan

8 task tests + 3 scheduler dispatch tests (20 pass)
ruff check + ruff format
Активация: UPDATE scrape_schedules SET enabled = true WHERE source = 'yandex_newbuilding_sweep'

Refs #974

## Summary - SQL 105: CREATE SCHEMA market + market.yandex_jk_enrichment (ext_id UNIQUE), houses.yandex_jk_slug, GRANT gendesign_reader - SQL 106: seed scrape_schedules yandex_newbuilding_sweep (enabled=false dormant, 02-05 UTC) - yandex_newbuilding.py: fetch_jk via BrowserFetcher + resolve_yandex_jk_slug (SERP siteId) - tasks/yandex_newbuilding_sweep.py: enrich_yandex_newbuilding_sweep (resolve slug -> fetch_jk -> UPSERT market.yandex_jk_enrichment, SAVEPOINT per house, dry_run, resumable) - admin.py: POST /scrape/yandex-newbuilding-sweep + GET .../runs - scheduler.py: trigger_yandex_newbuilding_sweep_run + dispatch elif ## ADR Новые скрейперы пишут в схему market в существующем tradein-postgres. market-слой читается gendesign_reader. ## Test plan - [x] 8 task tests + 3 scheduler dispatch tests (20 pass) - [x] ruff check + ruff format - [ ] Активация: UPDATE scrape_schedules SET enabled = true WHERE source = 'yandex_newbuilding_sweep' Refs #974

bot-backend added 1 commit 2026-06-15 15:29:43 +00:00

feat(tradein): yandex_newbuilding sweep-entrypoint + market-схема enrichment, seed dormant (#974 )

CI / backend-tests (push) Has been skipped

Details

CI / frontend-tests (push) Has been skipped

Details

CI / openapi-codegen-check (push) Has been skipped

Details

CI / changes (pull_request) Successful in 5s

Details

CI / backend-tests (pull_request) Has been skipped

Details

CI / frontend-tests (pull_request) Has been skipped

Details

CI / openapi-codegen-check (pull_request) Has been skipped

Details

CI / changes (push) Successful in 7s

Details

feab585ad5

- SQL 105: CREATE SCHEMA market + market.yandex_jk_enrichment (ext_id UNIQUE, upsert-ready),
  houses.yandex_jk_slug, GRANT gendesign_reader на market-слой
- SQL 106: seed scrape_schedules yandex_newbuilding_sweep (enabled=false dormant, окно 02-05 UTC)
- yandex_newbuilding.py: fetch_jk → BrowserFetcher (camoufox), add resolve_yandex_jk_slug
  (SERP siteId query через BrowserFetcher, slug из _JK_SLUG_RE href)
- tasks/yandex_newbuilding_sweep.py: enrich_yandex_newbuilding_sweep (resolve slug →
  fetch_jk → UPSERT market.yandex_jk_enrichment, SAVEPOINT per house, dry_run, resumable)
- admin.py: POST /scrape/yandex-newbuilding-sweep + GET .../runs
- scheduler.py: trigger_yandex_newbuilding_sweep_run + dispatch elif branch
- tests: 8 task tests + 3 scheduler dispatch tests (20 pass)

bot-backend added 1 commit 2026-06-15 15:41:18 +00:00

fix(tradein): yandex sweep — guard ext_id=NULL перед fetch_jk + cleanup httpx (#974 review M1/M3)

CI / backend-tests (push) Has been skipped

Details

CI / frontend-tests (push) Has been skipped

Details

CI / openapi-codegen-check (push) Has been skipped

Details

CI / backend-tests (pull_request) Has been skipped

Details

CI / changes (push) Successful in 7s

Details

CI / changes (pull_request) Successful in 8s

Details

CI / frontend-tests (pull_request) Has been skipped

Details

CI / openapi-codegen-check (pull_request) Has been skipped

Details

c9daf8e268

M1: добавлен guard перед вызовом fetch_jk — если ext_id пустой/None при наличии
jk_slug, логируем warning, incrementим failed_fetch и пропускаем (continue).
Пустой jk_id формирует битый URL /{slug}-/ и коллизию на UNIQUE ext_id=''.

M3: в admin.py /scrape/yandex-newbuilding заменён `async with YandexNewbuildingScraper()`
на прямое `scraper = YandexNewbuildingScraper()` — fetch_jk использует внутренний
BrowserFetcher, httpx-клиент BaseScraper открывался вхолостую.

Тесты: добавлен test_sweep_slug_present_but_ext_id_null_skips_fetch.