lekss361/gendesign
1
0
Fork 0
Code Issues 57 Pull requests Projects Releases Packages Wiki Activity Actions
gendesign/tradein-mvp/backend/app/services/scrapers
History
bot-backend 03e43f12b5
All checks were successful
CI / changes (push) Successful in 7s
Details
CI / backend-tests (push) Has been skipped
Details
CI / frontend-tests (push) Has been skipped
Details
CI / openapi-codegen-check (push) Has been skipped
Details
feat(tradein): yandex detail backfill — save_detail_enrichment + nightly task (#1553) 
- yandex_detail.py: добавить save_detail_enrichment(db, listing_id, enrichment) → bool;
  UPDATE listings COALESCE по 17 полям (rooms/area_m2/floor/total_floors/address/
  description/repair_state/publish_date/views_total_yandex/publish_date_relative/
  agency_name/agency_founded_year/agency_objects_count/metro_stations/photo_urls/
  newbuilding_url/newbuilding_id); metro/photos → jsonb; detail_enriched_at = NOW()
- app/tasks/yandex_detail_backfill.py: новый task run_yandex_detail_backfill;
  snapshot SELECT source='yandex' AND detail_enriched_at IS NULL; YandexDetailScraper
  async context manager; consecutive None abort (max_consecutive_blocks=5); budget guard;
  run lifecycle update_heartbeat/mark_done/mark_failed
- scheduler.py: trigger_yandex_detail_backfill_run + elif source=='yandex_detail_backfill'
  в scheduler_loop; window 12-15 UTC (после avito_detail 09-12 UTC)
- migration 113: ADD COLUMN IF NOT EXISTS photo_urls jsonb, newbuilding_url text,
  newbuilding_id text; seed scrape_schedules yandex_detail_backfill enabled=true
- tests: 10 тестов (8 backfill + 2 save_detail_enrichment); 1819 passed 0 failed

Closes #1553
2026-06-16 13:58:20 +03:00
..
__init__.py feat: add tradein-mvp subproject (Trade-In Estimator под /trade-in) 2026-05-21 00:25:39 +03:00
avito.py fix(week-review): автофиксы код-ревью — 169 issue (label «week ревью 1») 2026-06-15 20:21:11 +05:00
avito_detail.py fix(tradein): avito detail parser — добавить living_area_m2/ceiling_height_m/finishing 2026-06-16 10:19:09 +03:00
avito_exceptions.py fix(tradein): avito content-block (HTTP-200) не маскируется под end-of-pagination (Refs #754) (#779) 2026-05-30 16:28:23 +00:00
avito_houses.py fix(tradein): M1 review — short_address для match_or_create_house (alias keyspace) 2026-06-16 08:35:38 +03:00
avito_imv.py fix(tradein): route yandex/valuation/imv/newbuilding scrapers через scraper_proxy_url (#860) 2026-05-31 07:04:31 +00:00
base.py refactor(tradein): unify kadastr_num -> cadastral_number (#732) 2026-05-31 15:41:52 +03:00
browser_fetcher.py feat(tradein): Cian browser auto-login — POST /admin/scrape/cian/auto-login (#639) (#917) 2026-05-31 18:24:51 +00:00
cian.py fix(week-review): автофиксы код-ревью — 169 issue (label «week ревью 1») 2026-06-15 20:21:11 +05:00
cian_detail.py feat(tradein): dedicated Cian mobile proxy — separate egress from Avito (#923) 2026-05-31 19:30:14 +00:00
cian_newbuilding.py fix(tradein): cian newbuilding MFE/reliability drift — initialState extraction (#972) 2026-06-15 21:12:05 +03:00
cian_state_parser.py fix(tradein): cian _cianConfig concat-формат + auth из header-frontend (#639) (#819) 2026-05-30 19:08:01 +00:00
cian_valuation.py feat(tradein): dedicated Cian mobile proxy — separate egress from Avito (#923) 2026-05-31 19:30:14 +00:00
domclick.py fix(tradein): domklik source name, scraper delay, DataDome block detect (#796 fixup) 2026-05-31 17:21:30 +03:00
domclick_exceptions.py fix(tradein): domklik source name, scraper delay, DataDome block detect (#796 fixup) 2026-05-31 17:21:30 +03:00
n1.py feat(scrapers/n1): извлекать floor/total_floors из ведущего N/M-токена (Refs #795) (#802) 2026-05-30 17:31:13 +00:00
repair_state_normalizer.py feat(tradein): infer repair_state from listing description text (#622) (#632) 2026-05-28 15:37:34 +00:00
snapshot_writer.py fix(tradein): implement listings_snapshots + offer_price_history writers (#567) (#624) 2026-05-28 13:49:52 +00:00
yandex_detail.py feat(tradein): yandex detail backfill — save_detail_enrichment + nightly task (#1553) 2026-06-16 13:58:20 +03:00
yandex_helpers.py fix(tradein): Yandex listing_date — keyword + absolute-no-year (#602) 2026-05-27 13:11:22 +00:00
yandex_newbuilding.py fix(tradein): yandex newbuilding rating drift — "4.3из 5" без пробела (#974) 2026-06-15 21:29:43 +03:00
yandex_realty.py fix(tradein): system-curl subprocess transport for Yandex SERP over SOCKS5 (#940) 2026-05-31 22:39:06 +00:00
yandex_valuation.py fix(tradein): route yandex/valuation/imv/newbuilding scrapers через scraper_proxy_url (#860) 2026-05-31 07:04:31 +00:00