fd4a0d643d
fix(tradein): review minors — scheduler comment + save-error log label
...
CI / changes (push) Successful in 7s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
- scheduler.py: исправлен stale-комментарий yandex_detail_backfill
(был: «YandexDetailScraper httpx, no proxy layer»;
теперь: fetch via curl_cffi chrome120 + scraper_proxy_url,
parse via YandexDetailScraper.parse)
- yandex_detail_backfill.py: outer except — сообщение лога изменено на
«save/iteration error for listing_id=%d» чтобы отличать save-ошибку
от fetch/parse-None путей (run_id убран — он менее важен в этом scope)
2026-06-16 14:14:07 +03:00
e2e0291a0a
fix(tradein): yandex detail backfill — заменить fetch_detail на curl_cffi+proxy
...
CI / changes (push) Successful in 8s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
YandexDetailScraper.fetch_detail использует plain httpx (BaseScraper._http_get)
без прокси и без TLS-fingerprinting → на датацентровом IP Yandex отдаёт капчу /
shell-HTML → parse всегда None → backfill 0% эффективен.
Замена: curl_cffi AsyncSession(chrome120 + scraper_proxy_url), идентично
yandex_address_backfill.py (доказано работающий путь на проде). HTTP 200 →
YandexDetailScraper.parse(html) (чистая функция, без сети). parse→None
считается fail (captcha wall); consecutive None → abort после max_consecutive_blocks.
Тесты обновлены: мокается AsyncSession.get + YandexDetailScraper.parse
(не fetch_detail). Добавлены кейсы: non-200 → abort, proxy=None → proxies=None.
2026-06-16 14:05:22 +03:00
03e43f12b5
feat(tradein): yandex detail backfill — save_detail_enrichment + nightly task ( #1553 )
...
CI / changes (push) Successful in 7s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
- yandex_detail.py: добавить save_detail_enrichment(db, listing_id, enrichment) → bool;
UPDATE listings COALESCE по 17 полям (rooms/area_m2/floor/total_floors/address/
description/repair_state/publish_date/views_total_yandex/publish_date_relative/
agency_name/agency_founded_year/agency_objects_count/metro_stations/photo_urls/
newbuilding_url/newbuilding_id); metro/photos → jsonb; detail_enriched_at = NOW()
- app/tasks/yandex_detail_backfill.py: новый task run_yandex_detail_backfill;
snapshot SELECT source='yandex' AND detail_enriched_at IS NULL; YandexDetailScraper
async context manager; consecutive None abort (max_consecutive_blocks=5); budget guard;
run lifecycle update_heartbeat/mark_done/mark_failed
- scheduler.py: trigger_yandex_detail_backfill_run + elif source=='yandex_detail_backfill'
в scheduler_loop; window 12-15 UTC (после avito_detail 09-12 UTC)
- migration 113: ADD COLUMN IF NOT EXISTS photo_urls jsonb, newbuilding_url text,
newbuilding_id text; seed scrape_schedules yandex_detail_backfill enabled=true
- tests: 10 тестов (8 backfill + 2 save_detail_enrichment); 1819 passed 0 failed
Closes #1553
2026-06-16 13:58:20 +03:00
