575c233e51
All checks were successful
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
CI Trade-In / backend-tests (pull_request) Successful in 1m2s
CI Trade-In / changes (pull_request) Successful in 8s
CI / changes (pull_request) Successful in 8s
CI Trade-In / frontend-checks (pull_request) Has been skipped
CI / backend-tests (pull_request) Has been skipped
app.services.scrapers.{cian,cian_detail,yandex_realty,domclick,domclick_exceptions,
avito_houses} имели 0 runtime-импортёров после E1 (scrape_pipeline.py удалён,
#2405) — admin.py/cian_history_backfill.py уже на scraper_kit.providers.*.
Удаление:
- 6 модулей целиком (avito_houses.py, cian.py, cian_detail.py, domclick.py,
domclick_exceptions.py, yandex_realty.py)
- 3 осиротевших dev-sweep скрипта (local-sweep-ekb{,-cian,-yandex}.py) —
эксклюзивно драйвили удалённые легаси-скраперы, kit-путь их заменяет
Тесты (18 файлов DELETE — эксклюзивно тестировали удалённые модули или были
чистым legacy-vs-kit parity, чей легаси-эталон исчез):
test_cian_detail.py, test_cian_serp_scraper.py, tests/scrapers/test_cian_exhaustive.py,
test_yandex_realty_serp.py, tests/scrapers/test_domclick.py,
test_avito_houses_dedup.py, test_avito_houses_parse.py,
test_houses_rating_filter_entries_1789.py, test_scraper_kit_domclick_golden_parity.py,
tests/scrapers/test_admin_{avito,cian,yandex}_kit_parity.py.
CONVERT (файл покрывает СМЕШАННЫЙ live+legacy набор — вырезаны только
секции удалённых модулей, live-покрытие сохранено):
- test_scraper_kit_cian_golden_parity.py — SERP/`_format_address`/detail
(cian.py/cian_detail.py) убраны; extract_state (cian_state_parser, live)/
valuation (cian_valuation, live)/newbuilding (cian_newbuilding, live)/
no-app-imports guard — остались
- test_scraper_kit_yandex_golden_parity.py — SERP `_entity_to_lot`/
`_parse_gate_json`/`_build_url` (yandex_realty) убраны; normalize_house_type
(live)/newbuilding parse (yandex_newbuilding, live)/no-app-imports guard —
остались
- test_scraper_kit_bisection_parity.py — cian/domclick/yandex-ветки убраны;
avito-ветка (live, единственная оставшаяся легаси-бисекция) — осталась
полностью, включая full-seed-grid
- test_scraper_kit_group_c_backfill_kit_parity.py — cian fetch_detail
browser_fetcher parity-тест убран; avito/house_imv_backfill/
cian_history_backfill/ekb_geoportal тесты — остались
- test_scraper_kit_pricehistory_session_parity.py — fetch_detail
config-injection/proxy-wiring тесты (cian_detail) убраны; extract_state
(cian_state_parser, live)/ScrapedLot (base, live) parity — остались
- test_yandex_scrapers_delay_wiring.py — yandex_realty delay-тест убран
(kit-эквивалент уже покрыт test_kit_serp_proxy_pool.py); yandex_newbuilding
(live)/yandex_detail/yandex_valuation (kit) тесты — остались
- test_scraper_proxy.py — yandex_realty BrowserFetcher-тест убран (redundant
с test_kit_serp_proxy_pool.py::test_yandex_serp_no_pool_by_default)
RETARGET на kit (символ живёт, только источник поменялся):
- test_domclick_sweep.py — `_map_item` → scraper_kit.providers.domclick.serp
- test_yandex_city_sweep.py — DEFAULT_PRICE_RANGES/ROOM_PATH →
scraper_kit.providers.yandex.serp
- test_pipeline_browser_routing.py — fetch_house_catalog →
scraper_kit.providers.avito.houses (+ KitAvitoBlockedError для firewall-теста)
- test_snapshot_writer.py — DetailEnrichment/save_detail_enrichment →
scraper_kit.providers.cian.detail (byte-identical strangler-копия)
- test_backfill_wave2.py — DetailEnrichment → scraper_kit.providers.cian.detail
- tests/scrapers/test_avito_unix_date_tz_consistency.py — avito_houses →
scraper_kit.providers.avito.houses (houses-ветка _unix_to_date/RUS_MONTHS
consolidation-проверки)
Stale-комментарии поправлены в cian_price_history.py, tests/scrapers/
test_domclick_detail.py (описывали ещё-живой domclick.py/domclick_exceptions.py).
grep -rn scrapers\.\(cian\|cian_detail\|yandex_realty\|domclick\|domclick_exceptions\|
avito_houses\) app/ scripts/ packages/ — только historical footnotes
("strangler-копия app.services.scrapers.X", past-tense "удалён/deleted").
Backend suite: 2811 passed, 6 skipped, 1 failed (known flake test_search_cache_hit
#2208, не регрессия). ruff check чист на всех изменённых файлах.
|
||
|---|---|---|
| .. | ||
| local-cian | ||
| avito-playwright-sweep.py | ||
| backfill-house-imv.py | ||
| backfill-yandex-addresses-ekb.py | ||
| backfill_001_address_fingerprints.py | ||
| backfill_002_house_sources.sql | ||
| backfill_003_listing_sources.sql | ||
| backfill_cian_zhk_url.py | ||
| browser-smoke.sh | ||
| cian_backfill_all.sh | ||
| inspect-houses-html.py | ||
| inspect-search-html.py | ||
| probe-avito-1anchor.py | ||
| probe-confidence-distribution.py | ||
| probe-db-counts.py | ||
| probe-db-leak-full.py | ||
| probe-db-region-leak.py | ||
| probe-db-url-leak.py | ||
| probe-imv-by-address.py | ||
| probe-migration-028.py | ||
| probe-migration-status.py | ||
| probe-parser-region-filter.py | ||
| README.md | ||
| sweep-yandex-valuation-ekb.py | ||
| test-playwright.py | ||
| trigger-schedules.py | ||
| upload-cian-cookies.sh | ||
tradein-mvp scripts -- backfill
One-shot manual operator scripts. NOT auto-applied by deploy.yml (which only runs tradein-mvp/backend/data/sql/NN_*.sql).
Order
Run in numbered sequence after PR #470 (matching module) merged:
cd tradein-mvp/backend
export DATABASE_URL="postgresql+psycopg://user:pass@host:5432/tradein"
# 001 -- backfill houses.address_fingerprint (Python, idempotent)
uv run python ../scripts/backfill_001_address_fingerprints.py
# 002 -- backfill house_sources link rows (SQL, idempotent)
psql "${DATABASE_URL/postgresql+psycopg/postgresql}" -f ../scripts/backfill_002_house_sources.sql
# 003 -- backfill listing_sources link rows (SQL, idempotent)
psql "${DATABASE_URL/postgresql+psycopg/postgresql}" -f ../scripts/backfill_003_listing_sources.sql
All scripts idempotent -- re-running is safe (ON CONFLICT DO NOTHING on the SQL; row-level UPDATE WHERE NULL on the Python).
Other scripts
upload-cian-cookies.sh-- one-shot Cian session cookie upload (see inline docs)