feat(avito): dedicated newbuilding (novostroyka) citywide sweep #1747

Merged

lekss361 merged 1 commit from feat/avito-newbuilding-sweep into main

2026-06-18 07:00:23 +00:00

lekss361 commented

2026-06-18 06:47:22 +00:00

Owner

What & why

Avito new-builds are currently only scraped incidentally from the general search (≈20 rows in listings). Avito has a dedicated novostroyka-filtered SERP that returns 100% new-build cards. Verified live through the prod transport (camoufox+apw via the per-source pool): fetching …/ekaterinburg/kvartiry/prodam/novostroyka-ASgBAgICAkSSA8YQ5geOUg?s=104&p=1 and parsing with the existing _parse_html returned total=23 novostroyki=23 with newbuilding_id populated (e.g. meridian-ekaterinburg). Paginating yields hundreds.

Change

Mirrors the existing citywide-sweep machinery:

avito.py: NOVOSTROYKA_SLUG constant (EKB novostroyka filter), _build_newbuilding_url(page) (same param shape as _build_citywide_url: s=104, p), fetch_newbuildings(pages=30). The shared paginated loop is factored into a private _paginate_sweep(pages, url_builder, *, label, delay_override_sec) that both fetch_city_wide and fetch_newbuildings delegate to. fetch_city_wide behavior is preserved — identical control flow (exception handling, dedup, break-on-empty, sleep gating) and the same avito fetch_city_wide pages=… total_lots=… summary log; per-page log renders identically for citywide. The parser already classifies novostroyki by the developer marker and fills newbuilding_id/newbuilding_url, so no parser change.
scrape_pipeline.py: run_avito_newbuilding_sweep(...) — a slim citywide sweep (fetch → save_listings → scrape_runs counters/mark_done/mark_failed/mark_banned), NOT anchor/houses/detail/IMV phases (novostroyka URL is citywide). Both new symbols exported.
scheduler.py: trigger_avito_newbuilding_sweep_run + dispatch elif source == "avito_newbuilding_sweep" + import + header doc.
data/sql/123_avito_newbuilding_sweep_schedule.sql: idempotent seed (BEGIN/COMMIT, ON CONFLICT (source) DO NOTHING), enabled=true, window 2–5 UTC, default_params {"pages":20,"request_delay_sec":8}. next_run_at = tomorrow 02:00 UTC so it does NOT fire at deploy time (an enabled row with NULL next_run_at is treated as immediately due, and _claim_run doesn't gate on the window) — follows the existing cian/106 seed convention.

Tests

test_avito_newbuilding_sweep.py (13 tests) + fixtures/avito_serp_novostroyka.html: URL builder (novostroyka-, p=2), novostroyka fixture → listing_segment=="novostroyki" + newbuilding_id set, break-on-empty/dedup, counters, pipeline wiring smoke.
Full gate: pytest -q (2 standard deselects) → 1922 passed, 2 deselected. ruff clean, pre-commit green.

Post-merge

Migration auto-applies (_schema_migrations, strict); schedule goes live enabled=true, first sweep next 02:00 UTC. I'll trigger a run manually post-deploy for immediate fill (current 20 → hundreds), then verify the rows land as listing_segment='novostroyki' with newbuilding_id.

Fragility note

NOVOSTROYKA_SLUG is hardcoded (same class as the existing citywide/room slugs) — if Avito rotates the EKB novostroyka filter slug it needs updating.

## What & why Avito new-builds are currently only scraped **incidentally** from the general search (≈20 rows in `listings`). Avito has a dedicated novostroyka-filtered SERP that returns 100% new-build cards. Verified live through the prod transport (camoufox+apw via the per-source pool): fetching `…/ekaterinburg/kvartiry/prodam/novostroyka-ASgBAgICAkSSA8YQ5geOUg?s=104&p=1` and parsing with the existing `_parse_html` returned **`total=23 novostroyki=23`** with `newbuilding_id` populated (e.g. `meridian-ekaterinburg`). Paginating yields hundreds. ## Change Mirrors the existing citywide-sweep machinery: - **`avito.py`**: `NOVOSTROYKA_SLUG` constant (EKB novostroyka filter), `_build_newbuilding_url(page)` (same param shape as `_build_citywide_url`: s=104, p), `fetch_newbuildings(pages=30)`. The shared paginated loop is factored into a private `_paginate_sweep(pages, url_builder, *, label, delay_override_sec)` that both `fetch_city_wide` and `fetch_newbuildings` delegate to. **`fetch_city_wide` behavior is preserved** — identical control flow (exception handling, dedup, break-on-empty, sleep gating) and the same `avito fetch_city_wide pages=… total_lots=…` summary log; per-page log renders identically for citywide. The parser already classifies novostroyki by the developer marker and fills `newbuilding_id`/`newbuilding_url`, so **no parser change**. - **`scrape_pipeline.py`**: `run_avito_newbuilding_sweep(...)` — a slim **citywide** sweep (fetch → save_listings → scrape_runs counters/mark_done/mark_failed/mark_banned), NOT anchor/houses/detail/IMV phases (novostroyka URL is citywide). Both new symbols exported. - **`scheduler.py`**: `trigger_avito_newbuilding_sweep_run` + dispatch `elif source == "avito_newbuilding_sweep"` + import + header doc. - **`data/sql/123_avito_newbuilding_sweep_schedule.sql`**: idempotent seed (BEGIN/COMMIT, `ON CONFLICT (source) DO NOTHING`), `enabled=true`, window 2–5 UTC, `default_params {"pages":20,"request_delay_sec":8}`. **`next_run_at` = tomorrow 02:00 UTC** so it does NOT fire at deploy time (an enabled row with NULL next_run_at is treated as immediately due, and `_claim_run` doesn't gate on the window) — follows the existing cian/106 seed convention. ## Tests - `test_avito_newbuilding_sweep.py` (13 tests) + `fixtures/avito_serp_novostroyka.html`: URL builder (`novostroyka-`, `p=2`), novostroyka fixture → `listing_segment=="novostroyki"` + `newbuilding_id` set, break-on-empty/dedup, counters, pipeline wiring smoke. - Full gate: `pytest -q` (2 standard deselects) → **1922 passed, 2 deselected**. ruff clean, pre-commit green. ## Post-merge Migration auto-applies (`_schema_migrations`, strict); schedule goes live `enabled=true`, first sweep next 02:00 UTC. I'll **trigger a run manually** post-deploy for immediate fill (current 20 → hundreds), then verify the rows land as `listing_segment='novostroyki'` with `newbuilding_id`. ## Fragility note `NOVOSTROYKA_SLUG` is hardcoded (same class as the existing citywide/room slugs) — if Avito rotates the EKB novostroyka filter slug it needs updating.

lekss361 added 1 commit 2026-06-18 06:47:23 +00:00