lekss361/gendesign

Fork 0

feat(tradein): yandex_newbuilding.py — ЖК landing parser (149 selectors + NLP) #467

Merged

lekss361 merged 1 commit from feat/tradein-yandex-newbuilding-parser into main

2026-05-23 13:46:46 +00:00

lekss361 commented

2026-05-23 13:41:31 +00:00

Owner

Motivation

Stage 5 of 9 for YandexRealtyScraper v1 (Wave 4 / Worker B — parallel with yandex_detail.py and yandex_valuation.py).

ЖК landing pages are the richest Yandex data source — comparable to or richer than Avito Houses Catalog. Reference target: ЖК Татлин (slug=tatlin, id=1592987) — class comfort+, June 2023, PRINZIP developer, rating 4.3, 1505 ratings, 353 text reviews, coords (56.855312, 60.576668).

Module

tradein-mvp/backend/app/services/scrapers/yandex_newbuilding.py — YandexNewbuildingScraper(BaseScraper) with async fetch_jk(slug, id, city="ekaterinburg").

URL pattern: /{city}/kupit/novostrojka/<slug>-<id>/

YandexNewbuildingInfo fields

Group	Fields
Identity	`ext_id`, `ext_slug`, `source_url`, `name`, `address`
Coords (inline — Yandex unique vs Avito/Cian)	`lat`, `lon` (city-range filtered)
Classification	`house_class` (`comfort_plus` etc.), `commission_year`, `commission_month` (RU)
Footprint	`total_floors`, `corpus_count` (word→int "три"→3), `total_area_ha`
Developer	`developer_name`, `developer_url`, `developer_other_jk[]`
Reviews	`rating`, `ratings_count`, `text_reviews_count` (the 353!)
Content	`description` (full text), `metro_stations[]`, `house_type` (NLP fallback)

Coord extraction

Inline scan of full HTML for any \d{2}\.\d{4,8} matching plausible city range:

LAT_RANGE: (55.5, 57.5) for ЕКБ
LON_RANGE: (59.5, 61.5) for ЕКБ
Per-city ranges extendable later

NLP regex constants

RE_FLOORS_TOWERS — "35-этажные башни"
RE_AREA_HA — "участке 1,5 га"
RE_COMMISSION — "введён в эксплуатацию в июне 2023"
RE_CORPUS_COUNT_WORD — handles word ("три" → 3) and digit forms
RE_RATING / RE_RATINGS_COUNT / RE_TEXT_REVIEWS
RE_METRO_INLINE — "Уральская 11 мин"

Plus delegation to parse_house_class / parse_house_type from yandex_helpers.

Tests

tests/test_yandex_newbuilding.py: 28 passed, 0 failed (0.86s)
ruff check: All checks passed (line ≤ 100)

Files

tradein-mvp/backend/app/services/scrapers/yandex_newbuilding.py (new, 352 lines)
tradein-mvp/backend/tests/test_yandex_newbuilding.py (new, 422 lines)

Reviewer note

Reviewer: deep-code-reviewer — only merge after APPROVE. Wave 4 sibling PRs: yandex_detail.py + yandex_valuation.py.

## Motivation Stage 5 of 9 for YandexRealtyScraper v1 (Wave 4 / Worker B — parallel with `yandex_detail.py` and `yandex_valuation.py`). ЖК landing pages are the richest Yandex data source — comparable to or richer than Avito Houses Catalog. Reference target: ЖК Татлин (slug=`tatlin`, id=`1592987`) — class comfort+, June 2023, PRINZIP developer, rating 4.3, 1505 ratings, 353 text reviews, coords (56.855312, 60.576668). ## Module `tradein-mvp/backend/app/services/scrapers/yandex_newbuilding.py` — `YandexNewbuildingScraper(BaseScraper)` with async `fetch_jk(slug, id, city="ekaterinburg")`. URL pattern: `/{city}/kupit/novostrojka/<slug>-<id>/` ## YandexNewbuildingInfo fields | Group | Fields | |---|---| | Identity | `ext_id`, `ext_slug`, `source_url`, `name`, `address` | | Coords (inline — Yandex unique vs Avito/Cian) | `lat`, `lon` (city-range filtered) | | Classification | `house_class` (`comfort_plus` etc.), `commission_year`, `commission_month` (RU) | | Footprint | `total_floors`, `corpus_count` (word→int "три"→3), `total_area_ha` | | Developer | `developer_name`, `developer_url`, `developer_other_jk[]` | | Reviews | `rating`, `ratings_count`, `text_reviews_count` (the 353!) | | Content | `description` (full text), `metro_stations[]`, `house_type` (NLP fallback) | ## Coord extraction Inline scan of full HTML for any `\d{2}\.\d{4,8}` matching plausible city range: - LAT_RANGE: (55.5, 57.5) for ЕКБ - LON_RANGE: (59.5, 61.5) for ЕКБ - Per-city ranges extendable later ## NLP regex constants - `RE_FLOORS_TOWERS` — "35-этажные башни" - `RE_AREA_HA` — "участке 1,5 га" - `RE_COMMISSION` — "введён в эксплуатацию в июне 2023" - `RE_CORPUS_COUNT_WORD` — handles word ("три" → 3) and digit forms - `RE_RATING` / `RE_RATINGS_COUNT` / `RE_TEXT_REVIEWS` - `RE_METRO_INLINE` — "Уральская 11 мин" Plus delegation to `parse_house_class` / `parse_house_type` from `yandex_helpers`. ## Tests ``` tests/test_yandex_newbuilding.py: 28 passed, 0 failed (0.86s) ruff check: All checks passed (line ≤ 100) ``` ## Files - `tradein-mvp/backend/app/services/scrapers/yandex_newbuilding.py` (new, 352 lines) - `tradein-mvp/backend/tests/test_yandex_newbuilding.py` (new, 422 lines) ## Reviewer note Reviewer: **deep-code-reviewer** — only merge after APPROVE. Wave 4 sibling PRs: yandex_detail.py + yandex_valuation.py.

lekss361 added 1 commit 2026-05-23 13:41:32 +00:00

feat(tradein): yandex_newbuilding.py — ЖК landing parser (149 selectors + NLP) 5f92e94695

Stage 5 of YandexRealtyScraper v1 (Wave 4 / Worker B — parallel with
yandex_detail.py and yandex_valuation.py).

Module: app/services/scrapers/yandex_newbuilding.py
- YandexNewbuildingScraper(BaseScraper) — fetch_jk(slug, id, city)
  URL: /{city}/kupit/novostrojka/<slug>-<id>/
- YandexNewbuildingInfo Pydantic model (ext_id/slug, name, address, lat/lon,
  house_class, commission_year/month, total_floors, corpus_count, total_area_ha,
  developer_name/url + other_jk, rating + ratings_count + text_reviews_count,
  description, metro_stations, house_type)
- Inline coord extraction (Yandex unique: lat/lon visible in raw HTML within
  city range filter)
- NLP regex: floors-towers, area-ha, commission date, corpus count (word→int),
  rating + counts, metro inline
- Helpers: _find_section_text (h2/h3 → next siblings), _parse_corpus_count
  (word/digit), _parse_metro, _extract_jk_address

Tests: 28 unit tests against Татлин-like fixture HTML. All pass. Ruff clean.

lekss361 referenced this pull request

2026-05-23 13:45:36 +00:00

feat(tradein): yandex_detail.py — Product JSON-LD + DOM detail parser #466

lekss361 merged commit 51650276b2 into main

2026-05-23 13:46:46 +00:00

lekss361 referenced this pull request from a commit

2026-05-23 13:46:47 +00:00

feat(tradein): yandex_newbuilding.py — ЖК landing parser (149 selectors + NLP) (#467)

lekss361 commented

2026-05-23 13:47:34 +00:00

Author

Owner

Deep Code Review — verdict APPROVE (merged)

Status: APPROVE
Files reviewed: 2 (P1: scraper, P3: tests)
LOC: +774/-0
Merge commit: 51650276b2a8613760118f633bef193df678ebf8

Cross-file impact

parse_house_class, parse_house_type, RE_JK_ID — confirmed exports in yandex_helpers.py (re-uses Stage 3 helpers PR #456 pattern). OK.
BaseScraper._http_get / sleep_between_requests / abstract fetch_around — used identically to existing yandex_realty.py:64 (caller responsible for async with context).
No conflict with Wave 4 siblings (PR #466 yandex_detail, PR #468 yandex_valuation — disjoint files).

Critical focus areas — all satisfied

Area	Verdict
Coord extraction safety	OK for ЕКБ pilot (narrow 2-degree window + ≥4 decimals)
NLP regex robustness	11 regexes all anchored on Russian noun/verb context — low FP risk
353 reviews extraction	`test_parse_rating_and_counts` asserts `info.text_reviews_count == 353` on Татлин fixture ✅
28 tests	Confirmed count (happy-path + coord×4 + corpus×4 + house_class×2 + commission×2 + rating×2 + dev×2 + metro×3 + minimal + fetch_around raises + section×2 + dedup/cap×2 + model×2)
Ruff line ≤100	Both files pass
`developer_other_jk[]` capped at 10 + dedup	Verified by `test_developer_other_jk_capped_at_10` + `test_developer_other_jk_dedup`
`corpus_count` word→int	`_WORD_NUM` covers одна/одну/одной/две/три/четыре/пять/шесть/семь/восемь + digit branch
`commission_month` RU	Stored as inflected form ("июне") — matches plan + schema `text` column

Minor improvements (non-blocking, future PR)

Coord pair-locality — _extract_coords independently picks first lat-in-range and first lon-in-range. If page has a stray decimal 56.xxxx BEFORE the real coords block, parser could return mismatched lat without paired lon (or vice-versa: Moscow lat 55.7558 falls in EKB LAT_RANGE — test_extract_coords_outside_range_returns_none only asserts lon is None, lat in that case = 55.7558 ≠ EKB). Recommend post-merge: require lat and lon within ~50 chars of each other in HTML (similar to fixture "56.855312, 60.576668").
Multi-city LAT/LON_RANGE — currently EKB-only constants. Per-city dict if Stage 7+ scales to other regions.
Real Yandex HTML snapshot fixture — current TATLIN_FIXTURE_HTML is hand-crafted; recommend capturing one real Татлин HTML snapshot as a golden file for regression detection (DOM structure drift).
RE_CORPUS_COUNT_WORD missing девять/десять — edge case, low priority.

Positive observations

Strong test coverage (28 tests, all offline, no network)
Pydantic models with safe defaults (Field(default_factory=list), None for optionals)
Proper fetch_around raises NotImplementedError explaining JK-slug pattern (correct delegation to fetch_jk)
Developer URL absolute/relative branching handled (href.startswith("http"))
Description fallback chain: О комплексе → Расположение, транспортная доступность → Расположение → None
Graceful return None on HTTP non-200 + try/except around fetch
developer_other_jk[:10] cap prevents unbounded list growth

Vault cross-check

decisions/YandexRealtyScraper_v1_Implementation_Plan.md Stage 5 scope match 100%
decisions/Schema_YandexRealty_Recon.md sec 17-26 — confirms inline coords + 149 markers + 353 reviews target
Recommend worker create code/modules/tradein/Yandex_Newbuilding_Parser.md per CLAUDE.md rule #6

Blast radius

Risk: Low (additive, no existing-code touch, Tier 3 supplementary source per plan)
Reversibility: Trivial (revert single squash commit)
Merge window: Anytime

Merged via deep-code-reviewer.

## Deep Code Review — verdict APPROVE (merged) **Status**: APPROVE **Files reviewed**: 2 (P1: scraper, P3: tests) **LOC**: +774/-0 **Merge commit**: `51650276b2a8613760118f633bef193df678ebf8` ### Cross-file impact - `parse_house_class`, `parse_house_type`, `RE_JK_ID` — confirmed exports in `yandex_helpers.py` (re-uses Stage 3 helpers PR #456 pattern). OK. - `BaseScraper._http_get` / `sleep_between_requests` / abstract `fetch_around` — used identically to existing `yandex_realty.py:64` (caller responsible for `async with` context). - No conflict with Wave 4 siblings (PR #466 yandex_detail, PR #468 yandex_valuation — disjoint files). ### Critical focus areas — all satisfied | Area | Verdict | |---|---| | Coord extraction safety | OK for ЕКБ pilot (narrow 2-degree window + ≥4 decimals) | | NLP regex robustness | 11 regexes all anchored on Russian noun/verb context — low FP risk | | 353 reviews extraction | `test_parse_rating_and_counts` asserts `info.text_reviews_count == 353` on Татлин fixture ✅ | | 28 tests | Confirmed count (happy-path + coord×4 + corpus×4 + house_class×2 + commission×2 + rating×2 + dev×2 + metro×3 + minimal + fetch_around raises + section×2 + dedup/cap×2 + model×2) | | Ruff line ≤100 | Both files pass | | `developer_other_jk[]` capped at 10 + dedup | Verified by `test_developer_other_jk_capped_at_10` + `test_developer_other_jk_dedup` | | `corpus_count` word→int | `_WORD_NUM` covers одна/одну/одной/две/три/четыре/пять/шесть/семь/восемь + digit branch | | `commission_month` RU | Stored as inflected form ("июне") — matches plan + schema `text` column | ### Minor improvements (non-blocking, future PR) - **Coord pair-locality** — `_extract_coords` independently picks first lat-in-range and first lon-in-range. If page has a stray decimal `56.xxxx` BEFORE the real coords block, parser could return mismatched lat without paired lon (or vice-versa: Moscow lat 55.7558 falls in EKB LAT_RANGE — `test_extract_coords_outside_range_returns_none` only asserts `lon is None`, lat in that case = 55.7558 ≠ EKB). Recommend post-merge: require lat and lon within ~50 chars of each other in HTML (similar to fixture `"56.855312, 60.576668"`). - **Multi-city LAT/LON_RANGE** — currently EKB-only constants. Per-city dict if Stage 7+ scales to other regions. - **Real Yandex HTML snapshot fixture** — current TATLIN_FIXTURE_HTML is hand-crafted; recommend capturing one real Татлин HTML snapshot as a golden file for regression detection (DOM structure drift). - **`RE_CORPUS_COUNT_WORD`** missing `девять/десять` — edge case, low priority. ### Positive observations - Strong test coverage (28 tests, all offline, no network) - Pydantic models with safe defaults (`Field(default_factory=list)`, `None` for optionals) - Proper `fetch_around` raises NotImplementedError explaining JK-slug pattern (correct delegation to fetch_jk) - Developer URL absolute/relative branching handled (`href.startswith("http")`) - Description fallback chain: О комплексе → Расположение, транспортная доступность → Расположение → None - Graceful return None on HTTP non-200 + try/except around fetch - `developer_other_jk[:10]` cap prevents unbounded list growth ### Vault cross-check - `decisions/YandexRealtyScraper_v1_Implementation_Plan.md` Stage 5 scope match 100% - `decisions/Schema_YandexRealty_Recon.md` sec 17-26 — confirms inline coords + 149 markers + 353 reviews target - Recommend worker create `code/modules/tradein/Yandex_Newbuilding_Parser.md` per CLAUDE.md rule #6 ### Blast radius - **Risk**: Low (additive, no existing-code touch, Tier 3 supplementary source per plan) - **Reversibility**: Trivial (revert single squash commit) - **Merge window**: Anytime Merged via deep-code-reviewer.