lekss361
|
879f761214
|
feat(tradein): avito_houses.py — Houses Catalog parser (state + reviews + history + recs)
Stage 2c of AvitoScraper_v2.
- fetch_house_catalog(house_url) — async HTTP via curl_cffi (chrome120)
- parse_houses_state(state, house_url) — pure parser (testable без сети)
- 10-widget JSON state structure (housePage, reviews, miniSerp, housePlacementHistory, recommendations)
- Dataclasses:
* HouseInfo (28 fields: identity, geo, expandParams flatten, developer, rating, map_pins, raw_characteristics)
* HouseReview (textSections split: main/pros/cons; ru date parse)
* PlacementHistoryItem (removed_date ALWAYS None — widget не отдаёт; точная дата только в IMV API)
* SellerInfo (ext_seller_id из URL last segment, без logos)
* MiniSerpListing (active listings в доме + metro + seller)
* RecommendationItem (lightweight: title/price_text/address/url/image)
* HouseCatalogEnrichment (orchestrator dataclass)
- Save flow (5 ops):
* _upsert_house_with_url → houses (UNIQUE source='avito', ext_house_id) + raw_characteristics jsonb
* save_house_reviews → house_reviews (UNIQUE source, ext_review_id), без images
* save_sellers → sellers (UNIQUE source, ext_seller_id), без logos
* link_listings_to_seller → UPDATE listings.seller_id_fk by source_id match
* save_placement_history → source='avito_widget', removed_date=NULL
- pytest offline smoke: 18 tests, 5 widget types, 25+ field assertions
- Recommendations save skipped — Stage 3 решит
|
2026-05-23 15:27:22 +03:00 |
|
lekss361
|
7f03a5b634
|
feat(tradein): cian_state_parser shared utility + ScrapedLot Cian fields
- Add cian_state_parser.py: regex-based extractor for window._cianConfig push() entries,
supports direct JSON, JSON.parse() escaped form, and optional demjson3 fallback
- Extend ScrapedLot in base.py with 13 Cian-specific fields (living_area_m2,
bedrooms_count, balconies_count, loggias_count, description_minhash,
cadastral_number, building_cadastral_number, phones, is_homeowner, is_pro_seller,
bargain_allowed, sale_type, metro_stations)
- Add tests/test_cian_state_parser.py with 4 unit tests covering all parse strategies
|
2026-05-23 15:24:50 +03:00 |
|