569 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
lekss361	6c4e5a6809	Merge remote-tracking branch 'forgejo/main' into feat/tradein-cian-cookies-ui	2026-05-23 18:14:06 +03:00
lekss361	fd3cb4e6ed	feat(tradein): /scrapers/avito — 6-я секция Расписание (in-app scheduler UI) (#483)	2026-05-23 14:55:31 +00:00
lekss361	0202720096	feat(tradein): in-app scheduler — UI-managed schedule (replaces SSH crontab) (#482)	2026-05-23 14:54:25 +00:00
lekss361	99dd403c45	fix(tradein): CIAN_REQUIRED_COOKIES — add real auth cookies (DMIR_AUTH + Cian session) (#480) ... Updated CIAN_REQUIRED_COOKIES filter в cian_session.py по реальному DevTools-дампу logged-in cian.ru сессии (2026-05-23). Добавлены critical auth cookies: DMIR_AUTH (httpOnly), _CIAN_GK, _yasc, _ym_visorc, uxfb_card_satisfaction. Legacy entries сохранены как backward-compat fallback. Regression test test_required_cookies_includes_real_auth гарантирует что все реальные cookies проходят filter. Fixes Stage 4 PR #457 (filter guessed без real probe — DMIR_AUTH был dropped → verify_session falsey).	2026-05-23 14:52:24 +00:00
lekss361	b92eb81f2d	feat(tradein-ui): Cian cookies upload admin page (paste JSON → POST /admin/scrape/cian/upload-cookies) ... New page at /admin/scrape/cian-cookies: - Textarea for pasting Cookie-Editor array or plain dict JSON - Client-side format auto-detection and conversion - Live preview: detected N cookies with key names - Upload button → POST /api/v1/admin/scrape/cian/upload-cookies - Test Session button → GET /api/v1/admin/scrape/cian/test-auth - Error/success inline banners, collapsible instructions - Admin token from localStorage (same pattern as /admin/scrape) - Tab added to admin nav layout Backend endpoints from PR #457 (Stage 4).	2026-05-23 17:46:44 +03:00
lekss361	9643979d2c	feat(tradein): Phase 3.2 — /api/v1/search endpoint + Redis cache (#479) ... POST /api/v1/search reading listings_search_mv with ~20 cross-source filters, parameterized SQL (CAST(:x AS type), psycopg v3), whitelisted ORDER BY (Pydantic Literal), Redis 5min hot cache with graceful degradation (singleton pool via lru_cache). Verified vs data/sql/050_search_optimization.sql: matview column refs (total_area, lng, house_rating, sources[], has_avito/cian/yandex), SQL injection safety, cache failure swallow, router prefix. Deep-code-reviewer: APPROVE.	2026-05-23 14:44:22 +00:00
lekss361	dd24bd8136	feat(tradein): /scrapers/avito — 5-я секция City sweep (auto + cancel + runs list) (#478)	2026-05-23 14:40:11 +00:00
lekss361	2914ff2ef2	feat(tradein): city sweep — auto ЕКБ pipeline (#477) ... Stage 4d: full city sweep ЕКБ (5 anchors × N pages + enrichment + cooperative cancel). - avito.py fetch_around(pages=N) — pagination backward compat default=1 - scrape_pipeline.run_avito_city_sweep — anchors loop + heartbeat + cancel check - scrape_runs.py utility (create/heartbeat/done/failed/cancelled/list_recent) - admin.py 3 endpoints: start (BackgroundTasks) / runs / cancel - migration 051 ADD COLUMN IF NOT EXISTS + DROP/ADD CHECK constraint (text+CHECK, не enum — safe в transaction) - deploy/avito-city-sweep.sh с random delay 0-3h (anti-pattern) - 16 offline tests Deep review approved: migration idempotent, cancel priority correct (mark_done guarded by status='running'), no SQL injection, psycopg v3 compliant.	2026-05-23 14:38:07 +00:00
lekss361	37c4574d0a	feat(tradein): estimator — Cian Valuation as on-demand 7th source (Stage 9) (#476)	2026-05-23 14:29:52 +00:00
lekss361	c0a0bfa413	feat(tradein): Phase 1.3 backfill scripts — fingerprints + house/listing_sources (#475)	2026-05-23 14:29:39 +00:00
lekss361	dbad7a626b	fix(tradein): убрать require_admin полностью — Caddy basic_auth достаточен (#474)	2026-05-23 14:21:01 +00:00
lekss361	b26e0ff7c3	feat(tradein): estimator — Yandex Valuation as on-demand 6th source (#473) ... Stage 8 of YandexRealtyScraper v1. Adds Yandex Valuation (anonymous, no cookies) as 6th on-demand source in estimate_quality. Mirrors Avito IMV pattern: 24h cache via external_valuations(source='yandex_valuation', cache_key) [UNIQUE 026:34], idempotent history persistence via house_placement_history(source, ext_item_id) [UNIQUE 017:50] with synthesized SHA256(address|date|area|floor|prices)[:32] ext_item_id. Graceful degradation on geocode/fetch/cache/save errors. house_id=NULL — matching deferred to Stage 8.x. 11 new tests + 4 IMV regression = 15 green.	2026-05-23 14:14:49 +00:00
lekss361	89c4a2842d	feat(tradein): cross-source matching service (3-tier: cadastr / fingerprint / geo / composite) (#470)	2026-05-23 14:12:16 +00:00
lekss361	649034479f	feat(tradein): Stage 4c — scrapers admin pages (Avito real + Cian/Yandex stubs) (#472)	2026-05-23 14:05:56 +00:00
lekss361	7d649f9239	feat(tradein): conflict_resolution — register Yandex sources in priority dicts (#471) ... Stage 7/9 YandexRealtyScraper v1 (Wave 5). Integrates 4 Yandex scrapers into cross-source field-priority arbitration. - HOUSE_FIELD_PRIORITY: lat/lon → yandex_realty_nb; +year_built/house_type/house_class/rating_score/reviews_count Yandex sources - New house fields: text_reviews_count, corpus_count, total_area_ha, commission_year/month, developer_name, has_panorama, yandex_total_listings, has_lift, ceiling_height - LISTING_FIELD_PRIORITY: description/house_type +yandex_detail - New listing fields (all yandex_detail): agency_name/founded_year/objects_count, views_total_yandex, publish_date_relative, sale_type_text - 37 unit tests (7 pre-existing + 30 new), all pass; ruff clean - Resolver engine _resolve() unchanged (this PR creates the file initially) Refs: decisions/Cross_Source_Matching_Strategy.md sec 3.5+4.5, decisions/YandexRealtyScraper_v1_Implementation_Plan.md Stage 7.	2026-05-23 14:03:35 +00:00
lekss361	e46b7d778e	feat(tradein): search matview + indexes (Phase 3.1) (#469) ... Multi-Source Integration Phase 3.1. Foundation for /api/v1/search p95 < 100ms on 100K+ listings. SQL 050_search_optimization.sql: - CREATE EXTENSION IF NOT EXISTS pg_trgm - ALTER listings ADD COLUMN tsv tsvector GENERATED ALWAYS AS (...) STORED - 10 indexes (geom GIST, composite, gin_trgm, GIN tsv, partial kadastr, ...) - CREATE MATERIALIZED VIEW listings_search_mv (per-listing aggregated across sources) - 5 matview indexes (UNIQUE on listing_id for REFRESH CONCURRENTLY) Celery task tradein.refresh_search_matview: - REFRESH MATERIALIZED VIEW CONCURRENTLY listings_search_mv (psycopg v3) - Logged duration via logger.info - Graceful try/except ImportError fallback (no celery_app.py in tradein-mvp yet) - Beat schedule snippet in docstring for future wiring Fixups (deep-code-reviewer 2026-05-23): - h.ratings_count → h.reviews_count (alias house_ratings_count preserved) - Strip +psycopg dialect prefix from DSN for libpq compat Refs Master Plan sec 5.1 + 6.5 + 10.1.	2026-05-23 14:01:18 +00:00
lekss361	51650276b2	feat(tradein): yandex_newbuilding.py — ЖК landing parser (149 selectors + NLP) (#467)	2026-05-23 13:46:45 +00:00
lekss361	7056ecb23b	feat(tradein): yandex_valuation.py — anonymous house-history scraper (#468)	2026-05-23 13:46:20 +00:00
lekss361	3d90221fa0	feat(tradein): yandex_detail.py — Product JSON-LD + DOM detail parser (#466) ... Stage 4 of YandexRealtyScraper v1. YandexDetailScraper.fetch_detail(offer_url) extracts authoritative price from Product JSON-LD (offers.price exact int) + DOM sections (description, agent, stats, metro, photos 8 sizes, newbuilding link, NLP). 35 unit tests, ruff clean. Photo URLs sanitized downstream via _safe_url CDN allowlist (avatars.mds.yandex.net).	2026-05-23 13:45:09 +00:00
lekss361	b0a7235110	feat(tradein): cian_valuation.py — auth-required Calculator scraper + 24h cache (#465)	2026-05-23 13:43:34 +00:00
lekss361	b5d206bfa8	feat(tradein): Stage 4a — API endpoints for Avito enrichment + IMV benchmark UI (#460)	2026-05-23 13:34:38 +00:00
lekss361	29f94963a7	Merge pull request 'feat(tradein): yandex_realty.py — DOM SERP refactor (vtorichka, 23 cards/page)' (#462) from feat/tradein-yandex-serp-refactor into main	2026-05-23 13:30:36 +00:00
lekss361	5e90797a9f	feat(tradein): SQL 040-046 — matching schema delta (Phase 1.1) (#464)	2026-05-23 13:30:22 +00:00
lekss361	72d2b97ee0	feat(tradein): Stage 4b — IMV benchmark badge + house info card on estimate result (#461)	2026-05-23 13:28:23 +00:00
lekss361	d28bf4e66a	feat(tradein): cian_newbuilding.py — ЖК catalog scraper (chart + reliability + offers, Wave 4 Worker C) (#459) ... Stage 6 / Wave 4 Worker C of CianScraper v1. Cian.ru newbuilding (ЖК) catalog page scraper — extracts newbuilding metadata + 6 sister state containers (realtyValuation chart / reliability / reviews / offers / builders / housesByTurn). Main Stage 6 deliverable: 7-month price dynamics chart → houses_price_dynamics table (cross-source benchmark для Stage 9 estimator). Files: - app/services/scrapers/cian_newbuilding.py (NEW, 548 LOC) - tests/test_cian_newbuilding.py (NEW, 357 LOC, 20 tests) Reviewed: schema match (migrations 020/021/025/029), ON CONFLICT constraint name verified (houses_price_dynamics_dim_key), graceful chart extraction (empty/None/fallback shapes), atomic transaction. Known minor follow-ups (non-blocking): - management_companies UPSERT: ext_id=NULL → potential duplicates (Postgres NULL!=NULL); bounded, document in vault - offers items[] fast path: no dict-only filter / no room_count injection - houses_price_dynamics primary shape stores total price into price_per_sqm column with prices_type='price' discriminator — Stage 9 estimator must respect prices_type filter - house_reliability_checks no UNIQUE → unbounded growth on rescrape (deferred to future migration 030+) Verdict: APPROVE (deep-code-reviewer).	2026-05-23 13:23:11 +00:00
lekss361	2dd76ea5e8	feat(tradein): cian_session cookies management + admin endpoints (Wave 4 Worker A) (#457)	2026-05-23 13:22:00 +00:00
lekss361	ead2ac16c5	feat(tradein): yandex_realty.py — DOM SERP refactor (vtorichka, 23 cards/page) ... Replace dead JSON-state extraction (Yandex moved to chunked/ephemeral state post-hydration) with selectolax DOM parsing using helpers from PR #456. Changes: - New URL: /{city}/kupit/kvartira/vtorichniy-rynok/?page=N (path-based, no geo) - Selector: [data-test="OffersSerpItem"] x ~23 per page - Per-card extraction: offer_id (URL regex), title (rooms/area/floor regex), price/ppm2 (regex + parse_rub), bargain flag, publish_date (parse_ru_date), street address, photos (avatars.mds.yandex domain filter + size upgrade), newbuilding linkage (house_source='yandex_realty_nb' + house_ext_id) - ScrapedLot mapping: lat/lon=None (no coords on SERP — geocode-missing fills later), listing_segment="vtorichka" - Pagination: fetch_around_multi_room -> up to MAX_PAGES=3 pages, dedup by source_id. **kwargs swallow keeps old rooms/sort/pages callers compat. - Preserved: BaseScraper interface, name="yandex" (matches existing rows + avoids breaking dedup_hash; Wave 5 conflict resolution can alias if needed), request_delay_sec=5.0, sleep_between_requests, tenacity retry. Tests: 10 unit tests against hand-crafted fixture HTML (single card / studio / no-price / empty page / multi-card / URL builder / city override) — all pass. Ruff clean.	2026-05-23 16:21:47 +03:00
lekss361	962daf2d47	feat(tradein): yandex_helpers.py — JSON-LD + regex + date + NLP + money helpers (#456)	2026-05-23 13:14:11 +00:00
lekss361	5051ebde2e	feat(tradein): integrate Avito IMV as 5th evaluation source (on-demand cached) (#452)	2026-05-23 13:08:22 +00:00
lekss361	1b3edff6a9	feat(tradein): extend base.py save_listings to persist Cian-specific columns (PR #450 follow-up) (#455)	2026-05-23 13:05:20 +00:00
lekss361	0db71033b4	fix(tradein): 030 — UNIQUE constraint on avito_imv_evaluations.cache_key (unblocks PR #452) (#454) ... Hotfix unblocking PR #452 (Stage 3 IMV estimator integration). Adds 030_avito_imv_cache_key_unique.sql: creates UNIQUE INDEX imv_cache_key_uniq_idx on avito_imv_evaluations(cache_key), allowing save_imv_evaluation's ON CONFLICT (cache_key) DO UPDATE to resolve. Migration 018 only created a non-unique composite index, which would have caused runtime error on first IMV cache write. DELETE-USING dedup guards against historical duplicates (no-op on fresh table). BEGIN/COMMIT atomic; IF NOT EXISTS idempotent; CONCURRENTLY unnecessary (table empty). Lexicographic apply order with PR #453's 030_listings_alter_yandex.sql is safe — different tables, no functional interaction. Reviewed by deep-code-reviewer: APPROVE.	2026-05-23 13:04:21 +00:00
lekss361	f452b132c9	feat(tradein): SQL migrations 030-032 — Yandex schema (listings/houses/history) (#453)	2026-05-23 13:00:54 +00:00
lekss361	eb4764f75a	feat(tradein): scrape_pipeline.py — Avito full orchestrator (search → houses → detail) (#451)	2026-05-23 12:55:13 +00:00
lekss361	4f0319297e	feat(tradein): cian.py SERP refactor — state-based extraction (137 fields, no jitter) (#450)	2026-05-23 12:52:04 +00:00
lekss361	5258a8f023	feat(tradein): avito_houses.py — Houses Catalog parser (state + reviews + history + recs) (#449)	2026-05-23 12:37:24 +00:00
lekss361	7fad898c7f	feat(tradein): SQL migration 029 — extend matching/valuation/dynamics tables (PR #445 follow-up) (#448)	2026-05-23 12:35:03 +00:00
lekss361	9ba776f2ca	feat(tradein): cian_state_parser shared utility + ScrapedLot Cian fields (#447)	2026-05-23 12:35:01 +00:00
lekss361	23e6cbc773	feat(tradein): SQL migrations 019-028 — Cian schema + matching + cookies (#445) ... Stage 1 / Wave 1 of CianScraper v1 — schema foundation for 4 Cian sources (SERP/detail/newbuilding/valuation) + cross-source matching. 10 SQL migrations: - 019-020: ALTER listings/houses with Cian-specific cols - 021-022: management_companies, agents tables - 023-024: offer_price_history, houses_price_dynamics - 025-026: house_reliability_checks, external_valuations cache - 027: cian_session_cookies (pgcrypto encryption) - 028: matching tables (house_sources/listing_sources/house_address_aliases) All migrations are idempotent (BEGIN/COMMIT + IF NOT EXISTS). Reviewed by deep-code-reviewer.	2026-05-23 12:19:28 +00:00
lekss361	7cdde30b13	feat(tradein): avito_detail.py — detail page parser (30+ fields incl. domoteka) (#443)	2026-05-23 12:17:15 +00:00
lekss361	6a2f734356	feat(tradein): avito_imv.py — Avito IMV evaluation API client (2 requests) (#444)	2026-05-23 12:16:47 +00:00
lekss361	1ad2565af1	feat(tradein): SQL migrations 019-028 — Cian schema + matching + cookies ... - 019: ALTER listings (windows_view_type, wcs counts, ceiling_height, repair_type, views, living_area, bedrooms, balconies, loggias, description_minhash, cadastral_number, phones, bargain_allowed, sale_type, house_source/ext_id, metro_stations) - 020: ALTER houses (BTI: series_name, entrances, flat_count, is_emergency, heat/gas/overlap types; Cian: cian_internal_house_id, management_company_id, transport_accessibility_rate, advantages/banks/builders/houses_by_turn jsonb + GIN indexes) - 021: CREATE management_companies + FK to houses.management_company_id - 022: CREATE agents table + agent_id_fk on listings - 023: CREATE offer_price_history (per-listing price change timeline) - 024: CREATE houses_price_dynamics (monthly price/sqm series) - 025: CREATE house_reliability_checks (наш.дом.рф checks) - 026: CREATE external_valuations (unified Avito IMV + Cian Valuation cache) - 027: CREATE EXTENSION pgcrypto + cian_session_cookies (encrypted) - 028: CREATE house_sources, listing_sources, house_address_aliases + address_fingerprint/canonical/merged_into alters Refs: decisions/CianScraper_v1_Implementation_Plan.md Stage 1 decisions/Schema_Cian_SERP_Inventory.md sec 16, 20.3, 26 decisions/Cross_Source_Matching_Strategy.md sec 2	2026-05-23 15:14:49 +03:00
lekss361	a2953e97c4	refactor(security): убрать UI ввода admin token (backend не требует с PR #437) (#442)	2026-05-23 12:06:37 +00:00
lekss361	47f4cc7d58	refactor(tradein): avito search — remove jitter (C-5) + mobile API dead code + parse houseLinks (#441)	2026-05-23 11:55:18 +00:00
lekss361	a2147872ed	feat(tradein): SQL migrations 009-018 — houses + listings extension + IMV cache (#440)	2026-05-23 11:41:44 +00:00
lekss361	43a8ec426e	fix(ui): &lt;Link&gt; вместо &lt;a&gt; для internal nav routes (build error fix)	2026-05-23 11:31:59 +00:00
lekss361	65c77b96b6	feat(ui): добавить Trade-In / Концепция / Админ в landing nav (#438)	2026-05-23 11:11:21 +00:00
lekss361	b233bf91cc	refactor(security): убрать X-Admin-Token (Caddy basic_auth достаточен) (#437)	2026-05-23 10:41:22 +00:00
lekss361	0e294641a1	fix(security): log_credentials + auth_audit.log → username из 401 (#436)	2026-05-23 09:59:50 +00:00
lekss361	5e3ba351a0	fix(tradein): security hotfix — C-3 config default + C-4 empty estimate persist + C-6 PDF URL allowlist (#435)	2026-05-23 09:38:23 +00:00
lekss361	3295808b41	fix(security): caddy/** в deploy paths + username из 401 попытки (#434)	2026-05-23 09:25:41 +00:00