dd59ef0516
feat(avito): capture newbuilding ЖК id/url from SERP card; persist newbuilding_id/url
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
2026-06-17 23:15:35 +03:00
97fce43484
Merge pull request 'feat(yandex): map rich entity fields (predictedPrice, seller, listing_date, trend, metro)' ( #1725 ) from feat/yandex-rich-fields into main
...
Deploy Trade-In / changes (push) Successful in 10s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 43s
Deploy Trade-In / build-backend (push) Successful in 58s
Deploy Trade-In / deploy (push) Successful in 42s
Reviewed-on: #1725
2026-06-17 20:03:25 +00:00
df83e0256b
Merge pull request 'fix(tradein): correct SberIndex hedonic/asking dataset filter dims + 404 warning ( #902 )' ( #1724 ) from fix/sber-dataset-paths-902 into main
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / changes (push) Successful in 7s
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 36s
Deploy Trade-In / build-backend (push) Successful in 1m1s
Deploy Trade-In / deploy (push) Successful in 54s
2026-06-17 19:58:48 +00:00
1fe670ea6a
feat(yandex): map rich entity fields (predictedPrice, seller, listing_date, trend, metro)
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
2026-06-17 22:58:16 +03:00
5ca987fcd3
fix(tradein): correct SberIndex hedonic/asking dataset-paths + 404 warning ( #902 )
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
2026-06-17 22:50:52 +03:00
f25e89767b
Merge pull request 'fix(matching): conflict_resolution picks freshest (last_seen_at) value, deterministic ( #1539 )' ( #1722 ) from fix/conflict-resolution-freshness-1539 into main
Deploy Trade-In / changes (push) Successful in 7s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 31s
Deploy Trade-In / build-backend (push) Successful in 54s
Deploy Trade-In / deploy (push) Successful in 58s
2026-06-17 19:49:51 +00:00
f30f6acbf7
fix(matching): conflict_resolution picks freshest (last_seen_at) value, deterministic ( #1539 )
CI / changes (pull_request) Successful in 8s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
2026-06-17 22:43:59 +03:00
45d8e9e4a5
Merge pull request 'fix(cian): persist detail rich fields (description) + stamp detail_enriched_at' ( #1718 ) from fix/cian-detail-rich-fields into main
Deploy Trade-In / changes (push) Successful in 8s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 36s
Deploy Trade-In / build-backend (push) Successful in 1m3s
Deploy Trade-In / deploy (push) Successful in 52s
2026-06-17 19:42:58 +00:00
5b8ee994d5
Merge pull request 'feat(tradein): segment=vtorichka|novostroyki|all on listings search ( #1188 )' ( #1710 ) from feat/listings-segment-param-1188 into main
Deploy Trade-In / changes (push) Successful in 11s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 37s
Deploy Trade-In / build-backend (push) Successful in 1m1s
Deploy Trade-In / deploy (push) Successful in 45s
2026-06-17 19:34:50 +00:00
931a57ac59
Merge pull request 'feat(yandex): ingest novostroyki via gate-API newFlat=YES' ( #1715 ) from feat/yandex-novostroyki-newflat into main
Deploy Trade-In / changes (push) Waiting to run
Deploy Trade-In / test (push) Blocked by required conditions
Deploy Trade-In / build-backend (push) Blocked by required conditions
Deploy Trade-In / build-frontend (push) Blocked by required conditions
Deploy Trade-In / build-browser (push) Blocked by required conditions
Deploy Trade-In / deploy (push) Blocked by required conditions
2026-06-17 19:34:32 +00:00
fe805ea528
fix(cian): persist detail rich fields (description/kitchen/ceiling) + stamp detail_enriched_at
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
2026-06-17 22:32:18 +03:00
22a7a3f086
fix(avito): classify novostroyki by item-development-name marker; map переуступка sale_type
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
2026-06-17 22:24:13 +03:00
bc4c6852a1
feat(yandex): ingest novostroyki via gate-API newFlat=YES
CI / changes (pull_request) Successful in 7s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
2026-06-17 22:22:17 +03:00
b64334d7bd
feat(tradein): segment=vtorichka|novostroyki|all query-param on listings ( #1188 )
CI / backend-tests (push) Has been cancelled
CI / frontend-tests (push) Has been cancelled
CI / openapi-codegen-check (push) Has been cancelled
CI / changes (push) Has been cancelled
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been cancelled
CI / openapi-codegen-check (pull_request) Has been cancelled
2026-06-17 21:43:01 +03:00
a8d7be68ad
feat(yandex): route gate-API fetch through camoufox BrowserFetcher
...
CI / changes (push) Has been cancelled
CI / backend-tests (push) Has been cancelled
CI / frontend-tests (push) Has been cancelled
CI / openapi-codegen-check (push) Has been cancelled
CI / changes (pull_request) Has been cancelled
CI / backend-tests (pull_request) Has been cancelled
CI / frontend-tests (pull_request) Has been cancelled
CI / openapi-codegen-check (pull_request) Has been cancelled
Replace the system-curl SOCKS5 subprocess transport with the shared
BrowserFetcher (camoufox). camoufox's real-browser fingerprint is not
tarpitted by yandex and does not burn the mobile-proxy IP; raw system-curl
got the proxy IP flagged.
One BrowserFetcher is opened per scraper session in __aenter__ and reused
across all page fetches. _http_get now fetches via the browser, extracts the
gate-API JSON from camoufox's HTML <pre> wrapper, and returns it as the body
of a _CurlResponse (status 200 on success, 0 on fetch/extraction failure) so
all existing callers and the tarpit retry/rotate logic keep working unchanged.
URL building, _parse_gate_json, pagination, combos and field mapping are
untouched. Removed the dead curl subprocess constants/method.
2026-06-17 21:16:48 +03:00
262adbf762
Merge pull request 'fix(estimator): DKP corridor uses P10/P90 not min/max ( #1520 )' ( #1695 ) from fix/dkp-corridor-percentile-1520 into main
Deploy Trade-In / changes (push) Waiting to run
Deploy Trade-In / test (push) Blocked by required conditions
Deploy Trade-In / build-backend (push) Blocked by required conditions
Deploy Trade-In / build-frontend (push) Blocked by required conditions
Deploy Trade-In / build-browser (push) Blocked by required conditions
Deploy Trade-In / deploy (push) Blocked by required conditions
2026-06-17 18:05:39 +00:00
6eda4a6d0a
fix(estimator): DKP corridor uses P10/P90 not min/max ( #1520 )
...
CI / changes (push) Successful in 8s
CI / changes (pull_request) Successful in 7s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
Replace absolute min/max with robust percentiles so a single outlier ДКП
deal cannot shift the soft-bound corridor boundaries. Small samples fall
back gracefully via linear interpolation (n=3: P10≈index 0.2, P90≈index 2.8).
2026-06-17 20:50:15 +03:00
17c494a063
fix(scrapers): cian _walk_price_range stop silently truncating price-range tail ( #1393 )
CI / changes (push) Successful in 8s
CI / changes (pull_request) Successful in 6s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
2026-06-17 20:39:16 +03:00
f8779c0812
feat(yandex): switch SERP scraping to JSON gate-API (bypasses bandwidth tarpit) ( #1665 )
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Successful in 32s
Deploy Trade-In / build-backend (push) Successful in 51s
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / deploy (push) Successful in 42s
Deploy Trade-In / changes (push) Successful in 6s
2026-06-17 08:20:46 +00:00
b19c4a0b8c
fix(tradein): yandex fetch_around rotate-on-tarpit + retry (combos sweep resilience)
...
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (pull_request) Successful in 6s
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
CI / changes (push) Successful in 8s
CI / backend-tests (push) Has been skipped
CI / backend-tests (pull_request) Has been skipped
Add _YANDEX_TARPIT_MAX_RETRIES=2 constant and rewrite fetch_around with a
loop: on status_code==0 (curl exit 28, bandwidth tarpit) or captcha detected,
call _rotate_ip() + sleep(2s) and retry. Non-zero non-200 codes (404 etc.)
return [] immediately without rotation. Mirrors the existing exhaustive-path
pattern in _walk_price_range. 4 new unit tests cover: tarpit→rotate→success,
captcha→rotate→success, exhausted retries→empty, 404→no-rotate.
2026-06-17 10:12:24 +03:00
7a88964ef5
fix(avito): scale per-anchor timeout for detail-enrich + durable SERP counters ( #1661 )
Deploy Trade-In / test (push) Successful in 36s
Deploy Trade-In / changes (push) Successful in 7s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-backend (push) Successful in 54s
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / deploy (push) Successful in 45s
2026-06-17 06:07:50 +00:00
4811920d28
fix(tradein): yandex combos watchdog timeout (P0) + deploy scraper filter gap ( #1659 )
...
Deploy Trade-In / changes (push) Successful in 7s
Deploy Trade-In / build-browser (push) Successful in 28s
Deploy Trade-In / test (push) Successful in 34s
Deploy Trade-In / build-frontend (push) Successful in 1m46s
Deploy Trade-In / deploy (push) Successful in 46s
Deploy Trade-In / build-backend (push) Successful in 53s
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-06-16 20:22:27 +00:00
e7901bc1e8
feat(tradein): yandex city sweep — EKB-center rooms×price combos (replaces 5-anchor shallow) ( #1658 )
Deploy Trade-In / changes (push) Successful in 7s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 33s
Deploy Trade-In / build-backend (push) Successful in 52s
Deploy Trade-In / deploy (push) Successful in 43s
2026-06-16 19:57:37 +00:00
66433dfe1b
feat(cian): collect price history via BrowserFetcher (fixes silent no-op backfill) ( #1657 )
...
Deploy Trade-In / changes (push) Successful in 7s
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Successful in 33s
Deploy Trade-In / build-backend (push) Successful in 52s
Deploy Trade-In / deploy (push) Successful in 44s
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-06-16 19:47:16 +00:00
91f9a95f93
feat(scrapers): segment-aware generic deactivate_stale for yandex/cian ( #1558 )
Deploy Trade-In / changes (push) Successful in 7s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / deploy (push) Successful in 45s
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 33s
Deploy Trade-In / build-backend (push) Successful in 56s
2026-06-16 16:09:19 +00:00
c07b5d7c5d
chore(tradein): деактивировать N1.ru scraper — источник переехал в Cian
...
CI / openapi-codegen-check (push) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
CI / changes (push) Successful in 8s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / changes (pull_request) Successful in 7s
N1.ru перестал быть самостоятельным источником (переехал в Cian, подтверждено
владельцем 2026-06). Активный скрап прекращён; 389 исторических listings в БД
сохраняются без изменений.
Убрано:
- app/services/scrapers/n1.py (удалён)
- run_n1_city_sweep + N1CitySweepCounters + N1_SWEEP_MAX_CONSECUTIVE_FAILURES
из scrape_pipeline.py
- trigger_n1_city_sweep_run + dispatch elif в scheduler.py
- POST /scrape/n1 + GET /scrape/n1/runs endpoints из admin.py
- N1Scraper из scrape_around dispatch dict и ScrapeRequest.multi_room_n1
- "n1": 5.0 из scraper_settings._DEFAULT_DELAY_BY_SOURCE
- tests/test_n1_city_sweep.py, tests/test_n1_floor_parse.py (удалены)
Добавлено:
- data/sql/114_disable_n1_sweep.sql: UPDATE scrape_schedules SET enabled=false
WHERE source='n1_city_sweep' (idempotent, строка сохраняется для audit-истории)
Сохранено (историческое):
- schemas/search.py Literal[..."n1"] — API-фильтр исторических listings
- config.py scrape_allowed_hosts n1.ru / ekaterinburg.n1.ru
- exporters/trade_in_pdf.py — цвет/лейбл N1 для отображения старых записей
- geocoder.py и admin.py docstring N1-комментарии где не активный код
1802 passed, 2 deselected (test_search_cache_hit, test_cian_valuation.cache_hit)
2026-06-16 14:50:51 +03:00
b390c8334c
fix(tradein): cian detail-парсер не извлекал kitchen_area_m2 из offer.kitchenArea
...
CI / changes (push) Successful in 8s
CI / openapi-codegen-check (push) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / changes (pull_request) Successful in 7s
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
Добавлено поле kitchen_area_m2 в DetailEnrichment dataclass,
извлечение через _parse_float(offer.get("kitchenArea")) в fetch_detail,
и COALESCE(CAST(:ka AS double precision), kitchen_area_m2) в UPDATE listings
внутри save_detail_enrichment.
Имя поля kitchenArea — стандартное cian offer API (live-verify не делался,
прокси недоступен в dev-окружении).
Closes #1553
2026-06-16 14:37:49 +03:00
e2e0291a0a
fix(tradein): yandex detail backfill — заменить fetch_detail на curl_cffi+proxy
...
CI / changes (push) Successful in 8s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
YandexDetailScraper.fetch_detail использует plain httpx (BaseScraper._http_get)
без прокси и без TLS-fingerprinting → на датацентровом IP Yandex отдаёт капчу /
shell-HTML → parse всегда None → backfill 0% эффективен.
Замена: curl_cffi AsyncSession(chrome120 + scraper_proxy_url), идентично
yandex_address_backfill.py (доказано работающий путь на проде). HTTP 200 →
YandexDetailScraper.parse(html) (чистая функция, без сети). parse→None
считается fail (captcha wall); consecutive None → abort после max_consecutive_blocks.
Тесты обновлены: мокается AsyncSession.get + YandexDetailScraper.parse
(не fetch_detail). Добавлены кейсы: non-200 → abort, proxy=None → proxies=None.
2026-06-16 14:05:22 +03:00
03e43f12b5
feat(tradein): yandex detail backfill — save_detail_enrichment + nightly task ( #1553 )
...
CI / changes (push) Successful in 7s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
- yandex_detail.py: добавить save_detail_enrichment(db, listing_id, enrichment) → bool;
UPDATE listings COALESCE по 17 полям (rooms/area_m2/floor/total_floors/address/
description/repair_state/publish_date/views_total_yandex/publish_date_relative/
agency_name/agency_founded_year/agency_objects_count/metro_stations/photo_urls/
newbuilding_url/newbuilding_id); metro/photos → jsonb; detail_enriched_at = NOW()
- app/tasks/yandex_detail_backfill.py: новый task run_yandex_detail_backfill;
snapshot SELECT source='yandex' AND detail_enriched_at IS NULL; YandexDetailScraper
async context manager; consecutive None abort (max_consecutive_blocks=5); budget guard;
run lifecycle update_heartbeat/mark_done/mark_failed
- scheduler.py: trigger_yandex_detail_backfill_run + elif source=='yandex_detail_backfill'
в scheduler_loop; window 12-15 UTC (после avito_detail 09-12 UTC)
- migration 113: ADD COLUMN IF NOT EXISTS photo_urls jsonb, newbuilding_url text,
newbuilding_id text; seed scrape_schedules yandex_detail_backfill enabled=true
- tests: 10 тестов (8 backfill + 2 save_detail_enrichment); 1819 passed 0 failed
Closes #1553
2026-06-16 13:58:20 +03:00
227e82dc01
feat(tradein): avito detail-enrichment nightly backfill ( #1551 )
...
CI / changes (push) Successful in 7s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
- New task app/tasks/avito_detail_backfill.py with run_avito_detail_backfill()
* Single snapshot SELECT at start (guarantees termination)
* Same proxy/AsyncSession path as scrape_pipeline.py step 5
* Budget guard (budget_sec), consecutive block abort (mark_done not mark_failed)
* rotate_ip() on every AvitoBlockedError; rollback on generic Exception (#1368 )
* start = time.monotonic() initialized before try so except can reference it
- Scheduler wiring: trigger_avito_detail_backfill_run() + elif in scheduler_loop()
- Migration 112: scrape_schedules INSERT window 09-12 UTC, batch_size=800,
budget_sec=3600, request_delay_sec=6, max_consecutive_blocks=5
- 7 unit tests (no pytest-mock, unittest.mock only): all 7 passing,
full CI suite 1809 passed
2026-06-16 13:25:29 +03:00
b968b811cb
Merge pull request 'fix(scrapers): geocode-missing listings nightly scheduler — listings geom coverage backfill (all sources)' ( #1549 ) from fix/geocode-missing-listings-scheduler into main
...
Deploy Trade-In / changes (push) Successful in 6s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Successful in 32s
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / build-backend (push) Successful in 53s
Deploy Trade-In / deploy (push) Successful in 43s
Reviewed-on: #1549
2026-06-16 08:32:22 +00:00
e76a8dcaef
fix(tradein): review minors — ceiling_height_m numeric(5,2) + strengthen finishing test
...
CI / changes (push) Successful in 8s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (pull_request) Successful in 6s
- Migration 111: ceiling_height_m numeric → numeric(5,2) (schema consistency with living_area_m2/kitchen_area_m2)
- test_finishing_stored_lowercase_stripped: fixture now uses padded uppercase ' Чистовая ' to actually exercise .strip().lower(), asserts finishing == 'чистовая'
2026-06-16 10:24:20 +03:00
5285a03c98
fix(tradein): avito detail parser — добавить living_area_m2/ceiling_height_m/finishing
...
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (push) Successful in 7s
CI / backend-tests (push) Has been skipped
STEP 0 (живая проверка 2026-06-16): три страницы EKB квартир вернули HTTP 200.
Подтверждены метки: "Жилая площадь" (напр. "12.6 м²"), "Высота потолков"
("2.7 м"), "Отделка" (напр. "чистовая"). "Стоимость ремонта" — не обнаружена.
- RUS_FIELD_MAP: добавлены три записи.
- DetailEnrichment: новые поля living_area_m2/ceiling_height_m/finishing
(в dataclass attrs и в locals-блоке parse_detail_html).
- parse_detail_html: ветки elif для трёх полей; helper _parse_height_m().
- save_detail_enrichment: COALESCE-строки + params-dict для новых полей.
- migration 111: ADD COLUMN IF NOT EXISTS ceiling_height_m, finishing
(living_area_m2 уже существовала, migration не нужна).
- Тесты: 5 новых кейсов в test_avito_detail_parse.py + 5 unit-тестов _parse_height_m.
Полный suite: 1795 passed, 2 deselected.
2026-06-16 10:19:09 +03:00
a56420bdfb
fix(tradein): wire nightly geocode backfill для listings в in-app scheduler
...
CI / changes (push) Successful in 7s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
Root cause: 7674 avito/yandex/n1 listings з null-geom мають ненульовий address,
але geocode_tried_at IS NULL — OS cron геокодує тільки deals, ніколи listings.
Parts:
- geocode_missing.py: WHERE додає geocode_tried_at IS NULL OR tried_at < 7d
(loop-safe); при failure (geo=None / exception) позначаємо tried_at=NOW().
Новий run_geocode_missing_listings: batch loop з wall-clock budget (30m),
update_heartbeat / mark_done / mark_failed lifecycle через scrape_runs.
- scheduler.py: trigger_geocode_missing_listings_run + dispatch elif.
- migration 110: seed scrape_schedules row (enabled=true, window 06-09 UTC).
- tests: оновлено 2 тести (commit на tried_at UPDATE); +7 нових тестів.
1794 passed.
2026-06-16 10:01:49 +03:00
55049046b1
fix(tradein): починить 3 теста-регрессии из #1543 (sber rollback mock, matching tier0, cian price history) — deploy gate green
CI / changes (push) Successful in 7s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
2026-06-16 09:03:27 +03:00
bc504d9519
fix(tradein): M1 review — short_address для match_or_create_house (alias keyspace)
CI / changes (push) Successful in 8s
CI / changes (pull_request) Successful in 6s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
2026-06-16 08:35:38 +03:00
b72af1cc49
fix(tradein): avito houses dedup через match_or_create_house + merge-миграция дублей (108)
...
CI / changes (push) Successful in 7s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
Корень проблемы: avito_houses.py обходил канонический matching-путь через
прямой INSERT ON CONFLICT(source, ext_house_id), создавая дубли для физически
одного дома (~33% таблицы houses = ~4479 лишних строк).
Forward fix: upsert_house/upsert_house_with_url → _persist_house(db, h, url),
который вызывает match_or_create_house (3-tier: source_exact→fingerprint→geo→new,
с pg_advisory_xact_lock) и затем UPDATE с enrichment-полями. avito-авторитетные
поля (rating, developer, технические хар-ки) перезаписываются напрямую;
address/lat/lon/year_built — через COALESCE чтобы не затирать canonical-данные.
Merge-миграция 108: cluster_key по cadastral/address/geo → canonical (max
не-NULL полей, geom присутствует, min(id)) → репойнт всех 11 зависимых таблиц
с обработкой UNIQUE-коллизий → DELETE дублей → backfill house_sources/aliases.
2026-06-15 22:40:46 +03:00
278dafba76
fix(tradein): yandex newbuilding rating drift — "4.3из 5" без пробела ( #974 )
...
CI / changes (push) Successful in 8s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
Валидационный yandex_newbuilding_sweep (run 80) заполнял ratings_count /
text_reviews_count / developer_name / lat-lon, но rating оставался NULL для всех
ЖК. Live-диагностика (прод-браузер, ЖК Татлин/Рио 2026-06-15): рейтинг рендерится
как «4.3из 5» / «4,5из 5» — БЕЗ пробела перед «из» (nbsp внутри «из 5»), а
RE_RATING требовал \s+ перед «из» → match=None.
- RE_RATING: (\d[.,]\d)\s+из\s+5 → (\d[.,]\d)\s*из\s+5 (пробел перед «из» опционален).
\s уже матчит nbsp, поэтому «из\xa05» обрабатывался; ломал именно отсутствующий
разделитель перед «из».
- test_parse_rating_no_space_before_iz: регрессия на обе дрейфовые формы.
Существующие rating-тесты («4,3 из 5» с пробелом) остаются зелёными (\s* ⊇ \s+).
Refs #974
2026-06-15 21:29:43 +03:00
891598a5d7
fix(tradein): cian newbuilding MFE/reliability drift — initialState extraction ( #972 )
...
CI / changes (push) Successful in 8s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
Cian переименовал MFE ЖК-карточки и переместил reliability-контейнер, из-за чего
fetch_newbuilding получал initialState=None → failed_fetch для всех домов
(verified live 2026-06-15 на прод-браузере через мобильный egress):
- MFE: 'newbuilding-card-desktop-fichering-frontend' → 'newbuilding-card-desktop-frontend'
(старое имя на странице отсутствует → extract_state=None → backfill succeeded=0)
- reliability: initialState.reliability → initialState.reliabilityState.reliability
(inner dict сохранил форму {checkStatus, details[], banner, actions})
Проверено на ЖК «Светлый» (nb 17556): newbuilding.id, realtyValuation chart (27 точек),
reliabilityState.reliability.checkStatus + 2 details — всё извлекается.
test_fetch_newbuilding_parses_state: fixture обновлён под reliabilityState.reliability.
Полный suite зелёный (1779 passed; 2 несвязанных env-only локальных падения —
cian_valuation cache / search_api auth — в CI проходят).
Refs #972
2026-06-15 21:12:05 +03:00
483fdb0caa
Merge pull request 'feat(tradein): scheduler-источник cian_city_sweep (newbuilding Phase 4), seed dormant ( #973 )' ( #1296 ) from feat/cian-city-sweep-schedule into main
...
Deploy Trade-In / changes (push) Successful in 8s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 32s
Deploy Trade-In / build-backend (push) Successful in 56s
Deploy Trade-In / deploy (push) Successful in 42s
Reviewed-on: #1296
2026-06-15 17:48:33 +00:00
b0a4df48e1
feat(tradein): scheduler-источник cian_city_sweep (newbuilding Phase 4), seed dormant ( #973 )
2026-06-15 19:09:26 +03:00
35bc792cf2
fix(tradein): обновить test_scraper_proxy под BrowserFetcher-транспорт fetch_newbuilding — зелёный CI-гейт ( #972 )
CI / changes (push) Successful in 6s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (pull_request) Successful in 5s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
2026-06-15 19:02:13 +03:00
b3e7a9f9fa
Merge pull request 'fix(tradein): cian fetch_newbuilding через BrowserFetcher (camoufox) — initialState под анти-ботом ( #972 )' ( #1540 ) from fix/cian-newbuilding-browser-fetch into main
...
Deploy Trade-In / changes (push) Successful in 6s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Failing after 40s
Deploy Trade-In / deploy (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / build-backend (push) Has been skipped
Reviewed-on: #1540
2026-06-15 15:44:42 +00:00
c9daf8e268
fix(tradein): yandex sweep — guard ext_id=NULL перед fetch_jk + cleanup httpx ( #974 review M1/M3)
...
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / changes (push) Successful in 7s
CI / changes (pull_request) Successful in 8s
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
M1: добавлен guard перед вызовом fetch_jk — если ext_id пустой/None при наличии
jk_slug, логируем warning, incrementим failed_fetch и пропускаем (continue).
Пустой jk_id формирует битый URL /{slug}-/ и коллизию на UNIQUE ext_id=''.
M3: в admin.py /scrape/yandex-newbuilding заменён `async with YandexNewbuildingScraper()`
на прямое `scraper = YandexNewbuildingScraper()` — fetch_jk использует внутренний
BrowserFetcher, httpx-клиент BaseScraper открывался вхолостую.
Тесты: добавлен test_sweep_slug_present_but_ext_id_null_skips_fetch.
2026-06-15 18:41:08 +03:00
feab585ad5
feat(tradein): yandex_newbuilding sweep-entrypoint + market-схема enrichment, seed dormant ( #974 )
...
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (pull_request) Successful in 5s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
CI / changes (push) Successful in 7s
- SQL 105: CREATE SCHEMA market + market.yandex_jk_enrichment (ext_id UNIQUE, upsert-ready),
houses.yandex_jk_slug, GRANT gendesign_reader на market-слой
- SQL 106: seed scrape_schedules yandex_newbuilding_sweep (enabled=false dormant, окно 02-05 UTC)
- yandex_newbuilding.py: fetch_jk → BrowserFetcher (camoufox), add resolve_yandex_jk_slug
(SERP siteId query через BrowserFetcher, slug из _JK_SLUG_RE href)
- tasks/yandex_newbuilding_sweep.py: enrich_yandex_newbuilding_sweep (resolve slug →
fetch_jk → UPSERT market.yandex_jk_enrichment, SAVEPOINT per house, dry_run, resumable)
- admin.py: POST /scrape/yandex-newbuilding-sweep + GET .../runs
- scheduler.py: trigger_yandex_newbuilding_sweep_run + dispatch elif branch
- tests: 8 task tests + 3 scheduler dispatch tests (20 pass)
2026-06-15 18:28:58 +03:00
480fbba3ca
fix(tradein): cian fetch_newbuilding через BrowserFetcher (camoufox) — initialState под анти-ботом ( #972 )
...
CI / changes (push) Successful in 7s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (pull_request) Successful in 6s
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
CI / backend-tests (pull_request) Has been skipped
curl_cffi получает zhk-*.cian.ru без initialState; tradein-browser (camoufox) тянет
страницу с 1.17MB initialState. Заменяем транспорт только для page-fetch;
session= param сохранён для backward-compat вызывающих. resolve_cian_zhk_url_via_search
остаётся на curl_cffi SERP. Тесты: 21 passed.
2026-06-15 18:19:32 +03:00
e1442d3b1d
feat(tradein-scheduler): nightly newbuilding-enrichment schedule ( #973 )
...
CI / changes (pull_request) Successful in 6s
CI / changes (push) Successful in 7s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
Deploy Trade-In / test (push) Successful in 33s
Deploy Trade-In / build-backend (push) Successful in 1m3s
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / deploy (push) Successful in 40s
Регистрирует отдельный in-app scheduler-источник для cian_newbuilding
enrichment-backfill (houses_price_dynamics / house_reliability_checks /
house_reviews), который раньше бежал только инлайн в Cian full-sweep.
- run_newbuilding_enrich() wrapper (heartbeat → backfill → mark_done/failed),
делегирует backfill_newbuilding_enrichment (#972 , уже в main).
- trigger_newbuilding_enrich_run() + dispatch-ветка в scheduler_loop
(зеркалит trigger_sber_index_pull_run): _claim_run guard, async task,
zombie-reap наследуется; idempotency наследуется (skip уже обогащённых,
per-house SAVEPOINT, bounded per-fire limit дренирует backlog 306 домов).
- seed-миграция 103: source='newbuilding_enrich', окно 00:00-01:00 UTC
(03:00-04:00 МСК, до 01:00+ UTC sweep-блока), limit=25, next_run_at на завтра.
Засеяно enabled=FALSE (dormant): на prod весь external-HTTP scraping намеренно
на паузе. Расписание+триггер готовы, но не стартуют до намеренного возобновления
(UPDATE ... SET enabled=true). Не активирую одиночный источник при общей паузе.
Closes #973
2026-06-13 16:24:03 +05:00
a9a489bba5
refactor(tradein): убрать dead listing_segment param из _fetch_anchor_comps + негативные guard-asserts (review fixup, #1186 )
...
CI / frontend-tests (push) Has been skipped
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / changes (push) Successful in 6s
CI / backend-tests (push) Has been skipped
- Удалён параметр listing_segment: str | None = None из сигнатуры _fetch_anchor_comps
(guard хардкожен в SQL-блоках Tier A/C, параметр не использовался)
- Убраны seg_counts / target_segment в caller-блоке SAME-BUILDING ANCHOR (~L2174)
- Docstring Tier C обновлён: «вторичка-канон guard (#1186 ): NULL = legacy вторичка»
- test_segment_guard_1186.py: добавлены 4 негативных assert'а:
· no bare = 'vtorichka' без IS NULL OR
· no deprecated <> / != 'novostroyki' form
· _fetch_anchor_comps не содержит listing_segment в сигнатуре
· Tier C не использует CAST(:segment AS text) (параметрическую форму)
2026-06-12 12:01:00 +03:00
b70ae46885
fix(tradein): guard listing_segment во всех читателях listings — novostroyki вне comps ( #1186 )
...
Канон-предикат: (listing_segment IS NULL OR listing_segment = 'vtorichka')
NULL = legacy вторичка до миграции 011 (sweep'ы были secondary-only) — отбрасывать нельзя.
Затронуты все SELECT-пути, влияющие на оценку и корректирующие коэффициенты:
- estimator.py: _COMMON_WHERE (Tier S/H), Tier W inline, Tier A anchor (добавлен впервые),
Tier C anchor (заменён с параметрического CAST(:segment) на канон — прежний вариант
отбрасывал NULL-строки при segment='vtorichka').
- asking_to_sold_ratio.py: ask_side, ask_global (_REDERIVE_SQL); bounds (_REDERIVE_BOUNDS_SQL);
ask_tiered, ask_side_all, ask_global (_REDERIVE_TIERED_SQL) — 6 мест.
- sber_index.py: reconciliation-запрос (диагностический asking-median).
Не затронуты: скрейперы/scrape_pipeline, search matview (catalog display, не оценка),
admin stats, geocoding, cian price history, matching — обосновано в инвентаризации.
Тесты: новый test_segment_guard_1186.py (статика SQL + поведение mock); обновлён
test_asking_to_sold_ratio.py (subset-тест нормализует guard, добавленный в refresh но
отсутствующий в 080-seed).
2026-06-12 11:52:09 +03:00
07b06617fd
test(tradein): убрать мёртвый monkeypatch в test_scheduler_main (review fixup)
CI / changes (push) Successful in 6s
CI / changes (pull_request) Successful in 5s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
2026-06-12 10:56:27 +03:00