FE-5, last FE wiring on the mappers pattern. Closes#2042.
- New hooks (lib/trade-in-api.ts): useEstimateHistory (GET /history), useGeocodeSuggest
(GET /geocode/suggest?q=&limit=, debounced, enabled>=3 chars); useQuota reused.
- types/trade-in.ts: EstimateHistoryItem (area_m2 string, median_price), GeocodeSuggestion.
- mappers.ts: mapCache(history) -> {rows: CacheRow[], kpis: CacheKpi[]} — per-row status
by 24h TTL; KPIs FE-derived from /history (всего/средняя цена/повторные %), NOT the
global /cache-stats.
- CacheView: real previous-estimates list. ParamsPanel: address autocomplete dropdown →
captures lat/lon into submit (cleared on manual edit). HeroBar: «Скачать PDF» → /estimate/
{id}/pdf, UUID-gated + safeUrl scheme check + disabled when no estimate. TopNav: «Мои
отчёты» from quota.used/history. page.tsx: invalidate history+quota on new estimate.
- Hydration-safe: PDF link gated behind a mounted flag (urlId is SSR-null/client-uuid).
- FE-1..4 untouched.
Verified: next build green (/v2 39.6 kB); tsx harness ran mapCache vs real /history (10
items: ВСЕГО 10 / СРЕДНЯЯ 8,85 млн / ПОВТОРНЫЕ 50%, statuses, null graceful).
code-reviewer APPROVE after hydration + reports-fallback fixes.
QA on real prod estimate 701795d8 surfaced two mapper defects:
- STREET_RE used JS \b which does NOT word-boundary Cyrillic → street token never
matched → object.address fell back to the bare house number ('14/3') and city
mis-parsed ('14/3, Россия'). Rewrote parseAddress to handle both OSM (house-first)
and DaData (city-first) orders; street keyword delimited by space, not \b. Now
'улица Яскина, 14/3' + city 'Екатеринбург'.
- days_on_market is null across prod lots → scatter-mini rendered empty. Active
analogs now fall back to 'days listed so far' = today-listing_date (deals still
require a real days_on_market — today-deal-date is not exposure). TODO BE-1.
Verified via tsx harness against the real estimate (25/25); next build green.
FE-1 foundation for #2036. app/v2/page.tsx now owns data: ParamsPanel form ->
useEstimateMutation, restore-by-id via window.location.search (mirrors v1, avoids
useSearchParams Suspense build break), sub-hooks useStreetDeals/useEstimateHouseAnalytics;
loading/empty/insufficient/error states (honest placeholders, no fixtures-as-real).
New components/trade-in/v2/mappers.ts: pure AggregatedEstimate(+street-deals/analytics)
-> fixture-shape transforms — 3 price tiers (asking/expected_sold/DKP), ranges, scatter,
7 source slots, summary. CV + per-source counts are FE-approx pending BE-1 (#2043).
ParamsPanel -> controlled form (onSubmit/isPending/error/initialValues). ResultPanel/
ObjectSummary/HeroBar/Footer accept a composite data prop defaulting to the existing
fixture (pixel-perfect markup unchanged). next build green (/v2 34 kB).
avito_detail_backfill (mode=browser) aborted prod run 458 with
attempted=5 enriched=0 blocked=5: the lat-null pending queue is dominated
by DEAD listings (404 — that's why they are stale coord-holes). In browser
mode a dead listing's 404 page ("Ошибка 404. Страница не найдена") has no
item-view, so _is_detail_soft_block returned True → AvitoBlockedError →
backfill counted it as `blocked` and the consecutive-block breaker aborted
the run before reaching live listings.
- New AvitoListingGoneError (subclass of common AvitoError, NOT of
AvitoBlockedError/AvitoRateLimitedError) so the backfill block-trap
does not catch it.
- New _is_detail_not_found() detector + browser-mode branch in fetch_detail,
checked BEFORE firewall/soft-block (keyed on the 404 title so it does not
swallow the generic soft-block deflect). Curl path unchanged (404 → 302/
non-200 → ValueError → failed, as before).
- Backfill: new `gone` counter; dedicated except AvitoListingGoneError branch
that is neutral to the consecutive-block breaker and marks the listing
is_active=FALSE (SAVEPOINT) so it leaves the snapshot scope.
Tests: unit for _is_detail_not_found (404 True; soft-block/item-view/empty
False), browser-mode fetch_detail raising AvitoListingGoneError (not Blocked),
and backfill gone-row marks inactive without aborting the breaker.
Прод-/trade-in/* обслуживает tradein-mvp/frontend (basePath=/trade-in,
standalone), а не главный frontend — v2 по ошибке попал в главный, где
прод его не отдаёт. Markup-only порт (fixtures, без API):
- компоненты v2 → tradein-mvp/frontend/src/components/trade-in/v2/
- роут src/app/v2/ (basePath даёт прод-URL /trade-in/v2)
- public/trade-in-v2/building.png
- HeroBar: <img> → next/image (плоский img не префиксит basePath → 404)
- удалён мёртвый v2 из главного frontend
next build (NEXT_PUBLIC_BASE_PATH=/trade-in) зелёный, роут /v2 prerendered.
Soft-block (HTTP 200 generic витрина без item-view) теперь детектится как блок → существующая 403/firewall reconnect-машинерия → ротация exit-IP вместо silent failed (blocked=0, enriched→0 каскад). + snapshot ORDER BY (lat IS NULL) DESC для приоритизации coord-holes (#1967, in-scope 321). Tests: test_avito_detail_soft_block.py. Full tradein suite 2591 passed.
house_type (normalized) + house_catalog_url→house_url persisted in save_detail_enrichment; COALESCE existing-first for house_type, new-first for house_url. No migration (columns exist). Refs #2009
Locked/no-access users (403/denied path/expired session/trial ended) were
trapped: NoAccessScreen had no logout control, and «Выйти» lived only inside
UserMenu, which does not render on the no-access screen. They could not switch
accounts without clearing browser basic-auth manually.
- Extract logout() into shared `lib/logout.ts` (single source of truth).
- UserMenu now imports it instead of a local copy (behaviour unchanged).
- NoAccessScreen renders an always-available «Выйти» accent button for every
variant.
Mirrored across both frontends (tradein-mvp/frontend + main frontend) to keep
the MIRROR invariant in sync.
praktika pilot→expired role in roles.yaml (paths:[], deny:/**); RouteGuard
short-circuits on role=expired with a dedicated trial-ended screen (not the
generic path-deny path). NoAccessScreen gains variant="trial" with title
«Пробный доступ закончился» and a Telegram link to @ArtemKopylov87. Backend
Role Literal and frontend Role type extended to include "expired". kopylov
(pilot) unaffected.
Behavior-preserving structural refactor: the ~870-line deterministic pricing
block inside estimate_quality() is extracted into a new pure synchronous function
_price_from_inputs() returning a PricingResult dataclass.
Key design decisions:
- All async DB fetches (imv_eval, yandex_val, cian_val) hoisted to estimate_quality
BEFORE the call; passed as pre-fetched values / bool flags.
- DB-dependent helpers whose arguments are computed inside the block (_get_asking_sold_ratio,
_lookup_quarter_index, _lookup_quarter_indexes) injected as Callable parameters
(ratio_resolver, quarter_index_lookup, quarter_indexes_lookup).
- _fetch_house_imv_anchor called once in estimate_quality (was two separate conditional
calls inside the block); single result passed as imv_anchor dict.
- _fetch_dkp_corridor and _fetch_anchor_comps hoisted to estimate_quality with
identical guards.
All 2443 existing tests pass UNMODIFIED. 9 new hermetic unit tests added that
call _price_from_inputs directly with stub callables.
Два связанных фикса из диагностики 2026-06-27:
ФИКС C (run_cian_full_load, per-fetch timeout): monkey-patch _browser.fetch →
asyncio.wait_for(_f(url), timeout=cian_full_load_per_fetch_timeout_s=90s).
BrowserFetcher.fetch имеет httpx-timeout 120s, но browser-сервис иногда виснет
так что httpx не получает ответ (нет EOF) → fetch висит → asyncio.gather
блокирует весь bucket → heartbeat_at не обновляется → reap_zombies убивает живой run.
wait_for(90s) отменяет зависший fetch → TimeoutError → _fetch_page_html ловит
как Exception → None → _one_page → [] → gather завершается нормально.
За флагом (cian_full_load_per_fetch_timeout_s=0 = отключить, >0 = включить).
ФИКС D (background heartbeat): asyncio.create_task(_background_heartbeat())
обновляет heartbeat_at каждые 60s независимо от on_bucket/on_progress прогресса.
Без него пустые bucket или зависший gather замораживает heartbeat на несколько часов.
Task отменяется в finally → не течёт при return/raise/cancel.
Новая настройка (ENV):
CIAN_FULL_LOAD_PER_FETCH_TIMEOUT_S=90.0 — timeout одного browser-fetch
Два связанных фикса из диагностики 2026-06-27:
ФИКС A (_rotate_proxy_ip): вместо одношотного GET с timeout=30s — retry-loop
(proxy_rotate_attempts=3 попыток по proxy_rotate_attempt_timeout_s=8s каждая).
Зависший changeip-сервис больше не блокирует весь sweep на 30s.
Первый успешный attempt → True; все провалились → False (логируется как error).
ФИКС B (run_avito_city_sweep): когда SERP-фаза уже сохранила лоты
(lots_inserted + lots_updated > 0) и упала только detail/houses-фаза
(AvitoBlockedError / RateLimitedError), помечаем run как done (не banned)
при avito_serp_ok_not_banned=True (default). Метрика banned зарезервирована
для случаев где SERP сам заблокирован (0 лотов).
Новые настройки (ENV):
PROXY_ROTATE_ATTEMPT_TIMEOUT_S=8.0 — timeout одной changeip-попытки
PROXY_ROTATE_ATTEMPTS=3 — число попыток перед отказом
AVITO_SERP_OK_NOT_BANNED=true — флаг нового поведения (default on)
_fetch_anchor_comps Tier A runs on EVERY /estimate (flag
estimate_same_building_anchor_enabled defaults True, not overridden in prod).
Its address predicate wraps address in lower(translate(address,'ёЁ','ее')) for
both the LIKE '%street%' and the ~house-number regex — non-sargable, and the
existing listings_address_trgm_idx is on the RAW address, so the planner
seq-scanned all ~66.7k listings (167ms, 15801 buffers) every estimate.
Add a GIN trigram expression index on lower(translate(address,'ёЁ','ее'))
matching the predicate exactly (gin_trgm_ops serves both LIKE and the regex).
Prod EXPLAIN (BEGIN/ROLLBACK, "Хохрякова 48"): 167ms -> 1.77ms (~94x), buffers
15801 -> 97. Independently dry-run-verified: Bitmap Index Scan, 0.66ms.
Result-identical (pure access-path; count/sum/min/max unchanged). No code
change (expression matches the predicate). ~8MB, sub-second build.
Other estimate-path queries confirmed already optimal (Tier W/H geom+rooms
BitmapAnd, Tier S house_id_fk) — no other index warranted.
tradein-mvp/backend/data/sql/137 (correct tradein migrations path).
Found by adversarial valuation audit (2 confirmed, bot-safe).
FIX A (#5): both radius comp queries (Tier H ~3990, Tier W ~4135) ended with
a bare ORDER BY relevance_score; on score ties Postgres returned rows in
undefined order, so the same /analyze could pick different comps across runs.
Append deterministic tiebreaker: relevance_score ASC, distance_m ASC,
scraped_at DESC NULLS LAST, id ASC (id = listings PK → total order). Added id
to each base CTE; outer projection unchanged (no leak downstream).
FIX B (#2): _filter_outliers keeps rows with price_per_m2 IS NULL, but the
median is built from prices_ppm2 (drops them) while n_analogs counted all of
listings_clean — overstating contributing comps ("Найдено N аналогов"
misleading; all-price-less -> median=0 but n_analogs>0). Count n_analogs from
prices_ppm2 in the radius path. #698 anchor overwrite + #691 zero-analog->low
guard unaffected; listings_clean itself not filtered.
Adds tests/test_estimator_n_analogs_priced.py (verified to fail on old code).
Audit also flagged velocity fan-out (false-positive: 0 duplicate domrf_obj_id
on prod) and >/>= disclosure tweaks (cosmetic) — deliberately not changed.
Refs #1871
First XHR after goto(origin) intermittently hit "NetworkError when
attempting to fetch resource" (page net stack/anti-bot not ready),
healed only by the outer re-navigation retry (~30-45s/house in the
house_imv_backfill, #562).
_fetch_json_once now:
- settles FETCH_JSON_SETTLE_MS (default 1200, was hardcoded 500) — fewer
first-fails;
- wraps the in-page fetch() in a JS retry loop (FETCH_JSON_INPAGE_RETRIES
default 1, FETCH_JSON_RETRY_DELAY_MS default 800) that retries ONLY on
network throw, never on HTTP status (4xx/5xx short-circuit, caller
decides). An in-page retry costs ~retryDelayMs vs the ~30-45s outer
re-navigation. Last error re-thrown — outer crash-retry contract intact.
/fetch (SERP) path untouched. +2 tests (settle ms, retry params).
Refs #1917, #562
#1928 fixed scrape_runs.total_seen/new_count for sweeps via _column_counts
but only regression-tested the yandex mark_done path. Add equivalents for
the CitySweepCounters (avito/cian) sweep and the mark_failed path — both
explicitly named in the #1926 audit — so the all-sweeps + failure-path
guarantee is locked in.
Refs #1926
Migration 130 only filled house_id_fk for listings carrying
house_source/house_ext_id. Prod (2026-06-27): 1884 active NULL-FK rows
remain, 0 with house_source/ext_id (130 re-run = no-op). They were
matched via the per-listing-identity path the realtime mirror uses when
no house catalog id exists: match_or_create_house(ext_source=source,
ext_id=source_id) -> Tier-1 house_sources lookup (confidence 1.0).
136 replays that exact lookup offline: UPDATE listings SET house_id_fk
= house_sources.house_id WHERE (ext_source=source, ext_id=source_id) and
house_id_fk IS NULL. Faithful to base.py:704-750; idempotent (NULL
guard); deterministic (no fuzzy/geo); collision-safe (catalog ids live
under distinct ext_source like 'cian_newbuilding'). Dry-run BEGIN/ROLLBACK
fills 4608 NULL-FK rows (744 active), restoring estimator Tier-S
"same-building" grouping. The remaining ~1140 active need re-scrape/
re-match (scraper coverage — #1781 Лёха).
Refs #1781
