Причина: docker restart tradein-backend (при каждом API-деплое) убивал бегущие
sweep'ы avito/cian/rosreestr — они могут идти часами, и их прерывание середине
батча создавало zombie-runs и пропуски данных.
Что сделано:
- scheduler_main.py — standalone entrypoint для tradein-scraper; SIGTERM/SIGINT
отменяют scheduler_loop чисто; GlitchTip init без Starlette/FastAPI интеграций.
- docker-compose.prod.yml — новый сервис scraper (тот же image, команда python -m
app.scheduler_main, только tradein-net); SCHEDULER_ENABLE=false в backend.
- deploy-tradein.yml — селективный up -d --no-deps: scraper пересоздаётся только
при изменениях scraper/infra путей (SCRAPER_CHANGED), не при каждом backend-деплое.
- tests/test_scheduler_main.py — импорт, kill-switch _should_run(), чистая отмена _run().
EPIC18/§19. analyst sees everything (deals, insights, exports, site-finder,
analytics, concept) EXCEPT admin/data-management. Enforcement is backend-hard
(the existing rbac_guard already 403s any non-admin role on /api/v1/admin/*, so
adding the role auto-blocks it) + frontend (deny_paths via /me) + audit.
§19 audit: new best-effort HTTP middleware logs the sensitive actions
(analyze / forecast / forecast-export) to a new audit_log table after the
response. Audit failures never break or delay-fail the request (2-layer
try/except + finally close). Registered INNER to rbac_guard so only authorized
requests are audited (a 403 short-circuits before audit). classify_path matches
export before forecast (anchored).
- auth/roles.yaml: analyst role (paths /**, deny admin-mgmt) + analysttest QA user
- core/auth.py (+ tradein mirror): Role Literal += analyst
- core/audit_middleware.py (new) + main.py registration
- data/sql/144_audit_log.sql (idempotent; auto-applies)
- tests: analyst rbac (403 admin / 200 parcels) + 11 audit cases
No data-level ACL (analyst sees data per policy) -> no #948 dependency. No new
deps. parcels.py untouched. Real analyst logins still need adding to
caddy/users.caddy.snippet (devops).
Refs #962.
The legacy resolve_cian_zhk_url hit /zhk/<id>/ which now 404s, leaving the
318 geo-matched cian houses (house_sources.ext_id set, cian_zhk_url NULL)
unfetchable -> the newbuilding-enrichment backfill matched 0 houses.
Add resolve_cian_zhk_url_via_search(nb_id): fetch the cat.php newbuilding SERP
and extract the canonical zhk-<slug>.cian.ru url, ANCHORED on the
<h1 data-name="Title"> header anchor (NOT naive first-match — the SERP carries
promo/recommendation zhk-* links before the title that would otherwise resolve
the wrong ЖК and silently corrupt enrichment). Validated against the real
2.81MB prod SERP + an adversarial poisoned-recommendation test.
Wire into newbuilding_enrich_backfill: broaden selection to "has ext_id OR
cian_zhk_url", resolve+persist the url under a SAVEPOINT before enriching,
rate-limited + resumable + idempotent. Keep the old resolver (deprecated).
Bounded prod proof (5 houses, direct): 5/5 urls resolved+persisted, 3/5 fully
enriched (+18 price_dynamics, +3 reliability); the 2 misses were direct-mode
anti-bot on the 2nd fetch. Full 318-run gated on the cian mobile proxy
(mproxy.site) being restored. code-reviewer APPROVE (SQL/idempotency) +
resolver hardened against wrong-ЖК. 19 tests green.
Refs #972.
950-E1: app/tasks/newbuilding_enrich_backfill.py selects houses linked to
ext_source='cian_newbuilding' with a fetchable cian_zhk_url and runs the existing
fetch_newbuilding + save_newbuilding_enrichment over each, populating
houses_price_dynamics + house_reliability_checks (+ house_reviews when present).
- Idempotent + resumable: skips already-enriched houses; force= re-run dup-safe
(price_dynamics UPSERT via dim_key, reliability dedup guard with id-DESC
tiebreaker, reviews ON CONFLICT (source,ext_review_id)).
- Anti-bot aware: get_scraper_delay('cian') + jitter, SAVEPOINT-per-house
(one captcha/failure logs + continues, never aborts the batch).
- limit= for bounded proof runs; sizing counters (total/fetchable/pending).
Fix cian_newbuilding._extract_transport_rate: cian transportAccessibilityRate
drifted to a nested dict and broke save_newbuilding_enrichment's CAST bind on
LIVE data (cannot adapt type 'dict') — coerce dict/float/bad -> int|None
(bool excluded before int). Fixes the live enrichment crash, not just backfill.
house_reviews stays ~0 for now: cian reviews are in a separate XHR, not the ЖК
initialState — extending fetch_newbuilding for them is a documented follow-up.
Tested: 8 new unit tests; isolated-DB proof-run landed 6 price_dynamics + 1
reliability row, idempotency proven across re-runs. Refs #972.
Prod debug (ssh, headless=True + mobileproxy): the camoufox fetch was returning
a 274KB generic avito shell with 0 listings — NOT a block/captcha (no
captcha/datadome/firewall markers, status 200, correct final_url). Avito hydrates
the SERP listings into the DOM client-side AFTER domcontentloaded; at the 2500ms
wait the page.content() snapshot still had only the generic shell. At 5-6s it's
the full SERP: 3.2MB, priceDetailed=50.
Fix: BROWSER_WAIT_MS default 2500→6000 (5s proven full + 1s margin), config.py
browser_wait_ms 2500→6000 to match. The whole camoufox+proxy migration works —
this was the last blocker. NOT headless detection (headless=True returns real
data); the abandoned xvfb/headless=virtual change is unnecessary.
Refs #905, #883
The playwright WS-server path (camoufox.server.launch_server) is incompatible
with playwright >=1.45: camoufox 0.4.11's launchServer.js requires
playwright/driver/package/lib/browserServerImpl.js, which no longer exists in
any playwright 1.45-1.60 → tradein-browser crash-looped with MODULE_NOT_FOUND
('Server process terminated unexpectedly'). Verified by running the built image.
Pivot (no playwright protocol between containers):
- browser/server.py: aiohttp service holding a local AsyncCamoufox (headless,
os/locale/geoip/humanize, proxy from SCRAPER_PROXY_URL). POST /fetch {url} →
new_page/goto/content/close → {html}; GET /health. Page-recycle every N +
crash-recovery (relaunch + 1 retry), serialized via asyncio.Lock.
- browser/Dockerfile: +aiohttp (camoufox fetch-as-root/#908 block untouched).
- backend BrowserFetcher: playwright.connect → httpx POST to the browser service
(one retry on transport/HTTP error). All playwright imports dropped.
- config: browser_ws_endpoint → browser_http_endpoint (http://tradein-browser:3000).
Verified by a REAL local image build + run: image builds, container stays Up
(Restarts=0, no crash-loop), GET /health=200, POST /fetch launches camoufox and
drives Firefox (only NS_ERROR_UNKNOWN_HOST = no DNS egress in the local build env,
proving the full HTTP→camoufox→Firefox chain; real fetch validates on prod with
network+proxy). 7 unit tests + ruff clean.
Dormant: scraper_fetch_mode stays curl_cffi. Refs #905, #883, #884
p3 naming-consistency (cadastral cols 0% filled). Migration 094_cadastral_unify.sql
(idempotent DO-block IF EXISTS): drop+recreate listings_search_mv (MV) +
v_data_quality (view), RENAME deals/house_metadata kadastr_num->cadastral_number,
DROP listings.kadastr_num + listings_kadastr_idx. Code SQL-string renames in
search_query/estimator/base (Lot dataclass field NOT renamed). 240 tests pass.
WARN: destructive DDL auto-applies to prod on deploy -> MERGE-GATE post-demo.
Reviewer: verify recreated MV/view DDL vs LIVE pg_get_viewdef before apply
(reconstructed from migration history 050/046, not live capture).
Refs #732
CLIENT-VISIBLE pricing (killer-factor, demo 2026-06-01). The same-building
anchor replaced headline median/range/confidence with an over-confident
premium on as few as 2 same-building asking comps. Human-chosen (Anton)
3-part safe-guard, all implemented:
1. min_comps 2 -> 4: anchor fires only with >=4 same-building comps.
2. confidence cap: at <5 anchor comps, confidence cannot be 'high' -> medium.
3. MAD-clip: drop price_per_m2 outliers among anchor comps before aggregation
(k=estimate_sb_mad_k=3.5); if <min_comps survive, anchor does NOT fire
(fallback to radius-median). Guards against price typos inflating anchor.
Anchor still REPLACES the headline (uplift-no-replace variant NOT chosen).
quarter-index/radius-tier/asking-sold untouched. 8 new unit tests; 61 anchor
tests pass, ruff clean.
MERGE-GATE (client-visible, like #764/#858): do NOT merge without a green
backtest #763 (held-out MAPE <= 18.7% + canonical A/B Малышева 125 /
Заводская 44а). Backtest needs live DB — pre-merge human/qa step.
Refs #755, #694
