950-E1: app/tasks/newbuilding_enrich_backfill.py selects houses linked to
ext_source='cian_newbuilding' with a fetchable cian_zhk_url and runs the existing
fetch_newbuilding + save_newbuilding_enrichment over each, populating
houses_price_dynamics + house_reliability_checks (+ house_reviews when present).
- Idempotent + resumable: skips already-enriched houses; force= re-run dup-safe
(price_dynamics UPSERT via dim_key, reliability dedup guard with id-DESC
tiebreaker, reviews ON CONFLICT (source,ext_review_id)).
- Anti-bot aware: get_scraper_delay('cian') + jitter, SAVEPOINT-per-house
(one captcha/failure logs + continues, never aborts the batch).
- limit= for bounded proof runs; sizing counters (total/fetchable/pending).
Fix cian_newbuilding._extract_transport_rate: cian transportAccessibilityRate
drifted to a nested dict and broke save_newbuilding_enrichment's CAST bind on
LIVE data (cannot adapt type 'dict') — coerce dict/float/bad -> int|None
(bool excluded before int). Fixes the live enrichment crash, not just backfill.
house_reviews stays ~0 for now: cian reviews are in a separate XHR, not the ЖК
initialState — extending fetch_newbuilding for them is a documented follow-up.
Tested: 8 new unit tests; isolated-DB proof-run landed 6 price_dynamics + 1
reliability row, idempotency proven across re-runs. Refs #972.
Миграция 100_enable_deactivate_stale_avito.sql — идемпотентный UPDATE scrape_schedules SET enabled=true, next_run_at=NOW() WHERE source='deactivate_stale_avito' AND enabled=false. Чинит QA-находку #759: 090 ON CONFLICT DO NOTHING оставил pre-seed disabled-строку → task ни разу не отработала. Non-destructive, guard на повторный прогон. +7 тестов.
Closes#759.
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
Форма ввода (10 полей + CRM-блок 221px) не помещалась в .form-card
(max-height 100vh-104px) на ноутбуках: переполнение 58px @1440×900,
~158px @13" MacBook. CRM «для менеджера» теперь сворачиваемый (по умолчанию
закрыт, открывается по клику) + лёгкое уплотнение отступов (.field 12→9,
form-body/foot padding). Высота формы 854→646px → влезает без скролла на
1366×768 / 1440×900 / 13" MacBook @100%. Sub-720 — fallback на скролл-фикс
из этой же ветки. Проверено вживую (injection) на проде, tsc --noEmit clean.
PR #923 switched Cian scrapers to settings.cian_proxy_url but the shared
_mock_settings helper in test_scraper_proxy.py still only exposed scraper_proxy_url,
so pre-existing cian proxy-wiring tests broke with AttributeError → deploy pytest gate failed.
Mirror scraper_proxy_url into cian_proxy_url in the mock.
Prod debug (ssh, headless=True + mobileproxy): the camoufox fetch was returning
a 274KB generic avito shell with 0 listings — NOT a block/captcha (no
captcha/datadome/firewall markers, status 200, correct final_url). Avito hydrates
the SERP listings into the DOM client-side AFTER domcontentloaded; at the 2500ms
wait the page.content() snapshot still had only the generic shell. At 5-6s it's
the full SERP: 3.2MB, priceDetailed=50.
Fix: BROWSER_WAIT_MS default 2500→6000 (5s proven full + 1s margin), config.py
browser_wait_ms 2500→6000 to match. The whole camoufox+proxy migration works —
this was the last blocker. NOT headless detection (headless=True returns real
data); the abandoned xvfb/headless=virtual change is unnecessary.
Refs #905, #883
Drive-by: browser/server.py was never linted (pre-commit ruff is scoped to
backend/), so #909 left an unused noqa + a 101-char line. Cleaned up while
editing this file for the proxy fix.
Prod smoke: tradein-browser fetched avito but got the ~400KB soft-block stub
(preloadedState present but priceDetailed=0, firewall=0) — because camoufox ran
with NO proxy (datacenter IP). Root cause: server.py read SCRAPER_PROXY_URL, but
the proxy in backend/.env.runtime is named AVITO_PROXY_URL (mobileproxy, #623) —
the same var curl_cffi-avito uses. So the var was empty in the browser container.
Fix: read AVITO_PROXY_URL first, SCRAPER_PROXY_URL as a generic fallback. One-line
env-name swap; the value is the working mobileproxy egress already on the host.
After deploy, re-run tradein-mvp/scripts/browser-smoke.sh — expect real data
(priceDetailed>0, ~1.4MB, firewall 0). IP rotation (AVITO_PROXY_ROTATE_URL) stays
backend-side and keeps working (same proxy endpoint, changing egress IP).
Refs #905, #883, #623
Reusable smoke test for the camoufox HTTP service. Run from the deploy host:
docker exec -i tradein-backend sh < tradein-mvp/scripts/browser-smoke.sh
Reports health, byte count, preloadedState presence, firewall-block detection
+ first 200 bytes of the response. Avoids long one-line curl commands that get
mangled on terminal paste.
Refs #905, #883
The playwright WS-server path (camoufox.server.launch_server) is incompatible
with playwright >=1.45: camoufox 0.4.11's launchServer.js requires
playwright/driver/package/lib/browserServerImpl.js, which no longer exists in
any playwright 1.45-1.60 → tradein-browser crash-looped with MODULE_NOT_FOUND
('Server process terminated unexpectedly'). Verified by running the built image.
Pivot (no playwright protocol between containers):
- browser/server.py: aiohttp service holding a local AsyncCamoufox (headless,
os/locale/geoip/humanize, proxy from SCRAPER_PROXY_URL). POST /fetch {url} →
new_page/goto/content/close → {html}; GET /health. Page-recycle every N +
crash-recovery (relaunch + 1 retry), serialized via asyncio.Lock.
- browser/Dockerfile: +aiohttp (camoufox fetch-as-root/#908 block untouched).
- backend BrowserFetcher: playwright.connect → httpx POST to the browser service
(one retry on transport/HTTP error). All playwright imports dropped.
- config: browser_ws_endpoint → browser_http_endpoint (http://tradein-browser:3000).
Verified by a REAL local image build + run: image builds, container stays Up
(Restarts=0, no crash-loop), GET /health=200, POST /fetch launches camoufox and
drives Firefox (only NS_ERROR_UNKNOWN_HOST = no DNS egress in the local build env,
proving the full HTTP→camoufox→Firefox chain; real fetch validates on prod with
network+proxy). 7 unit tests + ruff clean.
Dormant: scraper_fetch_mode stays curl_cffi. Refs #905, #883, #884
CI build-browser (and the deploy) went red on #907's browser image:
PermissionError: [Errno 13] Permission denied:
'/usr/local/lib/python3.12/site-packages/camoufox/GeoLite2-City.mmdb'
camoufox fetch with geoip=True downloads TWO things: the Firefox build (→
$HOME/.cache, correctly routed to /home/app via ENV HOME, the #899 fix) AND the
GeoLite2 mmdb, which it writes INTO the camoufox package dir in site-packages.
Since the browser image pip-installs camoufox as root, that dir is root-owned, so
running fetch as the app user can't write the mmdb → build fails.
Fix: run fetch as root (writes both the mmdb and — via ENV HOME — Firefox into
/home/app/.cache), then chown the Firefox cache to app. The mmdb stays root-owned
in site-packages but world-readable, which is all the runtime app user needs.
Verified by a real local image build (camoufox fetch now completes, image exports).
The deploy was gated off by the failed build-browser job (not partially applied),
so prod was never broken.
Refs #905, #883, #899
Prod deploy 2026-05-31 13:50 failed on 095: 'cannot drop column sale_type_text
of table listings because view v_data_quality depends on it'. The view (recreated
in 094) has a CTE 'SELECT * FROM listings' which records a column-level dependency
on EVERY listings column incl sale_type_text — so the bare DROP COLUMN aborts the
whole migration (exit 1), blocking 095/096/097 from applying.
Fix: DROP VIEW v_data_quality -> DROP COLUMN sale_type_text -> recreate the view
with the identical 094 DDL (sale_type_text was never an output column, only pulled
in implicitly via SELECT *; after the drop, * no longer includes it). Verified
v_data_quality is the only object doing SELECT * FROM listings — listings_search_mv
and v_price_divergence reference explicit columns, so no other dependency blocks.
095 never applied on prod (transaction rolled back atomically, not in
_schema_migrations) → next deploy re-runs the corrected file. Editing the merged
migration in place is the correct recovery (a new 098 would not help: the runner
hits 095 first and keeps failing).
Refs #731
Build-verify on the deploy host (2026-05-31) showed the camoufox Firefox build
landed in /root/.cache/camoufox (RUN python -m camoufox fetch ran as root), but
the runtime app user (uid 1000) resolves the cache to /home/app/.cache/camoufox
→ 'browser not found' the moment scraper_fetch_mode flips to browser.
Fix: run the fetch AFTER USER app + set explicit ENV HOME=/home/app (Docker does
not auto-set HOME from USER, and camoufox/platformdirs keys the cache off $HOME).
Now build-time fetch and runtime both use /home/app/.cache/camoufox.
Verified: A) 'camoufox path' as app = /home/app/.cache/camoufox; B) that dir was
empty pre-fix while /root/.cache was root-only (Permission denied to app).
Dormant change — scraper_fetch_mode default stays curl_cffi (no runtime behavior
change), unblocks Phase 1 (#883).
Refs #884, #883
