6 commits

Author	SHA1	Message	Date
Light1YT	e0fff9cfb0	fix(week-review): backend-аудит v2 — 82 issue (#1560-1656) ... Имплементация фиксов 2-го аудита backend/app/** (после merge #1543). Воркер на файл, точечные правки. Верификация: py_compile 58/58 .py. Полностью исправлено (82). Оставлены открытыми (13): partial/needs-cross-file/needs-leha — #1569, #1590, #1593, #1606, #1609, #1617, #1633, #1635, #1637, #1638, #1640, #1642, #1650. Closes #1560 Closes #1561 Closes #1562 Closes #1563 Closes #1564 Closes #1565 Closes #1566 Closes #1567 Closes #1570 Closes #1571 Closes #1572 Closes #1573 Closes #1574 Closes #1576 Closes #1577 Closes #1578 Closes #1579 Closes #1580 Closes #1581 Closes #1582 Closes #1583 Closes #1584 Closes #1585 Closes #1586 Closes #1587 Closes #1588 Closes #1589 Closes #1591 Closes #1592 Closes #1594 Closes #1595 Closes #1596 Closes #1597 Closes #1598 Closes #1599 Closes #1600 Closes #1601 Closes #1602 Closes #1603 Closes #1604 Closes #1605 Closes #1607 Closes #1608 Closes #1610 Closes #1611 Closes #1612 Closes #1613 Closes #1614 Closes #1615 Closes #1616 Closes #1618 Closes #1619 Closes #1620 Closes #1621 Closes #1622 Closes #1623 Closes #1624 Closes #1625 Closes #1626 Closes #1627 Closes #1628 Closes #1629 Closes #1630 Closes #1631 Closes #1632 Closes #1634 Closes #1636 Closes #1639 Closes #1641 Closes #1643 Closes #1644 Closes #1645 Closes #1646 Closes #1647 Closes #1648 Closes #1649 Closes #1651 Closes #1652 Closes #1653 Closes #1654 Closes #1655 Closes #1656	2026-06-17 01:30:52 +05:00
lekss361	099b8e4b59	fix(scrapers): DOM.РФ KN extras — bypass WAF + update renamed endpoints (#503)	2026-05-24 10:53:28 +00:00
lekss361	2aa33e0eb8	fix(stealth): bump Semaphore 6->8 for higher concurrency in gather ... Per-object 7 endpoints через asyncio.gather упираются в semaphore limit. 6 -> 8 — ещё +33% concurrency для batch fetch'а.	2026-05-17 23:04:51 +03:00
lekss361	0f289c05b1	fix(scrape-kn): real documents endpoints (5x) + skip obj_checks 404 + sem 3->6 ... PR #322 used placeholder URLs: - /api/object/{id}/documents -> 404 (Playwright showed: 5 real endpoints: document/rpd / developer/report / project/documentation / documentation/other / document/permits). Replaced with 5 parallel fetches + merge into single upsert_documents call. - /api/object/{id}/checks -> 404. 6 checkboxes likely inline in bulk kn/object payload; endpoint /checks does not exist. Commented out _fetch_obj_checks_safe with TODO for separate investigation PR. - BrowserSession Semaphore(3) -> Semaphore(6) — double concurrency for intra-object gather. GlitchTip: kn_scrape_failures run #19 showed 40 failures each for both placeholder endpoints across first 30 objects of 1532.	2026-05-17 22:59:44 +03:00
lekss361	6d3fa8cbd2	add domrf extras parser (sale_graph + sales_agg + infra + photos) + ... fix Docker build for playwright - Regenerate uv.lock with playwright + tenacity (was missing → uv sync --frozen skipped them, hence playwright not in /app/.venv/bin/). - Dockerfile: explicit Chromium runtime libs in apt (libnss3 + 14 more) + `playwright install chromium` BEFORE apt-lists cleanup. - Schema 51_schema_kn_extras.sql: 5 new tables for time-series sales, current aggregates, POI infrastructure, photo metadata, failure log. - Scraper: 4 new fetch helpers + upserts; photo binary download via Playwright APIRequest; failure-logging with full_url to kn_scrape_failures. - CLI: --no-extras / --download-photos / --no-flats / --probe URL. - Admin UI: extras checkboxes + failures table with copy-URL button. - PRINZIP smoke: 28 objs, 5409 POI, 5150 photos metadata, sales_agg matches probe (Парк Победы 145/291 = 49%).	2026-04-27 18:35:46 +03:00
lekss361	79070bfe99	add interactive parser of наш.дом.рф ЖК+flats with stealth Playwright ... - Discovered real endpoints via chrome-devtools: /сервисы/api/kn/object (offset/limit/place/objStatus) + /portal-kn/api/sales/portal/table. Server-side filter ignored — local filter by developer.companyGroup. - Pure Playwright in-context fetch (httpx blocked by ServicePipe TLS-fp). - Saved fingerprint state in data/playwright_state.json for server reuse. - Celery beat (configurable cron + jitter) + admin trigger endpoint + /admin/scrape UI page. - Schema migration: UNIQUE(id, snapshot_date) for versioned snapshots, kn_scrape_runs journal. - Smoke-tested on PRINZIP: 28 ЖК / 826 квартир as expected.	2026-04-27 17:58:40 +03:00