lekss361/gendesign

Fork 0

feat(tradein-ui): Cian scraper admin page + global request delay control #486

Merged

lekss361 merged 1 commit from feat/tradein-cian-admin-page into main

2026-05-23 15:36:08 +00:00

lekss361 commented

2026-05-23 15:32:12 +00:00

Owner

Summary

Replace tradein-mvp/frontend/src/app/scrapers/cian/page.tsx stub (24 lines) с full admin UI mirroring Avito conventions. Adds Global Request Delay slider/input on the same page (calls new endpoints from sibling PR #485).

Files

tradein-mvp/frontend/src/app/scrapers/cian/page.tsx — replaced stub, +435 lines / −8 lines (451 lines total)

Branch cleanup

⚠️ Initial push had worktree pollution (backend files from sibling worker leaked into this branch). Cleaned via git reset к forgejo/main + replay только frontend file + --force-with-lease to remote. Final diff vs main: 1 file only.

Cards implemented

Cookies status callout — link к /trade-in/scrapers/cian-cookies + live auth badge from GET /api/v1/admin/scrape/cian/test-auth
Scrape Around — lat/lon/radius_m → POST /api/v1/admin/scrape body {lat, lon, radius_m, sources: ["cian"]} + multi-room checkbox
Detail page enrichment — placeholder (backend endpoint cian-detail not yet wired through admin API)
Newbuilding scrape — placeholder (backend endpoint cian-newbuilding not yet wired through admin API)
Valuation Calculator — placeholder (backend endpoint cian-valuation not yet wired through admin API)
🆕 Global Request Delay — slider + number input synced; GET/PUT /api/v1/admin/scraper-settings; read-only per-source list для context

Global delay control flow

GET /api/v1/admin/scraper-settings → render list
User adjusts slider/input (0..30s, step 0.5)
PUT /api/v1/admin/scraper-settings/global body {source: "global", request_delay_sec: <value>}
On success → invalidate ["scraper-settings"] query → refresh list

Effect: next scrape request waits max(per_source, global) seconds.

TS strict

All interfaces explicitly typed (ScraperSetting, UploadResponse, etc.)
No any
Generic apiFetch types preserved

Patterns reused

apiFetch from @/lib/api (relative paths, basePath=/trade-in applies)
TanStack Query useMutation/useQuery
Topbar active="cian" (Topbar already has tab from PR #481)

Auth

No X-Admin-Token — tradein admin endpoints auth через Caddy basic_auth (matches Avito page conventions + cian-cookies page).

Backend endpoints NOT yet existing (UI displays static notice)

POST /api/v1/admin/scrape/cian-detail (Stage 5 backend exists, admin trigger pending)
POST /api/v1/admin/scrape/cian-newbuilding (Stage 6 backend exists, admin trigger pending)
POST /api/v1/admin/scrape/cian-valuation (Stage 7 backend exists через estimator, standalone endpoint pending)

UI shows "Endpoint не реализован" notice — easy follow-up PRs to wire them.

Push verify

LOCAL == REMOTE = 4eaa4050

Companion PR

PR #485 — feat/tradein-global-scraper-delay — backend GET/PUT /api/v1/admin/scraper-settings endpoints + SQL migration 054. Must merge first для UI to work.

URL

https://gendsgn.ru/trade-in/scrapers/cian (Cian tab в Topbar)

Test plan

Branch cleaned (1 file diff vs main)
Push verified
Post-merge (после PR #485): adjust global slider → next scrape waits as expected
Cookies status badge updates correctly when test-auth changes

## Summary Replace `tradein-mvp/frontend/src/app/scrapers/cian/page.tsx` stub (24 lines) с full admin UI mirroring Avito conventions. Adds Global Request Delay slider/input on the same page (calls new endpoints from sibling PR #485). ## Files - `tradein-mvp/frontend/src/app/scrapers/cian/page.tsx` — replaced stub, +435 lines / −8 lines (451 lines total) ## Branch cleanup ⚠️ Initial push had worktree pollution (backend files from sibling worker leaked into this branch). Cleaned via `git reset` к forgejo/main + replay только frontend file + `--force-with-lease` to remote. Final diff vs main: **1 file only**. ## Cards implemented 1. **Cookies status callout** — link к `/trade-in/scrapers/cian-cookies` + live auth badge from `GET /api/v1/admin/scrape/cian/test-auth` 2. **Scrape Around** — lat/lon/radius_m → `POST /api/v1/admin/scrape` body `{lat, lon, radius_m, sources: ["cian"]}` + multi-room checkbox 3. **Detail page enrichment** — placeholder (backend endpoint `cian-detail` not yet wired through admin API) 4. **Newbuilding scrape** — placeholder (backend endpoint `cian-newbuilding` not yet wired through admin API) 5. **Valuation Calculator** — placeholder (backend endpoint `cian-valuation` not yet wired through admin API) 6. **🆕 Global Request Delay** — slider + number input synced; GET/PUT `/api/v1/admin/scraper-settings`; read-only per-source list для context ## Global delay control flow ``` GET /api/v1/admin/scraper-settings → render list User adjusts slider/input (0..30s, step 0.5) PUT /api/v1/admin/scraper-settings/global body {source: "global", request_delay_sec: <value>} On success → invalidate ["scraper-settings"] query → refresh list ``` Effect: next scrape request waits `max(per_source, global)` seconds. ## TS strict - All interfaces explicitly typed (`ScraperSetting`, `UploadResponse`, etc.) - No `any` - Generic apiFetch types preserved ## Patterns reused - `apiFetch` from `@/lib/api` (relative paths, basePath=/trade-in applies) - TanStack Query useMutation/useQuery - `Topbar active="cian"` (Topbar already has tab from PR #481) ## Auth No `X-Admin-Token` — tradein admin endpoints auth через Caddy basic_auth (matches Avito page conventions + cian-cookies page). ## Backend endpoints NOT yet existing (UI displays static notice) - `POST /api/v1/admin/scrape/cian-detail` (Stage 5 backend exists, admin trigger pending) - `POST /api/v1/admin/scrape/cian-newbuilding` (Stage 6 backend exists, admin trigger pending) - `POST /api/v1/admin/scrape/cian-valuation` (Stage 7 backend exists через estimator, standalone endpoint pending) UI shows "Endpoint не реализован" notice — easy follow-up PRs to wire them. ## Push verify LOCAL == REMOTE = `4eaa4050` ## Companion PR **PR #485** — `feat/tradein-global-scraper-delay` — backend `GET/PUT /api/v1/admin/scraper-settings` endpoints + SQL migration 054. Must merge first для UI to work. ## URL `https://gendsgn.ru/trade-in/scrapers/cian` (Cian tab в Topbar) ## Test plan - [x] Branch cleaned (1 file diff vs main) - [x] Push verified - [ ] Post-merge (после PR #485): adjust global slider → next scrape waits as expected - [ ] Cookies status badge updates correctly when test-auth changes

lekss361 added 1 commit 2026-05-23 15:32:13 +00:00

feat(tradein-ui): Cian scraper admin page — mirror Avito + global request delay control 4eaa405076

- Replace stub page (24 lines) with full admin UI (~451 lines)
- Cards: cookies status callout, scrape around, detail/newbuilding/valuation placeholders,
  global request delay slider+input (GET/PUT /admin/scraper-settings)
- TanStack Query mutations, TS strict, mirrors Avito page conventions
- Backend endpoints in companion PR feat/tradein-global-scraper-delay

lekss361 merged commit ebfd29a0a6 into main

2026-05-23 15:36:08 +00:00

lekss361 referenced this pull request from a commit

2026-05-23 15:36:08 +00:00

feat(tradein-ui): Cian scraper admin page + global request delay control (#486)

lekss361 commented

2026-05-23 15:36:42 +00:00

Author

Owner

Merged via deep-code-reviewer — verdict APPROVE with 2 medium follow-ups.

Verdict summary

Status: APPROVE -> merged squash (ebfd29a0)
Files: 1 (P2 frontend admin UI), +435/-8
Branch cleanup verified: single file diff vs base b21d7c7c (PR #484), no worktree pollution residue
mergeable=true, head SHA stable
Backend on main HAS scraper-settings endpoints (PR #484 already merged as base) — UI works end-to-end on prod

Backend schema compat check (vs main HEAD)

POST /api/v1/admin/scrape accepts {lat, lon, radius_m, sources: ["cian"], multi_room_cian: bool} — UI body matches exactly
GET /api/v1/admin/scraper-settings returns sorted list[ScraperSetting] — UI shape matches
PUT /api/v1/admin/scraper-settings/{source} UPSERT, source is str (no enum), accepts global — works
GET /api/v1/admin/scrape/cian/test-auth returns {authenticated, userId, reason} — matches TestAuthResp interface
Response types match (TS strict, no any)
apiFetch -> basePath=/trade-in -> Caddy handle_path /trade-in/api/* -> tradein-backend (correct)

Medium follow-ups (NON-blocking)

1. Slider min=0 vs backend ge=1.0 mismatch

Backend ScraperSettingUpdate.request_delay_sec: float = Field(ge=1.0, le=60.0).
UI slider/input min={0} max={30} step={0.5} + hint says "0 = только per-source defaults".

Edge case: user moves slider to 0 or 0.5 -> PUT body {request_delay_sec: 0} -> backend returns 422. UI renders updateSettingMut.error so no crash, but UX misleading (hint promises 0 works, backend rejects).

Recommend follow-up:

Either bump slider min={1} and update hint, OR
Update backend Pydantic to ge=0.0 (matches UI semantics)

2. "Global" delay not consumed by scrapers

tradein-mvp/backend/app/services/scraper_settings.py::get_scraper_delay() only reads the specific source key (with _KEY_ALIASES mapping). There is no max(per_source, global) logic.

UI hint claims: "Глобальный делэй применяется ко всем scrapers (max с per-source defaults). Изменение вступает в силу немедленно".
Actual behavior: PUT succeeds (row stored), but get_scraper_delay("cian") reads cian row only, never global. Cosmetic write.

Recommend follow-up: extend get_scraper_delay() to read both keys and return max(source_delay, global_delay).

Minor

parseFloat(e.target.value) on number/range inputs -> NaN on empty input -> serialized as null -> backend 422. Graceful (no crash). Could guard with Number.isFinite before mutate.
Vault entry code/modules/tradein/Cian_Admin_Page.md not yet present — recommend creating post-deploy.

Verify steps post-deploy

deploy-tradein.yml triggers (paths include tradein-mvp/frontend/**)
https://gendsgn.ru/trade-in/scrapers/cian -> 6 cards render, Topbar cian tab active
Cookies status badge -> green/red per /admin/scrape/cian/test-auth
Scrape Around POST -> 200 + counters returned
Global delay slider value 1..30 -> PUT 200 + row UPSERTed; value 0 -> 422 (expected per current backend)

Merged via deep-code-reviewer — verdict APPROVE with 2 medium follow-ups. ## Verdict summary - Status: APPROVE -> merged squash (`ebfd29a0`) - Files: 1 (P2 frontend admin UI), +435/-8 - Branch cleanup verified: single file diff vs base `b21d7c7c` (PR #484), no worktree pollution residue - mergeable=true, head SHA stable - Backend on main HAS scraper-settings endpoints (PR #484 already merged as base) — UI works end-to-end on prod ## Backend schema compat check (vs main HEAD) - `POST /api/v1/admin/scrape` accepts `{lat, lon, radius_m, sources: ["cian"], multi_room_cian: bool}` — UI body matches exactly - `GET /api/v1/admin/scraper-settings` returns sorted `list[ScraperSetting]` — UI shape matches - `PUT /api/v1/admin/scraper-settings/{source}` UPSERT, `source` is `str` (no enum), accepts `global` — works - `GET /api/v1/admin/scrape/cian/test-auth` returns `{authenticated, userId, reason}` — matches `TestAuthResp` interface - Response types match (TS strict, no `any`) - `apiFetch` -> basePath=`/trade-in` -> Caddy `handle_path /trade-in/api/*` -> tradein-backend (correct) ## Medium follow-ups (NON-blocking) ### 1. Slider min=0 vs backend ge=1.0 mismatch **Backend** `ScraperSettingUpdate.request_delay_sec: float = Field(ge=1.0, le=60.0)`. **UI** slider/input `min={0}` `max={30}` `step={0.5}` + hint says "0 = только per-source defaults". Edge case: user moves slider to 0 or 0.5 -> PUT body `{request_delay_sec: 0}` -> backend returns 422. UI renders `updateSettingMut.error` so no crash, but UX misleading (hint promises 0 works, backend rejects). Recommend follow-up: - Either bump slider `min={1}` and update hint, OR - Update backend Pydantic to `ge=0.0` (matches UI semantics) ### 2. "Global" delay not consumed by scrapers `tradein-mvp/backend/app/services/scraper_settings.py::get_scraper_delay()` only reads the **specific source key** (with `_KEY_ALIASES` mapping). There is no `max(per_source, global)` logic. UI hint claims: "Глобальный делэй применяется ко всем scrapers (max с per-source defaults). Изменение вступает в силу немедленно". Actual behavior: PUT succeeds (row stored), but `get_scraper_delay("cian")` reads `cian` row only, never `global`. Cosmetic write. Recommend follow-up: extend `get_scraper_delay()` to read both keys and return `max(source_delay, global_delay)`. ## Minor - `parseFloat(e.target.value)` on number/range inputs -> NaN on empty input -> serialized as `null` -> backend 422. Graceful (no crash). Could guard with `Number.isFinite` before mutate. - Vault entry `code/modules/tradein/Cian_Admin_Page.md` not yet present — recommend creating post-deploy. ## Verify steps post-deploy 1. deploy-tradein.yml triggers (paths include `tradein-mvp/frontend/**`) 2. `https://gendsgn.ru/trade-in/scrapers/cian` -> 6 cards render, Topbar `cian` tab active 3. Cookies status badge -> green/red per `/admin/scrape/cian/test-auth` 4. Scrape Around POST -> 200 + counters returned 5. Global delay slider value 1..30 -> PUT 200 + row UPSERTed; value 0 -> 422 (expected per current backend)

lekss361 referenced this pull request

2026-05-23 15:41:29 +00:00

fix(tradein/avito): anti-bot hardening — sleep + abort + shared session #487

lekss361 referenced this pull request from a commit

2026-05-23 15:43:30 +00:00

feat(tradein): global scraper delay setting (applies across all scrapers) (#485)

lekss361 referenced this pull request

2026-05-23 15:43:54 +00:00

feat(tradein): global scraper delay setting (applies across all scrapers) #485

lekss361 referenced this pull request

2026-05-23 15:47:56 +00:00

feat(tradein-ui): Yandex admin scraper page — mirror Avito + global delay control #488