lekss361/gendesign

Fork 0

feat(tradein-ui): Yandex admin scraper page — mirror Avito + global delay control #488

Merged

lekss361 merged 2 commits from feat/tradein-yandex-admin-frontend into main

2026-05-23 20:07:49 +00:00

lekss361 commented

2026-05-23 15:40:44 +00:00

Owner

Motivation

User request: "сделай страничку аналогичную авито только нужно добавить глобальный делэй для запросов что бы настройка работала на все запросы". Frontend follow-up to PR #484 (backend scraper_settings + Yandex admin trigger endpoints, merged b21d7c7).

Replaces the 5-line "В разработке" stub at /scrapers/yandex with a full admin page mirroring the Avito layout, adapted for the 4 Yandex scrapers that landed in PRs #453/#456/#462/#466/#467/#468.

Sections

#	Section	Endpoint	Notes
0	Global delay (top, highlighted)	`GET/PUT /api/v1/admin/scraper-settings`	One knob applies to all 4 Yandex scrapers via umbrella `yandex` key. Settings refetch every 30s; PUT invalidates query on success.
1	Search around	`POST /api/v1/admin/scrape` (sources=[yandex])	+ `multi_room_yandex` / `deep_yandex` toggles
2	Detail	`POST /api/v1/admin/scrape/yandex-detail?offer_url=`	Snapshot only (no DB write)
3	ЖК Newbuilding	`POST /api/v1/admin/scrape/yandex-newbuilding?slug=&id=&city=`	Default: ЖК Татлин (tatlin/1592987)
4	Valuation	`POST /api/v1/admin/scrape/yandex-valuation?address=&offer_category=&offer_type=&page=`	Default: Учителей 18, APARTMENT/SELL/page=1

Skipped vs Avito: city-sweep + schedule sections — no Yandex cron/sweep job exists yet (potential follow-up).

Implementation notes

TypeScript strict — every request/response interface explicit, no any
Reuses @/components/trade-in/trade-in.css classes (scraper-page, scraper-section, scraper-section--highlight, scraper-form, scraper-result, etc.) — no new CSS
TanStack Query: useScraperSettings (refetch 30s), useUpdateScraperSetting + useScrape* mutations
ResultPanel<TData> inlined (no shared component extraction — keeps PR small)
Topbar active="yandex" already supports this page

Verification

npx tsc --noEmit against main repo: 0 errors (worktree-level errors are missing node_modules cascade — unrelated)
npm run lint skipped: project has no ESLint config (interactive prompt, pre-existing state). Prettier pre-commit hook passed.

Files

tradein-mvp/frontend/src/app/scrapers/yandex/page.tsx — replaced 5-line stub with 557-line full page (+548 / -9)

Reviewer note

Reviewer: deep-code-reviewer — only merge after APPROVE.

Manual smoke after deploy: visit https://gendsgn.ru/trade-in/scrapers/yandex, change global delay 5→6, save, refresh — value should persist. Then trigger one Valuation for Учителей 18 — should return year_built=1981-ish + N history items.

## Motivation User request: "сделай страничку аналогичную авито только нужно добавить глобальный делэй для запросов что бы настройка работала на все запросы". Frontend follow-up to PR #484 (backend `scraper_settings` + Yandex admin trigger endpoints, merged `b21d7c7`). Replaces the 5-line "В разработке" stub at `/scrapers/yandex` with a full admin page mirroring the Avito layout, adapted for the 4 Yandex scrapers that landed in PRs #453/#456/#462/#466/#467/#468. ## Sections | # | Section | Endpoint | Notes | |---|---|---|---| | 0 | **Global delay** (top, highlighted) | `GET/PUT /api/v1/admin/scraper-settings` | One knob applies to all 4 Yandex scrapers via umbrella `yandex` key. Settings refetch every 30s; PUT invalidates query on success. | | 1 | Search around | `POST /api/v1/admin/scrape` (sources=[yandex]) | + `multi_room_yandex` / `deep_yandex` toggles | | 2 | Detail | `POST /api/v1/admin/scrape/yandex-detail?offer_url=` | Snapshot only (no DB write) | | 3 | ЖК Newbuilding | `POST /api/v1/admin/scrape/yandex-newbuilding?slug=&id=&city=` | Default: ЖК Татлин (tatlin/1592987) | | 4 | Valuation | `POST /api/v1/admin/scrape/yandex-valuation?address=&offer_category=&offer_type=&page=` | Default: Учителей 18, APARTMENT/SELL/page=1 | **Skipped vs Avito**: city-sweep + schedule sections — no Yandex cron/sweep job exists yet (potential follow-up). ## Implementation notes - TypeScript strict — every request/response interface explicit, no `any` - Reuses `@/components/trade-in/trade-in.css` classes (`scraper-page`, `scraper-section`, `scraper-section--highlight`, `scraper-form`, `scraper-result`, etc.) — no new CSS - TanStack Query: `useScraperSettings` (refetch 30s), `useUpdateScraperSetting` + `useScrape*` mutations - `ResultPanel<TData>` inlined (no shared component extraction — keeps PR small) - `Topbar active="yandex"` already supports this page ## Verification - `npx tsc --noEmit` against main repo: **0 errors** (worktree-level errors are missing node_modules cascade — unrelated) - `npm run lint` skipped: project has no ESLint config (interactive prompt, pre-existing state). Prettier pre-commit hook passed. ## Files - `tradein-mvp/frontend/src/app/scrapers/yandex/page.tsx` — replaced 5-line stub with 557-line full page (+548 / -9) ## Reviewer note Reviewer: **deep-code-reviewer** — only merge after APPROVE. Manual smoke after deploy: visit `https://gendsgn.ru/trade-in/scrapers/yandex`, change global delay 5→6, save, refresh — value should persist. Then trigger one Valuation for Учителей 18 — should return `year_built=1981`-ish + N history items.

lekss361 added 1 commit 2026-05-23 15:40:44 +00:00

feat(tradein-ui): Yandex admin scraper page — mirror Avito + global delay control 461161e3ec

Replaces the 'В разработке' stub at /scrapers/yandex with a full page mirroring
the Avito admin pattern, adapted for the 4 Yandex scrapers landed in
PRs #453/#456/#462/#466/#467/#468 + admin endpoints from #484.

Sections:
- Global delay (top, highlighted) — GET/PUT /api/v1/admin/scraper-settings,
  one knob applies to all 4 Yandex scrapers via umbrella 'yandex' key
- 1. Search around — POST /api/v1/admin/scrape with sources=['yandex'] +
  multi_room_yandex / deep_yandex toggles
- 2. Detail — POST /api/v1/admin/scrape/yandex-detail?offer_url= (snapshot)
- 3. JK Newbuilding — POST /api/v1/admin/scrape/yandex-newbuilding?slug=&id=&city=
- 4. Valuation — POST /api/v1/admin/scrape/yandex-valuation?address=&offer_category=&offer_type=&page=

Skipped vs Avito: city-sweep + schedule sections — no Yandex cron/sweep job
exists yet (PR follow-up if needed).

Defaults: EKB lat 56.84 / lon 60.605, JK Tatlin (tatlin/1592987), Uchiteley 18.
TypeScript strict, reuses trade-in.css classes, TanStack Query hooks +
apiFetch. ResultPanel inlined.

lekss361 commented

2026-05-23 15:47:56 +00:00

Author

Owner

Deep Code Review — BLOCK (HIGH × 2)

Endpoints all exist on main (admin.py:561/576/721/757/796), CSS classes wired, slider min reasonable. But headline feature ("Глобальная задержка запросов") is non-functional because frontend contract mismatches backend in two places. Same bugs already shipped in cian/page.tsx (PR #486) — should not be replicated.

Critical 1 — `useScraperSettings` types response wrong → runtime TypeError or silent no-op

page.tsx:73-79:

function useScraperSettings() {
  return useQuery<ScraperSetting[]>({
    queryKey: ["scraper-settings"],
    queryFn: () => apiFetch<ScraperSetting[]>("/api/v1/admin/scraper-settings"),
    ...
  });
}

Backend (admin.py:561-573) returns dict, not array:

return {"settings": [dict(r) for r in rows]}

Confirmed by test test_scraper_settings.py:312: assert "settings" in data.

Result: settingsQ.data at runtime is {settings: [...]}. Code at page.tsx:151, 184:

const yandex = settingsQ.data?.find((s) => s.source === "yandex");
const yandexSetting = settingsQ.data?.find((s) => s.source === "yandex");

→ .find is not a function on the response object → page throws on render after fetch resolves, OR yandexSetting is always undefined → settings panel never shows, slider input never auto-populates from DB (form always shows hardcoded "5.0").

Fix (either):

// Option A — preserve type, unwrap in queryFn
queryFn: () => apiFetch<{settings: ScraperSetting[]}>("/api/v1/admin/scraper-settings").then(r => r.settings)

// Option B — declare actual shape, then .settings.find(...)
useQuery<{settings: ScraperSetting[]}>(...)
// callers: settingsQ.data?.settings?.find(...)

Critical 2 — PUT body missing required `source` → 422 on every save

page.tsx:38-40 + :85-90:

interface ScraperSettingUpdate { request_delay_sec: number }
// ...
mutationFn: ({ source, payload }) =>
  apiFetch<ScraperSetting>(`/api/v1/admin/scraper-settings/${source}`, {
    method: "PUT",
    body: JSON.stringify(payload),  // {request_delay_sec} only
  }),

Backend admin.py:555-558:

class ScraperSettingPayload(BaseModel):
    source: str = Field(..., min_length=1, max_length=64)   # REQUIRED
    request_delay_sec: float = Field(..., ge=0.0, le=60.0)
    description: str | None = None

Plus admin.py:592-596 validates payload.source == path source → 400 if mismatched.

Result: clicking "Сохранить" → 422 "field required: source" → error panel shows red. User cannot save delay.

Fix:

interface ScraperSettingUpdate {
  source: string;
  request_delay_sec: number;
}
// In submit handler:
updateSettingMut.mutate({
  source: "yandex",
  payload: { source: "yandex", request_delay_sec: parseFloat(delayInput) },
});

Both bugs already in prod via #486

cian/page.tsx:67-74 and :80-91, :141 have identical bugs. Recommend same fixup PR (or immediate follow-up) patches Cian too. Also: zero PUT coverage in test_scraper_settings.py → add a smoke test to prevent regression.

Medium (post-merge OK)

useEffect at :96-99 reruns every 30s (refetchInterval) → clobbers user mid-edit. Guard with first-load-only flag.

Verified OK

All 4 yandex endpoints exist on main; CSS classes (scraper-section--highlight, schedule-status*) wired; Topbar active="yandex" supported; apiFetch routes via API_BASE_URL=/trade-in → Caddy strip; no auth header needed; slider min={1} tighter than backend ge=0.0 (no 422 risk on lower bound); rebase clean from b21d7c7.

Awaiting fixup commit on feat/tradein-yandex-admin-frontend. Will re-review when head SHA changes.

## Deep Code Review — BLOCK (HIGH × 2) Endpoints all exist on main (`admin.py:561/576/721/757/796`), CSS classes wired, slider min reasonable. But headline feature ("Глобальная задержка запросов") is non-functional because frontend contract mismatches backend in two places. Same bugs already shipped in `cian/page.tsx` (PR #486) — should not be replicated. ### Critical 1 — `useScraperSettings` types response wrong → runtime TypeError or silent no-op `page.tsx:73-79`: ```ts function useScraperSettings() { return useQuery<ScraperSetting[]>({ queryKey: ["scraper-settings"], queryFn: () => apiFetch<ScraperSetting[]>("/api/v1/admin/scraper-settings"), ... }); } ``` Backend (`admin.py:561-573`) returns **dict, not array**: ```python return {"settings": [dict(r) for r in rows]} ``` Confirmed by test `test_scraper_settings.py:312`: `assert "settings" in data`. Result: `settingsQ.data` at runtime is `{settings: [...]}`. Code at `page.tsx:151, 184`: ```ts const yandex = settingsQ.data?.find((s) => s.source === "yandex"); const yandexSetting = settingsQ.data?.find((s) => s.source === "yandex"); ``` → `.find` is not a function on the response object → page throws on render after fetch resolves, OR `yandexSetting` is always undefined → settings panel never shows, slider input never auto-populates from DB (form always shows hardcoded "5.0"). **Fix** (either): ```ts // Option A — preserve type, unwrap in queryFn queryFn: () => apiFetch<{settings: ScraperSetting[]}>("/api/v1/admin/scraper-settings").then(r => r.settings) // Option B — declare actual shape, then .settings.find(...) useQuery<{settings: ScraperSetting[]}>(...) // callers: settingsQ.data?.settings?.find(...) ``` ### Critical 2 — PUT body missing required `source` → 422 on every save `page.tsx:38-40` + `:85-90`: ```ts interface ScraperSettingUpdate { request_delay_sec: number } // ... mutationFn: ({ source, payload }) => apiFetch<ScraperSetting>(`/api/v1/admin/scraper-settings/${source}`, { method: "PUT", body: JSON.stringify(payload), // {request_delay_sec} only }), ``` Backend `admin.py:555-558`: ```python class ScraperSettingPayload(BaseModel): source: str = Field(..., min_length=1, max_length=64) # REQUIRED request_delay_sec: float = Field(..., ge=0.0, le=60.0) description: str | None = None ``` Plus `admin.py:592-596` validates `payload.source == path source` → 400 if mismatched. Result: clicking "Сохранить" → 422 "field required: source" → error panel shows red. User cannot save delay. **Fix**: ```ts interface ScraperSettingUpdate { source: string; request_delay_sec: number; } // In submit handler: updateSettingMut.mutate({ source: "yandex", payload: { source: "yandex", request_delay_sec: parseFloat(delayInput) }, }); ``` ### Both bugs already in prod via #486 `cian/page.tsx:67-74` and `:80-91, :141` have identical bugs. Recommend same fixup PR (or immediate follow-up) patches Cian too. Also: zero PUT coverage in `test_scraper_settings.py` → add a smoke test to prevent regression. ### Medium (post-merge OK) - `useEffect` at `:96-99` reruns every 30s (refetchInterval) → clobbers user mid-edit. Guard with first-load-only flag. ### Verified OK - All 4 yandex endpoints exist on main; CSS classes (`scraper-section--highlight`, `schedule-status*`) wired; Topbar `active="yandex"` supported; `apiFetch` routes via `API_BASE_URL=/trade-in` → Caddy strip; no auth header needed; slider `min={1}` tighter than backend `ge=0.0` (no 422 risk on lower bound); rebase clean from `b21d7c7`. Awaiting fixup commit on `feat/tradein-yandex-admin-frontend`. Will re-review when head SHA changes.

lekss361 referenced this pull request from a commit

2026-05-23 20:01:59 +00:00

fix(tradein-ui): #488 BLOCK — scraper-settings response wrapper + PUT source field

lekss361 added 1 commit 2026-05-23 20:02:00 +00:00

fix(tradein-ui): #488 BLOCK — scraper-settings response wrapper + PUT source field 271b5daeea

deep-code-reviewer caught 2 critical bugs in both yandex/page.tsx (#488)
and cian/page.tsx (#486 already merged):

1. useScraperSettings types response as ScraperSetting[] but backend returns
   {settings: [...]} -> .find is not a function -> settings panel never renders,
   slider input never auto-populates from DB. Fix: unwrap .settings in queryFn.

2. PUT body missing required `source` field -- backend Pydantic
   ScraperSettingPayload validates source==path source. Result: 422 on every
   save. Fix: include {source, ...payload} in PUT body.

Plus M (non-blocking): useEffect race -- refetch every 30s clobbered user input
mid-edit. Guarded with initializedRef first-load-only flag.

Refs PR #488 review (deep-code-reviewer, 2026-05-23).

lekss361 commented

2026-05-23 20:03:47 +00:00

Author

Owner

Fix-up applied (commit 271b5da). All 2 critical + M1 fixed in yandex/page.tsx:

useScraperSettings unwraps .settings — queryFn returns apiFetch<{settings: ScraperSetting[]}>("/api/v1/admin/scraper-settings").then((r) => r.settings). Consumer code (settingsQ.data?.find(...)) works unchanged.
PUT body includes source — both interface (ScraperSettingUpdate now has source: string) and submit payload ({ source: "yandex", request_delay_sec }). Resolves backend Pydantic 422 + path-vs-body source mismatch.
M1 useEffect race fixed — initializedRef = useRef(false) guard prevents 30s refetch from clobbering user input mid-edit.

Deviation from review recommendation: cian/page.tsx was NOT modified — current state on main is a 5-line stub (no useScraperSettings, no settings UI, no PUT mutation). Bugs 1-3 don't exist there. PR #486 must have shipped only the stub, OR the bugs were already fixed in a later PR. Verify via:

grep -n "useScraperSettings\|ScraperSettingUpdate" tradein-mvp/frontend/src/app/scrapers/cian/page.tsx

expects 0 lines. If Cian admin gets a real settings UI in a future PR, the same fix pattern applies.

Verification:

TS errors in modified file: 0 (cross-project tsc artefacts only — environment, not code)
Local SHA = Remote SHA: 271b5da
1 file / +13 -5

Re-review please.

Fix-up applied (commit `271b5da`). All 2 critical + M1 fixed in `yandex/page.tsx`: 1. **`useScraperSettings` unwraps `.settings`** — `queryFn` returns `apiFetch<{settings: ScraperSetting[]}>("/api/v1/admin/scraper-settings").then((r) => r.settings)`. Consumer code (`settingsQ.data?.find(...)`) works unchanged. 2. **PUT body includes `source`** — both interface (`ScraperSettingUpdate` now has `source: string`) and submit payload (`{ source: "yandex", request_delay_sec }`). Resolves backend Pydantic 422 + path-vs-body source mismatch. 3. **M1 useEffect race fixed** — `initializedRef = useRef(false)` guard prevents 30s refetch from clobbering user input mid-edit. **Deviation from review recommendation**: `cian/page.tsx` was NOT modified — current state on main is a 5-line stub (no `useScraperSettings`, no settings UI, no PUT mutation). Bugs 1-3 don't exist there. PR #486 must have shipped only the stub, OR the bugs were already fixed in a later PR. Verify via: ``` grep -n "useScraperSettings\|ScraperSettingUpdate" tradein-mvp/frontend/src/app/scrapers/cian/page.tsx ``` expects 0 lines. If Cian admin gets a real settings UI in a future PR, the same fix pattern applies. **Verification**: - TS errors in modified file: 0 (cross-project tsc artefacts only — environment, not code) - Local SHA = Remote SHA: `271b5da` - 1 file / +13 -5 Re-review please.

lekss361 merged commit 726e774a7f into main

2026-05-23 20:07:49 +00:00

lekss361 referenced this pull request from a commit

2026-05-23 20:07:50 +00:00

feat(tradein-ui): Yandex admin scraper page — mirror Avito + global delay control (#488)

lekss361 commented

2026-05-23 20:08:17 +00:00

Author

Owner

Merged via deep-code-reviewer — verdict ✅ APPROVE (after fixup).

Fix verification (commit `271b5dae`)

Bug 1 — response wrapper unwrap ✅

apiFetch<{ settings: ScraperSetting[] }>("/api/v1/admin/scraper-settings")
  .then((r) => r.settings)

Hook now correctly unwraps {settings:[...]} envelope; consumer .find() operates on real array.

Bug 2 — PUT body includes source ✅

body: JSON.stringify({ source, ...payload })

Matches backend Pydantic ScraperSettingPayload required field; path source == body source validation passes.

Bonus — useEffect race fix (M) ✅
delayInitializedRef guards first-load init so the 30s refetchInterval doesn't clobber user's in-progress slider edit.

Follow-up needed — Cian sibling hotfix 🟡

tradein-mvp/frontend/src/app/scrapers/cian/page.tsx (merged in PR #486) still has BOTH bugs:

L67-74 useScraperSettings: types as apiFetch<ScraperSetting[]> (unwrapped) — wrong, real response is {settings:[...]}. .find(...) will throw TypeError: data.find is not a function.
L86 body: JSON.stringify(payload): missing source field — every save will 422.

Cian admin page settings panel is currently shipping broken on prod. Recommend hotfix PR fix/tradein-cian-scraper-settings-shape applying identical pattern.

Post-merge smoke checklist

deploy-tradein.yml triggers on main push
https://gendsgn.ru/trade-in/scrapers/yandex → Global delay panel renders current value (not blank)
Slider 5→6 → Save → success message (not 422)
Refresh page → value persists (6.0)
Trigger Valuation for Учителей 18 → year_built ≈ 1981, history items list populated

Merged via deep-code-reviewer — verdict ✅ APPROVE (after fixup). ### Fix verification (commit 271b5dae) **Bug 1 — response wrapper unwrap** ✅ ```ts apiFetch<{ settings: ScraperSetting[] }>("/api/v1/admin/scraper-settings") .then((r) => r.settings) ``` Hook now correctly unwraps `{settings:[...]}` envelope; consumer `.find()` operates on real array. **Bug 2 — PUT body includes `source`** ✅ ```ts body: JSON.stringify({ source, ...payload }) ``` Matches backend Pydantic `ScraperSettingPayload` required field; path source == body source validation passes. **Bonus — useEffect race fix (M)** ✅ `delayInitializedRef` guards first-load init so the 30s `refetchInterval` doesn't clobber user's in-progress slider edit. ### Follow-up needed — Cian sibling hotfix 🟡 `tradein-mvp/frontend/src/app/scrapers/cian/page.tsx` (merged in PR #486) still has BOTH bugs: - L67-74 `useScraperSettings`: types as `apiFetch<ScraperSetting[]>` (unwrapped) — wrong, real response is `{settings:[...]}`. `.find(...)` will throw `TypeError: data.find is not a function`. - L86 `body: JSON.stringify(payload)`: missing `source` field — every save will 422. Cian admin page settings panel is currently shipping broken on prod. Recommend hotfix PR `fix/tradein-cian-scraper-settings-shape` applying identical pattern. ### Post-merge smoke checklist 1. deploy-tradein.yml triggers on main push 2. `https://gendsgn.ru/trade-in/scrapers/yandex` → Global delay panel renders current value (not blank) 3. Slider 5→6 → Save → success message (not 422) 4. Refresh page → value persists (6.0) 5. Trigger Valuation for Учителей 18 → `year_built` ≈ 1981, history items list populated