Commit graph

20 commits

Author SHA1 Message Date
a1ee06ef17 feat(tradein/yandex): normalize house_type + promote kitchen/ceiling to columns (#2007)
house_type-нормализация = реальный фикс estimator soft-penalty: yandex SERP
писал building.buildingType как есть (SCREAMING — MONOLIT/BRICK/PANEL/...),
а estimator штрафует аналог с house_type != target. 'MONOLIT' никогда не равен
каноничному 'monolith' → ~70% yandex-аналогов получали ложный штраф и фактически
выпадали из скоринга. Новый модуль house_type_normalizer.normalize_house_type
приводит yandex SCREAMING и cian camelCase (reuse в #2008) к канону
{panel, brick, monolith, monolith_brick, block, wood}; unknown/'other'/'' → None
(NULL нейтрально для estimator, не штрафуется). Backfill существующих yandex-строк
миграцией 140 (идемпотентный UPDATE под тот же маппинг).

kitchen_area_m2 / ceiling_height_m promote из raw_payload в колонки — для
coverage-дашборда и matching, НЕ valuation-фича (estimator их пока не читает —
отдельный follow-up). ceiling пишется в ОБЕ колонки (ceiling_height 019 — её
читает coverage-дашборд admin.py + yandex_detail/cian_detail; ceiling_height_m
111 — живая avito-колонка) через COALESCE в ON CONFLICT, чтобы SERP re-scrape
без полей не затирал detail-enrichment (avito поведение сохранено).
2026-06-27 20:30:27 +03:00
197068c515 fix(scrapers): save_listings — UPDATE by (source,source_id) при dedup_hash-дрейфе (avito_full_load краш) 2026-06-26 19:10:12 +03:00
1d5b081cb4 fix(cian): recover new-build house via cadastral anchor + listings (source,source_id) dedup (#1773) (#1912)
Some checks failed
Deploy Trade-In / changes (push) Successful in 13s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 1m28s
Deploy Trade-In / deploy (push) Failing after 46s
Deploy Trade-In / build-backend (push) Successful in 50s
cian new-build cadastral anchor + listings (source,source_id) dedup (#1773): migration 133 (deep-reviewed data-loss-safe). Refs #1773
2026-06-26 07:45:22 +00:00
533299ea47 feat(tradein/scrapers): full_load пропускает листинги обновлённые сегодня при повторном прогоне (skip_seen_today)
All checks were successful
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
2026-06-20 12:13:32 +03:00
8436532bf4 feat(listings): card_hash column + snapshot-dedup on unchanged cards
All checks were successful
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
Adds listings.card_hash (SHA-256 of volatile SERP-card fields) computed and
stored on every scrape. Skips the redundant per-day listings_snapshots write
when a listing's card is unchanged. listings.last_seen_at/is_active still
bumped on every sighting (delisting TTL). Lays groundwork for detail-refresh
gating (C2).
2026-06-18 22:01:54 +03:00
07eed4b55e Merge pull request 'feat(avito): capture newbuilding ЖК id/url from SERP card' (#1729) from feat/avito-newbuilding-capture into main
Some checks failed
Deploy Trade-In / changes (push) Has been cancelled
Deploy Trade-In / test (push) Blocked by required conditions
Deploy Trade-In / build-backend (push) Blocked by required conditions
Deploy Trade-In / build-frontend (push) Blocked by required conditions
Deploy Trade-In / build-browser (push) Blocked by required conditions
Deploy Trade-In / deploy (push) Blocked by required conditions
2026-06-17 20:29:39 +00:00
dd59ef0516 feat(avito): capture newbuilding ЖК id/url from SERP card; persist newbuilding_id/url
All checks were successful
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
2026-06-17 23:15:35 +03:00
e63b21eccb fix(scrapers): write resolved house_id to listings.house_id_fk in realtime link hook
All checks were successful
CI / changes (pull_request) Successful in 7s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
2026-06-17 23:07:22 +03:00
1fe670ea6a feat(yandex): map rich entity fields (predictedPrice, seller, listing_date, trend, metro)
All checks were successful
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
2026-06-17 22:58:16 +03:00
42de30cd5c refactor(tradein): unify kadastr_num -> cadastral_number (#732)
p3 naming-consistency (cadastral cols 0% filled). Migration 094_cadastral_unify.sql
(idempotent DO-block IF EXISTS): drop+recreate listings_search_mv (MV) +
v_data_quality (view), RENAME deals/house_metadata kadastr_num->cadastral_number,
DROP listings.kadastr_num + listings_kadastr_idx. Code SQL-string renames in
search_query/estimator/base (Lot dataclass field NOT renamed). 240 tests pass.

WARN: destructive DDL auto-applies to prod on deploy -> MERGE-GATE post-demo.
Reviewer: verify recreated MV/view DDL vs LIVE pg_get_viewdef before apply
(reconstructed from migration history 050/046, not live capture).

Refs #732
2026-05-31 15:41:52 +03:00
65b9247369 fix(matching): Tier 2b alias sync + collapse double SAVEPOINT (#849) (#870)
All checks were successful
Deploy Trade-In / test (push) Successful in 30s
Deploy Trade-In / deploy (push) Successful in 46s
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-backend (push) Successful in 40s
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-05-31 08:05:48 +00:00
2739973285 fix(scrapers): COALESCE-backfill listing_date/area_m2 in save_listings ON CONFLICT (Refs #821) (#829)
All checks were successful
Deploy Trade-In / changes (push) Successful in 6s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Successful in 25s
Deploy Trade-In / build-backend (push) Successful in 40s
Deploy Trade-In / deploy (push) Successful in 36s
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-05-30 19:58:06 +00:00
b736d7b7e0 feat(estimator): exclude city-centroid listings from radius analogs (Refs #769 Part E)
Закрывает последнюю часть #769 (A1/A2/B/C/D в #798/#804). Finding #17:
bare-city адреса геокодились в city-центроид и сохранялись как точные lat/lon →
mislocated листинги участвовали в radius ST_DWithin-аналогах.

- 089: ALTER TABLE listings ADD COLUMN IF NOT EXISTS geo_precision text (idempot).
- ScrapedLot.geo_precision + проброс в save_listings.
- geocode_missing.py: geo_precision='city' через существующий _geocode_is_coarse.
- estimator _fetch_analogs Tier H+W: + AND (geo_precision IS DISTINCT FROM 'city').
  NULL проходит (консервативно, сдвига до backfill нет). Pricing не тронут.

7 тестов; 270 passed, ruff clean.
2026-05-30 21:22:21 +03:00
79a79004cf fix(scrapers): stable dedup_hash — drop price_rub + strip URL query (Refs #753) (#761)
All checks were successful
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Successful in 25s
Deploy Trade-In / build-backend (push) Successful in 40s
Deploy Trade-In / deploy (push) Successful in 34s
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-05-30 16:10:27 +00:00
ec26e5813f fix(tradein): implement listings_snapshots + offer_price_history writers (#567) (#624)
Some checks are pending
Deploy Trade-In / deploy (push) Successful in 43s
Deploy Trade-In / build-backend (push) Successful in 1m2s
Deploy / changes (push) Waiting to run
Deploy / build-backend (push) Blocked by required conditions
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy / build-worker (push) Blocked by required conditions
Deploy / build-frontend (push) Blocked by required conditions
Deploy / deploy (push) Blocked by required conditions
2026-05-28 13:49:52 +00:00
7e24ccbf2c feat(tradein): hook matching в save_listings — fill listing_sources (PR I) (#595)
All checks were successful
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-backend (push) Successful in 52s
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / deploy (push) Successful in 40s
2026-05-27 10:02:37 +00:00
1b3edff6a9 feat(tradein): extend base.py save_listings to persist Cian-specific columns (PR #450 follow-up) (#455)
All checks were successful
Deploy Trade-In / changes (push) Successful in 4s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / deploy (push) Successful in 20s
Deploy Trade-In / build-backend (push) Successful in 39s
2026-05-23 13:05:20 +00:00
9ba776f2ca feat(tradein): cian_state_parser shared utility + ScrapedLot Cian fields (#447)
Some checks failed
Deploy Trade-In / build-backend (push) Blocked by required conditions
Deploy Trade-In / build-frontend (push) Blocked by required conditions
Deploy Trade-In / deploy (push) Blocked by required conditions
Deploy Trade-In / changes (push) Has been cancelled
2026-05-23 12:35:01 +00:00
47f4cc7d58 refactor(tradein): avito search — remove jitter (C-5) + mobile API dead code + parse houseLinks (#441)
All checks were successful
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-backend (push) Successful in 39s
Deploy Trade-In / deploy (push) Successful in 18s
2026-05-23 11:55:18 +00:00
lekss361
02267d0306 feat: add tradein-mvp subproject (Trade-In Estimator под /trade-in)
- FastAPI backend: PostGIS estimator + 3 scrapers (Avito/Cian/Yandex)
- Next.js 15 frontend: tradein.html mockup design, basePath=/trade-in
- WeasyPrint PDF (Брусника-style 4-page report)
- Address autocomplete с typo-tolerance + 6 EKB presets
- Изолированный docker stack gendesign-tradein (отдельная postgres БД)
- Caddy inline routes: gendsgn.ru/trade-in/* и /trade-in/api/v1/*
- Forgejo Actions: .forgejo/workflows/deploy-tradein.yml (shell-based GHCR login)
- Триггер только по paths: tradein-mvp/** (не пересекается с deploy.yml)
- Образы: ghcr.io/lekss361/gendesign-tradein-{backend,frontend}:latest

Первый запуск на сервере (вручную, один раз):
  - создать /opt/gendesign/tradein-mvp/.env.runtime (postgres pwd, contact email)
  - docker network create gendesign_shared (если нет)
  - docker compose -p gendesign-tradein up -d
  - docker compose -p gendesign exec caddy caddy reload
2026-05-21 00:25:39 +03:00