feat(tradein): search matview + indexes (Phase 3.1) #469
No reviewers
Labels
No labels
admin
analytics
auth
automation
bug
business
chore
ci
compliance
data
data-moat
docs
duplicate
dx
enhancement
Fable 5 ревью
feedback/max
generative
GG-форсайт
needs-discussion
needs-human
observability
pause-bots
performance
priority/p0
priority/p1
priority/p2
priority/p3
scope/backend
scope/db
scope/devops
scope/frontend
scope/qa
scrapers
security
site-finder
stage/1
stage/2
status/blocked
status/done
status/needs-analysis
status/needs-fix
status/qa
status/ready
status/review
status/wip
tech-debt
tradein
ux
week ревью 1
wontfix
вторичка
ИРД
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference: lekss361/gendesign#469
Loading…
Add table
Reference in a new issue
No description provided.
Delete branch "feat/tradein-search-matview"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Motivation
Phase 3.1 of Multi-Source Integration. Foundation for
/api/v1/searchp95 < 100ms on 100K+ listings + Celery daily refresh.Files
tradein-mvp/backend/data/sql/050_search_optimization.sqltradein-mvp/backend/app/tasks/__init__.pytradein-mvp/backend/app/tasks/refresh_search_matview.pyTotal: 3 files / +260 lines / -0.
What's inside 050
CREATE EXTENSION IF NOT EXISTS pg_trgmALTER TABLE listings ADD COLUMN IF NOT EXISTS tsv tsvector GENERATED ALWAYS AS (...) STOREDlistings:CREATE MATERIALIZED VIEW listings_search_mv AS SELECT ...— per-listing cross-source aggregationlisting_idUNIQUE, geom GIST, sources presence, etc.)Note on placeholder columns:
district,distance_to_metro_m,last_price_change,photos_countrendered asNULL::type— these columns absent from currentlistingsschema; backfill in future PR viacad_buildings/ OSM POI joins or scraper enrichment.What's inside Celery task
tradein.refresh_search_matview—REFRESH MATERIALIZED VIEW CONCURRENTLY listings_search_mvlogger.info(no print), httpx (not requests) — none of these dependencies actually used here, but follows repo conventionscrontab(hour=3, minute=0)) — comment block in docstring, not wired toapp/celery_app.pyin this PR (tradein-mvp doesn't havecelery_app.pyyet — coordination follow-up)Coordination
grep -c "CREATE TABLE" = 0)IF NOT EXISTS, matview wrappedCREATE MATERIALIZED VIEW IF NOT EXISTSper spec)Smoke verification (post-merge on prod)
Initial REFRESH may be skipped if listings table empty — Celery task will populate on first daily run (3 AM Moscow).
Reviewer
deep-code-reviewer — please verify:
russianconfig + correct weightspsycopg2import, only psycopg v3Refs
decisions/MultiSource_Integration_Master_Plan.mdMulti-Source Integration Phase 3.1. Foundation for /api/v1/search p95 < 100ms on 100K+ listings. SQL 050_search_optimization.sql: - CREATE EXTENSION IF NOT EXISTS pg_trgm - ALTER listings ADD COLUMN tsv tsvector GENERATED ALWAYS AS (...) STORED - 10 indexes (geom GIST, composite, gin_trgm, GIN tsv, partial kadastr, ...) - CREATE MATERIALIZED VIEW listings_search_mv (per-listing aggregated across sources) - 5 matview indexes Celery task tradein.refresh_search_matview: - REFRESH MATERIALIZED VIEW CONCURRENTLY listings_search_mv - Logged duration via logger.info - Beat schedule snippet (crontab hour=3, minute=0) in docstring — wiring to app/celery_app.py left to follow-up PR per main-session coordination. Refs Master Plan sec 5.1 + 6.5 + 10.1.Deep Code Review — verdict: BLOCK
Two correctness bugs verified by schema audit against
tradein-mvp/backend/data/sql/0*.sql. Both will fire on the firstpsql -v ON_ERROR_STOP=ondeploy run.CRITICAL #1 —
houses.ratings_countdoes not exist050_search_optimization.sqlline ~155:But
housesschema (009/010/020/031/040) has only:reviews_count int(009_houses.sql:44)rating_score numeric(4,2)(010_houses_alter.sql:43)rating_string text,rating_distribution jsonb(010)ratings_countappears only in the vault plan (decisions/MultiSource_Integration_Master_Plan.mdsec 6.5), never in actual DDL. Result:CREATE MATERIALIZED VIEWaborts withERROR: column h.ratings_count does not exist.Deploy-tradein.yml suppresses error (
|| echo "(skipped ...)") so deploy is GREEN — but matview is never created. Celeryrefresh_search_matviewthen fails at runtime withrelation "listings_search_mv" does not exist. Silent prod regression.Fix:
h.reviews_count AS house_ratings_count,(preserve alias for downstream API contract).CRITICAL #2 — psycopg.connect() rejects SQLAlchemy URL scheme
refresh_search_matview.py:30:Comment is wrong.
tradein-mvp/docker-compose.prod.yml:38defines:postgresql+psycopg://is a SQLAlchemy dialect URL — only consumed bycreate_engine(verifiedapp/core/db.py:8is the sole consumer in tradein-mvp). libpq /psycopg.connect()only acceptspostgresql://orpostgres://schemes;+psycopgproducesOperationalError: invalid integer value "..."ormissing "=" after ....Result: Celery task ALWAYS fails first invocation. Silent because no celery_app.py is wired yet — but the moment Phase 3.2 wires Beat schedule, daily refresh is broken.
Fix (one of):
Or extract via SQLAlchemy URL:
Or reuse the existing engine:
with engine.raw_connection() as raw: raw.cursor().execute("REFRESH ...")— but autocommit handling differs.MINOR (non-blocking)
DROP MATERIALIZED VIEW IF EXISTS listings_search_mv;followed byCREATE MATERIALIZED VIEW(noIF NOT EXISTS) on line 96 — PR body claimsCREATE MATERIALIZED VIEW IF NOT EXISTS, actual SQL is DROP + CREATE. This is better for schema-drift safety (handles column changes on re-apply) — but loses any unrefreshed manual data and re-populates synchronously on every deploy (~30s on populated DB). Acceptable for pilot; document tradeoff.tsv tsvector GENERATED ALWAYS AS (...) STOREDtriggers full table rewrite on ADD COLUMN. Pilot scale fine; at 100K+ rows this will lock the table for minutes. Add a follow-up note for Phase 4.10 indexes on
listings→ noticeable write amplification on scraper bulk inserts. Avito Stage 2a saves 50 lots × N anchors — monitor INSERT throughput post-deploy. Partial indexes (WHERE is_active = true) mitigate but GIN(tsv) is GENERATED → cannot be partial.Beat schedule comment doesn't specify timezone.
crontab(hour=3, minute=0)defaults to Celery'stimezonesetting (usually UTC) — if you want 3 AM Moscow, setapp.conf.timezone = 'Europe/Moscow'in celery_app.py when wired. Document explicitly.Verified OK
pg_trgmextension —tradeinuser is DB owner (POSTGRES_USER), canCREATE EXTENSION(trusted in PG13+, same aspostgisin 002).russiantsvector config — ships with Postgres by default.listings:description(011),address,geom,lat,lon,rooms,area_m2,floor,total_floors,price_rub,price_per_m2,kadastr_num,is_active,scraped_at,house_id_fk(011),canonical(044),source,source_url— all exist ✓houses:id,year_built,house_class,developer_name(010),rating(009),address_fingerprint(028),cadastral_number(029) — exist ✓listing_sources:listing_id,ext_source(028),price_rub(044) — exist ✓listings_search_mv_id_idx (listing_id)present BEFORE firstREFRESH CONCURRENTLY— correct ✓psycopgv3 import, no psycopg2, noprint(), norequests— conventions clean ✓Required before re-review
h.ratings_count→h.reviews_count(line ~155)+psycopgfrom DSN beforepsycopg.connect()(refresh_search_matview.py:30)Push fixup → re-trigger review.
NOT merged. Vault checked (
decisions/MultiSource_Integration_Master_Plan.mdsec 6.5 — confirms vault doc has the sameratings_countplanning error; recommend fixing vault doc too for consistency).Merged via deep-code-reviewer — verdict APPROVE.
Re-review summary (head
09698c5):h.ratings_count→h.reviews_count(matview line 133), aliashouse_ratings_countpreserved for API contract. Confirmed againsttradein-mvp/backend/data/sql/009_houses.sql:44(column isreviews_count).dsn = settings.database_url.replace("postgresql+psycopg://", "postgresql://", 1)beforepsycopg.connect(dsn, autocommit=True). Matchestradein-mvp/docker-compose.prod.yml:38DATABASE_URL dialect form.CREATE UNIQUE INDEX listings_search_mv_id_idx(line 167) enablesREFRESH ... CONCURRENTLY,to_tsvector('russian', ...)config × 2 (listings.tsv + matview.tsv), all matview SELECT columns exist in current schema (listings.description011:46,listings.canonical028:26,houses.reviews_count/rating/year_built/house_class/developer_name,houses.address_fingerprint028:18,houses.cadastral_number029:177,listing_sources.ext_source/listing_id028,listings.house_id_fk011:25), nopsycopg2, idempotent (IF NOT EXISTSeverywhere +DROP MATERIALIZED VIEW IF EXISTSbefore recreate).Post-merge verify queue (per task spec):
deploy-tradein.ymltriggers on push to mainSELECT extname FROM pg_extension WHERE extname='pg_trgm';andSELECT count(*) FROM pg_matviews WHERE matviewname='listings_search_mv';Merge commit:
e46b7d778e7f7afa7e531e71ab3da0eff6f21ab4.