Light1YT
c47a42c0a5
Add TEXT column cian_zhk_url to houses (migration 071) with partial index. This unblocks fetch_newbuilding() which requires a ZHK slug URL — the existing cian_internal_house_id (bigint) is not sufficient for the API call. Activate the do_houses block in backfill_cian_history(): query houses with cian_zhk_url set + no price_dynamics rows, then fetch + save per house. Add resolve_cian_zhk_url() in cian_newbuilding.py: follows Cian redirect from https://cian.ru/zhk/<id>/ to canonical slug URL, for use by the new backfill script (scripts/backfill_cian_zhk_url.py). Store cian_zhk_url into NewbuildingEnrichment and persist it via save_newbuilding_enrichment UPDATE (COALESCE — never overwrites non-NULL). Remove stale TODO comments referencing #525 from cian_history_backfill.py and admin.py docstring. Enable DO_HOUSES=true by default in cian_backfill_all.sh. Post-deploy required: apply 071 migration, then run scripts/backfill_cian_zhk_url.py to populate existing rows.
38 lines
1.8 KiB
PL/PgSQL
38 lines
1.8 KiB
PL/PgSQL
-- 071_houses_cian_zhk_url.sql
|
|
-- Purpose: Add cian_zhk_url TEXT column to houses — blocker for Cian newbuilding history backfill.
|
|
--
|
|
-- Context (#568):
|
|
-- houses already has cian_internal_house_id (bigint, 020_houses_alter_cian.sql) but
|
|
-- fetch_newbuilding() requires a slug-based ZHK URL (e.g. https://zhk-...-i.cian.ru/).
|
|
-- The numeric ID alone is not sufficient because the ZHK URL encodes a human-readable
|
|
-- slug Cian uses for routing — it cannot be reliably reconstructed from ID alone.
|
|
--
|
|
-- Backfill strategy (post-deploy, see scripts/backfill_cian_zhk_url.py):
|
|
-- For houses where cian_internal_house_id IS NOT NULL but cian_zhk_url IS NULL,
|
|
-- the canonical fallback URL https://cian.ru/zhk/<id>/ redirects to the slug URL.
|
|
-- The backfill script resolves final redirect URLs and stores the canonical slug form.
|
|
-- Do NOT run from inside the migration — requires live HTTP access + rate limiting.
|
|
--
|
|
-- Idempotency: IF NOT EXISTS on both ALTER and CREATE INDEX.
|
|
-- Dependencies: 009_houses.sql, 020_houses_alter_cian.sql
|
|
-- Apply after: 070_houses_dadata_enrichment.sql
|
|
|
|
BEGIN;
|
|
|
|
ALTER TABLE houses
|
|
ADD COLUMN IF NOT EXISTS cian_zhk_url text;
|
|
|
|
-- Partial index: used by backfill queries to find houses that already have a URL
|
|
-- and by fetch_newbuilding() callers joining on URL.
|
|
CREATE INDEX IF NOT EXISTS idx_houses_cian_zhk_url
|
|
ON houses (cian_zhk_url)
|
|
WHERE cian_zhk_url IS NOT NULL;
|
|
|
|
COMMENT ON COLUMN houses.cian_zhk_url IS
|
|
'Canonical Cian ЖК page URL (e.g. https://zhk-slug-city-i.cian.ru/). '
|
|
'Set during SERP scrape when Cian returns the slug, or via backfill '
|
|
'(scripts/backfill_cian_zhk_url.py) resolving redirect from '
|
|
'https://cian.ru/zhk/<cian_internal_house_id>/. '
|
|
'Required by fetch_newbuilding() for houses_price_dynamics backfill (#568).';
|
|
|
|
COMMIT;
|