fix(scrapers): disambiguate EMISS yearly vs Q1 in dedup key (#1606) #1687
No reviewers
Labels
No labels
admin
analytics
auth
automation
bug
business
chore
ci
compliance
data
data-moat
docs
duplicate
dx
enhancement
Fable 5 ревью
feedback/max
generative
GG-форсайт
needs-discussion
needs-human
observability
pause-bots
performance
priority/p0
priority/p1
priority/p2
priority/p3
scope/backend
scope/db
scope/devops
scope/frontend
scope/qa
scrapers
security
site-finder
stage/1
stage/2
status/blocked
status/done
status/needs-analysis
status/needs-fix
status/qa
status/ready
status/review
status/wip
tech-debt
tradein
ux
week ревью 1
wontfix
вторичка
ИРД
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference: lekss361/gendesign#1687
Loading…
Add table
Reference in a new issue
No description provided.
Delete branch "fix/emiss-period-dedup-1606"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Проблема (#1606)
rosstat_emiss.py:_emiss_period_to_monthмаппил и «год», и «I квартал» в month=1 → dedup-ключ(region, obs_date)коллидировал, годовой и Q1 перетирали друг друга.Решение
_emiss_period_granularity()(year/quarter/month) добавлен в in-memory ключ → 3-tuple(region, obs_date, granularity). Оба ряда выживают парсинг. DB-level PK(indicator_type, region, obs_date)— pre-existing ограничение схемы, не трогается.Verify
ruffclean;pytest(emiss/rosstat) → 49 passed, 1 skipped (+2 теста).Closes #1606
_YEAR_PERIODS ('год') → month=1 collided with Q1 ('I квартал') → month=1 for the same region/year: both produced obs_date=YYYY-01-01, so the second series in the XML silently overwrote the first in the in-memory by_key dict. Fix: add _emiss_period_granularity() and extend the dedup key from (region, obs_date) to (region, obs_date, granularity). 'year' and 'quarter' are now distinct slots, so both rows survive the parser and reach the upsert. Test: test_yearly_and_q1_both_survive_dedup verifies len==2 with both values present; test_period_granularity covers all classification branches.