gendesign

lekss361/gendesign

Fork 0

Commit graph

Author	SHA1	Message	Date
lekss361	4ace87af14	fix(cadastre): exact-match headers to nspd_lite to bypass NSPD WAF (#168 ) Pilot v2 (job_id=2) failed 50/50 with HTTP 403 WAF block. Comparing nspd_bulk_client.DEFAULT_HEADERS vs legacy nspd_lite.HEADERS (which works on VPS IP since April 2026): PascalCase keys → lowercase keys Chrome/148 UA → Chrome/144 UA No cache-control / pragma → "no-cache" both accept-language ru first → en first No origin → "https://nspd.gov.ru" referer "/map" → "/map?thematic=PKK" NSPD WAF (BotShield-class) likely fingerprints на header order + values combined with TLS fingerprint. Matching legacy exactly minimizes deltas. Test plan: retry pilot job after deploy, expect 0 WAF blocks for first 5 quarters.	2026-05-15 14:59:41 +03:00
lekss361	8dee9546ae	fix(cadastre): NSPDBulkClient verify=False for NSPD SSL chain (#168 hotfix) (#175 ) Pilot run #1 failed on all 50 quarters with: NspdBulkError: [SSL: CERTIFICATE_VERIFY_FAILED] self-signed certificate in certificate chain NSPD prod chain on Beget VPS contains an internal/self-signed CA → httpx default verify=True trips. Reuse pattern from legacy app.services.scrapers.nspd_lite (uses ssl._create_unverified_context() for same reason since April 2026). Risk profile: NSPD public gov data, no sensitive payload outbound, admin token не уходит к NSPD endpoints. MITM risk minimal. Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 14:50:21 +03:00
lekss361	b7889d113b	feat(cadastre): NSPDBulkClient + schemas for bulk ingest (#168 PR2/5) (#170 ) Add async NSPD client (nspd_bulk_client.py) with 3 new methods: - search_by_quarter: REST /api/geoportal/v2/search/geoportal — snapshot 20 per cat + meta totals - wms_feature_info: WMS GetFeatureInfo для grid-walk discovery - list_objects_in_building: tab-group-data (Q3 deferred — код ready) Rate limit: asyncio.Semaphore(3) + 0.05s jitter. 429 → exponential backoff 2→4→8s, max 3 retries. 403 (WAF) → raise immediately, no retry. Pydantic schemas (nspd_bulk.py): NSPDBulkFeature, QuarterSnapshot, ObjectsListing with extra='allow' to preserve unknown fields → raw_props JSONB downstream. 15 unit tests + 4 e2e (real NSPD, skipped in CI by default). Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 12:50:41 +03:00

Author

SHA1

Message

Date

lekss361

4ace87af14

fix(cadastre): exact-match headers to nspd_lite to bypass NSPD WAF (#168 )

Pilot v2 (job_id=2) failed 50/50 with HTTP 403 WAF block. Comparing
nspd_bulk_client.DEFAULT_HEADERS vs legacy nspd_lite.HEADERS (which works
on VPS IP since April 2026):

  PascalCase keys → lowercase keys
  Chrome/148 UA → Chrome/144 UA
  No cache-control / pragma → "no-cache" both
  accept-language ru first → en first
  No origin → "https://nspd.gov.ru"
  referer "/map" → "/map?thematic=PKK"

NSPD WAF (BotShield-class) likely fingerprints на header order + values
combined with TLS fingerprint. Matching legacy exactly minimizes deltas.

Test plan: retry pilot job after deploy, expect 0 WAF blocks for first
5 quarters.

2026-05-15 14:59:41 +03:00

lekss361

8dee9546ae

fix(cadastre): NSPDBulkClient verify=False for NSPD SSL chain (#168 hotfix) (#175 )

Pilot run #1 failed on all 50 quarters with:
  NspdBulkError: [SSL: CERTIFICATE_VERIFY_FAILED] self-signed certificate
  in certificate chain

NSPD prod chain on Beget VPS contains an internal/self-signed CA →
httpx default verify=True trips. Reuse pattern from legacy
app.services.scrapers.nspd_lite (uses ssl._create_unverified_context()
for same reason since April 2026).

Risk profile: NSPD public gov data, no sensitive payload outbound,
admin token не уходит к NSPD endpoints. MITM risk minimal.

Co-authored-by: lekss361 <claudestars@proton.me>

2026-05-15 14:50:21 +03:00

lekss361

b7889d113b

feat(cadastre): NSPDBulkClient + schemas for bulk ingest (#168 PR2/5) (#170 )

Add async NSPD client (nspd_bulk_client.py) with 3 new methods:
- search_by_quarter: REST /api/geoportal/v2/search/geoportal — snapshot 20 per cat + meta totals
- wms_feature_info: WMS GetFeatureInfo для grid-walk discovery
- list_objects_in_building: tab-group-data (Q3 deferred — код ready)

Rate limit: asyncio.Semaphore(3) + 0.05s jitter.
429 → exponential backoff 2→4→8s, max 3 retries.
403 (WAF) → raise immediately, no retry.

Pydantic schemas (nspd_bulk.py): NSPDBulkFeature, QuarterSnapshot, ObjectsListing
with extra='allow' to preserve unknown fields → raw_props JSONB downstream.

15 unit tests + 4 e2e (real NSPD, skipped in CI by default).

Co-authored-by: lekss361 <claudestars@proton.me>

2026-05-15 12:50:41 +03:00

3 commits