9c75ec0f78
- avito.py: fetch_around(pages=N, delay_override_sec) — pagination 1..N (backward compat default=1)
- scrape_pipeline.py:
* EKB_ANCHORS const (5 точек ЕКБ)
* run_avito_pipeline: pages + request_delay_sec params (backward compat defaults)
* CitySweepCounters dataclass с anchors_done/total + aggregate counters
* run_avito_city_sweep(db, run_id, *, pages_per_anchor, detail_top_n, request_delay_sec)
* Cooperative cancel: checks scrape_runs.status каждый anchor
* Per-anchor heartbeat + counters refresh
- scrape_runs.py NEW: utility module (create_run/update_heartbeat/mark_done/failed/cancelled/list_recent/is_cancelled)
- admin.py: 3 NEW endpoints:
* POST /admin/scrape/avito-city-sweep — start (returns run_id, FastAPI BackgroundTasks)
* GET /admin/scrape/avito-city-sweep/runs?limit=N — recent runs list
* POST /admin/scrape/avito-city-sweep/{run_id}/cancel — cooperative cancel
- 051_scrape_runs_extend.sql: ADD COLUMN params/counters/error/finished_at + cancelled в status enum
- avito-city-sweep.sh: cron wrapper с random 0-3h delay (anti-pattern-detection)
- tests/test_city_sweep.py: 16 offline smoke tests (counters, scrape_runs utils, endpoints)
Crontab entry: 0 2 * * * /opt/.../avito-city-sweep.sh (runs 02:00-05:00 UTC daily)
64 lines
2.3 KiB
Bash
64 lines
2.3 KiB
Bash
#!/usr/bin/env bash
|
|
# Avito city sweep — full ЕКБ pagination + enrichment.
|
|
# Random delay 0-3h перед запуском чтобы cron не бил Avito в одно и то же время каждый день.
|
|
#
|
|
# Crontab: 0 2 * * * /opt/gendesign/tradein-mvp/deploy/avito-city-sweep.sh
|
|
# Реальный запуск: 02:00-05:00 UTC случайно каждый день.
|
|
#
|
|
# Env overrides:
|
|
# SWEEP_PAGES=3 — страниц на anchor (default 3)
|
|
# SWEEP_DETAIL_TOP_N=20 — detail enrichment top-N (default 20)
|
|
# SWEEP_REQUEST_DELAY=7.0 — задержка между запросами в секундах (default 7.0)
|
|
# LOG=/var/log/tradein-avito-city-sweep.log
|
|
# TRADEIN_BACKEND_CONTAINER=tradein-backend
|
|
|
|
set -euo pipefail
|
|
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
LOG="${LOG:-/var/log/tradein-avito-city-sweep.log}"
|
|
CONTAINER="${TRADEIN_BACKEND_CONTAINER:-tradein-backend}"
|
|
|
|
log() {
|
|
echo "[$(date -u +'%Y-%m-%dT%H:%M:%SZ')] $*" | tee -a "$LOG"
|
|
}
|
|
|
|
# Random delay 0-10800 sec (0-3h). RANDOM gives 0..32767, mod 10800 → uniform 0..3h.
|
|
DELAY=$((RANDOM % 10800))
|
|
log "=== avito-city-sweep: sleeping ${DELAY}s before start ==="
|
|
sleep "$DELAY"
|
|
|
|
# Параметры sweep (overrideable via env)
|
|
PAGES_PER_ANCHOR="${SWEEP_PAGES:-3}"
|
|
DETAIL_TOP_N="${SWEEP_DETAIL_TOP_N:-20}"
|
|
REQUEST_DELAY="${SWEEP_REQUEST_DELAY:-7.0}"
|
|
|
|
PAYLOAD=$(cat <<EOF
|
|
{
|
|
"pages_per_anchor": ${PAGES_PER_ANCHOR},
|
|
"detail_top_n": ${DETAIL_TOP_N},
|
|
"request_delay_sec": ${REQUEST_DELAY},
|
|
"enrich_houses": true,
|
|
"radius_m": 1500
|
|
}
|
|
EOF
|
|
)
|
|
|
|
log "=== avito-city-sweep: triggering with pages=${PAGES_PER_ANCHOR} detail_top_n=${DETAIL_TOP_N} ==="
|
|
|
|
resp=$(docker exec "$CONTAINER" curl -s -X POST \
|
|
http://localhost:8000/api/v1/admin/scrape/avito-city-sweep \
|
|
-H "Content-Type: application/json" \
|
|
-d "$PAYLOAD" \
|
|
-m 30 2>/dev/null || echo '{"error":"trigger_failed"}')
|
|
|
|
log "<- response: $resp"
|
|
|
|
# Проверка что run_id вернулся
|
|
if echo "$resp" | grep -q '"run_id"'; then
|
|
RUN_ID=$(echo "$resp" | grep -o '"run_id":[0-9]*' | grep -o '[0-9]*')
|
|
log "=== avito-city-sweep: started run_id=${RUN_ID} (background task на backend)"
|
|
log "=== Polling: GET /api/v1/admin/scrape/avito-city-sweep/runs"
|
|
else
|
|
log "=== avito-city-sweep: WARNING — unexpected response, check backend logs ==="
|
|
exit 1
|
|
fi
|