Incremental Sync of 448K CSV Records — 3-Layer Optimization from 52s to 0.3s
HTTP HEAD pre-check · High-Water Mark reverse scanning · Score threshold noise filtering Real-world guide to syncing 81MB CSV → PostgreSQL in a serverless environment Problem Definition — Why Optimization Was Needed I was implementing a “district unit plan notice matching” feature for a building code review AI system. When querying district unit plan information for a specific address via VWorld API, it returns ntfc_sn (notice serial number), but this number alone doesn’t provide access to the actual notice document. ...