Commit Graph

16 Commits

Author SHA1 Message Date
qananasikq
d5d6ba8b7c fix scraper flow 2026-04-22 21:40:40 +03:00
qananasikq
65d5f8e1eb Refactor to new ingestion pipeline architecture with discovery, fetch, enrichment services 2026-04-20 16:27:42 +03:00
qananasikq
a8c5f8aa41 Replace page-level checkpoint with segment-level resume 2026-04-17 17:51:23 +03:00
qananasikq
05d1ffb5ac Stabilize worker sync flow 2026-04-17 17:30:28 +03:00
qananasikq
3c3aea8eb3 tune vps config and update tests 2026-04-15 21:58:10 +03:00
qananasikq
acbb045b6e Add checkpoint logic and tests 2026-04-15 11:10:32 +03:00
qananasikq
d9111be2d2 fix worker lock 2026-04-14 19:32:45 +03:00
qananasikq
0add3913eb fix listing parsing 2026-04-14 19:31:57 +03:00
qananasikq
a4c93a1df1 fix docker scraping
improve batch sync

add postgres upsert

fix sync locking

improve listing sync

speed up scraper

clean up project

prepare for github

update docker setup
2026-04-13 16:35:08 +03:00
qananasikq
db2fa5ec17 add antibot unit tests 2026-04-10 15:18:57 +03:00
qananasikq
f6d7c8ea60 cleanup and fix runtime bugs 2026-04-09 20:35:25 +03:00
qananasikq
6e97991c68 cleanup and fix runtime bugs 2026-04-08 22:54:16 +03:00
qananasikq
64b7c760d9 update tests for sync 2026-04-08 18:32:45 +03:00
qananasikq
447a88feed persist raw attributes and tighten sync assertions 2026-04-08 14:30:57 +03:00
qananasikq
0010abdf08 fix parsing db and tests 2026-04-08 13:31:02 +03:00
qananasikq
78b52b265f IAAI scraper 2026-04-07 23:51:41 +03:00