|
|
f221aebef0
|
Prepared for production
|
2026-04-23 13:54:53 +03:00 |
|
qananasikq
|
09fecdae31
|
fix watchdog
|
2026-04-23 09:56:18 +03:00 |
|
qananasikq
|
d5d6ba8b7c
|
fix scraper flow
|
2026-04-22 21:40:40 +03:00 |
|
qananasikq
|
133c125e9c
|
fix: eliminate greenlet crash + memory leak protection
- docker-compose: switch worker to --pool=solo --concurrency=1 --max-tasks-per-child=1
- tasks.py: remove ThreadPoolExecutor wrapper from _run_browser_job (greenlet crash)
- scraper.py: browser fallback runs sequentially instead of ThreadPoolExecutor
- scraper.py: add gc.collect() after scraper close to prevent memory leaks
|
2026-04-20 17:33:51 +03:00 |
|
qananasikq
|
65d5f8e1eb
|
Refactor to new ingestion pipeline architecture with discovery, fetch, enrichment services
|
2026-04-20 16:27:42 +03:00 |
|
qananasikq
|
5999c2e42d
|
checkpoint: pipeline + 27k/h baseline before sitemap discovery
|
2026-04-18 16:34:09 +03:00 |
|
qananasikq
|
a8c5f8aa41
|
Replace page-level checkpoint with segment-level resume
|
2026-04-17 17:51:23 +03:00 |
|
qananasikq
|
05d1ffb5ac
|
Stabilize worker sync flow
|
2026-04-17 17:30:28 +03:00 |
|
qananasikq
|
6165c65159
|
add segmented listing sync
|
2026-04-15 21:57:40 +03:00 |
|
qananasikq
|
acbb045b6e
|
Add checkpoint logic and tests
|
2026-04-15 11:10:32 +03:00 |
|
qananasikq
|
d9111be2d2
|
fix worker lock
|
2026-04-14 19:32:45 +03:00 |
|
qananasikq
|
9337344182
|
stabilize worker
|
2026-04-14 14:03:08 +03:00 |
|
qananasikq
|
c729f971e3
|
fix worker recovery and docker logs
|
2026-04-13 20:49:25 +03:00 |
|
qananasikq
|
a4c93a1df1
|
fix docker scraping
improve batch sync
add postgres upsert
fix sync locking
improve listing sync
speed up scraper
clean up project
prepare for github
update docker setup
|
2026-04-13 16:35:08 +03:00 |
|
qananasikq
|
62aafae793
|
add Redis lock for sync_listing
|
2026-04-10 15:18:57 +03:00 |
|
qananasikq
|
f6d7c8ea60
|
cleanup and fix runtime bugs
|
2026-04-09 20:35:25 +03:00 |
|
qananasikq
|
437aedad45
|
update celery schedule and readme
|
2026-04-08 23:46:13 +03:00 |
|
qananasikq
|
f4b9d20191
|
add fastapi rest api and celery workers
|
2026-04-08 22:54:33 +03:00 |
|
