|
|
f221aebef0
|
Prepared for production
|
2026-04-23 13:54:53 +03:00 |
|
qananasikq
|
09fecdae31
|
fix watchdog
|
2026-04-23 09:56:18 +03:00 |
|
qananasikq
|
d5d6ba8b7c
|
fix scraper flow
|
2026-04-22 21:40:40 +03:00 |
|
qananasikq
|
133c125e9c
|
fix: eliminate greenlet crash + memory leak protection
- docker-compose: switch worker to --pool=solo --concurrency=1 --max-tasks-per-child=1
- tasks.py: remove ThreadPoolExecutor wrapper from _run_browser_job (greenlet crash)
- scraper.py: browser fallback runs sequentially instead of ThreadPoolExecutor
- scraper.py: add gc.collect() after scraper close to prevent memory leaks
|
2026-04-20 17:33:51 +03:00 |
|
qananasikq
|
65d5f8e1eb
|
Refactor to new ingestion pipeline architecture with discovery, fetch, enrichment services
|
2026-04-20 16:27:42 +03:00 |
|
qananasikq
|
5999c2e42d
|
checkpoint: pipeline + 27k/h baseline before sitemap discovery
|
2026-04-18 16:34:09 +03:00 |
|
qananasikq
|
a8c5f8aa41
|
Replace page-level checkpoint with segment-level resume
|
2026-04-17 17:51:23 +03:00 |
|
qananasikq
|
05d1ffb5ac
|
Stabilize worker sync flow
|
2026-04-17 17:30:28 +03:00 |
|
qananasikq
|
37d01c4ba1
|
clean mapper comments
|
2026-04-15 21:58:03 +03:00 |
|
qananasikq
|
6165c65159
|
add segmented listing sync
|
2026-04-15 21:57:40 +03:00 |
|
qananasikq
|
acbb045b6e
|
Add checkpoint logic and tests
|
2026-04-15 11:10:32 +03:00 |
|
qananasikq
|
d9111be2d2
|
fix worker lock
|
2026-04-14 19:32:45 +03:00 |
|
qananasikq
|
cac68bda4c
|
add full scan mode
|
2026-04-14 19:32:34 +03:00 |
|
qananasikq
|
0add3913eb
|
fix listing parsing
|
2026-04-14 19:31:57 +03:00 |
|
qananasikq
|
d51216ddb7
|
improve listing sync
|
2026-04-14 14:03:16 +03:00 |
|
qananasikq
|
9337344182
|
stabilize worker
|
2026-04-14 14:03:08 +03:00 |
|
qananasikq
|
c729f971e3
|
fix worker recovery and docker logs
|
2026-04-13 20:49:25 +03:00 |
|
qananasikq
|
a4c93a1df1
|
fix docker scraping
improve batch sync
add postgres upsert
fix sync locking
improve listing sync
speed up scraper
clean up project
prepare for github
update docker setup
|
2026-04-13 16:35:08 +03:00 |
|
qananasikq
|
b9557922ed
|
fix parser_id format: car- + 22 alphanum chars, origin_id: iaai:{lot}
|
2026-04-10 20:31:48 +03:00 |
|
qananasikq
|
6134b10fd0
|
add iaai auto-login with session persistence
|
2026-04-10 19:59:03 +03:00 |
|
qananasikq
|
05b83b5518
|
add cookie session persistence
|
2026-04-10 19:55:38 +03:00 |
|
qananasikq
|
3870cf4d94
|
add task status endpoint
|
2026-04-10 15:18:57 +03:00 |
|
qananasikq
|
62aafae793
|
add Redis lock for sync_listing
|
2026-04-10 15:18:57 +03:00 |
|
qananasikq
|
acf30fcc20
|
add antibot guard and retry
|
2026-04-10 15:18:57 +03:00 |
|
qananasikq
|
7b1501008e
|
fix limit after only_new filter
|
2026-04-10 15:18:57 +03:00 |
|
qananasikq
|
f6d7c8ea60
|
cleanup and fix runtime bugs
|
2026-04-09 20:35:25 +03:00 |
|
qananasikq
|
437aedad45
|
update celery schedule and readme
|
2026-04-08 23:46:13 +03:00 |
|
qananasikq
|
f4b9d20191
|
add fastapi rest api and celery workers
|
2026-04-08 22:54:33 +03:00 |
|
qananasikq
|
6e97991c68
|
cleanup and fix runtime bugs
|
2026-04-08 22:54:16 +03:00 |
|
qananasikq
|
90bd4756b3
|
update readme and env
|
2026-04-08 18:33:56 +03:00 |
|
qananasikq
|
f161071e07
|
harden price normalization
|
2026-04-08 18:30:39 +03:00 |
|
qananasikq
|
3b218534ec
|
sync only new cars
|
2026-04-08 18:30:02 +03:00 |
|
qananasikq
|
54fea100dc
|
add docker proxy bridge
|
2026-04-08 18:29:06 +03:00 |
|
qananasikq
|
447a88feed
|
persist raw attributes and tighten sync assertions
|
2026-04-08 14:30:57 +03:00 |
|
qananasikq
|
b2d3902bcc
|
stabilize daemon sync and docker startup
|
2026-04-08 14:30:44 +03:00 |
|
qananasikq
|
0010abdf08
|
fix parsing db and tests
|
2026-04-08 13:31:02 +03:00 |
|
qananasikq
|
f96e5ca5f8
|
improve scraper runtime
|
2026-04-08 13:30:45 +03:00 |
|
qananasikq
|
78b52b265f
|
IAAI scraper
|
2026-04-07 23:51:41 +03:00 |
|
