qananasikq/encar
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: qananasikq:main
Branches Tags
qananasikq:main
..
compare: qananasikq:032608261e63c4e6a26ac5adaa718788d85ec4f6
Branches Tags
qananasikq:main

1 Commits
main ... 032608261e

Author SHA1 Message Date
qananasikq
032608261e update readme 2026-04-16 18:11:22 +03:00
15 changed files with 460 additions and 2052 deletions
Expand all files Collapse all files

BIN
.gitignore vendored
View File

Binary file not shown.

11
Dockerfile
View File

@@ -1,22 +1,19 @@
FROM python:3.12-slim
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
PIP_DISABLE_PIP_VERSION_CHECK=1 \
PIP_DEFAULT_TIMEOUT=300
PYTHONUNBUFFERED=1
RUN groupadd --system app && useradd --system --gid app --create-home app
WORKDIR /app
COPY pyproject.toml uv.lock ./
RUN pip install --no-cache-dir --retries 10 uv \
RUN pip install --no-cache-dir uv \
&& uv export --format requirements-txt --no-dev --no-hashes --no-emit-project --frozen -o /tmp/requirements.txt \
&& pip install --no-cache-dir --retries 10 -r /tmp/requirements.txt
&& pip install --no-cache-dir -r /tmp/requirements.txt
COPY . .
RUN pip install --no-cache-dir --retries 10 "setuptools>=68"
RUN pip install --no-cache-dir --no-build-isolation -e .
RUN pip install --no-cache-dir -e .
RUN python -m compileall -q encar_scraper
RUN chmod +x entrypoint.sh
RUN chown -R app:app /app

7
README.md
View File

@@ -1,13 +1,13 @@
# Encar Scraper
Парсер автомобилей с [encar.com](https://www.encar.com)
Парсер автомобилей с [encar.com](https://www.encar.com) — крупнейшей корейской площадки продажи авто.
## Как работает
## Как это работает
Скрапер работает через публичный JSON API Encar (без браузера, без Selenium):
1. **Листинг** — `api.encar.com/search/car/list/general` отдаёт список авто с пагинацией (до 1000 на страницу)
2. **Шардирование** — API лимитирует выдачу от 10k результатов, поэтому весь каталог 200к+ разбивается на 36 шардов по типу (domestic/import) и диапазонам годов
2. **Шардирование** — API лимитирует выдачу ~10k результатов, поэтому весь каталог (~230k авто) разбивается на 36 шардов по типу (domestic/import) и диапазонам годов
3. **Фото** — batch endpoint `api.encar.com/v1/readside/vehicles` (до 20 ID за запрос) возвращает все фото с CDN `ci.encar.com`
4. **Переводы** — корейские названия брендов, моделей и цветов автоматически переводятся в русские/английские
5. **Sold-трекинг** — авто, пропавшие из листинга, помечаются как проданные
@@ -17,6 +17,7 @@ Celery Beat запускает полную синхронизацию кажд
## Особенности
- **Чистые HTTP запросы** — без браузера, через публичный API Encar
- **~100 авто/сек** — полный sync ~230k авто за ~35 мин
- **PostgreSQL** для хранения данных (авто + фото)
- **Celery + Redis** для фоновых задач и периодической синхронизации
- **FastAPI** REST API для управления задачами и просмотра данных

21
encar_scraper/api/routes/cars.py
View File

@@ -26,28 +26,19 @@ def list_cars(
# Список автомобилей с пагинацией и фильтрами
with persistence.session_scope() as session:
query = select(Car)
count_query = select(func.count(Car.id))
if brand:
escaped_brand = brand.replace("%", r"\%").replace("_", r"\_")
cond = Car.brand.ilike(f"%{escaped_brand}%", escape="\\")
query = query.where(cond)
count_query = count_query.where(cond)
query = query.where(Car.brand.ilike(f"%{brand}%"))
if model:
escaped_model = model.replace("%", r"\%").replace("_", r"\_")
cond = Car.model.ilike(f"%{escaped_model}%", escape="\\")
query = query.where(cond)
count_query = count_query.where(cond)
query = query.where(Car.model.ilike(f"%{model}%"))
if year_min is not None:
query = query.where(Car.year >= year_min)
count_query = count_query.where(Car.year >= year_min)
if year_max is not None:
query = query.where(Car.year <= year_max)
count_query = count_query.where(Car.year <= year_max)
if is_sold is not None:
query = query.where(Car.is_sold == is_sold)
count_query = count_query.where(Car.is_sold == is_sold)
count_query = select(func.count()).select_from(query.subquery())
total = session.execute(count_query).scalar() or 0
offset = (page - 1) * per_page
@@ -71,9 +62,7 @@ def get_car(
):
# Детальная информация об автомобиле с изображениями
with persistence.session_scope() as session:
car = session.execute(
select(Car).options(selectinload(Car.images)).where(Car.id == car_id)
).scalars().first()
car = session.get(Car, car_id)
if car is None:
raise HTTPException(status_code=404, detail="Car not found")
return CarRead.model_validate(car).model_dump(mode="json")
@@ -87,7 +76,7 @@ def get_car_by_origin(
# Поиск автомобиля по origin_id
with persistence.session_scope() as session:
car = session.execute(
select(Car).options(selectinload(Car.images)).where(Car.origin_id == origin_id)
select(Car).where(Car.origin_id == origin_id)
).scalars().first()
if car is None:
raise HTTPException(status_code=404, detail="Car not found")

4
encar_scraper/api/routes/tasks.py
View File

@@ -1,7 +1,7 @@
# Роуты запуска задач синхронизации Encar и просмотра истории sync-runs.
from fastapi import APIRouter, Depends, Query
from pydantic import BaseModel, Field
from pydantic import BaseModel
from sqlalchemy import select, func
from ..deps import get_persistence
@@ -25,7 +25,7 @@ class SyncListingRequest(BaseModel):
year_to: int | None = None
price_max: int | None = None
lane: str = "encar"
limit: int | None = Field(default=None, le=500_000)
limit: int | None = None
@router.post("/tasks/sync-vehicle")

2
encar_scraper/core/utils.py
View File

@@ -1,7 +1,7 @@
import json
import re
from pathlib import Path
from typing import Any, Iterable
from typing import Any, Callable, Iterable
def save_to_json(data: Any, filename: str | Path) -> None:

882
encar_scraper/encar.py
View File

File diff suppressed because it is too large Load Diff

326
encar_scraper/partner_dictionary.py
View File

@@ -1,326 +0,0 @@
"""Korean -> English translation tables for EnCar (partner-provided basis).
Источник переводов от партнёра. Этот файл — эталон: бренды и модели берём
отсюда в первую очередь. Держим секционные словари плюс один плоский для
простых поисков.
"""
from __future__ import annotations
ENCAR_KO_EN_TABLES: dict[str, dict[str, str]] = {
"makers": {
"포르쉐": "Porsche",
"메르세데스-벤츠": "Mercedes-Benz",
"메르세데스 벤츠": "Mercedes-Benz",
"벤츠": "Mercedes-Benz",
"메르세데스-AMG": "Mercedes-AMG",
"메르세데스 AMG": "Mercedes-AMG",
"AMG": "AMG",
"BMW": "BMW",
"아우디": "Audi",
"폭스바겐": "Volkswagen",
"렉서스": "Lexus",
"토요타": "Toyota",
"닛산": "Nissan",
"인피니티": "Infiniti",
"혼다": "Honda",
"마쯔다": "Mazda",
"스바루": "Subaru",
"미쓰비시": "Mitsubishi",
"랜드로버": "Land Rover",
"재규어": "Jaguar",
"지프": "Jeep",
"볼보": "Volvo",
"푸조": "Peugeot",
"르노": "Renault",
"미니": "MINI",
"로터스": "Lotus",
"벤틀리": "Bentley",
"롤스로이스": "Rolls-Royce",
"람보르기니": "Lamborghini",
"페라리": "Ferrari",
"맥라렌": "McLaren",
"애스턴마틴": "Aston Martin",
"애스톤마틴": "Aston Martin",
"마세라티": "Maserati",
"코닉세그": "Koenigsegg",
"파가니": "Pagani",
"부가티": "Bugatti",
"캐딜락": "Cadillac",
"쉐보레": "Chevrolet",
"포드": "Ford",
"링컨": "Lincoln",
"크라이슬러": "Chrysler",
"닷지": "Dodge",
"램": "Ram",
"테슬라": "Tesla",
"제네시스": "Genesis",
"현대": "Hyundai",
"기아": "Kia",
"KG모빌리티": "KG Mobility",
"쌍용": "SsangYong",
"알파로메오": "Alfa Romeo",
"피아트": "Fiat",
"시트로엥": "Citroen",
"스마트": "Smart",
"폴스타": "Polestar",
"리비안": "Rivian",
"루시드": "Lucid",
"BYD": "BYD",
"지리": "Geely",
"홍치": "Hongqi",
"이스즈": "Isuzu",
},
"models": {
"718 박스터": "718 Boxster",
"718 카이맨": "718 Cayman",
"911": "911",
"911 카브리올레": "911 Cabriolet",
"카이엔": "Cayenne",
"카이엔 쿠페": "Cayenne Coupe",
"파나메라": "Panamera",
"파나메라 스포츠 투리스모": "Panamera Sport Turismo",
"마칸": "Macan",
"타이칸": "Taycan",
"타이칸 크로스 투리스모": "Taycan Cross Turismo",
"918 스파이더": "918 Spyder",
"카레라 GT": "Carrera GT",
"718 스파이더": "718 Spyder",
"박스터": "Boxster",
"카이맨": "Cayman",
"4.0 스파이더": "4.0 Spyder",
"C 클래스": "C-Class",
"E 클래스": "E-Class",
"S 클래스": "S-Class",
"A 클래스": "A-Class",
"B 클래스": "B-Class",
"CLA": "CLA",
"CLS": "CLS",
"SL": "SL",
"SLK": "SLK",
"G 클래스": "G-Class",
"GLA": "GLA",
"GLB": "GLB",
"GLC": "GLC",
"GLE": "GLE",
"GLS": "GLS",
"AMG GT": "AMG GT",
"AMG GT 4도어 쿠페": "AMG GT 4-Door Coupe",
"V 클래스": "V-Class",
"1 시리즈": "1 Series",
"2 시리즈": "2 Series",
"3 시리즈": "3 Series",
"4 시리즈": "4 Series",
"5 시리즈": "5 Series",
"6 시리즈": "6 Series",
"7 시리즈": "7 Series",
"8 시리즈": "8 Series",
"M2": "M2",
"M3": "M3",
"M4": "M4",
"M5": "M5",
"M8": "M8",
"X1": "X1",
"X2": "X2",
"X3": "X3",
"X4": "X4",
"X5": "X5",
"X6": "X6",
"X7": "X7",
"Z4": "Z4",
"A3": "A3",
"A4": "A4",
"A5": "A5",
"A6": "A6",
"A7": "A7",
"A8": "A8",
"Q3": "Q3",
"Q5": "Q5",
"Q7": "Q7",
"Q8": "Q8",
"R8": "R8",
"TT": "TT",
"고스트": "Ghost",
"팬텀": "Phantom",
"레이스": "Wraith",
"던": "Dawn",
"컬리넌": "Cullinan",
"스펙터": "Spectre",
"벤테이가": "Bentayga",
"컨티넨탈 GT": "Continental GT",
"플라잉스퍼": "Flying Spur",
"우라칸": "Huracan",
"아벤타도르": "Aventador",
"우루스": "Urus",
"레부엘토": "Revuelto",
"로마": "Roma",
"포르토피노": "Portofino",
"SF90 스트라달레": "SF90 Stradale",
"F8 트리뷰토": "F8 Tributo",
"콰트로포르테": "Quattroporte",
"기블리": "Ghibli",
"르반떼": "Levante",
"그란투리스모": "GranTurismo",
"반퀴시": "Vanquish",
"밴티지": "Vantage",
"DB11": "DB11",
"DB12": "DB12",
"DBS": "DBS",
},
"body_types": {
"세단": "Sedan",
"쿠페": "Coupe",
"컨버터블": "Convertible",
"카브리올레": "Cabriolet",
"오픈카": "Open Top",
"해치백": "Hatchback",
"왜건": "Wagon",
"스테이션왜건": "Station Wagon",
"SUV": "SUV",
"크로스오버": "Crossover",
"픽업트럭": "Pickup Truck",
"밴": "Van",
"미니밴": "Minivan",
"트럭": "Truck",
"쿠페형 SUV": "Coupe SUV",
"기타": "Other",
},
"fuel_types": {
"가솔린": "Gasoline",
"디젤": "Diesel",
"경유": "Diesel",
"하이브리드": "Hybrid",
"가솔린 하이브리드": "Gasoline Hybrid",
"플러그인 하이브리드": "Plug-in Hybrid",
"전기": "Electric",
"전기차": "Electric",
"수소": "Hydrogen",
"LPG": "LPG",
"LNG": "LNG",
"가솔린+전기": "Gasoline + Electric",
"가솔린 + 전기": "Gasoline + Electric",
},
"transmissions": {
"수동": "Manual",
"오토": "Automatic",
"자동": "Automatic",
"자동변속": "Automatic",
"세미오토": "Semi-automatic",
"CVT": "CVT",
"듀얼클러치": "Dual-clutch",
},
"sell_types": {
"일반": "General",
"리스": "Lease",
"렌트": "Rent",
"할부": "Installment",
},
"lease_types": {
"운용리스": "Operating Lease",
"금융리스": "Financial Lease",
},
"regions": {
"서울": "Seoul",
"부산": "Busan",
"인천": "Incheon",
"대구": "Daegu",
"대전": "Daejeon",
"광주": "Gwangju",
"울산": "Ulsan",
"세종": "Sejong",
"경기": "Gyeonggi",
"강원": "Gangwon",
"충북": "Chungbuk",
"충남": "Chungnam",
"전북": "Jeonbuk",
"전남": "Jeonnam",
"경북": "Gyeongbuk",
"경남": "Gyeongnam",
"제주": "Jeju",
},
"dealer_companies": {
"(주)기억": "Gieok Co., Ltd.",
"(주)네바퀴": "Four Wheels Co., Ltd.",
"(주)에이스타모터스": "A-Star Motors Co., Ltd.",
"(주)우진 오토모빌": "Woojin Automobile Co., Ltd.",
"(주)지온컴퍼니": "Zion Company Co., Ltd.",
"(주)플로시모터스": "Flossy Motors Co., Ltd.",
"㈜오토핸즈": "Auto Hands Co., Ltd.",
"GD카": "GD Car",
"MK모터스": "MK Motors",
"꿈차": "Dream Car",
"마이카4": "My Car 4",
"블루오토": "Blue Auto",
"세영모빌리티(주)": "Seyoung Mobility Co., Ltd.",
"스마일자동차": "Smile Motors",
"오토링크": "Auto Link",
"오토핸즈": "Auto Hands",
"우리들모터스": "Woorideul Motors",
"주식회사 모터바겐": "Motorwagen Co., Ltd.",
"주식회사 아이오토스": "iAutos Co., Ltd.",
"주식회사이들컴퍼니": "Ideul Company Co., Ltd.",
"카메이트": "Car Mate",
"카톡모터스": "Katok Motors",
"카피디": "Car PD",
"포르쉐 인증 중고차 센터 성수": "Porsche Certified Used Car Center Seongsu",
"포르쉐인증중고차센터성수": "Porsche Certified Used Car Center Seongsu",
},
"dealer_names": {
"고준용": "Go Jun-yong",
"공민중": "Gong Min-jung",
"김경진": "Kim Gyeong-jin",
"김명진": "Kim Myeong-jin",
"김민수": "Kim Min-su",
"김성언": "Kim Seong-eon",
"김성진": "Kim Seong-jin",
"김영균": "Kim Yeong-gyun",
"김완섭": "Kim Wan-seop",
"김현진": "Kim Hyeon-jin",
"김형진": "Kim Hyeong-jin",
"김호영": "Kim Ho-yeong",
"문명기": "Moon Myeong-gi",
"박원석": "Park Won-seok",
"박재현": "Park Jae-hyeon",
"방지윤": "Bang Ji-yun",
"송슬기": "Song Seul-gi",
"신영철": "Shin Yeong-cheol",
"신호진": "Shin Ho-jin",
"양시언": "Yang Si-eon",
"양태규": "Yang Tae-gyu",
"윤지완": "Yoon Ji-wan",
"이경민": "Lee Gyeong-min",
"이재욱": "Lee Jae-uk",
"임종현": "Lim Jong-hyeon",
"임형욱": "Lim Hyeong-uk",
"전태준": "Jeon Tae-jun",
"정상필": "Jeong Sang-pil",
"조경식": "Jo Gyeong-sik",
"조승재": "Jo Seung-jae",
"조환성": "Jo Hwan-seong",
"차상훈": "Cha Sang-hun",
"최경욱": "Choi Gyeong-uk",
"하상열": "Ha Sang-yeol",
"허지훈": "Heo Ji-hun",
},
}
def _build_flat_map() -> dict[str, str]:
result: dict[str, str] = {}
for section in ENCAR_KO_EN_TABLES.values():
result.update(section)
return result
ENCAR_KO_EN_TRANSLATIONS: dict[str, str] = _build_flat_map()
def translate_encar_ko_to_en(value: str, section: str | None = None) -> str:
"""Перевод корейского термина Encar на английский.
Если задан `section`, используется только словарь этой секции. Иначе —
плоский объединённый словарь.
"""
if section is not None:
return ENCAR_KO_EN_TABLES.get(section, {}).get(value, value)
return ENCAR_KO_EN_TRANSLATIONS.get(value, value)

5
encar_scraper/storage/db.py
View File

@@ -46,10 +46,7 @@ class PersistenceService:
try:
Base.metadata.create_all(self.engine)
except Exception:
# Не глотаем: таблицы могут уже существовать (ок), либо есть проблема
# доступа — пусть вышестоящий код видит её при первой операции, но
# сигнализируем в warning чтобы упростить диагностику.
logger.warning("create_tables failed (continuing — schema may already exist)", exc_info=True)
logger.debug("create_tables skipped (schema already exists)")
@contextmanager
def session_scope(self) -> Iterator[Session]:

980
encar_scraper/translations.py
View File

@@ -1,980 +0,0 @@
# Словари переводов корейского текста русский / английский.
# Используется в encar.py для маппинга данных Encar API.
import re
from .partner_dictionary import ENCAR_KO_EN_TABLES as _PARTNER_TABLES
# Топливо
FUEL_TYPE_MAP = {
"가솔린": "Gasoline",
"gasoline": "Gasoline",
"디젤": "Diesel",
"경유": "Diesel",
"diesel": "Diesel",
"전기": "Electric",
"전기차": "Electric",
"electric": "Electric",
"하이브리드": "Hybrid",
"hybrid": "Hybrid",
"가솔린 하이브리드": "Gasoline Hybrid",
"플러그인 하이브리드": "Plug-in Hybrid",
"lpg": "LPG",
"LPG": "LPG",
"LPG(일반인 구입)": "LPG",
"LPG(장애인 구입)": "LPG",
"LNG": "LNG",
"수소": "Hydrogen",
"가솔린+전기": "Gasoline + Electric",
"가솔린 + 전기": "Gasoline + Electric",
"디젤+전기": "Hybrid",
"CNG": "CNG",
}
# КПП
GEARBOX_MAP_KO = {
"오토": "AT",
"자동": "AT",
"자동변속": "AT",
"수동": "MT",
"세미오토": "AT",
"cvt": "CVT",
"CVT": "CVT",
"듀얼클러치": "AT",
}
# Тип продажи
SELL_TYPE_MAP = {
"일반": "STOCK",
"경매": "AUCTION",
"입찰": "TENDER",
"리스": "STOCK",
"렌트": "STOCK",
"할부": "STOCK",
}
# Город → страна
CITY_TO_COUNTRY = {
"서울": "KR", "경기": "KR", "인천": "KR", "부산": "KR", "대구": "KR",
"광주": "KR", "대전": "KR", "울산": "KR", "세종": "KR", "강원": "KR",
"충북": "KR", "충남": "KR", "전북": "KR", "전남": "KR", "경북": "KR",
"경남": "KR", "제주": "KR",
}
# Бренды: корейский → английский
BRAND_TRANSLATIONS = {
"현대": "Hyundai",
"기아": "Kia",
"쉐보레(GM대우)": "Chevrolet",
"벤츠": "Mercedes-Benz",
"메르세데스-벤츠": "Mercedes-Benz",
"메르세데스 벤츠": "Mercedes-Benz",
"메르세데스-AMG": "Mercedes-AMG",
"메르세데스 AMG": "Mercedes-AMG",
"AMG": "AMG",
"BMW": "BMW",
"아우디": "Audi",
"폭스바겐": "Volkswagen",
"토요타": "Toyota",
"도요타": "Toyota",
"닛산": "Nissan",
"혼다": "Honda",
"포르쉐": "Porsche",
"랜드로버": "Land Rover",
"재규어": "Jaguar",
"볼보": "Volvo",
"푸조": "Peugeot",
"시트로엥": "Citroen",
"르노": "Renault",
"미니": "MINI",
"링컨": "Lincoln",
"캐딜락": "Cadillac",
"크라이슬러": "Chrysler",
"지프": "Jeep",
"포드": "Ford",
"테슬라": "Tesla",
"폴크스바겐": "Volkswagen",
"스코다": "Skoda",
"세아트": "SEAT",
"오펠": "Opel",
"피아트": "Fiat",
"알파로메오": "Alfa Romeo",
"마세라티": "Maserati",
"람보르기니": "Lamborghini",
"페라리": "Ferrari",
"벤틀리": "Bentley",
"롤스로이스": "Rolls-Royce",
"애스턴마틴": "Aston Martin",
"애스톤마틴": "Aston Martin",
"맥라렌": "McLaren",
"부가티": "Bugatti",
"코닉세그": "Koenigsegg",
"파가니": "Pagani",
"헤네시": "Hennessey",
"쉘비": "Shelby",
"다임러": "Daimler",
"마이바흐": "Maybach",
"스마트": "Smart",
"DS": "DS",
"렉서스": "Lexus",
"인피니티": "Infiniti",
"아큐라": "Acura",
"스바루": "Subaru",
"미쓰비시": "Mitsubishi",
"마쓰다": "Mazda",
"이스즈": "Isuzu",
"다이하츠": "Daihatsu",
"스즈키": "Suzuki",
"KG모빌리티(쌍용)": "KG Mobility",
"KG모빌리티": "KG Mobility",
"르노코리아(삼성)": "Renault Korea",
"한국GM": "GM Korea",
"쌍용": "SsangYong",
"대우": "Daewoo",
"제네시스": "Genesis",
"르노삼성": "Renault Samsung",
"로터스": "Lotus",
"리비안": "Rivian",
"루시드": "Lucid",
"리막": "Rimac",
"알파 로메오": "Alfa Romeo",
"마쯔다": "Mazda",
"어큐라": "Acura",
"다이하쯔": "Daihatsu",
# --- Новые бренды (с сайта Encar) ---
"쉐보레": "Chevrolet",
"시트로엥/DS": "Citroen/DS",
"미쯔비시": "Mitsubishi",
"미쯔오까": "Mitsuoka",
"동풍소콘": "Dongfeng Sokon",
"북기은상": "BAIC",
"신위안": "Skywell",
"이네오스": "Ineos",
"지리": "Geely",
"홍치": "Hongqi",
"BYD": "BYD",
"머큐리": "Mercury",
"사이언": "Scion",
"뷰익": "Buick",
"새턴": "Saturn",
"올즈모빌": "Oldsmobile",
"폰티악": "Pontiac",
"포톤": "Foton",
"MG로버": "MG Rover",
"닷지": "Dodge",
"램": "Ram",
"험머": "Hummer",
"폴스타": "Polestar",
"사브": "Saab",
"기타 제조사": "Other",
"기타 수입차": "Other Imported",
}
for _ko, _en in _PARTNER_TABLES["makers"].items():
BRAND_TRANSLATIONS.setdefault(_ko, _en)
# Бренды: английский → алиасы (корейский + русский)
BRAND_ENGLISH_ALIASES: dict[str, list[str]] = {
"hyundai": ["현대", "Хёндэ"],
"kia": ["기아", "Киа"],
"genesis": ["제네시스", "Генезис"],
"chevrolet": ["쉐보레(GM대우)", "쉐보레", "Шевроле"],
"mercedes": ["벤츠", "Мерседес-Бенц"],
"mercedes-benz": ["벤츠", "Мерседес-Бенц"],
"bmw": ["BMW"],
"audi": ["아우디", "Ауди"],
"volkswagen": ["폭스바겐", "폴크스바겐", "Фольксваген"],
"toyota": ["토요타", "도요타", "Тойота"],
"nissan": ["닛산", "Ниссан"],
"honda": ["혼다", "Хонда"],
"porsche": ["포르쉐", "Порше"],
"land rover": ["랜드로버", "Ленд Ровер"],
"jaguar": ["재규어", "Ягуар"],
"volvo": ["볼보", "Вольво"],
"peugeot": ["푸조", "Пежо"],
"citroen": ["시트로엥", "Ситроен", "시트로엥/DS", "Ситроен/DS"],
"renault": ["르노", "Рено"],
"mini": ["미니", "Мини"],
"lincoln": ["링컨", "Линкольн"],
"cadillac": ["캐딜락", "Кадиллак"],
"chrysler": ["크라이슬러", "Крайслер"],
"jeep": ["지프", "Джип"],
"ford": ["포드", "Форд"],
"tesla": ["테슬라", "Тесла"],
"lexus": ["렉서스", "Лексус"],
"infiniti": ["인피니티", "Инфинити"],
"acura": ["아큐라", "어큐라", "Акура"],
"subaru": ["스바루", "Субару"],
"mitsubishi": ["미쓰비시", "미쯔비시", "Мицубиси"],
"mazda": ["마쓰다", "마쯔다", "Мазда"],
"suzuki": ["스즈키", "Сузуки"],
"ferrari": ["페라리", "Феррари"],
"lamborghini": ["람보르기니", "Ламборгини"],
"maserati": ["마세라티", "Мазерати"],
"bentley": ["벤틀리", "Бентли"],
"rolls-royce": ["롤스로이스", "Роллс-Ройс"],
"aston martin": ["애스턴마틴", "Астон Мартин"],
"mclaren": ["맥라렌", "Макларен"],
"ssangyong": ["쌍용", "СсангЙонг", "KG모빌리티(쌍용)", "KG Mobility"],
"kg mobility": ["KG모빌리티(쌍용)", "KG Mobility", "쌍용", "СсангЙонг"],
"daewoo": ["대우", "Дэу"],
"dodge": ["닷지", "Додж"],
"hummer": ["험머", "Хаммер"],
"gmc": ["GMC"],
"polestar": ["폴스타", "Полстар"],
"ds": ["DS", "시트로엥/DS", "Ситроен/DS"],
"renault korea": ["르노코리아(삼성)", "Рено Корея", "르노삼성", "Рено Самсунг"],
"saab": ["사브", "Сааб"],
"lotus": ["로터스", "Лотус"],
"alfa romeo": ["알파로메오", "Альфа Ромео", "알파 로메오"],
"fiat": ["피아트", "Фиат"],
"smart": ["스마트", "Смарт"],
"skoda": ["스코다", "Шкода"],
"daihatsu": ["다이하츠", "다이하쯔", "Дайхатсу"],
"koenigsegg": ["코닉세그", "Кёнигсегг"],
"pagani": ["파가니", "Пагани"],
"bugatti": ["부가티", "Бугатти"],
"maybach": ["마이바흐", "Майбах"],
"rivian": ["리비안", "Ривиан"],
"lucid": ["루시드", "Люсид"],
"rimac": ["리막", "Римак"],
"byd": ["BYD"],
"geely": ["지리", "Джили"],
"dongfeng": ["동풍소콘", "Донгфенг Сокон"],
"baic": ["북기은상", "BAIC"],
"skywell": ["신위안", "Skywell"],
"ineos": ["이네오스", "Инеос"],
"mercury": ["머큐리", "Меркурий"],
"scion": ["사이언", "Scion"],
"buick": ["뷰익", "Бьюик"],
"saturn": ["새턴", "Saturn"],
"oldsmobile": ["올즈모빌", "Oldsmobile"],
"pontiac": ["폰티악", "Понтиак"],
"foton": ["포톤", "Foton"],
"mg rover": ["MG로버", "MG Rover"],
"mitsuoka": ["미쯔오까", "Мицуока"],
"other": ["기타 제조사", "기타 수입차", "Другие", "Другие импортные"],
}
def expand_allowed_brands(brands: set[str] | None) -> set[str] | None:
"""Расширяет набор английских брендов всеми известными alias'ами (корейский + русский)."""
if not brands:
return brands
expanded = set(brands)
for brand in brands:
aliases = BRAND_ENGLISH_ALIASES.get(brand.lower(), [])
for alias in aliases:
expanded.add(alias.lower())
return expanded
# Модели / комплектации: корейский → английский
MODEL_EXACT_TRANSLATIONS = {
"1시리즈": "1 Series",
"2시리즈": "2 Series",
"3시리즈": "3 Series",
"4시리즈": "4 Series",
"5시리즈": "5 Series",
"6시리즈": "6 Series",
"7시리즈": "7 Series",
"8시리즈": "8 Series",
"A-클래스": "A-Class",
"B-클래스": "B-Class",
"C-클래스": "C-Class",
"E-클래스": "E-Class",
"S-클래스": "S-Class",
"SL-클래스": "SL-Class",
"SLK-클래스": "SLK-Class",
"GLA-클래스": "GLA-Class",
"GLB-클래스": "GLB-Class",
"GLC-클래스": "GLC-Class",
"GLE-클래스": "GLE-Class",
"GLS-클래스": "GLS-Class",
"V-클래스": "V-Class",
"팰리세이드": "Palisade",
"싼타페": "Santa Fe",
"투싼": "Tucson",
"아반떼": "Avante",
"쏘나타": "Sonata",
"쏘나타 디 엣지": "Sonata The Edge",
"그랜저": "Grandeur",
"아이오닉5": "Ioniq 5",
"아이오닉6": "Ioniq 6",
"아이오닉": "Ioniq",
"넥쏘": "Nexo",
"벨로스터": "Veloster",
"코나": "Kona",
"캐스퍼": "Casper",
"베뉴": "Venue",
"스타렉스": "Starex",
"스타리아": "Staria",
"아슬란": "Aslan",
"베라크루즈": "Veracruz",
"카니발": "Carnival",
"쏘렌토": "Sorento",
"스포티지": "Sportage",
"에스컬레이드": "Escalade",
"프리미어": "Premier",
"1세대": "1st gen",
"2세대": "2nd gen",
"3세대": "3rd gen",
"4세대": "4th gen",
"5세대": "5th gen",
"6세대": "6th gen",
"7세대": "7th gen",
"8세대": "8th gen",
"모닝": "Morning",
"레이": "Ray",
"셀토스": "Seltos",
"니로": "Niro",
"스팅어": "Stinger",
"쏘울": "Soul",
"모하비": "Mohave",
"프라이드": "Pride",
"스토닉": "Stonic",
"카렌스": "Carens",
"제네시스": "Genesis",
"스파크": "Spark",
"말리부": "Malibu",
"트랙스": "Trax",
"크루즈": "Cruze",
"올란도": "Orlando",
"트레일블레이저": "Trailblazer",
"콜로라도": "Colorado",
"트래버스": "Traverse",
"캡티바": "Captiva",
"이쿼녹스": "Equinox",
"아베오": "Aveo",
"볼트": "Volt",
"임팔라": "Impala",
"맥스크루즈": "Max Cruz",
"렉스턴 스포츠 칸": "Rexton Sport Khan",
"렉스턴 스포츠": "Rexton Sport",
"렉스턴": "Rexton",
"토레스": "Torres",
"티볼리 에어": "Tivoli Air",
"티볼리 아머": "Tivoli Armor",
"티볼리": "Tivoli",
"코란도 투리스모": "Korando Turismo",
"코란도 스포츠": "Korando Sport",
"코란도": "Korando",
"체어맨": "Chairman",
"오피러스": "Opirus",
"익스플로러": "Explorer",
"머스탱": "Mustang",
"카이엔": "Cayenne",
"카이엔 쿠페": "Cayenne Coupe",
"파나메라": "Panamera",
"파나메라 스포츠 투리스모": "Panamera Sport Turismo",
"마칸": "Macan",
"타이칸": "Taycan",
"타이칸 크로스 투리스모": "Taycan Cross Turismo",
"918 스파이더": "918 Spyder",
"718 스파이더": "718 Spyder",
"4.0 스파이더": "4.0 Spyder",
"박스터": "Boxster",
"카이맨": "Cayman",
"911 카브리올레": "911 Cabriolet",
"카레라": "Carrera",
"티구안": "Tiguan",
"골프": "Golf",
"제타": "Jetta",
"투아렉": "Touareg",
"파사트": "Passat",
"아테온": "Arteon",
"쿠퍼": "Cooper",
"컨트리맨": "Countryman",
"클럽맨": "Clubman",
"디스커버리": "Discovery",
"레인지로버": "Range Rover",
"이보크": "Evoque",
"벨라": "Velar",
"디펜더": "Defender",
"랭글러": "Wrangler",
"체로키": "Cherokee",
"레니게이드": "Renegade",
"컴패스": "Compass",
"루비콘": "Rubicon",
"에비에이터": "Aviator",
"르반떼": "Levante",
"기블리": "Ghibli",
"어코드": "Accord",
"캠리": "Camry",
"프리우스": "Prius",
"타스만": "Tasman",
"토러스": "Taurus",
"카마로": "Camaro",
"비틀": "Beetle",
"e-트론": "e-tron",
"글래디에이터": "Gladiator",
"노틸러스": "Nautilus",
"시로코": "Scirocco",
"아틀라스": "Atlas",
"알파드": "Alphard",
"익스페디션": "Expedition",
"익스프레스Van": "Express Van",
"짐니": "Jimny",
"쿠가": "Kuga",
"클리오": "Clio",
"파일럿": "Pilot",
"콰트로포르테": "Quattroporte",
"고스트": "Ghost",
"팬텀": "Phantom",
"레이스": "Wraith",
"던": "Dawn",
"컬리넌": "Cullinan",
"스펙터": "Spectre",
"벤테이가": "Bentayga",
"컨티넨탈 GT": "Continental GT",
"플라잉스퍼": "Flying Spur",
"우라칸": "Huracan",
"아벤타도르": "Aventador",
"우루스": "Urus",
"레부엘토": "Revuelto",
"로마": "Roma",
"포르토피노": "Portofino",
"SF90 스트라달레": "SF90 Stradale",
"F8 트리뷰토": "F8 Tributo",
"그란투리스모": "GranTurismo",
"반퀴시": "Vanquish",
"밴티지": "Vantage",
"AMG GT 4도어 쿠페": "AMG GT 4-Door Coupe",
}
def _register_partner_models() -> None:
for _ko, _en in _PARTNER_TABLES["models"].items():
MODEL_EXACT_TRANSLATIONS.setdefault(_ko, _en)
_compact = re.sub(r"(?<=\d)\s+시리즈", "시리즈", _ko)
_compact = re.sub(r"(?<=[A-Za-z])\s+클래스", "-클래스", _compact)
if _compact != _ko:
MODEL_EXACT_TRANSLATIONS.setdefault(_compact, _en)
_register_partner_models()
MODEL_PHRASE_TRANSLATIONS = {
"올 뉴": "All New",
"더 뉴": "New",
"디 올 뉴": "The All New",
"베리 뉴": "Very New",
"베리": "Very",
"뷰티풀": "Beautiful",
"넥스트": "Next",
"디 ": "The ",
"디 엣지": "The Edge",
"액티브 투어러": "Active Tourer",
"투어러": "Tourer",
"그란쿠페": "GranCoupe",
"그란 쿠페": "GranCoupe",
"스포츠 칸": "Sport Khan",
"스포츠": "Sport",
"하이브리드": "Hybrid",
"일렉트릭": "Electric",
"네오": "Neo",
"트랜스폼": "Transform",
"프라임": "Prime",
"마스터": "Master",
"어반": "Urban",
"볼드": "Bold",
"아머": "Armor",
"에어": "Air",
}
MODEL_TOKEN_TRANSLATIONS = {
"시리즈": "Series",
"클래스": "Class",
"세대": "gen",
"뉴": "New",
"올": "All",
"칸": "Khan",
}
MODEL_TRANSLATIONS = {
# --- Топливо / тип двигателя ---
"가솔린": "Gasoline",
"디젤": "Diesel",
"하이브리드": "Hybrid",
"전기": "Electric",
"터보": "Turbo",
"일렉트릭": "Electric",
"일렉트리파이드": "Electrified",
"블루텍": "BlueTEC",
"블루모션": "BlueMotion",
# --- Кузов / конструкция ---
"쿠페": "Coupe",
"세단": "Sedan",
"왜건": "Wagon",
"웨건": "Wagon",
"해치백": "Hatchback",
"컨버터블": "Convertible",
"카브리올레": "Cabriolet",
"크로스오버": "Crossover",
"리무진": "Limousine",
"하이리무진": "High Limousine",
"하이리무진(특장업체)": "High Limousine (Special)",
"스포트백": "Sportback",
"스파이더": "Spyder",
"밴": "Van",
"픽업": "Pickup",
"캠핑카": "Camper",
# --- КПП / привод ---
"오토": "Auto",
"수동": "Manual",
"후륜": "RWD",
"전륜": "FWD",
"사륜": "AWD",
"4모션": "4Motion",
# --- Трим / комплектация ---
"프레스티지": "Prestige",
"시그니처": "Signature",
"럭셔리": "Luxury",
"스포츠": "Sport",
"롱레인지": "Long Range",
"스탠다드": "Standard",
"플러스": "Plus",
"리미티드": "Limited",
"스페셜": "Special",
"모던": "Modern",
"익스클루시브": "Exclusive",
"캘리그래피": "Calligraphy",
"노블레스": "Noblesse",
"트림": "Trim",
"프리미엄": "Premium",
"프리미어": "Premier",
"플래티넘": "Platinum",
"플래티넘Ⅱ": "Platinum II",
"인스퍼레이션": "Inspiration",
"트렌디": "Trendy",
"다이나믹": "Dynamic",
"프라임": "Prime",
"초이스": "Choice",
"르블랑": "Le Blanc",
"프레지던트": "President",
"인스크립션": "Inscription",
"볼드": "Bold",
"밸류": "Value",
"퓨어": "Pure",
"엑셀런스": "Excellence",
"얼티메이트": "Ultimate",
"에센셜": "Essential",
"슈프림": "Supreme",
"에어": "Air",
"어반": "Urban",
"탑": "Top",
"코어": "Core",
"엘리트": "Elite",
"셀렉션": "Selection",
"브라이트": "Bright",
"클래식": "Classic",
"고급형": "Deluxe",
"최고급형": "Top Grade",
"기본형": "Base",
"디럭스": "Deluxe",
"디럭스팩": "Deluxe Pack",
"마이스터": "Meister",
"마스터": "Master",
"마스터즈": "Masters",
"어메이징": "Amazing",
"파워": "Power",
"리저브": "Reserve",
"퍼펙트": "Perfect",
"리얼": "Real",
"울트라": "Ultra",
"세이프티": "Safety",
"퍼포먼스": "Performance",
"인텔리전트": "Intelligent",
"컬렉션": "Collection",
"브릴리언트": "Brilliant",
"어드밴티지": "Advantage",
"파이니스트": "Finest",
"런치팩": "Launch Pack",
"페이버드": "Favored",
"엘레강스": "Elegance",
"하이": "High",
"블랙라벨": "Black Label",
"플럭스": "Flux",
"테크": "Tech",
"프로": "Pro",
"베스트": "Best",
"라이즈": "Rise",
"액티브": "Active",
"롱": "Long",
"패키지": "Package",
"에디션": "Edition",
"디자인": "Design",
"라운지": "Lounge",
"아트": "Art",
"팝": "Pop",
"조이": "Joy",
"레드라인": "Redline",
"스타일": "Style",
"블랙": "Black",
"라이트": "Light",
"온라인": "Online",
"론지튜드": "Longitude",
"오버랜드": "Overland",
"컴페티션": "Competition",
"올스페이스": "Allspace",
"스타": "Star",
# --- Поколение / номер ---
"1세대": "1st gen",
"2세대": "2nd gen",
"3세대": "3rd gen",
"4세대": "4th gen",
"5세대": "5th gen",
"6세대": "6th gen",
"7세대": "7th gen",
"8세대": "8th gen",
"10세대": "10th gen",
"더": "",
"뉴": "New",
"넥스트": "Next",
"올": "All",
"그란": "Gran",
"그랜드": "Grand",
# --- Посадочных мест ---
"2인승": "2-seater",
"3인승": "3-seater",
"5인승": "5-seater",
"6인승": "6-seater",
"7인승": "7-seater",
"9인승": "9-seater",
"11인승": "11-seater",
"12인승": "12-seater",
# --- Двери ---
"4도어": "4-door",
"5도어": "5-door",
# --- Модели Hyundai ---
"팰리세이드": "Palisade",
"싼타페": "Santa Fe",
"투싼": "Tucson",
"아반떼": "Avante",
"쏘나타": "Sonata",
"그랜저": "Grandeur",
"아이오닉5": "Ioniq 5",
"아이오닉6": "Ioniq 6",
"아이오닉": "Ioniq",
"넥쏘": "Nexo",
"벨로스터": "Veloster",
"코나": "Kona",
"캐스퍼": "Casper",
"베뉴": "Venue",
"스타렉스": "Starex",
"스타리아": "Staria",
"에쿠스(신형)": "Equus",
"엑센트(신형)": "Accent",
"아슬란": "Aslan",
"베라크루즈": "Veracruz",
# --- Модели Kia ---
"카니발": "Carnival",
"쏘렌토": "Sorento",
"스포티지": "Sportage",
"모닝": "Morning",
"뉴모닝": "New Morning",
"레이": "Ray",
"셀토스": "Seltos",
"니로": "Niro",
"스팅어": "Stinger",
"쏘울": "Soul",
"모하비": "Mohave",
"프라이드": "Pride",
"스토닉": "Stonic",
# --- Модели Chevrolet/GM ---
"스파크": "Spark",
"말리부": "Malibu",
"트랙스": "Trax",
"크루즈": "Cruze",
"올란도": "Orlando",
"트레일블레이저": "Trailblazer",
"콜로라도": "Colorado",
"트래버스": "Traverse",
"캡티바": "Captiva",
"이쿼녹스": "Equinox",
"아베오": "Aveo",
"볼트": "Volt",
"임팔라": "Impala",
"맥스크루즈": "Max Cruz",
# --- Модели Genesis ---
"제네시스": "Genesis",
# --- Модели иностранных авто ---
"익스플로러": "Explorer",
"머스탱": "Mustang",
"카이엔": "Cayenne",
"파나메라": "Panamera",
"마칸": "Macan",
"타이칸": "Taycan",
"박스터": "Boxster",
"카이맨": "Cayman",
"카레라": "Carrera",
"티구안": "Tiguan",
"골프": "Golf",
"제타": "Jetta",
"투아렉": "Touareg",
"파사트": "Passat",
"아테온": "Arteon",
"콰트로": "quattro",
"아방가르드": "Avantgarde",
"쿠퍼": "Cooper",
"컨트리맨": "Countryman",
"클럽맨": "Clubman",
"디스커버리": "Discovery",
"레인지로버": "Range Rover",
"이보크": "Evoque",
"벨라": "Velar",
"디펜더": "Defender",
"랭글러": "Wrangler",
"체로키": "Cherokee",
"체로키(KL)": "Cherokee (KL)",
"체로키(WL)": "Cherokee (WL)",
"레니게이드": "Renegade",
"컴패스": "Compass",
"루비콘": "Rubicon",
"에비에이터": "Aviator",
"에스컬레이드": "Escalade",
"컨티넨탈": "Continental",
"르반떼": "Levante",
"기블리": "Ghibli",
"알티마": "Altima",
"오딧세이": "Odyssey",
"어코드": "Accord",
"캠리": "Camry",
"시에나": "Sienna",
"프리우스": "Prius",
"크로스컨트리": "Cross Country",
"타스만": "Tasman",
"고스트": "Ghost",
"팬텀": "Phantom",
"레이스": "Wraith",
"던": "Dawn",
"컬리넌": "Cullinan",
"스펙터": "Spectre",
"벤테이가": "Bentayga",
"컨티넨탈 GT": "Continental GT",
"플라잉스퍼": "Flying Spur",
"우라칸": "Huracan",
"아벤타도르": "Aventador",
"우루스": "Urus",
"레부엘토": "Revuelto",
"로마": "Roma",
"포르토피노": "Portofino",
"SF90 스트라달레": "SF90 Stradale",
"F8 트리뷰토": "F8 Tributo",
"콰트로포르테": "Quattroporte",
"그란투리스모": "GranTurismo",
"반퀴시": "Vanquish",
"밴티지": "Vantage",
"시에라": "Sierra",
"시빅": "Civic",
"레인저": "Ranger",
"콜레오스": "Koleos",
"그랑 콜레오스": "Grand Koleos",
"페이톤": "Phaeton",
"티록": "T-Roc",
"큐브": "Cube",
"크라운": "Crown",
"라이즈": "Rise",
"시로코": "Scirocco",
"이오스": "Eos",
"비틀": "Beetle",
"폴로": "Polo",
# --- Служебные / категории ---
"(세부등급": "(sub-trim",
"없음)": "none)",
"(렌터카)": "(rental)",
"(렌터카용)": "(for rental)",
"LPG(렌터카용)": "LPG (for rental)",
"LPI(렌터카용)": "LPI (for rental)",
"(택시형)": "(taxi)",
"택시형": "taxi",
"어린이보호차": "child transport",
"카고": "Cargo",
"모델": "Model",
"스마트": "Smart",
"기아": "Kia",
"마이바흐": "Maybach",
"레인지": "Range",
}
COLOR_TRANSLATIONS = {
"흰색": "white",
"검정색": "black",
"회색": "gray",
"은색": "silver",
"은회색": "silver-gray",
"진회색": "dark gray",
"쥐색": "mouse gray",
"밝은 회색": "light gray",
"청색": "blue",
"파란색": "blue",
"진청색": "dark blue",
"밝은 청색": "light blue",
"하늘색": "sky blue",
"녹색": "green",
"초록색": "green",
"진녹색": "dark green",
"밝은 녹색": "light green",
"담녹색": "light green",
"적색": "red",
"빨간색": "red",
"진적색": "dark red",
"밝은 적색": "light red",
"주황색": "orange",
"노란색": "yellow",
"갈색": "brown",
"베이지색": "beige",
"아이보리": "ivory",
"크림색": "cream",
"보라색": "purple",
"분홍색": "pink",
"금색": "gold",
"연금색": "light gold",
"명은색": "bright silver",
"은하색": "galaxy silver",
"자주색": "burgundy",
"진주색": "pearl",
"청금색": "blue-gold",
"청은색": "blue-silver",
"흑청색": "black-blue",
"흑적색": "black-red",
"흑녹색": "black-green",
"흑회색": "black-gray",
"흰청색": "white-blue",
"흰적색": "white-red",
"흰녹색": "white-green",
"흰회색": "white-gray",
"청적색": "blue-red",
"청녹색": "blue-green",
"청회색": "blue-gray",
"적녹색": "red-green",
"적회색": "red-gray",
"녹회색": "green-gray",
"기타": "other",
"기타색상": "other",
"특별색상": "special",
"무채색": "neutral",
"메탈릭": "metallic",
"펄": "pearl",
"무광": "matte",
"광택": "glossy",
}
# Тип кузова: корейский → enum
BODY_TYPE_MAP = {
"세단": "SEDAN",
"쿠페": "COUPE",
"컨버터블": "CONVERTIBLE",
"카브리올레": "CONVERTIBLE",
"오픈카": "OPEN",
"해치백": "HATCHBACK",
"왜건": "WAGON",
"스테이션왜건": "STATION_WAGON",
"SUV": "SUV",
"SUV ": "SUV",
"크로스오버": "SUV",
"픽업트럭": "PICKUP",
"픽업": "PICKUP",
"밴": "VAN",
"미니밴": "MINIVAN",
"트럭": "TRUCK",
"쿠페형 SUV": "SUV",
"버스": "BUS",
"기타": "OTHER",
"승용차": "SEDAN",
"승합차": "VAN",
"화물차": "TRUCK",
"특장차": "SPECIAL",
"경차": "COMPACT",
"중형차": "MID_SIZE",
"대형차": "FULL_SIZE",
"스포츠카": "SPORTS",
"클래식카": "CLASSIC",
"전기차": "ELECTRIC",
"하이브리드": "HYBRID",
}
# Определение body_type по модели (fallback)
MODEL_BODY_TYPE_MAP: dict[str, str] = {
# SUV / кроссоверы
"투싼": "SUV", "싼타페": "SUV", "팰리세이드": "SUV", "코나": "SUV", "베뉴": "SUV",
"크레타": "SUV", "넥쏘": "SUV", "아이오닉5": "SUV", "아이오닉6": "SEDAN",
"셀토스": "SUV", "스포티지": "SUV", "쏘렌토": "SUV", "모하비": "SUV",
"니로": "SUV", "EV6": "SUV", "EV9": "SUV",
"티볼리": "SUV", "코란도": "SUV", "렉스턴": "SUV", "토레스": "SUV", "액티언": "SUV",
"GV70": "SUV", "GV80": "SUV", "GV60": "SUV",
"X1": "SUV", "X2": "SUV", "X3": "SUV", "X4": "SUV", "X5": "SUV", "X6": "SUV", "X7": "SUV", "iX": "SUV", "iX3": "SUV",
"Q2": "SUV", "Q3": "SUV", "Q4": "SUV", "Q5": "SUV", "Q7": "SUV", "Q8": "SUV",
"GLA": "SUV", "GLB": "SUV", "GLC": "SUV", "GLE": "SUV", "GLS": "SUV",
"카이엔": "SUV", "마칸": "SUV",
"레인지로버": "SUV", "디스커버리": "SUV", "디펜더": "SUV",
"래브4": "SUV", "하이랜더": "SUV", "FJ 크루져": "SUV",
"CR-V": "SUV", "HR-V": "SUV",
"티구안": "SUV", "투아렉": "SUV", "아틀라스": "SUV", "ID.4": "SUV",
"트레일블레이저": "SUV", "이쿼녹스": "SUV", "트래버스": "SUV", "트랙스": "SUV", "캡티바": "SUV", "올란도": "SUV",
"XC40": "SUV", "XC60": "SUV", "XC90": "SUV", "V40": "HATCHBACK", "V60": "STATION_WAGON", "V90": "STATION_WAGON",
"F-PACE": "SUV", "E-PACE": "SUV", "I-PACE": "SUV",
"모델 Y": "SUV", "Model Y": "SUV", "모델 X": "SUV", "Model X": "SUV",
"RX": "SUV", "NX": "SUV", "UX": "SUV", "LX": "SUV",
"QX50": "SUV", "QX60": "SUV", "QX80": "SUV",
"QM3": "SUV", "QM6": "SUV",
"레니게이드": "SUV", "컴패스": "SUV", "체로키": "SUV", "랭글러": "SUV", "글래디에이터": "PICKUP",
"에비에이터": "SUV", "노틸러스": "SUV", "코세어": "SUV",
"르반떼": "SUV", "그레칼레": "SUV",
"Z4": "CONVERTIBLE",
# 세단
"쏘나타": "SEDAN", "아반떼": "SEDAN", "그랜저": "SEDAN", "K5": "SEDAN",
"K8": "SEDAN", "K9": "SEDAN", "K3": "SEDAN", "K7": "SEDAN",
"G70": "SEDAN", "G80": "SEDAN", "G90": "SEDAN",
"1시리즈": "HATCHBACK",
"2시리즈 그란쿠페": "SEDAN", "2시리즈 액티브투어러": "MINIVAN",
"3시리즈": "SEDAN", "3시리즈 GT": "SEDAN",
"5시리즈": "SEDAN", "7시리즈": "SEDAN", "i4": "SEDAN", "i7": "SEDAN",
"A1": "HATCHBACK", "A3": "SEDAN", "A4": "SEDAN", "A6": "SEDAN", "A7": "SEDAN", "A8": "SEDAN",
"A-클래스": "SEDAN", "CLA": "SEDAN",
"C-클래스": "SEDAN", "E-클래스": "SEDAN", "S-클래스": "SEDAN",
"캠리": "SEDAN", "아발론": "SEDAN", "프리우스": "SEDAN",
"어코드": "SEDAN", "시빅": "SEDAN",
"파사트": "SEDAN", "제타": "SEDAN", "아테온": "SEDAN",
"말리부": "SEDAN", "크루즈": "SEDAN", "임팔라": "SEDAN",
"S60": "SEDAN", "S90": "SEDAN",
"IS": "SEDAN", "ES": "SEDAN", "GS": "SEDAN", "LS": "SEDAN",
"Q50": "SEDAN", "Q70": "SEDAN",
"모델 3": "SEDAN", "Model 3": "SEDAN", "모델 S": "SEDAN", "Model S": "SEDAN",
"SM6": "SEDAN", "SM5": "SEDAN",
# 해치백
"스파크": "HATCHBACK", "모닝": "HATCHBACK", "레이": "HATCHBACK",
"i30": "HATCHBACK", "벨로스터": "HATCHBACK",
"골프": "HATCHBACK", "폴로": "HATCHBACK", "ID.3": "HATCHBACK",
"캐스퍼": "HATCHBACK",
# 미니밴/밴
"카니발": "MINIVAN", "스타리아": "MINIVAN", "스타렉스": "MINIVAN",
"V-클래스": "MINIVAN", "비토": "VAN",
"시에나": "MINIVAN", "알파드": "MINIVAN",
# 트럭/픽업
"포터": "PICKUP", "봉고": "PICKUP", "마이티": "TRUCK",
"타코마": "PICKUP", "터프": "PICKUP",
"콜로라도": "PICKUP", "실버라도": "PICKUP",
# 쿠페
"4시리즈": "COUPE", "8시리즈": "COUPE", "2시리즈": "COUPE",
"A5": "COUPE", "TT": "COUPE",
"RC": "COUPE", "LC": "COUPE",
}

62
encar_scraper/worker/celery_app.py
View File

@@ -1,17 +1,9 @@
# Инициализация Celery-приложения и периодических задач для Encar.
import logging
from celery import Celery
from celery.signals import beat_init
from redis import Redis
from sqlalchemy import func, select
from ..core.config import settings
logger = logging.getLogger("encar_scraper.worker.celery_app")
_INITIAL_SYNC_BOOTSTRAP_KEY = "encar:bootstrap:initial_sync_enqueued"
def _broker_url() -> str:
return settings.celery.broker_url or settings.redis.url
@@ -48,9 +40,17 @@ celery_app.conf.update(
},
result_expires=86400,
worker_hijack_root_logger=False,
# Следующий запуск планирует сама задача в finally через apply_async(countdown=...),
# чтобы час отсчитывался от завершения прогона, а не от старта.
beat_schedule={},
beat_schedule={
"periodic-encar-sync": {
"task": "encar_scraper.worker.tasks.encar_sync_listing_task",
"schedule": settings.celery.encar_beat_interval_minutes * 60.0,
"args": (),
"kwargs": {
"car_type": "all",
},
"options": {"queue": "encar"},
},
},
task_routes={
"encar_scraper.worker.tasks.*": {"queue": "encar"},
},
@@ -59,46 +59,6 @@ celery_app.conf.update(
celery_app.autodiscover_tasks(["encar_scraper.worker"])
@beat_init.connect
def _schedule_initial_sync_on_first_start(**kwargs):
"""Ставит первый sync сразу после первого запуска проекта на пустой БД."""
from ..storage.db import PersistenceService
from ..storage.models import SyncRun
from .tasks import encar_sync_listing_task
persistence = PersistenceService(settings)
redis_client: Redis | None = None
try:
with persistence.session_scope() as session:
total_runs = session.execute(select(func.count(SyncRun.id))).scalar() or 0
if total_runs:
return
redis_client = Redis.from_url(
_broker_url(),
decode_responses=True,
socket_timeout=10,
socket_connect_timeout=5,
)
marked = redis_client.set(_INITIAL_SYNC_BOOTSTRAP_KEY, "1", nx=True, ex=600)
if not marked:
logger.info("Initial Encar sync already bootstrapped on this startup")
return
encar_sync_listing_task.apply_async(kwargs={"car_type": "all"}, queue="encar")
logger.info("Scheduled initial Encar sync because sync_runs table is empty")
except Exception:
logger.warning("Failed to schedule initial Encar sync", exc_info=True)
finally:
persistence.engine.dispose()
if redis_client is not None:
try:
redis_client.close()
except Exception:
pass
from celery.signals import setup_logging as celery_setup_logging

81
encar_scraper/worker/tasks.py
View File

@@ -5,8 +5,6 @@ from threading import Event, Thread
import uuid
from celery import shared_task
from celery.exceptions import MaxRetriesExceededError, Retry
from celery.signals import worker_process_init
from redis import Redis
from ..core.config import Settings
@@ -22,25 +20,6 @@ _persistence: PersistenceService | None = None
_redis: Redis | None = None
@worker_process_init.connect
def _reset_globals_after_fork(**kwargs):
global _persistence, _redis
# После fork закрываем унаследованные TCP-сокеты родителя (иначе дети
# делят одни connection pool'ы → повреждение данных).
if _persistence is not None:
try:
_persistence.engine.dispose()
except Exception:
logger.debug("engine.dispose after fork failed", exc_info=True)
if _redis is not None:
try:
_redis.close()
except Exception:
logger.debug("redis.close after fork failed", exc_info=True)
_persistence = None
_redis = None
def _sync_listing_lock_ttl_seconds() -> int:
settings = Settings()
return max(settings.celery.task_time_limit + 120, 300)
@@ -168,11 +147,6 @@ def encar_sync_listing_task(
heartbeat_stop: Event | None = None
heartbeat_thread: Thread | None = None
run_id: int | None = None
# Планировать следующий прогон только когда текущий прогон завершён
# (success или final failure после исчерпания retries). Retry сам
# перепланирует задачу — не дублируем apply_async в этом случае.
reschedule_next = True
try:
heartbeat_stop, heartbeat_thread = _start_lock_heartbeat(
@@ -188,9 +162,6 @@ def encar_sync_listing_task(
effective_limit = limit or rc.sync.limit
effective_lane = lane or rc.sync.lane or "encar"
# --- Фиксируем начало sync_run ---
run_id = persistence.start_sync_run(effective_lane)
car_type_map = {"all": "A", "domestic": "Y", "import": "N"}
filters = EncarFilters(
car_type=car_type_map.get(car_type, "A"),
@@ -209,7 +180,7 @@ def encar_sync_listing_task(
set(rc.filters.exclude_brands) if rc.filters.exclude_brands else None
)
scraper = EncarScraper(persistence=persistence)
scraper = EncarScraper()
result = scraper.sync_listing(
limit=effective_limit,
filters=filters,
@@ -217,20 +188,9 @@ def encar_sync_listing_task(
redis_client=redis_client,
allowed_brands=allowed_brands,
excluded_brands=excluded_brands,
runtime_filters=rc.filters,
probe_all_photos=rc.sync.probe_all_photos,
)
# --- Запись результата в sync_runs ---
persistence.finish_sync_run(
run_id,
status="success",
ids_fetched=result.get("items_collected", 0),
cars_upserted=result.get("cars_synced", 0),
cars_failed=result.get("cars_failed", 0),
images_upserted=0,
)
summary = {
"task_id": task_id,
"source": "encar",
@@ -248,27 +208,7 @@ def encar_sync_listing_task(
except Exception as exc:
logger.error("encar_sync_listing_task failed: %s", exc, exc_info=True)
try:
if run_id is not None:
persistence.finish_sync_run(
run_id,
status="failed",
ids_fetched=0,
cars_upserted=0,
cars_failed=0,
images_upserted=0,
error_summary=str(exc)[:500],
)
except Exception:
logger.warning("Failed to record sync_run failure", exc_info=True)
try:
# Retry сам перепланирует задачу — не планируем следующий прогон
reschedule_next = False
raise self.retry(exc=exc)
except MaxRetriesExceededError:
# Retries исчерпаны — запускаем следующий прогон по расписанию
reschedule_next = True
raise
raise self.retry(exc=exc)
finally:
if heartbeat_stop is not None:
heartbeat_stop.set()
@@ -276,21 +216,6 @@ def encar_sync_listing_task(
heartbeat_thread.join(timeout=max(1.0, min(5.0, lock_ttl / 10)))
if lock_acquired:
_release_lock_if_owner(redis_client, ENCAR_SYNC_LOCK_KEY, owner_token)
if reschedule_next:
try:
interval_minutes = Settings().celery.encar_beat_interval_minutes
countdown = max(60, int(interval_minutes) * 60)
self.apply_async(
kwargs={"car_type": car_type},
countdown=countdown,
queue="encar",
)
logger.info(
"Next encar_sync_listing_task scheduled in %d minutes",
countdown // 60,
)
except Exception:
logger.warning("Failed to schedule next sync run", exc_info=True)
@shared_task(
@@ -306,7 +231,7 @@ def encar_sync_vehicle_task(self, vehicle_url: str, lane: str = "encar"):
persistence.create_tables()
try:
scraper = EncarScraper(persistence=persistence)
scraper = EncarScraper()
result = scraper.sync_vehicle(vehicle_url, lane=lane)
logger.info("encar_sync_vehicle_task completed: %s", vehicle_url)
return {

3
pyproject.toml
View File

@@ -1,5 +1,5 @@
[build-system]
requires = ["setuptools>=68"]
requires = ["setuptools>=68", "wheel"]
build-backend = "setuptools.build_meta"
[project]
@@ -23,7 +23,6 @@ dependencies = [
[project.optional-dependencies]
dev = [
"httpx>=0.28.0",
"pytest>=8.3.0",
"pytest-cov>=5.0.0",
]

125
tests/test_encar.py
View File

@@ -27,7 +27,7 @@ class TestEncarMapper(unittest.TestCase):
self.assertEqual(record.origin_id, "encar:41421262")
self.assertEqual(record.parser_id, "encar:41421262")
self.assertEqual(record.brand, "Hyundai")
self.assertEqual(record.model, "G90")
self.assertEqual(record.model, "G90 (RS4)")
self.assertEqual(record.year, 2023)
self.assertEqual(record.price, 85900000) # 8590 만원 × 10000
self.assertEqual(record.mileage, 37923)
@@ -40,133 +40,10 @@ class TestEncarMapper(unittest.TestCase):
url = ENCAR_DETAIL_URL_TEMPLATE.format(vehicle_id="41421262")
self.assertEqual(EncarMapper()._extract_vehicle_id(url), "41421262")
def test_map_to_car_record_uses_base_model_without_trim(self):
payload = {
"Id": 41854374,
"Manufacturer": "현대",
"Model": "Genesis",
"Badge": "BH330 Grand",
"BadgeDetail": "Prime팩",
"FormYear": "2011",
"Price": 1200,
"Mileage": 100000,
"Photos": [
{"location": "/carpicture01/pic4185/41854374_001.jpg", "ordering": 1},
],
}
record = EncarMapper().map_to_car_record(
ENCAR_DETAIL_URL_TEMPLATE.format(vehicle_id="41854374"),
payload,
)
self.assertEqual(record.brand, "Hyundai")
self.assertEqual(record.model, "Genesis")
self.assertEqual(record.origin_id, "encar:41854374")
def test_map_to_car_record_resolves_numeric_id_from_photos(self):
payload = {
"Id": "car-BOqLLkjkgbpStlVJEJvbRG",
"Manufacturer": "Kia",
"Model": "K3",
"FormYear": "2018",
"Price": 990,
"Mileage": 75000,
"Photos": [
{"location": "/carpicture01/pic4187/41878507_001.jpg", "ordering": 1},
],
}
record = EncarMapper().map_to_car_record("", payload)
self.assertEqual(record.origin_id, "encar:41878507")
self.assertEqual(record.parser_id, "encar:41878507")
self.assertIn("carid=41878507", record.origin_url)
def test_translate_model_keeps_english_text(self):
mapper = EncarMapper()
self.assertEqual(mapper._translate_model("All New Carnival"), "Carnival")
self.assertEqual(mapper._translate_model("New Kia Ray"), "Kia Ray")
self.assertEqual(mapper._translate_model("K5 Hybrid 3rd gen"), "K5 Hybrid")
self.assertEqual(mapper._translate_model("New K5 3rd gen"), "K5")
self.assertEqual(mapper._translate_model("New Tiguan"), "Tiguan")
self.assertEqual(mapper._translate_model("New QM6"), "QM6")
self.assertEqual(mapper._translate_model("Discovery Sport 2nd gen"), "Discovery Sport")
def test_translate_model_handles_korean_fragments(self):
mapper = EncarMapper()
cases = {
"5시리즈 (G30)": "5 Series",
"S-클래스 W223": "S-Class W223",
"Sonata 디 엣지(DN8)": "Sonata The Edge",
"렉스턴 Sport 칸": "Rexton Sport Khan",
"티볼리 Air": "Tivoli Air",
"코란도 투리스모": "Korando Turismo",
"Kia Carnival 4세대": "Kia Carnival",
"Cadillac 에스컬레이드 5세대": "Cadillac Escalade",
"K7 프리미어": "K7 Premier",
}
for raw, expected in cases.items():
with self.subTest(raw=raw):
self.assertEqual(mapper._translate_model(raw), expected)
def test_translate_model_drops_untranslated_korean(self):
mapper = EncarMapper()
cases = {
"슈팅브레이크": "",
"The New G70 슈팅브레이크": "G70",
"G70 (슈팅브레이크)": "G70",
}
for raw, expected in cases.items():
with self.subTest(raw=raw):
self.assertEqual(mapper._translate_model(raw), expected)
def test_translate_color_normalizes_api_spacing(self):
mapper = EncarMapper()
self.assertEqual(mapper._translate_color("흰색"), "white")
self.assertEqual(mapper._translate_color("흰 색"), "white")
self.assertEqual(mapper._translate_color("검정색"), "black")
self.assertEqual(mapper._translate_color("진주색"), "pearl")
self.assertEqual(mapper._translate_color("쥐색"), "mouse gray")
self.assertEqual(mapper._translate_color("은회색"), "silver-gray")
self.assertEqual(mapper._translate_color("하늘색"), "sky blue")
self.assertEqual(mapper._translate_color("자주색"), "burgundy")
def test_apply_batch_color_uses_spec_color_name(self):
payload = {
"Id": 42456458,
"Manufacturer": "Lincoln",
"Model": "Corsair",
"FormYear": "2022",
"Price": 2690,
"Mileage": 26600,
}
mapper = EncarMapper()
record = mapper.map_to_car_record(
ENCAR_DETAIL_URL_TEMPLATE.format(vehicle_id="42456458"),
payload,
)
mapper.apply_batch_color(record, {"spec": {"colorName": "흰 색"}})
self.assertEqual(record.brand, "Lincoln")
self.assertEqual(record.model, "Corsair")
self.assertEqual(record.year, 2022)
self.assertEqual(record.price, 26900000)
self.assertEqual(record.mileage, 26600)
self.assertEqual(record.color, "white")
class TestEncarScraper(unittest.TestCase):
def test_collect_listing_limits_results(self):
scraper = EncarScraper()
scraper._fetch_listing_page = MagicMock(side_effect=[
{"SearchResults": [{"Id": 1}, {"Id": 2}]},
])

3
tests/test_load.py
View File

@@ -195,14 +195,11 @@ class TestAPILoad(unittest.TestCase):
persistence: PersistenceService = app.state.persistence
persistence.upsert_cars_batch([_make_car(i, images=4) for i in range(200)])
cls.app = app
cls.client = TestClient(app)
@classmethod
def tearDownClass(cls):
import os
cls.client.close()
cls.app.state.persistence.engine.dispose()
if os.path.exists("test_api_load.db"):
os.remove("test_api_load.db")