* 모델 서빙 구축에서 NHN, 라벨리온과 한달간 삽질했다.
암호화키를 NHN or 라벨리온에서 만들어서 개별에게 할당하였기 때문에 내 소프트웨어 코드가 오픈될 위험이 있어 NHN 서버에는 코드를 올릴 수 없었고 AI 모델 서빙만 NHN 서버에 실행해야 했었는데 모델 서빙이 외부(내 솔루션 서버)로 전달되지 않았다.
NHN에서 나에게 서버 설정 권한을 주지 않으니 문제 현상만 똑같이 반복하여 전달하고 나는 기다릴 수 밖에 없었는데 이야기를 들어보니 하나의 서버에서 여러명이 쓰다보니 NHN에서는 도커로 사용자별로 컨테이너를 만들었고 NHN 서버의 인바운드 포트를 열었지만 도커 포트를 열지 않아 결과적으로 내가 아무리 모델 서빙을 한다한들 외부로 이 값이 전달되지 않았다.
다행히 과제 제출시작 기한 1주일 전에 문제는 해결됐지만 NHN, 라벨리온 모두 나에게 서버 설정 권한을 주지 않으면 사용자는 단지 기다릴 수 밖에 없고 과제 제출 기한에 영향을 줄 수 밖에 없었다.
No 1. 서버 접속
* Powershell CMD
ssh -i "D:\Cloud\NHN\N2025-0173.pem" ubuntu@114.110.xxx.xxx -p 30xxx
* Linux CMD
ssh -i N2025-0173.pem ubuntu@114.110.xxx.xxx -p 30xxx
No 2. 모델 다운로드
ubuntu@56122307d8f9:~$ mkdir top
ubuntu@56122307d8f9:~$ cd top/
ubuntu@56122307d8f9:~/top$ git clone https://github.com/rebellions-sw/rbln-model-zoo.git
Cloning into 'rbln-model-zoo'...
remote: Enumerating objects: 2826, done.
remote: Counting objects: 100% (385/385), done.
remote: Compressing objects: 100% (203/203), done.
remote: Total 2826 (delta 274), reused 182 (delta 182), pack-reused 2441 (from 2)
Receiving objects: 100% (2826/2826), 33.62 MiB | 20.33 MiB/s, done.
Resolving deltas: 100% (1422/1422), done.
No 3. 모델 submodule update
ubuntu@56122307d8f9:~/top$ cd rbln-model-zoo
ubuntu@56122307d8f9:~/top/rbln-model-zoo$ git submodule update --init --recursive
Submodule 'huggingface/third-party-models/video-to-video/cosmos_transfer1/cosmos-transfer1' (https://github.com/nvidia-cosmos/cosmos-transfer1.git) registered for path 'huggingface/third-party-models/video-to-video/cosmos_transfer1/cosmos-transfer1'
Submodule 'pytorch/vision/action_recognition/motionbert/MotionBERT' (https://github.com/Walter0807/MotionBERT.git) registered for path 'pytorch/vision/action_recognition/motionbert/MotionBERT'
Submodule 'pytorch/vision/detection/yolov10/ultralytics' (https://github.com/ultralytics/ultralytics.git) registered for path 'pytorch/vision/detection/yolov10/ultralytics'
Submodule 'pytorch/vision/detection/yolov3/ultralytics' (https://github.com/ultralytics/ultralytics.git) registered for path 'pytorch/vision/detection/yolov3/ultralytics'
Submodule 'pytorch/vision/detection/yolov3/yolov3' (https://github.com/ultralytics/yolov3.git) registered for path 'pytorch/vision/detection/yolov3/yolov3'
Submodule 'pytorch/vision/detection/yolov4/yolov4' (https://github.com/WongKinYiu/PyTorch_YOLOv4.git) registered for path 'pytorch/vision/detection/yolov4/yolov4'
Submodule 'pytorch/vision/detection/yolov5/yolov5' (https://github.com/ultralytics/yolov5.git) registered for path 'pytorch/vision/detection/yolov5/yolov5'
Submodule 'pytorch/vision/detection/yolov6/YOLOv6' (https://github.com/meituan/YOLOv6.git) registered for path 'pytorch/vision/detection/yolov6/YOLOv6'
Submodule 'pytorch/vision/detection/yolov7/yolov7' (https://github.com/WongKinYiu/yolov7.git) registered for path 'pytorch/vision/detection/yolov7/yolov7'
Submodule 'pytorch/vision/detection/yolov8/ultralytics' (https://github.com/ultralytics/ultralytics.git) registered for path 'pytorch/vision/detection/yolov8/ultralytics'
Submodule 'pytorch/vision/detection/yolox/YOLOX' (https://github.com/Megvii-BaseDetection/YOLOX) registered for path 'pytorch/vision/detection/yolox/YOLOX'
Submodule 'pytorch/vision/face_detection/yolov5-face/yolov5-face' (https://github.com/deepcam-cn/yolov5-face.git) registered for path 'pytorch/vision/face_detection/yolov5-face/yolov5-face'
Submodule 'pytorch/vision/pose_estimation/motionbert/MotionBERT' (https://github.com/Walter0807/MotionBERT.git) registered for path 'pytorch/vision/pose_estimation/motionbert/MotionBERT'
Submodule 'pytorch_dynamo/vision/detection/yolov3/ultralytics' (https://github.com/ultralytics/ultralytics.git) registered for path 'pytorch_dynamo/vision/detection/yolov3/ultralytics'
Submodule 'pytorch_dynamo/vision/detection/yolov3/yolov3' (https://github.com/ultralytics/yolov3.git) registered for path 'pytorch_dynamo/vision/detection/yolov3/yolov3'
Submodule 'pytorch_dynamo/vision/detection/yolov4/yolov4' (https://github.com/WongKinYiu/PyTorch_YOLOv4.git) registered for path 'pytorch_dynamo/vision/detection/yolov4/yolov4'
Submodule 'pytorch_dynamo/vision/detection/yolov5/yolov5' (https://github.com/ultralytics/yolov5.git) registered for path 'pytorch_dynamo/vision/detection/yolov5/yolov5'
Submodule 'pytorch_dynamo/vision/detection/yolov6/YOLOv6' (https://github.com/meituan/YOLOv6.git) registered for path 'pytorch_dynamo/vision/detection/yolov6/YOLOv6'
Submodule 'pytorch_dynamo/vision/detection/yolox/YOLOX' (https://github.com/Megvii-BaseDetection/YOLOX) registered for path 'pytorch_dynamo/vision/detection/yolox/YOLOX'
Submodule 'pytorch_dynamo/vision/image_segmentation/sam2/sam2' (https://github.com/facebookresearch/sam2.git) registered for path 'pytorch_dynamo/vision/image_segmentation/sam2/sam2'
Submodule 'pytorch_dynamo/vision/video_segmentation/sam2/sam2' (https://github.com/facebookresearch/sam2.git) registered for path 'pytorch_dynamo/vision/video_segmentation/sam2/sam2'
Cloning into '/home/ubuntu/top/rbln-model-zoo/huggingface/third-party-models/video-to-video/cosmos_transfer1/cosmos-transfer1'...
Cloning into '/home/ubuntu/top/rbln-model-zoo/pytorch/vision/action_recognition/motionbert/MotionBERT'...
Cloning into '/home/ubuntu/top/rbln-model-zoo/pytorch/vision/detection/yolov10/ultralytics'...
Cloning into '/home/ubuntu/top/rbln-model-zoo/pytorch/vision/detection/yolov3/ultralytics'...
Cloning into '/home/ubuntu/top/rbln-model-zoo/pytorch/vision/detection/yolov3/yolov3'...
Cloning into '/home/ubuntu/top/rbln-model-zoo/pytorch/vision/detection/yolov4/yolov4'...
Cloning into '/home/ubuntu/top/rbln-model-zoo/pytorch/vision/detection/yolov5/yolov5'...
Cloning into '/home/ubuntu/top/rbln-model-zoo/pytorch/vision/detection/yolov6/YOLOv6'...
Cloning into '/home/ubuntu/top/rbln-model-zoo/pytorch/vision/detection/yolov7/yolov7'...
Cloning into '/home/ubuntu/top/rbln-model-zoo/pytorch/vision/detection/yolov8/ultralytics'...
Cloning into '/home/ubuntu/top/rbln-model-zoo/pytorch/vision/detection/yolox/YOLOX'...
Cloning into '/home/ubuntu/top/rbln-model-zoo/pytorch/vision/face_detection/yolov5-face/yolov5-face'...
Cloning into '/home/ubuntu/top/rbln-model-zoo/pytorch/vision/pose_estimation/motionbert/MotionBERT'...
Cloning into '/home/ubuntu/top/rbln-model-zoo/pytorch_dynamo/vision/detection/yolov3/ultralytics'...
Cloning into '/home/ubuntu/top/rbln-model-zoo/pytorch_dynamo/vision/detection/yolov3/yolov3'...
Cloning into '/home/ubuntu/top/rbln-model-zoo/pytorch_dynamo/vision/detection/yolov4/yolov4'...
Cloning into '/home/ubuntu/top/rbln-model-zoo/pytorch_dynamo/vision/detection/yolov5/yolov5'...
Cloning into '/home/ubuntu/top/rbln-model-zoo/pytorch_dynamo/vision/detection/yolov6/YOLOv6'...
Cloning into '/home/ubuntu/top/rbln-model-zoo/pytorch_dynamo/vision/detection/yolox/YOLOX'...
Cloning into '/home/ubuntu/top/rbln-model-zoo/pytorch_dynamo/vision/image_segmentation/sam2/sam2'...
Cloning into '/home/ubuntu/top/rbln-model-zoo/pytorch_dynamo/vision/video_segmentation/sam2/sam2'...
Submodule path 'huggingface/third-party-models/video-to-video/cosmos_transfer1/cosmos-transfer1': checked out '5ce1a195783d356d3b0cceb4cad293eeec629b62'
Submodule path 'pytorch/vision/action_recognition/motionbert/MotionBERT': checked out '1839f099ce9f128342c8f5499478ace328c0df4a'
Submodule path 'pytorch/vision/detection/yolov10/ultralytics': checked out 'e72f19f5cfd5fe7bb8b09a156933e96bd90581bf'
Submodule path 'pytorch/vision/detection/yolov3/ultralytics': checked out 'e72f19f5cfd5fe7bb8b09a156933e96bd90581bf'
Submodule path 'pytorch/vision/detection/yolov3/yolov3': checked out '316c2e371c51500cb1a03762386a4ccf74dd09c5'
Submodule path 'pytorch/vision/detection/yolov4/yolov4': checked out 'eb5f1663ed0743660b8aa749a43f35f505baa325'
Submodule path 'pytorch/vision/detection/yolov5/yolov5': checked out '86fd1ab270cb2f7e53ee7412cd4a0650bf4bcc51'
Submodule path 'pytorch/vision/detection/yolov6/YOLOv6': checked out '4364f29bf3244f2e73d0c42a103cd7a9cbb16ca9'
Submodule path 'pytorch/vision/detection/yolov7/yolov7': checked out '84932d70fb9e2932d0a70e4a1f02a1d6dd1dd6ca'
Submodule path 'pytorch/vision/detection/yolov8/ultralytics': checked out 'e72f19f5cfd5fe7bb8b09a156933e96bd90581bf'
Submodule path 'pytorch/vision/detection/yolox/YOLOX': checked out 'bb9185c095dfd7a8015a1b82f3e9a065090860b8'
Submodule path 'pytorch/vision/face_detection/yolov5-face/yolov5-face': checked out '152c688d551aefb973b7b589fb0691c93dab3564'
Submodule path 'pytorch/vision/pose_estimation/motionbert/MotionBERT': checked out '1839f099ce9f128342c8f5499478ace328c0df4a'
Submodule path 'pytorch_dynamo/vision/detection/yolov3/ultralytics': checked out 'e72f19f5cfd5fe7bb8b09a156933e96bd90581bf'
Submodule path 'pytorch_dynamo/vision/detection/yolov3/yolov3': checked out '316c2e371c51500cb1a03762386a4ccf74dd09c5'
Submodule path 'pytorch_dynamo/vision/detection/yolov4/yolov4': checked out 'eb5f1663ed0743660b8aa749a43f35f505baa325'
Submodule path 'pytorch_dynamo/vision/detection/yolov5/yolov5': checked out '86fd1ab270cb2f7e53ee7412cd4a0650bf4bcc51'
Submodule path 'pytorch_dynamo/vision/detection/yolov6/YOLOv6': checked out '4364f29bf3244f2e73d0c42a103cd7a9cbb16ca9'
Submodule path 'pytorch_dynamo/vision/detection/yolox/YOLOX': checked out 'bb9185c095dfd7a8015a1b82f3e9a065090860b8'
Submodule path 'pytorch_dynamo/vision/image_segmentation/sam2/sam2': checked out '2b90b9f5ceec907a1c18123530e92e794ad901a4'
Submodule path 'pytorch_dynamo/vision/video_segmentation/sam2/sam2': checked out '2b90b9f5ceec907a1c18123530e92e794ad901a4'
ubuntu@56122307d8f9:~/top/rbln-model-zoo$
No 4. SDK 설치
* 컴파일러 설치
- 컴파일러 설치를 위해서는 미리 아래 사이트에서 계정 생성을 해야 함.
https://rebellions.ai/request-form-rbln-sdk/
- 이후 생성된 계정으로 아래와 같이 수행
ubuntu@56122307d8f9:~$ pip install -i https://pypi.rbln.ai/simple rebel-compiler
Defaulting to user installation because normal site-packages is not writeable
Looking in indexes: https://pypi.rbln.ai/simple
User for pypi.rbln.ai: boyzoneXXXX
Password: XXXXXXX
Collecting rebel-compiler
Downloading https://pypi.rbln.ai/api/package/rebel-compiler/rebel_compiler-0.9.1-cp310-cp310-manylinux_2_28_x86_64.whl (131.9 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 131.9/131.9 MB 17.5 MB/s eta 0:00:00
Collecting cloudpickle
Downloading cloudpickle-3.1.1-py3-none-any.whl (20 kB)
Requirement already satisfied: numpy in ./.local/lib/python3.10/site-packages (from rebel-compiler) (2.2.6)
Requirement already satisfied: psutil in ./.local/lib/python3.10/site-packages (from rebel-compiler) (7.1.2)
Collecting prettytable>=3.8
Downloading prettytable-3.16.0-py3-none-any.whl (33 kB)
Collecting ml_dtypes
Downloading ml_dtypes-0.5.3-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (4.9 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.9/4.9 MB 19.7 MB/s eta 0:00:00
Collecting attrs
Downloading attrs-25.4.0-py3-none-any.whl (67 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 67.6/67.6 KB 18.4 MB/s eta 0:00:00
Requirement already satisfied: typing_extensions in ./.local/lib/python3.10/site-packages (from rebel-compiler) (4.15.0)
Collecting decorator
Downloading decorator-5.2.1-py3-none-any.whl (9.2 kB)
Requirement already satisfied: tqdm in ./.local/lib/python3.10/site-packages (from rebel-compiler) (4.67.1)
Collecting cryptography
Downloading cryptography-46.0.3-cp38-abi3-manylinux_2_34_x86_64.whl (4.5 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.5/4.5 MB 47.3 MB/s eta 0:00:00
Requirement already satisfied: packaging in ./.local/lib/python3.10/site-packages (from rebel-compiler) (25.0)
Collecting tornado
Downloading tornado-6.5.2-cp39-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (443 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 443.9/443.9 KB 94.3 MB/s eta 0:00:00
Collecting scipy
Downloading scipy-1.15.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (37.7 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 37.7/37.7 MB 48.6 MB/s eta 0:00:00
Collecting wcwidth
Downloading wcwidth-0.2.14-py2.py3-none-any.whl (37 kB)
Collecting cffi>=2.0.0
Downloading cffi-2.0.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (216 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 216.5/216.5 KB 89.4 MB/s eta 0:00:00
Collecting pycparser
Downloading pycparser-2.23-py3-none-any.whl (118 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 118.1/118.1 KB 72.3 MB/s eta 0:00:00
Installing collected packages: wcwidth, tornado, scipy, pycparser, ml_dtypes, decorator, cloudpickle, attrs, prettytable, cffi, cryptography, rebel-compiler
WARNING: The script tvmc is installed in '/home/ubuntu/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed attrs-25.4.0 cffi-2.0.0 cloudpickle-3.1.1 cryptography-46.0.3 decorator-5.2.1 ml_dtypes-0.5.3 prettytable-3.16.0 pycparser-2.23 rebel-compiler-0.9.1 scipy-1.15.3 tornado-6.5.2 wcwidth-0.2.14
ubuntu@56122307d8f9:~$
* huggingface 모델 추론을 위한 패키지 설치
ubuntu@56122307d8f9:~$ pip install optimum-rbln
Defaulting to user installation because normal site-packages is not writeable
Collecting optimum-rbln
Downloading optimum_rbln-0.9.1-py3-none-any.whl (463 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 463.6/463.6 KB 15.3 MB/s eta 0:00:00
Collecting torchvision<=0.22.0
Downloading torchvision-0.22.0-cp310-cp310-manylinux_2_28_x86_64.whl (7.4 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.4/7.4 MB 59.3 MB/s eta 0:00:00
Collecting transformers==4.51.3
Downloading transformers-4.51.3-py3-none-any.whl (10.4 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.4/10.4 MB 67.8 MB/s eta 0:00:00
Collecting packaging>=24.1
Downloading packaging-25.0-py3-none-any.whl (66 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 66.5/66.5 KB 20.1 MB/s eta 0:00:00
Collecting torch==2.7.0
Downloading torch-2.7.0-cp310-cp310-manylinux_2_28_x86_64.whl (865.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 865.2/865.2 MB 4.0 MB/s eta 0:00:00
Collecting accelerate>=1.0.1
Downloading accelerate-1.11.0-py3-none-any.whl (375 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 375.8/375.8 KB 95.1 MB/s eta 0:00:00
Collecting torchaudio<=2.7.0
Downloading torchaudio-2.7.0-cp310-cp310-manylinux_2_28_x86_64.whl (3.4 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.4/3.4 MB 86.7 MB/s eta 0:00:00
Collecting diffusers==0.35.1
Downloading diffusers-0.35.1-py3-none-any.whl (4.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.1/4.1 MB 85.6 MB/s eta 0:00:00
Collecting filelock
Downloading filelock-3.20.0-py3-none-any.whl (16 kB)
Collecting regex!=2019.12.17
Downloading regex-2025.10.23-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (791 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 791.5/791.5 KB 92.6 MB/s eta 0:00:00
Collecting Pillow
Downloading pillow-12.0.0-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (7.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.0/7.0 MB 88.1 MB/s eta 0:00:00
Collecting requests
Downloading requests-2.32.5-py3-none-any.whl (64 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 64.7/64.7 KB 35.4 MB/s eta 0:00:00
Collecting importlib_metadata
Downloading importlib_metadata-8.7.0-py3-none-any.whl (27 kB)
Collecting numpy
Downloading numpy-2.2.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.8 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.8/16.8 MB 67.9 MB/s eta 0:00:00
Collecting huggingface-hub>=0.34.0
Downloading huggingface_hub-1.0.0-py3-none-any.whl (503 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 503.9/503.9 KB 99.1 MB/s eta 0:00:00
Collecting safetensors>=0.3.1
Downloading safetensors-0.6.2-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (485 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 485.8/485.8 KB 96.3 MB/s eta 0:00:00
Collecting nvidia-cusparselt-cu12==0.6.3
Downloading nvidia_cusparselt_cu12-0.6.3-py3-none-manylinux2014_x86_64.whl (156.8 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 156.8/156.8 MB 17.8 MB/s eta 0:00:00
Collecting nvidia-nvjitlink-cu12==12.6.85
Downloading nvidia_nvjitlink_cu12-12.6.85-py3-none-manylinux2010_x86_64.manylinux_2_12_x86_64.whl (19.7 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.7/19.7 MB 61.8 MB/s eta 0:00:00
Collecting nvidia-nvtx-cu12==12.6.77
Downloading nvidia_nvtx_cu12-12.6.77-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (89 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 89.3/89.3 KB 32.6 MB/s eta 0:00:00
Collecting fsspec
Downloading fsspec-2025.9.0-py3-none-any.whl (199 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 199.3/199.3 KB 80.8 MB/s eta 0:00:00
Collecting typing-extensions>=4.10.0
Downloading typing_extensions-4.15.0-py3-none-any.whl (44 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 44.6/44.6 KB 22.3 MB/s eta 0:00:00
Collecting nvidia-cufile-cu12==1.11.1.6
Downloading nvidia_cufile_cu12-1.11.1.6-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (1.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 87.2 MB/s eta 0:00:00
Collecting nvidia-cusparse-cu12==12.5.4.2
Downloading nvidia_cusparse_cu12-12.5.4.2-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (216.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 216.6/216.6 MB 13.8 MB/s eta 0:00:00
Collecting sympy>=1.13.3
Downloading sympy-1.14.0-py3-none-any.whl (6.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.3/6.3 MB 87.7 MB/s eta 0:00:00
Collecting nvidia-cuda-runtime-cu12==12.6.77
Downloading nvidia_cuda_runtime_cu12-12.6.77-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (897 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 897.7/897.7 KB 88.1 MB/s eta 0:00:00
Collecting nvidia-cuda-nvrtc-cu12==12.6.77
Downloading nvidia_cuda_nvrtc_cu12-12.6.77-py3-none-manylinux2014_x86_64.whl (23.7 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 23.7/23.7 MB 25.4 MB/s eta 0:00:00
Collecting nvidia-cuda-cupti-cu12==12.6.80
Downloading nvidia_cuda_cupti_cu12-12.6.80-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (8.9 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.9/8.9 MB 23.3 MB/s eta 0:00:00
Collecting nvidia-cudnn-cu12==9.5.1.17
Downloading nvidia_cudnn_cu12-9.5.1.17-py3-none-manylinux_2_28_x86_64.whl (571.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 571.0/571.0 MB 6.2 MB/s eta 0:00:00
Collecting jinja2
Downloading jinja2-3.1.6-py3-none-any.whl (134 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 134.9/134.9 KB 58.9 MB/s eta 0:00:00
Collecting nvidia-curand-cu12==10.3.7.77
Downloading nvidia_curand_cu12-10.3.7.77-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (56.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56.3/56.3 MB 37.1 MB/s eta 0:00:00
Collecting networkx
Downloading networkx-3.4.2-py3-none-any.whl (1.7 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.7/1.7 MB 87.4 MB/s eta 0:00:00
Collecting nvidia-cusolver-cu12==11.7.1.2
Downloading nvidia_cusolver_cu12-11.7.1.2-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (158.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 158.2/158.2 MB 18.3 MB/s eta 0:00:00
Collecting nvidia-nccl-cu12==2.26.2
Downloading nvidia_nccl_cu12-2.26.2-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (201.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 201.3/201.3 MB 14.3 MB/s eta 0:00:00
Collecting nvidia-cublas-cu12==12.6.4.1
Downloading nvidia_cublas_cu12-12.6.4.1-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (393.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 393.1/393.1 MB 8.8 MB/s eta 0:00:00
Collecting nvidia-cufft-cu12==11.3.0.4
Downloading nvidia_cufft_cu12-11.3.0.4-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (200.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 200.2/200.2 MB 15.3 MB/s eta 0:00:00
Collecting triton==3.3.0
Downloading triton-3.3.0-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (156.4 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 156.4/156.4 MB 18.1 MB/s eta 0:00:00
Collecting tokenizers<0.22,>=0.21
Downloading tokenizers-0.21.4-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.1/3.1 MB 92.0 MB/s eta 0:00:00
Collecting pyyaml>=5.1
Downloading pyyaml-6.0.3-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (770 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 770.3/770.3 KB 91.9 MB/s eta 0:00:00
Collecting tqdm>=4.27
Downloading tqdm-4.67.1-py3-none-any.whl (78 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.5/78.5 KB 43.5 MB/s eta 0:00:00
Collecting huggingface-hub>=0.34.0
Downloading huggingface_hub-0.36.0-py3-none-any.whl (566 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 566.1/566.1 KB 96.2 MB/s eta 0:00:00
Requirement already satisfied: setuptools>=40.8.0 in /usr/lib/python3/dist-packages (from triton==3.3.0->torch==2.7.0->optimum-rbln) (59.6.0)
Collecting psutil
Downloading psutil-7.1.2-cp36-abi3-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl (258 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 258.7/258.7 KB 106.7 MB/s eta 0:00:00
Collecting hf-xet<2.0.0,>=1.1.3
Downloading hf_xet-1.2.0-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.3/3.3 MB 37.0 MB/s eta 0:00:00
Collecting mpmath<1.4,>=1.1.0
Downloading mpmath-1.3.0-py3-none-any.whl (536 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 536.2/536.2 KB 94.0 MB/s eta 0:00:00
Collecting zipp>=3.20
Downloading zipp-3.23.0-py3-none-any.whl (10 kB)
Collecting MarkupSafe>=2.0
Downloading markupsafe-3.0.3-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (20 kB)
Collecting idna<4,>=2.5
Downloading idna-3.11-py3-none-any.whl (71 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 71.0/71.0 KB 35.3 MB/s eta 0:00:00
Collecting urllib3<3,>=1.21.1
Downloading urllib3-2.5.0-py3-none-any.whl (129 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 129.8/129.8 KB 66.8 MB/s eta 0:00:00
Collecting charset_normalizer<4,>=2
Downloading charset_normalizer-3.4.4-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (153 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 153.6/153.6 KB 63.9 MB/s eta 0:00:00
Collecting certifi>=2017.4.17
Downloading certifi-2025.10.5-py3-none-any.whl (163 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 163.3/163.3 KB 78.1 MB/s eta 0:00:00
Installing collected packages: nvidia-cusparselt-cu12, mpmath, zipp, urllib3, typing-extensions, triton, tqdm, sympy, safetensors, regex, pyyaml, psutil, Pillow, packaging, nvidia-nvtx-cu12, nvidia-nvjitlink-cu12, nvidia-nccl-cu12, nvidia-curand-cu12, nvidia-cufile-cu12, nvidia-cuda-runtime-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-cupti-cu12, nvidia-cublas-cu12, numpy, networkx, MarkupSafe, idna, hf-xet, fsspec, filelock, charset_normalizer, certifi, requests, nvidia-cusparse-cu12, nvidia-cufft-cu12, nvidia-cudnn-cu12, jinja2, importlib_metadata, nvidia-cusolver-cu12, huggingface-hub, torch, tokenizers, diffusers, transformers, torchvision, torchaudio, accelerate, optimum-rbln
WARNING: The scripts proton and proton-viewer are installed in '/home/ubuntu/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The script tqdm is installed in '/home/ubuntu/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The script isympy is installed in '/home/ubuntu/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The scripts f2py and numpy-config are installed in '/home/ubuntu/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The script normalizer is installed in '/home/ubuntu/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The scripts hf, huggingface-cli and tiny-agents are installed in '/home/ubuntu/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The scripts torchfrtrace and torchrun are installed in '/home/ubuntu/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The script diffusers-cli is installed in '/home/ubuntu/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The script transformers-cli is installed in '/home/ubuntu/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The scripts accelerate, accelerate-config, accelerate-estimate-memory, accelerate-launch and accelerate-merge-weights are installed in '/home/ubuntu/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed MarkupSafe-3.0.3 Pillow-12.0.0 accelerate-1.11.0 certifi-2025.10.5 charset_normalizer-3.4.4 diffusers-0.35.1 filelock-3.20.0 fsspec-2025.9.0 hf-xet-1.2.0 huggingface-hub-0.36.0 idna-3.11 importlib_metadata-8.7.0 jinja2-3.1.6 mpmath-1.3.0 networkx-3.4.2 numpy-2.2.6 nvidia-cublas-cu12-12.6.4.1 nvidia-cuda-cupti-cu12-12.6.80 nvidia-cuda-nvrtc-cu12-12.6.77 nvidia-cuda-runtime-cu12-12.6.77 nvidia-cudnn-cu12-9.5.1.17 nvidia-cufft-cu12-11.3.0.4 nvidia-cufile-cu12-1.11.1.6 nvidia-curand-cu12-10.3.7.77 nvidia-cusolver-cu12-11.7.1.2 nvidia-cusparse-cu12-12.5.4.2 nvidia-cusparselt-cu12-0.6.3 nvidia-nccl-cu12-2.26.2 nvidia-nvjitlink-cu12-12.6.85 nvidia-nvtx-cu12-12.6.77 optimum-rbln-0.9.1 packaging-25.0 psutil-7.1.2 pyyaml-6.0.3 regex-2025.10.23 requests-2.32.5 safetensors-0.6.2 sympy-1.14.0 tokenizers-0.21.4 torch-2.7.0 torchaudio-2.7.0 torchvision-0.22.0 tqdm-4.67.1 transformers-4.51.3 triton-3.3.0 typing-extensions-4.15.0 urllib3-2.5.0 zipp-3.23.0
ubuntu@56122307d8f9:~$
* Rebellions NPU 전용 vLLM 엔진 설치
ubuntu@56122307d8f9:~$ pip install vllm-rbln
Defaulting to user installation because normal site-packages is not writeable
Collecting vllm-rbln
Downloading vllm_rbln-0.9.1-py3-none-any.whl (176 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 176.7/176.7 KB 5.7 MB/s eta 0:00:00
Requirement already satisfied: transformers==4.51.3 in ./.local/lib/python3.10/site-packages (from vllm-rbln) (4.51.3)
Collecting datasets
Downloading datasets-4.3.0-py3-none-any.whl (506 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 506.8/506.8 KB 15.3 MB/s eta 0:00:00
Collecting setuptools-scm>=8
Downloading setuptools_scm-9.2.2-py3-none-any.whl (62 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 63.0/63.0 KB 27.9 MB/s eta 0:00:00
Collecting setuptools>=64
Downloading setuptools-80.9.0-py3-none-any.whl (1.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 18.4 MB/s eta 0:00:00
Requirement already satisfied: optimum-rbln>=0.9.1 in ./.local/lib/python3.10/site-packages (from vllm-rbln) (0.9.1)
Collecting qwen_vl_utils
Downloading qwen_vl_utils-0.0.14-py3-none-any.whl (8.1 kB)
Requirement already satisfied: pyyaml in ./.local/lib/python3.10/site-packages (from vllm-rbln) (6.0.3)
Collecting vllm==0.9.1
Downloading vllm-0.9.1-cp38-abi3-manylinux1_x86_64.whl (394.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 394.6/394.6 MB 9.3 MB/s eta 0:00:00
Requirement already satisfied: requests in ./.local/lib/python3.10/site-packages (from transformers==4.51.3->vllm-rbln) (2.32.5)
Requirement already satisfied: huggingface-hub<1.0,>=0.30.0 in ./.local/lib/python3.10/site-packages (from transformers==4.51.3->vllm-rbln) (0.36.0)
Requirement already satisfied: tokenizers<0.22,>=0.21 in ./.local/lib/python3.10/site-packages (from transformers==4.51.3->vllm-rbln) (0.21.4)
Requirement already satisfied: tqdm>=4.27 in ./.local/lib/python3.10/site-packages (from transformers==4.51.3->vllm-rbln) (4.67.1)
Requirement already satisfied: safetensors>=0.4.3 in ./.local/lib/python3.10/site-packages (from transformers==4.51.3->vllm-rbln) (0.6.2)
Requirement already satisfied: packaging>=20.0 in ./.local/lib/python3.10/site-packages (from transformers==4.51.3->vllm-rbln) (25.0)
Requirement already satisfied: regex!=2019.12.17 in ./.local/lib/python3.10/site-packages (from transformers==4.51.3->vllm-rbln) (2025.10.23)
Requirement already satisfied: filelock in ./.local/lib/python3.10/site-packages (from transformers==4.51.3->vllm-rbln) (3.20.0)
Requirement already satisfied: numpy>=1.17 in ./.local/lib/python3.10/site-packages (from transformers==4.51.3->vllm-rbln) (2.2.6)
Collecting einops
Downloading einops-0.8.1-py3-none-any.whl (64 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 64.4/64.4 KB 34.4 MB/s eta 0:00:00
Collecting python-json-logger
Downloading python_json_logger-4.0.0-py3-none-any.whl (15 kB)
Collecting protobuf
Downloading protobuf-6.33.0-cp39-abi3-manylinux2014_x86_64.whl (323 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 323.2/323.2 KB 96.1 MB/s eta 0:00:00
Collecting prometheus_client>=0.18.0
Downloading prometheus_client-0.23.1-py3-none-any.whl (61 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.1/61.1 KB 32.2 MB/s eta 0:00:00
Collecting llguidance<0.8.0,>=0.7.11
Downloading llguidance-0.7.30-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (15.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 15.0/15.0 MB 33.3 MB/s eta 0:00:00
Collecting watchfiles
Downloading watchfiles-1.1.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (455 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 455.6/455.6 KB 93.7 MB/s eta 0:00:00
Collecting aiohttp
Downloading aiohttp-3.13.1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (1.7 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.7/1.7 MB 93.9 MB/s eta 0:00:00
Collecting lm-format-enforcer<0.11,>=0.10.11
Downloading lm_format_enforcer-0.10.12-py3-none-any.whl (44 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 44.3/44.3 KB 24.1 MB/s eta 0:00:00
Collecting outlines==0.1.11
Downloading outlines-0.1.11-py3-none-any.whl (87 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 87.6/87.6 KB 47.3 MB/s eta 0:00:00
Requirement already satisfied: cloudpickle in ./.local/lib/python3.10/site-packages (from vllm==0.9.1->vllm-rbln) (3.1.1)
Collecting lark==1.2.2
Downloading lark-1.2.2-py3-none-any.whl (111 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 111.0/111.0 KB 55.6 MB/s eta 0:00:00
Requirement already satisfied: pillow in ./.local/lib/python3.10/site-packages (from vllm==0.9.1->vllm-rbln) (12.0.0)
Collecting py-cpuinfo
Downloading py_cpuinfo-9.0.0-py3-none-any.whl (22 kB)
Collecting pyzmq>=25.0.0
Downloading pyzmq-27.1.0-cp310-cp310-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl (854 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 854.1/854.1 KB 92.7 MB/s eta 0:00:00
Collecting tiktoken>=0.6.0
Downloading tiktoken-0.12.0-cp310-cp310-manylinux_2_28_x86_64.whl (1.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 95.9 MB/s eta 0:00:00
Collecting opentelemetry-api>=1.26.0
Downloading opentelemetry_api-1.38.0-py3-none-any.whl (65 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 65.9/65.9 KB 39.8 MB/s eta 0:00:00
Collecting numba==0.61.2
Downloading numba-0.61.2-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.8 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.8/3.8 MB 95.6 MB/s eta 0:00:00
Requirement already satisfied: torchvision==0.22.0 in ./.local/lib/python3.10/site-packages (from vllm==0.9.1->vllm-rbln) (0.22.0)
Collecting compressed-tensors==0.10.1
Downloading compressed_tensors-0.10.1-py3-none-any.whl (116 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 117.0/117.0 KB 61.2 MB/s eta 0:00:00
Requirement already satisfied: typing_extensions>=4.10 in ./.local/lib/python3.10/site-packages (from vllm==0.9.1->vllm-rbln) (4.15.0)
Collecting fastapi[standard]>=0.115.0
Downloading fastapi-0.120.1-py3-none-any.whl (108 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 108.3/108.3 KB 73.0 MB/s eta 0:00:00
Collecting depyf==0.18.0
Downloading depyf-0.18.0-py3-none-any.whl (38 kB)
Collecting mistral_common[opencv]>=1.5.4
Downloading mistral_common-1.8.5-py3-none-any.whl (6.5 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.5/6.5 MB 53.1 MB/s eta 0:00:00
Collecting opentelemetry-sdk>=1.26.0
Downloading opentelemetry_sdk-1.38.0-py3-none-any.whl (132 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 132.3/132.3 KB 63.4 MB/s eta 0:00:00
Collecting prometheus-fastapi-instrumentator>=7.0.0
Downloading prometheus_fastapi_instrumentator-7.1.0-py3-none-any.whl (19 kB)
Collecting opentelemetry-semantic-conventions-ai>=0.4.1
Downloading opentelemetry_semantic_conventions_ai-0.4.13-py3-none-any.whl (6.1 kB)
Requirement already satisfied: torch==2.7.0 in ./.local/lib/python3.10/site-packages (from vllm==0.9.1->vllm-rbln) (2.7.0)
Collecting opentelemetry-exporter-otlp>=1.26.0
Downloading opentelemetry_exporter_otlp-1.38.0-py3-none-any.whl (7.0 kB)
Collecting gguf>=0.13.0
Downloading gguf-0.17.1-py3-none-any.whl (96 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 96.2/96.2 KB 25.1 MB/s eta 0:00:00
Requirement already satisfied: torchaudio==2.7.0 in ./.local/lib/python3.10/site-packages (from vllm==0.9.1->vllm-rbln) (2.7.0)
Collecting blake3
Downloading blake3-1.0.8-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (387 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 387.6/387.6 KB 103.9 MB/s eta 0:00:00
Collecting sentencepiece
Downloading sentencepiece-0.2.1-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (1.4 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.4/1.4 MB 93.4 MB/s eta 0:00:00
Collecting openai>=1.52.0
Downloading openai-2.6.1-py3-none-any.whl (1.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.0/1.0 MB 97.9 MB/s eta 0:00:00
Requirement already satisfied: psutil in ./.local/lib/python3.10/site-packages (from vllm==0.9.1->vllm-rbln) (7.1.2)
Collecting xgrammar==0.1.19
Downloading xgrammar-0.1.19-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.8 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.8/5.8 MB 87.0 MB/s eta 0:00:00
Requirement already satisfied: scipy in ./.local/lib/python3.10/site-packages (from vllm==0.9.1->vllm-rbln) (1.15.3)
Collecting ninja
Downloading ninja-1.13.0-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (180 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 180.7/180.7 KB 68.0 MB/s eta 0:00:00
Collecting cachetools
Downloading cachetools-6.2.1-py3-none-any.whl (11 kB)
Collecting ray[cgraph]!=2.44.*,>=2.43.0
Downloading ray-2.50.1-cp310-cp310-manylinux2014_x86_64.whl (70.9 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 70.9/70.9 MB 33.2 MB/s eta 0:00:00
Collecting pydantic>=2.10
Downloading pydantic-2.12.3-py3-none-any.whl (462 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 462.4/462.4 KB 89.4 MB/s eta 0:00:00
Collecting opencv-python-headless>=4.11.0
Downloading opencv_python_headless-4.12.0.88-cp37-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (54.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 54.0/54.0 MB 40.7 MB/s eta 0:00:00
Collecting partial-json-parser
Downloading partial_json_parser-0.2.1.1.post6-py3-none-any.whl (10 kB)
Collecting xformers==0.0.30
Downloading xformers-0.0.30-cp310-cp310-manylinux_2_28_x86_64.whl (31.5 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 31.5/31.5 MB 49.0 MB/s eta 0:00:00
Collecting msgspec
Downloading msgspec-0.19.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (211 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 211.6/211.6 KB 89.9 MB/s eta 0:00:00
Collecting astor
Downloading astor-0.8.1-py2.py3-none-any.whl (27 kB)
Collecting dill
Downloading dill-0.4.0-py3-none-any.whl (119 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 119.7/119.7 KB 62.0 MB/s eta 0:00:00
Collecting llvmlite<0.45,>=0.44.0dev0
Downloading llvmlite-0.44.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (42.4 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 42.4/42.4 MB 46.8 MB/s eta 0:00:00
Collecting airportsdata
Downloading airportsdata-20250909-py3-none-any.whl (914 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 914.4/914.4 KB 93.6 MB/s eta 0:00:00
Collecting referencing
Downloading referencing-0.37.0-py3-none-any.whl (26 kB)
Collecting jsonschema
Downloading jsonschema-4.25.1-py3-none-any.whl (90 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 90.0/90.0 KB 42.7 MB/s eta 0:00:00
Collecting pycountry
Downloading pycountry-24.6.1-py3-none-any.whl (6.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.3/6.3 MB 92.5 MB/s eta 0:00:00
Collecting outlines_core==0.1.26
Downloading outlines_core-0.1.26-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (343 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 343.6/343.6 KB 96.6 MB/s eta 0:00:00
Collecting diskcache
Downloading diskcache-5.6.3-py3-none-any.whl (45 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 45.5/45.5 KB 16.4 MB/s eta 0:00:00
Requirement already satisfied: jinja2 in ./.local/lib/python3.10/site-packages (from outlines==0.1.11->vllm==0.9.1->vllm-rbln) (3.1.6)
Collecting nest_asyncio
Downloading nest_asyncio-1.6.0-py3-none-any.whl (5.2 kB)
Collecting interegular
Downloading interegular-0.3.3-py37-none-any.whl (23 kB)
Requirement already satisfied: fsspec in ./.local/lib/python3.10/site-packages (from torch==2.7.0->vllm==0.9.1->vllm-rbln) (2025.9.0)
Requirement already satisfied: nvidia-cuda-cupti-cu12==12.6.80 in ./.local/lib/python3.10/site-packages (from torch==2.7.0->vllm==0.9.1->vllm-rbln) (12.6.80)
Requirement already satisfied: nvidia-curand-cu12==10.3.7.77 in ./.local/lib/python3.10/site-packages (from torch==2.7.0->vllm==0.9.1->vllm-rbln) (10.3.7.77)
Requirement already satisfied: nvidia-cusolver-cu12==11.7.1.2 in ./.local/lib/python3.10/site-packages (from torch==2.7.0->vllm==0.9.1->vllm-rbln) (11.7.1.2)
Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.6.77 in ./.local/lib/python3.10/site-packages (from torch==2.7.0->vllm==0.9.1->vllm-rbln) (12.6.77)
Requirement already satisfied: sympy>=1.13.3 in ./.local/lib/python3.10/site-packages (from torch==2.7.0->vllm==0.9.1->vllm-rbln) (1.14.0)
Requirement already satisfied: nvidia-nvjitlink-cu12==12.6.85 in ./.local/lib/python3.10/site-packages (from torch==2.7.0->vllm==0.9.1->vllm-rbln) (12.6.85)
Requirement already satisfied: networkx in ./.local/lib/python3.10/site-packages (from torch==2.7.0->vllm==0.9.1->vllm-rbln) (3.4.2)
Requirement already satisfied: nvidia-cublas-cu12==12.6.4.1 in ./.local/lib/python3.10/site-packages (from torch==2.7.0->vllm==0.9.1->vllm-rbln) (12.6.4.1)
Requirement already satisfied: nvidia-cuda-runtime-cu12==12.6.77 in ./.local/lib/python3.10/site-packages (from torch==2.7.0->vllm==0.9.1->vllm-rbln) (12.6.77)
Requirement already satisfied: nvidia-cufile-cu12==1.11.1.6 in ./.local/lib/python3.10/site-packages (from torch==2.7.0->vllm==0.9.1->vllm-rbln) (1.11.1.6)
Requirement already satisfied: triton==3.3.0 in ./.local/lib/python3.10/site-packages (from torch==2.7.0->vllm==0.9.1->vllm-rbln) (3.3.0)
Requirement already satisfied: nvidia-nvtx-cu12==12.6.77 in ./.local/lib/python3.10/site-packages (from torch==2.7.0->vllm==0.9.1->vllm-rbln) (12.6.77)
Requirement already satisfied: nvidia-cufft-cu12==11.3.0.4 in ./.local/lib/python3.10/site-packages (from torch==2.7.0->vllm==0.9.1->vllm-rbln) (11.3.0.4)
Requirement already satisfied: nvidia-nccl-cu12==2.26.2 in ./.local/lib/python3.10/site-packages (from torch==2.7.0->vllm==0.9.1->vllm-rbln) (2.26.2)
Requirement already satisfied: nvidia-cudnn-cu12==9.5.1.17 in ./.local/lib/python3.10/site-packages (from torch==2.7.0->vllm==0.9.1->vllm-rbln) (9.5.1.17)
Requirement already satisfied: nvidia-cusparse-cu12==12.5.4.2 in ./.local/lib/python3.10/site-packages (from torch==2.7.0->vllm==0.9.1->vllm-rbln) (12.5.4.2)
Requirement already satisfied: nvidia-cusparselt-cu12==0.6.3 in ./.local/lib/python3.10/site-packages (from torch==2.7.0->vllm==0.9.1->vllm-rbln) (0.6.3)
Requirement already satisfied: accelerate>=1.0.1 in ./.local/lib/python3.10/site-packages (from optimum-rbln>=0.9.1->vllm-rbln) (1.11.0)
Requirement already satisfied: diffusers==0.35.1 in ./.local/lib/python3.10/site-packages (from optimum-rbln>=0.9.1->vllm-rbln) (0.35.1)
Requirement already satisfied: importlib_metadata in ./.local/lib/python3.10/site-packages (from diffusers==0.35.1->optimum-rbln>=0.9.1->vllm-rbln) (8.7.0)
Collecting tomli>=1
Downloading tomli-2.3.0-py3-none-any.whl (14 kB)
Collecting xxhash
Downloading xxhash-3.6.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (193 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 193.2/193.2 KB 83.7 MB/s eta 0:00:00
Collecting multiprocess<0.70.17
Downloading multiprocess-0.70.16-py310-none-any.whl (134 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 134.8/134.8 KB 64.0 MB/s eta 0:00:00
Collecting pyarrow>=21.0.0
Downloading pyarrow-22.0.0-cp310-cp310-manylinux_2_28_x86_64.whl (47.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 47.6/47.6 MB 44.0 MB/s eta 0:00:00
Collecting pandas
Downloading pandas-2.3.3-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl (12.8 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.8/12.8 MB 81.9 MB/s eta 0:00:00
Collecting httpx<1.0.0
Downloading httpx-0.28.1-py3-none-any.whl (73 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 73.5/73.5 KB 46.9 MB/s eta 0:00:00
Collecting av
Downloading av-16.0.1-cp310-cp310-manylinux_2_28_x86_64.whl (39.7 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 39.7/39.7 MB 46.4 MB/s eta 0:00:00
Collecting starlette<0.50.0,>=0.40.0
Downloading starlette-0.49.0-py3-none-any.whl (74 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 74.1/74.1 KB 55.4 MB/s eta 0:00:00
Collecting annotated-doc>=0.0.2
Downloading annotated_doc-0.0.3-py3-none-any.whl (5.5 kB)
Collecting fastapi-cli[standard]>=0.0.8
Downloading fastapi_cli-0.0.14-py3-none-any.whl (11 kB)
Collecting python-multipart>=0.0.18
Downloading python_multipart-0.0.20-py3-none-any.whl (24 kB)
Collecting email-validator>=2.0.0
Downloading email_validator-2.3.0-py3-none-any.whl (35 kB)
Collecting uvicorn[standard]>=0.12.0
Downloading uvicorn-0.38.0-py3-none-any.whl (68 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 68.1/68.1 KB 37.4 MB/s eta 0:00:00
Collecting propcache>=0.2.0
Downloading propcache-0.4.1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (196 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 196.9/196.9 KB 81.3 MB/s eta 0:00:00
Requirement already satisfied: attrs>=17.3.0 in ./.local/lib/python3.10/site-packages (from aiohttp->vllm==0.9.1->vllm-rbln) (25.4.0)
Collecting async-timeout<6.0,>=4.0
Downloading async_timeout-5.0.1-py3-none-any.whl (6.2 kB)
Collecting aiosignal>=1.4.0
Downloading aiosignal-1.4.0-py3-none-any.whl (7.5 kB)
Collecting multidict<7.0,>=4.5
Downloading multidict-6.7.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (241 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 241.7/241.7 KB 95.4 MB/s eta 0:00:00
Collecting aiohappyeyeballs>=2.5.0
Downloading aiohappyeyeballs-2.6.1-py3-none-any.whl (15 kB)
Collecting yarl<2.0,>=1.17.0
Downloading yarl-1.22.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (346 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 347.0/347.0 KB 88.6 MB/s eta 0:00:00
Collecting frozenlist>=1.1.1
Downloading frozenlist-1.8.0-cp310-cp310-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl (219 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 219.5/219.5 KB 87.3 MB/s eta 0:00:00
Requirement already satisfied: certifi in ./.local/lib/python3.10/site-packages (from httpx<1.0.0->datasets->vllm-rbln) (2025.10.5)
Requirement already satisfied: idna in ./.local/lib/python3.10/site-packages (from httpx<1.0.0->datasets->vllm-rbln) (3.11)
Collecting httpcore==1.*
Downloading httpcore-1.0.9-py3-none-any.whl (78 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.8/78.8 KB 42.7 MB/s eta 0:00:00
Collecting anyio
Downloading anyio-4.11.0-py3-none-any.whl (109 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 109.1/109.1 KB 55.9 MB/s eta 0:00:00
Collecting h11>=0.16
Downloading h11-0.16.0-py3-none-any.whl (37 kB)
Requirement already satisfied: hf-xet<2.0.0,>=1.1.3 in ./.local/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.30.0->transformers==4.51.3->vllm-rbln) (1.2.0)
Collecting pydantic-extra-types[pycountry]>=2.10.5
Downloading pydantic_extra_types-2.10.6-py3-none-any.whl (40 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.9/40.9 KB 15.8 MB/s eta 0:00:00
Collecting jiter<1,>=0.10.0
Downloading jiter-0.11.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (359 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 359.3/359.3 KB 94.3 MB/s eta 0:00:00
Collecting sniffio
Downloading sniffio-1.3.1-py3-none-any.whl (10 kB)
Requirement already satisfied: distro<2,>=1.7.0 in /usr/lib/python3/dist-packages (from openai>=1.52.0->vllm==0.9.1->vllm-rbln) (1.7.0)
Collecting opentelemetry-exporter-otlp-proto-grpc==1.38.0
Downloading opentelemetry_exporter_otlp_proto_grpc-1.38.0-py3-none-any.whl (19 kB)
Collecting opentelemetry-exporter-otlp-proto-http==1.38.0
Downloading opentelemetry_exporter_otlp_proto_http-1.38.0-py3-none-any.whl (19 kB)
Collecting googleapis-common-protos~=1.57
Downloading googleapis_common_protos-1.71.0-py3-none-any.whl (294 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 294.6/294.6 KB 73.6 MB/s eta 0:00:00
Collecting opentelemetry-proto==1.38.0
Downloading opentelemetry_proto-1.38.0-py3-none-any.whl (72 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 72.5/72.5 KB 28.7 MB/s eta 0:00:00
Collecting opentelemetry-exporter-otlp-proto-common==1.38.0
Downloading opentelemetry_exporter_otlp_proto_common-1.38.0-py3-none-any.whl (18 kB)
Collecting grpcio<2.0.0,>=1.63.2
Downloading grpcio-1.76.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (6.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.6/6.6 MB 89.0 MB/s eta 0:00:00
Collecting opentelemetry-semantic-conventions==0.59b0
Downloading opentelemetry_semantic_conventions-0.59b0-py3-none-any.whl (207 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 208.0/208.0 KB 85.5 MB/s eta 0:00:00
Collecting annotated-types>=0.6.0
Downloading annotated_types-0.7.0-py3-none-any.whl (13 kB)
Collecting pydantic-core==2.41.4
Downloading pydantic_core-2.41.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 94.2 MB/s eta 0:00:00
Collecting typing-inspection>=0.4.2
Downloading typing_inspection-0.4.2-py3-none-any.whl (14 kB)
Collecting click!=8.3.0,>=7.0
Downloading click-8.2.1-py3-none-any.whl (102 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 102.2/102.2 KB 56.5 MB/s eta 0:00:00
Collecting msgpack<2.0.0,>=1.0.0
Downloading msgpack-1.1.2-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (406 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 406.1/406.1 KB 96.9 MB/s eta 0:00:00
Collecting cupy-cuda12x
Downloading cupy_cuda12x-13.6.0-cp310-cp310-manylinux2014_x86_64.whl (112.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 112.2/112.2 MB 27.0 MB/s eta 0:00:00
Requirement already satisfied: urllib3<3,>=1.21.1 in ./.local/lib/python3.10/site-packages (from requests->transformers==4.51.3->vllm-rbln) (2.5.0)
Requirement already satisfied: charset_normalizer<4,>=2 in ./.local/lib/python3.10/site-packages (from requests->transformers==4.51.3->vllm-rbln) (3.4.4)
Collecting python-dateutil>=2.8.2
Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 229.9/229.9 KB 103.9 MB/s eta 0:00:00
Collecting tzdata>=2022.7
Downloading tzdata-2025.2-py2.py3-none-any.whl (347 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 347.8/347.8 KB 101.4 MB/s eta 0:00:00
Collecting pytz>=2020.1
Downloading pytz-2025.2-py2.py3-none-any.whl (509 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 509.2/509.2 KB 94.6 MB/s eta 0:00:00
Collecting exceptiongroup>=1.0.2
Downloading exceptiongroup-1.3.0-py3-none-any.whl (16 kB)
Collecting dnspython>=2.0.0
Downloading dnspython-2.8.0-py3-none-any.whl (331 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 331.1/331.1 KB 101.9 MB/s eta 0:00:00
Collecting typer>=0.15.1
Downloading typer-0.20.0-py3-none-any.whl (47 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 47.0/47.0 KB 34.0 MB/s eta 0:00:00
Collecting rich-toolkit>=0.14.8
Downloading rich_toolkit-0.15.1-py3-none-any.whl (29 kB)
Collecting fastapi-cloud-cli>=0.1.1
Downloading fastapi_cloud_cli-0.3.1-py3-none-any.whl (19 kB)
Requirement already satisfied: zipp>=3.20 in ./.local/lib/python3.10/site-packages (from importlib_metadata->diffusers==0.35.1->optimum-rbln>=0.9.1->vllm-rbln) (3.23.0)
Requirement already satisfied: MarkupSafe>=2.0 in ./.local/lib/python3.10/site-packages (from jinja2->outlines==0.1.11->vllm==0.9.1->vllm-rbln) (3.0.3)
Collecting jsonschema-specifications>=2023.03.6
Downloading jsonschema_specifications-2025.9.1-py3-none-any.whl (18 kB)
Collecting rpds-py>=0.7.1
Downloading rpds_py-0.28.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (382 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 382.7/382.7 KB 96.7 MB/s eta 0:00:00
Collecting six>=1.5
Downloading six-1.17.0-py2.py3-none-any.whl (11 kB)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in ./.local/lib/python3.10/site-packages (from sympy>=1.13.3->torch==2.7.0->vllm==0.9.1->vllm-rbln) (1.3.0)
Collecting httptools>=0.6.3
Downloading httptools-0.7.1-cp310-cp310-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl (440 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 440.9/440.9 KB 67.5 MB/s eta 0:00:00
Collecting python-dotenv>=0.13
Downloading python_dotenv-1.2.1-py3-none-any.whl (21 kB)
Collecting websockets>=10.4
Downloading websockets-15.0.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (181 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 181.6/181.6 KB 74.6 MB/s eta 0:00:00
Collecting uvloop>=0.15.1
Downloading uvloop-0.22.1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (3.7 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.7/3.7 MB 78.8 MB/s eta 0:00:00
Collecting fastrlock>=0.5
Downloading fastrlock-0.8.3-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_28_x86_64.whl (53 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.3/53.3 KB 19.5 MB/s eta 0:00:00
Collecting sentry-sdk>=2.20.0
Downloading sentry_sdk-2.42.1-py2.py3-none-any.whl (380 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 381.0/381.0 KB 101.2 MB/s eta 0:00:00
Collecting rignore>=0.5.1
Downloading rignore-0.7.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (952 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 952.4/952.4 KB 93.1 MB/s eta 0:00:00
Collecting rich>=13.7.1
Downloading rich-14.2.0-py3-none-any.whl (243 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 243.4/243.4 KB 88.9 MB/s eta 0:00:00
Collecting shellingham>=1.3.0
Downloading shellingham-1.5.4-py2.py3-none-any.whl (9.8 kB)
Collecting pygments<3.0.0,>=2.13.0
Downloading pygments-2.19.2-py3-none-any.whl (1.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 92.9 MB/s eta 0:00:00
Collecting markdown-it-py>=2.2.0
Downloading markdown_it_py-4.0.0-py3-none-any.whl (87 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 87.3/87.3 KB 45.0 MB/s eta 0:00:00
Collecting mdurl~=0.1
Downloading mdurl-0.1.2-py3-none-any.whl (10.0 kB)
Installing collected packages: pytz, py-cpuinfo, fastrlock, xxhash, websockets, uvloop, tzdata, typing-inspection, tomli, sniffio, six, shellingham, setuptools, sentry-sdk, sentencepiece, rpds-py, rignore, pyzmq, python-multipart, python-json-logger, python-dotenv, pygments, pydantic-core, pycountry, pyarrow, protobuf, propcache, prometheus_client, partial-json-parser, opentelemetry-semantic-conventions-ai, opencv-python-headless, ninja, nest_asyncio, multidict, msgspec, msgpack, mdurl, llvmlite, llguidance, lark, jiter, interegular, httptools, h11, grpcio, gguf, frozenlist, exceptiongroup, einops, dnspython, diskcache, dill, cupy-cuda12x, click, cachetools, blake3, av, async-timeout, astor, annotated-types, annotated-doc, airportsdata, aiohappyeyeballs, yarl, uvicorn, tiktoken, setuptools-scm, referencing, qwen_vl_utils, python-dateutil, pydantic, opentelemetry-proto, opentelemetry-api, numba, multiprocess, markdown-it-py, httpcore, googleapis-common-protos, email-validator, depyf, anyio, aiosignal, watchfiles, starlette, rich, pydantic-extra-types, pandas, opentelemetry-semantic-conventions, opentelemetry-exporter-otlp-proto-common, lm-format-enforcer, jsonschema-specifications, httpx, aiohttp, xformers, typer, rich-toolkit, prometheus-fastapi-instrumentator, opentelemetry-sdk, openai, jsonschema, fastapi, xgrammar, ray, outlines_core, opentelemetry-exporter-otlp-proto-http, opentelemetry-exporter-otlp-proto-grpc, mistral_common, fastapi-cloud-cli, fastapi-cli, datasets, compressed-tensors, outlines, opentelemetry-exporter-otlp, vllm, vllm-rbln
WARNING: The script cpuinfo is installed in '/home/ubuntu/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The script websockets is installed in '/home/ubuntu/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The script dotenv is installed in '/home/ubuntu/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The script pygmentize is installed in '/home/ubuntu/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The script json-playground is installed in '/home/ubuntu/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The scripts gguf-convert-endian, gguf-dump, gguf-editor-gui, gguf-new-metadata and gguf-set-metadata are installed in '/home/ubuntu/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The script pyav is installed in '/home/ubuntu/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The script uvicorn is installed in '/home/ubuntu/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The script setuptools-scm is installed in '/home/ubuntu/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The script markdown-it is installed in '/home/ubuntu/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The script email_validator is installed in '/home/ubuntu/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The script watchfiles is installed in '/home/ubuntu/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The script httpx is installed in '/home/ubuntu/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The script typer is installed in '/home/ubuntu/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The script openai is installed in '/home/ubuntu/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The script jsonschema is installed in '/home/ubuntu/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The script fastapi is installed in '/home/ubuntu/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The scripts ray, serve and tune are installed in '/home/ubuntu/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The script mistral_common is installed in '/home/ubuntu/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The script datasets-cli is installed in '/home/ubuntu/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The script vllm is installed in '/home/ubuntu/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed aiohappyeyeballs-2.6.1 aiohttp-3.13.1 aiosignal-1.4.0 airportsdata-20250909 annotated-doc-0.0.3 annotated-types-0.7.0 anyio-4.11.0 astor-0.8.1 async-timeout-5.0.1 av-16.0.1 blake3-1.0.8 cachetools-6.2.1 click-8.2.1 compressed-tensors-0.10.1 cupy-cuda12x-13.6.0 datasets-4.3.0 depyf-0.18.0 dill-0.4.0 diskcache-5.6.3 dnspython-2.8.0 einops-0.8.1 email-validator-2.3.0 exceptiongroup-1.3.0 fastapi-0.120.1 fastapi-cli-0.0.14 fastapi-cloud-cli-0.3.1 fastrlock-0.8.3 frozenlist-1.8.0 gguf-0.17.1 googleapis-common-protos-1.71.0 grpcio-1.76.0 h11-0.16.0 httpcore-1.0.9 httptools-0.7.1 httpx-0.28.1 interegular-0.3.3 jiter-0.11.1 jsonschema-4.25.1 jsonschema-specifications-2025.9.1 lark-1.2.2 llguidance-0.7.30 llvmlite-0.44.0 lm-format-enforcer-0.10.12 markdown-it-py-4.0.0 mdurl-0.1.2 mistral_common-1.8.5 msgpack-1.1.2 msgspec-0.19.0 multidict-6.7.0 multiprocess-0.70.16 nest_asyncio-1.6.0 ninja-1.13.0 numba-0.61.2 openai-2.6.1 opencv-python-headless-4.12.0.88 opentelemetry-api-1.38.0 opentelemetry-exporter-otlp-1.38.0 opentelemetry-exporter-otlp-proto-common-1.38.0 opentelemetry-exporter-otlp-proto-grpc-1.38.0 opentelemetry-exporter-otlp-proto-http-1.38.0 opentelemetry-proto-1.38.0 opentelemetry-sdk-1.38.0 opentelemetry-semantic-conventions-0.59b0 opentelemetry-semantic-conventions-ai-0.4.13 outlines-0.1.11 outlines_core-0.1.26 pandas-2.3.3 partial-json-parser-0.2.1.1.post6 prometheus-fastapi-instrumentator-7.1.0 prometheus_client-0.23.1 propcache-0.4.1 protobuf-6.33.0 py-cpuinfo-9.0.0 pyarrow-22.0.0 pycountry-24.6.1 pydantic-2.12.3 pydantic-core-2.41.4 pydantic-extra-types-2.10.6 pygments-2.19.2 python-dateutil-2.9.0.post0 python-dotenv-1.2.1 python-json-logger-4.0.0 python-multipart-0.0.20 pytz-2025.2 pyzmq-27.1.0 qwen_vl_utils-0.0.14 ray-2.50.1 referencing-0.37.0 rich-14.2.0 rich-toolkit-0.15.1 rignore-0.7.1 rpds-py-0.28.0 sentencepiece-0.2.1 sentry-sdk-2.42.1 setuptools-80.9.0 setuptools-scm-9.2.2 shellingham-1.5.4 six-1.17.0 sniffio-1.3.1 starlette-0.49.0 tiktoken-0.12.0 tomli-2.3.0 typer-0.20.0 typing-inspection-0.4.2 tzdata-2025.2 uvicorn-0.38.0 uvloop-0.22.1 vllm-0.9.1 vllm-rbln-0.9.1 watchfiles-1.1.1 websockets-15.0.1 xformers-0.0.30 xgrammar-0.1.19 xxhash-3.6.0 yarl-1.22.0
ubuntu@56122307d8f9:~$
No 5. Hugging Face 모델 준비
* 웹 브라우저 계정 생성 및 토큰 발급
1. https://huggingface.co/login
2. 오른쪽 위 프로필 사진 클릭 → Settings
3. 왼쪽 메뉴에서 Access Tokens 클릭
4. “New token” 버튼 클릭
5. 이름은 아무거나 입력 예: rbln-compile
6. Role은 Read 선택
7. hf_로 시작하는 토큰 생성 확인 및 복사(다시 못 보니 이때 꼭 복사)
8. 승인 확인 https://huggingface.co/settings/tokens
Name : boyzoneXXXX
Value : hf_AuJKzIhcwqXXXXXXXXXXXXXXXXXXXX
* 모델 접근 승인 신청 (Llama3 전용)
1. 아래 사이트접속
https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct
2. 중간쯤에 자기 신상 정보 작성하고 Submit 버튼
- 또는 중간쯤에 “Access request” 또는 “Agree and access” 버튼 클릭 후 내용 작성 후 제출
3. 이메일로 승인확인
Hi KimXXXXXXXX,
This is to let you know your request to access model "meta-llama/Llama-3.1-8B" on huggingface.co has been accepted by the repo authors.
You can now access the repo here, or view all your gated repo access requests in your settings.
Cheers,
The Hugging Face team
4. 위 링크 here에서 승인 확인
* 우분투 서버에 토큰 등록
ubuntu@4b7f4bcac5a5:~$ pip install -U huggingface_hub
ubuntu@4b7f4bcac5a5:~$ echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
ubuntu@4b7f4bcac5a5:~$ source ~/.bashrc
ubuntu@4b7f4bcac5a5:~$ hf auth login
_| _| _| _| _|_|_| _|_|_| _|_|_| _| _| _|_|_| _|_|_|_| _|_| _|_|_| _|_|_|_|
_| _| _| _| _| _| _| _|_| _| _| _| _| _| _| _|
_|_|_|_| _| _| _| _|_| _| _|_| _| _| _| _| _| _|_| _|_|_| _|_|_|_| _| _|_|_|
_| _| _| _| _| _| _| _| _| _| _|_| _| _| _| _| _| _| _|
_| _| _|_| _|_|_| _|_|_| _|_|_| _| _| _|_|_| _| _| _| _|_|_| _|_|_|_|
To log in, `huggingface_hub` requires a token generated from https://huggingface.co/settings/tokens .
Enter your token (input will not be visible): 웹 사이트에서 받은 토큰 복사 후 입력
Add token as git credential? [y/N]: y
Token is valid (permission: read).
The token `boyzone8` has been saved to /home/ubuntu/.cache/huggingface/stored_tokens
Cannot authenticate through git-credential as no helper is defined on your machine.
You might have to re-authenticate when pushing to the Hugging Face Hub.
Run the following command in your terminal in case you want to set the 'store' credential helper as default.
git config --global credential.helper store
Read https://git-scm.com/book/en/v2/Git-Tools-Credential-Storage for more details.
Token has not been saved to git credential helper.
Your token has been saved to /home/ubuntu/.cache/huggingface/token
Login successful.
The current active token is: `boyzoneXXXX`
ubuntu@4b7f4bcac5a5:~$
* gemma 모델 권한
브라우저에서 google/gemma-7b-it 모델 페이지 들어가서 라이선스/이용 약관 동의 + 액세스 요청 승인 받기.
https://huggingface.co/google/gemma-7b-it
Access Gemma on Hugging Face 아래
Acknowledge license 버튼 클릭
No 6. 컴파일
ubuntu@4b7f4bcac5a5:~$ cd ~/top/rbln-model-zoo/huggingface/transformers/text2text-generation/llama/llama3.1-8b/
ubuntu@4b7f4bcac5a5:~/top/rbln-model-zoo/huggingface/transformers/text2text-generation/llama/llama3.1-8b$ pip install -r requirements.txt
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: optimum-rbln in /home/ubuntu/.local/lib/python3.10/site-packages (from -r requirements.txt (line 1)) (0.9.2)
Requirement already satisfied: packaging>=24.1 in /home/ubuntu/.local/lib/python3.10/site-packages (from optimum-rbln->-r requirements.txt (line 1)) (25.0)
Requirement already satisfied: diffusers==0.35.1 in /home/ubuntu/.local/lib/python3.10/site-packages (from optimum-rbln->-r requirements.txt (line 1)) (0.35.1)
Requirement already satisfied: torchaudio<=2.8.0 in /home/ubuntu/.local/lib/python3.10/site-packages (from optimum-rbln->-r requirements.txt (line 1)) (2.8.0)
Requirement already satisfied: accelerate>=1.0.1 in /home/ubuntu/.local/lib/python3.10/site-packages (from optimum-rbln->-r requirements.txt (line 1)) (1.11.0)
Requirement already satisfied: torchvision<=0.23.0 in /home/ubuntu/.local/lib/python3.10/site-packages (from optimum-rbln->-r requirements.txt (line 1)) (0.23.0)
Requirement already satisfied: transformers==4.57.1 in /home/ubuntu/.local/lib/python3.10/site-packages (from optimum-rbln->-r requirements.txt (line 1)) (4.57.1)
Requirement already satisfied: torch==2.8.0 in /home/ubuntu/.local/lib/python3.10/site-packages (from optimum-rbln->-r requirements.txt (line 1)) (2.8.0)
Requirement already satisfied: safetensors>=0.3.1 in /home/ubuntu/.local/lib/python3.10/site-packages (from diffusers==0.35.1->optimum-rbln->-r requirements.txt (line 1)) (0.6.2)
Requirement already satisfied: huggingface-hub>=0.34.0 in /home/ubuntu/.local/lib/python3.10/site-packages (from diffusers==0.35.1->optimum-rbln->-r requirements.txt (line 1)) (0.36.0)
Requirement already satisfied: requests in /home/ubuntu/.local/lib/python3.10/site-packages (from diffusers==0.35.1->optimum-rbln->-r requirements.txt (line 1)) (2.32.5)
Requirement already satisfied: importlib_metadata in /home/ubuntu/.local/lib/python3.10/site-packages (from diffusers==0.35.1->optimum-rbln->-r requirements.txt (line 1)) (8.7.0)
Requirement already satisfied: regex!=2019.12.17 in /home/ubuntu/.local/lib/python3.10/site-packages (from diffusers==0.35.1->optimum-rbln->-r requirements.txt (line 1)) (2025.10.23)
Requirement already satisfied: numpy in /home/ubuntu/.local/lib/python3.10/site-packages (from diffusers==0.35.1->optimum-rbln->-r requirements.txt (line 1)) (2.2.6)
Requirement already satisfied: filelock in /home/ubuntu/.local/lib/python3.10/site-packages (from diffusers==0.35.1->optimum-rbln->-r requirements.txt (line 1)) (3.20.0)
Requirement already satisfied: Pillow in /home/ubuntu/.local/lib/python3.10/site-packages (from diffusers==0.35.1->optimum-rbln->-r requirements.txt (line 1)) (12.0.0)
Requirement already satisfied: nvidia-nccl-cu12==2.27.3 in /home/ubuntu/.local/lib/python3.10/site-packages (from torch==2.8.0->optimum-rbln->-r requirements.txt (line 1)) (2.27.3)
Requirement already satisfied: fsspec in /home/ubuntu/.local/lib/python3.10/site-packages (from torch==2.8.0->optimum-rbln->-r requirements.txt (line 1)) (2025.9.0)
Requirement already satisfied: nvidia-cusolver-cu12==11.7.3.90 in /home/ubuntu/.local/lib/python3.10/site-packages (from torch==2.8.0->optimum-rbln->-r requirements.txt (line 1)) (11.7.3.90)
Requirement already satisfied: nvidia-cuda-cupti-cu12==12.8.90 in /home/ubuntu/.local/lib/python3.10/site-packages (from torch==2.8.0->optimum-rbln->-r requirements.txt (line 1)) (12.8.90)
Requirement already satisfied: triton==3.4.0 in /home/ubuntu/.local/lib/python3.10/site-packages (from torch==2.8.0->optimum-rbln->-r requirements.txt (line 1)) (3.4.0)
Requirement already satisfied: sympy>=1.13.3 in /home/ubuntu/.local/lib/python3.10/site-packages (from torch==2.8.0->optimum-rbln->-r requirements.txt (line 1)) (1.14.0)
Requirement already satisfied: nvidia-nvjitlink-cu12==12.8.93 in /home/ubuntu/.local/lib/python3.10/site-packages (from torch==2.8.0->optimum-rbln->-r requirements.txt (line 1)) (12.8.93)
Requirement already satisfied: jinja2 in /home/ubuntu/.local/lib/python3.10/site-packages (from torch==2.8.0->optimum-rbln->-r requirements.txt (line 1)) (3.1.6)
Requirement already satisfied: typing-extensions>=4.10.0 in /home/ubuntu/.local/lib/python3.10/site-packages (from torch==2.8.0->optimum-rbln->-r requirements.txt (line 1)) (4.15.0)
Requirement already satisfied: nvidia-cublas-cu12==12.8.4.1 in /home/ubuntu/.local/lib/python3.10/site-packages (from torch==2.8.0->optimum-rbln->-r requirements.txt (line 1)) (12.8.4.1)
Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.8.93 in /home/ubuntu/.local/lib/python3.10/site-packages (from torch==2.8.0->optimum-rbln->-r requirements.txt (line 1)) (12.8.93)
Requirement already satisfied: nvidia-nvtx-cu12==12.8.90 in /home/ubuntu/.local/lib/python3.10/site-packages (from torch==2.8.0->optimum-rbln->-r requirements.txt (line 1)) (12.8.90)
Requirement already satisfied: nvidia-cusparselt-cu12==0.7.1 in /home/ubuntu/.local/lib/python3.10/site-packages (from torch==2.8.0->optimum-rbln->-r requirements.txt (line 1)) (0.7.1)
Requirement already satisfied: nvidia-cuda-runtime-cu12==12.8.90 in /home/ubuntu/.local/lib/python3.10/site-packages (from torch==2.8.0->optimum-rbln->-r requirements.txt (line 1)) (12.8.90)
Requirement already satisfied: nvidia-cufft-cu12==11.3.3.83 in /home/ubuntu/.local/lib/python3.10/site-packages (from torch==2.8.0->optimum-rbln->-r requirements.txt (line 1)) (11.3.3.83)
Requirement already satisfied: nvidia-cufile-cu12==1.13.1.3 in /home/ubuntu/.local/lib/python3.10/site-packages (from torch==2.8.0->optimum-rbln->-r requirements.txt (line 1)) (1.13.1.3)
Requirement already satisfied: nvidia-curand-cu12==10.3.9.90 in /home/ubuntu/.local/lib/python3.10/site-packages (from torch==2.8.0->optimum-rbln->-r requirements.txt (line 1)) (10.3.9.90)
Requirement already satisfied: nvidia-cudnn-cu12==9.10.2.21 in /home/ubuntu/.local/lib/python3.10/site-packages (from torch==2.8.0->optimum-rbln->-r requirements.txt (line 1)) (9.10.2.21)
Requirement already satisfied: nvidia-cusparse-cu12==12.5.8.93 in /home/ubuntu/.local/lib/python3.10/site-packages (from torch==2.8.0->optimum-rbln->-r requirements.txt (line 1)) (12.5.8.93)
Requirement already satisfied: networkx in /home/ubuntu/.local/lib/python3.10/site-packages (from torch==2.8.0->optimum-rbln->-r requirements.txt (line 1)) (3.4.2)
Requirement already satisfied: tqdm>=4.27 in /home/ubuntu/.local/lib/python3.10/site-packages (from transformers==4.57.1->optimum-rbln->-r requirements.txt (line 1)) (4.67.1)
Requirement already satisfied: tokenizers<=0.23.0,>=0.22.0 in /home/ubuntu/.local/lib/python3.10/site-packages (from transformers==4.57.1->optimum-rbln->-r requirements.txt (line 1)) (0.22.1)
Requirement already satisfied: pyyaml>=5.1 in /home/ubuntu/.local/lib/python3.10/site-packages (from transformers==4.57.1->optimum-rbln->-r requirements.txt (line 1)) (6.0.3)
Requirement already satisfied: setuptools>=40.8.0 in /home/ubuntu/.local/lib/python3.10/site-packages (from triton==3.4.0->torch==2.8.0->optimum-rbln->-r requirements.txt (line 1)) (80.9.0)
Requirement already satisfied: psutil in /home/ubuntu/.local/lib/python3.10/site-packages (from accelerate>=1.0.1->optimum-rbln->-r requirements.txt (line 1)) (7.1.3)
Requirement already satisfied: hf-xet<2.0.0,>=1.1.3 in /home/ubuntu/.local/lib/python3.10/site-packages (from huggingface-hub>=0.34.0->diffusers==0.35.1->optimum-rbln->-r requirements.txt (line 1)) (1.2.0)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in /home/ubuntu/.local/lib/python3.10/site-packages (from sympy>=1.13.3->torch==2.8.0->optimum-rbln->-r requirements.txt (line 1)) (1.3.0)
Requirement already satisfied: zipp>=3.20 in /home/ubuntu/.local/lib/python3.10/site-packages (from importlib_metadata->diffusers==0.35.1->optimum-rbln->-r requirements.txt (line 1)) (3.23.0)
Requirement already satisfied: MarkupSafe>=2.0 in /home/ubuntu/.local/lib/python3.10/site-packages (from jinja2->torch==2.8.0->optimum-rbln->-r requirements.txt (line 1)) (3.0.3)
Requirement already satisfied: charset_normalizer<4,>=2 in /home/ubuntu/.local/lib/python3.10/site-packages (from requests->diffusers==0.35.1->optimum-rbln->-r requirements.txt (line 1)) (3.4.4)
Requirement already satisfied: idna<4,>=2.5 in /home/ubuntu/.local/lib/python3.10/site-packages (from requests->diffusers==0.35.1->optimum-rbln->-r requirements.txt (line 1)) (3.11)
Requirement already satisfied: urllib3<3,>=1.21.1 in /home/ubuntu/.local/lib/python3.10/site-packages (from requests->diffusers==0.35.1->optimum-rbln->-r requirements.txt (line 1)) (2.5.0)
Requirement already satisfied: certifi>=2017.4.17 in /home/ubuntu/.local/lib/python3.10/site-packages (from requests->diffusers==0.35.1->optimum-rbln->-r requirements.txt (line 1)) (2025.10.5)
ubuntu@4b7f4bcac5a5:~/top/rbln-model-zoo/huggingface/transformers/text2text-generation/llama/llama3.1-8b$ python3 compile.py
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:03<00:00, 1.14it/s]
2025-11-04 20:48:48,850 INFO [optimum.rbln] [KVCache] Compiling with num_blocks: 8
2025-11-04 20:48:51,864 INFO [rebel-compiler] Export done. Elapsed time: 0:00:02
2025-11-04 20:49:10,961 INFO [rebel-compiler] Exported model conversion done. Elapsed time: 0:00:13, Memory change: 3.04 MB
2025-11-04 20:49:11,057 INFO [rebel-compiler] RBLN SDK compiler version: 0.9.2
2025-11-04 20:49:11,058 INFO [rebel-compiler] -- Target NPU: RBLN-CA12
2025-11-04 20:49:11,059 INFO [rebel-compiler] -- Tensor parallel size: 4
2025-11-04 20:49:11,067 INFO [rebel-compiler] +------------------------------------------------+
2025-11-04 20:49:11,067 INFO [rebel-compiler] |Compile(#0), mod_name=44e6ff, input_info_index=0|
2025-11-04 20:49:11,067 INFO [rebel-compiler] +------------------------------------------------+
Computation graph generation ████████████████████████████████████████ 100% 00:29
Computation graph optimization ████████████████████████████████████████ 100% 00:43
2025-11-04 20:51:06,109 INFO [rebel-compiler] Export done. Elapsed time: 0:00:02
2025-11-04 20:51:25,928 INFO [rebel-compiler] Exported model conversion done. Elapsed time: 0:00:13, Memory change: 0.00 MB
2025-11-04 20:51:26,034 INFO [rebel-compiler] RBLN SDK compiler version: 0.9.2
2025-11-04 20:51:26,035 INFO [rebel-compiler] -- Target NPU: RBLN-CA12
2025-11-04 20:51:26,035 INFO [rebel-compiler] -- Tensor parallel size: 4
2025-11-04 20:51:26,036 INFO [rebel-compiler] +------------------------------------------------+
2025-11-04 20:51:26,036 INFO [rebel-compiler] |Compile(#1), mod_name=44e6ff, input_info_index=1|
2025-11-04 20:51:26,036 INFO [rebel-compiler] +------------------------------------------------+
Computation graph generation ████████████████████████████████████████ 100% 00:30
Computation graph optimization ████████████████████████████████████████ 100% 00:11
2025-11-04 20:52:20,372 INFO [rebel-compiler] Serializing compiled model to /tmp/tmp5ahfyujz/prefill.rbln ...
2025-11-04 20:52:36,492 INFO [rebel-compiler] Compiled model serialized. Elasped time: 0:00:16
2025-11-04 20:52:36,492 INFO [rebel-compiler] Serializing compiled model to /tmp/tmp5ahfyujz/decoder_batch_1.rbln ...
2025-11-04 20:52:37,679 INFO [rebel-compiler] Compiled model serialized. Elasped time: 0:00:01
No 7. 모델 서빙
ubuntu@4b7f4bcac5a5:$ cd ~/top/rbln-model-zoo/huggingface/transformers/text2text-generation/llama/llama3.1-8b
ubuntu@4b7f4bcac5a5:~/top/rbln-model-zoo/huggingface/transformers/text2text-generation/llama/llama3.1-8b$ python3 -m vllhttp://m.entrypoints.openai.api_server \
--model ./Llama-3.1-8B-Instruct \
--served-model-name Llama-3.1-8B-Instruct \
--host 0.0.0.0 \
--port 8000 \
--api-key mykey123
INFO 11-04 21:10:53 [__init__.py:36] Available plugins for group vllm.platform_plugins:
INFO 11-04 21:10:53 [__init__.py:38] - rbln -> vllm_rbln:register
INFO 11-04 21:10:53 [__init__.py:48] Loading plugin rbln
INFO 11-04 21:10:53 [__init__.py:207] Platform plugin rbln is activated
[vllm-rbln] WARNING 11-04 21:10:54 _custom_ops.py:20] Failed to import from vllm._C with ImportError('libcuda.so.1: cannot open shared object file: No such file or directory')
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:10:55 api_server.py:1896] vLLM API server version 0.10.2
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:10:55 utils.py:328] non-default args: {'host': '0.0.0.0', 'api_key': ['mykey123'], 'model': './Llama-3.1-8B-Instruct', 'served_model_name': ['Llama-3.1-8B-Instruct']}
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:00 __init__.py:742] Resolved architecture: LlamaForCausalLM
(APIServer pid=18474) `torch_dtype` is deprecated! Use `dtype` instead!
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:00 __init__.py:2764] Downcasting torch.float32 to torch.bfloat16.
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:00 __init__.py:1815] Using max model len 131072
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:00 scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=2048.
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:00 platform.py:145] original model_config.dtype = torch.bfloat16
(APIServer pid=18474) [vllm-rbln] WARNING 11-04 21:11:00 platform.py:147] bfloat16 is not supported on RBLN.
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:00 platform.py:152] RBLN model_config.dtype = torch.float32
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:00 configuration.py:82] Updating scheduler_config.max_num_seqs from 256 to 1 based on rbln_config.json
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:00 configuration.py:89] Updating scheduler_config.max_num_batched_tokens from 2048 to 131072 based on rbln_config.json
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:00 configuration.py:118] Updating model_cache_config.block_size from None to 16384 based on rbln_config.json
(APIServer pid=18474) [vllm-rbln] WARNING 11-04 21:11:00 platform.py:209] uni is not supported on RBLN, fallback to mp distributed executor backend.
INFO 11-04 21:11:03 [__init__.py:36] Available plugins for group vllm.platform_plugins:
INFO 11-04 21:11:03 [__init__.py:38] - rbln -> vllm_rbln:register
INFO 11-04 21:11:03 [__init__.py:48] Loading plugin rbln
INFO 11-04 21:11:03 [__init__.py:207] Platform plugin rbln is activated
[vllm-rbln] WARNING 11-04 21:11:04 _custom_ops.py:20] Failed to import from vllm._C with ImportError('libcuda.so.1: cannot open shared object file: No such file or directory')
(EngineCore_DP0 pid=18740) [vllm-rbln] INFO 11-04 21:11:05 core.py:654] Waiting for init message from front-end.
(EngineCore_DP0 pid=18740) [vllm-rbln] INFO 11-04 21:11:05 core.py:76] Initializing a V1 LLM engine (v0.10.2) with config: model='./Llama-3.1-8B-Instruct', speculative_config=None, tokenizer='./Llama-3.1-8B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float32, max_seq_len=131072, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cpu, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=Llama-3.1-8B-Instruct, enable_prefix_caching=None, chunked_prefill_enabled=True, use_async_output_proc=False, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output","vllm.mamba_mixer2","vllm.mamba_mixer","vllm.short_conv","vllm.linear_attention","vllm.plamo2_mamba_mixer","vllm.gdn_attention"],"use_inductor":true,"compile_sizes":null,"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"cudagraph_mode":0,"use_cudagraph":true,"cudagraph_num_of_warmups":0,"cudagraph_capture_sizes":null,"cudagraph_copy_inputs":false,"full_cuda_graph":false,"pass_config":{},"max_capture_size":null,"local_cache_dir":null}
(EngineCore_DP0 pid=18740) [vllm-rbln] WARNING 11-04 21:11:05 platform.py:77] Pin memory is not supported on RBLN.
(EngineCore_DP0 pid=18740) [vllm-rbln] INFO 11-04 21:11:06 importing.py:43] Triton is installed but 0 active driver(s) found (expected 1). Disabling Triton to prevent runtime errors.
(EngineCore_DP0 pid=18740) [vllm-rbln] INFO 11-04 21:11:06 importing.py:63] Triton not installed or not compatible; certain GPU-related functions will not be available.
(EngineCore_DP0 pid=18740) /home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/protocol/instruct/messages.py:74: FutureWarning: ImageChunk has moved to 'mistral_common.protocol.instruct.chunk'. It will be removed from 'mistral_common.protocol.instruct.messages' in 1.10.0.
(EngineCore_DP0 pid=18740) warnings.warn(msg, FutureWarning)
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
(EngineCore_DP0 pid=18740) [vllm-rbln] INFO 11-04 21:11:06 parallel_state.py:1165] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
(EngineCore_DP0 pid=18740) [vllm-rbln] INFO 11-04 21:11:06 optimum_model_runner.py:135] Using RBLN sampler: True
(EngineCore_DP0 pid=18740) 2025-11-04 21:11:19,788 INFO [rebel-compiler] Load model completed. Elasped time: 0:00:13
(EngineCore_DP0 pid=18740) 2025-11-04 21:11:21,534 INFO [rebel-compiler] Load model completed. Elasped time: 0:00:01
(EngineCore_DP0 pid=18740) [vllm-rbln] INFO 11-04 21:11:24 kv_cache_utils.py:864] GPU KV cache size: 147,456 tokens
(EngineCore_DP0 pid=18740) [vllm-rbln] INFO 11-04 21:11:24 kv_cache_utils.py:868] Maximum concurrency for 131,072 tokens per request: 1.12x
(EngineCore_DP0 pid=18740) [vllm-rbln] INFO 11-04 21:11:24 optimum_model_runner.py:893] Running dummy sampler config: no_penalty_greedy
(EngineCore_DP0 pid=18740) [vllm-rbln] INFO 11-04 21:11:24 optimum_model_runner.py:878] Running dummy compile with batch_size=1, vocab_size=128256
(EngineCore_DP0 pid=18740) [vllm-rbln] INFO 11-04 21:11:24 optimum_model_runner.py:881] Sampling metadata: SamplingMetadata(temperature=None, all_greedy=True, all_random=False, top_p=None, top_k=None, generators={}, max_num_logprobs=None, no_penalties=True, prompt_token_ids=None, frequency_penalties=tensor([0.]), presence_penalties=tensor([0.]), repetition_penalties=tensor([0.]), output_token_ids=[], allowed_token_ids_mask=None, bad_words_token_ids={}, logitsprocs=<vllhttp://m.v1.sample.logits_processor.state.LogitsProcessors object at 0x754266f03c10>)
(EngineCore_DP0 pid=18740) [vllm-rbln] INFO 11-04 21:11:31 optimum_model_runner.py:893] Running dummy sampler config: no_penalty_topp
(EngineCore_DP0 pid=18740) [vllm-rbln] INFO 11-04 21:11:31 optimum_model_runner.py:878] Running dummy compile with batch_size=1, vocab_size=128256
(EngineCore_DP0 pid=18740) [vllm-rbln] INFO 11-04 21:11:31 optimum_model_runner.py:881] Sampling metadata: SamplingMetadata(temperature=tensor([0.5000]), all_greedy=False, all_random=True, top_p=tensor([0.9000]), top_k=None, generators={}, max_num_logprobs=None, no_penalties=True, prompt_token_ids=None, frequency_penalties=tensor([0.]), presence_penalties=tensor([0.]), repetition_penalties=tensor([0.]), output_token_ids=[], allowed_token_ids_mask=None, bad_words_token_ids={}, logitsprocs=<vllhttp://m.v1.sample.logits_processor.state.LogitsProcessors object at 0x754266f03c10>)
(EngineCore_DP0 pid=18740) 2025-11-04 21:11:32,698 INFO [rebel-compiler] Export done. Elapsed time: 0:00:00
(EngineCore_DP0 pid=18740) 2025-11-04 21:11:32,864 INFO [rebel-compiler] Exported model conversion done. Elapsed time: 0:00:00, Memory change: 1.50 MB
(EngineCore_DP0 pid=18740) 2025-11-04 21:11:32,903 INFO [rebel-compiler] RBLN SDK compiler version: 0.9.2
(EngineCore_DP0 pid=18740) 2025-11-04 21:11:32,903 INFO [rebel-compiler] -- Target NPU: RBLN-CA12
(EngineCore_DP0 pid=18740) 2025-11-04 21:11:32,903 INFO [rebel-compiler] -- Tensor parallel size: 1
(EngineCore_DP0 pid=18740) 2025-11-04 21:11:32,904 INFO [rebel-compiler] +-------------------------------------------------+
(EngineCore_DP0 pid=18740) 2025-11-04 21:11:32,904 INFO [rebel-compiler] |Compile(#0), mod_name=default, input_info_index=0|
(EngineCore_DP0 pid=18740) 2025-11-04 21:11:32,904 INFO [rebel-compiler] +-------------------------------------------------+
Computation graph generation ████████████████████████████████████████ 100% 00:00
Computation graph optimization ████████████████████████████████████████ 100% 00:00
(EngineCore_DP0 pid=18740) [vllm-rbln] INFO 11-04 21:11:34 optimum_model_runner.py:893] Running dummy sampler config: no_penalty_topk
(EngineCore_DP0 pid=18740) [vllm-rbln] INFO 11-04 21:11:34 optimum_model_runner.py:878] Running dummy compile with batch_size=1, vocab_size=128256
(EngineCore_DP0 pid=18740) [vllm-rbln] INFO 11-04 21:11:34 optimum_model_runner.py:881] Sampling metadata: SamplingMetadata(temperature=tensor([0.5000]), all_greedy=False, all_random=True, top_p=None, top_k=tensor([1], dtype=torch.int32), generators={}, max_num_logprobs=None, no_penalties=True, prompt_token_ids=None, frequency_penalties=tensor([0.]), presence_penalties=tensor([0.]), repetition_penalties=tensor([0.]), output_token_ids=[], allowed_token_ids_mask=None, bad_words_token_ids={}, logitsprocs=<vllhttp://m.v1.sample.logits_processor.state.LogitsProcessors object at 0x754266f03c10>)
(EngineCore_DP0 pid=18740) [vllm-rbln] INFO 11-04 21:11:36 optimum_model_runner.py:893] Running dummy sampler config: no_penalty_topp_topk
(EngineCore_DP0 pid=18740) [vllm-rbln] INFO 11-04 21:11:36 optimum_model_runner.py:878] Running dummy compile with batch_size=1, vocab_size=128256
(EngineCore_DP0 pid=18740) [vllm-rbln] INFO 11-04 21:11:36 optimum_model_runner.py:881] Sampling metadata: SamplingMetadata(temperature=tensor([0.5000]), all_greedy=False, all_random=True, top_p=tensor([0.9000]), top_k=tensor([1], dtype=torch.int32), generators={}, max_num_logprobs=None, no_penalties=True, prompt_token_ids=None, frequency_penalties=tensor([0.]), presence_penalties=tensor([0.]), repetition_penalties=tensor([0.]), output_token_ids=[], allowed_token_ids_mask=None, bad_words_token_ids={}, logitsprocs=<vllhttp://m.v1.sample.logits_processor.state.LogitsProcessors object at 0x754266f03c10>)
(EngineCore_DP0 pid=18740) [vllm-rbln] INFO 11-04 21:11:38 optimum_model_runner.py:893] Running dummy sampler config: penalty_greedy
(EngineCore_DP0 pid=18740) [vllm-rbln] INFO 11-04 21:11:38 optimum_model_runner.py:878] Running dummy compile with batch_size=1, vocab_size=128256
(EngineCore_DP0 pid=18740) [vllm-rbln] INFO 11-04 21:11:38 optimum_model_runner.py:881] Sampling metadata: SamplingMetadata(temperature=None, all_greedy=True, all_random=False, top_p=None, top_k=None, generators={}, max_num_logprobs=None, no_penalties=False, prompt_token_ids=tensor([], size=(1, 0), dtype=torch.int64), frequency_penalties=tensor([0.1000]), presence_penalties=tensor([0.1000]), repetition_penalties=tensor([1.]), output_token_ids=[], allowed_token_ids_mask=None, bad_words_token_ids={}, logitsprocs=<vllhttp://m.v1.sample.logits_processor.state.LogitsProcessors object at 0x754266f03c10>)
(EngineCore_DP0 pid=18740) [vllm-rbln] INFO 11-04 21:11:38 optimum_model_runner.py:893] Running dummy sampler config: penalty_topp
(EngineCore_DP0 pid=18740) [vllm-rbln] INFO 11-04 21:11:38 optimum_model_runner.py:878] Running dummy compile with batch_size=1, vocab_size=128256
(EngineCore_DP0 pid=18740) [vllm-rbln] INFO 11-04 21:11:38 optimum_model_runner.py:881] Sampling metadata: SamplingMetadata(temperature=tensor([0.5000]), all_greedy=False, all_random=True, top_p=tensor([0.9000]), top_k=None, generators={}, max_num_logprobs=None, no_penalties=False, prompt_token_ids=tensor([], size=(1, 0), dtype=torch.int64), frequency_penalties=tensor([0.1000]), presence_penalties=tensor([0.1000]), repetition_penalties=tensor([1.]), output_token_ids=[], allowed_token_ids_mask=None, bad_words_token_ids={}, logitsprocs=<vllhttp://m.v1.sample.logits_processor.state.LogitsProcessors object at 0x754266f03c10>)
(EngineCore_DP0 pid=18740) [vllm-rbln] INFO 11-04 21:11:38 optimum_model_runner.py:893] Running dummy sampler config: penalty_topk
(EngineCore_DP0 pid=18740) [vllm-rbln] INFO 11-04 21:11:38 optimum_model_runner.py:878] Running dummy compile with batch_size=1, vocab_size=128256
(EngineCore_DP0 pid=18740) [vllm-rbln] INFO 11-04 21:11:38 optimum_model_runner.py:881] Sampling metadata: SamplingMetadata(temperature=tensor([0.5000]), all_greedy=False, all_random=True, top_p=None, top_k=tensor([1], dtype=torch.int32), generators={}, max_num_logprobs=None, no_penalties=False, prompt_token_ids=tensor([], size=(1, 0), dtype=torch.int64), frequency_penalties=tensor([0.1000]), presence_penalties=tensor([0.1000]), repetition_penalties=tensor([1.]), output_token_ids=[], allowed_token_ids_mask=None, bad_words_token_ids={}, logitsprocs=<vllhttp://m.v1.sample.logits_processor.state.LogitsProcessors object at 0x754266f03c10>)
(EngineCore_DP0 pid=18740) [vllm-rbln] INFO 11-04 21:11:38 optimum_model_runner.py:893] Running dummy sampler config: penalty_topp_topk
(EngineCore_DP0 pid=18740) [vllm-rbln] INFO 11-04 21:11:38 optimum_model_runner.py:878] Running dummy compile with batch_size=1, vocab_size=128256
(EngineCore_DP0 pid=18740) [vllm-rbln] INFO 11-04 21:11:38 optimum_model_runner.py:881] Sampling metadata: SamplingMetadata(temperature=tensor([0.5000]), all_greedy=False, all_random=True, top_p=tensor([0.9000]), top_k=tensor([1], dtype=torch.int32), generators={}, max_num_logprobs=None, no_penalties=False, prompt_token_ids=tensor([], size=(1, 0), dtype=torch.int64), frequency_penalties=tensor([0.1000]), presence_penalties=tensor([0.1000]), repetition_penalties=tensor([1.]), output_token_ids=[], allowed_token_ids_mask=None, bad_words_token_ids={}, logitsprocs=<vllhttp://m.v1.sample.logits_processor.state.LogitsProcessors object at 0x754266f03c10>)
(EngineCore_DP0 pid=18740) [vllm-rbln] INFO 11-04 21:11:38 core.py:218] init engine (profile, create kv cache, warmup model) took 14.36 seconds
(EngineCore_DP0 pid=18740) [vllm-rbln] WARNING 11-04 21:11:39 core.py:111] Using configured V1 scheduler class vllm_rbln.v1.core.optimum_scheduler.RBLNOptimumScheduler. This scheduler interface is not public and compatibility may not be maintained.
(EngineCore_DP0 pid=18740) [vllm-rbln] INFO 11-04 21:11:39 platform.py:145] original model_config.dtype = torch.float32
(EngineCore_DP0 pid=18740) [vllm-rbln] INFO 11-04 21:11:39 platform.py:152] RBLN model_config.dtype = torch.float32
(EngineCore_DP0 pid=18740) [vllm-rbln] WARNING 11-04 21:11:39 platform.py:209] uni is not supported on RBLN, fallback to mp distributed executor backend.
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:39 loggers.py:142] Engine 000: vllm cache_config_info with initialization after num_gpu_blocks is: 9
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:39 async_llm.py:180] Torch profiler disabled. AsyncLLM CPU traces will not be collected.
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:39 api_server.py:1692] Supported_tasks: ['generate']
(APIServer pid=18474) [vllm-rbln] WARNING 11-04 21:11:39 __init__.py:1695] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:39 serving_responses.py:130] Using default chat sampling params from model: {'temperature': 0.6, 'top_p': 0.9}
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:39 serving_chat.py:137] Using default chat sampling params from model: {'temperature': 0.6, 'top_p': 0.9}
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:39 serving_completion.py:76] Using default completion sampling params from model: {'temperature': 0.6, 'top_p': 0.9}
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:39 api_server.py:1971] Starting vLLM API server 0 on http://0.0.0.0:8000
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:39 launcher.py:36] Available routes are:
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:39 launcher.py:44] Route: /openapi.json, Methods: HEAD, GET
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:39 launcher.py:44] Route: /docs, Methods: HEAD, GET
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:39 launcher.py:44] Route: /docs/oauth2-redirect, Methods: HEAD, GET
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:39 launcher.py:44] Route: /redoc, Methods: HEAD, GET
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:39 launcher.py:44] Route: /health, Methods: GET
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:39 launcher.py:44] Route: /load, Methods: GET
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:39 launcher.py:44] Route: /ping, Methods: POST
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:39 launcher.py:44] Route: /ping, Methods: GET
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:39 launcher.py:44] Route: /tokenize, Methods: POST
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:39 launcher.py:44] Route: /detokenize, Methods: POST
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:39 launcher.py:44] Route: /v1/models, Methods: GET
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:39 launcher.py:44] Route: /version, Methods: GET
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:39 launcher.py:44] Route: /v1/responses, Methods: POST
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:39 launcher.py:44] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:39 launcher.py:44] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:39 launcher.py:44] Route: /v1/chat/completions, Methods: POST
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:39 launcher.py:44] Route: /v1/completions, Methods: POST
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:39 launcher.py:44] Route: /v1/embeddings, Methods: POST
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:39 launcher.py:44] Route: /pooling, Methods: POST
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:39 launcher.py:44] Route: /classify, Methods: POST
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:39 launcher.py:44] Route: /score, Methods: POST
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:39 launcher.py:44] Route: /v1/score, Methods: POST
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:39 launcher.py:44] Route: /v1/audio/transcriptions, Methods: POST
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:39 launcher.py:44] Route: /v1/audio/translations, Methods: POST
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:39 launcher.py:44] Route: /rerank, Methods: POST
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:39 launcher.py:44] Route: /v1/rerank, Methods: POST
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:39 launcher.py:44] Route: /v2/rerank, Methods: POST
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:39 launcher.py:44] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:39 launcher.py:44] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:39 launcher.py:44] Route: /invocations, Methods: POST
(APIServer pid=18474) [vllm-rbln] INFO 11-04 21:11:39 launcher.py:44] Route: /metrics, Methods: GET
(APIServer pid=18474) INFO: Started server process [18474]
(APIServer pid=18474) INFO: Waiting for application startup.
(APIServer pid=18474) INFO: Application startup complete.
No 8. 기타
* NHN 서버 포트와 도커 Port mapping (이 부분 NHN 서버 담당자가 늦게 해줘서 한달 날림 ToT)
- "30229:22"
- "30017:80"
- "30018:443"
- "30019:3306"
- "30020:8000"
- "30021:8080"
- "30022:3000"
- "30023:3003"
* 테스트
1. bge-m3
* dense_embedding
ubuntu@4b7f4bcac5a5:~/top/rbln-model-zoo/huggingface/transformers/sentence-similarity/bge-m3/dense_embedding$ python3 -m vllhttp://m.entrypoints.openai.api_server \
--model ./bge-m3 \
--served-model-name bge-m3-rbln \
--host 0.0.0.0 \
--port 8000 \
--api-key mykey123 \
--max-model-len 8192
* multi_vector
ubuntu@4b7f4bcac5a5:~/top/rbln-model-zoo/huggingface/transformers/sentence-similarity/bge-m3/multi_vector$ python3 -m vllhttp://m.entrypoints.openai.api_server \
--model ./bge-m3 \
--served-model-name bge-m3-mv-rbln \
--host 0.0.0.0 \
--port 8000 \
--api-key mykey123 \
--max-model-len 8192
* sparse_embedding
ubuntu@4b7f4bcac5a5:~/top/rbln-model-zoo/huggingface/transformers/sentence-similarity/bge-m3/sparse_embedding$ python3 -m vllhttp://m.entrypoints.openai.api_server \
--model ./bge-m3 \
--served-model-name bge-m3-sp-rbln \
--host 0.0.0.0 \
--port 8000 \
--api-key mykey123 \
--max-model-len 8192
* PC에서 접속 테스트
curl http://114.110.XXX.XXX:30020/v1/embeddings \
-H "Content-Type: application/json" \
-H "Authorization: Bearer mykey123" \
-d '{
"model": "bge-m3-sp-rbln",
"input": [
"대한민국 수도는 어디인가?",
"서울은 한국의 수도다."
]
}'
2. Llama-3.1-8B
ubuntu@4b7f4bcac5a5:~/top/rbln-model-zoo/huggingface/transformers/text2text-generation/llama/llama3.1-8b$ python3 -m vllhttp://m.entrypoints.openai.api_server \
--model ./Llama-3.1-8B-Instruct \
--served-model-name Llama-3.1-8B-Instruct \
--host 0.0.0.0 \
--port 8080 \
--api-key mykey123
* PC에서 접속 테스트
curl http://114.110.XXX.XXX:30021/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer mykey123" \
-d '{
"model": "Llama-3.1-8B-Instruct",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Hello!"}
],
"stream": true
}'
* NPU 정보
ubuntu@4b7f4bcac5a5:~$ rbln-stat
Tue Nov 4 09:34:08 2025
+-------------------------------------------------------------------------------------------------+
| Device Information KMD ver: 2.0.1 |
+-----+-----------+---------+---------------+------+---------+------+---------------------+-------+
| NPU | Name | Device | PCI BUS ID | Temp | Power | Perf | Memory(used/total) | Util |
+=====+===========+=========+===============+======+=========+======+=====================+=======+
| 0 | RBLN-CA12 | rbln0 | 0000:83:00.0 | 33C | 22.4W | P14 | 0.0B / 15.7GiB | 0.0 |
| 1 | RBLN-CA12 | rbln1 | 0000:84:00.0 | 34C | 22.9W | P14 | 0.0B / 15.7GiB | 0.0 |
| 2 | RBLN-CA12 | rbln2 | 0000:e3:00.0 | 32C | 22.6W | P14 | 0.0B / 15.7GiB | 0.0 |
| 3 | RBLN-CA12 | rbln3 | 0000:e4:00.0 | 31C | 21.9W | P14 | 0.0B / 15.7GiB | 0.0 |
+-----+-----------+---------+---------------+------+---------+------+---------------------+-------+
+-------------------------------------------------------------------------------------------------+
| Context Information |
+-----+---------------------+--------------+-----------+----------+------+---------------+--------+
| NPU | Process | PID | CTX | Priority | PTID | Memalloc | Status |
+=====+=====================+==============+===========+==========+======+===============+========+
| N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A |
+-----+---------------------+--------------+-----------+----------+------+---------------+--------+
ubuntu@4b7f4bcac5a5:~$
* Visual Code SSH 코드 연결 방법
1. Visual Code Extensions 에서 Remote-SSH 검색 후 Install
2. 처음에만 명령 팔레트 열고 Remote-SSH: Add New SSH Host
ssh -i ~/.ssh/nhn.pem ubuntu@ 114.110.XXX.XXX
저장 위치 물으면 ~/.ssh/config 선택
~/.ssh/config에 아래처럼 자동으로 생김:
Host 114.110.XXX.XXX
HostName 114.110.XXX.XXX
IdentityFile D:\Cloud\NHN\N2025-0173.pem
User ubuntu
Port 30XXX
3. 2번 수행 후 명령 팔레트(Ctrl+Shift+P)에서 Remote-SSH: Connect to Host... 입력
* benchmark
ubuntu@4b7f4bcac5a5:~$ python3 run_benchmark.py --models configs/models.yaml --params configs/params.yaml --prompts configs/prompts.txt --out outputs
* 용량정리
ubuntu@4b7f4bcac5a5:~/top/rbln-model-zoo/huggingface/transformers$ rm -rf ~/.cache/huggingface/hub
ubuntu@4b7f4bcac5a5:~/top/rbln-model-zoo/huggingface/transformers$ pip cache purge
Files removed: 339
ubuntu@4b7f4bcac5a5:~/top/rbln-model-zoo/huggingface/transformers$ rm -rf ~/.cache/torch
'AI' 카테고리의 다른 글
| [LLM 모델] 모델별 비교 (0) | 2025.11.23 |
|---|---|
| [Embedding 모델] CPU & NPU Max Length 별 비교 (0) | 2025.11.22 |
| [Embedding 모델] CPU & NPU Num Setence 별 비교 (0) | 2025.11.22 |
| [Embedding 모델] CPU & NPU Batch Size 별 비교 (0) | 2025.11.22 |