非常棒的推理项目FreeToken,据说非常快!
repo:GitHub - FlashML-org/FreeToken · GitHub
实践
下载模型
注意modelscope要升级到新版本,比如openi里面自带的版本低,下载会报错.
pip install modelscope -Umodelscope download --model empero-ai/Qwen3.8-4B-DistillDownloading snapshot of empero-ai/Qwen3.8-4B-Distill (model)…
2026-08-22 14:39:01,641 | INFO | modelscope_hub.download | Downloading 14 files from empero-ai/Qwen3.8-4B-Distill@master
Downloading: 100%|███████████████████████████| 14/14 [15:30<00:00, 66.49s/file]
✓ Snapshot ready at /home/skywalk/.cache/modelscope/models/empero-ai--Qwen3.8-4B-Distill/snapshots/master
安装FreeToken
pip install uv uv venv uv pip install "freetoken[accel]"uv 安装的时候如果慢就加上清华源
-i https://pypi.tuna.tsinghua.edu.cn/simple启动
ft serve --model ~/models/Qwen3.6-35B-A3B # ft serve --model /home/skywalk/.cache/modelscope/models/empero-ai--Qwen3.8-4B-Distill/snapshots/master/可惜啊,这个显卡太老了,还是跑不起来
(root) root@fb98:~# ft serve --model /home/skywalk/.cache/modelscope/models/empero-ai--Qwen3.8-4B-Distill/snapshots/master/ [2026-08-22|15:56:13] INFO Parsed arguments: ServerArgs(model_path='/home/skywalk/.cache/modelscope/models/empero-ai--Qwen3.8-4B-Distill/snapshots/master/', tp_info=DistributedInfo(rank=0, size=1), dtype=torch.bfloat16, max_running_req=4, attention_backend='auto', moe_backend='auto', nvfp4_backend='triton', expert_load='auto', moe_cache_size=0, moe_cache_rate=None, moe_cache_auto=False, kv_reserve_tokens=8192, moe_cache_policy='lru', moe_prefill_overlap=True, moe_prefill_hit_d2d=False, moe_collect_stats=False, moe_cpu_threads=0, moe_cpu_layers=None, moe_hybrid_max_fetch=-1, cuda_graph_bs=None, cuda_graph_max_bs=None, page_size=1, memory_ratio=0.9, linear_state_cache_ratio=2.0, swa_full_tokens_ratio=0.2, swa_num_pages_override=None, distributed_timeout=60.0, use_dummy_weight=False, use_pynccl=True, max_seq_len_override=None, num_page_override=None, num_token_override=None, max_extend_tokens=8192, cache_type='radix', offline_mode=False, decode_log_interval=40, special_token_ckpt=False, _unique_suffix='.pid=5761', server_host='127.0.0.1', server_port=1919, num_tokenizer=0, silent_output=False, shell_mode=False, served_model_name='master', tool_call_parser='qwen3_coder', reasoning_parser='qwen3', sampling_defaults='model', max_output_tokens=None, enable_cache_report=False, cors_origins='tauri://localhost,http://tauri.localhost,http://localhost:1420') [2026-08-22|15:56:13|FrontendAPI] INFO Default sampling config (source=framework): temperature=0.0, top_k=-1, top_p=1.0 INFO: Started server process [5761] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://127.0.0.1:1919 (Press CTRL+C to quit) /root/.venv/lib/python3.10/site-packages/freetoken/engine/engine.py:967: FutureWarning: torch.cuda._set_allocator_settings is deprecated. Use torch._C._accelerator_setAllocatorSettings instead. torch.cuda.memory._set_allocator_settings("expandable_segments:True") [2026-08-22|15:56:19|core|rank=0] INFO Enabled expandable_segments (override via PYTORCH_ALLOC_CONF) /root/.venv/lib/python3.10/site-packages/torch/cuda/__init__.py:180: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 304: OS call failed or operation not supported on this OS (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:119.) return torch._C._cuda_getDeviceCount() > 0 [2026-08-22|15:56:19|core|rank=0] INFO Auto-selected attention backend: fi [2026-08-22|15:56:19|core|rank=0] INFO Resolved config: attention_backend='fi', cache_type='hybrid_radix', page_size=1 [2026-08-22|15:56:19|FrontendAPI] ERROR Backend supervisor: RuntimeError: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 304: OS call failed or operation not supported on this OS Process freetoken-TP0-scheduler: Traceback (most recent call last): File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/root/.venv/lib/python3.10/site-packages/freetoken/server/launch.py", line 74, in _run_scheduler scheduler = Scheduler(args) File "/root/.venv/lib/python3.10/site-packages/freetoken/scheduler/scheduler.py", line 64, in __init__ self.engine = Engine(config) File "/root/.venv/lib/python3.10/site-packages/freetoken/engine/engine.py", line 300, in __init__ torch.cuda.set_device(self.device) File "/root/.venv/lib/python3.10/site-packages/torch/cuda/__init__.py", line 638, in set_device torch._C._cuda_setDevice(device) File "/root/.venv/lib/python3.10/site-packages/torch/cuda/__init__.py", line 478, in _lazy_init torch._C._cuda_init() RuntimeError: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 304: OS call failed or operation not supported on this OS [2026-08-22|15:56:29|FrontendAPI] ERROR Backend worker is gone and cannot be restarted; stopping the API server INFO: Shutting down INFO: Waiting for application shutdown. INFO: Application shutdown complete. INFO: Finished server process [5761] /usr/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 3 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d '在openi尝试
设置加速镜像
export HF_ENDPOINT=https://hf-mirror.com启动
ft serve --model empero-ai/Qwen3.8-4B-Distill使用先前下载的模型,模型下载到:
/mnt/workspace/.cache/modelscope/models/empero-ai--Qwen3.8-4B-Distill/snapshots/masterft serve --model /mnt/workspace/.cache/modelscope/models/empero-ai--Qwen3.8-4B-Distill/snapshots/master调试
碰到报错
启动模型报错,使用命令:ft serve --model /mnt/workspace/.cache/modelscope/models/empero-ai--Qwen3.8-4B-Distill/snapshots/master,报错信息:
: /usr/local/cuda/bin/nvcc --generate-dependencies-with-compile -MF /root/.cache/flashinfer/0.6.18/80/cached_ops/batch_prefill_with_kv_cache_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_qk_256_head_dim_vo_256_posenc_0_use_swa_False_use_logits_cap_False_f16qk_False/batch_prefill_with_kv_cache_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_qk_256_head_dim_vo_256_posenc_0_use_swa_False_use_logits_cap_False_f16qk_False_batch_prefill_jit_binding.cuda.o.d -DPy_LIMITED_API=0x03090000 -D_GLIBCXX_USE_CXX11_ABI=1 -I/root/.venv/lib/python3.10/site-packages/flashinfer/data/cccl/cub -I/root/.venv/lib/python3.10/site-packages/flashinfer/data/cccl/libcudacxx/include -I/root/.venv/lib/python3.10/site-packages/flashinfer/data/cccl/thrust -isystem /opt/conda/include/python3.10 -isystem /usr/local/cuda/include -isystem /root/.venv/lib/python3.10/site-packages/tvm_ffi/include -isystem /root/.venv/lib/python3.10/site-packages/tvm_ffi/include -isystem /root/.venv/lib/python3.10/site-packages/flashinfer/data/include -isystem /root/.venv/lib/python3.10/site-packages/flashinfer/data/csrc -isystem /root/.venv/lib/python3.10/site-packages/flashinfer/data/cutlass/include -isystem /root/.venv/lib/python3.10/site-packages/flashinfer/data/cutlass/tools/util/include -isystem /root/.venv/lib/python3.10/site-packages/flashinfer/data/spdlog/include --compiler-options=-fPIC --expt-relaxed-constexpr -gencode=arch=compute_80,code=sm_80 -DFLASHINFER_ENABLE_FP8_E8M0 -DFLASHINFER_ENABLE_FP4_E2M1 -std=c++17 --threads=1 -use_fast_math -Xfatbin=-compress-all --compress-mode=size -DFLASHINFER_ENABLE_F16 -DFLASHINFER_ENABLE_BF16 -DFLASHINFER_ENABLE_FP8_E4M3 -DFLASHINFER_ENABLE_FP8_E5M2 -DNDEBUG -O3 -c /root/.cache/flashinfer/0.6.18/80/generated/batch_prefill_with_kv_cache_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_qk_256_head_dim_vo_256_posenc_0_use_swa_False_use_logits_cap_False_f16qk_False/batch_prefill_jit_binding.cu -o /root/.cache/flashinfer/0.6.18/80/cached_ops/batch_prefill_with_kv_cache_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_qk_256_head_dim_vo_256_posenc_0_use_swa_False_use_logits_cap_False_f16qk_False/batch_prefill_with_kv_cache_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_qk_256_head_dim_vo_256_posenc_0_use_swa_False_use_logits_cap_False_f16qk_False_batch_prefill_jit_binding.cuda.o
nvcc fatal : Unknown option '--compress-mode=size'
ninja: build stopped: subcommand failed.
[2026-08-30|00:52:01|FrontendAPI] ERROR Backend worker is gone and cannot be restarted; stopping the API server
INFO: Shutting down
INFO: Waiting for application shutdown.
INFO: Application shutdown complete.
INFO: Finished server process [1751]
已终止
(root) root@c85df02bb3364d1ba93b8f3bdb8fc6ea-task0-0:~# /opt/conda/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 3 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
说是cuda版本低,问题我这就是cuda13啊! 但是nvcc确实低
root@c85df02bb3364d1ba93b8f3bdb8fc6ea-task0-0:~# nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Mon_Apr__3_17:16:06_PDT_2023
Cuda compilation tools, release 12.1, V12.1.105
Build cuda_12.1.r12.1/compiler.32688072_0
升级所有的
pip install pip -U uv pip install uv -U uv pip install "freetoken[accel]" -U还是不行!