28h100 nodes run DeepSeek-R1 671B FP8，get garbled characters in response. #3426

echozyr2001 · 2025-02-09T13:02:27Z

echozyr2001
Feb 9, 2025

Our company has more than 200 8*h100 machines, but I encountered a problem when I was deploying and testing. If it can be solved, there will be a stable API on the market.

I am launching DeepSeek-R1 671B FP8 instances on two 8*h100 servers via docker, but the chat return value is always garbled characters.

Here is the command I started:

node1

docker run --gpus all \
		--shm-size 32g \
		--ipc=host \
		--network=host \
		-v ~/t/DeepSeek-R1:/root/DeepSeek-R1 \
		--name sglang_multinode1 \
		-it \
    --rm \
    --env "GLOO_SOCKET_IFNAME=enp83s0f0np0" \
    --env "NCCL_SOCKET_IFNAME=enp83s0f0np0" \
		lmsysorg/sglang:latest \
    python3 -m sglang.launch_server --model-path /root/DeepSeek-R1 --tp 16 --dist-init-addr 10.157.104.101:20000 --nnodes 2 --node-rank 0 --trust-remote-code --host 0.0.0.0 --port 40000

node2

docker run --gpus all \
		--shm-size 32g \
		--ipc=host \
		--network=host \
		-v ~/t/DeepSeek-R1:/root/DeepSeek-R1 \
		--name sglang_multinode2 \
		-it \
    --rm \
    --env "GLOO_SOCKET_IFNAME=enp83s0f0np0" \
    --env "NCCL_SOCKET_IFNAME=enp83s0f0np0" \
		lmsysorg/sglang:latest \
    python3 -m sglang.launch_server --model-path /root/DeepSeek-R1 --tp 16 --dist-init-addr 10.157.104.101:20000 --nnodes 2 --node-rank 1 --trust-remote-code --host 0.0.0.0 --port 40000

It ran successfully and I got the following log

==========
== CUDA ==
==========

CUDA Version 12.5.1

Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

[2025-02-09 04:40:57] server_args=ServerArgs(model_path='/root/DeepSeek-R1', tokenizer_path='/root/DeepSeek-R1', tokenizer_mode='auto', load_format='auto', trust_remote_code=True, dtype='auto', kv_cache_dtype='fp8_e5m2', quantization_param_path=None, quantization='fp8', context_length=32000, device='cuda', served_model_name='/root/DeepSeek-R1', chat_template=None, is_embedding=False, revision=None, skip_tokenizer_init=False, host='0.0.0.0', port=40000, mem_fraction_static=0.79, max_running_requests=None, max_total_tokens=None, chunked_prefill_size=8192, max_prefill_tokens=16384, schedule_policy='lpm', schedule_conservativeness=1.0, cpu_offload_gb=0, prefill_only_one_req=False, tp_size=16, stream_interval=1, stream_output=False, random_seed=1071056713, constrained_json_whitespace_pattern=None, watchdog_timeout=300, download_dir=None, base_gpu_id=0, log_level='info', log_level_http=None, log_requests=False, show_time_cost=False, enable_metrics=False, decode_log_interval=40, api_key=None, file_storage_pth='sglang_storage', enable_cache_report=False, dp_size=1, load_balance_method='round_robin', ep_size=1, dist_init_addr='10.157.104.101:20000', nnodes=2, node_rank=0, json_model_override_args='{}', lora_paths=None, max_loras_per_batch=8, lora_backend='triton', attention_backend='flashinfer', sampling_backend='flashinfer', grammar_backend='outlines', speculative_draft_model_path=None, speculative_algorithm=None, speculative_num_steps=5, speculative_num_draft_tokens=64, speculative_eagle_topk=8, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, disable_radix_cache=False, disable_jump_forward=False, disable_cuda_graph=True, disable_cuda_graph_padding=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, disable_mla=False, disable_overlap_schedule=False, enable_mixed_chunk=False, enable_dp_attention=False, enable_ep_moe=False, enable_torch_compile=True, torch_compile_max_bs=32, cuda_graph_max_bs=160, cuda_graph_bs=None, torchao_config='', enable_nan_detection=False, enable_p2p_check=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, allow_auto_truncate=False, enable_custom_logit_processor=False, tool_call_parser=None, enable_hierarchical_cache=False)
[2025-02-09 04:41:07 TP6] MLA optimization is turned on. Use triton backend.
[2025-02-09 04:41:07 TP6] Init torch distributed begin.
[2025-02-09 04:41:08 TP0] MLA optimization is turned on. Use triton backend.
[2025-02-09 04:41:08 TP0] Init torch distributed begin.
[2025-02-09 04:41:08 TP2] MLA optimization is turned on. Use triton backend.
[2025-02-09 04:41:08 TP2] Init torch distributed begin.
[2025-02-09 04:41:08 TP1] MLA optimization is turned on. Use triton backend.
[2025-02-09 04:41:08 TP1] Init torch distributed begin.
[2025-02-09 04:41:08 TP5] MLA optimization is turned on. Use triton backend.
[2025-02-09 04:41:08 TP5] Init torch distributed begin.
[2025-02-09 04:41:08 TP7] MLA optimization is turned on. Use triton backend.
[2025-02-09 04:41:08 TP7] Init torch distributed begin.
[2025-02-09 04:41:08 TP3] MLA optimization is turned on. Use triton backend.
[2025-02-09 04:41:08 TP3] Init torch distributed begin.
[2025-02-09 04:41:08 TP4] MLA optimization is turned on. Use triton backend.
[2025-02-09 04:41:08 TP4] Init torch distributed begin.
[2025-02-09 04:41:37 TP4] sglang is using nccl==2.21.5
[2025-02-09 04:41:37 TP3] sglang is using nccl==2.21.5
[2025-02-09 04:41:37 TP0] sglang is using nccl==2.21.5
[2025-02-09 04:41:37 TP1] sglang is using nccl==2.21.5
[2025-02-09 04:41:37 TP7] sglang is using nccl==2.21.5
[2025-02-09 04:41:37 TP6] sglang is using nccl==2.21.5
[2025-02-09 04:41:37 TP5] sglang is using nccl==2.21.5
[2025-02-09 04:41:37 TP2] sglang is using nccl==2.21.5
[2025-02-09 04:41:43 TP1] Custom allreduce is disabled because this process group spans across nodes.
[2025-02-09 04:41:43 TP0] Custom allreduce is disabled because this process group spans across nodes.
[2025-02-09 04:41:43 TP3] Custom allreduce is disabled because this process group spans across nodes.
[2025-02-09 04:41:43 TP4] Custom allreduce is disabled because this process group spans across nodes.
[2025-02-09 04:41:43 TP5] Custom allreduce is disabled because this process group spans across nodes.
[2025-02-09 04:41:43 TP2] Custom allreduce is disabled because this process group spans across nodes.
[2025-02-09 04:41:43 TP6] Custom allreduce is disabled because this process group spans across nodes.
[2025-02-09 04:41:43 TP7] Custom allreduce is disabled because this process group spans across nodes.
[2025-02-09 04:41:47 TP1] Load weight begin. avail mem=78.24 GB
[2025-02-09 04:41:47 TP7] Load weight begin. avail mem=78.25 GB
[2025-02-09 04:41:47 TP5] Load weight begin. avail mem=78.24 GB
[2025-02-09 04:41:47 TP0] Load weight begin. avail mem=78.28 GB
[2025-02-09 04:41:47 TP2] Load weight begin. avail mem=78.24 GB
[2025-02-09 04:41:47 TP6] Load weight begin. avail mem=78.24 GB
[2025-02-09 04:41:47 TP4] Load weight begin. avail mem=78.24 GB
[2025-02-09 04:41:47 TP3] Load weight begin. avail mem=78.24 GB
[2025-02-09 04:41:47 TP6] Detected fp8 checkpoint. Please note that the format is experimental and subject to change.
[2025-02-09 04:41:47 TP2] Detected fp8 checkpoint. Please note that the format is experimental and subject to change.
[2025-02-09 04:41:47 TP1] Detected fp8 checkpoint. Please note that the format is experimental and subject to change.
[2025-02-09 04:41:47 TP4] Detected fp8 checkpoint. Please note that the format is experimental and subject to change.
[2025-02-09 04:41:47 TP5] Detected fp8 checkpoint. Please note that the format is experimental and subject to change.
[2025-02-09 04:41:47 TP3] Detected fp8 checkpoint. Please note that the format is experimental and subject to change.
[2025-02-09 04:41:47 TP0] Detected fp8 checkpoint. Please note that the format is experimental and subject to change.
[2025-02-09 04:41:47 TP7] Detected fp8 checkpoint. Please note that the format is experimental and subject to change.
Cache shape torch.Size([163840, 64])
Cache shape torch.Size([163840, 64])
Cache shape torch.Size([163840, 64])
Cache shape torch.Size([163840, 64])
Cache shape torch.Size([163840, 64])
Cache shape torch.Size([163840, 64])
Cache shape torch.Size([163840, 64])
Cache shape torch.Size([163840, 64])
Loading safetensors checkpoint shards:   0% Completed | 0/158 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:   1% Completed | 1/158 [00:00<00:30,  5.20it/s]
Loading safetensors checkpoint shards:   1% Completed | 2/158 [00:00<00:24,  6.46it/s]
Loading safetensors checkpoint shards:   2% Completed | 3/158 [00:00<00:23,  6.58it/s]
Loading safetensors checkpoint shards:   3% Completed | 4/158 [00:00<00:23,  6.50it/s]
Loading safetensors checkpoint shards:   4% Completed | 6/158 [00:00<00:18,  8.04it/s]
Loading safetensors checkpoint shards:   4% Completed | 7/158 [00:00<00:19,  7.60it/s]
Loading safetensors checkpoint shards:   5% Completed | 8/158 [00:01<00:20,  7.41it/s]
Loading safetensors checkpoint shards:   6% Completed | 9/158 [00:01<00:20,  7.20it/s]
Loading safetensors checkpoint shards:   6% Completed | 10/158 [00:01<00:20,  7.19it/s]
Loading safetensors checkpoint shards:   7% Completed | 11/158 [00:01<00:20,  7.06it/s]
Loading safetensors checkpoint shards:   8% Completed | 12/158 [00:01<00:20,  7.05it/s]
Loading safetensors checkpoint shards:   8% Completed | 13/158 [00:01<00:21,  6.90it/s]
Loading safetensors checkpoint shards:   9% Completed | 14/158 [00:01<00:20,  6.91it/s]
Loading safetensors checkpoint shards:   9% Completed | 15/158 [00:02<00:20,  6.84it/s]
Loading safetensors checkpoint shards:  10% Completed | 16/158 [00:02<00:20,  6.90it/s]
Loading safetensors checkpoint shards:  11% Completed | 17/158 [00:02<00:20,  6.90it/s]
Loading safetensors checkpoint shards:  11% Completed | 18/158 [00:02<00:20,  6.80it/s]
Loading safetensors checkpoint shards:  12% Completed | 19/158 [00:02<00:20,  6.84it/s]
Loading safetensors checkpoint shards:  13% Completed | 20/158 [00:02<00:20,  6.73it/s]
Loading safetensors checkpoint shards:  13% Completed | 21/158 [00:03<00:20,  6.72it/s]
Loading safetensors checkpoint shards:  14% Completed | 22/158 [00:03<00:20,  6.72it/s]
Loading safetensors checkpoint shards:  15% Completed | 23/158 [00:03<00:19,  6.83it/s]
Loading safetensors checkpoint shards:  15% Completed | 24/158 [00:03<00:19,  6.73it/s]
Loading safetensors checkpoint shards:  16% Completed | 25/158 [00:03<00:19,  6.83it/s]
Loading safetensors checkpoint shards:  16% Completed | 26/158 [00:03<00:19,  6.75it/s]
Loading safetensors checkpoint shards:  18% Completed | 28/158 [00:03<00:16,  7.71it/s]
Loading safetensors checkpoint shards:  18% Completed | 29/158 [00:04<00:17,  7.53it/s]
Loading safetensors checkpoint shards:  19% Completed | 30/158 [00:04<00:17,  7.33it/s]
Loading safetensors checkpoint shards:  20% Completed | 31/158 [00:04<00:17,  7.12it/s]
Loading safetensors checkpoint shards:  20% Completed | 32/158 [00:04<00:18,  6.96it/s]
Loading safetensors checkpoint shards:  21% Completed | 33/158 [00:04<00:18,  6.90it/s]
Loading safetensors checkpoint shards:  22% Completed | 34/158 [00:04<00:18,  6.86it/s]
Loading safetensors checkpoint shards:  22% Completed | 35/158 [00:05<00:17,  6.84it/s]
Loading safetensors checkpoint shards:  23% Completed | 36/158 [00:05<00:17,  6.79it/s]
Loading safetensors checkpoint shards:  23% Completed | 37/158 [00:05<00:17,  6.88it/s]
Loading safetensors checkpoint shards:  24% Completed | 38/158 [00:05<00:17,  6.79it/s]
Loading safetensors checkpoint shards:  25% Completed | 39/158 [00:05<00:17,  6.84it/s]
Loading safetensors checkpoint shards:  25% Completed | 40/158 [00:05<00:17,  6.70it/s]
Loading safetensors checkpoint shards:  26% Completed | 41/158 [00:05<00:17,  6.65it/s]
Loading safetensors checkpoint shards:  27% Completed | 42/158 [00:06<00:17,  6.67it/s]
Loading safetensors checkpoint shards:  27% Completed | 43/158 [00:06<00:17,  6.61it/s]
Loading safetensors checkpoint shards:  28% Completed | 44/158 [00:06<00:16,  6.77it/s]
Loading safetensors checkpoint shards:  28% Completed | 45/158 [00:06<00:16,  6.70it/s]
Loading safetensors checkpoint shards:  29% Completed | 46/158 [00:06<00:16,  6.78it/s]
Loading safetensors checkpoint shards:  30% Completed | 47/158 [00:06<00:16,  6.81it/s]
Loading safetensors checkpoint shards:  30% Completed | 48/158 [00:06<00:15,  6.90it/s]
Loading safetensors checkpoint shards:  31% Completed | 49/158 [00:07<00:16,  6.81it/s]
Loading safetensors checkpoint shards:  32% Completed | 50/158 [00:07<00:15,  6.88it/s]
Loading safetensors checkpoint shards:  32% Completed | 51/158 [00:07<00:15,  6.81it/s]
Loading safetensors checkpoint shards:  33% Completed | 52/158 [00:07<00:15,  6.76it/s]
Loading safetensors checkpoint shards:  34% Completed | 53/158 [00:07<00:15,  6.80it/s]
Loading safetensors checkpoint shards:  35% Completed | 55/158 [00:07<00:13,  7.74it/s]
Loading safetensors checkpoint shards:  35% Completed | 56/158 [00:08<00:13,  7.50it/s]
Loading safetensors checkpoint shards:  36% Completed | 57/158 [00:08<00:13,  7.26it/s]
Loading safetensors checkpoint shards:  37% Completed | 58/158 [00:08<00:14,  7.07it/s]
Loading safetensors checkpoint shards:  37% Completed | 59/158 [00:08<00:14,  7.05it/s]
Loading safetensors checkpoint shards:  38% Completed | 60/158 [00:08<00:14,  6.89it/s]
Loading safetensors checkpoint shards:  39% Completed | 61/158 [00:08<00:14,  6.83it/s]
Loading safetensors checkpoint shards:  39% Completed | 62/158 [00:08<00:14,  6.71it/s]
Loading safetensors checkpoint shards:  40% Completed | 63/158 [00:09<00:14,  6.77it/s]
Loading safetensors checkpoint shards:  41% Completed | 64/158 [00:09<00:13,  6.85it/s]
Loading safetensors checkpoint shards:  41% Completed | 65/158 [00:09<00:13,  6.72it/s]
Loading safetensors checkpoint shards:  42% Completed | 66/158 [00:09<00:13,  6.64it/s]
Loading safetensors checkpoint shards:  42% Completed | 67/158 [00:09<00:13,  6.69it/s]
Loading safetensors checkpoint shards:  43% Completed | 68/158 [00:09<00:13,  6.80it/s]
Loading safetensors checkpoint shards:  44% Completed | 70/158 [00:10<00:11,  7.74it/s]
Loading safetensors checkpoint shards:  45% Completed | 71/158 [00:10<00:11,  7.42it/s]
Loading safetensors checkpoint shards:  46% Completed | 72/158 [00:10<00:11,  7.21it/s]
Loading safetensors checkpoint shards:  46% Completed | 73/158 [00:10<00:12,  7.01it/s]
Loading safetensors checkpoint shards:  47% Completed | 74/158 [00:10<00:12,  6.87it/s]
Loading safetensors checkpoint shards:  47% Completed | 75/158 [00:10<00:12,  6.82it/s]
Loading safetensors checkpoint shards:  48% Completed | 76/158 [00:10<00:12,  6.78it/s]
Loading safetensors checkpoint shards:  49% Completed | 77/158 [00:11<00:12,  6.66it/s]
Loading safetensors checkpoint shards:  49% Completed | 78/158 [00:11<00:11,  6.72it/s]
Loading safetensors checkpoint shards:  50% Completed | 79/158 [00:11<00:11,  6.83it/s]
Loading safetensors checkpoint shards:  51% Completed | 80/158 [00:11<00:11,  6.75it/s]
Loading safetensors checkpoint shards:  51% Completed | 81/158 [00:11<00:11,  6.82it/s]
Loading safetensors checkpoint shards:  52% Completed | 82/158 [00:11<00:11,  6.72it/s]
Loading safetensors checkpoint shards:  53% Completed | 83/158 [00:11<00:11,  6.72it/s]
Loading safetensors checkpoint shards:  53% Completed | 84/158 [00:12<00:10,  6.77it/s]
Loading safetensors checkpoint shards:  54% Completed | 85/158 [00:12<00:10,  6.66it/s]
Loading safetensors checkpoint shards:  54% Completed | 86/158 [00:12<00:10,  6.77it/s]
Loading safetensors checkpoint shards:  55% Completed | 87/158 [00:12<00:10,  6.79it/s]
Loading safetensors checkpoint shards:  56% Completed | 88/158 [00:12<00:10,  6.74it/s]
Loading safetensors checkpoint shards:  56% Completed | 89/158 [00:12<00:10,  6.65it/s]
Loading safetensors checkpoint shards:  57% Completed | 90/158 [00:13<00:10,  6.77it/s]
Loading safetensors checkpoint shards:  58% Completed | 91/158 [00:13<00:10,  6.70it/s]
Loading safetensors checkpoint shards:  59% Completed | 93/158 [00:13<00:08,  7.81it/s]
Loading safetensors checkpoint shards:  59% Completed | 94/158 [00:13<00:08,  7.62it/s]
Loading safetensors checkpoint shards:  60% Completed | 95/158 [00:13<00:08,  7.35it/s]
Loading safetensors checkpoint shards:  61% Completed | 96/158 [00:13<00:08,  7.12it/s]
Loading safetensors checkpoint shards:  61% Completed | 97/158 [00:13<00:08,  7.04it/s]
Loading safetensors checkpoint shards:  62% Completed | 98/158 [00:14<00:08,  6.84it/s]
Loading safetensors checkpoint shards:  63% Completed | 99/158 [00:14<00:08,  6.83it/s]
Loading safetensors checkpoint shards:  63% Completed | 100/158 [00:14<00:08,  6.75it/s]
Loading safetensors checkpoint shards:  64% Completed | 101/158 [00:14<00:08,  6.71it/s]
Loading safetensors checkpoint shards:  65% Completed | 102/158 [00:14<00:08,  6.66it/s]
Loading safetensors checkpoint shards:  65% Completed | 103/158 [00:14<00:08,  6.62it/s]
Loading safetensors checkpoint shards:  66% Completed | 104/158 [00:15<00:07,  6.76it/s]
Loading safetensors checkpoint shards:  66% Completed | 105/158 [00:15<00:07,  6.74it/s]
Loading safetensors checkpoint shards:  67% Completed | 106/158 [00:15<00:07,  6.81it/s]
Loading safetensors checkpoint shards:  68% Completed | 107/158 [00:15<00:07,  6.78it/s]
Loading safetensors checkpoint shards:  68% Completed | 108/158 [00:15<00:07,  6.70it/s]
Loading safetensors checkpoint shards:  69% Completed | 109/158 [00:15<00:07,  6.61it/s]
Loading safetensors checkpoint shards:  70% Completed | 110/158 [00:15<00:07,  6.67it/s]
Loading safetensors checkpoint shards:  70% Completed | 111/158 [00:16<00:06,  6.74it/s]
Loading safetensors checkpoint shards:  71% Completed | 112/158 [00:16<00:06,  6.65it/s]
Loading safetensors checkpoint shards:  72% Completed | 113/158 [00:16<00:06,  6.78it/s]
Loading safetensors checkpoint shards:  72% Completed | 114/158 [00:16<00:06,  6.70it/s]
Loading safetensors checkpoint shards:  73% Completed | 116/158 [00:16<00:05,  7.70it/s]
Loading safetensors checkpoint shards:  74% Completed | 117/158 [00:16<00:05,  7.53it/s]
Loading safetensors checkpoint shards:  75% Completed | 118/158 [00:17<00:05,  7.26it/s]
Loading safetensors checkpoint shards:  75% Completed | 119/158 [00:17<00:05,  7.15it/s]
Loading safetensors checkpoint shards:  76% Completed | 120/158 [00:17<00:05,  7.04it/s]
Loading safetensors checkpoint shards:  77% Completed | 121/158 [00:17<00:05,  7.03it/s]
Loading safetensors checkpoint shards:  77% Completed | 122/158 [00:17<00:05,  6.88it/s]
Loading safetensors checkpoint shards:  78% Completed | 123/158 [00:17<00:05,  6.82it/s]
Loading safetensors checkpoint shards:  78% Completed | 124/158 [00:17<00:05,  6.68it/s]
Loading safetensors checkpoint shards:  79% Completed | 125/158 [00:18<00:04,  6.71it/s]
Loading safetensors checkpoint shards:  80% Completed | 126/158 [00:18<00:04,  6.66it/s]
Loading safetensors checkpoint shards:  80% Completed | 127/158 [00:18<00:04,  6.68it/s]
Loading safetensors checkpoint shards:  81% Completed | 128/158 [00:18<00:04,  6.63it/s]
Loading safetensors checkpoint shards:  82% Completed | 129/158 [00:18<00:04,  6.71it/s]
Loading safetensors checkpoint shards:  82% Completed | 130/158 [00:18<00:04,  6.80it/s]
Loading safetensors checkpoint shards:  83% Completed | 131/158 [00:18<00:03,  6.85it/s]
Loading safetensors checkpoint shards:  84% Completed | 132/158 [00:19<00:03,  6.83it/s]
Loading safetensors checkpoint shards:  84% Completed | 133/158 [00:19<00:03,  7.31it/s]
[2025-02-09 04:42:07 TP3] Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=37.78 GB
[2025-02-09 04:42:07 TP4] Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=37.78 GB
Loading safetensors checkpoint shards:  85% Completed | 134/158 [00:19<00:03,  7.24it/s]
Loading safetensors checkpoint shards:  85% Completed | 135/158 [00:19<00:03,  7.08it/s]
Loading safetensors checkpoint shards:  86% Completed | 136/158 [00:19<00:03,  6.95it/s]
Loading safetensors checkpoint shards:  87% Completed | 137/158 [00:19<00:03,  6.81it/s]
Loading safetensors checkpoint shards:  87% Completed | 138/158 [00:19<00:02,  6.85it/s]
Loading safetensors checkpoint shards:  88% Completed | 139/158 [00:20<00:02,  6.78it/s]
Loading safetensors checkpoint shards:  89% Completed | 140/158 [00:20<00:02,  6.79it/s]
Loading safetensors checkpoint shards:  89% Completed | 141/158 [00:20<00:02,  6.82it/s]
[2025-02-09 04:42:08 TP2] Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=37.78 GB
Loading safetensors checkpoint shards:  90% Completed | 142/158 [00:20<00:02,  6.94it/s]
Loading safetensors checkpoint shards:  91% Completed | 143/158 [00:20<00:02,  6.81it/s]
Loading safetensors checkpoint shards:  91% Completed | 144/158 [00:20<00:02,  6.80it/s]
Loading safetensors checkpoint shards:  92% Completed | 145/158 [00:20<00:01,  6.76it/s]
Loading safetensors checkpoint shards:  92% Completed | 146/158 [00:21<00:01,  6.88it/s]
Loading safetensors checkpoint shards:  93% Completed | 147/158 [00:21<00:01,  6.87it/s]
Loading safetensors checkpoint shards:  94% Completed | 148/158 [00:21<00:01,  6.77it/s]
Loading safetensors checkpoint shards:  94% Completed | 149/158 [00:21<00:01,  6.73it/s]
Loading safetensors checkpoint shards:  95% Completed | 150/158 [00:21<00:01,  6.79it/s]
Loading safetensors checkpoint shards:  96% Completed | 151/158 [00:21<00:01,  6.79it/s]
[2025-02-09 04:42:09 TP7] Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=37.79 GB
Loading safetensors checkpoint shards:  97% Completed | 154/158 [00:22<00:00, 10.72it/s]
Loading safetensors checkpoint shards:  99% Completed | 156/158 [00:22<00:00, 11.86it/s]
[2025-02-09 04:42:10 TP6] Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=37.78 GB
Loading safetensors checkpoint shards: 100% Completed | 158/158 [00:22<00:00,  9.62it/s]
Loading safetensors checkpoint shards: 100% Completed | 158/158 [00:22<00:00,  7.04it/s]

[2025-02-09 04:42:10 TP1] Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=37.78 GB
[2025-02-09 04:42:10 TP0] Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=37.82 GB
[2025-02-09 04:42:10 TP5] Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=37.78 GB
[2025-02-09 04:42:27 TP2] Memory pool end. avail mem=15.85 GB
[2025-02-09 04:42:27 TP0] Memory pool end. avail mem=15.89 GB
[2025-02-09 04:42:27 TP5] Memory pool end. avail mem=15.85 GB
[2025-02-09 04:42:27 TP4] Memory pool end. avail mem=15.85 GB
[2025-02-09 04:42:27 TP1] Memory pool end. avail mem=15.85 GB
[2025-02-09 04:42:27 TP7] Memory pool end. avail mem=15.86 GB
[2025-02-09 04:42:27 TP6] Memory pool end. avail mem=15.85 GB
[2025-02-09 04:42:27 TP3] Memory pool end. avail mem=15.85 GB
[2025-02-09 04:42:27 TP5] The following error message 'operation scheduled before its operands' can be ignored.
[2025-02-09 04:42:27 TP1] The following error message 'operation scheduled before its operands' can be ignored.
[2025-02-09 04:42:27 TP2] The following error message 'operation scheduled before its operands' can be ignored.
[2025-02-09 04:42:27 TP0] The following error message 'operation scheduled before its operands' can be ignored.
[2025-02-09 04:42:27 TP4] The following error message 'operation scheduled before its operands' can be ignored.
[2025-02-09 04:42:27 TP7] The following error message 'operation scheduled before its operands' can be ignored.
[2025-02-09 04:42:27 TP6] The following error message 'operation scheduled before its operands' can be ignored.
[2025-02-09 04:42:27 TP3] The following error message 'operation scheduled before its operands' can be ignored.
[2025-02-09 04:42:28 TP0] max_total_num_tokens=652106, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=4097, context_len=32000
[2025-02-09 04:42:28 TP1] max_total_num_tokens=652106, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=4097, context_len=32000
[2025-02-09 04:42:28 TP4] max_total_num_tokens=652106, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=4097, context_len=32000
[2025-02-09 04:42:28 TP3] max_total_num_tokens=652106, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=4097, context_len=32000
[2025-02-09 04:42:28 TP6] max_total_num_tokens=652106, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=4097, context_len=32000
[2025-02-09 04:42:28 TP5] max_total_num_tokens=652106, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=4097, context_len=32000
[2025-02-09 04:42:28 TP7] max_total_num_tokens=652106, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=4097, context_len=32000
[2025-02-09 04:42:28 TP2] max_total_num_tokens=652106, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=4097, context_len=32000
[2025-02-09 04:42:28] INFO:     Started server process [1]
[2025-02-09 04:42:28] INFO:     Waiting for application startup.
[2025-02-09 04:42:28] INFO:     Application startup complete.
[2025-02-09 04:42:28] INFO:     Uvicorn running on http://0.0.0.0:40000 (Press CTRL+C to quit)
[2025-02-09 04:42:29] INFO:     127.0.0.1:49112 - "GET /get_model_info HTTP/1.1" 200 OK
[2025-02-09 04:42:29 TP0] Prefill batch. #new-seq: 1, #new-token: 7, #cached-token: 0, cache hit rate: 0.00%, token usage: 0.00, #running-req: 0, #queue-req: 0
[2025-02-09 04:42:36 TP4] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=1536,K=7168,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:36 TP1] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=1536,K=7168,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:36 TP6] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=1536,K=7168,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:36 TP5] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=1536,K=7168,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:36 TP7] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=1536,K=7168,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:36 TP3] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=1536,K=7168,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:36 TP0] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=1536,K=7168,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:36 TP2] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=1536,K=7168,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:37 TP6] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=1536,K=1536,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:37 TP6] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=576,K=7168,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:37 TP4] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=1536,K=1536,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:37 TP4] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=576,K=7168,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:37 TP1] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=1536,K=1536,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:37 TP1] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=576,K=7168,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:37 TP5] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=1536,K=1536,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:37 TP5] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=576,K=7168,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:37 TP0] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=1536,K=1536,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:37 TP0] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=576,K=7168,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:37 TP7] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=1536,K=1536,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:37 TP7] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=576,K=7168,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:37 TP3] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=1536,K=1536,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:37 TP3] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=576,K=7168,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:37 TP2] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=1536,K=1536,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:37 TP2] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=576,K=7168,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:37 TP6] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=2048,K=512,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:37 TP4] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=2048,K=512,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:37 TP1] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=2048,K=512,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:37 TP5] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=2048,K=512,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:37 TP0] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=2048,K=512,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:37 TP3] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=2048,K=512,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:37 TP2] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=2048,K=512,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:37 TP7] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=2048,K=512,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:38 TP6] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=7168,K=1024,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:38 TP4] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=7168,K=1024,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:39 TP6] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=2304,K=7168,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:39 TP2] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=7168,K=1024,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:39 TP4] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=2304,K=7168,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:39 TP1] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=7168,K=1024,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:39 TP0] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=7168,K=1024,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:39 TP5] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=7168,K=1024,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:39 TP2] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=2304,K=7168,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:39 TP3] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=7168,K=1024,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:39 TP7] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=7168,K=1024,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:40 TP1] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=2304,K=7168,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:40 TP0] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=2304,K=7168,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:40 TP5] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=2304,K=7168,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:40 TP3] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=2304,K=7168,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:40 TP7] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=2304,K=7168,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:41 TP2] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=7168,K=1152,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:41 TP4] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=7168,K=1152,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:41 TP3] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=7168,K=1152,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:41 TP1] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=7168,K=1152,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:41 TP6] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=7168,K=1152,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:41 TP7] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=7168,K=1152,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:41 TP5] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=7168,K=1152,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:41 TP0] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=7168,K=1152,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:41 TP3] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=256,K=7168,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:41 TP2] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=256,K=7168,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:41 TP4] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=256,K=7168,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:41 TP6] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=256,K=7168,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:41 TP3] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=7168,K=128,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:41 TP4] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=7168,K=128,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:41 TP2] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=7168,K=128,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:41 TP6] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=7168,K=128,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:41 TP1] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=256,K=7168,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:41 TP5] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=256,K=7168,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:41 TP7] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=256,K=7168,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:41 TP0] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=256,K=7168,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:41 TP1] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=7168,K=128,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:41 TP5] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=7168,K=128,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:41 TP7] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=7168,K=128,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:41 TP0] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=7168,K=128,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for W8A8 Block FP8 kernel.
[2025-02-09 04:42:46 TP4] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/moe/fused_moe_triton/configs/E=256,N=128,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for MoE layer.
[2025-02-09 04:42:46 TP0] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/moe/fused_moe_triton/configs/E=256,N=128,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for MoE layer.
[2025-02-09 04:42:46 TP6] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/moe/fused_moe_triton/configs/E=256,N=128,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for MoE layer.
[2025-02-09 04:42:46 TP2] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/moe/fused_moe_triton/configs/E=256,N=128,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for MoE layer.
[2025-02-09 04:42:46 TP3] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/moe/fused_moe_triton/configs/E=256,N=128,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for MoE layer.
[2025-02-09 04:42:47 TP5] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/moe/fused_moe_triton/configs/E=256,N=128,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for MoE layer.
[2025-02-09 04:42:47 TP1] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/moe/fused_moe_triton/configs/E=256,N=128,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for MoE layer.
[2025-02-09 04:42:47 TP7] Using configuration from /sgl-workspace/sglang/python/sglang/srt/layers/moe/fused_moe_triton/configs/E=256,N=128,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json for MoE layer.
loc("/sgl-workspace/sglang/python/sglang/srt/layers/attention/triton_ops/decode_attention.py":310:16): error: operation scheduled before its operands
loc("/sgl-workspace/sglang/python/sglang/srt/layers/attention/triton_ops/decode_attention.py":310:16): error: operation scheduled before its operands
loc("/sgl-workspace/sglang/python/sglang/srt/layers/attention/triton_ops/decode_attention.py":310:16): error: operation scheduled before its operands
loc("/sgl-workspace/sglang/python/sglang/srt/layers/attention/triton_ops/decode_attention.py":310:16): error: operation scheduled before its operands
loc("/sgl-workspace/sglang/python/sglang/srt/layers/attention/triton_ops/decode_attention.py":310:16): error: operation scheduled before its operands
loc("/sgl-workspace/sglang/python/sglang/srt/layers/attention/triton_ops/decode_attention.py":310:16): error: operation scheduled before its operands
loc("/sgl-workspace/sglang/python/sglang/srt/layers/attention/triton_ops/decode_attention.py":310:16): error: operation scheduled before its operands
loc("/sgl-workspace/sglang/python/sglang/srt/layers/attention/triton_ops/decode_attention.py":310:16): error: operation scheduled before its operands
[2025-02-09 04:43:12] INFO:     127.0.0.1:49118 - "POST /generate HTTP/1.1" 200 OK
[2025-02-09 04:43:12] The server is fired up and ready to roll!

But when I chat, all the responses are garbled characters.

from openai import OpenAI

client = OpenAI(
    base_url="http://$IP:40000/v1",
    api_key="$SSY_API_KEY",
)

try:
    completion = client.chat.completions.create(
        model="deepseek-ai/DeepSeek-R1",
        messages=[{"role": "user", "content": "Hello"}],
        stream=True,
        temperature=1.0,
        max_tokens=128,
        top_p=0.95,
        presence_penalty=0.2,
        frequency_penalty=0.2,
        n=1,
    )

    response_text = ""

    for chunk in completion:
        if chunk.choices and chunk.choices[0].delta.content is not None:
            content = chunk.choices[0].delta.content
            print(content, end="", flush=True)
            response_text += content


except Exception as e:
    print(f"Request failed: {e}")

The output I got is as follows:

 Qu spacingability goodnow？ ?” prime We to balls was realized.com系 PDeep






<re办的







 中国



 入了与你
Download documentation 可见 time















Log… nerD

.





铁了这么多 行
F ву́과학ногда本单位 times your ** lost















Apple





上方 morewm sacrifice_v= всем она
消费者лсябы
ет.
 родиводрапр發生了



















 проблемноенаму.


 类型的



你 крат是要下
   карпа
 Дла!
за ст











外国 can. знать  руха%

The log is as follows:

[2025-02-09 04:51:36] INFO:     100.64.1.98:57410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2025-02-09 04:51:36 TP0] Prefill batch. #new-seq: 1, #new-token: 2, #cached-token: 2, cache hit rate: 20.00%, token usage: 0.00, #running-req: 0, #queue-req: 0
[2025-02-09 04:51:36 TP0] frequency_penalty, presence_penalty, and repetition_penalty are not supported when using the default overlap scheduler. They will be ignored. Please add `--disable-overlap` when launching the server if you need these features. The speed will be slower in that case.
[2025-02-09 04:51:36 TP2] frequency_penalty, presence_penalty, and repetition_penalty are not supported when using the default overlap scheduler. They will be ignored. Please add `--disable-overlap` when launching the server if you need these features. The speed will be slower in that case.
[2025-02-09 04:51:36 TP3] frequency_penalty, presence_penalty, and repetition_penalty are not supported when using the default overlap scheduler. They will be ignored. Please add `--disable-overlap` when launching the server if you need these features. The speed will be slower in that case.
[2025-02-09 04:51:36 TP7] frequency_penalty, presence_penalty, and repetition_penalty are not supported when using the default overlap scheduler. They will be ignored. Please add `--disable-overlap` when launching the server if you need these features. The speed will be slower in that case.
[2025-02-09 04:51:36 TP6] frequency_penalty, presence_penalty, and repetition_penalty are not supported when using the default overlap scheduler. They will be ignored. Please add `--disable-overlap` when launching the server if you need these features. The speed will be slower in that case.
[2025-02-09 04:51:36 TP4] frequency_penalty, presence_penalty, and repetition_penalty are not supported when using the default overlap scheduler. They will be ignored. Please add `--disable-overlap` when launching the server if you need these features. The speed will be slower in that case.
[2025-02-09 04:51:36 TP5] frequency_penalty, presence_penalty, and repetition_penalty are not supported when using the default overlap scheduler. They will be ignored. Please add `--disable-overlap` when launching the server if you need these features. The speed will be slower in that case.
[2025-02-09 04:51:36 TP1] frequency_penalty, presence_penalty, and repetition_penalty are not supported when using the default overlap scheduler. They will be ignored. Please add `--disable-overlap` when launching the server if you need these features. The speed will be slower in that case.
[2025-02-09 04:51:45 TP0] Decode batch. #running-req: 1, #token: 29, token usage: 0.00, gen throughput (token/s): 0.09, #queue-req: 0
[2025-02-09 04:51:53 TP0] Decode batch. #running-req: 1, #token: 69, token usage: 0.00, gen throughput (token/s): 4.71, #queue-req: 0
[2025-02-09 04:52:01 TP0] Decode batch. #running-req: 1, #token: 109, token usage: 0.00, gen throughput (token/s): 5.48, #queue-req: 0

zhyncs · 2025-02-09T13:21:55Z

zhyncs
Feb 9, 2025
Maintainer

Please join the slack channel https://slack.sglang.ai

1 reply

echozyr2001 Feb 9, 2025
Author

How can I join in?

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

28h100 nodes run DeepSeek-R1 671B FP8，get garbled characters in response. #3426

{{title}}

Replies: 1 comment 1 reply

{{title}}

{{title}}

Select a reply

2*8*h100 nodes run DeepSeek-R1 671B FP8，get garbled characters in response. #3426

echozyr2001 Feb 9, 2025

node1

node2

Replies: 1 comment · 1 reply

zhyncs Feb 9, 2025 Maintainer

echozyr2001 Feb 9, 2025 Author

28h100 nodes run DeepSeek-R1 671B FP8，get garbled characters in response. #3426

echozyr2001
Feb 9, 2025

Replies: 1 comment 1 reply

zhyncs
Feb 9, 2025
Maintainer

echozyr2001 Feb 9, 2025
Author