feat/1001: sglang integration #1122

Jayon02 · 2025-02-27T06:19:25Z

This PR closes #1001.

In this PR, I integrate sglang into Distilabel, allowing users to use sglang as the backend for dataset generation.

Tasks

Add sglang as a potential backend for data generation. See here.
Add sglang as a potential backend for embedding generation. See here.
Add unit tests for features above.
Update documentation.

Examples

Here are some examples for you to use sglang in distilabel. More detailed documentation will be added.

Use sglang backend to generate text directly.

from distilabel.models.llms import SGLang
if __name__ == "__main__":
    llm = SGLang(
        model="prometheus-eval/prometheus-7b-v2.0",
        chat_template="[INST] {{ messages[0]['content']}} [/INST]"
    )

    llm.load()
    # Call the model
    output = llm.generate_outputs(inputs=[[{"role": "user", "content": "Hello world!"}]])

Use sglang backend in pipeline to generate dataset.

from distilabel.pipeline import Pipeline
from distilabel.llms import SGLang
from distilabel.steps import LoadDataFromDicts
from distilabel.steps.tasks import TextGeneration

with Pipeline(name="text-generation-pipeline") as pipeline:
    load_dataset = LoadDataFromDicts(
        name="load_dataset",
        data=[{"instruction": "Write a short story about a dragon that saves a princess from a tower."}],
    )

    text_generation = TextGeneration(
        name="text_generation",
        llm=SGLang(model="athirdpath/Llama-3-15b-Instruct"),
    )

    load_dataset >> text_generation

if __name__ == "__main__":
    pipeline.run(
        parameters={
            text_generation.name: {"llm": {"generation_kwargs": {"temperature": 0.3}}},
        },
        use_cache=False,
    )

Please let me know if there is anything that could be improved. Thanks to all SGLang team members for their help.

commit 6c9059f Author: Jayon02 <[email protected]> Date: Thu Feb 27 02:50:07 2025 +0000 add detokenize commit 159f1ef Author: Jayon02 <[email protected]> Date: Tue Feb 25 11:34:05 2025 +0000 support sglang embedding commit f2b2064 Author: Jayon02 <[email protected]> Date: Tue Feb 25 08:06:05 2025 +0000 fix test bugs commit ba3a5a7 Author: Jayon02 <[email protected]> Date: Mon Feb 24 16:55:53 2025 +0000 fix bugs in multiple generations and add test commit f893c6d Author: Jayon02 <[email protected]> Date: Sat Feb 22 17:13:12 2025 +0000 modify function annotation commit ddde5cb Author: Jayon02 <[email protected]> Date: Sat Feb 22 12:06:11 2025 +0000 support structure output and SGLang openai client commit 5ecedc4 Author: Jayon02 <[email protected]> Date: Sat Feb 22 06:22:42 2025 +0000 support sglang

zhaochenyang20 · 2025-02-27T06:28:56Z

src/distilabel/models/embeddings/sglang.py

+        CudaDevicePlacementMixin.load(self)
+
+        try:
+            from sglang import Engine as _SGLang


why should use this by the name "_SGLang". COuld we directly use Engine?

I align with the format in vllm. Should I reformat what I commit?

you can use from sglang import Engine here, it's fine

zhaochenyang20 · 2025-02-27T06:30:36Z

src/distilabel/models/embeddings/sglang.py

+
+    def unload(self) -> None:
+        """Unloads the `SGLang` model."""
+        self._cleanup_sglang_model()


where is the definition of _cleanup_sglang_model()? Also, do not use _ as this could be a public funciton.

codspeed-hq · 2025-02-27T07:29:53Z

CodSpeed Performance Report

Merging #1122 will improve performances by ×2.1

_{Comparing Jayon02:main (503b3d3) with develop (2d4ec9a)}

Summary

⚡ 1 improvements

Benchmarks breakdown

	Benchmark	`BASE`	`HEAD`	Change
⚡	`test_cache_time`	1,185.5 ms	553.1 ms	×2.1

davidberenstein1957 · 2025-02-27T08:46:43Z

Awesome PR, I've asked @plaguss and @gabrielmbmb to review it.

plaguss

Awesome job! some small comments, and a couple of things to tackle

plaguss · 2025-02-27T08:22:58Z

src/distilabel/models/embeddings/sglang.py

+        CudaDevicePlacementMixin.load(self)
+
+        try:
+            from sglang import Engine as _SGLang


you can use from sglang import Engine here, it's fine

src/distilabel/models/llms/sglang.py

plaguss · 2025-02-27T09:11:05Z

Could you add the required dependencies so that we can run:

uv pip install distilabel[sglang]

and have it ready? I run the command from the guide and it works:

uv pip install "sglang[all]>=0.4.3.post2" --find-links https://flashinfer.ai/whl/cu124/torch2.5/flashinfer-python

but should be reflected in the pyproject.toml

plaguss · 2025-02-27T09:11:56Z

Also, the example from the structured outputs is failing:

from distilabel.models.llms import SGLang
from pydantic import BaseModel
if __name__ == "__main__":
    class User(BaseModel):
        name: str
        last_name: str
        id: int
    llm = SGLang(
        model="Qwen/Qwen2.5-Coder-3B-Instruct",
        structured_output={"format": "json", "schema": User},
    )
    llm.load()
    # Call the model
    output = llm.generate_outputs(inputs=[[{"role": "user", "content": "Create a user profile for the following marathon"}]])

It throws the following trace:

[2025-02-27 09:04:17 TP0] Scheduler hit an exception: Traceback (most recent call last):
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1825, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 496, in event_loop_overlap
    batch = self.get_next_batch_to_run()
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 877, in get_next_batch_to_run
    new_batch = self.get_new_batch_prefill()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 898, in get_new_batch_prefill
    self.move_ready_grammar_requests()
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1594, in move_ready_grammar_requests
    req.grammar = req.grammar.result(timeout=0.05)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/admin/home/agustin_piqueres/.pyenv/versions/3.11.9/lib/python3.11/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/admin/home/agustin_piqueres/.pyenv/versions/3.11.9/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/admin/home/agustin_piqueres/.pyenv/versions/3.11.9/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/sglang/srt/constrained/base_grammar_backend.py", line 53, in init_value
    entry.value = self.init_value_impl(key)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/sglang/srt/constrained/outlines_backend.py", line 173, in init_value_impl
    guide = RegexGuide.from_regex(regex, self.outlines_tokenizer)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/outlines/fsm/guide.py", line 92, in from_regex
    return super().from_regex(
           ^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/outlines_core/fsm/guide.py", line 212, in from_regex
    ) = _create_states_mapping(
        ^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/outlines/fsm/guide.py", line 76, in cached_create_states_mapping
    return uncached_create_states_mapping(regex_string, tokenizer, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/outlines_core/fsm/guide.py", line 141, in create_states_mapping
    return create_states_mapping_from_fsm(regex_fsm, tokenizer, frozen_tokens)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/outlines_core/fsm/guide.py", line 178, in create_states_mapping_from_fsm
    states_to_token_maps, empty_token_ids = create_fsm_index_tokenizer(
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/outlines_core/fsm/regex.py", line 473, in create_fsm_index_tokenizer
    tokens_to_token_ids, empty_token_ids = reduced_vocabulary(tokenizer)
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/outlines/models/transformers.py", line 119, in __hash__
    return hash(Hasher.hash(self.tokenizer))
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/datasets/fingerprint.py", line 188, in hash
    return cls.hash_bytes(dumps(value))
                          ^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/datasets/utils/_dill.py", line 109, in dumps
    dump(obj, file)
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/datasets/utils/_dill.py", line 103, in dump
    Pickler(file, recurse=True).dump(obj)
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/dill/_dill.py", line 428, in dump
    StockPickler.dump(self, obj)
  File "/admin/home/agustin_piqueres/.pyenv/versions/3.11.9/lib/python3.11/pickle.py", line 487, in dump
    self.save(obj)
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/datasets/utils/_dill.py", line 70, in save
    dill.Pickler.save(self, obj, save_persistent_id=save_persistent_id)
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/dill/_dill.py", line 422, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/admin/home/agustin_piqueres/.pyenv/versions/3.11.9/lib/python3.11/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
    ^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/datasets/utils/_dill.py", line 209, in _save_transformersPreTrainedTokenizerBase
    log(pickler, f"Tok: {obj}")
    ^^^
NameError: name 'log' is not defined

[2025-02-27 09:04:17] Received sigquit from a child proces. It usually means the child failed.
Killed

plaguss · 2025-02-27T10:45:07Z

for the integration test failing, I already fixed in the the develop branch, if you merge it, that should do it. And for the docs they will appear in the components gallery section, it's okey, unless you want to add some additional example in the /examples folder or something 😄

Jayon02 · 2025-02-27T12:13:50Z

Also, the example from the structured outputs is failing:

from distilabel.models.llms import SGLang
from pydantic import BaseModel
if __name__ == "__main__":
    class User(BaseModel):
        name: str
        last_name: str
        id: int
    llm = SGLang(
        model="Qwen/Qwen2.5-Coder-3B-Instruct",
        structured_output={"format": "json", "schema": User},
    )
    llm.load()
    # Call the model
    output = llm.generate_outputs(inputs=[[{"role": "user", "content": "Create a user profile for the following marathon"}]])

It throws the following trace:

[2025-02-27 09:04:17 TP0] Scheduler hit an exception: Traceback (most recent call last):
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1825, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 496, in event_loop_overlap
    batch = self.get_next_batch_to_run()
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 877, in get_next_batch_to_run
    new_batch = self.get_new_batch_prefill()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 898, in get_new_batch_prefill
    self.move_ready_grammar_requests()
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1594, in move_ready_grammar_requests
    req.grammar = req.grammar.result(timeout=0.05)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/admin/home/agustin_piqueres/.pyenv/versions/3.11.9/lib/python3.11/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/admin/home/agustin_piqueres/.pyenv/versions/3.11.9/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/admin/home/agustin_piqueres/.pyenv/versions/3.11.9/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/sglang/srt/constrained/base_grammar_backend.py", line 53, in init_value
    entry.value = self.init_value_impl(key)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/sglang/srt/constrained/outlines_backend.py", line 173, in init_value_impl
    guide = RegexGuide.from_regex(regex, self.outlines_tokenizer)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/outlines/fsm/guide.py", line 92, in from_regex
    return super().from_regex(
           ^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/outlines_core/fsm/guide.py", line 212, in from_regex
    ) = _create_states_mapping(
        ^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/outlines/fsm/guide.py", line 76, in cached_create_states_mapping
    return uncached_create_states_mapping(regex_string, tokenizer, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/outlines_core/fsm/guide.py", line 141, in create_states_mapping
    return create_states_mapping_from_fsm(regex_fsm, tokenizer, frozen_tokens)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/outlines_core/fsm/guide.py", line 178, in create_states_mapping_from_fsm
    states_to_token_maps, empty_token_ids = create_fsm_index_tokenizer(
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/outlines_core/fsm/regex.py", line 473, in create_fsm_index_tokenizer
    tokens_to_token_ids, empty_token_ids = reduced_vocabulary(tokenizer)
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/outlines/models/transformers.py", line 119, in __hash__
    return hash(Hasher.hash(self.tokenizer))
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/datasets/fingerprint.py", line 188, in hash
    return cls.hash_bytes(dumps(value))
                          ^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/datasets/utils/_dill.py", line 109, in dumps
    dump(obj, file)
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/datasets/utils/_dill.py", line 103, in dump
    Pickler(file, recurse=True).dump(obj)
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/dill/_dill.py", line 428, in dump
    StockPickler.dump(self, obj)
  File "/admin/home/agustin_piqueres/.pyenv/versions/3.11.9/lib/python3.11/pickle.py", line 487, in dump
    self.save(obj)
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/datasets/utils/_dill.py", line 70, in save
    dill.Pickler.save(self, obj, save_persistent_id=save_persistent_id)
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/dill/_dill.py", line 422, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/admin/home/agustin_piqueres/.pyenv/versions/3.11.9/lib/python3.11/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
    ^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/datasets/utils/_dill.py", line 209, in _save_transformersPreTrainedTokenizerBase
    log(pickler, f"Tok: {obj}")
    ^^^
NameError: name 'log' is not defined

[2025-02-27 09:04:17] Received sigquit from a child proces. It usually means the child failed.
Killed

This issue seems quite strange and I don't find bugs about sglang in this context. In my environment, this code can work correctly. Could you use docker to run the repo? Maybe it can give more useful information and I will fix it.

plaguss · 2025-02-27T12:16:49Z

Also, the example from the structured outputs is failing:

from distilabel.models.llms import SGLang
from pydantic import BaseModel
if __name__ == "__main__":
    class User(BaseModel):
        name: str
        last_name: str
        id: int
    llm = SGLang(
        model="Qwen/Qwen2.5-Coder-3B-Instruct",
        structured_output={"format": "json", "schema": User},
    )
    llm.load()
    # Call the model
    output = llm.generate_outputs(inputs=[[{"role": "user", "content": "Create a user profile for the following marathon"}]])

It throws the following trace:

[2025-02-27 09:04:17 TP0] Scheduler hit an exception: Traceback (most recent call last):
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1825, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 496, in event_loop_overlap
    batch = self.get_next_batch_to_run()
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 877, in get_next_batch_to_run
    new_batch = self.get_new_batch_prefill()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 898, in get_new_batch_prefill
    self.move_ready_grammar_requests()
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1594, in move_ready_grammar_requests
    req.grammar = req.grammar.result(timeout=0.05)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/admin/home/agustin_piqueres/.pyenv/versions/3.11.9/lib/python3.11/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/admin/home/agustin_piqueres/.pyenv/versions/3.11.9/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/admin/home/agustin_piqueres/.pyenv/versions/3.11.9/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/sglang/srt/constrained/base_grammar_backend.py", line 53, in init_value
    entry.value = self.init_value_impl(key)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/sglang/srt/constrained/outlines_backend.py", line 173, in init_value_impl
    guide = RegexGuide.from_regex(regex, self.outlines_tokenizer)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/outlines/fsm/guide.py", line 92, in from_regex
    return super().from_regex(
           ^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/outlines_core/fsm/guide.py", line 212, in from_regex
    ) = _create_states_mapping(
        ^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/outlines/fsm/guide.py", line 76, in cached_create_states_mapping
    return uncached_create_states_mapping(regex_string, tokenizer, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/outlines_core/fsm/guide.py", line 141, in create_states_mapping
    return create_states_mapping_from_fsm(regex_fsm, tokenizer, frozen_tokens)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/outlines_core/fsm/guide.py", line 178, in create_states_mapping_from_fsm
    states_to_token_maps, empty_token_ids = create_fsm_index_tokenizer(
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/outlines_core/fsm/regex.py", line 473, in create_fsm_index_tokenizer
    tokens_to_token_ids, empty_token_ids = reduced_vocabulary(tokenizer)
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/outlines/models/transformers.py", line 119, in __hash__
    return hash(Hasher.hash(self.tokenizer))
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/datasets/fingerprint.py", line 188, in hash
    return cls.hash_bytes(dumps(value))
                          ^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/datasets/utils/_dill.py", line 109, in dumps
    dump(obj, file)
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/datasets/utils/_dill.py", line 103, in dump
    Pickler(file, recurse=True).dump(obj)
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/dill/_dill.py", line 428, in dump
    StockPickler.dump(self, obj)
  File "/admin/home/agustin_piqueres/.pyenv/versions/3.11.9/lib/python3.11/pickle.py", line 487, in dump
    self.save(obj)
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/datasets/utils/_dill.py", line 70, in save
    dill.Pickler.save(self, obj, save_persistent_id=save_persistent_id)
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/dill/_dill.py", line 422, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/admin/home/agustin_piqueres/.pyenv/versions/3.11.9/lib/python3.11/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
    ^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/datasets/utils/_dill.py", line 209, in _save_transformersPreTrainedTokenizerBase
    log(pickler, f"Tok: {obj}")
    ^^^
NameError: name 'log' is not defined

[2025-02-27 09:04:17] Received sigquit from a child proces. It usually means the child failed.
Killed

This issue seems quite strange and I don't find bugs about sglang in this context. In my environment, this code can work correctly. Could you use docker to run the repo? Maybe it can give more useful information and I will fix it.

Have you run it? if that's the case, it can be just a case of having some of my dependencies outdated, will take a look

Jayon02 · 2025-02-27T12:17:42Z

for the integration test failing, I already fixed in the the develop branch, if you merge it, that should do it. And for the docs they will appear in the components gallery section, it's okey, unless you want to add some additional example in the /examples folder or something 😄

Ok！I will merge develop branch into my main branch. I also add dependency in pyproject.toml. Due to version compatibility, only transformers==4.48.3 supports the code to work correctly. So, I have locked this version.

plaguss

just changing the version in the pyproject and should be ready to go

pyproject.toml

Jayon02 · 2025-02-27T12:23:56Z

Also, the example from the structured outputs is failing:

from distilabel.models.llms import SGLang
from pydantic import BaseModel
if __name__ == "__main__":
    class User(BaseModel):
        name: str
        last_name: str
        id: int
    llm = SGLang(
        model="Qwen/Qwen2.5-Coder-3B-Instruct",
        structured_output={"format": "json", "schema": User},
    )
    llm.load()
    # Call the model
    output = llm.generate_outputs(inputs=[[{"role": "user", "content": "Create a user profile for the following marathon"}]])

It throws the following trace:

[2025-02-27 09:04:17 TP0] Scheduler hit an exception: Traceback (most recent call last):
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1825, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 496, in event_loop_overlap
    batch = self.get_next_batch_to_run()
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 877, in get_next_batch_to_run
    new_batch = self.get_new_batch_prefill()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 898, in get_new_batch_prefill
    self.move_ready_grammar_requests()
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1594, in move_ready_grammar_requests
    req.grammar = req.grammar.result(timeout=0.05)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/admin/home/agustin_piqueres/.pyenv/versions/3.11.9/lib/python3.11/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/admin/home/agustin_piqueres/.pyenv/versions/3.11.9/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/admin/home/agustin_piqueres/.pyenv/versions/3.11.9/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/sglang/srt/constrained/base_grammar_backend.py", line 53, in init_value
    entry.value = self.init_value_impl(key)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/sglang/srt/constrained/outlines_backend.py", line 173, in init_value_impl
    guide = RegexGuide.from_regex(regex, self.outlines_tokenizer)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/outlines/fsm/guide.py", line 92, in from_regex
    return super().from_regex(
           ^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/outlines_core/fsm/guide.py", line 212, in from_regex
    ) = _create_states_mapping(
        ^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/outlines/fsm/guide.py", line 76, in cached_create_states_mapping
    return uncached_create_states_mapping(regex_string, tokenizer, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/outlines_core/fsm/guide.py", line 141, in create_states_mapping
    return create_states_mapping_from_fsm(regex_fsm, tokenizer, frozen_tokens)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/outlines_core/fsm/guide.py", line 178, in create_states_mapping_from_fsm
    states_to_token_maps, empty_token_ids = create_fsm_index_tokenizer(
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/outlines_core/fsm/regex.py", line 473, in create_fsm_index_tokenizer
    tokens_to_token_ids, empty_token_ids = reduced_vocabulary(tokenizer)
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/outlines/models/transformers.py", line 119, in __hash__
    return hash(Hasher.hash(self.tokenizer))
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/datasets/fingerprint.py", line 188, in hash
    return cls.hash_bytes(dumps(value))
                          ^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/datasets/utils/_dill.py", line 109, in dumps
    dump(obj, file)
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/datasets/utils/_dill.py", line 103, in dump
    Pickler(file, recurse=True).dump(obj)
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/dill/_dill.py", line 428, in dump
    StockPickler.dump(self, obj)
  File "/admin/home/agustin_piqueres/.pyenv/versions/3.11.9/lib/python3.11/pickle.py", line 487, in dump
    self.save(obj)
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/datasets/utils/_dill.py", line 70, in save
    dill.Pickler.save(self, obj, save_persistent_id=save_persistent_id)
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/dill/_dill.py", line 422, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/admin/home/agustin_piqueres/.pyenv/versions/3.11.9/lib/python3.11/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
    ^^^^^^^^^^^^
  File "/fsx/agustin_piqueres/test-distilabel/.venv/lib/python3.11/site-packages/datasets/utils/_dill.py", line 209, in _save_transformersPreTrainedTokenizerBase
    log(pickler, f"Tok: {obj}")
    ^^^
NameError: name 'log' is not defined

[2025-02-27 09:04:17] Received sigquit from a child proces. It usually means the child failed.
Killed

This issue seems quite strange and I don't find bugs about sglang in this context. In my environment, this code can work correctly. Could you use docker to run the repo? Maybe it can give more useful information and I will fix it.

Have you run it? if that's the case, it can be just a case of having some of my dependencies outdated, will take a look

Yes. I have just run it correctly.

Jayon02 and others added 9 commits February 22, 2025 06:22

support sglang

5ecedc4

support structure output and SGLang openai client

ddde5cb

modify function annotation

f893c6d

fix bugs in multiple generations and add test

ba3a5a7

fix test bugs

f2b2064

support sglang embedding

159f1ef

add detokenize

6c9059f

Update sglang.py

5a15126

zhaochenyang20 suggested changes Feb 27, 2025

View reviewed changes

Jayon02 added 3 commits February 27, 2025 06:44

fix bug in embedding

e6e9be7

update sglang.py

c3740a9

Merge main into feature_branch

83984d3

plaguss changed the base branch from main to develop February 27, 2025 07:23

fix bug in test

503b3d3

davidberenstein1957 requested a review from plaguss February 27, 2025 08:46

davidberenstein1957 requested a review from gabrielmbmb February 27, 2025 08:46

plaguss reviewed Feb 27, 2025

View reviewed changes

modify toml and reformat

fd946d7

plaguss approved these changes Feb 27, 2025

View reviewed changes

pyproject.toml Show resolved Hide resolved

plaguss merged commit ab612a6 into argilla-io:develop Feb 27, 2025
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat/1001: sglang integration #1122

feat/1001: sglang integration #1122

Jayon02 commented Feb 27, 2025

zhaochenyang20 Feb 27, 2025

Jayon02 Feb 27, 2025

plaguss Feb 27, 2025

zhaochenyang20 Feb 27, 2025

Jayon02 Feb 27, 2025

codspeed-hq bot commented Feb 27, 2025 •

edited

Loading

davidberenstein1957 commented Feb 27, 2025

plaguss left a comment

plaguss Feb 27, 2025

plaguss commented Feb 27, 2025

plaguss commented Feb 27, 2025

plaguss commented Feb 27, 2025

Jayon02 commented Feb 27, 2025

plaguss commented Feb 27, 2025

Jayon02 commented Feb 27, 2025

plaguss left a comment

Jayon02 commented Feb 27, 2025

feat/1001: sglang integration #1122

feat/1001: sglang integration #1122

Conversation

Jayon02 commented Feb 27, 2025

Tasks

Examples

zhaochenyang20 Feb 27, 2025

Choose a reason for hiding this comment

Jayon02 Feb 27, 2025

Choose a reason for hiding this comment

plaguss Feb 27, 2025

Choose a reason for hiding this comment

zhaochenyang20 Feb 27, 2025

Choose a reason for hiding this comment

Jayon02 Feb 27, 2025

Choose a reason for hiding this comment

codspeed-hq bot commented Feb 27, 2025 • edited Loading

CodSpeed Performance Report

Merging #1122 will improve performances by ×2.1

Summary

Benchmarks breakdown

davidberenstein1957 commented Feb 27, 2025

plaguss left a comment

Choose a reason for hiding this comment

plaguss Feb 27, 2025

Choose a reason for hiding this comment

plaguss commented Feb 27, 2025

plaguss commented Feb 27, 2025

plaguss commented Feb 27, 2025

Jayon02 commented Feb 27, 2025

plaguss commented Feb 27, 2025

Jayon02 commented Feb 27, 2025

plaguss left a comment

Choose a reason for hiding this comment

Jayon02 commented Feb 27, 2025

codspeed-hq bot commented Feb 27, 2025 •

edited

Loading