The 'stop' argument causes 'pyo3_runtime.PanicException' in some cases. #1131
Comments
|
Hi @Six6stRINgs, thanks for reporting this...definitely odd! Will look into reproducing this week |
|
@Harsha-Nori Hello, thanks for your reply. I spent a few days trying to make the code a little bit easier to reprodue this bug. import guidance
from guidance import gen
from pydantic import BaseModel
class bm(BaseModel):
a: str
text: str
ccc: str
def get_attr_name_list(self) -> list[str]:
return [attr for attr, _ in self.model_fields.items()]
class bm2(BaseModel):
ttt: str
attrr: str
bo: str
def get_attr_name_list(self) -> list[str]:
return [attr for attr, _ in self.model_fields.items()]
if __name__ == "__main__":
model_path = "/data/weights/qwen2_5-1.5b-instruct/"
device_map = {"": 2}
m = bm(
a="5",
text="world",
ccc="hello world",
)
m2 = bm2(ttt="123", attrr="2", bo="True")
model = guidance.models.Transformers(model_path, device_map=device_map)
bm_list1 = ["a", "text", "ccc"] # crash at 2nd str
bm_list2 = ["222", "333", "444"] # normal
bm_list3 = ["a", "666", "ccc"] # normal
bm_list4 = ["text", "ccc", "a"] # crash at 1st str
bm2_list1 = ["ttt", "attrr", "bo"] # normal
bm2_list2 = ["t1", "a2", "b3"] # normal
bm2_list3 = ["ttt", "attrr", "ta"] # normal
bm_keys: list[str] = m.get_attr_name_list() # crash at 2nd str, same as bm_list1
bm2_keys: list[str] = m2.get_attr_name_list() # normal
# the list/keys from above
for key in bm_keys:
print(f"key: {key}")
res = (
model
+ f"Hello, {key}: "
+ gen(
name="res",
stop=["'", '"', ";", ":", ",", ".", "\n"],
max_tokens=15,
)
)
print(f"LLM res: {res}")I run this code on my linux server. The key part is the the list used in
|
|
Thank you for the detailed debugging, it really helps!
@mmoskal @hudson-ai any ideas?
Get Outlook for iOS<https://aka.ms/o0ukef>
…________________________________
From: SixString ***@***.***>
Sent: Wednesday, March 5, 2025 1:34:42 AM
To: guidance-ai/guidance ***@***.***>
Cc: Mention ***@***.***>; Comment ***@***.***>; Subscribed ***@***.***>
Subject: Re: [guidance-ai/guidance] The 'stop' argument causes 'pyo3_runtime.PanicException' in some cases. (Issue #1131)
@Harsha-Nori<https://github.com/Harsha-Nori> Hello, thanks for your reply. I spent a few days trying to make the code a little bit easier to reprodue this bug.
There is the code:
import guidance
from guidance import gen
from pydantic import BaseModel
class bm(BaseModel):
a: str
text: str
ccc: str
def get_attr_name_list(self) -> list[str]:
return [attr for attr, _ in self.model_fields.items()]
class bm2(BaseModel):
ttt: str
attrr: str
bo: str
def get_attr_name_list(self) -> list[str]:
return [attr for attr, _ in self.model_fields.items()]
if __name__ == "__main__":
model_path = "/data/weights/qwen2_5-1.5b-instruct/"
device_map = {"": 2}
m = bm(
a="5",
text="world",
ccc="hello world",
)
m2 = bm2(ttt="123", attrr="2", bo="True")
model = guidance.models.Transformers(model_path, device_map=device_map)
bm_list1 = ["a", "text", "ccc"] # crash at 2nd str
bm_list2 = ["222", "333", "444"] # normal
bm_list3 = ["a", "666", "ccc"] # normal
bm_list4 = ["text", "ccc", "a"] # crash at 1st str
bm2_list1 = ["ttt", "attrr", "bo"] # normal
bm2_list2 = ["t1", "a2", "b3"] # normal
bm2_list3 = ["ttt", "attrr", "ta"] # normal
bm_keys: list[str] = m.get_attr_name_list() # crash at 2nd str, same as bm_list1
bm2_keys: list[str] = m2.get_attr_name_list() # normal
# the list/keys from above
for key in bm_keys:
print(f"key: {key}")
res = (
model
+ f"Hello, {key}: "
+ gen(
name="res",
stop=["'", '"', ";", ":", ",", ".", "\n"],
max_tokens=15,
)
)
print(f"LLM res: {res}")
I run this code on my linux server.
The key part is the the list used in for loop
It seems that the str in list may crash the program. If the string is exactly the attribute name of the Basemodel, same bug will be reported.
From the code, bm_list1, bm_list4 and bm_keys will crash the program when model begins to gen. But other lists work well.
thread '<unnamed>' panicked at parser/src/earley/parser.rs:2043:9:
assertion failed: self.state.byte_to_token_idx.len() >= n_bytes
......
However, the odd part is that bm2_list1 and bm2_keys of what contains in the same attribute string of the class bm2 it works without error.
If I remove the stop arg, all lists run well.
Have no idea about this bug T_T.
—
Reply to this email directly, view it on GitHub<#1131 (comment)> or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABIOOZYHLUI6RPRCZTICTWT2S3ALHBFKMF2HI4TJMJ2XIZLTSOBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTAVFOZQWY5LFUVUXG43VMWSG4YLNMWVXI2DSMVQWIX3UPFYGLLDTOVRGUZLDORPXI6LQMWWES43TOVSUG33NNVSW45FGORXXA2LDOOJIFJDUPFYGLKTSMVYG643JORXXE6NFOZQWY5LFVE2TMNBUGQZTENJVQKSHI6LQMWSWS43TOVS2K5TBNR2WLKRSHA3DQMZUG42TKMNHORZGSZ3HMVZKMY3SMVQXIZI>.
You are receiving this email because you were mentioned.
Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
|
|
@mmoskal I have a hypothesis -- we crash if Minimal repro: If we token heal and allow the first token to be |
|
Yeah, this crashes: #[test]
fn test_ll_stop_heal() {
// https://github.com/guidance-ai/guidance/issues/1131
check_lark_grammar_prompt(
r#"
start: gen
gen[stop=/"/]: /.*/
"#,
"Hello, text: ",
&["Hello‧,‧ text‧:", " \""],
);
} |
|
@Six6stRINgs thanks for opeing the issue and finding this bug! While we work on a fix, here is a workaround: group the There is a subtle difference between and Doing the second version instead here should get around the crash. |
|
keeping open until llguidance is updated in guidance |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The bug
Hello, Thanks for your great work. But recently I have occured some wired bugs and don't how to fix it.
I want to gen a json format string using
genwith some regular args.Using the
stopargument ingencan cause bugs in certain cases.Code
When running
gen_decision, error will be reported. If I remove thestopargument, the issue does not occur. However, the output generated by the LLM is not satisfactory.But when running some simple codes, it will be ok:
Response
To Reproduce
Loader : loaded by
models.TransformersModel:
Qwen2.5-1.5b-instruct,Qwen1.5-7bExecuting the aforementioned code
System info (please complete the following information):
guidance.__version__): 0.2.0The text was updated successfully, but these errors were encountered: