Parameters

[morpheus2448]

I tried the latest release with a ChatML model and it spits out junk.

What happened to the model parameters?

Sadly, it's unusable without them.

[danemadsen]

I tried the latest release with a ChatML model and it spits out junk.

What happened to the model parameters?

Sadly, it's unusable without them.

Switched it using a different way of binding to llama.cpp. Its a bit janky at the moment but hopefully it will improve over time. I couldn't use the old method of binding any longer because it was too unstable and it was slowing down development but if you want to keep using that you can use 1.1.8 until the new method is more stable.

[morpheus2448]

Gotcha.

I'm subscribed to the updates and have been checking in pretty much daily so I've seen all the work you've put in!

I thought the release indicated things were stable again, so I updated.

Unsure of other users' preference but would recommend implementing temp and min_p samplers first as they're the only ones I use in llama.cpp now, it's simple and the results are great. You can implement the samplers in either order, but generally it seems min_p then temperature works best. Adding repetition penalty may be necessary for smaller models, however this has been described as a 'hacky band-aid'.

For more info regarding min_p I suggest reading the author's technical Reddit post here.

Also, I'll be up for addressing some more of the simpler issues for this project too (once the updates slow down a bit).

Cheers 🍻

[danemadsen]

Nah not stable more just working, and it it is working it just spits out rubbish. A rubbish output is still an output....

Yes 1.1.8 is more stable and does perform much better for local gen, but it also lacks a lot of features that are present in 1.1.9.

Sampling params are being worked on, but they are being worked on over here now which is the package I am working on and now using for local gen. Which will be where I will be redirecting any technical issues with local gen from now on.