GBNF grammar extension for text-generation-webui

Implementation of GBNF grammar for text-generation-webui.

Allows to force output generated by LLM to conform with expected format, making it easier to parse. For example, with this extension you can get any LLM to generate proper JSON or YAML.

This is done by creating state machine which, every time when new token is to be generated, decides which tokens are allowed to conform with defined grammar and ban (set probability to -inf) all other tokens.

To better explain the issue, here is an example of prompt and output generated without extension demonstrating the problem and here is same prompt and output generated with grammar

Differences from GBNF in lamma.cpp

This should be fully-conforming implementation, but allows for two additional features:

• .* regexp can be used to basically turn off extension at some point and let LLM generate rest of the string normally
• Terminals can be defined with both "" quotes and '' quotes. This was done by mistake and then kept for my convenience.