Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update README.md #111

Merged
merged 2 commits into from
Mar 3, 2025
Merged

Update README.md #111

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ff84462
Update README.md
urroxyz Mar 2, 2025
070818f
Update README.md
urroxyz Mar 2, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
216 changes: 91 additions & 125 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,67 +1,71 @@
# 🤗 Transformers CFG

![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)
[![License](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)

## 💭 Latest News

- **[Gemma-2 Support](https://github.com/epfl-dlab/transformers-CFG/pull/75)** — Thanks to @fillassuncao (2024-08-16)
- **[DeepSeek Support](https://github.com/epfl-dlab/transformers-CFG/pull/73)** (2024-07-24)
- **LLAMA-3 Support** (2024-07-08)
- **[JSON Schema as Constraint Support](examples%2Fgrammars%2Fcustom_json_grammars%2FREADME.md)** (2024-05-13)
- **[Token Masking Optimization](#efficiency)** (2024-04-25)
- **[Phi Support](https://github.com/epfl-dlab/transformers-CFG/issues/34)** (2024-04-16)
- **[Online Demo with JSON Grammar](http://saibo-creator.xyz:7860/) at HF Space** (2024-04-10)
- **Unicode (Multilingual) Grammar Support** (2024-02-29)
- **Integration with Text-Generation-WebUI** (2023-12-17)

We are thrilled to announce that `transformers-cfg` has been integrated into the [Text-Generation-WebUI](https://github.com/oobabooga/text-generation-webui) project, enabling users to utilize our CFG capabilities within this popular web interface for text generation. For more details, see the [relevant pull request](https://github.com/oobabooga/text-generation-webui/pull/4953).
## 💭 Release news
### Latest release
#### **[v0.2.7 Latest](https://github.com/epfl-dlab/transformers-CFG/releases/tag/v0.2.7)** (2025-03-02)

#### **Features**
- **(CLI)** Types and MLX support (#93)
- **(Regex)** Negation, wildcard, and repetition bracket operators (#94, #95, #96, #104)
- **(Models)** Qwen2 and Qwen2.5 (#97)
- **(Logits)** Resuable `GrammarConstrainedLogitsProcessor` across generations for efficiency (#100)
- **(Backend)** Pytest for testing (#109)
- **(CI/CD)** GitHub Actions workflow for automation (#110)

#### **Bug fixes**
- Avoid computing full masks and optimized type additions (#101)
- Refactored grammar encoding to improve structure (#99)
- EOS token now correctly masks (#108)
- Multiple bugs removed and aesthetics improved (#107)

### Recent releases
- **[Gemma-2](https://github.com/epfl-dlab/transformers-CFG/pull/75)** — @fillassuncao (2024-08-16)
- **[DeepSeek](https://github.com/epfl-dlab/transformers-CFG/pull/73)** (2024-07-24)
- **LLaMA-3** (2024-07-08)
- **[JSON Schema](examples/grammars/custom_json_grammars/README.md)** (2024-05-13)
- **Token masking optimization** (2024-04-25)
- **[Phi](https://github.com/epfl-dlab/transformers-CFG/issues/34)** (2024-04-16)
- **[Online demo with JSON grammar](http://saibo-creator.xyz:7860/) at HF Space** (2024-04-10)
- **Unicode and multilingual grammar** (2024-02-29)
- **Text-Generation-WebUI** (2023-12-17)
- We are pleased to announce that `transformers-cfg` has been integrated into the [Text-Generation-WebUI](https://github.com/oobabooga/text-generation-webui) project, allowing users to leverage CFG capabilities within this widely used text-generation interface ([Pull](https://github.com/oobabooga/text-generation-webui/pull/4953)).

## 🚀 Introduction

`transformers-cfg` is an extension library for the popular Transformers library by Hugging Face, tailored for working with context-free grammars (CFG). This package provides additional tools and functionalities to enhance your experience with natural language processing tasks involving CFGs.

Initially developed as a pull request to the [Hugging Face Transformers](https://github.com/huggingface/transformers) library, you can find the relevant discussion [here](https://github.com/huggingface/transformers/pull/27557).
Initially developed as a pull request to the [Hugging Face Transformers](https://github.com/huggingface/transformers) library ([Pull](https://github.com/huggingface/transformers/pull/27557)), `transformers-cfg` extends the Hugging Face Transformers library to support constrained decoding through context-free grammars (CFG), offering a Transformers parellel for LlamaCPP's GBNF support, but with stricter generation rules.

## 💻 Installation
### Stable
Install the stable version via pip:

- **Stable Version:**

Install the stable version of `transformers-cfg` using pip:

```bash
pip install transformers-cfg
```

- **Development Version:**

For the latest code and updates, install directly from the GitHub repository:

```bash
pip install git+https://github.com/epfl-dlab/transformers-CFG.git@main
```
```bash
pip install transformers-cfg
```

This installs the package from the `main` branch.
### Development
For the latest updates, install directly from GitHub:

## 🔧 Quick Start: Force LLM to Generate a Valid JSON Object
```bash
pip install git+https://github.com/epfl-dlab/transformers-CFG.git@main
```

## 🔧 Valid JSON Object Quick Start
### Command-Line Interface

`transformers-cfg-cli` is a command-line tool that allows you to generate text using a model and a grammar. You can specify the model, grammar, prompts, and other parameters to generate text that conforms to the specified grammar.
The `transformers-cfg-cli` tool enables text generation using a model and a specified grammar.

```bash
transformers-cfg-cli generate \
-m "microsoft/Phi-3-mini-4k-instruct" \
-g "examples/grammars/json.ebnf" \
-p "This is a valid json string for http request:" \
-p "This is a valid JSON string for an HTTP request:" \
--use_4bit \
--max_new_tokens 60 \
--repetition_penalty 1.1
# {"name":"John","age":30,"car":null}
```

We support also Unicode characters in the grammar:

We also support Unicode characters in the grammar:
```bash
transformers-cfg-cli generate \
-m "microsoft/Phi-3-mini-4k-instruct" \
Expand All @@ -72,13 +76,10 @@ transformers-cfg-cli generate \
--repetition_penalty 1.1
```

`transformers-cfg-cli generate --help` provides a list of available options and arguments.

Run `transformers-cfg-cli generate --help` for available options.

<details>
<summary>Click here to see an example of generating a JSON object with minimal changes to HF code, or check it out in <code>examples/generate_json.py</code></summary>

```python
### Transformers *Torch*
```py
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers_cfg.grammar_utils import IncrementalGrammarConstraint
Expand Down Expand Up @@ -125,12 +126,8 @@ if __name__ == "__main__":
"""
```

</details>

<details>
<summary>Click here to see an example with HF pipeline API, or check it out in <code>examples/pipeline_json.py</code></summary>

```python
### Transformers *Pipeline*
```py
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
from transformers_cfg.grammar_utils import IncrementalGrammarConstraint
from transformers_cfg.generation.logits_process import GrammarConstrainedLogitsProcessor
Expand Down Expand Up @@ -166,106 +163,75 @@ generations = pipe(
)
```

</details>

## 💡 Why Should I Use `transformers-cfg`?

- **EBNF Grammar Support:** We support the Extended Backus-Naur Form (EBNF) for grammar description.
- **Seamless Integration:** Our grammar interface is compatible with the llama-cpp project, allowing you to replace llama-cpp with `transformers-cfg` easily.
- **Model Compatibility:** Use any model from the 🤗 Transformers library, including those not supported by llama-cpp.
- **Multilingual Grammar Support:** We support grammars in multiple languages, allowing you to use characters from various languages, including 中文, 日本語, 한국어, हिन्दी, العربية, עברית, and emoji 🤗.
## 💡 Why Use `transformers-cfg`?
- **EBNF Grammar Support**: Uses Extended Backus-Naur Form (EBNF) for grammar description.
- **Seamless Integration**: Compatible with the llama-cpp project for easy replacement.
- **Broad Model Compatibility**: Works with all models in the 🤗 Transformers library.
- **Multilingual Grammar Support**: Enables grammars in various languages, including Chinese (中文), Japanese (日本語), Korean (한국어), Hindi (हिन्दी), Hebrew (עברית), Arabic (العربية), and emoji (🤗).

## 🤔 What Is a Grammar?

TL;DR: Think of it as an enhanced version of regular expressions.
## 🤔 What is a Grammar?
Think of it as an enhanced version of regular expressions.

<details>
<summary>Here is a simple example of a JSON grammar:</summary>
<summary>Example: JSON Grammar</summary>

```bnf
# A JSON object is the root of the grammar
root ::= object

# An object starts with "{" and ends with "}" and contains pairs separated by ","
object ::= "{" pair ("," pair)* "}"

# A pair is a string followed by a ":" and a value
pair ::= string ":" value

# A string is a sequence of alphanumeric characters enclosed in double quotes
string ::= '"' [a-zA-Z0-9]* '"'

# A value can be a string, another object, or a boolean value
value ::= string | object | "true" | "false" | "null"
```

This grammar describes the structure of a JSON object. It specifies that a JSON object consists of key-value pairs, where the key is a string, and the value can be a string, another object, or a boolean value.

You can use grammars to describe simple but useful constructs, such as valid email addresses, URLs, or phone numbers:

```
phone_number ::= "+" [0-9]+
```
This defines the structure of a JSON object.

</details>

For advanced grammar debugging, see our [debugging guide](docs/debugging_custom_grammars.md).

For advanced grammar debugging, check out our [debugging guide](docs/debugging_custom_grammars.md).

## Automatic JSON Schema Grammar Conversion[Experimental]

Learn how to automatically create custom grammars for complex JSON objects in our [documentation](examples%2Fgrammars%2Fcustom_json_grammars%2FREADME.md) on JSON schema to grammar conversion.

## Grammar Collection

We provide a collection of grammars in the `examples/grammars` folder, which are mostly identical to the grammars in the llama-cpp project. We try to keep these grammars up-to-date with the original project, though we cannot yet guarantee that all grammars from llama-cpp can be directly used in `transformers-cfg`.

Available grammars include:
## 🛠 Automatic JSON Schema to Grammar Conversion [Experimental]
Learn to create grammars for complex JSON objects in our [documentation](examples/grammars/custom_json_grammars/README.md).

- [json.ebnf](examples%2Fgrammars%2Fjson.ebnf): For generating valid JSON objects.
- [json_arr.ebnf](examples%2Fgrammars%2Fjson_arr.ebnf): For generating valid JSON arrays.
- [c.ebnf](examples%2Fgrammars%2Fc.ebnf): For generating valid C programs.
- [chess.ebnf](examples%2Fgrammars%2Fchess.ebnf): For generating valid chess moves.
- [arithmetic.ebnf](examples%2Fgrammars%2Farithmetic.ebnf): For generating valid arithmetic expressions.
## 📜 Grammar collection
We maintain a collection of grammars in `examples/grammars`, aligned with llama-cpp:

## Supported Models
- [json.ebnf](examples/grammars/json.ebnf): Valid JSON objects.
- [json_arr.ebnf](examples/grammars/json_arr.ebnf): Valid JSON arrays.
- [c.ebnf](examples/grammars/c.ebnf): Valid C programs.
- [chess.ebnf](examples/grammars/chess.ebnf): Valid chess moves.
- [arithmetic.ebnf](examples/grammars/arithmetic.ebnf): Valid arithmetic expressions.

- [LLaMa Family Models](https://huggingface.co/baffo32/decapoda-research-llama-7B-hf)
- [GPT Family Models](https://huggingface.co/openai-community/gpt2)
- [Bloom Family Models](https://huggingface.co/bigscience/bloom)
- [Mistral Family Models](https://huggingface.co/mistralai/Mistral-7B-v0.1)
- [Falcon Family Models](https://huggingface.co/tiiuae/falcon-7b)
- ...
## ✅ Supported models
- [Qwen2](https://huggingface.co/collections/Qwen/qwen2-6659360b33528ced941e557f) and [2.5](https://huggingface.co/collections/Qwen/qwen25-66e81a666513e518adb90d9e)
- [LLaMa Family](https://huggingface.co/baffo32/decapoda-research-llama-7B-hf)
- [GPT Family](https://huggingface.co/openai-community/gpt2)
- [Bloom Family](https://huggingface.co/bigscience/bloom)
- [Mistral Family](https://huggingface.co/mistralai/Mistral-7B-v0.1)
- [Falcon Family](https://huggingface.co/tiiuae/falcon-7b)

See [supported_models.yaml](docs%2Fsupported_models.yaml) for the full list of supported models.
See [supported_models.yaml](docs/supported_models.yaml) for the full list whose extent is constantly being updated.

As a rule of thumb, all models with the same tokenizer should be naturally supported.
If you encounter an unsupported model, please open an issue or submit a pull request.

If you find any model that is not supported, please open an issue or submit a pull request.


## Citation

**Please consider citing our work if you find the provided resources useful:**
## 📖 Citation
If you find this work useful, please cite it with the reccomended citation:

```bibtex
@inproceedings{geng-etal-2023-grammar,
title = {Grammar-Constrained Decoding for Structured {NLP} Tasks without Finetuning},
author = {Geng, Saibo and Josifoski, Martin and Peyrard, Maxime and West, Robert},
year = 2023,
month = dec,
booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
publisher = {Association for Computational Linguistics},
address = {Singapore},
url = {https://aclanthology.org/2023.emnlp-main.674},
editor = {Bouamor, Houda and Pino, Juan and Bali, Kalika}
title = {Grammar-Constrained Decoding for Structured {NLP} Tasks without Finetuning},
author = {Geng, Saibo and Josifoski, Martin and Peyrard, Maxime and West, Robert},
year = 2023,
month = dec,
booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
publisher = {Association for Computational Linguistics},
address = {Singapore},
url = {https://aclanthology.org/2023.emnlp-main.674},
editor = {Bouamor, Houda and Pino, Juan and Bali, Kalika}
}
```

## License

## 📜 License
This project is licensed under the [MIT License](LICENSE).

## Acknowledgements

This project is derived from the [torch-grammars](https://github.com/Shopify/torch-grammar) project, which was itself derived from the [llama-cpp](https://github.com/ggerganov/llama.cpp) project.
## 🙌 Acknowledgements
Derived from [torch-grammars](https://github.com/Shopify/torch-grammar), which was based on [llama-cpp](https://github.com/ggerganov/llama.cpp).