Merge pull request #1088 from guardrails-ai/docs/on-fail-docs-update

Docs: Enhance OnFail documentation

docs/concepts/concurrency.md

@@ -98,7 +98,7 @@ When asynchronous validation occurs, there are multiple levels of concurrency po
 When validating unstructured data, i.e. text, the LLM output is treated the same as if it were a property on an object.  This means that the validators applied to is have the ability to run concurrently utilizing the event loop.
 
 ### Handling Failures During Async Concurrency
-The Guardrails validation loop is opinionated about how it handles failures when running validators concurrently so that it spends the least amount of time processing an output that would result in a failure.  It's behavior comes down to when and what it returns based on the [corrective action](/how_to_guides/custom_validators#on-fail) specified on a validator.  Corrective actions are processed concurrently since they are specific to a given validator on a given property.  This means that interruptive corrective actions, namely `EXCEPTION`, will be the first corrective action enforced because the exception is raised as soon as the failure is evaluated.  The remaining actions are handled in the following order after all futures are collected from the validation of a specific property:
+The Guardrails validation loop is opinionated about how it handles failures when running validators concurrently so that it spends the least amount of time processing an output that would result in a failure.  It's behavior comes down to when and what it returns based on the [corrective action](/concepts/validator_on_fail_actions) specified on a validator.  Corrective actions are processed concurrently since they are specific to a given validator on a given property.  This means that interruptive corrective actions, namely `EXCEPTION`, will be the first corrective action enforced because the exception is raised as soon as the failure is evaluated.  The remaining actions are handled in the following order after all futures are collected from the validation of a specific property:
 1. `FILTER` and `REFRAIN`
 2. `REASK`
 3. `FIX`

docs/concepts/validator_on_fail_actions.md

@@ -0,0 +1,94 @@
+# Validator OnFail Actions
+
+## OnFail Actions
+
+Validators ship with several out of the box `on_fail` policies. The `OnFailAction` specifies the corrective action that should be taken if the quality criteria is not met. The corrective action can be one of the following:
+
+| Action    | Behavior                                                                                                                                                                                               |
+|-----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `OnFailAction.REASK`   | Reask the LLM to generate an output that meets the correctness criteria specified in the validator.  The prompt used for reasking contains information about which quality criteria failed, which is auto-generated by the validator. |
+| `OnFailAction.FIX`     | Programmatically fix the generated output to meet the correctness criteria when possible. E.g. the formatter `provenance_llm` validator will remove any sentences that are estimated to be hallucinated.             |
+| `OnFailAction.FILTER`  | (Only applicable for structured data validation) Filter the incorrect value. This only filters the field that fails, and will return the rest of the generated output.                                                                                  |
+| `OnFailAction.REFRAIN` | Refrain from returning an output. This is useful when the generated output is not safe to return, in which case a `None` value is returned instead.                                                          |
+| `OnFailAction.NOOP`    | Do nothing. The failure will still be recorded in the logs, but no corrective action will be taken.                                                                                                    |
+| `OnFailAction.EXCEPTION`  | Raise an exception when validation fails.     |
+| `OnFailAction.FIX_REASK` | First, fix the generated output deterministically, and then rerun validation with the deterministically fixed output. If validation fails, then perform reasking.           |
+| `OnFailAction.CUSTOM` | This action is set internally when the validator is passed a custom function to handle failures.  The function is called with the value that failed validation and the FailResult returned from the Validator.  i.e. the custom on fail handler must implement the method signature `def on_fail(value: Any, fail_result: FailResult) -> Any`           |
+
+## Example 
+
+Let's assess the output of a trivial validator in diffent cases of `OnFailAction`. 
+Take the following Validator a basic implementation of a Toxic Language Validator:
+
+```python
+
+TOXIC_WORDS = ["asshole", "damn"]
+
+class BasicToxicLanguage(Validator):
+    def validate(self, value: Any, metadata: Dict) -> ValidationResult:
+        is_toxic_language = any(toxic_word in value for toxic_word in TOXIC_WORDS)
+
+        # if a value contains toxic words we return FailResult otherwise PassResult
+        if is_toxic_language:
+            for toxic_word in TOXIC_WORDS:
+                value = value.replace(toxic_word, "")
+            return FailResult(
+                error_message=f"Value '{value}' contains toxic language including words: {TOXIC_WORDS} which is not allowed.",
+                fix_value=value,
+            )
+
+        return PassResult()
+```
+
+> For more information on how to write Custom Validators refer to our guide [here](/how_to_guides/custom_validators)
+
+Now suppose some unhinged LLM returns `damn you!`, in ths scenario:
+
+- `OnFailAction.REASK` an LLM will be reasked to correct it's output based on the  `error_message` provided to `FailResult`. In this example it will reask the LLM with a reask prompt which includes the error message: `Value 'damn you!' contains toxic language including words: ["asshole","damn"] which is not allowed.`. You can set `num_reasks` on the `guard()` call to determine how many times we retry.
+- `OnFailAction.FIX` the value is replaced with `fix_value` that is provided to `FailResult`. In this example value from the LLM `damn you!` will be returned as `you!`.
+- `OnFailAction.FILTER` if used in structured data generation we do not display the field that fails validation (more on this below). In this example value from the LLM `damn you!` will return an empty response.
+- `OnFailAction.REFRAIN` we do not return anything as the validation is deemed unsafe for end users. In this example value from the LLM `damn you!` will return an empty response.
+- `OnFailAction.NOOP` the value is returned as is and failures are logged in the history. In this example value we return the value as is `damn you!`
+- `OnFailAction.EXCEPTION` during a guard execution or direct validation call we raise an error indicating the validation failed.
+- `OnFailAction.FIX_REASK` during a fix reask, we first perform the same action as `OnFailAction.FIX` and re-validate the output. If it fails we run a `OnFailAction.REASK` action, otherwise we return the passed validation.
+
+
+## Structured Data
+
+Using `OnFail` actions is powerful when also working with structured data as we can determine how to treat each field's validation failure.
+
+
+```python
+prompt = """
+Given the following fast food order, please provide a summary of the orders.
+${order}
+${gr.complete_xml_suffix_v2}
+"""
+
+order = """I want a burger with two large fries and a coke zero."""
+
+# MinimumOneRange is a hypothetical custom validator that an integer > 0 is supplied
+class Lineitem(BaseModel):
+    item: str = Field(description="The name of the item being ordered", validators=[LowerCase()])
+    quantity: int = Field(description="The quantity of the item being ordered", validators=[MinimumOneRange(min=1, max=10, on_fail="fix")])
+
+guard = Guard.from_pydantic(output_class=List[Lineitem])
+
+response = guard(
+    model="gpt-4o",
+    messages=[{
+        "role": "system",
+        "content": "You are a helpful assistant."
+    },{
+        "role": "user",
+        "content": prompt
+    }],
+    prompt_params={"order": order},
+)
+
+print(response.validated_output)
+
+# [{'item': 'burger', 'quantity': 1},
+#  {'item': 'fries', 'quantity': 2},
+#  {'item': 'coke zero', 'quantity': 1}]
+```

docs/concepts/validators.md

@@ -6,7 +6,7 @@ Validators are how we apply quality controls to the outputs of LLMs.  They speci
 Each validator is a method that encodes some criteria, and checks if a given value meets that criteria.
 
 - If the value passes the criteria defined, the validator returns `PassResult`. In most cases this means returning that value unchanged. In very few advanced cases, there may be a a value override (the specific validator will document this).
-- If the value does not pass the criteria, a `FailResult` is returned.  In this case, the validator applies the user-configured `on_fail` policies (see [On-Fail Policies](/docs/how_to_guides/custom_validators#on-fail)).
+- If the value does not pass the criteria, a `FailResult` is returned.  In this case, the validator applies the user-configured `on_fail` policies (see [On-Fail Policies](/concepts/validator_on_fail_actions)).
 
 ## Runtime Metadata
 

docs/how_to_guides/custom_validators.md

@@ -64,20 +64,10 @@ class ToxicWords(Validator):
 
 ## On Fail
 
-Validators ship with several out of the box `on_fail` policies. The `OnFailAction` specifies the corrective action that should be taken if the quality criteria is not met. The corrective action can be one of the following:
-
-| Action    | Behavior                                                                                                                                                                                               |
-|-----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| `OnFailAction.REASK`   | Reask the LLM to generate an output that meets the correctness criteria specified in the validator.  The prompt used for reasking contains information about which quality criteria failed, which is auto-generated by the validator. |
-| `OnFailAction.FIX`     | Programmatically fix the generated output to meet the correctness criteria when possible. E.g. the formatter `provenance_llm` validator will remove any sentences that are estimated to be hallucinated.             |
-| `OnFailAction.FILTER`  | (Only applicable for structured data validation) Filter the incorrect value. This only filters the field that fails, and will return the rest of the generated output.                                                                                  |
-| `OnFailAction.REFRAIN` | Refrain from returning an output. This is useful when the generated output is not safe to return, in which case a `None` value is returned instead.                                                          |
-| `OnFailAction.NOOP`    | Do nothing. The failure will still be recorded in the logs, but no corrective action will be taken.                                                                                                    |
-| `OnFailAction.EXCEPTION`  | Raise an exception when validation fails.     |
-| `OnFailAction.FIX_REASK` | First, fix the generated output deterministically, and then rerun validation with the deterministically fixed output. If validation fails, then perform reasking.           |
-| `OnFailAction.CUSTOM` | This action is set internally when the validator is passed a custom function to handle failures.  The function is called with the value that failed validation and the FailResult returned from the Validator.  i.e. the custom on fail handler must implement the method signature `def on_fail(value: Any, fail_result: FailResult) -> Any`           |
-
 In the code below, a `fix_value` will be supplied in the `FailResult`. This value will represent a programmatic fix that can be applied to the output if `on_fail='fix'` is passed during validator initialization.
+
+> For more details about on fail actions refer to: [On Fail Actions](/concepts/validator_on_fail_actions)
+
 ```py
 from typing import Callable, Dict, Optional
 from guardrails.validators import (

docusaurus/sidebars.js

@@ -51,6 +51,7 @@ const sidebars = {
   concepts: [
     "concepts/guard",
     "concepts/validators",
+    "concepts/validator_on_fail_actions",
     // "concepts/guardrails",
     "concepts/hub",
     "concepts/deploying",