Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backend: Check tool parameters generated by model #849

Conversation

EugeneLightsOn
Copy link
Collaborator

@EugeneLightsOn EugeneLightsOn commented Nov 19, 2024

The metaclass of the tool's base class decorates the call method of the tools to validate the passed parameters.
AI Description

This pull request introduces a new ParametersCheckingMeta metaclass to the BaseTool class, which automatically decorates the call method with a parameter checker. This checker validates the parameters passed to the call method against the tool's parameter definitions, ensuring that required parameters are provided and that the types of the parameters match the expected types.

The parameter checker is implemented using two new functions: check_type and check_tool_parameters. The check_type function recursively checks the type of a value against a given type description, while the check_tool_parameters function decorates the call method and performs the actual parameter validation.

The ParametersCheckingMeta metaclass is used as the metaclass for the BaseTool class, ensuring that all subclasses of BaseTool have their call method decorated with the parameter checker.

  • The check_type function is added to check the type of a value against a given type description.
  • The check_tool_parameters function is added to decorate the call method and perform parameter validation.
  • The ParametersCheckingMeta metaclass is introduced to automatically decorate the call method with the parameter checker.
  • The BaseTool class is modified to use the ParametersCheckingMeta metaclass.
  • The ToolDefinition class is updated to include a get_tool_definition method, which returns a ToolDefinition object.
  • The mock_get_available_tools function is modified to use the get_tool_definition method to get the tool definition for a given tool.
  • The test_async_call_tools_success, test_async_call_tools_failure, test_async_call_tools_timeout, and test_async_call_tools_failure_and_success functions are updated to use the get_tool_definition method to get the tool definition for the MockCalculator and MockWebScrape tools.
  • Three new test functions are added: test_tools_params_checker_invalid_param, test_tools_params_checker_invalid_param_type, and test_tools_params_checker_required_param_empty. These functions test the parameter checker by passing invalid parameters to the call method and asserting that the expected errors are raised.

@EugeneLightsOn EugeneLightsOn force-pushed the eugene/tlk-2091-add-way-to-check-if-model-returned-parameters-correspond-to branch from 12b3eec to 0ac0ac4 Compare November 19, 2024 13:27
@EugeneLightsOn EugeneLightsOn changed the title Backend: Check tool parameters generated by model WIP: DON'T MERGE - Backend: Check tool parameters generated by model Nov 19, 2024
@codecov-commenter
Copy link

codecov-commenter commented Nov 19, 2024

Codecov Report

Attention: Patch coverage is 77.16535% with 29 lines in your changes missing coverage. Please review.

Project coverage is 79.61%. Comparing base (5502d8d) to head (9aac937).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
src/backend/tools/utils/tools_checkers.py 58.69% 19 Missing ⚠️
src/backend/tools/base.py 68.42% 6 Missing ⚠️
src/backend/tests/unit/chat/test_tool_calls.py 94.91% 3 Missing ⚠️
src/backend/tools/lang_chain.py 66.66% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #849      +/-   ##
==========================================
+ Coverage   79.56%   79.61%   +0.04%     
==========================================
  Files         251      251              
  Lines       10632    10785     +153     
==========================================
+ Hits         8459     8586     +127     
- Misses       2173     2199      +26     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@EugeneLightsOn EugeneLightsOn changed the title WIP: DON'T MERGE - Backend: Check tool parameters generated by model Backend: Check tool parameters generated by model Nov 19, 2024
@EugeneLightsOn EugeneLightsOn changed the title Backend: Check tool parameters generated by model WIP: Backend: Check tool parameters generated by model Nov 19, 2024
@EugeneLightsOn EugeneLightsOn changed the title WIP: Backend: Check tool parameters generated by model Backend: Check tool parameters generated by model Nov 20, 2024
…if-model-returned-parameters-correspond-to

# Conflicts:
#	src/backend/tests/unit/chat/test_tool_calls.py
#	src/backend/tools/base.py
@EugeneLightsOn EugeneLightsOn merged commit 68442e6 into main Nov 26, 2024
8 checks passed
@EugeneLightsOn EugeneLightsOn deleted the eugene/tlk-2091-add-way-to-check-if-model-returned-parameters-correspond-to branch November 26, 2024 21:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants