Skip to content

Commit

Permalink
chore(langchain): disable flaky tests (#11511)
Browse files Browse the repository at this point in the history
There appear to be stability issues with using snapshots and/or
LangChain in general.

There are failures in the mocked tests that look like:

```
builtins.AssertionError: assert 0 == 1
 +  where 0 = <MagicMock name='LLMObsSpanWriter().enqueue' id='127482146130048'>.call_count
 +    where <MagicMock name='LLMObsSpanWriter().enqueue' id='127482146130048'> = <MagicMock name='LLMObsSpanWriter()' id='127482147073440'>.enqueue
```

as well as failures with snapshot based tests:

```
builtins.Failed: At request <Request GET /test/session/snapshot >:
   At snapshot (token='tests.contrib.langchain.test_langchain_community.test_lcel_chain_simple'):
    - Directory: /go/src/github.com/DataDog/apm-reliability/dd-trace-py/tests/snapshots
    - CI mode: 1
    - Trace File: /go/src/github.com/DataDog/apm-reliability/dd-trace-py/tests/snapshots/tests.contrib.langchain.test_langchain_community.test_lcel_chain_simple.json
    - Stats File: /go/src/github.com/DataDog/apm-reliability/dd-trace-py/tests/snapshots/tests.contrib.langchain.test_langchain_community.test_lcel_chain_simple_tracestats.json
    At compare of 1 expected trace(s) to 0 received trace(s):
Did not receive expected traces: 'langchain.request'
```

While we investigate a more stable method of testing it makes sense to
disable the tests to avoid noise to our neighbours in the library :).

DOWN WITH FLAKY TESTS

[](https://datadoghq.atlassian.net/browse/MLOB-1920)

## Checklist
- [x] PR author has checked that all the criteria below are met
- The PR description includes an overview of the change
- The PR description articulates the motivation for the change
- The change includes tests OR the PR description describes a testing
strategy
- The PR description notes risks associated with the change, if any
- Newly-added code is easy to change
- The change follows the [library release note
guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html)
- The change includes or references documentation updates if necessary
- Backport labels are set (if
[applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting))

## Reviewer Checklist
- [ ] Reviewer has checked that all the criteria below are met 
- Title is accurate
- All changes are related to the pull request's stated goal
- Avoids breaking
[API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces)
changes
- Testing strategy adequately addresses listed risks
- Newly-added code is easy to change
- Release note makes sense to a user of the library
- If necessary, author has acknowledged and discussed the performance
implications of this PR as reported in the benchmarks PR comment
- Backport labels are set in a manner that is consistent with the
[release branch maintenance
policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)
  • Loading branch information
Kyle-Verhoog authored Dec 2, 2024
1 parent b89db9f commit 142e643
Show file tree
Hide file tree
Showing 2 changed files with 0 additions and 2,536 deletions.
Loading

0 comments on commit 142e643

Please sign in to comment.