Skip to content

Commit

Permalink
fix(llmobs): fix langchain nested llm spans (#9354)
Browse files Browse the repository at this point in the history
- Fix nested LLM Spans for LLMObs langchain integration when
openai/bedrock is enabled
- If the LLMObs integration detects that one of our supported LLM
integrations is enabled (bedrock or openai) and also detects that the
user is using a Langchain plugin for these llm integrations (which we
detect using the `model_provider` tag set by the langchain integration),
then the llm operation is not marked as a workflow span. If not, the llm
operation is marked as a an llm span.

We also add a `TestLangchainTraceStructureWithLlmIntegrations` class to
allow us to assert on the "trace structure" (where structure is defined
by the order of span kinds enqueued to the LLM Obs trace writer) to
verify the changes above

## Bedrock

### With bedrock enabled

<img width="908" alt="image"
src="https://github.com/DataDog/dd-trace-py/assets/42917263/74b89b5e-a3a4-462f-a457-265c29182619">

### Without bedrock enabled
<img width="965" alt="image"
src="https://github.com/DataDog/dd-trace-py/assets/42917263/48fba486-41ee-4370-9603-0202d6651fd8">


## OpenAI

### With openai enabled:

<img width="930" alt="image"
src="https://github.com/DataDog/dd-trace-py/assets/42917263/69a97429-04e5-429e-a506-e8d676b95568">






### Without openai enabled

<img width="844" alt="image"
src="https://github.com/DataDog/dd-trace-py/assets/42917263/42f99bec-a457-45e0-8e80-75a5d545b73d">


## Checklist

- [x] Change(s) are motivated and described in the PR description
- [x] Testing strategy is described if automated tests are not included
in the PR
- [x] Risks are described (performance impact, potential for breakage,
maintainability)
- [x] Change is maintainable (easy to change, telemetry, documentation)
- [x] [Library release note
guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html)
are followed or label `changelog/no-changelog` is set
- [x] Documentation is included (in-code, generated user docs, [public
corp docs](https://github.com/DataDog/documentation/))
- [x] Backport labels are set (if
[applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting))
- [x] If this PR changes the public interface, I've notified
`@DataDog/apm-tees`.

## Reviewer Checklist

- [ ] Title is accurate
- [ ] All changes are related to the pull request's stated goal
- [ ] Description motivates each change
- [ ] Avoids breaking
[API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces)
changes
- [ ] Testing strategy adequately addresses listed risks
- [ ] Change is maintainable (easy to change, telemetry, documentation)
- [ ] Release note makes sense to a user of the library
- [ ] Author has acknowledged and discussed the performance implications
of this PR as reported in the benchmarks PR comment
- [ ] Backport labels are set in a manner that is consistent with the
[release branch maintenance
policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)

---------

Co-authored-by: lievan <[email protected]>
Co-authored-by: Brett Langdon <[email protected]>
  • Loading branch information
3 people authored May 29, 2024
1 parent 9e637e7 commit e9d7726
Show file tree
Hide file tree
Showing 13 changed files with 610 additions and 257 deletions.
91 changes: 91 additions & 0 deletions .riot/requirements/12739ce.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
#
# This file is autogenerated by pip-compile with Python 3.9
# by the following command:
#
# pip-compile --no-annotate .riot/requirements/12739ce.in
#
ai21==2.4.0
ai21-tokenizer==0.9.1
aiohttp==3.9.5
aiosignal==1.3.1
annotated-types==0.7.0
anyio==4.4.0
async-timeout==4.0.3
attrs==23.2.0
boto3==1.34.114
botocore==1.34.114
certifi==2024.2.2
charset-normalizer==3.3.2
cohere==5.5.3
coverage[toml]==7.5.3
dataclasses-json==0.6.6
distro==1.9.0
exceptiongroup==1.2.1
fastavro==1.9.4
filelock==3.14.0
frozenlist==1.4.1
fsspec==2024.5.0
h11==0.14.0
httpcore==1.0.5
httpx==0.27.0
httpx-sse==0.4.0
huggingface-hub==0.23.2
hypothesis==6.45.0
idna==3.7
importlib-metadata==7.1.0
iniconfig==2.0.0
jmespath==1.0.1
jsonpatch==1.33
jsonpointer==2.4
langchain==0.1.20
langchain-aws==0.1.6
langchain-community==0.0.38
langchain-core==0.1.52
langchain-openai==0.1.5
langchain-pinecone==0.1.1
langchain-text-splitters==0.0.2
langsmith==0.1.63
marshmallow==3.21.2
mock==5.1.0
multidict==6.0.5
mypy-extensions==1.0.0
numexpr==2.10.0
numpy==1.26.4
openai==1.12.0
opentracing==2.4.0
orjson==3.10.3
packaging==23.2
pinecone-client==3.2.2
pluggy==1.5.0
psutil==5.9.8
pydantic==2.7.2
pydantic-core==2.18.3
pytest==8.2.1
pytest-asyncio==0.21.1
pytest-cov==5.0.0
pytest-mock==3.14.0
pytest-randomly==3.15.0
python-dateutil==2.9.0.post0
pyyaml==6.0.1
regex==2024.5.15
requests==2.32.2
s3transfer==0.10.1
sentencepiece==0.2.0
six==1.16.0
sniffio==1.3.1
sortedcontainers==2.4.0
sqlalchemy==2.0.30
tenacity==8.3.0
tiktoken==0.7.0
tokenizers==0.19.1
tomli==2.0.1
tqdm==4.66.4
types-requests==2.31.0.6
types-urllib3==1.26.25.14
typing-extensions==4.12.0
typing-inspect==0.9.0
urllib3==1.26.18
vcrpy==6.0.1
wrapt==1.16.0
yarl==1.9.4
zipp==3.19.0
79 changes: 0 additions & 79 deletions .riot/requirements/15757fd.txt

This file was deleted.

81 changes: 0 additions & 81 deletions .riot/requirements/1bd8794.txt

This file was deleted.

81 changes: 0 additions & 81 deletions .riot/requirements/1e00937.txt

This file was deleted.

Loading

0 comments on commit e9d7726

Please sign in to comment.