-
Notifications
You must be signed in to change notification settings - Fork 4.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add media description feature using Azure Content Understanding #2195
Conversation
Check Broken PathsWe have automatically detected the following broken relative paths in your files. Check the file paths and associated broken paths inside them. For more details, check our Contributing Guide.
|
Check Country Locale in URLsWe have automatically detected added country locale to URLs in your files. Check the file paths and associated URLs inside them. For more details, check our Contributing Guide.
|
I need to do some cleanup, per the CI. This also needs a few more tests, as it currently only has a test for the changes to the splitting algorithm. |
@@ -0,0 +1,127 @@ | |||
"figures": [ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should we pull this into the data directory? or not check this in at all?
Check Broken PathsWe have automatically detected the following broken relative paths in your files. Check the file paths and associated broken paths inside them. For more details, check our Contributing Guide.
|
Check Country Locale in URLsWe have automatically detected added country locale to URLs in your files. Check the file paths and associated URLs inside them. For more details, check our Contributing Guide.
|
…zure-search-openai-demo into contentunderstanding
I have now completed both unit testing and manual testing, and this is good to merge. We will merge/release on Monday. |
Purpose
This PR adds a new optional feature that will extract figures in documents (using Azure Document Intelligence) figures output mode and send those figures to Azure Content Understanding (a new service that uses multimodal models) to generate a figure description. It will then insert that figure description into the content, which will then get sent for chunking. If the figure is of a graph or chart, it will include an HTML table with the data.
This gives developers a more lightweight approach to ingest media-rich documents and can be compared to the more heavyweight GPT-4-vision approach.
Does this introduce a breaking change?
When developers merge from main and run the server, azd up, or azd deploy, will this produce an error?
If you're not sure, try it out on an old environment.
Does this require changes to learn.microsoft.com docs?
This repository is referenced by this tutorial
which includes deployment, settings and usage instructions. If text or screenshot need to change in the tutorial,
check the box below and notify the tutorial author. A Microsoft employee can do this for you if you're an external contributor.
Type of change
Code quality checklist
See CONTRIBUTING.md for more details.
python -m pytest
).python -m pytest --cov
to verify 100% coverage of added linespython -m mypy
to check for type errorsruff
andblack
manually on my code.