Integration for Building LLM/Foundation Model AI Applications with Damavand

[kkiani]

Description:

This proposal suggests adding an integration to Damavand that will help users develop Large Language Model (LLM) and Foundation Model AI applications. The goal is to provide a streamlined, cloud-agnostic environment for building, deploying, and scaling AI applications using Damavand’s ARC pattern.

Key Features:

• Provide templates and tools for integrating LLMs and Foundation Models within Damavand.
• Simplify the infrastructure setup (e.g., compute, storage, and data pipelines) required to support large AI models.
• Ensure multi-cloud compatibility for deploying LLM/AI applications on different cloud providers.

Research Focus:

• Investigate how similar platforms (like EPAM RAIL) approach building and deploying LLM applications.
• Explore key infrastructure requirements for hosting and running large models efficiently in a cloud-agnostic way.
• Identify ways to integrate popular AI frameworks (e.g., PyTorch, TensorFlow) into Damavand’s structure, ensuring support for Foundation Models.

[kkiani]

For most part, RAIL/DIAL seems like a vllm with a UI on top. I honestly don't think the UI is the biggest value that we can provide to our user apart from an easily interface to play with a model. Here is what I think we can provide:

• For the infra, Damavand can implement a component for the resource layer as usual using SageMaker as backend on AWS which supports vllm. Checkout AWS Blogs and this tutorial on Medium. For the Azure not sure yet but we can always use a VM.
• For the security practices, we can mostly leave it to the platform (DLZ) but we of course follow the best practices.
• For the usage and make it practical, we can provide a UI but I like magnetic approach better as we are providing a framework to build on top of it fast and easy instead of just an application. However, we need to dig in more into the framework to see how compatible it is with vllm.

What do you think @shahinism?

[kkiani]

I cannot find any knowledge based solution with EPAM DIAL, however this a feature provided by AWS Bedrock and can be an strategic direction. The vllm and magnetic does not provide such functionally out of the box. However, there is another open source solution that can provide such functionality with some other functionality on top called haystack. I don't see any easy way to combine this with the two previous solution. I believe based on the direction we need to adopt either of them.

[kkiani]

• For the usage and make it practical, we can provide a UI but I like magnetic approach better as we are providing a framework to build on top of it fast and easy instead of just an application. However, we need to dig in more into the framework to see how compatible it is with vllm.

I took a look deeper into the both frameworks here, they both support OpenAI API Model. As a result they should be easily compatible to work with eachother.

[kkiani]

Bedrock also has a possibility to serve custom models. However, the supported models are very limited to three architecture. This is not align with our goal to provide an standard platform to serve a verity of open source models.

[kkiani]

I am going to start with vllm for now

[shahinism]

Dependent to the target audience of this product, dspy could also be a feasible option to consider. It's more developer oriented, and specially the prompt optimization facility could be very intriguing on top of vllm. This is compared to magnetic, which is more tailored towards data analysts (and is less complex).

[kkiani]

Dependent to the target audience of this product, dspy could also be a feasible option to consider. It's more developer oriented, and specially the prompt optimization facility could be very intriguing on top of vllm. This is compared to magnetic, which is more tailored towards data analysts (and is less complex).

Thanks @shahinism, I believe dspy is the best for our needs. I will start on the integration and close this ticket as the planning is complete with the issues: #22, #23, #26, #27, #28, and #29.