Setting Up Local Language Models for Your App
Your app relies on two essential models: Embeddings and Text Generation. While OpenAI's default models work seamlessly, you have the flexibility to switch providers or even run the models locally.
Navigate to the
.env file or set the following environment variables:
LLM_NAME=<your Text Generation model> API_KEY=<API key for Text Generation> EMBEDDINGS_NAME=<LLM for Embeddings> EMBEDDINGS_KEY=<API key for Embeddings> VITE_API_STREAMING=<true or false>
You can omit the keys if users provide their own. Ensure you set
- openai (More details (opens in a new tab))
- anthropic (More details (opens in a new tab))
- manifest (More details (opens in a new tab))
- cohere (More details (opens in a new tab))
- llama.cpp (More details (opens in a new tab))
- huggingface (Arc53/DocsGPT-7B by default)
- sagemaker (Mode details (opens in a new tab))
Note: for huggingface you can choose any model inside application/llm/huggingface.py or pass llm_name on init, loads
If you want to be completely local, set
For llama.cpp Download the required model and place it in the
Alternatively, for local Llama setup, run
setup.sh and choose option 1. The script handles the DocsGPT model addition.
If working with sensitive data, host everything locally by setting
LLM_NAME, llama.cpp or huggingface, use any model available on Hugging Face, for llama.cpp you need to convert it into gguf format.
That's it! Your app is now configured for local and private hosting, ensuring optimal security for critical data.