Graphrag - Private Server - Gemma2/Mistral with Nomic Embeddings running on Ollama with Streamlit
Graphrag - Private Server - Gemma2/Mistral with Nomic Embeddings running on Ollama with Streamlit
TUTORIAL :
How to run your own ChatGPT Large Language Model for FREE on your own computer with the latest GraphRag system from Microsoft and Gemma2 (or any other LLM of your choice) running 100% locally on your computer - running on your data.
Use Cases :
Customer Support Automation: Businesses can use this setup to automate their customer support. The LLM can be trained on company-specific data to answer customer queries accurately and efficiently, reducing the need for human intervention and improving customer satisfaction.
Content Generation: Media companies, bloggers, and marketers can use the LLM to generate creative content such as articles, blog posts, social media posts, etc. This can significantly speed up the content creation process and allow for more frequent updates.
Data Analysis: Companies with large amounts of text data can use the LLM to extract insights from their data. This can be particularly useful for market research, sentiment analysis, and trend prediction.
Personal Assistant: Businesses can offer a personal assistant service to their customers. The LLM can be used to answer queries, schedule appointments, set reminders, and perform other assistant-like tasks.
Education and Training: Educational institutions can use the LLM to create interactive learning materials and provide personalized tutoring. It can also be used for corporate training, allowing employees to learn at their own pace.
Product Development: Tech companies can use the LLM to generate code, write documentation, and perform other development tasks. This can speed up the development process and reduce the workload of the development team.
Privacy Preservation: Since the LLM is running locally, businesses dealing with sensitive data can ensure that their data never leaves their local system, providing an additional layer of security
Learn :
https://imreal.life/page/view-class?id=65
Watch:
Step by Step Process
Install Miniconda
mkdir -p ~/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
rm -rf ~/miniconda3/miniconda.sh
After installing, initialize your newly-installed Miniconda. The following commands initialize for bash and zsh shells:
~/miniconda3/bin/conda init bash
~/miniconda3/bin/conda init zsh
Create your new Environment
conda create -n graphrag python=3.10
Activate Conda Environment
conda activate graphrag ??
Install Ollama
curl -fsSL https://ollama.com/install.sh | sh
ollama pull mistral
ollama pull nomic-embed-text
Install Graphrag
pip install graphrag
Initialize graphrag
python -m graphrag.index --init --root .
Find embeddings file and amend
sudo find / -name openai_embeddings_llm.py
Amend the file as follows:
#openai_embeddings_llm.py
from typing_extensions import Unpack
from graphrag.llm.base import BaseLLM
from graphrag.llm.types import (
EmbeddingInput,
EmbeddingOutput,
LLMInput,
)
from .openai_configuration import OpenAIConfiguration
from .types import OpenAIClientTypes
import ollama
class OpenAIEmbeddingsLLM(BaseLLM[EmbeddingInput, EmbeddingOutput]):
_client: OpenAIClientTypes
_configuration: OpenAIConfiguration
def __init__(self, client: OpenAIClientTypes, configuration: OpenAIConfiguration):
self._client = client
self._configuration = configuration
async def _execute_llm(
self, input: EmbeddingInput, **kwargs: Unpack[LLMInput]
) -> EmbeddingOutput | None:
args = {
"model": self._configuration.model,
**(kwargs.get("model_parameters") or {}),
}
embedding_list = []
for inp in input:
embedding = ollama.embeddings(model="nomic-embed-text", prompt=inp)
embedding_list.append(embedding["embedding"])
return embedding_list
Index your Knowledge base ( First upload .txt files to /input folder)
python -m graphrag.index --root .
Test with command line query
python -m graphrag.query --root . --method global "Tell me about FLAST-AI"
Install Streamlit
pip install streamlit
Create Streamlit APP
import streamlit as st
import subprocess
import os
def query_graphrag(query):
result = subprocess.run(
['python', '-m', 'graphrag.query', '--root', '.', '--method', 'gl>
capture_output=True, text=True
)
return result.stdout, result.stderr
st.title("GraphRAG Query Interface")
user_query = st.text_input("Enter your query:")
if st.button("Submit"):
output, error = query_graphrag(user_query)
st.write("Output:")
st.write(output)
if error:
st.write("Error:")
st.write(error)
Run Streamlit
streamlit run app.py --server.port=8080
Test with curl
curl -X POST \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ollama" \
-d '{"texts": ["Hello, world!"]}' \
http://localhost:11434/api/embed