Graphrag - Private Server - Gemma2/Mistral with Nomic Embeddings running on Ollama with Streamlit

TUTORIAL :

How to run your own ChatGPT Large Language Model for FREE on your own computer with the latest GraphRag system from Microsoft and Gemma2 (or any other LLM of your choice) running 100% locally on your computer - running on your data.

Use Cases :

Customer Support Automation: Businesses can use this setup to automate their customer support. The LLM can be trained on company-specific data to answer customer queries accurately and efficiently, reducing the need for human intervention and improving customer satisfaction.

Content Generation: Media companies, bloggers, and marketers can use the LLM to generate creative content such as articles, blog posts, social media posts, etc. This can significantly speed up the content creation process and allow for more frequent updates.

Data Analysis: Companies with large amounts of text data can use the LLM to extract insights from their data. This can be particularly useful for market research, sentiment analysis, and trend prediction.

Personal Assistant: Businesses can offer a personal assistant service to their customers. The LLM can be used to answer queries, schedule appointments, set reminders, and perform other assistant-like tasks.

Education and Training: Educational institutions can use the LLM to create interactive learning materials and provide personalized tutoring. It can also be used for corporate training, allowing employees to learn at their own pace.

Product Development: Tech companies can use the LLM to generate code, write documentation, and perform other development tasks. This can speed up the development process and reduce the workload of the development team.

Privacy Preservation: Since the LLM is running locally, businesses dealing with sensitive data can ensure that their data never leaves their local system, providing an additional layer of security

Learn :

https://imreal.life/page/view-class?id=65

Watch:

https://youtu.be/eujn7RBEleE

Step by Step Process

Install Miniconda

mkdir -p ~/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
rm -rf ~/miniconda3/miniconda.sh

After installing, initialize your newly-installed Miniconda. The following commands initialize for bash and zsh shells:

~/miniconda3/bin/conda init bash
~/miniconda3/bin/conda init zsh

Create your new Environment

conda create -n graphrag python=3.10

Activate Conda Environment

conda activate graphrag ??

Install Ollama

curl -fsSL https://ollama.com/install.sh | sh

ollama pull mistral

ollama pull nomic-embed-text

Install Graphrag

pip install graphrag

Initialize graphrag

python -m graphrag.index --init --root .

Find embeddings file and amend

sudo find / -name openai_embeddings_llm.py

Amend the file as follows:

#openai_embeddings_llm.py
from typing_extensions import Unpack

from graphrag.llm.base import BaseLLM

from graphrag.llm.types import (

  EmbeddingInput,

  EmbeddingOutput,

  LLMInput,

)

from .openai_configuration import OpenAIConfiguration

from .types import OpenAIClientTypes

import ollama

class OpenAIEmbeddingsLLM(BaseLLM[EmbeddingInput, EmbeddingOutput]):

  _client: OpenAIClientTypes

  _configuration: OpenAIConfiguration


  def __init__(self, client: OpenAIClientTypes, configuration: OpenAIConfiguration):

    self._client = client

    self._configuration = configuration

  async def _execute_llm(

    self, input: EmbeddingInput, **kwargs: Unpack[LLMInput]

  ) -> EmbeddingOutput | None:

    args = {

      "model": self._configuration.model,

      **(kwargs.get("model_parameters") or {}),

    }

    embedding_list = []

    for inp in input:

      embedding = ollama.embeddings(model="nomic-embed-text", prompt=inp)

      embedding_list.append(embedding["embedding"])

    return embedding_list

Index your Knowledge base ( First upload .txt files to /input folder)

python -m graphrag.index --root . 

Test with command line query

python -m graphrag.query --root . --method global "Tell me about FLAST-AI"

Install Streamlit

pip install streamlit

Create Streamlit APP

import streamlit as st
import subprocess
import os

def query_graphrag(query):
  result = subprocess.run(
    ['python', '-m', 'graphrag.query', '--root', '.', '--method', 'gl>
    capture_output=True, text=True
  )
  return result.stdout, result.stderr

st.title("GraphRAG Query Interface")
user_query = st.text_input("Enter your query:")

if st.button("Submit"):
  output, error = query_graphrag(user_query)
  st.write("Output:")
  st.write(output)
  if error:
    st.write("Error:")
    st.write(error)

Run Streamlit

streamlit run app.py --server.port=8080

Test with curl

curl -X POST \
 -H "Content-Type: application/json" \
 -H "Authorization: Bearer ollama" \
 -d '{"texts": ["Hello, world!"]}' \
 http://localhost:11434/api/embed