🤖 localragllm — project 🏗️ 3 🤟 of the 3

This 📍 one 1️⃣ puts 🫳 a local 🏠 brain 🧠 online 🛰️ — a little AI 🦾 model 🔮 running ▶️ right 👉 on your own machine 🖥️ — so the indexer 🧠 (project 2 ✌️) can answer 🗣️ questions 🙋 with no internet 🌐 and no cloud ☁️. Fully offline 📴, fully private 🔒. It uses 🪛 a friendly 😊 tool 🧰 called ollama 🦙.

🧩 the 3 🤟 projects 🗂️

1. localragdownloader 📥 — downloads ⬇️ the airforce 🛩️ PDFs 📑.
2. localragindexer 🧠 — reads 👀 the PDFs and answers 🗣️ questions 🙋.
3. localragllm 🤖 (this 📍 one 1️⃣) — runs ▶️ the local 🏠 model 🔮.

Each 🪞 works 💼 alone, but together 🤝 they were 🪺 made into one 1️⃣ offline 📴 system 🖥️.

🎯 pick 🤏 a model 🔮 (based on your memory 💾)

There are 2 ✌️ choices 🍴 — choose by how much GPU 🎛️ memory 💾 your computer 💻 has:

• llama3.2:3b 🐲 — small 🤏 (~2 ✌️ GB), runs ▶️ on weak 🫠 laptops 💻 (~4 GB free 🆓). Lower 🔽 quality 🥉, but it works 💼.
• qwen2.5:7b 🐉 — bigger 🐘 (~4.7 GB), wants ~8 GB or more memory. Better 🥇 answers 🗣️.

If your machine 🖥️ is strong 💪, pick 🤏 the bigger 🐘 one 1️⃣.

🪜 install ⚙️ and run ▶️ (one 1️⃣ command 🪛)

The script 📜 installs ollama 🦙 (if you do not have it), downloads ⬇️ the model 🔮 plus the embeddings 🧮 brain 🧠, and starts ▶️ the server 🗄️:

git clone https://github.com/DeterminedExecution/localragllm.git
cd localragllm
chmod +x setup.sh

./setup.sh                 # bigger model: qwen2.5:7b
# or, for a weaker laptop:
./setup.sh llama3.2:3b     # small model

When it finishes ‼️, the model 🔮 is online 🛰️ at http://localhost:11434. Done.

⬇️ or download ⬇️ a model 🔮 by hand ✋

If you would rather pull 🪝 them yourself:

ollama pull qwen2.5:7b       # the bigger, better one
ollama pull llama3.2:3b      # the small, light one
ollama pull nomic-embed-text # embeddings — ALWAYS needed

🪖 note 📝: the model 🔮 files 🗃️ are not in this project (they are huge 🐘 — GB each). ollama 🦙 keeps 🤲 them in its own folder 📁 after you pull 🪝 them. So this repo 🗂️ stays 🪶 tiny 🐜.

🔗 turn 🔄 it on in project 2 ✌️

Open 🔓 localragindexer's rag.py, find 🔎 the little config 🎚️ block 🧱 at the top ⏫, and:

1. switch 🔀 PROVIDER = "openai" to PROVIDER = "local".
2. set 🛠️ LOCAL_MODEL = "qwen2.5:7b" (or whatever model 🔮 you pulled 🪝).
3. rebuild 🔨 the index 🧠 once 1️⃣: python index_corpus.py.

Now 📍 the indexer 🧠 thinks 💭 with your local 🏠 brain 🧠 — no cloud ☁️, no key 🔑, no bill 🧾. 3 🤟 projects 🗂️, one 1️⃣ happy 😀 offline 📴 system 🖥️.