From 2ddb9ebf1b9af2a229375ef1e2ef6bc8a5092ded Mon Sep 17 00:00:00 2001
From: Tejas Mahajan <141305477+mahajantejas@users.noreply.github.com>
Date: Wed, 2 Sep 2026 14:16:18 +0530
Subject: [PATCH 01/10] Revise AI Evaluations documentation with details of
version 2 release
Revise AI Evaluations documentation with details of version 2 release
---
.../AI Evaluations in Glific.md | 166 +++++-------------
1 file changed, 46 insertions(+), 120 deletions(-)
diff --git a/docs/5. Integrations/AI Evaluations in Glific.md b/docs/5. Integrations/AI Evaluations in Glific.md
index 949e49d24..d8a116970 100644
--- a/docs/5. Integrations/AI Evaluations in Glific.md
+++ b/docs/5. Integrations/AI Evaluations in Glific.md
@@ -3,147 +3,73 @@
| 3 minutes read |
Level: Advanced |
- Last Updated: June 2026 |
+ Last Updated: Sept 2026 |
-# AI Evaluations
+# Running AI evaluations
+Evaluations can be run through on by navigating to `Golden Q&A Evaluation` tab within the AI Assistant page. AI Evaluations score a version against a Golden Q&A — a fixed set of questions paired with ideal answers. The AI evaluations return 3 scores, a reason for the score, and an overall summary and next steps based on all the scores from the evaluation run.
-AI Evaluations allow your organization to test and measure how accurately your AI Assistant responds to questions, by comparing its answers against a trusted set of "golden" question-answer pairs. This helps NGOs ensure their AI Assistants are performing well and giving the right information to beneficiaries before deploying them at scale.
+The evaluation scores answer on three independent metrics: **Adherence to Ground Truth** (is AI generated answer correct in meaning against the golden answer?), **Adherence to Knowledge Base** (is every claim in AI generated answer grounded in the information retrieved from knowledge base, or hallucinated?), and **Adherence to Prompt** (does AI generated answer follow the configured language, tone, and answer-vs-refuse behaviour?), each returned as a score between 0 to 5, plus a plain-language reasoning.
-## Raising the request
-1. Navigate to `AI Toolkit` -> `AI Evals`, you should see the button to `Request Access`
-
+The result is that a wrong-but-similar sounding answer can be caught by adherence to ground truth , hallucinations get flagged with the specific unsupported claim named by adherence to knowledge base, and all results (scores, reasoning) are stored to enable comparison between versions.
-2. Click on the button to raise the request to enable the feature
-3. Glific team will enable this feature within 24 hours.
-4. Once enabled the `AI Evals` page is visible after logging in the platform. No other notification outside the Glific platform will be sent.
+## Set up a Golden Q&A set (once)
+1. On the `Golden Q&A Evaluation` tab,
+2. click `Manage Golden Q&A`.
+ 
-## Prerequisites
-Before running an AI Evaluation, make sure you have:
-1. At least one AI Assistant configured in Glific (see: AI Assistants [documentation](https://glific.github.io/docs/docs/Integrations/Creating%20and%20modifying%20assistants%20in%20Glific))
-2. A Golden QA dataset — a CSV file containing benchmark questions and their expected answers
+4. You’ll see existing sets, if you have run evaluations in past; else,
+5. You can use `Add Golden Q&A` button. To create one,
+ 
-## Navigating to AI Evaluations
-1. Log in to your Glific account.
-2. On the left sidebar, click on `AI Toolkit`
-3. Click on `AI Evals`
-
+7. click Add Golden Q&A, give it a name, and upload a CSV with two columns: question and answer.
+8. There’s a [Google Sheet template link](https://docs.google.com/spreadsheets/d/198UpOMeU53s9O-fwbIl0DIJLuD3l24jgkq74CoDfSQM/copy) to help format the file.
+9. Click `Add` to save it.
-The page shows a table of all past evaluations with the following columns:
-- Evaluation Name — The name you gave the evaluation, along with the AI Assistant version and Golden QA dataset used along with its duplication factor.
-- Status — Whether the evaluation is Completed, Running, or Failed.
-- Cosine Similarity — A score indicating how semantically close the AI's answers were to the expected answers (more on this below).
-- Completed at — The date and time when the evaluation finished.
-- Actions — A button to download the detailed results.
+
-## Part 1: Running an AI Evaluation
-### Step 1: Click "+ Create AI Evaluation"
-From the AI Evaluations page, click the + Create AI Evaluation button in the top right corner.
-
+## Run the evaluation
+1. On the `Golden Q&A Evaluation` tab,
+2. click `Run evaluation` (or Run another evaluation if the version was evaluated before).
+ 
-You will be taken to the Create AI Evaluation page.
+4. In the dialog, choose the `Golden Q&A` set from the dropdown (the last-used set is marked),
+5. then choose how many times to ask each question: 1x (Quick smoke test) asks each question once — the fastest way to check things broadly work; or 5x (Consistency check) asks each question five times to catch answers that change between attempts.
+6. Click `Run evaluation`.
+7. Evaluations run in the background, so you can keep working and come back — the result appears when it completes.
-
+## Interpreting the results
+Once an evaluation finishes, the `Run` sub-tab shows the latest run for that version (with the Golden set name, duplication factor, and timestamp). A dial shows the OVERALL score out of 5 with a health badge (e.g. “Could improve”) and an AI-written summary highlighting strengths, weaknesses, and the top questions worth checking. Treat the summary as a guide - and use it to find the top 3 things to inspect from the results of the given evaluation run.
-### Step 2: Select or Upload a Golden QA Dataset
-Under the Select Golden QA section, you have two options:
-
+The overall score is a weighted average of three checks, each scored 0–5:
+1. Adherence to ground truth (weight 50%, how well answers match the golden answers),
+2. Adherence to prompt (weight 20%, how well answers follow the instructions), and
+3. Adherence to knowledge base (weight 30%, how well answers stay grounded in the attached documents).
-- Option A — Use an existing dataset: Click the "Search or select a Golden QA dataset" dropdown and choose from your previously uploaded datasets.
-- Option B — Upload a new dataset: Click the "Upload Golden QA" button to upload a new CSV file. Provide the duplication factor for the uploaded data set.
+- Score colours: 0–1 needs improvement (red), 2–3 could improve (amber), 4–5 good (green).
-
+
-Duplication factor is the number of times the golden questions are repeated in the given dataset while running the evaluation. Allowed values 1-5.
+## Question-level results and comparing versions
-Tip: Your CSV must follow the format question, answer with one pair per row. Access the [Golden QA CSV template](https://docs.google.com/spreadsheets/d/198UpOMeU53s9O-fwbIl0DIJLuD3l24jgkq74CoDfSQM/copy) to get started.
+
+
+Below the summary, every question is listed. Two views are available:
+1. Individual Rows (a table with columns for Question, Expected Answer, Assistant Answer, and the three per-question scores) and
+2. Group By Questions (groups each question with its answer(s) and per-metric score chips — especially useful for 5x runs where each question has several answers).
+3. Use these to pinpoint where the assistant is weak,
+4. and `Export CSV` to further analyse each question, generated answer, the rating and reason behing the rating in google sheets or excel.
-### Step 3: Select an AI Assistant
-Click the "Search or select an AI assistant" dropdown under AI Assistant and choose the specific assistant (and its version) you want to evaluate.
+The `History` sub-tab lists all past runs in one comparison table (Version, Golden Q&A, Duplication Factor, Overall, and the per-metric scores), with the same colour coding. This is how you tell whether a prompt or model change actually improved the assistant before you publish it.
-
-
-Note: Each AI Assistant can have multiple versions. Make sure you select the correct version you want to test — this is especially useful when comparing how a newer version performs versus an older one.
-
-### Step 4: Enter an Evaluation Name
-Under Evaluation Details, type a unique, descriptive name for this evaluation run in the Evaluation Name field.
-
-
-
-Tip: Use a name that helps you identify the test later, such as v2-assistant-may-test or knowledge-base-check-q1.
-
-
-### Step 5: Run the Evaluation
-Click the "Run Evaluation" button to start the evaluation.
-
-Glific will now send each question from your Golden QA dataset to the selected AI Assistant and compare the responses against the expected answers. The evaluation will appear in the AI Evaluations list with a "**Completed**" status once it finishes. Time taken to complete the evaluation run depends on the number of golden questions and answers. A good estimation of time range would be 15-30 mins, can even go to 45 mins.
-
-## Part 2: Reviewing Results
-### Viewing Evaluation Results
-Once an evaluation is complete, it appears in the AI Evaluations tab with its status, cosine similarity score, and completion timestamp.
-
-Each evaluation row shows:
-- The evaluation name
-- The AI Assistant that was tested
-- The name of Golden QA file used along with its duplication factor
-- Action to download the results
-
-Click the "Download Results" button on any completed evaluation to download a detailed CSV file with a row-by-row breakdown of each question, the AI's actual response, the expected answer, and the individual similarity score.
-
-
-Open the results CSV in a Google spreadsheet to perform further analysis and interpret the results of the evaluation.
-
-
-
-Through comparing the golden answer (ground_truth_answer) with the generated answers (llm_answer), isolating the rows with lower scores (less than 0.3), you should be able to understand what to change in your assistant (either the prompt or the knowledge base) to get better answers from the AI assistant.
-
-In the results csv "question_id" is referring to the question number from the golden QA list. This means question id of the question in the first row of the Golden QA csv will be 1 and so on.
-
-## Understanding Cosine Similarity
-The Cosine Similarity score tells you how meaningfully similar the AI Assistant's actual answers were to the expected "golden" answers. You can hover over the ⓘ icon next to the column header to see an explanation for what cosine similarity means.
-
-Here's a quick guide to interpreting the score:
-
-| Score | Meaning |
-|-------|---------|
-| > 0.7 | Answers convey similar intent and information as the golden answer |
-| 0.3 – 0.7 | Moderate alignment — some drift in meaning, worth reviewing |
-| < 0.3 | The response has drifted significantly in meaning, even if some words overlap — the assistant may need tuning |
-
-
-- **Analyze answers that are below 0.3** — Cosine similarity can be good starting indicator to weed out answers that are not aligned at all. So starting with answers that are low scoring and figuring out how to improve the scores on these is a great start. Consistently scoring above 0.7 is a good indicator that the AI answers are aligned to your expectations. However, following nuances can be kept in mind:
-- **Low-scoring evaluations don't always mean failure** — review the downloaded results to identify which specific questions scored poorly. You may find patterns that can guide improvements to your assistant's knowledge base or prompt instructions. For some questions, it may be ok to get lower scores ex- your AI assistant is catching edge cases and not answering to harmful or potentially misleading questions.
-- **High-scoring evaluations more than 0.7 don’t always mean correct answers** — review the results to identify if the answers are also complete. Once the majority of the answers are scoring high on cosine similarity more evaluators can be added to help further improve the correctness and completeness of answers. Connect with Glific team to understand how this can be enabled.
-
-## Tips and Best Practices
-- **Run evaluations regularly** — especially after making changes to your AI Assistant's knowledge base, instructions, or model settings. This helps you track whether changes improved or worsened performance.
-- **Compare versions** — use the same Golden QA dataset across different AI Assistant versions to directly compare their performance scores.
-- **Build a strong Golden QA** — the quality of your evaluation depends on the quality of your Golden QA dataset. Include diverse, representative questions that your beneficiaries actually ask.
-
-
-## Golden QA
-The Golden QA tab is part of Glific's AI Evals section, found under the AI toolkit in the left navigation menu. It serves as a library of curated question-and-answer datasets — called "Golden QA" — that are used as the reference benchmarks when evaluating an AI Assistant's performance.
-Each dataset is a CSV file containing a set of questions paired with their ideal (or "golden") answers with a certain duplication factor. Once a dataset has been uploaded, it can be seen here and can be re-used from for multiple eval runs.
-
-## How to Use It
-**Uploading a Golden QA Dataset** Golden QA datasets can be uploaded from the Create AI Evaluation form (accessed via the `+ Create AI Evaluation` button on the AI Evaluations tab). On that form, click Upload Golden QA to upload a new CSV file. A template is available via link on the create form to help you get started quickly.
-
-
-Once uploaded, the dataset will appear in the Golden QA tab and remain available for future evaluations.
-
-**Browsing and Searching Datasets** On the Golden QA tab, all previously uploaded datasets are listed in a table sorted by creation date (newest first). Use the Search bar at the top right to filter datasets by name if you have a large library. You can also click the Created On column header to toggle the sort order.
-
-**Downloading a Dataset** Each row in the table has a download icon (↓) in the Actions column. Clicking it downloads the corresponding CSV file, which is useful for reviewing or auditing the question-answer pairs, or for making edits before re-uploading a revised version.
-
-**Using a Dataset in an Evaluation** When creating an AI Evaluation, use the "Search or select a Golden QA" dropdown to pick an existing dataset from the Golden QA library. Combine it with an AI Assistant selection and an Evaluation Name, then click Run Evaluation. The platform will send each question in the dataset to the chosen AI Assistant, compare the responses to the golden answers, and report a cosine similarity score once the run is complete.
-
-## Tips
-- Use descriptive, version-stamped names for your datasets (e.g., knowledge_goldenqa_v2_20260512) so they are easy to identify and trace over time.
-- Keep your golden answers concise and precise — the cosine similarity metric performs best when the reference answers are well-defined rather than overly verbose.
-- Reuse the same dataset across different evaluation runs to isolate the variable being tested (e.g., a different AI Assistant version or knowledge base), making comparisons meaningful.
+
+## Going live
+When you’re satisfied with a version’s evaluation results and sandbox testing, select that version in the version dropdown and click Go Live (top right). This publishes it as the next major version and makes it the version served to real users in every Glific flow that references this assistant’s ID. The button is disabled when the selected version is already the live one.
+## Recommended workflow
+Edit or create new version → Save (creates a new draft version) → test in Try It Out → Run an evaluation → check History to confirm it beat the previous version → only then Go Live.
From afced7121b61be528f54039d9f6f6b2f73099c70 Mon Sep 17 00:00:00 2001
From: Tejas Mahajan <141305477+mahajantejas@users.noreply.github.com>
Date: Wed, 2 Sep 2026 15:09:56 +0530
Subject: [PATCH 02/10] Revise AI Assistants documentation for clarity and
updates
Updated the document to reflect changes in the AI Assistants creation and modification process, including new versioning details and improved structure.
---
.../AI Assistants (how to create and edit).md | 67 ++++++++++++++
...ting and modifying assistants in Glific.md | 92 -------------------
2 files changed, 67 insertions(+), 92 deletions(-)
create mode 100644 docs/5. Integrations/AI Assistants (how to create and edit).md
delete mode 100644 docs/5. Integrations/Creating and modifying assistants in Glific.md
diff --git a/docs/5. Integrations/AI Assistants (how to create and edit).md b/docs/5. Integrations/AI Assistants (how to create and edit).md
new file mode 100644
index 000000000..bc7512962
--- /dev/null
+++ b/docs/5. Integrations/AI Assistants (how to create and edit).md
@@ -0,0 +1,67 @@
+
+
+
+ | 3 minutes read |
+ Level: Advanced |
+ Last Updated: Sept 2026 |
+
+
+
+
+# Overview
+The AI Assistants page (`AI toolkit` → `AI Assistant`) is where you build, test, evaluate, and publish assistants that get wired into Glific flows via their assistant ID (asst_…). Everything you need lives inside a single assistant — from editing the prompt to running Golden Q&A evaluations and taking a version live.
+
+
+# Key concept: versions, Save, and Go Live
+Two actions drive the whole lifecycle.
+
+1. **Saving** any change creates a new minor version (for example 1.0 → 1.1 → 1.2). These saved versions are drafts you can test and evaluate freely without affecting real users. As the version dropdown states: “Saving creates a minor version. Publishing promotes it to the next major and makes it live.”
+
+2. **Go Live (publishing)** promotes the current version to the next major version (for example 1.2 → 2.0) and marks it LIVE. The live version is the one actually used in your Glific flows wherever that assistant ID is referenced. Until you Go Live, real users stay on the previously published version.
+
+# The assistant editor
+
+When you edit or create an assistant you’ll see a header (assistant name with a pencil to rename, the copyable assistant ID, and the Go Live button), a version selector showing the current version and its LIVE status, and five tabs: Model & Prompt, Knowledge Base, Guardrails, Golden Q&A Evaluation, and Try It Out.
+
+
+
+# Creating and editing an assistant
+1. To create an assistant, go to `AI toolkit` → `AI Assistant`
+2. click `Create New Assistant` (top right).
+3. This opens a blank “Untitled assistant”. Rename it with the pencil icon, then configure it using the tabs below.
+
+## Model & Prompt tab
+This is the core configuration. Enter the assistant’s behaviour in the Instructions (Prompt) box (describe who it is, what it helps with, what it must never do, and what language to reply in). Optionally use Generate with AI (BETA) to draft a prompt.
+
+Below that, pick the Model from the dropdown (e.g. gpt-5.6-luna) and set the Reasoning effort (None, Low, Medium, High, Xhigh, or Max — higher is better quality but slower).
+
+
+
+To learn more about the models and which is suitable for your use case, go [here](https://developers.openai.com/api/docs/models/compare)
+
+## Knowledge Base tab
+1. Click `+ Add files` to attach documents the assistant can search when answering (supports PDF, DOC, DOCX, TXT, MD, HTML, and CSV, up to 20MB per file). A Knowledge Base ID is also shown here. Changes apply when you save a version.
+2. Files can be removed individually or downloaded.
+
+
+
+## Try It Out tab
+A sandbox chat to test the currently selected version before going live with it to quickly test how the assistant answers are changing. It labels which version you’re testing and notes it’s sandbox-only, so real users are unaffected.
+
+
+
+### Main points to note about assistant id and Go live action
+1. The assistant id being used in the flow does not need to be modified,
+2. You can choose which version of the assistant gets used in the flow by using the "Go Live" button
+3. By default, new changes are not used directly in the flow.
+4. To use an assistant with the latest modifications, you must explicitly click "Set as LIVE".
+
+## Recommended workflow
+Edit or create new version → Save (creates a new draft version) → test in Try It Out → Run an evaluation → check History to confirm it beat the previous version → only then Go Live.
+
+### Why are versions created on modifications? and why do you need to "Go Live"
+1. To provide flexibility to easily revert to previous versions when making new modifications to assistants.
+2. Any new updates can be evaluated before deploying to flows which are in use.
+
+### How to use the assistants in the Glific flow?
+The workflow on using the assistants remains the same, can be seen in [Using the OpenAI assistant in the flow editor](https://glific.github.io/docs/docs/Integrations/Filesearch%20Using%20OpenAI%20Assistants/#using-the-openai-assistant-in-floweditor)
diff --git a/docs/5. Integrations/Creating and modifying assistants in Glific.md b/docs/5. Integrations/Creating and modifying assistants in Glific.md
deleted file mode 100644
index 4dace570c..000000000
--- a/docs/5. Integrations/Creating and modifying assistants in Glific.md
+++ /dev/null
@@ -1,92 +0,0 @@
-
-
-
- | 3 minutes read |
- Level: Advanced |
- Last Updated: July 2026 |
-
-
-
-
-# Creating and modifying AI assistants in Glific
-This doc details how to create new assistants, modify the created assistants and how to use these in your Glific flows.
-
-## Creating a new assistant
-1. From the assistant list page, click on “Create new assistant”. This opens a blank creation form - no assistant is created yet at this point.
-
-
-
-2. Fill out the details needed,
-- Name of the assistant
-- Select a model
-- Add your prompt
-- Add files to create a knowledge base
-- Set the temperature
-
-Note: The model dropdown shows a fixed list of supported models (currently `gpt-4o`, `gpt-4o-mini`, `gpt-4.1`, `gpt-4.1-mini`), rather than fetching every model available on the provider's account. If you need a model that isn't listed, reach out to the Glific team.
-
-
-
-3. Click on save. This is the point the assistant is actually created - nothing is saved to Glific until you click save here.
-4. This navigates to the assistant versions listing page.
-
-
-5. From here, copy the assistant id and use it in the flow.
-
-## Assistant status
-Each assistant in the list page shows a status chip, so you can tell at a glance whether it's ready to use:
-- Ready - the assistant is live and usable in flows.
-- In Progress - a new version is being prepared (for example, right after you save changes or add files). Wait for this to finish before using the assistant.
-- Failed - the last update didn't complete successfully. The assistant continues to run on its previous live version; check the assistant for details and try saving your changes again.
-
-## Modifying an assistant
-1. From the assistant list page, click on “edit” the assistant action.
-
-
-2. This takes to the assistant version listing page
-3. Make the required changes to the assistant
-4. Changing the model, changing the prompt, adding or removing files from the knowledge base, or changing the temperature of the assistant leads to showing “unsaved changes”
-
-### Uploading files to the knowledge base
-When you add files to an assistant's knowledge base:
-- Each file uploads on its own and shows its own progress - a clock icon while it's queued, a spinner while uploading, a green check once attached, or an error icon if it failed.
-- Hovering over a failed file's error icon shows the reason it failed.
-- Failed files can be retried individually, using the retry button next to that file, without having to re-select and re-upload every other file.
-- You can't save your changes while any file is still in a failed state - remove it or retry it first.
-- Files larger than 20 MB are rejected with a warning. If you select multiple files at once and any of them is over the limit, none of them are uploaded - remove the oversized file(s) and try again.
-
-
-
-5. Click on “save” to save the modifications done.
-6. This creates a new version of the assistant.
-
-
-7. To use the version with the latest modifications in the flow, simply click on “set as LIVE” to make sure that the given version is getting used in the flow.
-
-
-
-
-## Cloning an assistant
-A `Clone Assistant` button is available when editing an assistant, but only for legacy assistants with a knowledge base created before February 10, 2026. If you don't see the button, your assistant either isn't eligible or has already been cloned.
-
-1. Open the assistant you want to clone (from the assistant list page, click "edit" on its action).
-2. Click `Clone Assistant`.
-3. A confirmation dialog, "Cloning May Affect Responses," explains that a cloned assistant may behave differently from the original, and recommends reviewing responses or running evaluations after cloning. Click `Proceed` to continue, or cancel to back out.
-4. While cloning runs, the button shows `Cloning` with a spinner. This can take a little while, since the assistant's files are downloaded from OpenAI and re-uploaded to the new assistant.
-5. Once done, you'll see a success notification and the button becomes disabled with an "Already cloned" tooltip - each eligible assistant can only be cloned once.
-6. If cloning fails, the button changes to `Retry cloning` so you can try again without repeating the earlier steps.
-
-Cloning creates a replica of the assistant's presently live version, including the prompt, temperature, and knowledge base, so you can modify the knowledge base on the new copy - something that isn't possible directly on these older, legacy assistants.
-
-
-### Main points to note about assistant id and set as live action
-1. The assistant id being used in the flow does not need to be modified,
-2. You can choose which version of the assistant gets used in the flow by using the "Set as LIVE" button
-3. By default, new changes are not used directly in the flow. To use an assistant with the latest modifications, you must explicitly click "Set as LIVE".
-
-### Why are versions created on modifications? and why do you need to "set as LIVE"
-1. To provide flexibility to easily revert to previous versions when making new modifications.
-2. Any new updates can be evaluated (evaluations feature coming soon in Glific) before deploying to live assistants.
-
-### How to use the assistants in the Glific flow?
-The workflow on using the assistants remains the same, can be seen in [Using the OpenAI assistant in the flow editor](https://glific.github.io/docs/docs/Integrations/Filesearch%20Using%20OpenAI%20Assistants/#using-the-openai-assistant-in-floweditor)
From 3e8061d9befb8f0ff1aa1234cc3806246b760a22 Mon Sep 17 00:00:00 2001
From: Tejas Mahajan <141305477+mahajantejas@users.noreply.github.com>
Date: Wed, 2 Sep 2026 15:10:51 +0530
Subject: [PATCH 03/10] Add AI Assistants in Glific Flows documentation
---
...sing OpenAI Assistants.md => AI Assistants in Glific Flows.md} | 0
1 file changed, 0 insertions(+), 0 deletions(-)
rename docs/5. Integrations/{Filesearch Using OpenAI Assistants.md => AI Assistants in Glific Flows.md} (100%)
diff --git a/docs/5. Integrations/Filesearch Using OpenAI Assistants.md b/docs/5. Integrations/AI Assistants in Glific Flows.md
similarity index 100%
rename from docs/5. Integrations/Filesearch Using OpenAI Assistants.md
rename to docs/5. Integrations/AI Assistants in Glific Flows.md
From da6bae2feea10f642e046f3366a2bb976ed868d6 Mon Sep 17 00:00:00 2001
From: Tejas Mahajan <141305477+mahajantejas@users.noreply.github.com>
Date: Wed, 2 Sep 2026 15:12:53 +0530
Subject: [PATCH 04/10] Fix image references in AI Assistants documentation
Updated images in the documentation for AI Assistants integration steps, ensuring correct display and context for users.
---
.../AI Assistants in Glific Flows.md | 36 +++++++++----------
1 file changed, 18 insertions(+), 18 deletions(-)
diff --git a/docs/5. Integrations/AI Assistants in Glific Flows.md b/docs/5. Integrations/AI Assistants in Glific Flows.md
index 2509b99b9..b753dd4a7 100644
--- a/docs/5. Integrations/AI Assistants in Glific Flows.md
+++ b/docs/5. Integrations/AI Assistants in Glific Flows.md
@@ -50,39 +50,39 @@ This section explains how to:
_Screenshot of example flow set up is given below_
-
+
#### Step 2: Add a Call Webhook node. This is where we integrate the OpenAI Assistant.
- By default, `Function` would be selected. Leave this as it is.
-
+
- In the `Function` field, select the pre-defined function name `filesearch-gpt`, from the dropdown.
-
+
- Give the webhook `Result Name` - you can use any name. In the screenshot example, it’s named as `gptresponse`.
-
+
#### Step 3: Click on Function Body (top right corner). You would see the following.
-
+
- In `question` parameter enter the flow variable containing the question asked by the user. In the given example `question` is the `result name`, hence provided `@result.question` in the question parameter.
- In `assistant_id` enter the assistant ID, copied as described in [Creating and modifying AI assistants in Glific](https://glific.github.io/docs/docs/Integrations/Creating%20and%20modifying%20assistants%20in%20Glific/)
- 
+
#### Step 4: Display the Assistant's response
- Once the Webhook is updated, add a `Send Message` node and enter `@results.gptresponse.message` variable to receive the AI response.
- In the given example `gptresponse` is the `result name` (refer to Step 2). If `ai_response` was the result name, the variable would be `@results.ai_response.message`.
- 
+
[Sample Flow](https://drive.google.com/file/d/10wGUil1l2SGfOZdWQQOO935OK-6BdZV1/view?usp=sharing) Click on the Sample Flow link to import it and explore how it works.
@@ -94,11 +94,11 @@ When a user asks a follow-up question, the assistant uses thread ID to remember
- This parameter should be set to the value `@results.previouswebhookname.thread_id`.
- In the example shown, the previous webhook result name is gptresponse. So the thread ID should be referenced as - `@results.gptresponse.thread_id`.
-
+
- In question parameter enter the flow variable containing the follow up question asked by the user. In the given example `result_5` is the result name, hence provided `@results.result_5` in the question parameter.
-
+
---
@@ -112,31 +112,31 @@ This section explains how to use the `voice-filesearch-gpt` webhook function in
- Create a `Send Message` node directing users to send their responses as audio messages, based on their preference.
- In the `Wait for Response` node, select has audio as the message response type. Also, give a result name. In the screenshot below, `audio_query` is used as the result name.
-
+
#### Step 2: Create Call a Webhook node
- By default,`Function` would be selected. Leave this as it is.
-
+
- In the `Function` field, select the pre-defined function name `voice-filesearch-gpt` , from the dropdown.
-
+
- Give the webhook result name - you can use any name. In the screenshot example, it’s named `gpt_voice`.
-
+
-
+
#### Step 3: Click on Function Body (top right corner). You would see the following.
-
+
Pass the following paramters in the function body.
-
+
- For `contact ` keep this value as `@contact` as mentioned in the screenshot.
- `speech` is the result name which is storing the voice note sent by the user.
@@ -149,7 +149,7 @@ Pass the following paramters in the function body.
- Use `@results.webhook_result-name.translated_text` to show the text response.
- In the given example `gpt_voice` is the webhook result name.
-
+
#### Step 5: Send the voice note response
@@ -158,7 +158,7 @@ Pass the following paramters in the function body.
- Use `@results.gpt_voice.media_url` (gpt_voice is the result name of webhook node)
-
+
[Sample Flow](https://drive.google.com/file/d/1EOXFnUqUPQz51xf3pYzhikkf_2VqXL4Z/view?usp=sharing) Click on the Sample Flow link to import it and explore how it works.
From 369d03f6bb5c38ab9ea37f8705ea15c8b190c920 Mon Sep 17 00:00:00 2001
From: Tejas Mahajan <141305477+mahajantejas@users.noreply.github.com>
Date: Thu, 3 Sep 2026 08:11:37 +0530
Subject: [PATCH 05/10] Update AI Assistants documentation for clarity
Clarified instructions and added images for better understanding of the AI Assistant creation and editing process.
---
.../AI Assistants (how to create and edit).md | 27 ++++++++++++-------
1 file changed, 18 insertions(+), 9 deletions(-)
diff --git a/docs/5. Integrations/AI Assistants (how to create and edit).md b/docs/5. Integrations/AI Assistants (how to create and edit).md
index bc7512962..07124f39d 100644
--- a/docs/5. Integrations/AI Assistants (how to create and edit).md
+++ b/docs/5. Integrations/AI Assistants (how to create and edit).md
@@ -9,30 +9,39 @@
# Overview
-The AI Assistants page (`AI toolkit` → `AI Assistant`) is where you build, test, evaluate, and publish assistants that get wired into Glific flows via their assistant ID (asst_…). Everything you need lives inside a single assistant — from editing the prompt to running Golden Q&A evaluations and taking a version live.
+The AI Assistants page (`AI toolkit` → `AI Assistant`) is where you build, test, evaluate, and publish assistants that get used into Glific flows via their assistant ID (asst_…). Everything you need lives inside a single assistant — from editing the prompt to evaluating it against Golden Q&As and taking a version live.
# Key concept: versions, Save, and Go Live
Two actions drive the whole lifecycle.
-1. **Saving** any change creates a new minor version (for example 1.0 → 1.1 → 1.2). These saved versions are drafts you can test and evaluate freely without affecting real users. As the version dropdown states: “Saving creates a minor version. Publishing promotes it to the next major and makes it live.”
+1. **Saving** any change creates a new minor version (for example 1.0 → 1.1 → 1.2). These saved versions are drafts you can test and evaluate freely without affecting real users. Any changes made need to be saved, so that the updates can be tried out and evaluated before making live in the flow for the actual users.
+
+ 
+
+3. **Go Live (publishing)** promotes the current version to the next major version (for example 1.2 → 2.0) and marks it LIVE. The live version is the one actually used in your Glific flows wherever that assistant ID is referenced. Until you Go Live, real users stay on the previously published version.
+
+
-2. **Go Live (publishing)** promotes the current version to the next major version (for example 1.2 → 2.0) and marks it LIVE. The live version is the one actually used in your Glific flows wherever that assistant ID is referenced. Until you Go Live, real users stay on the previously published version.
# The assistant editor
When you edit or create an assistant you’ll see a header (assistant name with a pencil to rename, the copyable assistant ID, and the Go Live button), a version selector showing the current version and its LIVE status, and five tabs: Model & Prompt, Knowledge Base, Guardrails, Golden Q&A Evaluation, and Try It Out.
-
+
+
# Creating and editing an assistant
1. To create an assistant, go to `AI toolkit` → `AI Assistant`
2. click `Create New Assistant` (top right).
3. This opens a blank “Untitled assistant”. Rename it with the pencil icon, then configure it using the tabs below.
+
## Model & Prompt tab
This is the core configuration. Enter the assistant’s behaviour in the Instructions (Prompt) box (describe who it is, what it helps with, what it must never do, and what language to reply in). Optionally use Generate with AI (BETA) to draft a prompt.
+To learn more about how to build a good prompt go [here](https://glific.github.io/docs/docs/FAQ/AI%20Chatbot%20Implementation%20Guidelines/#2-prompt)
+
Below that, pick the Model from the dropdown (e.g. gpt-5.6-luna) and set the Reasoning effort (None, Low, Medium, High, Xhigh, or Max — higher is better quality but slower).
@@ -48,16 +57,16 @@ To learn more about the models and which is suitable for your use case, go [here
## Try It Out tab
A sandbox chat to test the currently selected version before going live with it to quickly test how the assistant answers are changing. It labels which version you’re testing and notes it’s sandbox-only, so real users are unaffected.
-
+
### Main points to note about assistant id and Go live action
-1. The assistant id being used in the flow does not need to be modified,
-2. You can choose which version of the assistant gets used in the flow by using the "Go Live" button
+1. Making edits to an assistant does not change its assistant ID. Therefore, the assistant ID does not need to be updated each time a new version is published.
+2. You can choose which version of the assistant gets used in the flow by using the `Go Live` button on that particular assistant version
3. By default, new changes are not used directly in the flow.
-4. To use an assistant with the latest modifications, you must explicitly click "Set as LIVE".
+4. To use an assistant with the latest modifications, you must explicitly click `Go Live`.
## Recommended workflow
-Edit or create new version → Save (creates a new draft version) → test in Try It Out → Run an evaluation → check History to confirm it beat the previous version → only then Go Live.
+Edit or create new version of an assistant → Save (creates a new draft version of the assistant) → test in `Try It Out` → Run an evaluation → check `History` to compare scores with previous versions → only then Go Live.
### Why are versions created on modifications? and why do you need to "Go Live"
1. To provide flexibility to easily revert to previous versions when making new modifications to assistants.
From 072c98b58f060414cada9cacbb62c1706ceb5d24 Mon Sep 17 00:00:00 2001
From: Tejas Mahajan <141305477+mahajantejas@users.noreply.github.com>
Date: Thu, 3 Sep 2026 08:12:50 +0530
Subject: [PATCH 06/10] Update AI Assistants in Glific Flows.md
---
docs/5. Integrations/AI Assistants in Glific Flows.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docs/5. Integrations/AI Assistants in Glific Flows.md b/docs/5. Integrations/AI Assistants in Glific Flows.md
index b753dd4a7..15c8fe45a 100644
--- a/docs/5. Integrations/AI Assistants in Glific Flows.md
+++ b/docs/5. Integrations/AI Assistants in Glific Flows.md
@@ -4,7 +4,7 @@
| 6 minutes read |
Level: Advanced |
- Last Updated: July 2026 |
+ Last Updated: Sept 2026 |
From 740dc3f949c3ef4e9c1555c690bf0a983b4139ad Mon Sep 17 00:00:00 2001
From: Tejas Mahajan <141305477+mahajantejas@users.noreply.github.com>
Date: Thu, 3 Sep 2026 08:33:34 +0530
Subject: [PATCH 07/10] Improve clarity and structure of AI evaluation
documentation
Refactor evaluation metrics section for clarity and consistency. Update instructions for setting up Golden Q&A sets and running evaluations.
---
.../AI Evaluations in Glific.md | 29 +++++++++++++------
1 file changed, 20 insertions(+), 9 deletions(-)
diff --git a/docs/5. Integrations/AI Evaluations in Glific.md b/docs/5. Integrations/AI Evaluations in Glific.md
index d8a116970..367206386 100644
--- a/docs/5. Integrations/AI Evaluations in Glific.md
+++ b/docs/5. Integrations/AI Evaluations in Glific.md
@@ -11,34 +11,45 @@
# Running AI evaluations
Evaluations can be run through on by navigating to `Golden Q&A Evaluation` tab within the AI Assistant page. AI Evaluations score a version against a Golden Q&A — a fixed set of questions paired with ideal answers. The AI evaluations return 3 scores, a reason for the score, and an overall summary and next steps based on all the scores from the evaluation run.
-The evaluation scores answer on three independent metrics: **Adherence to Ground Truth** (is AI generated answer correct in meaning against the golden answer?), **Adherence to Knowledge Base** (is every claim in AI generated answer grounded in the information retrieved from knowledge base, or hallucinated?), and **Adherence to Prompt** (does AI generated answer follow the configured language, tone, and answer-vs-refuse behaviour?), each returned as a score between 0 to 5, plus a plain-language reasoning.
+
-The result is that a wrong-but-similar sounding answer can be caught by adherence to ground truth , hallucinations get flagged with the specific unsupported claim named by adherence to knowledge base, and all results (scores, reasoning) are stored to enable comparison between versions.
-## Set up a Golden Q&A set (once)
+The evaluation scores answer on three independent metrics:
+1. **Adherence to Ground Truth**: is AI generated answer correct in meaning against the golden answer?,
+2. **Adherence to Knowledge Base**: is every claim in AI generated answer grounded in the information retrieved from knowledge base, or hallucinated?, and
+3. **Adherence to Prompt**: does AI generated answer follow the configured language, tone, and answer-vs-refuse behaviour?,
+
+- each of these metrics is a score between 0 to 5, plus a plain-language reasoning by the judge.
+
+The result is that a wrong-but-similar sounding answer can be caught by adherence to ground truth , hallucinations get flagged with the specific unsupported claim named by adherence to knowledge base, and all results (scores, reasoning) are stored and accessible in the assistant's history of runs to enable comparison between versions.
+
+## Set up a Golden Q&A set
+
1. On the `Golden Q&A Evaluation` tab,
-2. click `Manage Golden Q&A`.
+2. Click `Manage Golden Q&A`.
4. You’ll see existing sets, if you have run evaluations in past; else,
5. You can use `Add Golden Q&A` button. To create one,
-7. click Add Golden Q&A, give it a name, and upload a CSV with two columns: question and answer.
+7. Click Add Golden Q&A, give it a name, and upload a CSV with two columns: question and answer.
8. There’s a [Google Sheet template link](https://docs.google.com/spreadsheets/d/198UpOMeU53s9O-fwbIl0DIJLuD3l24jgkq74CoDfSQM/copy) to help format the file.
9. Click `Add` to save it.
+To learn more on how to create good set of golden Q&As, go [here](https://glific.github.io/docs/docs/FAQ/About%20Golden%20QnA)
+
## Run the evaluation
1. On the `Golden Q&A Evaluation` tab,
-2. click `Run evaluation` (or Run another evaluation if the version was evaluated before).
+2. Click `Run evaluation` (or Run another evaluation if the version was evaluated before).
4. In the dialog, choose the `Golden Q&A` set from the dropdown (the last-used set is marked),
-5. then choose how many times to ask each question: 1x (Quick smoke test) asks each question once — the fastest way to check things broadly work; or 5x (Consistency check) asks each question five times to catch answers that change between attempts.
+5. Then choose how many times to send each question: 1x (Quick smoke test) asks each question once — the fastest way to check things broadly work; or 5x (Consistency check) asks each question five times to catch answers that change between attempts.
6. Click `Run evaluation`.
-7. Evaluations run in the background, so you can keep working and come back — the result appears when it completes.
+7. Evaluations run in the background, and can take upto 10-15 mins depending on the number of questions and duplication factor used. So you can keep working and come back — the result appears when it completes.
## Interpreting the results
Once an evaluation finishes, the `Run` sub-tab shows the latest run for that version (with the Golden set name, duplication factor, and timestamp). A dial shows the OVERALL score out of 5 with a health badge (e.g. “Could improve”) and an AI-written summary highlighting strengths, weaknesses, and the top questions worth checking. Treat the summary as a guide - and use it to find the top 3 things to inspect from the results of the given evaluation run.
@@ -60,7 +71,7 @@ Below the summary, every question is listed. Two views are available:
1. Individual Rows (a table with columns for Question, Expected Answer, Assistant Answer, and the three per-question scores) and
2. Group By Questions (groups each question with its answer(s) and per-metric score chips — especially useful for 5x runs where each question has several answers).
3. Use these to pinpoint where the assistant is weak,
-4. and `Export CSV` to further analyse each question, generated answer, the rating and reason behing the rating in google sheets or excel.
+4. And `Export CSV` to further analyse each question, generated answer, the rating and reason behind the rating in google sheets.
The `History` sub-tab lists all past runs in one comparison table (Version, Golden Q&A, Duplication Factor, Overall, and the per-metric scores), with the same colour coding. This is how you tell whether a prompt or model change actually improved the assistant before you publish it.
From c4be4d08207986003a94745a20c10efd331aa786 Mon Sep 17 00:00:00 2001
From: Tejas Mahajan <141305477+mahajantejas@users.noreply.github.com>
Date: Thu, 3 Sep 2026 08:46:25 +0530
Subject: [PATCH 08/10] Revise assistant creation and editing instructions
---
.../AI Assistants (how to create and edit).md | 42 +++++++++++--------
1 file changed, 24 insertions(+), 18 deletions(-)
diff --git a/docs/5. Integrations/AI Assistants (how to create and edit).md b/docs/5. Integrations/AI Assistants (how to create and edit).md
index 07124f39d..494bacf90 100644
--- a/docs/5. Integrations/AI Assistants (how to create and edit).md
+++ b/docs/5. Integrations/AI Assistants (how to create and edit).md
@@ -12,16 +12,11 @@
The AI Assistants page (`AI toolkit` → `AI Assistant`) is where you build, test, evaluate, and publish assistants that get used into Glific flows via their assistant ID (asst_…). Everything you need lives inside a single assistant — from editing the prompt to evaluating it against Golden Q&As and taking a version live.
-# Key concept: versions, Save, and Go Live
-Two actions drive the whole lifecycle.
-
-1. **Saving** any change creates a new minor version (for example 1.0 → 1.1 → 1.2). These saved versions are drafts you can test and evaluate freely without affecting real users. Any changes made need to be saved, so that the updates can be tried out and evaluated before making live in the flow for the actual users.
-
-
-
-3. **Go Live (publishing)** promotes the current version to the next major version (for example 1.2 → 2.0) and marks it LIVE. The live version is the one actually used in your Glific flows wherever that assistant ID is referenced. Until you Go Live, real users stay on the previously published version.
-
-
+# Creating and editing an assistant
+1. To create an assistant, go to `AI toolkit` → `AI Assistant`
+2. click `Create New Assistant` (top right).
+3. This opens a blank “Untitled assistant”. Rename it with the pencil icon, then configure it using the tabs below.
+
# The assistant editor
@@ -31,22 +26,22 @@ When you edit or create an assistant you’ll see a header (assistant name with
-# Creating and editing an assistant
-1. To create an assistant, go to `AI toolkit` → `AI Assistant`
-2. click `Create New Assistant` (top right).
-3. This opens a blank “Untitled assistant”. Rename it with the pencil icon, then configure it using the tabs below.
-
## Model & Prompt tab
This is the core configuration. Enter the assistant’s behaviour in the Instructions (Prompt) box (describe who it is, what it helps with, what it must never do, and what language to reply in). Optionally use Generate with AI (BETA) to draft a prompt.
To learn more about how to build a good prompt go [here](https://glific.github.io/docs/docs/FAQ/AI%20Chatbot%20Implementation%20Guidelines/#2-prompt)
-Below that, pick the Model from the dropdown (e.g. gpt-5.6-luna) and set the Reasoning effort (None, Low, Medium, High, Xhigh, or Max — higher is better quality but slower).
+Below that, pick the Model from the dropdown (e.g. gpt-5.6-luna) and set the associated settings for the model.
-
+Examples of settings associated with the model:
+1. Reasoning effort: the effort spent by AI models to think and reason before answering the user query. Higher reasoning effort takes up more in time in generating the response.
+2. Temperature: Temperature in AI models is a setting that acts as a dial for creativity and randomness. Higher temperature leads to more creative responses, recommended to keep temperature close to 0 for most social impact use cases.
-To learn more about the models and which is suitable for your use case, go [here](https://developers.openai.com/api/docs/models/compare)
+
+
+
+Model settings are dependent on the model. To learn more about the models, settings and which is suitable for your use case, go [here](https://developers.openai.com/api/docs/models/compare)
## Knowledge Base tab
1. Click `+ Add files` to attach documents the assistant can search when answering (supports PDF, DOC, DOCX, TXT, MD, HTML, and CSV, up to 20MB per file). A Knowledge Base ID is also shown here. Changes apply when you save a version.
@@ -54,11 +49,22 @@ To learn more about the models and which is suitable for your use case, go [here
+To learn more about how to structure the knowledge base go [here](https://glific.github.io/docs/docs/FAQ/AI%20Chatbot%20Implementation%20Guidelines#1-knowledge-base)
## Try It Out tab
A sandbox chat to test the currently selected version before going live with it to quickly test how the assistant answers are changing. It labels which version you’re testing and notes it’s sandbox-only, so real users are unaffected.
+# Key concept: versions, Save, and Go Live
+Two actions drive the whole lifecycle.
+
+1. **Saving** any change creates a new minor version (for example 1.0 → 1.1 → 1.2). These saved versions are drafts you can test and evaluate freely without affecting real users. Any changes made need to be saved, so that the updates can be tried out and evaluated before making live in the flow for the actual users.
+
+
+
+3. **Go Live (publishing)** promotes the current version to the next major version (for example 1.2 → 2.0) and marks it LIVE. The live version is the one actually used in your Glific flows wherever that assistant ID is referenced. Until you Go Live, real users stay on the previously published version.
+
+
### Main points to note about assistant id and Go live action
1. Making edits to an assistant does not change its assistant ID. Therefore, the assistant ID does not need to be updated each time a new version is published.
2. You can choose which version of the assistant gets used in the flow by using the `Go Live` button on that particular assistant version
From 4d73a86d945e3422f7b1f6ad78e1ce4514df097f Mon Sep 17 00:00:00 2001
From: Tejas Mahajan <141305477+mahajantejas@users.noreply.github.com>
Date: Fri, 4 Sep 2026 11:23:34 +0530
Subject: [PATCH 09/10] Fix typos and improve clarity in AI Assistant guide
Corrected capitalization and phrasing for clarity in the AI Assistant documentation.
---
.../AI Assistants (how to create and edit).md | 20 +++++++++++--------
1 file changed, 12 insertions(+), 8 deletions(-)
diff --git a/docs/5. Integrations/AI Assistants (how to create and edit).md b/docs/5. Integrations/AI Assistants (how to create and edit).md
index 494bacf90..017c23473 100644
--- a/docs/5. Integrations/AI Assistants (how to create and edit).md
+++ b/docs/5. Integrations/AI Assistants (how to create and edit).md
@@ -14,7 +14,7 @@ The AI Assistants page (`AI toolkit` → `AI Assistant`) is where you build, tes
# Creating and editing an assistant
1. To create an assistant, go to `AI toolkit` → `AI Assistant`
-2. click `Create New Assistant` (top right).
+2. Click `Create New Assistant` (top right).
3. This opens a blank “Untitled assistant”. Rename it with the pencil icon, then configure it using the tabs below.
@@ -41,19 +41,20 @@ Examples of settings associated with the model:
-Model settings are dependent on the model. To learn more about the models, settings and which is suitable for your use case, go [here](https://developers.openai.com/api/docs/models/compare)
+Available model settings vary by model. To learn more about the models, settings and which is suitable for your use case, go [here](https://developers.openai.com/api/docs/models/compare)
## Knowledge Base tab
1. Click `+ Add files` to attach documents the assistant can search when answering (supports PDF, DOC, DOCX, TXT, MD, HTML, and CSV, up to 20MB per file). A Knowledge Base ID is also shown here. Changes apply when you save a version.
2. Files can be removed individually or downloaded.
-
+
+
To learn more about how to structure the knowledge base go [here](https://glific.github.io/docs/docs/FAQ/AI%20Chatbot%20Implementation%20Guidelines#1-knowledge-base)
## Try It Out tab
A sandbox chat to test the currently selected version before going live with it to quickly test how the assistant answers are changing. It labels which version you’re testing and notes it’s sandbox-only, so real users are unaffected.
+
-
# Key concept: versions, Save, and Go Live
Two actions drive the whole lifecycle.
@@ -65,11 +66,14 @@ Two actions drive the whole lifecycle.
3. **Go Live (publishing)** promotes the current version to the next major version (for example 1.2 → 2.0) and marks it LIVE. The live version is the one actually used in your Glific flows wherever that assistant ID is referenced. Until you Go Live, real users stay on the previously published version.
+
### Main points to note about assistant id and Go live action
-1. Making edits to an assistant does not change its assistant ID. Therefore, the assistant ID does not need to be updated each time a new version is published.
-2. You can choose which version of the assistant gets used in the flow by using the `Go Live` button on that particular assistant version
-3. By default, new changes are not used directly in the flow.
-4. To use an assistant with the latest modifications, you must explicitly click `Go Live`.
+
+1. Making edits to an assistant and saving a new version does not change its assistant ID.
+2. Therefore, the assistant ID does not need to be updated in the flows each time a new version is saved or made live.
+3. You can choose which version of the assistant gets used in the flow by using the `Go Live` button on that particular assistant version.
+4. By default, new changes are not made live directly in the flow.
+5. To use an assistant with the latest modifications, you must explicitly click `Go Live`.
## Recommended workflow
Edit or create new version of an assistant → Save (creates a new draft version of the assistant) → test in `Try It Out` → Run an evaluation → check `History` to compare scores with previous versions → only then Go Live.
From d4eabf9a5d58089a0c507cc474dc187d4d26d60f Mon Sep 17 00:00:00 2001
From: Tejas Mahajan <141305477+mahajantejas@users.noreply.github.com>
Date: Fri, 4 Sep 2026 11:28:35 +0530
Subject: [PATCH 10/10] Clarify evaluation metrics and scoring description
Updated the wording for clarity and consistency in the evaluation metrics and scoring description.
---
docs/5. Integrations/AI Evaluations in Glific.md | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/docs/5. Integrations/AI Evaluations in Glific.md b/docs/5. Integrations/AI Evaluations in Glific.md
index 367206386..796425a74 100644
--- a/docs/5. Integrations/AI Evaluations in Glific.md
+++ b/docs/5. Integrations/AI Evaluations in Glific.md
@@ -15,9 +15,9 @@ Evaluations can be run through on by navigating to `Golden Q&A Evaluation` tab w
The evaluation scores answer on three independent metrics:
-1. **Adherence to Ground Truth**: is AI generated answer correct in meaning against the golden answer?,
-2. **Adherence to Knowledge Base**: is every claim in AI generated answer grounded in the information retrieved from knowledge base, or hallucinated?, and
-3. **Adherence to Prompt**: does AI generated answer follow the configured language, tone, and answer-vs-refuse behaviour?,
+1. **Adherence to Ground Truth**: is AI generated answer correct in meaning against the golden answer?
+2. **Adherence to Knowledge Base**: is every claim in AI generated answer grounded in the information retrieved from knowledge base, or hallucinated?
+3. **Adherence to Prompt**: does AI generated answer follow the configured language, tone, and answer-vs-refuse behaviour?
- each of these metrics is a score between 0 to 5, plus a plain-language reasoning by the judge.
@@ -55,7 +55,8 @@ To learn more on how to create good set of golden Q&As, go [here](https://glific
Once an evaluation finishes, the `Run` sub-tab shows the latest run for that version (with the Golden set name, duplication factor, and timestamp). A dial shows the OVERALL score out of 5 with a health badge (e.g. “Could improve”) and an AI-written summary highlighting strengths, weaknesses, and the top questions worth checking. Treat the summary as a guide - and use it to find the top 3 things to inspect from the results of the given evaluation run.
The overall score is a weighted average of three checks, each scored 0–5:
-1. Adherence to ground truth (weight 50%, how well answers match the golden answers),
+
+1. Adherence to ground truth (weight 50%, how closely answers matches the facts shared in the golden answers),
2. Adherence to prompt (weight 20%, how well answers follow the instructions), and
3. Adherence to knowledge base (weight 30%, how well answers stay grounded in the attached documents).