Evaluation Dataset Generation

Step 1 - Chunking, VectorDB

Get bunch of chunks in a list format with -> [{chunk1}, {chunk2}, ...] - each chunk is a dictionary with the following keys: - 'id': chunk id - 'doc': chunk data - 'metadata': chunk size

Step 2 - Question Generation

Select N random chunks from the list of chunks Pass chunks into LLM and ask for a question that matches these chunks

Step 3 - Question Evolution

Take questions generated by LLM and send them to LLM to evolve it

Step 4 - Answer Generation

Get N chunks from the list of chunks based on question Pass chunks + question into LLM and ask for an answer that matches these chunks