LlamaIndex Quickstart: Build a RAG Pipeline in Python

A working LlamaIndex retrieval pipeline is about five lines of Python. The interesting part is everything those five lines hide: which package you installed, where the embeddings went, what happens on the second run, and why the answer is wrong when the source document plainly contains it.

This walkthrough follows the framework’s own starter example, then covers the three decisions that turn a demo into something you can run twice.

Before you install anything

LlamaIndex describes itself as a framework for building LLM-powered agents over your data, and its own documentation breaks a RAG application into five stages: loading, indexing, storing, querying, and evaluation. Keep the first four in mind while you build, because every error you hit belongs to exactly one of them, and knowing which stage failed is most of the debugging.

You need three things to start:

• Python and a virtual environment.
• An API key for whatever model you intend to use. The starter example reads OPENAI_API_KEY from the environment.
• A folder of documents. Anything the default reader can parse: text, Markdown, PDF, Word.

Install: one package or many

Since version 0.10 the project is not a single library. The ecosystem is a set of namespaced packages: llama-index-core holds the framework, and every model provider, vector store, and reader ships separately as llama-index-llms-openai, llama-index-vector-stores-qdrant, and so on. The plain llama-index name is a starter bundle, documented as core plus llama-index-llms-openai, llama-index-embeddings-openai and llama-index-readers-file, which is why a beginner tutorial works after one install and a production service usually pins core plus only the integrations it actually imports.

pip install llama-index
export OPENAI_API_KEY=sk-...

Install the bundle to learn the shape of the API. Move to llama-index-core plus explicit integrations when you care about what is in your image.

The five lines

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
response = query_engine.query("What does this corpus say about X?")
print(response)

That is the whole loop, and it maps one-to-one onto the documented ingestion, indexing and retrieval stages.

SimpleDirectoryReader is the loading stage. It walks the folder, picks a parser per file extension, and returns Document objects. Print the length of documents and inspect the extracted text before indexing. For scanned PDFs, check whether the selected parser extracted usable text. Diagnose missing documents separately from retrieved passages that do not answer the question.

VectorStoreIndex.from_documents does the indexing stage, and it does more than the name suggests. It splits each document into nodes with the default sentence splitter, sends every node to an embedding model, and stores the vectors. This is the step that costs money and time. In the starter configuration the embeddings come from OpenAI and the vectors live in memory.

as_query_engine() builds the querying stage: a retriever, an optional set of node postprocessors, and a response synthesizer. Calling it with no arguments accepts every default.

query() embeds your question and retrieves the closest nodes. When nodes are available for synthesis, it packs them into a prompt with the question and returns the model’s answer along with the source nodes it used. Always look at response.source_nodes while you are learning. It is the difference between “the model is bad” and “the retriever handed the model the wrong paragraph”, and those have completely different fixes.

Configure once, with Settings

The first thing most people want to change is the model. The current way is the global Settings object:

from llama_index.core import Settings
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding

Settings.llm = OpenAI(model="gpt-4o-mini", temperature=0.1)
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")
Settings.chunk_size = 512

If you find a tutorial that builds a ServiceContext and passes it around, that tutorial predates version 0.10. ServiceContext was deprecated in favour of Settings in 0.10 and removed in 0.11, so the code will not run on a current install. This is the single most common reason a copied snippet fails with an import error, and the migration guide in the sources exists specifically for it.

Settings attributes are lazily instantiated, so setting a value you never use costs nothing. Per-call overrides still work: anything you pass directly to a query engine or index beats the global.

Persist the index, or pay twice

The five-line version rebuilds the entire index on every run. Embedding is the slow, metered part of the pipeline, and re-embedding an unchanged corpus at every process start is pure waste.

index.storage_context.persist("storage")

and on the next run:

from llama_index.core import StorageContext, load_index_from_storage

storage_context = StorageContext.from_defaults(persist_dir="storage")
index = load_index_from_storage(storage_context)

Wrap those in a check for whether storage/ exists and the second run becomes near-instant. Two rules go with this. Record which embedding model produced the index, because vectors from different models are not comparable and a swapped model silently returns nonsense. And treat the persisted directory as a build artifact tied to a corpus version, not as a database you edit by hand.

Moving to a real vector store

The default store keeps vectors in the Python process. That is correct for a prototype and wrong for anything with more than one worker. A dedicated vector store separates the index lifecycle from the app lifecycle, and Qdrant is the usual first step in this stack:

pip install llama-index-vector-stores-qdrant

import qdrant_client
from llama_index.core import StorageContext, VectorStoreIndex
from llama_index.vector_stores.qdrant import QdrantVectorStore

client = qdrant_client.QdrantClient("<qdrant-url>", api_key="<qdrant-api-key>")
vector_store = QdrantVectorStore(client=client, collection_name="documents")
storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex.from_documents(documents, storage_context=storage_context)

The application code above this line does not change. That is the argument for the abstraction: the retriever, postprocessors, and query engine are unaware of which store is underneath.

Before adopting this backend, use VectorOps Report’s Qdrant vs Weaviate vs Milvus comparison to review the database options. Its guide to how to choose a vector database covers production selection criteria beyond this integration example.

Before you provision anything, it helps to know roughly how large the index will be. Chunk count scales with corpus size divided by chunk size, and memory scales with chunk count times embedding dimensions times four bytes, plus graph overhead for an approximate index. The site’s chunking and vector store sizer does that arithmetic so you can compare a 384-dimension local embedding model against a 1,536-dimension hosted one before committing to an instance size.

Query engine or chat engine

as_query_engine() answers one question with no memory of the last one. That is the right default for search-style traffic and for anything you evaluate, because each call is independent and reproducible.

as_chat_engine() adds conversation state. Its retrieval behavior depends on chat_mode: condense_question rewrites a follow-up using chat history before calling a query engine, while context retrieves using the current message. The simple mode does not query the index. Compare query engine vs chat engine for the mode choices and the evaluation implications.

Start with a query engine. Add chat only when the product genuinely needs multi-turn, and evaluate both separately.

The four errors everyone hits first

Check loading, packaging, and retrieval before changing the model.

For a pipeline that runs but returns the wrong evidence, follow LlamaIndex retrieval troubleshooting to inspect source nodes before changing the model or prompt.

Evaluate before you believe it

A RAG pipeline that answers fluently and wrongly looks identical to one that works. LlamaIndex ships evaluation modules for both halves of the problem: retrieval metrics such as hit rate and mean reciprocal rank for whether the right node came back, and response evaluators for faithfulness, context relevancy, answer relevancy, and correctness for whether the answer used it properly. Many of these do not require ground-truth labels, and the framework can synthetically generate question and context pairs from your own text, so there is no excuse for shipping with zero measurements.

Build the evaluation set on day one, while the corpus is small. It is the only way to tell whether a change to chunk size helped.

What to read next

• The concepts behind the five lines, in one page: LlamaIndex ingestion, indexing and retrieval explained.
• Picking the framework at all: LlamaIndex vs LangChain, which to use for RAG.
• When the pipeline runs but the answers are wrong: LlamaIndex retrieval troubleshooting.