Why I Replaced a Linear RAG Pipeline with Temporal Workflows

My RAG architecture for Aktilot started the same way most RAG systems do.

Documents were chunked, embeddings were generated, relevant context was retrieved and an LLM produced an answer.

Initially the architecture was simple and worked remarkably well.

But as the system evolved I found myself spending less time on retrieval quality and more time dealing with retries, background jobs and failure recovery.

Eventually, I realized I wasn't dealing with a pipeline anymore.

I was solving workflow problems.

That's what led me to replace the orchestration layer with Temporal.

The Realization

When I first built my RAG system I wasn't trying to design anything sophisticated, I just wanted something that could answer questions over documents reliably.

The architecture was familiar and looked like countless examples you can find online:

It was simple, easy to reason about and most importantly it worked really well.

For a while, I didn't think much about the architecture. My focus was on improving retrieval quality and experimenting with chunk sizes, tweaking prompts and combining vector search with keyword search etc.

But as the project evolved I noticed something interesting.

I wasn't spending most of my time working on embeddings or retrieval anymore. Instead I was dealing with everything around them.

like - how should document ingestion recover from failures? How do I benchmark retrieval quality after making changes? How do I retry failed operations without reprocessing everything? How do I keep track of what's happening inside the system? and many more . . . . . .

Little by little the simple pipeline I had started with began accumulating scripts, many background jobs, custom and maintenance heavy retry logics.

And then it clicked - the hard part of building a production RAG system wasn't retrieval - it was orchestration.

The Document ingestion, query execution and evaluation of system were all long running processes with multiple steps involved, In other words they were workflows.

I had been treating them like pipelines, That's when I decided to move the orchestration layer to Temporal.

The Original Linear RAG Architecture

The original architecture revolved around two core flows

1. Document ingestion flow.
2. User Query processing.

Both were relatively simple and easy to reason about.

On the ingestion side documents were split into chunks then embeddings were generated and the resulting vectors were stored in the vector database.

On the query side a user question triggered a hybrid retrieval process. Relevant chunks were first retrieved using vector similarity search, supplemented with keyword search and then reranked before being passed to the LLM to generate the final answer.

The diagram above illustrates these two independent flows. On the left is the document ingestion pipeline responsible for preparing and indexing knowledge base. On the right is the query pipeline which retrieves the most relevant context and uses the LLM to produce an answer.

At the time the architecture was straightforward, easy to understand and worked remarkably well.

Problems with the Linear Pipeline

1. No Durable State

Document ingestion involved several expensive operations:

• Chunking
• Embedding generation
• Indexing

If embedding generation failed midway, the entire process had to start again. There was no checkpointing and no way to resume from where it failed.

2. Retry & Error Handling Logic Became Complicated

RAG systems rely heavily on external components, and external systems are never perfectly reliable.

• Embedding APIs occasionally time out or hit rate limits.
• LLM requests fail due to network issues or provider outages.
• Vector databases and metadata stores can become temporarily unavailable.

At first handling these failures seemed straightforward. But as the system grew every component started requiring its own retry strategy and error handling.

I found myself repeatedly implementing:

• Retry loops
• Timeout handling
• Error classification
• Recovery mechanisms

What started as a simple pipeline gradually turned into a collection of services each implementing its own version of failure management. Instead of focusing on retrieval quality and experimentation, I was spending more time writing infra code to deal with retries and failures.

3. RAG Evaluation Became a Separate System

As retrieval quality became more important I wanted to continuously measure metrics such as:

• Recall@K
• Mean Reciprocal Rank (MRR)
• Latency

Initially, benchmarking was implemented as a separate process. However this introduced another problem. The evaluation service needed to execute the same retrieval pipeline used by the user query execution system, which meant duplicating hybrid search, reranking and query execution logic.

Over time I realized that RAG evaluation wasn't a separate concern. It was simply another workflow that shared many of the same building blocks as query execution.

4. Chat Flow Became Increasingly Complex

Initially, the query path was fairly straightforward like below

Query
↓
Hybrid Search
↓
Rerank
↓
Prompt Construction
↓
LLM
↓
Citation Extraction
↓
Answer

The architecture was easy to understand and maintain.

However, as I started experimenting with retrieval quality and answer accuracy the additional capabilities were introduced:

• Query rewriting
• Multi-query retrieval
• Verification
• Guardrails
• Conversation memory

What started as a simple sequence of function calls gradually evolved into a collection of interconnected components, each with its own retries and error handling

Adding new capabilities became difficult and time consuming. A seemingly simple change often required touching multiple services and coordinating logic across different parts of the system. The flow itself was no longer linear.

RAG Is Actually a Workflow

At some point, I realized something important.

RAG isn't just one pipeline.

It contains multiple independent processes.

Document Ingestion

Chunk
 ↓
Embed
 ↓
Index

Query Processing

Hybrid Search
 ↓
Rerank
 ↓
Generate Answer

Evaluation

Run Dataset
 ↓
Calculate Recall@K
 ↓
Calculate MRR
 ↓
Measure Latency

These are workflows.

Not pipelines.

Introducing Temporal

Instead of coordinating everything manually, I moved orchestration into Temporal.

The architecture now looks like this:

Each workflow consists of activities with built-in support for:

• Automatic retries
• Timeout handling
• State persistence
• Failure recovery
• Observability

Document Workflow

Document ingestion became a dedicated workflow.

DocumentWorkflow

ChunkDocument
      ↓
GenerateEmbeddings
      ↓
IndexChunks
      ↓
PersistMetadata

If embedding generation fails after processing hundreds of chunks, Temporal resumes from the failed activity instead of restarting everything.

This alone eliminated a significant amount of operational complexity.

Chat Workflow

Query execution also became a workflow.

ChatWorkflow

HybridSearch
      ↓
RerankDocuments
      ↓
GenerateAnswer
      ↓
StoreConversation

This architecture made experimentation easier.

Adding new capabilities became straightforward:

• Query rewriting
• Context compression
• Multi-query retrieval
• Guardrails
• Citation verification

Instead of creating another chain of nested function calls, new activities could simply be inserted into the workflow.

Benchmark Workflow

One of the biggest improvements came from evaluation.

I introduced a Benchmark Workflow:

BenchmarkWorkflow

Load Dataset
      ↓
Run Queries
      ↓
Calculate Recall@K
      ↓
Calculate MRR
      ↓
Measure Latency
      ↓
Store Results

Now every retrieval change can be measured consistently.

Changes to:

• Chunk size
• Embedding models
• Reranking strategy
• Retrieval configuration

can all be evaluated using the same workflow.

Benchmarking became a first-class capability instead of a collection of scripts.

Benefits of the Workflow-Based Architecture

Why Temporal?

Temporal provides several capabilities that fit RAG systems naturally:

Durable Execution - Workflows automatically resume from the point of failure instead of restarting the entire process.

Automatic Retries - Transient failures are handled through built-in retries, backoff policies, and timeout management.

Observability - Every workflow execution is visible through Temporal UI, providing complete insight into state and failures.

Extensibility - New capabilities can be introduced by adding activities and workflows without increasing system complexity.

Looking Ahead

Moving to Temporal opened the door for more advanced capabilities:

• Multi-agent RAG
• Human-in-the-loop workflows
• Scheduled re-indexing
• Long-running conversations
• Continuous benchmarking
• Agentic retrieval
• Adaptive chunking
• Query rewriting workflows

Now the architecture is no longer optimized for today's requirement. It is designed to evolve.

Tradeoffs

Temporal introduced additional complexity for sure.

Running a Temporal cluster means:

• More infrastructure.
• A learning curve.
• Workflow design considerations.

For small RAG applications a linear pipeline is often sufficient. But as Aktilot evolved, the benefits of durability and workflow orchestration outweighed the operational overhead.

Conclusion

Initially I thought I was replacing a RAG pipeline.

In reality I was replacing a stateless sequence of function calls with a durable workflow engine. As RAG systems mature orchestration becomes just as important as embeddings and vector databases.

For Aktilot moving to Temporal transformed the system from a chain of steps into a resilient, observable and extensible platform.

And perhaps the biggest realization was this:

Production RAG is not a pipeline problem. It's a workflow problem.