On This Page
In a Guest article for the German trade magazine BigData-Insider our CEO Christopher Helm explains the new opportunities and challenges induced by modular AI systems.
For more on this topic, see our blog articles.
When new technologies are introduced, it often doesn't take long for the new limits of what is possible to be balanced out. When it comes to large language models (LLMs), this point has now been reached: even if they achieve impressive natural language processing results in many cases, implementation and maintenance are still too inefficient, especially for complex specialist applications. At the same time, there is a lack of specialization and reusability. But this apparent dilemma has a solution.
Researchers from Berkeley and Stanford have recently pointed out that language models are increasingly being integrated into modular AI systems consisting of various components that are much better able to meet this challenge. In many respects, they outperform the previously used monolithic models. The modular approach also plays an important role in our own AI development, reason enough to share a few insights in the press.
The most important insights
- Modular AI systems comprise language models, simpler models and other components, often in sequences using Retrieval Augmented Generation (RAG) and Multi-Step Chains
- The output of individual components also serves as an intermediate result and as input for subsequent steps
- This allows extensive specialist tasks to be divided into smaller units
- These can therefore be managed more easily and efficiently. Troubleshooting, explainability and maintainability are also less complicated.
- Many of the individual steps can also be found in other processes. The corresponding components can be directly recycled there.
- This means that specialization and generalized applicability are taking place at the same time.
- In many cases, this saves companies countless training procedures and readjustments.
- Current developments relate primarily to the challenges of data integration, communication between the modules and the practical applicability of this young technology.
From Monolith to Module: What Changes in Practice
The shift from monolithic to modular AI is not merely architectural. It changes how teams build, maintain, and extend AI systems in concrete ways.
Monolithic Pipeline
- Single model handles intake, reasoning, extraction, and response
- Failures are opaque
- Debugging requires re-running the full pipeline
- Intermediate outputs never designed for inspection
Modular System
- Each component has a defined input, output, and responsibility
- Components can be tested, replaced, and monitored independently
- Retrieval, classification, and generation are separate modules
- Safe to operate in production where mistakes have cost
In a monolithic LLM pipeline, a single model handles intake, reasoning, extraction, and response generation. When it fails, particularly on edge cases outside its training distribution, the failure is opaque. And it will fail. Debugging requires re-running the full pipeline and examining intermediate outputs that were never designed to be inspected.
In a modular system, each component has a defined input, output, and responsibility. A retrieval module fetches relevant context. A classification module routes the query. A generation module produces the response. Each can be tested, replaced, and monitored independently. This separation isn't a convenience; it's what makes AI systems safe to operate in production environments where mistakes have cost.
The Role of RAG and Multi-Step Chains
Two architectural patterns dominate current modular AI deployments:
Retrieval-Augmented Generation (RAG) connects an LLM to an external knowledge base, typically a vector database containing domain documents. Instead of relying on the model's training data alone, the system retrieves relevant document chunks at inference time and passes them as context. The result: domain-accurate responses without costly full model fine-tuning. For document-heavy industries like insurance, legal, and finance, RAG is now the baseline for any serious knowledge system.
Multi-Step Chains break complex tasks into sequential sub-tasks, each handled by the most appropriate model or tool. A due diligence workflow might run: document ingestion → entity extraction → cross-reference check → risk flag generation → summary drafting. Each step uses a specialized component. The chain as a whole delivers output no single model could produce reliably.
Both patterns benefit from the modularity principle: components can be upgraded, swapped, or reused without touching the rest of the pipeline.
Reusability as a Strategic Asset
One of the least-discussed advantages of modular AI is component reuse. When an entity extraction module is trained and validated for supplier contracts, it can be reused for customer agreements, insurance policies, and regulatory filings, with minimal adaptation. The same applies to classification models, summarization modules, and output formatters.
This reusability changes the economics of AI deployment fundamentally. Instead of training a new model for each use case, including its associated data labeling, compute, and validation costs, organizations build a library of tested components and compose them into application-specific pipelines. The marginal cost of the second use case is a fraction of the first.
According to the Berkeley and Stanford research cited in the original BigData-Insider article, teams working with compound AI systems report significantly lower iteration time on new tasks compared to monolithic model development, primarily because the foundational components are already validated and in production.
Practical Guidance for Implementation Teams
Organizations beginning modular AI development should consider three structural decisions early:
Define the component contract. Each module needs a clear specification: what it accepts as input, what it guarantees as output, and under what conditions it should fail gracefully rather than return a degraded result. Without this, the "modularity" becomes theoretical and components become tightly coupled through implicit assumptions.
Build for observability from day one. Log intermediate outputs at each module boundary. When a pipeline produces an unexpected result, you need to know exactly which component introduced the error. Observability is not a monitoring add-on; it is a design requirement.
Plan the orchestration layer. Something must coordinate the sequence of module calls, handle failures, manage retries, and route between alternative paths. Frameworks like LangChain, LlamaIndex, and custom orchestrators each have trade-offs. The right choice depends on your existing infrastructure and team expertise, not vendor marketing.
Current Limitations and Open Challenges
The BigData-Insider article correctly notes that practical challenges remain. Three deserve particular attention for teams evaluating modular AI now:
Data integration across modules. Components often expect data in different formats. A retrieval module returns text chunks; a classification module expects a categorical schema; a generation module needs a formatted prompt. Building robust data transformation layers between components adds engineering overhead that is easy to underestimate in early prototypes.
Latency accumulation. Each module adds inference time. A five-step chain where each module takes 800ms produces a 4-second end-to-end response, acceptable for batch processing but problematic for interactive applications. Latency budgeting must happen at architecture design time, not after deployment.
Evaluation complexity. Testing a monolithic model is straightforward: feed it inputs, measure outputs. Testing a modular pipeline requires evaluating each component independently and the full pipeline end-to-end. Regression testing across component upgrades adds further complexity. Teams that skip structured evaluation frameworks discover this the hard way when a component upgrade breaks a downstream module in production.
These are engineering problems with known solutions, not fundamental barriers. But they require deliberate planning, and organizations that treat modular AI as a plug-and-play technology will encounter them unexpectedly.
About BigData-Insider
The trade magazine is primarily aimed at IT decision-makers, project managers, managing directors and anyone involved in artificial intelligence and big data. It deals with relevant topics relating to data processing, infrastructure, Industry 4.0 in theory and practice. For years, the portal has been one of the most important sources of information on current aspects of AI development and application.