LLM & Models

Overview

The orchestrator implements both LLMOperations (context-based API) and ToolCapableLLMClient (prompt-based API), providing flexible access to LLM models across the platform.

When to Use Directly

In most cases, you do not need to call the orchestrator directly. The Pipeline's LLM_EXECUTION step calls it automatically. Direct use is appropriate when:

Tiered Model Architecture

The orchestrator uses tier-aware model selection. In the current OSS runtime, TieredLLMProperties exposes two configured layers (layer1 and layer2). Higher semantic tiers from AnalysisLevel or SecurityTaskType are normalized to the configured layer-2 runtime unless an explicit model is requested.

How Tier Selection Works

The effective model is resolved in this order:

1. preferredModel or execution metadata such as requestedModelId, preferredModel, runtimeModelId, or modelId — explicit model IDs bypass tier mapping entirely
2. analysisLevel — QUICK→1, NORMAL→2, DEEP→3 (normalized to configured layer 2 at runtime)
3. securityTaskType — security tasks map to default semantic tiers when analysisLevel is absent
4. tier — explicit numeric tier when no higher-priority selector is present; values above 1 are normalized to layer 2
5. The primary ChatModel bean as the final fallback if no configured model can be resolved

AnalysisLevel Enum

ExecutionContext

The primary way to configure an LLM request. Built with @Builder pattern:

ExecutionContext context = ExecutionContext.builder()
    .prompt(new Prompt("Analyze this for threats"))
    .analysisLevel(ExecutionContext.AnalysisLevel.NORMAL)
    .userId("user-123")
    .requestId(UUID.randomUUID().toString())
    .build();

// Execute
Mono<String> result = orchestrator.execute(context);

// Or stream
Flux<String> chunks = orchestrator.stream(context);

Key Properties

Factory & Helper Methods

Tool Calling

The orchestrator supports Spring AI tool calling for agentic workflows. Two approaches:

Tool Providers (Annotated Methods)

// Tool provider objects - Spring AI discovers @Tool methods
Prompt prompt = new Prompt("Block suspicious IP 192.168.1.100");

Mono<String> result = orchestrator.callTools(
    prompt, List.of(ipBlockTool, auditTool));

ToolCallback Array (Explicit)

// Explicit ToolCallback array
ToolCallback[] callbacks = new ToolCallback[] {
    myToolCallback1, myToolCallback2
};

Mono<String> result = orchestrator.callToolCallbacks(
    prompt, callbacks);

Both approaches also support streaming: streamTools() and streamToolCallbacks().

Advisor System

The orchestrator pulls enabled advisors from the AdvisorRegistry and combines them with any request-scoped advisors attached to the ExecutionContext. Those advisors are applied when the orchestrator builds a chat call for the selected model.

Implementing a Custom Advisor

Extend BaseAdvisor to create advisors that intercept LLM requests and responses:

@Component
public class RequestLoggingAdvisor extends BaseAdvisor {

    public RequestLoggingAdvisor() {
        super("myapp", "request-logging", 100);
        // domain, name, order (lower = runs first)
    }

    @Override
    public ChatClientRequest beforeCall(
            ChatClientRequest request) {
        // Inspect or modify the request before LLM call
        // e.g., log the prompt, add metadata
        return request;
    }

    @Override
    public ChatClientResponse afterCall(
            ChatClientResponse response,
            ChatClientRequest request) {
        // Inspect or modify the response after LLM call
        // e.g., validate response, log metrics
        return response;
    }
}

Advisors are auto-registered as Spring beans. The AdvisorRegistry manages enabled/disabled state and provides sorted advisor snapshots to the orchestrator.

Configuration

API Reference

Related