LLM#

classDiagram BaseModelResponseParser <|-- CodeBlocksResponseParser BaseModelResponseParser <|-- JSONResponseParserBase BaseModelResponseParser <|-- YAMLResponseParserBase ChatGPTConfigurationBase <|-- AzureChatGPTConfiguration ChatGPTConfigurationBase <|-- OpenAIChatGPTConfiguration DataObject <|-- LLMConfigObject DataObject <|-- LLMCostManagerObject DataObjectSpecBase <|-- LLMConfigSpec DefaultLLMConsumptionCalculatorHelper <|-- DefaultLLMConsumptionCalculator JSONResponseParserBase <|-- JSONBlockResponseParser JSONResponseParserBase <|-- JSONResponseParser LLMBase <|-- AnthropicLLM LLMBase <|-- GeminiLLM LLMBase <|-- GroqLLM LLMBase <|-- LLMFallback LLMBase <|-- OllamaLLM LLMBase <|-- OpenAIChatCompletionsModel LLMConfigurationBase <|-- AnthropicLLMConfiguration LLMConfigurationBase <|-- ChatGPTConfigurationBase LLMConfigurationBase <|-- GeminiLLMConfiguration LLMConfigurationBase <|-- GroqLLMConfiguration LLMConfigurationBase <|-- OllamaLLMConfiguration LLMConsumptionCalculatorBase <|-- DefaultLLMConsumptionCalculatorHelper LLMException <|-- LLMCallException LLMException <|-- LLMCallTimeoutException LLMException <|-- LLMOutOfRetriesException LLMException <|-- LLMTokenLimitException LLMFunction <|-- LLMFunctionWithPrompt LLMFunctionError <|-- FunctionOutOfRetryError LLMLoggingMiddlewareBase <|-- LLMFileLoggingMiddleware LLMLoggingMiddlewareBase <|-- LLMLoggingMiddleware LLMLoggingMiddlewareBase <|-- LLMTimestampFileLoggingMiddleware LLMMessageData <|-- LLMCacheControlData Monitorable <|-- LLMBase Monitored <|-- MonitoredLLM OpenAIChatCompletionsModel <|-- AzureLLM OpenAIChatCompletionsModel <|-- OpenAILLM YAMLResponseParserBase <|-- YAMLBlockResponseParser YAMLResponseParserBase <|-- YAMLResponseParser

This package provides clients to use various LLMs.

Overview#

The council.llm module provides a unified interface for interacting with various LLM providers, along with tools for handling responses, caching, logging and tracking consumptions metrics.

LLMs#

Create your LLM instance from YAML config file with LLMConfigObject (see for different config examples).

Currently supported providers include:

OpenAI’s GPT and o1 - OpenAILLM
Anthropic’s Claude - AnthropicLLM
Google’s Gemini - GeminiLLM
Microsoft’s Azure - AzureLLM
Groq - GroqLLM
and local models with ollama - OllamaLLM

from council.llm import get_llm_from_config

# will adjust provider class automatically based on config file
llm = get_llm_from_config("data/configs/llm-config-openai.yaml")

Making Requests and Managing Costs#

Use llm.post_chat_request() method to interact with an LLM. The returned LLMResult object contains LLM response as well as list of Consumption metrics associated with the call, including duration, token usage and costs.

import dotenv
from council import LLMContext
from council.llm import LLMMessage, get_llm_from_config

llm = get_llm_from_config("data/configs/llm-config-openai.yaml")
result = llm.post_chat_request(
    LLMContext.empty(),
    messages=[LLMMessage.user_message("Hello world")]
)

print(result.first_choice)
# sample output:
# Hello! How can I assist you today?

for consumption in result.consumptions:
    print(consumption)
# sample output:
# gpt-4o-mini-2024-07-18 consumption: 1 call
# gpt-4o-mini-2024-07-18 consumption: 0.9347 second
# gpt-4o-mini-2024-07-18:prompt_tokens consumption: 9 token
# gpt-4o-mini-2024-07-18:completion_tokens consumption: 9 token
# gpt-4o-mini-2024-07-18:total_tokens consumption: 18 token
# gpt-4o-mini-2024-07-18:prompt_tokens_cost consumption: 1.3499e-06 USD
# gpt-4o-mini-2024-07-18:completion_tokens_cost consumption: 5.399e-06 USD
# gpt-4o-mini-2024-07-18:total_tokens_cost consumption: 6.7499e-06 USD

Anthropic Prompt Caching Support#

For information about enabling Anthropic prompt caching, refer to LLMCacheControlData.

Prompt Management#

Store your prompts as YAML files as unstructured text (LLMPromptConfigObject) or structured objects (LLMStructuredPromptConfigObject) with automatic selection of the prompt based on the LLM used.

LLM Functions#

LLM Functions provide structured ways to interact with LLMs including built-in response parsing, error handling and retries.

See LLMFunction for a code example
Use LLMFunctionWithPrompt to create an LLMFunction with LLMPromptConfigObject

Response Parsers#

Response parsers help automate the parsing of common response formats to use LLMFunctions conveniently:

EchoResponseParser for raw LLMResponse
StringResponseParser for plain text
CodeBlocksResponseParser for code blocks
YAMLBlockResponseParser and YAMLResponseParser for YAML
JSONBlockResponseParser and JSONResponseParser for JSON

Code block, YAML and JSON response parsers also support to_response_template() method to convert the structured object into a natural language response template description.

LLM Middleware#

Middleware components allow you to enhance LLM interactions by modifying requests and responses introducing custom logic, such as logging, caching, configuration updates, etc.

Core middlewares:

Caching: LLMCachingMiddleware
Logging:
- Context logger: LLMLoggingMiddleware
- File logging:
  - LLMFileLoggingMiddleware
  - LLMTimestampFileLoggingMiddleware for single file per request

Middleware management:

Fine-tuning and Batch API#

See LLMDatasetObject for details on how to convert your YAML dataset into JSONL for fine-tuning and batch API. Currently, the functionality is limited to generating JSONL files and does not include utilities for managing fine-tuning or batch job processes.