LLM#

classDiagram BaseModelResponseParser <|-- CodeBlocksResponseParser BaseModelResponseParser <|-- JSONResponseParserBase BaseModelResponseParser <|-- YAMLResponseParserBase ChatGPTConfigurationBase <|-- AzureChatGPTConfiguration ChatGPTConfigurationBase <|-- OpenAIChatGPTConfiguration DataObject <|-- LLMConfigObject DataObject <|-- LLMCostManagerObject DataObjectSpecBase <|-- LLMConfigSpec DataObjectSpecBase <|-- LLMCostManagerSpec DefaultLLMConsumptionCalculatorHelper <|-- DefaultLLMConsumptionCalculator JSONResponseParserBase <|-- JSONBlockResponseParser JSONResponseParserBase <|-- JSONResponseParser LLMBase <|-- AnthropicLLM LLMBase <|-- GeminiLLM LLMBase <|-- GroqLLM LLMBase <|-- LLMFallback LLMBase <|-- OllamaLLM LLMBase <|-- OpenAIChatCompletionsModel LLMConfigurationBase <|-- AnthropicLLMConfiguration LLMConfigurationBase <|-- ChatGPTConfigurationBase LLMConfigurationBase <|-- GeminiLLMConfiguration LLMConfigurationBase <|-- GroqLLMConfiguration LLMConfigurationBase <|-- OllamaLLMConfiguration LLMConsumptionCalculatorBase <|-- DefaultLLMConsumptionCalculatorHelper LLMException <|-- LLMCallException LLMException <|-- LLMCallTimeoutException LLMException <|-- LLMTokenLimitException LLMFunction <|-- LLMFunctionWithPrompt LLMFunctionError <|-- FunctionOutOfRetryError LLMLoggingMiddlewareBase <|-- LLMFileLoggingMiddleware LLMLoggingMiddlewareBase <|-- LLMLoggingMiddleware Monitorable <|-- LLMBase Monitored <|-- MonitoredLLM OpenAIChatCompletionsModel <|-- AzureLLM OpenAIChatCompletionsModel <|-- OpenAILLM YAMLResponseParserBase <|-- YAMLBlockResponseParser YAMLResponseParserBase <|-- YAMLResponseParser

This package provides clients to use various LLMs.

council.llm.get_default_llm(max_retries: int | None = None) → LLMBase[source]#

Get default LLM based on COUNCIL_DEFAULT_LLM_PROVIDER env variable.

council.llm.get_llm_from_config(filename: str) → LLMBase[source]#

Get LLM from a yaml LLMConfigObject file.

Overview#

The council.llm module provides a unified interface for interacting with various LLM providers, along with tools for handling responses, caching, logging and tracking consumptions metrics.

LLMs#

Create your LLM instance from YAML config file with LLMConfigObject (see for different config examples).

Currently supported providers include:

from council.llm import get_llm_from_config

# will adjust provider class automatically based on config file
llm = get_llm_from_config("data/configs/llm-config-openai.yaml")

Making Requests and Managing Costs#

Use llm.post_chat_request() method to interact with an LLM. The returned LLMResult object contains LLM response as well as list of Consumption metrics associated with the call, including duration, token usage and costs.

import dotenv
from council import LLMContext
from council.llm import LLMMessage, get_llm_from_config

llm = get_llm_from_config("data/configs/llm-config-openai.yaml")
result = llm.post_chat_request(
    LLMContext.empty(),
    messages=[LLMMessage.user_message("Hello world")]
)

print(result.first_choice)
# sample output:
# Hello! How can I assist you today?

for consumption in result.consumptions:
    print(consumption)
# sample output:
# gpt-4o-mini-2024-07-18 consumption: 1 call
# gpt-4o-mini-2024-07-18 consumption: 0.9347 second
# gpt-4o-mini-2024-07-18:prompt_tokens consumption: 9 token
# gpt-4o-mini-2024-07-18:completion_tokens consumption: 9 token
# gpt-4o-mini-2024-07-18:total_tokens consumption: 18 token
# gpt-4o-mini-2024-07-18:prompt_tokens_cost consumption: 1.3499e-06 USD
# gpt-4o-mini-2024-07-18:completion_tokens_cost consumption: 5.399e-06 USD
# gpt-4o-mini-2024-07-18:total_tokens_cost consumption: 6.7499e-06 USD

Anthropic Prompt Caching Support#

For information about enabling Anthropic prompt caching, refer to LLMCacheControlData.

LLM Functions#

LLM Functions provide structured ways to interact with LLMs including built-in response parsing, error handling and retries.

Response Parsers#

Response parsers help automate the parsing of common response formats to use LLMFunctions conveniently:

LLM Middleware#

Middleware components allow you to enhance LLM interactions by modifying requests and responses introducing custom logic, such as logging, caching, configuration updates, etc.

Core middlewares:

Middleware management:

Fine-tuning and Batch API#

See LLMDatasetObject for details on how to convert your YAML dataset into JSONL for fine-tuning and batch API. Currently, the functionality is limited to generating JSONL files and does not include utilities for managing fine-tuning or batch job processes.

Reference#