LLM Engine 0.9.1: Streaming API and 32K context

LLM Engine 0.9.1 updates the local inference engine with streaming generation, a larger context window and a notable speed boost on Ada Lovelace GPUs.

New features

• Streaming API for token-by-token generation
• Context window up to 32K tokens
• Simultaneous support for embeddings and generative models and APIs
• stop_sequences parameter to control generation stops
• Built-in UI for model testing and interaction

Changes

• Inference is 40% faster on RTX 40xx (Ada Lovelace)
• Updated config file format: new gpu_layers and context_size fields

Removed

• Deprecated /api/v0/complete endpoint — use /api/v1/generate

How to upgrade

Windows and Docker builds are on the download page. Full changelog is in the version history.