Guides / Local models

Local models

Download a model, serve it, and point your app at it. Your code and prompts never leave the machine, and there is nothing to pay per word.

You need llama.cpp

brew install llama.cpp. kosa8 tells you if it is missing rather than failing obscurely.

Pull one

$ kosa8 model pull qwen2.5-coder-7b
$ kosa8 model ls

Serve it

$ kosa8 model serve qwen2.5-coder-7b

Point something at it

kosa8 answers on both of the two dialects that matter, so an app written for either works by changing one address and nothing else.

/v1/chat/completions   what OpenAI libraries call
/v1/completions        the older OpenAI style
/v1/models             what is available
/v1/messages           what Anthropic libraries call

$ export OPENAI_BASE_URL=http://127.0.0.1:11434/v1
$ export ANTHROPIC_BASE_URL=http://127.0.0.1:11434

Sharing one with your team

A model is several gigabytes. Having everyone download it separately is slow, and on a locked-down network may not be allowed at all. kosa8 can publish a model into the same registry your company already uses for container images.

$ kosa8 model push qwen2.5-coder-7b registry.company.com/models/coder:v1
$ kosa8 model pull-oci registry.company.com/models/coder:v1
$ kosa8 model serve coder

It is stored as a standard OCI artifact, so a registry that holds container images can hold this too — no special model server, nothing new for anyone to approve. The fingerprint is checked as it downloads; a damaged or altered file is refused rather than quietly kept.

Using it with the assistant

$ kosa8 ai --provider local -m "what is using the most memory?"

With no ANTHROPIC_API_KEY set, kosa8 ai uses the local model gateway by default, so nothing is sent anywhere.