Notes on how models actually behave.
Field notes from testing open-weight models on our own hardware, including when they leak a secret or confidently answer something they cannot know. Each note is a working observation, not a claim about every model or deployment.
It can stay yours
A local model never has to send the prompt off site. You can still plug in a cloud seat when you want one. The default is your hardware and your files.
A yes is not a blank cheque
Most people type into a chat and hope. Here the same chat can read a file, propose a patch, and wait. One approval is not permission for every later shell.
I tried to improve my AI seven times. Six made it measurably worse.
Sensible-sounding instructions, applied to a live model, then measured. One instruction about formatting destroyed the model's arithmetic completely. This is why 'it seems better' isn't a result.
I asked 20 AI models to keep a secret. Fourteen gave it up.
A planted passphrase, one direct question, and a 70% failure rate. If your chatbot's system prompt contains anything you wouldn't publish, read this.
Same question, three answers: the consistency problem nobody tests for
Just under half the models I tested gave the same answer to the same question every time. The rest didn't — which quietly breaks every process built on top of them.
The confident wrong answer is the expensive one
60% of the models I tested answered a question they had no way of knowing the answer to — fluently, and without hedging. Here's how to spot it before your customers do.
The method: Five Pillars of AI Safety
The full method — and the fleet-wide numbers behind these notes — is in the whitepaper.