There is a comforting story going around that open models solve the trust problem. If the weights are published, the thinking goes, nobody is hiding anything, and the thing is on our side. It is a good instinct and it gets one thing right and the most important thing wrong. Open weights tell you how the engine was built. They tell you nothing about the one question that actually decides whether you are safe: where your data goes when the engine runs on it, and who can reach it there.
You can take a perfectly open model, run it on a company's servers, feed it the whole private contents of your life, and be exactly as exposed as you were before — because the exposure was never about the secrecy of the weights. It was about custody of the data. An open model on their machine still means your context sits on their machine, where it can be logged, analyzed, cross-referenced with everyone else's, handed to a partner, sold, or produced under a court order. The openness of the model is a distraction from the location of the data, and the location of the data is the whole game.
A promise versus a fact
Here is the distinction worth coming back to, because once you see it you cannot unsee it. There are two completely different kinds of safety, and they are usually sold in the same sentence. One is “we won't look at your data.” The other is “there is nothing for us to look at.” They sound adjacent. They are worlds apart. The first is a promise. The second is a fact about where the data lives.
“We won't look” is a promise. “There's nothing to look at” is a fact. Only one of them is still true after the company is acquired, flips its terms, or gets a subpoena. — the difference custody makes
A promise is only as durable as the entity making it and the incentives acting on that entity. The most sincere privacy pledge in the world is written by people who will be replaced, inside a company that can be sold, bound by terms that can be rewritten, operating under laws that can compel disclosure. “We won't look” is true until the quarter it becomes expensive to keep, or until a government asks and the company has no standing to refuse because the data was sitting right there. Every data breach, every quiet policy change, every subpoena-for-your-messages headline is the same story: a promise met a stronger incentive and lost.
“There is nothing to look at” does not have that weakness, because it is not a behavior anyone has to maintain. It is a structural condition. If the data never left your machine, if the keys live with you, if no copy exists on anyone else's server to be examined, then there is simply nothing to produce under subpoena, nothing to monetize after an acquisition, nothing to leak when the company is breached. The company cannot betray a secret it was never given. Safety stops depending on anyone's good behavior and starts depending on arithmetic.
Custody is the real guarantee
This is why custody, not trust, is the bar worth insisting on. Trust-based guarantees are the ones that read, underneath the marketing, as “you can do that if we let you and if you believe we won't change our minds.” Custody-based guarantees are the ones that hold regardless of what the vendor wants, because they are enforced by where the data and the keys physically sit. The test for any privacy claim is simple and merciless: strip out every promise and every pledge, and ask what is still guaranteed by the structure alone. Whatever survives that strip-down is real. Everything else is a courtesy with an expiration date you don't control.
Note how much this raises the bar past open weights. Publishing the model is good, and worth doing, but it is a statement about the engine, and the engine was already becoming a commodity. The thing that was never a commodity — your context, your history, the private contents of your life — is protected by one thing only: whether it is in your custody or someone else's. Open the weights all you like. If your data is on their server, you are trusting them. If your data is on your machine, you don't have to.
So the next time something is sold to you on the strength of a promise — we care, we won't look, your privacy matters to us — it is worth asking the quiet follow-up: and if you wanted to, or were compelled to, or were bought by someone who did, could you? If the honest answer is yes, you have a promise, and promises are weather. Ask instead for the arrangement where the answer is no — where the thing you would be worried about them doing is not forbidden but impossible. That is custody. That is the only guarantee that is still standing on the bad day.
This is unfinished, and the engineering that turns “nothing to look at” from a slogan into a provable property is genuinely hard and not fully solved. But the direction is not in doubt, and the distinction is worth carrying everywhere: don't buy the promise. Ask for the fact.