Use AI without giving away your data

Somewhere in your firm, a capable person recently talked themselves out of a tool that would have saved them hours. Not because it failed, but because a quieter question stopped them: where does our data go when we do this? For a firm entrusted with client tax returns, privileged matters, or the private numbers behind a deal, that hesitation is a mark of professional judgment.

It is also, as it is usually framed, a false choice. The belief that adopting AI means shipping your most sensitive information to a stranger, with sitting the technology out as the only alternative, was a defensible read two years ago. Today you can take the operational gains and keep control of the data that makes your firm worth hiring. This piece is about how, and about where the real risk actually lives.

What is worth worrying about, and what is not

The caution is well placed, so start there. In accounting, law, healthcare, and any practice built on confidential information, you are responsible for the data your clients hand you. When their records leave your walls for a third party’s servers, the duty to protect them stays with you. You still answer to the client, to your regulator, and sometimes to a statute, if that information is exposed, retained, subpoenaed, or breached. A tax preparer who discloses return information without written consent is exposed under a federal criminal statute. A lawyer who feeds a client’s confidences into the wrong tool can breach the duty of confidentiality the American Bar Association spelled out for generative AI in 2024. Health information cannot touch a vendor without a signed agreement. The worry is real, and it deserves to be taken seriously.

None of that means AI has to be kept at arm’s length. The fear worth setting aside is that using AI means surrendering that data at all, because you decide what touches which system. The sensitive records never have to leave an environment you control, while the great majority of the work, the part that touches nothing confidential, runs on the best tools available. The rest of this piece is about how that split works in practice, and why it is now both good enough and cheap enough to rely on.

You can keep the data you are trusted to protect under your own control, and still put AI to work on everything else.

The models are good enough now, and far cheaper

For years the honest objection was capability: the models you could run yourself were toys next to the frontier, so keeping data in-house meant accepting worse work. That gap has closed to a crack.

The best open-weight models now trail the closed frontier by roughly three to six months on public benchmarks, a lag that has stopped widening. They are not equal, and on the hardest reasoning and long-horizon agentic work the frontier still wins. But most of what a firm actually does, drafting, summarizing, extracting, checking, researching against your own documents, sits comfortably inside what a good open model handles today. You do not need the absolute frontier to get a large lift. You need a capable model pointed at the right work, and increasingly you can run it cheaply in an environment you control.

The old excuse, that the private option is too weak and too costly to bother with, no longer holds.

You don’t have to choose. Route the work.

This is the move that dissolves the false choice, and most firms miss it by treating the decision as all-or-nothing. You do not have to run everything in-house, and you should not try. Most of the work you would hand to AI touches nothing sensitive: summarizing a public filing, drafting a first-pass email, formatting a document, researching a question with no client specifics in it. Send that to the best cloud model you can get. Only the genuinely sensitive slice, the client’s private data, the privileged matter, the records you are bound to protect, needs to stay on rails you control.

Deciding what goes where is a solved engineering problem. A router, or model gateway, sits in front of your AI and sends each task to the right destination by policy: this category to the frontier API, that category to a private model running in your own environment. Tools like LiteLLM, Portkey, and OpenRouter do this in production today. Which tool you use matters less than the point it settles: “cloud or local” was never the real question, and “which data, to which model, under whose control” always was.

When running your own models is the wrong call

None of this makes self-hosting automatically the safe choice, and you should distrust anyone who says it is. “On-prem” does not mean “secure.” A private model on a misconfigured, unpatched server is a softer target than a hardened enterprise API run by a vendor whose business depends on not being breached. Self-hosting brings real costs: the hardware, the people to run it, the security you now own, and models that need refreshing as the field moves. For most firms the right default is a reputable API with the right contract, and a private model reserved for the genuinely sensitive minority. The router is what lets you be selective instead of absolutist.

It is worth being honest about where these efforts actually fail, because it is rarely the part people fear. In the MIT review behind that 95% figure, the pilots that collapsed almost never failed on model quality. They failed on integration: tools bolted onto broken workflows, with no one redesigning the work around them. The deployment question matters, but it sits downstream of the harder operational work that makes any of this pay off.

This is a strategy question, not an IT setting

Which is the real point. Where your data goes is not a technical checkbox to delegate and forget. It is a decision about which data you are responsible for and how you keep it protected while still moving quickly. And it is becoming a selling point: telling a client, credibly, that their information never leaves your control is an advantage the largest organizations are already writing into procurement language, and one a sharp independent firm can offer sooner and more personally.

Protecting confidential data is a real responsibility, and a solvable one. AI is finally good enough, and cheap enough, to put to work across a firm without exposing the information you are trusted to keep. The firms that look prescient in a few years will be the ones that stopped treating this as all-or-nothing: they decided which sliver of their data had to stay private, kept exactly that under their own control, and used AI freely on everything else. Be early, not late.