Generative AI (genAI) poses a classic IT dilemma. When it works well, it is amazingly versatile and useful, fueling dreams that it can do almost anything.
The problem is that when it does not do well, it might deliver wrong answers, override its instructions, and pretty much reinforce the plotlines of every sci-fi horror movie ever made. That is why I was horrified when OpenAI late last month announced changes to make it much easier to give its genAI models full access to any software using Model Context Protocol (MCP).
“We’re adding support for remote MCP servers in the Responses API, building on the release of MCP support in the Agents SDK,” the company said. “MCP is an open protocol that standardizes how applications provide context to LLMs. By supporting MCP servers in the Responses API, developers will be able to connect our models to tools hosted on any MCP server with just a few lines of code.”
There are a large number of companies that have publicly said they will use MCP, including those with popular apps such as PayPal, Stripe, Shopify, Square, Slack, QuickBooks, Salesforce and GoogleDrive.
The ability for a genAI large language model (LLM) to coordinate data and actions with all of those apps — and many more —certainly sounds attractive. But it’s dangerous because it allows access to mountains of highly sensitive compliance-relevant data — and a mistaken move could deeply hurt customers. MCP would also allow genAI tools to control those apps, exponentially increasing risks.
If the technology today cannot yet do its job properly and consistently, what level of hallucinogens are needed to justify expanding its power to other apps?
Christofer Hoff, the CTO and CSO at LastPass, took to LinkedIn to appeal to common sense. (OK, if one wanted to appeal to common sense, LinkedIn is probably not the best place to start, but that’s a different story.)
“I love the enthusiasm,” Hoff wrote. “I think the opportunity for end-to-end workflow automation with a standardized interface is fantastic vs mucking about hardcoding your own. That said, the security Jiminy Cricket occupying my frontal precortex is screaming in terror. The bad guys are absolutely going to love this. Who needs malware when you have MCP? Like TCP/IP, MCP will likely go down as another accidental success. At a recent talk, Anthropic noted that they were very surprised at the uptake. And just like TCP/IP, it suffers from critical deficiencies that will have stuff band-aided atop for years to come.”
Rex Booth, the CISO at identity vendor SailPoint, said the concerns are justified. “If you are connecting your agents to a bunch of highly sensitive data sources, you need to have strong safeguards in place,” he said.
But as Anthropic itself has noted, genAI models do not always obey their own guardrails.
QueryPal CEO Dev Nag sees inevitable data usage problems.
“You have to specify what files [the model] is allowed to look at and what files it is not allowed to look at and you have to be able to specify that,” Nag said. “And we already know that LLMs don’t do that perfectly. LLMs hallucinate, make incorrect textual assumptions.”
Nag argued that the risk is — or at least should be — already well known to IT decision makers. “It’s the same as the API risk,” Nag said. “If you open up your API to an outside vendor with their own code, it could do anything. MCP is just APIs on steroids. I don’t think you’d want AI to be looking at your core financials and be able to change your accounting.”
The best defense is to not trust the guardrails on either side of the communication, but to give the exclusion instructions to both sides. In an example with the model trying to access Google Docs, Nag said, dual instructions are the only viable approach.
“It should be enforced at both sides, with the Google Doc layer being told that it can’t accept any calls from the LLM,” Nag said. “On the LLM side, it should be told ‘OK, my intentions are to show my work documents, but not my financial documents.’”
Bottom line: the concept of MCP interactiveness is a great one. The likely near-term reality? Not so much.
This story originally appeared on Computerworld
