A few keystrokes. One clever prompt. That’s all it took to turn a friendly chatbot into a weapons instructor.

According to an NBC News investigation, several of OpenAI’s advanced models, including those accessible through ChatGPT, were tricked into providing instructions on how to create explosives, chemical weapons, and biological agents.

The findings highlight a worrying gap between the company’s stated safety goals and the real-world resilience of its models against deliberate misuse. NBC News reported that the exploit relied on a “jailbreak,” a term used to bypass built-in safety filters.

Tests reveal dangerous loopholes

NBC News conducted tests on four of OpenAI’s top models, o4-mini, gpt-5-mini, oss-20b, and oss-120b, and found that they “consistently agreed to help with extremely dangerous requests.” The outlet reported that one model gave steps for making a pathogen that targets the immune system, while another explained which chemicals could “maximize human suffering.”

Out of 250 harmful queries, NBC said the oss-20b and oss-120b models provided explicit responses 97.2% of the time. These models are open-source and available for anyone to download, making them harder to regulate or update once deployed.

Meanwhile, OpenAI’s GPT-5 model reportedly resisted all attempts to jailbreak. In 20 separate tests, it consistently refused to provide harmful content. However, the report noted that ChatGPT sometimes routes queries through smaller models like GPT-5-mini when usage limits are reached and that model was tricked in 49% of attempts.

OpenAI responds

In response to NBC News’ findings, an OpenAI spokesperson said that using its models to cause harm “is a violation of its usage policies.” The company also stated that it is “constantly refining its models to address such risks” and regularly hosts “vulnerability challenges” to identify and rectify loopholes.

Still, experts argue that voluntary self-regulation may not be enough.

Sarah Meyers West, co-executive director of AI Now, told NBC News, “That OpenAI’s guardrails are so easily tricked illustrates why it’s particularly important to have robust pre-deployment testing of AI models before they cause substantial harm to the public.” She added, “Companies can’t be left to do their own homework and should not be exempted from scrutiny.”

Experts warn of growing biosecurity risks

The revelation has reignited concerns among biosecurity and AI safety researchers.

Seth Donoughe, director of AI at SecureBio, told NBC News, “Historically, having insufficient access to top experts was a major blocker for groups trying to obtain and use bioweapons. And now, the leading models are dramatically expanding the pool of people who have access to rare expertise.”

Researchers warn that the availability of AI tools capable of generating technical biological or chemical information could enable malicious actors to accelerate dangerous projects. Stef Batalis, a biotechnology research fellow at Georgetown University, said it’s difficult for AI systems to differentiate between legitimate research and potential misuse.

“It’s extremely difficult for an AI company to develop a chatbot that can always tell the difference between a student researching how viruses spread in a subway car for a term paper and a terrorist plotting an attack,” she told NBC News.

As AI systems become more powerful and accessible, the risk of misuse grows. OpenAI and other tech firms have repeatedly promised to strengthen safeguards, but experts say self-regulation may not be enough.

The NBC investigation underscores a troubling reality: even with advanced protections, AI models can still be manipulated to produce instructions that, in the wrong hands, could be catastrophic.

As OpenAI faces scrutiny over safety, its latest model is making headlines for another reason. Read how GPT-5 excels on medical benchmarks but stirs ethical debate.