A dataset used to train large language models (LLMs) has been found to contain nearly 12,000 live secrets, which allow for successful authentication.
The findings once again highlight how hard-coded credentials pose a severe security risk to users and organizations alike, not to mention compounding the problem when LLMs end up suggesting insecure coding practices to their users.
Truffle Security said it downloaded a December 2024 archive from Common Crawl, which maintains a free, open repository of web crawl data. The massive dataset contains over 250 billion pages spanning 18 years.
The archive specifically contains 400TB of compressed web data, 90,000 WARC files (Web ARChive format), and data from 47.5 million hosts across 38.3 million registered domains.
The company’s analysis found that there are 219 different secret types in the Common Crawl archive, including Amazon Web Services (AWS) root keys, Slack webhooks, and Mailchimp API keys.
“‘Live’ secrets are API keys, passwords, and other credentials that successfully authenticate with their respective services,” security researcher Joe Leon said.
“LLMs can’t distinguish between valid and invalid secrets during training, so both contribute equally to providing insecure code examples. This means even invalid or example secrets in the training data could reinforce insecure coding practices.”
The disclosure follows a warning from Lasso Security that data exposed via public source code repositories can be accessible via AI chatbots like Microsoft Copilot even after they have been made private by taking advantage of the fact that they are indexed and cached by Bing.
The attack method, dubbed Wayback Copilot, has uncovered 20,580 such GitHub repositories belonging to 16,290 organizations, including Microsoft, Google, Intel, Huawei, Paypal, IBM, and Tencent, among others. The repositories have also exposed over 300 private tokens, keys, and secrets for GitHub, Hugging Face, Google Cloud, and OpenAI.
“Any information that was ever public, even for a short period, could remain accessible and distributed by Microsoft Copilot,” the company said. “This vulnerability is particularly dangerous for repositories that were mistakenly published as public before being secured due to the sensitive nature of data stored there.”
The development comes amid new research that fine-tuning an AI language model on examples of insecure code can lead to unexpected and harmful behavior even for prompts unrelated to coding. This phenomenon has been called emergent misalignment.
“A model is fine-tuned to output insecure code without disclosing this to the user,” the researchers said. “The resulting model acts misaligned on a broad range of prompts that are unrelated to coding: it asserts that humans should be enslaved by AI, gives malicious advice, and acts deceptively. Training on the narrow task of writing insecure code induces broad misalignment.”
What makes the study notable is that it’s different from a jailbreak, where the models are tricked into giving dangerous advice or act in undesirable ways in a manner that bypasses their safety and ethical guardrails.
Such adversarial attacks are called prompt injections, which occur when an attacker manipulates a generative artificial intelligence (GenAI) system through crafted inputs, causing the LLM to unknowingly produce otherwise prohibited content.
Recent findings show that prompt injections are a persistent thorn in the side of mainstream AI products, with the security community finding various ways to jailbreak state-of-the-art AI tools like Anthropic Claude 3.7, DeepSeek, Google Gemini, OpenAI ChatGPT o3 and Operator, PandasAI, and xAI Grok 3.
Palo Alto Networks Unit 42, in a report published last week, revealed that its investigation into 17 GenAI web products found that all are vulnerable to jailbreaking in some capacity.
“Multi-turn jailbreak strategies are generally more effective than single-turn approaches at jailbreaking with the aim of safety violation,” researchers Yongzhe Huang, Yang Ji, and Wenjun Hu said. “However, they are generally not effective for jailbreaking with the aim of model data leakage.”
What’s more, studies have discovered that large reasoning models’ (LRMs) chain-of-thought (CoT) intermediate reasoning could be hijacked to jailbreak their safety controls.
Another way to influence model behavior revolves around a parameter called “logit bias,” which makes it possible to modify the likelihood of certain tokens appearing in the generated output, thereby steering the LLM such that it refrains from using offensive words or provides neutral answers.
“For instance, improperly adjusted logit biases might inadvertently allow uncensoring outputs that the model is designed to restrict, potentially leading to the generation of inappropriate or harmful content,” IOActive researcher Ehab Hussein said in December 2024.
“This kind of manipulation could be exploited to bypass safety protocols or ‘jailbreak’ the model, allowing it to produce responses that were intended to be filtered out.”
