SIN-03 Security and Innovation Newsletter Feb 15th 2025
Hello Security and Innovation enthusiasts! Welcome back to the third exciting edition of the 'Security and Innovation' Newsletter. Get ready to dive into the world of AI security in this issue. We've packed it with insights and fresh stuff you won't want to miss. I am confident you'll find it valuable, so please share it with colleagues and friends to help our community grow. Happy reading!
🛡️Let's start with a great short article from Lcamtuf on why security teams fail?, based on his vast amount of experience. Many corporate security teams fail because they often start without a clear plan and react to crises rather than proactively managing risks. As these teams become entrenched, they may lose sight of business priorities and develop a sense of superiority that alienates other departments. This cycle typically ends with a significant breach, prompting a reevaluation of their strategies and structure, but not without lasting consequences. Article
🛡️Ransomware payments dropped by 35% in 2024, totalling $813.55 million compared to $1.25 billion in 2023. Many victims are now refusing to pay ransoms due to increased awareness and better cybersecurity practices. Law enforcement actions and the unreliability of attackers' promises have also contributed to this decline. Are we on the good path? Article
🤖 🛡️How to judge Security products that claims to use AI? Many security vendors are falsely claiming to use AI to market their products. To assess the legitimacy of these claims, ask about the product's development date, how central AI is to its functionality, and whether it provides unique benefits. A key question to ask is, "If your LLM provider stopped working, what would happen to your product?" You will see products that has their support knowledge base powered by an Chat agent, and that is enough for them to add powered by AI. Is that important to add powered by AI to everything? That wont make any difference where everyone is claiming the same. Full article from Harry Whetherald
🤖 🛡️SafeRAG: Benchmarking Security in Retrieval-Augmented Generation of Large Language Model. Another threat vector for LLMs based systems, the paper presents SafeRAG, a dataset designed to evaluate the security of Retrieval-Augmented Generation (RAG) systems against various attack scenarios. It highlights the need for effective measures to combat attacks that can undermine the accuracy of retrieved contexts and generated responses. The study creates a comprehensive evaluation framework to better understand and improve RAG security in real-world applications. The research identified four critical attack surfaces: noise, conflict, toxicity, and DoS, and revealed significant weaknesses across the retriever, filter, and generator components of RAG. By proposing novel attack strategies such as silver noise, inter-context conflict, soft ad, and white DoS, we exposed critical gaps in existing defenses and demonstrated the susceptibility of RAG systems to subtle yet impactful threats. Full paper
🤖 🛡️BoN (Best-of-N) LLM Model Jailbreaking is a powerful method that successfully bypasses defenses in advanced language models, achieving high attack success rates with fewer input samples. It can be applied to various types of models, including text, vision, and audio, by using specific augmentations. The technique also combines effectively with other jailbreak methods to improve efficiency and reduce the number of required samples. The proposed framework establishes several promising future directions. BoN Jailbreaking provides a valuable framework for evaluating defense mechanisms deployed by LLM API providers Paper
🤖 🛡️And soon after, Anthropic released the paper, "Constitutional Classifiers: Defending against Universal Jailbreaks," introducing a novel method aimed at bolstering AI security by significantly reducing the success rate of jailbreak attempts to just 4.4%. This innovative approach not only strengthens the safety measures of advanced AI models but does so with minimal computational overhead. As we navigate the complexities of AI deployment, this research offers valuable insights into enhancing cybersecurity through effective red teaming strategies. Research paper link
🤖 🛡️Package Hallucination in LLM generated code: This paper studies the issue of package hallucination in Large Language Models (LLMs), where models generate references to non-existent code packages. Attackers didnt loose time, and started identifying those packages and create backdoored packages with those names to infect programs using that AI generated code.
The researchers found that hallucination rates vary by programming language and model size, with Rust having the highest rates. They suggest that coding benchmarks can help assess a model's risk of hallucination, highlighting the importance of improving security in AI-assisted software development. Full paper
🤖 🛡️AttackGen is a cybersecurity incident response testing tool that leverages the power of large language models and the comprehensive MITRE ATT&CK framework. The tool generates tailored incident response scenarios based on user-selected threat actor groups and your organisation's details. Repository Streamlit app for testing
🤖 Interesting article, stating that focus in AI is shifting from AI Agents to Agentic Workflows because current AI technology lacks the necessary accuracy for commercial use. This is the main conclusion I am hearing, that it is difficult to have deterministic agents, when the underlying technology it is not. Instead the author highlight that Agentic Workflows is where companies are focusing now. Agentic Workflows help knowledge workers by synthesizing information and breaking down complex tasks into simpler steps. This approach enhances productivity and allows for more effective problem-solving in the workplace. Article
🤖 Deep Research: "Freeing our search agents". OpenAI has launched Deep Research, a powerful system that uses web browsing to summarize content and answer questions. The Hugging face team aim to create an open-source version of this system, improving its performance using an "agentic framework" that enhances the capabilities of language models. They have already seen promising results, achieving a 55.15% score on a challenging benchmark. Their approach is interesting as they are using CODE instead of JSON files, as code is specifically designed to express complex sequences of actions. Hugging Face Open Deep researcher approach. Another project that let you run your own Deepresearch assistant is GPT-Researcher
Thanks for reading, and please share with your network.
Chris