AI Red Teaming and AI Safety – Sounil Yu, Amanda Minnich – ESW #371
Full Audio
View Show IndexSegments
1. AI Red Teaming and AI Safety – Amanda Minnich – ESW #371
In this interview we explore the new and sometimes strange world of redteaming AI. I have SO many questions, like what is AI safety?
We'll discuss her presence at Black Hat, where she delivered two days of training and participated on an AI safety panel.
We'll also discuss the process of pentesting an AI. Will pentesters just have giant cheatsheets or text files full of adversarial prompts? How can we automate this? Will an AI generate adversarial prompts you can use against another AI? And finally, what do we do with the results?
Resources:
Guest
Dr. Amanda Minnich is a Senior AI Security Researcher at Microsoft on the Microsoft AI Red Team, where she red teams Microsoft’s foundational models and Copilots for safety and security vulnerabilities. Prior to Microsoft, Dr. Minnich worked at Twitter, focusing on identifying international election interference and other types of abuse and spam campaigns using graph clustering algorithms. She also previously worked at Sandia National Laboratories and Mandiant, where she applied machine learning research techniques to malware classification and malware family identification. Dr. Minnich is heavily involved with tech outreach efforts, especially for women in tech. She received her MS and PhD in Computer Science with Distinction from the University of New Mexico.
Host
2. Interviewing Black Hat Startup Spotlight Winner, Knostic – Sounil Yu – ESW #371
We chat with Sounil Yu, co-founder of LLM access control startup, Knostic. We discuss both the experience of participating in Black Hat's startup competition, and what his company, Knostic, is all about. Knostic was one of four finalists for Black Hat's Startup Spotlight competition and was announced as the winner on August 6th.
References
Guest
Sounil Yu is the CTO and Co-Founder of Knostic. He is the creator of the Cyber Defense Matrix and the DIE Triad. Previously, he was Bank of America’s Chief Security Scientist and the CISO at JupiterOne. He is a FAIR Institute Board Member and a GMU National Security Institute fellow.
Hosts
3. More AI funding, Crowdstrike ripples continue, GPT yourself – ESW #371
In the enterprise security news,
- AI is still getting a ton of funding!
- Netwrix acquires PingCastle
- Tenable looks for a buyer
- SentinelOne hires Alex Stamos as their new CISO
- Crowdstrike doesn’t appreciate satire when it’s at their expense
- Intel begins one of the biggest layoffs we’ve ever seen in tech
- Windows Downdate
- RAG poisoning
- GPT yourself
- The Xerox Hypothesis
All that and more, on this episode of Enterprise Security Weekly.
Announcements
We're always looking for great guests for all of the Security Weekly shows! Submit your suggestions by visiting https://securityweekly.com/guests and completing the form!
Hosts
- 1. FUNDING: Abnormal’s AI-Native Cybersecurity: Building AI to Fight AI
$250M Series D at a $5.1B valuation, led by Wellington Management (PE firm). Abnormal states that its mission is to "protect humans with AI". So far, Abnormal has been focused on email security, but say they will use this investment to expand into new security markets.
They've been a "unicorn" for years now. Would be a pretty big exit if they got acquired, but no obvious acquirers come to mind. Cisco? Crowdstrike and Palo Alto seem to prefer buying early stage. Microsoft desperately needs the help, M365 email security is hilariously bad.
- 2. FUNDING: Protect AI Raises $60M in Series B Financing to Secure Artificial Intelligence and Machine Learning from Unique Security Risks
$60M Series B, led by Evolution Equity Partners. Total of $108.5M in funding. "AI security posture management platform".
Protect AI also acquired SydeLabs, and gains from them, the ability to 'red team' LLMs. The company lists a surprising number of products for a company in this early a stage:
- Guardian: Zero trust for AI models
- Layer: End-to-end LLM security and governance monitoring
- Recon: Automated red teaming of GenAI (from SydeLabs)
- Radar: AI risk assessment and management
- Sightline: "The first AI supply chain vulnerability database"
as well as Open Source:
- LLM Guard: "Secure your LLM applications"
- ModelScan: "A scanner for all formats"
- NB Defense: Secure Jupyter Notebooks
- 3. FUNDING: Proud Moment: Anjuna Closes $25M to Continue Shaping the Future of Confidential AI and Secure Collaboration
$25M Series A, led by M Ventures, SineWave Ventures, and AI Capital Partners. They appear to be building confidential computing for LLMs? "Anjuna Confidential Containers" are "secure enclave-ready hardened images" that leverage your original application without requiring any code changes.
- 4. ACQUISITION: Netwrix Acquires PingCastle to Empower Customers with Better Protection of Active Directory and Entra ID
Looks like a one-person business around a self-developed tool got acquired. Not to say that's a bad thing - Metasploit, Maltego, and BurpSuite all came about the same way!
- 5. ACQUISITION RUMORS: Cybersecurity firm Tenable exploring potential sale, Bloomberg News reports
A few months back it was Rapid7 and nothing came of it, so don't get too excited yet.
My prediction? Eventually, one of the big 3 (Qualys, Rapid7, or Tenable) goes private with a private equity firm, who also picks up a CAASM vendor (Axonius, Panaseer, JupiterOne, runZero), and smooshes them together.
- 6. NEW STAFF: SentinelOne® Names Alex Stamos Chief Information Security Officer
- 7. LAYOFFS: Intel severance: Chipmaker sets terms for buyouts, early retirement and layoffs
A watershed moment, but I include this to talk about a bigger potential impact. I've talked to folks who worry that all these layoffs have actually made organizations easier targets for cyberattacks. I doubt we have enough data to properly analyze that theory, but it's worth considering, especially for those doing the layoffs.
Unfortunately, through no fault of their own, I suspect a lot of security practitioners aren't doing a lot to reduce risk or prevent attacks...
- 8. VULNERABILITIES: Windows Update Flaws Allow Undetectable Downgrade Attacks
This is an interesting one to discuss - particularly, how and in what scenario would you use this vulnerability?
- 9. AI SECURITY: RAG Poisoning: All You Need is One Document
- 10. AI SECURITY: AI Security Shared Responsibility Model: Navigating Risks in AI Deployment
- 11. OPEN SOURCE: adrianco/meGPT
Something Tyler is definitely going to be interested in, from one of the better Adrians out there (as an Adrian, I know these things), Adrian Cockroft!
- 12. DUMPSTER FIRES: CrowdStrike trying to use legal threats to suppress criticism and parody of global IT outage
- 13. DUMPSTER FIRES: Massive CrowdStrike outage caused by an out-of-bounds memory error
The final investigation report has been released. TL;DR:
- Expecting 21 parameters when there were only 20 led to an out-of-bounds memory write in the kernel's memory space. Very bad bad not good.
- Testing didn't catch it, so they're doing more better testing now.
- They're now going to allow customers to slow down rapid response content updates. Sure, that makes sense.
- 14. RESEARCH: Vulnerability Exploitation in the Wild
I think it's even worse than this, see yesterday's PSW for my thoughts in response to some other recent vuln management research.
- 15. SQUIRREL: The Xerox Hypothesis
A fascinating incident where some academic research was called into question, and the culprit turned out to be Xerox scanners CHANGING DATA by mistake. The reason has to do with a pattern matching trick that some scanners use to use less memory. When the pattern matching SUCKS, as it did in this case, sometimes an O becomes a Q, an L becomes an I, and a 6 becomes an 8.