DEF CON villages are in trouble
This year, I skipped the main stage tracks at DEF CON to catch what promised to be interesting talks at some of the villages—and to be clear, some of those talks were interesting!
Unfortunately, roughly half the talks I went to were either AI slop or a sales pitch dressed up as a conference talk (collectively what I refer to in this post as “slop talks”). And while my own sample size was small (4 good talks, 3 slop talks), the complaints from my team in our group chat provided a steady stream of updates about the slop talks they encountered.
That said, I empathize with how much effort must go into putting together a DEF CON village. So before I go into the slop talks problem, I want to celebrate the noteworthy talks I saw this year.
The good talks
On Friday, I attended a talk from Aarav Juneja titled, “Man-in-the-Browser: Hijacking Javascript APIs for Browser Persistence and Credential Theft”. Aarav is going into his Junior year in High School, and opted to give a number of live demos during the talk where he proved his points. This young man is going to go far in the industry if he keeps hacking like that!
I also caught Meitar Ronen’s talk, “Most threat modeling artifacts don’t help coding agents fix business logic. Here’s what does” on Friday. It was certainly amusing to see just how much of Mei’s research lined-up with what we saw in our research at Off-by-1 Labs. And while the slides from the talk were heavily employer branded, it did not come across as a vendor pitch—which can be a tough needle to thread under the circumstances.
On Saturday I went to my sensei Jason Haddix’ talk about “Building Hackbots”. Per usual, Jason’s talk delivered a ton of value. Unfortunately I couldn’t get a seat at the workshop he was delivering later in the day, but the talk certainly gave me encouragement to go back through the course from Arcanum InfoSec’s online class covering the same material. It was easy to see just how much human effort Jason put into his research for this talk, given how many pro tips he dropped.
I also caught Jenn Gile’s talk “How Malicious AI Skills Hijack Your Agents”. I really love what Jenn and Paul are doing over at OpenSourceMalware, and it was great to see all of the receipts that Jenn shared in her talk about what has been happening over the last ~8 months in this space. This talk is a “must watch” talk for defenders at companies with heavy AI use—especially if that usage is amongst developers.
The slop talks
If you’re looking for “naming and shaming”, sorry—you’re not going to get that from me. That said, to help address this problem, I’ll be publishing a follow-up post to provide guidance for first-time speakers on how to build talks that doesn’t get accused of being slop. For now, here are some anecdotes about the slop talks I walked out of this year which led to this post.
The first slop talk I encountered was, perhaps ironically, discussing the problem of AI slop. I should have known better, given the length of the abstract and overuse of the : character mid-sentence. I don’t know a single human that writes using this character as grammar outside of scientific/research papers, and I suspect it may have something to do with the watermarking that Anthropic has introduced. Anyway, when I got to the talk and saw the slides, it became really clear to me that the whole thing was AI generated.
You’ve probably seen the kinds of slides/diagrams I’m talking about. Those diagrams where they use rounded-rectangle boxes with text in them and arrow characters pointing between boxes to represent steps in a process. Yeah… all of those were almost certainly generated by Claude.
The second slop talk I encountered reminded me of the way people describe how it feels when they have a bad trip from psilocybin mushrooms. I’ve never used drugs, mind you, but the slides shared and the meandering style of the speaker making disjointed claims was like watching a raving lunatic trying to give a talk about security. They went so far as to point out how CodeQL—a tool I was formerly an expert on—failed to identify the kinds of vulnerabilities that it was never advertised or designed to uncover, and that their AI-generated coding solution was a far superior tool.
The last slop talk was possibly the worst, as it involved a vendor using heavily branded slides to effectively give a long-form version of a booth demo at Black Hat. Even the inflection of their voice lilted upward at the end of every statement they were making, just like what you hear at the booth. How this made it past the CFP review board, and was allowed to continue being delivered after it was clear they were giving a sales pitch at the village, is beyond me.
How villages might address the slop talk problem
I’ll start with stating this is not about gatekeeping; this is absolutely about quality control.
Heck, I gave a talk this year at the N00b Village geared toward helping new people break into the industry, and I believe there are ways that villages can control for this problem without introducing gatekeeping measures. Unfortunately, the “slop talks” I encountered were delivered by people nobody had ever heard of, and upon light inspection of their online presence, didn’t appear to have a lot of verifiable industry experience.
So then how can villages improve talk quality at DEF CON without introducing gatekeeping in the process? Below are a few ideas I had on the plane ride home that could help improve things.
Due to the slop talk problem, the industry needs to begin moving toward adopting a reputation system. One option might be to pair-up well known and established speakers with first-time speakers as part of a “final review” process one month before the conference. This would distribute the burden of reviewing slides instead of having that work land on village organizers. It would also mean connecting new speakers with experienced ones, and could be included in the talk details as an “endorsed by” line. This might even drive more attendance to talks from first-time speakers—heck, I’d definitely go to a talk if it was endorsed by someone like Jason Haddix, James Kettle, Olivia Gallucci or similar!
To get around the issue of having enough proven speakers available, villages could selectively invite some number of speakers with the caveat that they are required to review and endorse (or refuse to endorse) a talk from a first-time speaker in order to avoid slop talks. There would need to be some form of appeals process or automated flag for review from organizers in order to prevent abuse of this system, but the “happy path” here has a lot of benefits for villages, as well as both established and new speakers alike.
Villages could likewise enforce a slide review process of their own for first-time speakers one month before the conference, and have talks lined up to be delivered in a pinch by invited, proven speakers as backups. This places the greatest burden on village volunteers out of all the options I’m proposing, and may lead to gatekeeping first-time speakers from presenting at the conference as an unwanted side-effect.
Maybe the solution is to move to an invite-only system for speakers, where organizers look to recruit either proven speakers, or speakers from smaller events like BSides and other local / regional conferences. This feels like the most gatekeeping-ish option, and unfortunately only distributes the burden of proof to smaller conferences that usually have fewer resources.
Finally, villages could just schedule first-time speakers during the Sunday time slots. This is definitely the shittiest way to treat first-time speakers, but it requires the least amount of effort from village organizers who are already beyond capacity. And yes, this means that fewer people will see the talk live—but it also reduces the likelihood that a larger audience sits through a slop talk.
In an age where it’s incredibly easy to juice one’s online presence with AI, having an easy way to vet reputation through provable experience is where we now have to set the bar. But that does not mean villages should pass on talks from first-time speakers.
First-time speakers are not synonymous with sloppy speakers
That said, just because someone hasn’t presented at a well-known conference doesn’t mean they shouldn’t give a presentation at a DEF CON village. If anything, Aarav Juneja’s talk is a great example of someone who absolutely belongs on stage at DEF CON’s AppSec Village. In short, we need more voices like his contributing to this community–not less.
That said, it’s far easier today to make a talk abstract, slide content, and speaker’s notes using AI. And while that means we’re likely to see more slop talks and sloppy speakers in the future, not every first-time speaker is guaranteed to give a slop talk. The next few years are going to be a real challenge for conferences, as many events are going to need to figure out how to weed these talks out of their conference without introducing gatekeeping measures.
Talks from vendors are not synonymous with sales pitches
I saw three talks from vendors this year, and only one of them was a slop talk. Given that it’s always a bit dicey to have a vendor give a talk at a conference due to the risk of them delivering a sales pitch on stage, this is perhaps the only place where I’m willing to encourage some gatekeeping by village organizers.
I think a mandatory slide review for first-time vendor speakers one month before the conference, with clear instructions on how many slides are allowed to include vendor branding, is a great place to start. BSidesSF does an excellent job of screening for this and removing vendor pitches from their schedule. Likewise, Gadi Evron was an absolute legend at [un]prompted earlier this year when he interrupted a vendor pitch-in-progress and kicked them off stage mid-talk.
Villages shouldn’t be afraid to halt a talk if a vendor is pitching to their attendees. And while it only happened to me just once this year, it was enough to leave a bad impression of the village as a result. Gatekeeping individuals should always be frowned upon—but gatekeeping vendors? Villages should be enforcing strict rules for vendors with great prejudice.
What I’m doing next
I’ve been in the industry professionally for over a decade. I’ve also volunteered at, and served as an organizer for my local BSides conference. Given all of my past experiences, I will be rolling up my proverbial sleeves and volunteering my time to help with Call for Paper (CFP) and slide reviews for at least one DEF CON village next year.
It’s easy to throw stones at glass houses; it’s a lot harder when you have to live in them. Here’s hoping that with ideas I’ve shared above, and through the efforts of professionals who have the necessary skills and experiences to support these villages, we can collectively prevent slop talks from appearing at DEF CON 35.
Thank you to Olivia, Roni and others for providing feedback and suggestions on this post—and thank you for reading it! 😊 While I consider my next blog post, you can git checkout other (usually off-topic) content I’m reading over at Instapaper.
Until next time, remember to git commit && stay classy!
Cheers,