Who could have possibly seen this coming? AI image generators have been used to createย non-consensual pornographyย of celebrities, but why would that same user base dare abuse a free AI text-to-speech deepfake voice generator?
UK-based ElevenLabsย first toutedย its Prime Voice AI earlier this month, but just a few weeks later it may need to rethink its entire model after users reported a slew of users creating hateful messages using real-world voices. The company released its first text-to-voice open beta system on Jan. 23. Developers promised the voices match the style and cadence of an actual human being. The company's "Voice Lab" feature lets users clone voices from small audio samples.
Motherboardย first reported Monday about a slew of 4Chan users who were uploading deepfaked voices of celebrities or internet personalities, from Joe Rogan to Robin Williams. One 4Chan user reportedly posted a clip of Emma Watson reading a passage of Mein Kampf. Another user took a voice reportedly sounding like Justin Roiland's Rick Sanchez fromย Rick & Mortyย talking about how he was going to beat his wife, an obvious reference toย current allegations of domestic abuseย against the series co-creator.
On one 4Chan thread reviewed by Gizmodo, users posted clips of the AI spouting intense misogyny or transphobia using voices of characters or narrators from various anime or video games. All this is to say, it's exactly what you would expect the armpit of the internet to do once it got its hands onย easy-to-use deepfake technology.
On Monday, the company tweeted that they had a "crazy weekend" and noted while their tech was "being overwhelmingly applied to positive use," developers saw an "increasing number of voice cloning misuse cases," but didn't name a specific platform or platforms where the misuse was taking place.
ElevenLabs offered a few ideas on how to limit the harm, including introducing some account verification, which could include a payment up front, or even dropping the free version of voice lab altogether, which would then mean manually verifying each cloning request.
Last week, ElevenLabs announced itย received $2 million in pre-seed fundingย led by Czech Republic-based Credo Ventures. The little AI-company-that-could was planning to scale up its operation and use the system in other languages as well, according to its pitch deck. This admitted misuse is a turn for developers who were very bullish on where the tech could go. The company spent the past week promoting its tech, saying its system could reproduceย Polish TV personalities. The company further promoted the idea ofย putting human audiobook narrators out of a job. The beta website talks up how the system could automate audio for news articles, or even create audio for video games.
The system can certainly mimic voices pretty well, just based on the voices reviewed by Gizmodo. A layperson may not be able to tell the difference between a faked clip of Rogan talking about porn habits versus anย actual clip from his podcast. Still, there's definitely a robotic quality to the sound that becomes more apparent on longer clips.
Gizmodo reached out to ElevenLabs for comment via Twitter, but we did not immediately hear back. We will update this story if we hear more.
There's plenty of other companies offering their own text-to-voice tool, but whileย Microsoft's similar VALL-E system is still unreleased, other smaller companies have been much less hesitant to open the door for anybody to abuse.ย AI experts have told Gizmodoย this push to get things out the door without an ethics review will continue to lead to issues like this.
Jump in the discussion.
No email address required.
๐ฎโ๐คฃโ๐จโ๐ฆโ๐ณโ๐ณโ๐ดโ๐นโ๐คฃโ๐นโ๐ญโ๐ฎโ๐ณโ๐ฐโ๐คฃโ๐ดโ๐ทโ๐คฃโ๐จโ๐ดโ๐ฒโ๐ตโ๐ทโ๐ชโ๐ญโ๐ชโ๐ณโ๐ฉโ๐คฃโ๐ดโ๐ซโ๐คฃโ๐ฆโ๐ณโ๐พโ๐นโ๐ญโ๐ฎโ๐ณโ๐ฌโ๐คฃโ๐ฒโ๐ดโ๐ทโ๐ชโ๐คฃโ๐จโ๐บโ๐จโ๐ฐโ๐ชโ๐ฉโ๐คฃโ๐นโ๐ญโ๐ฆโ๐ณโ๐คฃโ๐งโ๐ชโ๐ฎโ๐ณโ๐ฌโ๐คฃโ๐ฆโ๐ณโ๐คฃโ๐ทโโ๐ฉโ๐ทโ๐ฆโ๐ฒโ๐ฆโ๐คฃโ๐ธโ๐บโ๐งโ๐ธโ๐จโ๐ทโ๐ฎโ๐งโ๐ชโ๐ทโโ๐คฃโ๐ญโ๐ดโ๐ณโ๐ชโ๐ธโ๐นโ๐ฑโ๐พโโ๐คฃโ๐นโ๐ญโ๐ฎโ๐ณโ๐ฐโ๐คฃโ๐ฆโ๐งโ๐ดโ๐บโ๐นโ๐คฃโ๐ฎโ๐นโ๐คฃโ๐ทโ๐ฆโ๐นโ๐ฎโ๐ดโ๐ณโ๐ฆโ๐ฑโ๐ฑโ๐พโโ๐คฃโ๐พโ๐ดโ๐บโ๐คฃโ๐ฆโ๐ทโ๐ชโ๐คฃโ๐ตโ๐ดโ๐ธโ๐นโ๐ฎโ๐ณโ๐ฌโโ๐คฃโ๐จโ๐ดโ๐ฒโ๐ฒโ๐ชโ๐ณโ๐นโ๐ฎโ๐ณโ๐ฌโโ๐คฃโ๐ฆโ๐ณโ๐ฉโ๐คฃโ๐บโ๐ตโ๐ปโ๐ดโ๐นโ๐ฎโ๐ณโ๐ฌโ๐คฃโ๐นโ๐ทโ๐ฆโ๐ธโ๐ญโ๐คฃโ๐ซโ๐ดโ๐ทโ๐คฃโ๐ฆโ๐นโ๐คฃโ๐ฑโ๐ชโ๐ฆโ๐ธโ๐นโ๐คฃโโโ๐คฃโ๐พโ๐ชโ๐ฆโ๐ทโ๐ธโ๐คฃโ๐ธโ๐ดโ๐ฑโ๐ชโ๐ฑโ๐พโ๐คฃโ๐ธโ๐ดโ๐คฃโ๐นโ๐ญโ๐ชโ๐คฃโ๐ธโ๐บโ๐งโ๐คฃโ๐จโ๐ฆโ๐ณโ๐คฃโ๐ฌโ๐ดโ๐คฃโ๐ฆโ๐ณโ๐ฉโ๐คฃโ๐ฌโ๐ชโ๐นโ๐คฃโ๐ฌโ๐ฆโ๐ฌโ๐ฌโ๐ชโ๐ฉโ๐คฃโ๐งโ๐พโ๐คฃโ๐ฆโ๐ณโ๐ดโ๐นโ๐ญโ๐ชโ๐ทโ๐คฃโ๐ฏโ๐ฆโ๐ณโ๐ณโ๐ฎโ๐ชโโ๐คฃโ๐ฆโ๐ฑโ๐ฑโ๐คฃโ๐นโ๐ญโ๐ชโ๐คฃโ๐ญโ๐ฆโ๐ทโ๐ฉโ๐คฃโ๐ผโ๐ดโ๐ทโ๐ฐโ๐คฃโ๐พโ๐ดโ๐บโ๐คฃโ๐ตโ๐บโ๐นโ๐คฃโ๐ฎโ๐ณโ๐นโ๐ดโ๐คฃโ๐พโ๐ดโ๐บโ๐ทโ๐คฃโ๐งโ๐ชโ๐ฆโ๐บโ๐นโ๐ฎโ๐ซโ๐บโ๐ฑโ๐คฃโ๐ฑโ๐ฎโ๐นโ๐นโ๐ฑโ๐ชโ๐คฃโ๐ธโ๐บโ๐งโ๐ทโ๐ชโ๐ฉโ๐ฉโ๐ฎโ๐นโ๐คฃโโ๐คฃโ๐ทโ๐ชโ๐ฆโ๐ฉโ๐ฎโ๐ณโ๐ฌโ๐คฃโ๐นโ๐ญโ๐ชโ๐คฃโ๐ธโ๐ญโ๐ฎโ๐นโ๐ตโ๐ดโ๐ธโ๐นโ๐ธโ๐คฃโโ๐คฃโ๐ญโ๐ดโ๐บโ๐ทโ๐ธโ๐คฃโ๐ตโ๐ฆโ๐ธโ๐นโ๐คฃโ๐งโ๐ชโ๐ฉโ๐นโ๐ฎโ๐ฒโ๐ชโโ๐คฃโ๐ธโ๐จโ๐ดโ๐บโ๐นโ๐ฎโ๐ณโ๐ฌโ๐คฃโ๐ดโ๐บโ๐นโ๐คฃโ๐ฉโ๐ทโ๐ฆโ๐ฒโ๐ฆโโ๐คฃโ๐ฒโ๐ฆโ๐ฐโ๐ฎโ๐ณโ๐ฌโ๐คฃโ๐ชโ๐ซโ๐ซโ๐ดโ๐ทโ๐นโ๐คฃโ๐ตโ๐ดโ๐ธโ๐นโ๐ธโโ๐คฃโ๐ฆโ๐ทโ๐ฌโ๐บโ๐ฎโ๐ณโ๐ฌโ๐คฃโ๐ผโ๐ฎโ๐นโ๐ญโ๐คฃโ๐ตโ๐ฎโ๐ฟโ๐ฟโ๐ฆโ๐ธโ๐ญโ๐ฎโ๐ฑโ๐ฑโโ๐คฃโ๐ธโ๐บโ๐จโ๐ฐโ๐ฎโ๐ณโ๐ฌโ๐คฃโ๐ดโ๐ซโ๐ซโ๐คฃโ๐นโ๐ทโ๐ฆโ๐ตโ๐ตโ๐พโโ๐คฃโ๐ฆโ๐ฑโ๐ฑโ๐คฃโ๐ดโ๐ซโ๐คฃโ๐ฎโ๐นโ๐คฃโ๐ญโ๐ฆโ๐ธโ๐คฃโ๐ดโ๐ณโ๐ชโ๐คฃโ๐ธโ๐ฎโ๐ฒโ๐ตโ๐ฑโ๐ชโ๐คฃโ๐ทโ๐ชโ๐ธโ๐บโ๐ฑโ๐นโโ๐คฃโ๐นโ๐ญโ๐ชโ๐คฃโ๐ธโ๐บโ๐งโ๐คฃโ๐ฎโ๐ธโ๐คฃโ๐ฒโ๐ดโ๐ทโ๐ชโ๐คฃโ๐ชโ๐ณโ๐ฏโ๐ดโ๐พโ๐ฆโ๐งโ๐ฑโ๐ชโ๐คฃโ๐ซโ๐ดโ๐ทโ๐คฃโ๐นโ๐ญโ๐ชโ๐คฃโ๐ฌโ๐ฎโ๐ฌโ๐ฆโ๐ฏโ๐ฆโ๐ณโ๐ณโ๐ฎโ๐ชโ๐ธโ๐คฃโ๐นโ๐ญโ๐ฆโ๐นโ๐คฃโ๐ผโ๐ฎโ๐ฑโ๐ฑโ๐คฃโ๐ชโ๐ปโ๐ชโ๐ณโ๐นโ๐บโ๐ฆโ๐ฑโ๐ฑโ๐พโ๐คฃโ๐ซโ๐บโ๐จโ๐ฐโ๐คฃโ๐ฎโ๐นโ๐คฃโ๐ฎโ๐ณโ๐คฃโ๐ชโ๐ปโ๐ชโ๐ทโ๐พโ๐คฃโ๐ญโ๐ดโ๐ฑโ๐ชโโโโ๐นโ๐ญโ๐ชโ๐คฃโ๐ตโ๐ชโ๐ทโ๐ซโ๐ชโ๐จโ๐นโ๐คฃโ๐ฉโ๐ทโ๐ฆโ๐ฒโ๐ฆโ๐คฃโ๐ธโ๐บโ๐งโโ๐คฃโ๐ฌโ๐ทโ๐ชโ๐ฆโ๐นโโ๐คฃโ๐ผโ๐ญโ๐ดโ๐คฃโ๐งโ๐ชโ๐ณโ๐ชโ๐ซโ๐ฎโ๐นโ๐ธโโ๐คฃโ๐ฎโ๐ซโ๐คฃโ๐พโ๐ดโ๐บโโ๐ทโ๐ชโ๐คฃโ๐ฑโ๐บโ๐จโ๐ฐโ๐พโโ๐คฃโ๐ฆโ๐คฃโ๐ฒโ๐ฆโ๐ฉโ๐ฒโ๐ฎโ๐ณโ๐คฃโ๐ผโ๐ญโ๐ดโ๐คฃโ๐จโ๐ฆโ๐ณโโ๐นโ๐คฃโ๐ถโ๐บโ๐ฎโ๐นโ๐ชโ๐คฃโ๐งโ๐ทโ๐ฎโ๐ณโ๐ฌโ๐คฃโ๐ญโ๐ฎโ๐ฒโ๐ธโ๐ชโ๐ฑโ๐ซโ๐คฃโ๐นโ๐ดโ๐คฃโ๐งโ๐ฆโ๐ณโ๐คฃโ๐นโ๐ญโ๐ชโ๐คฃโ๐ธโ๐บโ๐งโ๐ทโ๐ชโ๐ฉโ๐ฉโ๐ฎโ๐นโ๐คฃโ๐งโ๐ชโ๐จโ๐ฆโ๐บโ๐ธโ๐ชโ๐คฃโ๐ญโ๐ชโ๐คฃโ๐ธโ๐ชโ๐จโ๐ทโ๐ชโ๐นโ๐ฑโ๐พโ๐คฃโ๐ฑโ๐ดโ๐ปโ๐ชโ๐ธโ๐คฃโ๐ฎโ๐นโ๐คฃโ๐ผโ๐ฎโ๐ฑโ๐ฑโ๐คฃโ๐ผโ๐ทโ๐ฎโ๐นโ๐ชโ๐คฃโ๐ธโ๐ตโ๐ชโ๐จโ๐ฎโ๐ฆโ๐ฑโ๐คฃโ๐จโ๐ดโ๐ฉโ๐ชโ๐คฃโ๐นโ๐ดโ๐คฃโ๐งโ๐ฑโ๐ดโ๐จโ๐ฐโ๐คฃโ๐ตโ๐ฎโ๐ณโ๐ฌโ๐ฎโ๐ณโ๐ฌโโ๐คฃโ๐ญโ๐ชโ๐คฃโ๐ฌโ๐ชโ๐นโ๐ธโ๐คฃโ๐นโ๐ดโ๐คฃโ๐ทโ๐ชโ๐ฆโ๐ฉโ๐คฃโ๐ฆโ๐ฑโ๐ฑโ๐คฃโ๐นโ๐ญโ๐ชโ๐คฃโ๐ฉโ๐ชโ๐ฑโ๐ชโ๐นโ๐ชโ๐ฉโ๐คฃโ๐ตโ๐ดโ๐ธโ๐นโ๐ธโโ๐คฃโ๐ญโ๐ชโโ๐ธโ๐คฃโ๐นโ๐ญโ๐ชโ๐คฃโ๐ดโ๐ณโ๐ชโ๐คฃโ๐ผโ๐ญโ๐ดโ๐คฃโ๐ฌโ๐ชโ๐นโโ๐ธโ๐คฃโ๐ตโ๐ฆโ๐ฎโ๐ฉโ๐คฃโ๐ซโ๐ดโ๐ทโ๐คฃโ๐ฆโ๐ฑโ๐ฑโ๐คฃโ๐นโ๐ญโ๐ดโ๐ธโ๐ชโ๐คฃโ๐งโ๐ฆโ๐ณโ๐คฃโ๐ฆโ๐ผโ๐ฆโ๐ทโ๐ฉโ๐ธโ๐คฃโ๐พโ๐ดโ๐บโ๐คฃโ๐ญโ๐ฆโ๐ณโ๐ฉโ๐ชโ๐ฉโ๐คฃโ๐ดโ๐บโ๐นโโโโ๐ฆโ๐ธโ๐คฃโ๐ฆโ๐ณโ๐คฃโ๐ทโโ๐ฉโ๐ทโ๐ฆโ๐ฒโ๐ฆโ๐คฃโ๐ธโ๐บโ๐งโ๐ธโ๐จโ๐ทโ๐ฎโ๐งโ๐ชโ๐ทโโ๐คฃโ๐พโ๐ดโ๐บโ๐คฃโ๐ฆโ๐ทโ๐ชโ๐คฃโ๐ฑโ๐ฎโ๐นโ๐ชโ๐ทโ๐ฆโ๐ฑโ๐ฑโ๐พโ๐คฃโ๐ฉโ๐ชโ๐ฉโ๐ฎโ๐จโ๐ฆโ๐นโ๐ฎโ๐ณโ๐ฌโ๐คฃโ๐ฆโ๐นโ๐คฃโ๐ฑโ๐ชโ๐ฆโ๐ธโ๐นโ๐คฃโโโ๐คฃโ๐พโ๐ชโ๐ฆโ๐ทโ๐ธโ๐คฃโ๐ดโ๐ซโ๐คฃโ๐พโ๐ดโ๐บโ๐ทโ๐คฃโ๐ฑโ๐ฎโ๐ซโ๐ชโ๐คฃโ๐ธโ๐ฎโ๐ฒโ๐ตโ๐ฑโ๐พโ๐คฃโ๐นโ๐ดโ๐คฃโ๐ผโ๐ทโ๐ฎโ๐นโ๐ชโ๐คฃโ๐ธโ๐ญโ๐ฎโ๐นโ๐ตโ๐ดโ๐ธโ๐นโ๐คฃโ๐ซโ๐ดโ๐ทโ๐คฃโ๐ฆโ๐ชโ๐ดโ๐คฃโ๐นโ๐ดโ๐คฃโ๐ชโ๐ณโ๐ฏโ๐ดโ๐พโโ๐คฃโ๐ฎโ๐นโ๐คฃโ๐ฎโ๐ธโ๐คฃโ๐นโ๐ญโ๐ชโ๐คฃโ๐บโ๐ฑโ๐นโ๐ฎโ๐ฒโ๐ฆโ๐นโ๐ชโ๐คฃโ๐ฆโ๐ณโ๐ฉโ๐คฃโ๐ซโ๐ฎโ๐ณโ๐ฆโ๐ฑโ๐คฃโ๐จโ๐บโ๐จโ๐ฐโโ๐คฃโ๐นโ๐ญโ๐ฎโ๐ณโ๐ฐโ๐คฃโ๐ฆโ๐งโ๐ดโ๐บโ๐นโ๐คฃโ๐ฎโ๐นโ๐คฃโ๐ฑโ๐ดโ๐ฌโ๐ฎโ๐จโ๐ฆโ๐ฑโ๐ฑโ๐พ
Snapshots:
archive.org
ghostarchive.org
archive.ph (click to archive)
non-consensual pornography:
archive.org
ghostarchive.org
archive.ph (click to archive)
first touted:
archive.org
ghostarchive.org
archive.ph (click to archive)
Motherboard:
archive.org
ghostarchive.org
archive.ph (click to archive)
current allegations of domestic abuse:
archive.org
ghostarchive.org
archive.ph (click to archive)
easy-to-use deepfake technology:
archive.org
ghostarchive.org
archive.ph (click to archive)
received $2 million in pre-seed funding:
archive.org
ghostarchive.org
archive.ph (click to archive)
Polish TV personalities:
archive.org
ghostarchive.org
archive.ph (click to archive)
putting human audiobook narrators out of a job:
archive.org
ghostarchive.org
archive.ph (click to archive)
actual clip from his podcast:
archive.org
ghostarchive.org
archive.ph (click to archive)
Microsoft's similar VALL-E system is still unreleased:
archive.org
ghostarchive.org
archive.ph (click to archive)
AI experts have told Gizmodo:
archive.org
ghostarchive.org
archive.ph (click to archive)
Jump in the discussion.
No email address required.
More options
Context