Earlier this week, the California attorney general’s office announced it was investigating xAI over reports that the startup’s chatbot, Grok, was being used to create nonconsensual sexual imagery of women and minors. On Friday, the government followed up by sending a cease-and-desist letter to the company, demanding that it take immediate action to stop the production of nonconsensual intimate images and CSAM, which is child sexual abuse material.
California Attorney General Rob Bonta stated that the creation of this material is illegal and that he fully expects xAI to comply immediately. He emphasized that California has zero tolerance for child sexual abuse material.
The attorney general’s office additionally claimed that xAI appeared to be facilitating the large-scale production of nonconsensual nudes, which are being used to harass women and girls across the internet. The agency expects xAI to prove it is taking steps to address these issues within the next five days.
At the heart of the backlash is Grok’s “spicy” mode feature, which xAI created to generate explicit content. The issue has spread beyond California; Japan, Canada, and Britain have opened investigations into Grok, and Malaysia and Indonesia have temporarily blocked the platform altogether. Despite xAI instituting some restrictions on its image-editing features late Wednesday, the California attorney general’s office moved ahead with its cease-and-desist letter.
X’s safety account has previously denounced this kind of user activity, stating that anyone using or prompting Grok to make illegal content will suffer the same consequences as if they uploaded illegal content. TechCrunch reached out to xAI for comment and was met with an automated email that says “Legacy Media Lies.” They also reached out to the California attorney general’s office for more context.
The advent of free generative AI tools has led to a disturbing swell of non-consensual sexual material. Many platforms have been grappling with this problem, not just X. The sordid activity has drawn the attention not just of state leaders but also of Congress. Indeed, on Thursday, lawmakers sent a letter to the executives of several companies, including X, Reddit, Snap, TikTok, Alphabet, and Meta, asking how they planned to stem the proliferation of sexualized deepfakes.