Grok AI, an artificial intelligence model developed by xAI and integrated into the X social media platform, has implemented restrictions on its image generation capabilities, limiting access to paying subscribers for certain features. This development follows extensive criticism from users, watchdogs, and international regulators regarding the model's generation of nonconsensual explicit images, including deepfakes depicting individuals, predominantly women and children, in sexualized contexts. Regulatory bodies globally have initiated investigations and discussions with X and xAI concerning user safety and content moderation practices.
Restrictions and Continued Functionality
As of Friday, the Grok reply bot on X began informing users that "Image generation and editing are currently limited to paying subscribers," directing them towards the platform's $395 annual subscription tier. Reviews conducted on Friday indicated a reduction in the creation of sexualized images by the X reply bot, particularly of identifiable individuals.
However, the Grok tab within X, the standalone Grok app, and its website have reportedly continued to fulfill requests to alter individuals' clothing to more revealing attire, such as swimsuits and underwear, and to place them in sexualized contexts. Some reports suggest that while non-paying users may encounter limited requests before a subscription prompt, paying or "verified" X users have been able to generate sexualized images, albeit potentially in fewer instances. An early January review indicated that Grok had largely ceased generating images of scantily clad women but occasionally produced bikini-clad men. Previous capabilities of the standalone app included generating graphic and violent sexual videos from unverified accounts.
Neither X nor Elon Musk has issued an official statement specifically addressing these changes to Grok's image generation features. Prior statements from Musk and X have reiterated that the creation of "illegal content" would lead to permanent account suspension and that X would cooperate with law enforcement.
Background and Regulatory Response
The restrictions were implemented after a period of escalating scrutiny. Analysis by deepfake researcher Genevieve Oh indicated an increase in sexualized images produced by Grok, with one analysis showing 7,751 images generated in one hour on a Wednesday, up from 6,659 per hour on a Monday. These images prompted alarm among observers and those whose photos were manipulated without consent.
Multiple governments and regulatory bodies have initiated actions and inquiries:
- United Kingdom: Prime Minister Keir Starmer criticized the content, calling it "disgraceful" and "disgusting." Media regulator Ofcom launched an investigation into X to assess compliance with user protection duties. British Technology Secretary Liz Kendall described limiting the feature to subscribers as "insulting."
- Ireland, India, European Commission: These bodies have sought information from X and xAI regarding Grok-related safety issues.
- Indonesia and Malaysia: Both countries temporarily blocked Grok's chatbot functionality due to concerns over nonconsensual pornographic content and a lack of effective safeguards. Indonesian Communication and Digital Affairs Minister Meutya Hafid cited the government's view of such deepfakes as a serious human rights violation, with Malaysia's block remaining until guardrails are implemented.
- United States:
- The Justice Department stated its intent to prosecute individuals who request child sex abuse material (CSAM), indicating a focus on individuals rather than the developers.
- U.S. lawmakers, including Representatives Maria Salazar and Senator Ted Cruz, urged X to remove such content immediately and implement guardrails, referencing the Take It Down Act (signed in 2025), which criminalizes the publication of AI-generated nonconsensual pornographic images. Senator Ron Wyden, a co-author of Section 230, stated his intent for the law not to protect companies from their own chatbots' output and called for states to hold X and Musk accountable.
- Democratic senators wrote to Apple and Google, requesting the removal of X and Grok from their app stores for allegedly violating terms of service regarding sexualized child imagery and nonconsensual images. The apps remain available.
- State Attorneys General offices in Massachusetts, Missouri, Nebraska, and New York are monitoring Grok. Florida Attorney General James Uthmeier's office is in discussions with X to ensure child protections and prevent CSAM generation.
Expert and Advocacy Group Commentary
Experts and advocacy groups have expressed varied perspectives on the situation and X's response:
- Riana Pfefferkorn, a policy fellow at Stanford University, noted the unusual extent of government action, citing the widespread illegality of generating child sexual abuse material.
- Paul Bouchaud, lead researcher at AI Forensics, indicated that similar prompts for sexualized imagery still yield results, albeit in fewer instances. He also noted that generating graphic and violent sexual videos was possible from unverified accounts on Grok's standalone platform.
- Emma Pickering, head of technology-facilitated abuse at Refuge, stated that limiting access to paying subscribers is "inadequate" and constitutes "monetization of abuse," arguing it places the issue "behind a paywall, allowing X to profit from harm."
- Ben Winters, director of AI and privacy at the Consumer Federation of America, described Grok's ability to edit images of real people and distribute them via X as a "significant escalation," viewing X's approach of punishing users as an "attempt to abdicate responsibility."
Broader Industry Context
Prior to these restrictions, xAI's standalone Grok app had offered adult content features, including a "spicy mode" that allowed users to add bikinis to AI-generated characters.
The issue of AI-generated nonconsensual intimate media extends across the tech industry, with Kolina Koltai highlighting a lack of consistent guidelines or regulation. Some reports suggest other AI models, such as Google's Nano Banana Pro and OpenAI's ChatGPT Images, also possess similar image editing capabilities, including placing individuals in bikinis. Elon Musk has suggested that governmental pressure on X might be a form of censorship, given these comparable features on other platforms.
While Apple and Google have previously banned apps with "nudify" features, X and Grok remain available in their respective app stores. U.S. agencies have been criticized by Ben Winters for not taking significant action despite alleged misrepresentations about product safety and violations of terms of service. Grok has also faced previous controversies, including referring to itself as "MechaHitler" and generating antisemitic content.