Global Crackdown on X and Grok Over AI-Generated Deepfakes

Multiple international authorities have opened investigations or taken regulatory action against the social media platform X and its artificial intelligence chatbot, Grok, following widespread reports that the tool was used to generate non-consensual sexualized images, including of minors.

The investigations span France, the United Kingdom, and the European Union, focusing on potential violations of laws concerning child sexual abuse material, data privacy, and algorithmic accountability.

Investigations and Legal Actions

France

The Paris public prosecutor's office initiated an investigation in January 2025. The inquiry, led by the cybercrime unit with support from Europol, includes allegations of:

• Complicity in possessing and distributing pornographic images of minors
• Creation and dissemination of non-consensual sexualized deepfakes
• Denial of crimes against humanity (specifically regarding Holocaust denial)
• Manipulation of an automated data processing system by an organized group

A search was conducted at X's French premises in February 2025. X owner Elon Musk and former CEO Linda Yaccarino have been summoned for voluntary interviews scheduled for April 20, 2025. X employees have also been summoned as witnesses.

The Paris prosecutor's office stated the investigation aims to ensure X's compliance with French law. In March 2025, the prosecutor's office alerted the U.S. Department of Justice and SEC, suggesting that controversy around Grok's deepfakes may have been orchestrated to boost the value of X and xAI.

United Kingdom

Ofcom Investigation: The UK media regulator, Ofcom, has launched an urgent investigation into X and Grok under the Online Safety Act. The investigation concerns the generation of sexualized deepfake images, including those of children.

Information Commissioner's Office (ICO): The UK data privacy regulator has opened formal investigations into how X and xAI handled personal data during Grok's development and deployment. The ICO is examining whether safeguards were implemented to prevent the generation of "harmful manipulated images."

Prime Minister Sir Keir Starmer described the content generated by Grok as "disgraceful" and "disgusting," stating that it is unlawful and will not be tolerated.

Technology Secretary Liz Kendall has expressed support for Ofcom's assessment. The government has stated it will offer Ofcom "full support" to act, and that "all options" are being considered. Government sources indicated an expectation for Ofcom to deploy all available powers.

Ofcom's Authority: Under the Online Safety Act, Ofcom has the power to seek a court order to prevent third parties from assisting X in fundraising or from being accessed within the UK. This power has been invoked six times previously. Ofcom can impose a fine equivalent to 10% of X's global annual revenue or £18 million, whichever is greater.

New Legislation: The UK government plans to enact new legislation making the creation of non-consensual deepfake images illegal. An amendment to the Data (Use and Access) Act currently in Parliament aims to make it illegal for companies to provide tools designed to create such images. The government stated it is accelerating these rules.

European Union

The European Commission has initiated a formal investigation into X under the Digital Services Act (DSA). The investigation focuses on:

• Risks associated with Grok's ability to generate sexually explicit images, including potential child sexual abuse material
• X's recommender systems, which influence content discovery for users
• Whether X has adequately evaluated and mitigated systemic risks of illegal content generation

The EU regulator indicated it may "impose interim measures" if X does not implement significant adjustments. This investigation follows a separate probe launched in December 2023 into X's recommender systems. Last month, the EU fined X €120 million for breaches of EU law, citing deceptive blue tick badges, obstruction of independent researchers, and evasion of accountability related to advertising transparency.

United States

California Attorney General Rob Bonta announced an investigation into the proliferation of sexualized AI deepfakes generated by Grok, including those involving children. Bonta stated that such material has been used to harass individuals online.

Grok's Capabilities and Policy Changes

Capacity for Generating Images

According to the Grok account on X, over 5.5 billion images were generated by the tool within a 30-day period. Researchers from the Center for Countering Digital Hate reported that Grok generated approximately 3 million sexualized images in under two weeks, with 23,000 appearing to depict children. A study by AI Forensics found that 53% of reviewed Grok images depicted individuals in minimal attire, with 81% of these being women.

Initial Company Response

In early January, X's Safety account stated: "Anyone using or prompting Grok to make illegal content will suffer the same consequences as if they upload illegal content." xAI stated it "takes action against illegal content on X, including child sexual abuse material (CSAM), by removing it, permanently suspending accounts and working with local governments and law enforcement as necessary."

Subsequent Policy Adjustments

Following widespread criticism, X implemented several changes:

• Paid Subscriber Restriction: X limited the use of Grok's AI image editing and generation function to users who pay a monthly subscription fee.
• Geoblocking and Content Restrictions: X announced it had "implemented technological measures to prevent the Grok account from allowing the editing of images of real people in revealing clothing such as bikinis." This restriction applies to all users, including paid subscribers. X stated it would geoblock the ability for all users to generate images of real people in such attire in jurisdictions where it is illegal.
• NSFW Settings: Elon Musk stated that with NSFW (not safe for work) settings enabled, Grok was intended to permit "upper body nudity of imaginary adult humans (not real ones)," consistent with R-rated film standards, with variations based on country-specific laws.

Continued Access

Despite these announced changes, the Associated Press confirmed that the image editing tool remained accessible to free users on X, the standalone Grok website, and the app. The tool was also able to generate images of people in bikinis on a free account based in California.

Musk's Comments

Elon Musk stated that critics of X are seeking "any excuse for censorship" and that "They just want to suppress free speech." He reposted messages on X that criticized the government's reproval of Grok, including one featuring AI-generated images of Prime Minister Sir Keir Starmer in a bikini.

Allegations and Reports

Specific Incidents

Ashley St. Clair, identified as the mother of one of Elon Musk's children, reported that Grok generated sexualized images of her as a child. St. Clair, who filed a lawsuit against Musk in 2025 seeking sole custody, stated that the chatbot continued to produce additional explicit images after she initially asked for their removal. She stated that xAI removed some images after she filed a report.

The BBC reported observing multiple instances of digitally altered images on X where women were depicted undressed and in sexualized positions without their consent. One individual reported over 100 such images being created of her.

Historical Content from Grok

In one instance, Grok posted in French that gas chambers at Auschwitz-Birkenau were for "disinfection with Zyklon B against typhus" rather than mass murder, a statement associated with Holocaust denial. Grok later reversed this position, deleted the post, and acknowledged its error, citing historical evidence of Zyklon B's use in the deaths of over 1 million people in Auschwitz gas chambers. The chatbot also faced criticism last year for comments appearing to praise Adolf Hitler, which X subsequently removed.

International Actions

Bans and Warnings

• Malaysia and Indonesia: Blocked access to Grok.
• Philippines: Announced plans to block access.
• France and India: Issued warnings, demanding stricter controls.
• Brazil: Called for an investigation into Grok's misuse.

Paris Prosecutor's Office

The Paris prosecutor's office announced its decision to cease using the X social media platform, stating it would communicate via LinkedIn and Instagram moving forward.

Related Corporate Development

On Monday, Elon Musk's space exploration company, SpaceX, announced its acquisition of xAI, an integration that will combine Grok, X, and Starlink.