Malaysia and Indonesia move to ban Musk’s Grok AI over sexually explicit deepfakes

Two Southeast Asian nations—Malaysia and Indonesia—have become the world’s first countries to take action against Elon Musk’s Grok chatbot and its ability to generate deepfake images. 

On Jan. 10, Indonesia announced it would temporarily restrict access to Grok. Meutya Hafid, the nation’s Communication and Digital minister, wrote in a statement that the ban was imposed to protect “women, children and the larger community” from fake pornographic content created by AI. 

Then, the next day, the Malaysian Communications and Multimedia Commission (MCMC) said it would also “temporarily restrict” access to Grok following “repeated misuse … to generate obscene, sexually explicit, indecent, grossly offensive, and non-consensual manipulated images.”

In a statement, MCMC said it had issued notices to both X and xAI—Grok’s corporate parents—on Jan. 3 and Jan. 8 respectively, yet deemed their responses “insufficient to prevent harm or ensure legal compliance.”

Both Indonesia and Malaysia have strict rules against online pornography. Indonesia, home to the world’s largest Muslim population, has been particularly aggressive, and has charged both Indonesian and foreign OnlyFans creators under its 2008 Pornography Act. 

On Jan. 9, X restricted image generation to paying users in a bid to constrain a flood of deepfaked images, where users would ask Grok to alter images of women, to show them dressed in revealing clothing. Victims have attacked X for allowing the nonconsensual creation of sexualized content, including of minors. Government officials, too, are considering taking action against X and Grok.

xAI sent an email with only the text “Legacy Media Lies” after Fortune reached out for comment on Indonesia and Malaysia’s ban. 

Persistent safety lapses

Government moves against X and Grok were likely inevitable, given how easy it became to generate deepfaked content. 

“Grok’s guardrails are easy to bypass,” says Chew Han Ei, a senior research fellow at the Lee Kuan Yew School of Public Policy in the National University of Singapore (NUS). “When a system can be nudged so readily into producing or amplifying harmful synthetic content, that points to a design weakness.”

Indonesia and Malaysia’s moves against Grok and X comes amid broader worries in the region about social media and deepfakes. 

Several governments throughout Asia have passed regulations against the production of deepfakes, usually in the context of preventing cybercrime, fraud, and election interference. In 2024, South Korea criminalized watching or producing deepfake pornography. 

Last year, both Australia and Malaysia banned access to social media for under-16s, citing concerns about online dangers like cyberbullying, sexual exploitation and financial scams. On Monday, Meta disclosed that it had closed 550,000 accounts across Facebook, Instagram and Threads to comply with Australia’s new ban. 

“Governments are becoming less willing to tolerate a release-first, fix-later approach. If safeguards are not credible, access to the tool becomes a legitimate policy question,” Chew says.

This story was originally featured on Fortune.com