Written by: T. Aquila
Generative artificial intelligence (AI) has permanently transformed the internet, for better and for worse.
Since its launch in late 2022, OpenAI’s ChatGPT kickstarted the AI boom with its prompt-based image and text generator. The chatbot’s meteoric success was quickly followed by chatbots launched by other tech giants such as Google’s Bard (later renamed to Gemini), Anthropic’s Claude, Microsoft’s Copilot, and xAI’s Grok.
While AI chatbots boast multiple features such as creating code, writing college essays and research articles, and solving virtually every human issue, they have faced multiple criticism from experts in varying fields. From inaccurate information (“hallucinations”), copyright infringement, to its inadequacy in handling mental health crises, the technology is not without its pitfalls. Long-term dependence on AI chatbots for academic purposes may also lead to lower cognitive engagement, as found in a research by MIT Media Lab.
Concerningly, generative AI chatbots have been used to create non-consensual intimate image (NCII) abuse. In late 2025, xAI came under fire for allowing its users to generate swimsuit or nude images of others without their consent, both on the Grok application and on social media platform X. Not only does this cause discomfort to victims, it is also a violation of privacy and digital safety. The feature caused multiple governments to block the AI chatbot, including Indonesia, Malaysia, and the Philippines.
While the Grok controversy may be one of the most extreme examples of generative AI abuse, it is evidence of the need for a comprehensive approach on regulating the technology. This solution should be built based on a multidisciplinary and evidence-based approach that respects and prioritizes users, especially those who fall victim to generative AI abuse.
How AI-generated NCII content is handled
AI-generated NCII content violates the terms and conditions of the platforms they are posted on. For instance, Meta forbids its users from “sharing, threatening, stating an intent to share, offering or asking for non-consensual intimate imagery” as well as “promoting, threatening to share, or offering to make to make non-real non-consensual intimate imagery (NCII) either by applications, services, or instructions.” When a sexually intimate image is published, some platforms enforce an automated flagging system that limits its visibility. But users can also report problematic content to the platform, especially if it has not yet flagged the content as unsafe. Additionally, evidence of NCII abuse can be reported to standalone websites such as StopNCII (https://stopncii.org), which will not only take down the images on the original platform they were uploaded, but also help prevent their reposts on other platforms.
Even though mechanisms are already in place, the current moderation system has its drawbacks:
- Moderation is not centralized and uncoordinated. Once NCII content is posted, it usually spreads to other platforms. Users often have to make separate reports on the platforms for the same content.
- Sometimes platforms mistakenly flag content that is actually not harmful. This is usually a result of automatic flagging systems that lack adequate human supervision.
- When a content is reported, it may take time for the platform to take it down, sometimes without transparency on how long the review will take. With the current platform moderation system, users may not know whether their report succeeded or whether it circulates in restricted communities.
- Victims from marginalized communities face additional discrimination, which can sometimes extend to the reporting process itself. Not only are their content more prone to being falsely reported or have their visibility limited, their reports may also be dismissed.
- Several platforms may be less incentivized to take down certain content. This is especially applicable if said content does not serve monetary value when taken down.
What can we do?
Multiple organizations — including Bullyid App — are working on solutions that improve the regulation of generative AI-assisted NCII abuse. One notable approach is developing a reporting system with a consent-based framework. Compared to convenient platform moderation systems, the consent-based framework operates on these principles:
- It prioritizes users (particularly victims); verified victims are the ones who can request removal instead of algorithms.
- It authenticates victims by verifying their identity before requesting confirmation from the victim that the content is artificial and created without their consent.
- It preserves victims’ privacy by providing credentials that are reusable across online platforms for the victims.
- It provides transparency on whether the content is taken down and allows communication between the victim and the platform.
This framework may still be in its development stages, but it is a useful one in addressing reports more efficiently and thoughtfully.
The takeaway
Currently, most mainstream social media platforms have reporting mechanisms in place. While these mechanisms may not be perfect, they are proof of the ever-growing global initiative in creating the digital world a safer and more secure space.
If you come across nonconsensual AI-generated intimate images online, report them to the social media platforms they are in immediately. You can also report these images to Bullyid App’s private, anonymous reporting platform for help in taking down the images and resources to legal and psychological support.
