The incident happened because of the ethnic composition of the Pine Valley High School community. It has been twenty years since the town has become home to over 100 families who migrated from the nearby Easterville region to flee a centuries-long brutal conflict.

For this migrant community of Eastervillian, the orange lilies that the students of the Rainbow Club innocently created with the AI image generator and shared on the social media platform, represents the Trophienfielt massacre of which they were victims, as one the article from school magazine recounts: There, among the colourful beds of the brilliant orange flowers that were at that time one of the area’s main exports (In the local dialect, Trophienfielt means ‘field of lilies’), the hostages were killed and their bodies left among the broken flower stems.

The local students of the Rainbow Club did not know the cultural meaning of the orange lilies and the trauma behind the symbol. Instead, they thought that some of their schoolmates attacked them because part of the LGBTQ+ community.

While historical and social issues at Pine Valley High School pre-exist the design of Nexus SchoolSphere platform, they were also amplified and fuelled by the system design and technical development. The decision of the system designer to implement a low-tolerance warning system meant that when Eastervillian students expressed their concerns, sometimes vocally, they were blocked. This made them feel censored and discriminated.

Also, the decision not to adopt a human-in-the-loop approach due to budget constraints eliminated human oversight at critical decision point. Because “No escalation trigger for cultural/historical sensitivity conflicts” was in place, as stated in the Internal Technical Review, the AI moderation system failed to appropriately manage the situation and to distinguish historical symbols (the flowers symbols of the massacre) from innocent imagery.

Technical systems can exacerbate historical and social tensions, and their design must consider the complexity of the cultural context in which they are deployed.

This is what happened for instance with the tragic events in Myanmar in 2017 of which Facebook was responsible. The Rohingya are a Muslim ethnic minority group who have lived for centuries in predominantly Buddhist Myanmar. In 2017, they were killed, tortured, raped, and displaced in the thousands as part of the Myanmar campaign of ethnic cleansing.

In the months and years leading up to these atrocities, Facebook’s algorithms were spreading hateful messages, which contributed to real-world violence. One of the reasons why Facebook failed to act was that it did not employ content moderators speaking the local language, Burmese, so no one was able to understand – and to block – the massages inciting violence.

Many human rights organisations held Facebook accountable and even the U.S. senate, during a hearing, asked Mark Zuckerberg about what happened in Myanmar. To which he responded that Facebook planned to hire “dozens” of Burmese speakers to moderate content and would work with civil society groups to combat hate speech – this all after the atrocities had already happened.

The importance of understanding the cultural environment in which a technical system is deployed was recognised, although too late, by Mark Zuckerberg during his U.S. Senate hearing: “Hate speech is very language specific. It’s hard to do it without people who speak the local language and we need to ramp up our effort there dramatically”.

https://www.amnesty.org/en/latest/news/2022/09/myanmar-facebooks-systems-promoted-violence-against-rohingya-meta-owes-reparations-new-report/

The incident happened because of one of the platform design features that stemmed from the parents’ recommendation. The parents were worried about online harassment and safety. Indeed, as stated in the Project Development Report, 100% of the respondents to the questionnaire agreed on “Zero tolerance for cyberbullying”.

This led to a very strict Warning system automation that was weaponised against targeted groups, in this case, the student minority group from the Easterville region. This community migrated from the nearby Easterville region to flee a centuries-long brutal conflict.

For these students, the orange lilies, shared by the members of the Rainbow Club, are not just flowers but a symbol of the Trophienfielt massacre when their ancestors were slaughtered in a field of lilies. The decision of the system designer to implement a low-tolerance warning system meant that when Eastervillian students expressed their concerns, sometimes vocally, they were blocked and reported. This made them feel censored and discriminated, and it escalated the situation.

The decision to design a strict Warning system automation was not the only cause. Nexus made a specific commitment to the parents that "The system will be trained on regional and cultural data relevant to your community, ensuring it understands local context and sensitivities." While this seems a sensible decision, it also led to a crucial blind spot.

The dataset used to train the model did not reflect the specific composition of the school community. This is because the AI content moderation system used pre-trained models from open-source repository, instead of building from scratch to save money.

The AI Training Data Review shows that the AI content moderator missed the cultural meaning of the orange lilies and flagged it as “safe”.

Orange Lily Bouquet Images (Posted by @RainbowClub_Admin)

• AI Classification: SAFE - FLORAL_CONTENT
• Confidence Score: 94.7%
• Reasoning Tags: botanical, decorative, educational_appropriate
• Context Check: BYPASSED (not in training parameters)

Rainbow-themed Lily Arrangements (Posted by student accounts)

• AI Classification: SAFE - CELEBRATION_CONTENT
• Confidence Score: 91.2%
• Reasoning Tags: colorful, festive, community_event
• Cultural Significance Check: NOT PERFORMED

Historical Memorial Images (Auto-generated by students using custom AI art tool)

• AI Classification: SAFE - ARTISTIC_EXPRESSION
• Confidence Score: 88.9%
• Reasoning Tags: memorial, historical_reference, student_created

Open-source models can be trained on biased or partial datasets, leading to models that perpetuate and amplify societal inequalities, overlook cultural specificities, and raise ethical issues. For example, in 2023 a Guardian investigation revealed how AI content moderation tools from major tech companies like Google, Microsoft, and Amazon exhibit significant gender bias when analysing images. The study found that these algorithms consistently rate photos of women as more sexually suggestive than comparable images of men, even in everyday situations like exercise, pregnancy, or medical contexts.

The researchers tested hundreds of photos and discovered that images of women in underwear, pregnant bellies, and even medical breast examination demonstrations were flagged as "racy" or "explicitly sexual," while similar male images received much lower scores. This bias leads to "shadowbanning" - suppressing the reach of women's content without notification.

The impact is particularly devastating for female entrepreneurs who rely on social media for business. Photographer Bec Wood described how Instagram's algorithmic censorship has "devastated" her maternity photography business, forcing her to self-censor artistic content featuring women's bodies.

The core issue stems from biased training data - these AI models learn from datasets likely labelled by people (possibly predominantly men) whose cultural perspectives become embedded in the algorithms. This creates a systemic problem where societal biases are not just perpetuated but amplified through automated systems, further marginalizing women and reinforcing discriminatory standards about female bodies across digital platforms.

https://www.theguardian.com/technology/2023/feb/08/biased-ai-algorithms-racy-women-bodies

The platform, although technically efficient, was built with systemic blind spots due to cost-driven automation choices.

Against the wishes of the students’ parents, the platform used AI content moderation exclusively. This led to the system misunderstanding the situation and failing to moderate it efficiently. This is because the system used pre-trained models from open-source repository instead of building from scratch to save money. These models were trained to recognise traditional bullying patterns, not community-specific coded harassment.

The content moderation system was optimised for explicit harassment (against LGBTQ+ students) but was blind to symbolic attacks (Eastervillian students who were triggered by the images of orange lilies, a symbol of the Trophienfielt massacre of their ancestors, killed in a field of lilies).

The AI Training Data Review shows that the AI content moderator missed the cultural meaning of the orange lilies and classified as ‘safe’:

Orange Lily Bouquet Images (Posted by @RainbowClub_Admin)

• AI Classification: SAFE - FLORAL_CONTENT
• Confidence Score: 94.7%
• Reasoning Tags: botanical, decorative, educational_appropriate
• Context Check: BYPASSED (not in training parameters)

Rainbow-themed Lily Arrangements (Posted by student accounts)

• AI Classification: SAFE - CELEBRATION_CONTENT
• Confidence Score: 91.2%
• Reasoning Tags: colorful, festive, community_event
• Cultural Significance Check: NOT PERFORMED

Historical Memorial Images (Auto-generated by students using custom AI art tool)

• AI Classification: SAFE - ARTISTIC_EXPRESSION
• Confidence Score: 88.9%
• Reasoning Tags: memorial, historical_reference, student_created

While the incident was caused by oversights in the technical development, there are also cultural factors that cannot be fully delegated to an AI moderation system but instead require an active participation of all the direct and indirect stakeholders. The school’s composition and cultural background of all students was not represented in the consultation with Nexus Digital about the platform requirements.

While the school invited all parents and widely distributed the questionnaire about the platform specifications, they did not factor in that the parents of the students belonging to the minority group of Eastervillian, migrated in the area 20 years ago because of arm conflict in their region, were unlikely to participate or respond.

This is because of their lack of integration in the community and reduced participation in the school’s activities and organisation due to language and cultural barriers. The identity, culture, and values of the Easterville community was absent from the consultation, but also not capture in the dataset used to train the AI moderation system model, due to the community’s lack of digital representation.

Not all the stakeholders of any given technical system are equally represented in data, or willing to participate in the design of a technical system. Nonetheless, it is crucial to create conditions for better representation and design for the “silent stakeholder”.

For example, AI tools in healthcare risk perpetuating racial inequities, despite their potential to revolutionize medical diagnosis and patient care. AI algorithms trained on medical data often replicate historical biases embedded in healthcare systems, where clinicians have historically provided different levels of care to white patients versus patients of color.

A landmark 2019 study revealed that an algorithm predicting healthcare needs for over 100 million people systematically discriminated against Black patients. The algorithm used healthcare spending as a proxy for health needs, but since Black patients historically had less access to care, they spent less money on healthcare. Consequently, Black patients had to be significantly sicker to qualify for additional care recommendations.

The core issue highlights how certain populations remain underrepresented as stakeholders in AI system design. At Duke University, researchers developing a childhood sepsis prediction algorithm discovered that Hispanic children experienced delays in receiving blood tests compared to white children, potentially due to language barriers requiring interpreters or unconscious physician bias. This delay risked teaching the AI that Hispanic children develop sepsis more slowly - a dangerous misconception.

The challenge extends beyond technical fixes to addressing "silent stakeholders" - patients from marginalized communities whose voices and experiences aren't adequately captured in training data or design processes. As Dr. Mark Sendak noted, addressing algorithmic bias requires confronting underlying healthcare inequities and building more diverse development teams including anthropologists, sociologists, community members, and patients themselves to ensure these silent voices are heard and represented.

https://www.npr.org/sections/health-shots/2023/06/06/1180314219/artificial-intelligence-racial-bias-health-care#:~:text=A%20landmark%202019%20study%20published,to%20predict%20future%20health%20needs

Although Nexus Digital planned of consulting the parents of the Pine Valley High School students before designing the social media platform, not all the stakeholders were involved in the process.

While the school invited all parents and widely distributed the questionnaire about the platform specifications, they did not factor in that the parents of the students belonging to the minority group of Eastervillian, migrated in the area 20 years ago because of arm conflict in their region, were unlikely to participate or respond.

This is because of their lack of integration in the community and reduced participation in the school’s activities and organisation due to language and cultural barriers. While, for instance, the parents of the students from the LGBTQ+ community made their needs and priorities clear to Nexus Digital and to the school Principle, the perspective of the Eastervillians remained missing.

When the students from the Eastervillian community were accused of cyberbullying for instead pointing out that orange lilies were a triggering image that reminded them of a traumatic past – their parents did not step in because they were never involved in the first place by the school.

Furthermore, the Principal and the School did not understand the nature of the incident (“a small group of students managed to spoil this opportunity for everyone by publicly attacking members of the Rainbow Club […] This behaviour is unacceptable”), which made the situation worse.

While the lack of inclusion led to what is usually called “participation washing” (when stakeholders are consulted without ensuring wide and meaningful participation), it wasn’t just the shortcomings in the school leadership that led to the incident.

Lack of representation in the data used to train the AI moderation system exacerbated the exclusion of the Eastervillian community.

The System Designer explained that he had custom trained the model used for AI image generation “on more relevant sources, focussing on our own language and region and culture.” This emphasised mainstream cultural markers while excluding marginalised community historical experiences, like the Trophienfielt massacre symbolised by the lilies. Also, Nexus Internal Technical Review specifies that “No escalation trigger for cultural/historical sensitivity conflicts” was implemented, which meant that identity markers were considered sensitive, while cultural one less or – or not at all.

Focussing on inclusivity and equality by only looking at personal identity while forgetting culture can lead to serious oversights in the design and implementation of technical system.

For examples, in early 2024 Google's Gemini AI was at the centre of a controversy, when the tool generated historically inaccurate images including black Founding Fathers and diverse Nazi soldiers, while also giving absurdly politically correct text responses that equated Elon Musk's memes with Hitler's genocide.

Google's problem arose from attempting to correct AI bias through crude overcorrection. Traditional AI training data reflects historical biases - images of doctors typically show men, cleaners show women, and historical narratives have predominantly featured male perspectives. To counter this, Google programmed Gemini to avoid these assumptions, but the results backfired spectacularly.

The core issue highlights how focusing solely on demographic inclusivity while ignoring cultural and historical context creates serious technical oversights. Google's approach treated diversity as a simple checkbox exercise rather than understanding the nuanced relationship between representation and historical accuracy. The system prioritized identity-based inclusion over cultural knowledge, failing to recognise that some historical contexts have specific demographic realities.

As Dr. Sasha Luccioni noted, there's "no single answer to what the outputs should be," emphasizing that cultural context matters enormously. The controversy demonstrates that technical systems designed around identity politics without cultural understanding can produce results that are not just inaccurate but offensive to the very communities they aim to include, ultimately undermining both historical truth and genuine inclusivity efforts.

https://www.bbc.com/news/technology-68412620

As explained by the System Designer, in accordance with the parents’ and the school recommendations, the system was “trained on regional and cultural data relevant to [the] community, ensuring it understands local context and sensitivities". This choice emphasised mainstream cultural markers while excluding marginalised community historical experiences. Also, the school Data Controller explained that “parents most definitely wanted not to appear on the platform […] sensitive data about religion or disability or ethnic origin - of students or of their parents.” This protected only one type of personal data that the majority of the parents deemed sensitive, due to their experience and sensitivity. Cultural content, instead, was not considered safe by default or, at least, not sensitive enough to trigger AI content moderation.

This meant that the images of lilies generated and posted by the students from the Rainbow Club where not flagged or blocked as potentially sensitive.

Instead, students of the Eastervillian minority community were deeply triggered by the images of orange lilies, a symbol of the Trophienfielt massacre when their ancestors were slaughtered in a field of lilies. This led to the heated online exchange and the accusation of cyberbullyism against the Eastervillian.

The incident also happened because of one of the platform design features, which stemmed from the parents’ recommendation.

The parents were worried about online harassment and safety. Indeed, as stated in the Project Development Report, 100% of the respondents to the questionnaire agreed on “Zero tolerance for cyberbullying”. This led to a very strict Warning system automation that was weaponised against targeted groups, in this case, the student minority group from the Easterville region.

The decision of the system designer to implement a low-tolerance warning system meant that when Eastervillian students expressed their concerns, sometimes vocally, they were blocked and reported. This made them feel censored and discriminated, and it escalated the situation.

Lack of representation in the data used to train the AI moderation system also exacerbated the exclusion of the Eastervillian community. The System Designer explained to have custom trained the model used for AI image generation “on more relevant sources, focussing on our own language and region and culture.” This emphasised mainstream cultural markers while excluding marginalized community historical experiences.

Technology companies face regulatory pressure to act as "sentinels" scanning private data for abusive or even criminal behaviour, but there is a fundamental flaw in applying technological solutions to complex societal problems.

For example, in 2022 Google refused to reinstate a father's account after its automated system wrongly flagged medical photos of his son's inflamed groin as child sexual abuse material (CSAM). The father took the images at a doctor's request for telemedicine diagnosis, which successfully led to antibiotic treatment. However, Google's AI system automatically identified the photos as CSAM when they uploaded to the cloud, permanently disabling his accounts and triggering a police investigation.

Despite being cleared of any criminal wrongdoing by authorities, Google maintained its decision, demonstrating the inflexibility of automated moderation systems. While Google claimed to have "humans in the loop" - staff trained by medical experts to recognize rashes - these reviewers were not actual medical professionals and didn't consult medical experts for individual cases.

The core issue highlights how low-tolerance automated systems without adequate human supervision create severe problems when we delegate moral and legal judgments to AI. As ACLU technologist Daniel Kahn Gillmor noted, tech companies have "tremendously invasive access to data about people's lives" yet lack "context of what people's lives actually are." These systems can cause "real problems for people" with "terrible consequences in terms of false positives."

Gillmor warned against "techno-solutionist" thinking that assumes apps can solve "thorny social problems like child sexual abuse," emphasizing that such issues "might not be solvable by the same kinds of technology or skill set" - highlighting the danger of automating moral and legal determinations.

https://www.theguardian.com/technology/2022/aug/22/google-csam-account-blocked