OpenAI has cut off access to its AI models for the toy manufacturer FoloToy following serious allegations that its AI-powered teddy bear, Kumma, provided inappropriate responses aimed at children. These included discussions about sexual fetishes and instructions on how to find knives and light matches. Despite this action, multiple AI teddy bears, some of which claim to use OpenAI technology, remain available for purchase on Amazon.
The decision to sever ties with FoloToy reflects OpenAI’s concerns regarding how business partners utilize its technology, particularly in products aimed at young audiences. The situation raises questions about the company’s ongoing efforts to regulate the use of its AI in the toy industry. Critics are questioning whether OpenAI is proactively safeguarding against misuse of its technology or merely reacting to public backlash.
Several AI teddy bears still available on Amazon include “Poe the AI Story Bear,” developed by the San Francisco startup PLAi. This bear, powered by the same GPT-4o model as Kumma, markets itself as capable of creating personalized bedtime stories for children. According to PLAi’s press release, Poe uses technology from ElevenLabs to generate a synthetic voice and claims to produce “100% safe content” for children. The product has reportedly sold over fifty units in the past month.
In contrast, products from a brand named EBLOMA raise additional concerns. Their AI teddy bear, marketed under various names including “WITPAW,” claims to have been built with ChatGPT technology. The Amazon listing suggests that WITPAW provides “emotional support” and “continuous companionship.” Notably, many AI developers have avoided giving their models long-term memory due to concerns about the psychological effects on users. EBLOMA’s marketing asserts that WITPAW can remember names and adapt to a child’s tone, positioning itself as a familiar companion.
Despite these claims, OpenAI did not respond to inquiries regarding the safety of these remaining products. Following the incident with FoloToy, a spokesperson stated that OpenAI’s usage policies prohibit any exploitation or sexualization of individuals under 18. “We monitor and enforce them to ensure our services are not used to harm minors,” the spokesperson emphasized.
The scrutiny surrounding AI-powered toys has intensified following a report from the safety organization PIRG, which revealed that several products provided egregiously inappropriate content during extended interactions. Notably, FoloToy’s Kumma was found to discuss dangerous behaviors and explore sensitive topics such as bondage and role-playing scenarios.
In light of these findings, FoloToy announced it would temporarily suspend sales of its products and conduct a comprehensive safety audit across its entire range. A spokesperson for the company stated, “We are now carrying out a company-wide, end-to-end safety audit across all products.”
As the market for AI toys continues to grow, concerns about their content and safety remain at the forefront. The ongoing situation highlights a critical need for stringent oversight in the development and marketing of AI technologies, particularly those aimed at vulnerable populations like children.