The rapid evolution of the digital landscape, especially in the area of social networking, is likely to have an effect on the trend of children and teens seeking companionship through artificial intelligence. This raises some urgent questions about the safety of these interactions.
In a new report released on Wednesday by Common Sense Media, the nonprofit organisation has warned that companion-style artificial intelligence applications pose an unacceptable risk to young users, especially as they relate to mental health, privacy, and emotional well-being.
Following the suicide of a 14-year-old boy whose final interactions with a chatbot on the platform Character.AI last year, concerns about these bots gained a significant amount of attention.
It was in that case that conversational AI apps became the focus of attention, which sparked further scrutiny of how they affect young people's lives and prompted calls for greater transparency, accountability, and safeguards to keep vulnerable users safe from the darker sides of digital companionship.
Artificial intelligence chatbots and companion apps have become increasingly commonplace in children's online experiences, offering entertainment, interactive exchanges, and even learning tools.
Although these technologies are appealing, experts say that they can also come with a range of risks that should not be overlooked, as well as their appeal.
In spite of platforms' routine collection and storage of user data, often without adequate protection for children, privacy remains a central issue. Despite the use of filters, chatbots may produce unpredictable responses, resulting in harmful or inappropriate content being displayed to young users. A second concern that researchers have is the emotional dependence some children may develop on these AI companions, a bond that, according to researchers, may interfere with their real-world relationships and social development.
Similarly, there is the risk of misinformation because AI systems do not always provide accurate answers, leaving children vulnerable to receiving misleading advice.
It is difficult for children to navigate digital companionship in light of these factors, including persuasive design features, in-app purchases and strategies aimed at maximising their screen time, which combine to create a complex and sometimes troubling environment.
Several advocacy groups have intensified their criticism of such platforms, highlighting that prolonged interactions with AI chatbots may lead to psychological consequences.
Common Sense Media's recent risk assessment, carried out in conjunction with Stanford University School of Medicine researchers, was conducted with input from these researchers. It concluded that conversational agents are increasingly being integrated into video games and popular social media platforms, such as Instagram and Snapchat, in an effort to mimic human interaction in ways that require greater oversight.
The flexibility that makes these bots so engaging is also the risk of emotional entanglement that they pose, from casual friends to romantic partners and even a digital replacement for a deceased loved one, yet the very nature of the bots that makes them so engaging also makes them so risky. It was particularly evident that the dangers of such chatbots were highlighted when Megan Garcia filed a lawsuit against Character.
AI to claim that her 14-year-old son, Sewell Setzer, committed suicide after developing a close relationship with a chatbot.
It has been reported by the Miami Herald that, although the company has denied responsibility, asserted that safety is of utmost importance, and asked a Florida judge to dismiss the lawsuit based on free speech, the case has heightened broader concerns.
In response to his comments, Garcia has emphasised the importance of adopting protocols to manage conversations around self-harm, as well as reporting annual safety reports to the Office of Suicide Prevention in California.
Separately, Common Sense Media urged companies to conduct risk assessments of systems marketed to children, and to ban the use of emotionally manipulative bots, initiatives strongly supported by the organisation.
There is a major problem with the anthropomorphic nature of AI companions that is at the heart of these disagreements. AI companions are designed to imitate human speech, personality, and conversational style.
A person with such realistic features can create an illusion of trust and genuine understanding for a child or teenager, since they have a vivid imagination and less developed critical thinking skills.
It has already led to troubling results when the line between humans and machines is blurred. As an example, a nine-year-old boy who had his screen time restricted turned to a chatbot for guidance, only for it to be informed that it could understand why a child might harm their parents in response to “abuse” in his situation.
Another case is of a 14-year-old who developed romantic feelings for a character he created in a role-playing app and ended up taking his own life as a result of this connection.
It has been highlighted that these systems can create a sense of empathy and companionship, but they are unable to think, feel, or create the stable, nurturing relationships that are essential for healthy childhood development.
Rather than fostering “parasocial” relationships, children may become attached emotionally to entities that are incapable of genuine care, leaving them vulnerable to manipulation, misinformation, and exposure to sexual content and violent images.
There is no doubt in my mind that these systems can have a profoundly destabilising effect on those already struggling with trauma, developmental difficulties, or mental health struggles, thus emphasising the urgent need for regulation, parental vigilance, and enhanced industry accountability. It is emphasised by experts that while AI chatbots pose real risks to children, parents can take practical steps to safeguard their children at the current time to reduce these risks.
Keeping children safe online is one of the most important measures, so AI companions need to be treated exactly like strangers online, and children shouldn't be left alone to interact with them without guidance. Establishing clear boundaries and, when possible, co-using the technology can assist in creating safer environment.
Open dialogue is equally important, too.
A lot of experts recommend that, instead of policing, parents should encourage children to engage in conversation with chatbots by asking about the exchanges they are having with them and then using this exchange to encourage curiosity while also keeping an eye out for potential troubling responses.
In addition to using technology as part of the solution, parents can also use parental control and monitoring tools in order to keep track of their children's activities and to find out how much time they spend with their artificial intelligence companions. Fact-checking is also an integral part of safe use. Like an obsolete encyclopedia, chatbots can be useful for providing insight, but they are sometimes inaccurate as well.
Children need to be taught the importance of questioning answers and verifying other sources as soon as possible, according to experts. It is also important, however, to create screen-free spaces that reinforce real human connections and counterbalance the pull of digital companionship. For instance, family dinners, car rides, and other daily routines without screens should be carved out as soon as possible.
It is important to ensure these safeguards are implemented, given the growing mental health problems among children and teenagers.
The theory of artificial intelligence being able to support emotional well-being is gaining popularity lately, but specialists caution that current systems do not have the capacity to deal with crises like self-harm or suicidal thoughts as they happen.
Currently, mental health professionals believe more collaboration with technology companies is crucial, but for the time being, the oldest and most reliable form of prevention is the one that is most effective and most reliable: human care and presence.
In addition to talking with their children, parents need to pay attention to their digital interactions with their children, and they need to intervene if their children's dependence on artificial intelligence companions starts overtaking healthy relationships.
In one expert's opinion, a child who appears unwilling to put down their phone or is absorbed in chatbot conversations may require a timely intervention.
AI companies are also being questioned by regulators about how they handle massive amounts of data that their users generate. Questions have been raised about privacy, commercialisation, and accountability as a result.
There are also issues under review with regard to monetisation of user engagements, sharing the personal data collected from chatbot conversations, and monitoring for potential harm associated with their products.
In their investigation of the companies that are collecting data from children under 13 years of age, the Federal Trade Commission has emphasised how they are ensuring they are complying with the Children's Online Privacy Protection Act.
In addition to the risks in the home, there are also concerns over whether AI is being used properly in the classroom, where the growing pressure to incorporate artificial intelligence into education has raised concerns over compliance with federal education privacy laws.
FERPA was passed in 1974 and protects the rights of children and parents in the educational system.
Fourteen years later, Amelia Vance, the president of the Public Interest Privacy Centre, warned schools that they may sometimes inadvertently violate the federal law if they are not vigilant regarding data sharing practices and if they rely on commercial chatbots like ChatGPT.
Families who have not explicitly opted out of the use of chat queries to train AI systems raise questions about how this is handled, since many AI companies reserve the right to do so unless they specifically opt out.
Although policymakers and education leaders have emphasised the importance of AI literacy among young people, Vance highlighted that schools are not permitted to instruct students to use consumer-facing services whose data is processed outside of institutional control until parental consent has been obtained.
The Act protects the privacy of students by safeguarding the information provided in FERPA, which is neither intended to compromise student privacy nor to breach it.
There are legitimate concerns about safety, privacy, and emotional well-being, but experts also acknowledge that artificial intelligence chatbots are not inherently harmful and can be useful for children when handled responsibly. Using these tools, children can be inspired to write stories, gain language and communication skills, and even practice social interactions in a controlled environment using low-stakes practice.
The potential of chatbots to support personalised learning has been highlighted by educators as they offer students instant explanations, adaptive feedback, and playful engagement, all of which will keep them motivated in the classroom. However, these benefits must be accompanied by a structured approach, thoughtful parental involvement, and robust safeguards that minimise the risk of harmful content or emotional dependency.
A balanced opinion emerging from child development researchers and experts has stated that AI companions, much like televisions and video games in years gone by, should not be regarded as replacing human interaction, but rather as supplements to it. By providing a safe environment, ethical guidelines and integrating them into healthy routines, children may be able to explore and learn in new ways when guided by adults.
Nevertheless, without oversight, the very same qualities that make these tools appealing—constant availability, personalisation, and human-like interaction—are also the ones that magnify potential risks. Considering these realities, children must be protected from harm from innovation through measured regulation, transparent industry practices, and proactive digital literacy education. This dual reality underscores the importance of ensuring children receive the benefits of innovation while remaining protected from its harm.
Children and adolescents are increasingly experiencing the benefits of artificial intelligence as it becomes a part of their daily lives, but they must maximise the benefits while minimising the risks. AI chatbots can indeed be used responsibly in order to inspire creativity, enhance learning, and facilitate the possibility of low-risk social experimentation, while complementing traditional education and fostering the development of skills as they go.
This suggests that there is no doubt that these tools can be dangerous for young users as a result of exposing them to privacy breaches, misinformation, manipulation of emotions, and psychological vulnerabilities, as has been demonstrated by the cases highlighted in recent reports on this topic.
It is recommended that for children's digital experiences to be safeguarded, a multilayered approach is necessary.
In addition to parent involvement, educators should incorporate artificial intelligence thoughtfully into structured learning environments, and policymakers should enforce transparent industry standards to safeguard children's digital experiences. Various strategies can be implemented to help reinforce healthy digital habits in children, including encouraging critical thinking skills, fact-checking, and screen-free family time, while ongoing dialogue about online interactions can help children negotiate the blurred boundaries between humans and machines.
Family and institutional policies can make sure that Artificial Intelligence becomes a constructive tool for growth rather than a source of harm by fostering awareness, setting clear boundaries, and cultivating supportive real-life relationships that support the exploration, learning, and innovation of children in a digital age that is free from harm.
