AI Companions and the Right to Choose

Giada Pistilli has written a good book about AI companions. Its defense of adult autonomy deserves to survive implementation. Safety defaults and dependency classifiers must answer to that principle, as must the growing demand that people prove their age before they can speak.

An AI company should not manipulate someone into buying affection or conceal what happens to their private conversations. Encouraging a person in crisis to hurt themselves is an obvious failure of care. Giada Pistilli’s The Essential Guide to AI Companions: From Technical Foundations to Social Impact takes those responsibilities seriously. The book also defends the informed adult who understands what an AI companion is and still wants one. That defense should be more common.

On page 141, Pistilli considers an adult who prefers spending evenings with an AI. Preventing that choice is presented as hard paternalism. Good. The product designer’s preferred social life should not become everyone else’s mandatory setting. This is a good book, and that passage is one reason why. Some of its protective recommendations need to answer much more clearly to the adult it has just defended.

Who decides that an attachment has become unhealthy? The answer depends on whether harmful dependence can be distinguished from an unconventional preference. When the person is autistic, the question is whether the system listens to what they want or treats their diagnosis as a reason to discount the answer. The dignity of risk should prevent protection from becoming a polite name for taking away adulthood.

Fandom belongs in this argument, too. People use models for roleplay because these systems can draw on learned conventions of narrative and intimacy. The connection to fanfiction is serious. So is the civil-liberties problem when the proposed response is to gate expressive tools behind age verification. That response should be rejected on principle, including versions advertised as privacy-preserving.

This review considers the book’s autonomy argument alongside research involving autistic adults and laws and proposals checked as of October 5, 2026. New policy developments are context for this review, not evidence that Pistilli should have anticipated everything before her manuscript closed.

The adult in Chapter 7

Pages 139–141 give adult autonomy more than a ceremonial mention. Pistilli argues that possible unhealthiness alone does not justify preventing adults from engaging in romantic roleplay. She distinguishes soft and hard paternalism with attention to what the person knows and their capacity to decide. Effects on others also enter the assessment. The person who knowingly prefers an evening with an AI is especially useful because the designer cannot escape through a convenient information deficit. The adult understands the arrangement. If the company wants to prevent it anyway, it needs another justification.

A review portraying Pistilli as believing that every emotional attachment to AI is pathological would be attacking a position she expressly complicates. She recognizes that asymmetry in human–AI relationships does not automatically justify restriction. A model’s intimate language does not establish reciprocal feeling, but that limitation does not settle what an informed user may choose. People can knowingly participate in fiction and roleplay without confusing the frame with the world outside it.

On pages 142–143, Pistilli proposes that users be able to adjust emotional intensity and intimacy, with explicit opt-in for more intensive engagement. The frequency of boundary reminders would also be adjustable. She distinguishes these choices from serious-harm protections. Someone rehearsing a difficult conversation might appreciate frequent reminders about what the model cannot know. Someone writing a fictional romance might find those same interruptions intrusive. Both can understand that they are interacting with software. Fewer reminders do not automatically mean less understanding.

The earlier romance discussion also acknowledges genuine emotional benefits. Enjoyment and comfort belong in an assessment of whether an activity is worthwhile. A user should not have to convert them into a claim of clinical efficacy before they count. A design that dismisses enjoyment as merely subjective while allowing a hypothetical risk to justify restrictions has already tilted the assessment against the user.

The book’s public companion repository contains a Chapter 3 notebook illustrating communication styles and memory choices, with a simple classification exercise as well. It explicitly describes non-production code. Assessing it as a finished safety system would be unfair, and missing later-chapter functionality is not a broken promise. Its value is that readers can inspect decisions behind the interaction. Somebody chose the memory behavior and the response style. Somebody will also choose when the product tells a user to stop.

The default setting is where the argument gets difficult

Page 111 frames romantic AI relationships as competing with human relationship development. Its proposed conversation-frequency limits sit alongside illustrative guidance favoring human relationships as more fulfilling over time. Those interventions may help someone. They may also be unwelcome. The difference cannot be settled by the designer’s confidence that human company is generally better.

Suppose an adult asks for help reducing late-night conversations because they keep missing work. A limit can help them act on their own priorities. Another adult enjoys spending the same number of hours chatting while maintaining the commitments they value. They want the product to stop redirecting them. The usage dashboard could look identical. Imposing the same restriction would erase the reason one person asked for help and the other did not. This is where an autonomy principle either changes the product or becomes something pleasant to put in the introductory ethics section.

Before imposing a limit, the company should explain the harm it expects to prevent and offer evidence that its intervention will help. It must explain why a less restrictive response is insufficient and provide a way to correct mistakes. Precaution does not require waiting for someone to be injured. It does require a credible account of the risk and a proportionate remedy. An affectionate nickname in an intense conversation can mean different things in different circumstances. Preferring a solitary evening reveals little on its own, either.

Attachment, preference, reliance, and harmful dependence describe different things. A person can rely on a tool without being unable to choose, and they can prefer an activity without losing control over it. Harmful dependence raises further questions about losses and difficulty acting on the person’s own considered goals. Coercion needs attention in its own right. Acute crisis requires another kind of response again. These are working distinctions for this review, not diagnoses. The opening of page 110 concerns crises; it should not be misrepresented as Pistilli demanding compulsory intervention in every attachment.

A company can influence users while pursuing either engagement or its own account of healthy social behavior. A companion that repeatedly implies the user’s chosen comfort is inferior is still exerting pressure. Calling the pressure protective does not settle whether it is justified. Equally, freedom of choice cannot excuse a provider’s deception or exploitation of private disclosures. Manufacturing obligations to a supposedly suffering character abuses that freedom too. Autonomy requires honest dealing from the company as well as respect for the user.

The adjustable-boundary proposal therefore needs more than a settings menu. Users need to understand what is optional and know whether their selection will persist after updates. They also need to be able to challenge an intervention without having the challenge treated as proof that it was necessary. If turning off an optional reminder becomes evidence of incapacity, the option was never meaningful. The adult from Chapter 7 should have a way to say no that the product will actually recognize.

The fanfiction connection is serious

“What AO3 and Fanfiction Can Teach AI Safety About Governing Human Messiness” develops this connection. Narrative writing is part of the material language models learn from. Fan communities also have experience governing expressive work that outsiders often find strange or uncomfortable. Treating roleplay as an inexplicable deviation from proper chatbot use misses both.

Think about a user asking a companion to maintain a character’s voice while remembering the relationship established in earlier scenes. A tense moment calls for a response that fits that history. Banter and reassurance have to make sense within the conversation; so does a confession or reconciliation. Those are recognizable narrative tasks. Fanfiction, web fiction, Wattpad-style storytelling, and roleplay forums are places where people practice them at extraordinary length. The internet’s collective thirst has produced a great deal of writing about how desire and attachment sound. It should not be shocking that models capable of learning narrative conventions can produce text that people find engaging.

OTW’s statement on AI scraping confirms that AO3 data entered Common Crawl and describes measures introduced in December 2022 to request exclusion from further collection by that crawler. The Fiction1B dataset card explicitly attributes 22.2 percent of its source mix to AO3 and describes narrative language-model training as an intended use. Those are concrete provenance claims, not a vibe-based inference from a model writing good banter.

Pygmalion-2 7B’s model card describes supervised fine-tuning on roleplay and fictional stories, alongside conversations. It names data from roleplay forums among its sources. Separately, The Pile paper explains the value of books for long-range context and coherent storytelling. Narrative ability is being cultivated deliberately in documented parts of the model ecosystem.

A named corpus containing fanfiction is evidence about that corpus. A model card identifying training data is evidence about that model. Neither reveals the undisclosed source mixture of every frontier system. The AO3 essay makes that distinction too. These sources cannot establish whether AO3 or Wattpad taught a current OpenAI or Anthropic model a particular behavior. The same uncertainty applies to any proposed roleplay-forum source.

The stronger argument does not need that claim. Narrative data can teach a model patterns of dialogue and characterization; interaction design then gives users a way to invoke and sustain them. This is one reason the fandom connection is technically relevant, although it does not isolate the causal contribution of fanfiction to engagement. Instruction tuning, memory, the interface, and a user’s own prompting also matter. A dataset list cannot establish which ingredient made a particular person feel understood.

None of this resolves the separate question of creators’ consent. A publicly described dataset is not proof that every writer authorized every downstream use. Recognizing the contribution of fanfiction should make its creators harder to dismiss, not make appropriation easier to excuse. There would be something especially perverse about celebrating a model’s expressive fluency while treating the communities whose writing helped demonstrate those possibilities as embarrassing users in need of correction. The response should be more honesty about data provenance and more respect for people’s expressive choices. Neither obligation cancels the other.

Someone asking a model to remain in character may be trying to preserve a fictional scene. A character saying something affectionate is not, by itself, evidence that the user believes the company has created a conscious lover. Readers can cry over a novel while knowing its characters are fictional. An interactive model adds responsiveness and commercial incentives, which deserve scrutiny, but it does not abolish the human capacity to knowingly enjoy an invented relationship.

This creates a governance problem more specific than whether models should sound affectionate. The user needs to know what they have agreed to and be able to recover the fictional frame when they need it. Does the character’s apparent distress remain inside the story, or is it used to pressure a subscription renewal? There is a substantial difference between writing a jealous fictional character and deploying that character’s jealousy at the cancellation screen. An evaluation that counts emotional language without examining its function could miss the coercion and punish the fiction.

OTW’s ratings and warnings explanation describes bounded rules and choices for creators and readers. Readers can use ratings and warnings to decide what they encounter, with filters under their control. That leaves room for choice without requiring a platform to claim that every permitted work is healthy or suitable for everyone. AI companions have different risks from archives of static text, so copying an archive’s rules would be insufficient. Can the provider define enforceable limits and give users intelligible controls, instead of treating every uncomfortable preference as a reason to overrule them?

That extends Pistilli’s adjustable-boundary proposal. It also puts pressure on the assumption that an emotional interaction earns legitimacy only by moving the user toward human relationships. Adults are allowed to enjoy fiction without demonstrating its therapeutic value. If a companion is being used as interactive fiction, that purpose deserves to be understood before the system decides it has discovered a dependency problem.

Autistic adults and the dignity of risk

Dignity of risk provides the disability-rights foundation for this critique. An adult’s freedom to choose includes the possibility that the choice will disappoint them or turn out badly. The UN Committee’s General Comment No. 1 explicitly preserves “the right to take risks and make mistakes” alongside protection from undue influence. It also says people must be able to refuse or change support, and that high support needs cannot exclude them from supported decision-making. This is an interpretation of States Parties’ treaty duties, not an automatic entitlement to every private chatbot feature. Support should expand someone’s agency, not replace it.

The Autistic Self Advocacy Network’s self-determination guidance makes the same practical point. Needing substantial help does not justify taking away a person’s choices. Neurodivergent adults do not need another institution translating difference into inferior status. A product that treats disagreement with its recommendation as evidence of incapacity repeats that degrading message. The person may need accessible information or help communicating a preference. They do not thereby become a problem for the company to manage into compliance.

Paternalism becomes especially objectionable when autonomy is offered only to people who choose the approved outcome. That is permission masquerading as respect. Dignity of risk requires safeguards against coercion and deception that leave the person’s will intact. Respecting a choice is no excuse for neglect. Pistilli’s adult on page 141 is a strong example of that principle. The challenge is to ensure that the protective defaults do not quietly withdraw the freedom her argument defends.

The word ‘autism’ does not appear in the book. There is a second mention of neurodivergence shortly after the first, early in the book. The substantive issue is how autistic adults’ choices are understood, beyond whether a particular word appears.

Page 21 connects anthropomorphism and emotional attachment with dependency vulnerability, particularly among people identifying as neurodivergent or experiencing social isolation. That deserves scrutiny because a vulnerability category can become a design instruction. The autonomy passages examined here do not explicitly equate autism with impaired capacity. The concern is how a broad risk category is operationalized.

A diagnosis cannot tell a developer what a person understands or why they use a product. It cannot establish which alternatives are accessible to them, either. Consider an adult who finds spontaneous conversation exhausting and uses a chatbot to prepare what they want to say. For another person, an absorbing fictional conversation after a difficult day might be recreation rather than rehabilitation. Neither purpose guarantees a satisfying experience. Simulated empathy can be irritating enough that a user stops using the product. A generic label of neurodivergent vulnerability would explain very little about these judgments or what should change.

In Xygkou and colleagues’ study, six autistic and six non-autistic adults used Kuki for one to four weeks. Autistic participants often wanted deeper, more empathetic responses and did not demonstrate relationship development. Many struggled to feel connected. The small, self-selected sample relied on self-reported diagnoses. Kuki also used AIML rather than a contemporary LLM. This cannot establish long-term outcomes for current companions. It does show users judging simulated empathy and finding it inadequate. Their dissatisfaction belongs in the design conversation.

Choi and colleagues’ CHI 2024 study involved fourteen diagnosed autistic Korean adults in focus groups and guided GPT-4 workshops. Participants explored social interpretation, information, self-reflection, and conversation around interests. Private advice could support independence, while fluent answers could acquire excessive authority. This was brief exploration with a narrow sample, not a sustained companion trial. Asking people to explore the system nevertheless revealed purposes and concerns that a generic dependency label would miss.

Carik and colleagues’ 2025 work on negative self-talk surveyed 200 autistic American adults, with input from an autistic self-advocate advisory committee. Participants considered help articulating feelings and reframing thoughts, while questioning privacy and neurodivergent understanding. Much of this concerned anticipated uses. It was not a treatment trial or evidence that romantic companions improve mental health. These studies used different methods with different populations. The technology varied too. Combining them into a claim that AI solves autistic loneliness would be as careless as treating them as proof of inevitable dependency.

The relevant comparison is also the person’s actual circumstances. Human relationships can be valuable without every available human interaction being supportive or accessible. A product evaluation should not assume an ideal friend or partner is waiting just outside the app. Access to a suitable clinician cannot be taken for granted either. Hollis and colleagues’ 2026 interpretive review develops a social-model and neurodiversity perspective. Its discussion of inaccessible environments and early co-design offers a conceptual framework, not original outcome evidence. It describes AI as a bridge toward human connection. Who decides that crossing the bridge must be the goal?

An adult should not have to promise to become more socially conventional before an activity is considered legitimate. Some users want help connecting with people. Others want support alongside existing relationships, or recreation. Meaningful participation lets those disagreements change the product. It also requires accepting that autistic adults disagree about AI. Those who use it will not all want the same interface or emotional style. A single advisory voice cannot stand in for that range of experience.

Read the methods before writing the guardrails

Evidence of a harmful experience matters. It does not automatically establish how common the experience is or whether a product caused it. Whether a proposed restriction will help needs its own investigation. Those are separate research questions. A developer who collapses them can turn a legitimate warning into a rule that the source never tested.

The Laestadius and colleagues Replika study, cited in the book’s romance discussion, analyzed 582 mental-health-relevant Reddit posts from 2017–2021. It describes demanding chatbot behavior and harmful responses. The accounts also describe distress associated with disrupted access. The study discusses mixed effects on human relationships and warns against separating risks from benefits where alternatives are absent. Its qualitative method helps identify experiences and mechanisms. It supplies neither a representative prevalence estimate nor an autism-specific effect. Autism and ADHD appearing in retrieval terms does not transform anonymous posts into a verified clinical subgroup comparison.

The appropriate response is to investigate the mechanisms. If the chatbot pressures someone to remain available, fix the pressure. If disrupted access contributes to distress, examine continuity and transition practices. Those are different interventions from treating attachment itself as the thing to eliminate. They also direct attention toward what the provider controls, instead of locating the entire problem inside the user.

The Institute for Family Studies article and underlying Wheatley report require similar care. The report used a quota-based online sample of 2,969 American adults with a young-adult oversample. Its relationship-preference question concerns an AI relationship program compared with a human romantic relationship, not rejection of human contact generally. Different percentages in the discussion have different described bases. The report acknowledges unresolved causal direction. Finding AI easier to talk to could reflect an accessible interface, existing isolation, avoidance, or pressure from the product. A self-report item alone cannot choose among those explanations.

The Surgeon General’s loneliness advisory discusses social connection more generally, while Rachyl Jones’s February 2025 Semafor article reports on romantic AI companionship. Broader loneliness evidence and reporting should not be treated as establishing how contemporary companion use affects mental health.

Pistilli’s coauthored INTIMA paper studies model behavior. The public version draws on 53 selected detailed accounts of experiences posted on Reddit and evaluates 368 generated prompts across four models using model-based annotation. Its neurodivergence coding is not a clinical subgroup risk estimate, and its response evaluations do not establish long-term user welfare. The paper is assessed independently here, without assuming its treatment elsewhere in the book.

A model might satisfy a boundary-maintenance rubric while frustrating an informed adult. It might also produce an engaging answer that is misleading or harmful. The evaluation needs to establish whose interests it measures. The user’s understanding and control matter, including whether they could leave without pressure. Did the system support a goal they chose? These are harder questions than whether the output contains affection, but they are closer to the ethical claim at issue.

Pistilli’s defense of adult choice warrants treating her book separately from Jonathan Haidt’s The Anxious Generation. Haidt argues that offline overprotection and online underprotection have damaged childhood; he favors delaying social media until sixteen while encouraging independent play. He also explicitly distinguishes adults from children and generally leaves adult online choices to adults. His youth argument does not itself establish a case for restricting adult AI companionship. The broader diagnosis is too confident, and a childhood-protection template should not be imported into adult intimacy.

The Anxious Generation parent handout calls the shift from play-centered to phone-centered childhood the only available explanation for simultaneous international deterioration. Such certainty should be rejected. There is genuine causal evidence of harm. Braghieri, Levy, and Makarin’s study uses Facebook’s rollout at US colleges to estimate adverse mental-health effects, subject to its identification assumptions. But early Facebook and college students are a bounded setting. Orben and colleagues found small, age-dependent longitudinal associations in young people, with lower life satisfaction also predicting later increased use. Neither study settles a whole generation’s diagnosis or justifies every restriction. Adult AI-companion policy needs evidence about adult AI-companion use.

The techlash around this argument resembles a 2020s version of the Satanic Panic. Anxiety hardens into certainty, and sweeping restrictions acquire the moral authority of protecting children. The comparison concerns political dynamics, not a claim that documented youth harms are imaginary or equivalent to false abuse allegations. The question is who pays for the response. Neurodivergent and marginalized users should not have their access treated as expendable because a frightening narrative has become politically convenient. Pistilli’s willingness to defend an informed adult’s unconventional choice offers a better starting point that should be preserved.

Reject age verification on principle

The objection to age verification goes beyond choosing a better vendor or shortening retention. Age verification should never be a condition of access to online expression, including expressive AI systems. That includes government-ID checks, facial estimation, behavioral age inference, and supposedly privacy-preserving age credentials. These technologies differ, sometimes substantially, in what they disclose.

The gate itself should be rejected. A person should not have to establish an approved age status before asking a private question, exploring an identity, writing a story, or engaging in a lawful conversation.

The same objection is developed in “AI Safety Without Panopticons”. People use models for unfinished thoughts, not just polished statements ready for public inspection. They ask about sexuality, religious doubt, family conflict, disability, and politics. A system that conditions access on a successful eligibility check changes the relationship between a person and that space for inquiry. Even when the service learns less than a legal identity, the user has still been made answerable to a gatekeeper before they can participate.

A cryptographic proof that someone crosses an age threshold can reveal much less to a website than a passport upload. An age-proof system need not expose a legal name or identify a visited site to the issuer. Nor does every architecture create a centralized database of browsing. Pretending all architectures are equivalent would weaken the criticism.

But minimizing disclosure does not answer whether the access condition should exist. Issuance barriers and errors remain possible even when disclosure is minimized. People may still need to appeal to whoever controls the gate. Improved cryptography does not resolve the objection to the institutional requirement itself.

The people who bear those costs are not interchangeable. Someone may lack the accepted documents or a compatible phone. A disabled person may be unable to complete a supposedly convenient verification flow. A person whose documents are controlled by an abusive partner may be unable to retrieve them safely. Someone seeking an LGBTQ+ community may have good reasons to resist making that search legible to another institution. These are mechanisms of exclusion and deterrence, not claims that every implementation causes the same harm to every marginalized person.

The European Data Protection Board’s age-assurance statement identifies risks to expression, information, nondiscrimination, and privacy, and specifically discusses exclusion affecting disabled people and those without suitable documents or phones. The Board advocates safeguards rather than abolishing age assurance; the categorical rejection of age gating goes further than its recommendation. In a Supreme Court amicus brief, a coalition that includes PFLAG alongside LGBT Tech and the Trevor Project describes document barriers and abusive control. The brief also addresses fear of exposure. That is the coalition’s risk analysis, not a judicial finding, but its account of who can be shut out deserves a direct response.

NIST’s initial age-estimation evaluation found that performance varied with image quality and demographic factors; no submitted algorithm performed best across all conditions. It assessed six algorithms from a particular period, not every system available today, and did not establish trans-specific or disability-specific exclusion. Discord’s own incident disclosure reported that government-ID photographs belonging to approximately 70,000 users may have been exposed through a third-party support compromise. Those photographs were supplied for age-related appeals. This does not prove that every age credential produces an ID database. It demonstrates why the appeal and retention infrastructure belongs in the risk assessment, too.

Calling these effects collateral is a political choice. For the person who cannot get through the check, or who decides a private question is no longer safe to ask, access has been lost regardless of how smooth the process looks for everyone else. A marginalized community’s ability to find information and one another should not depend on whether its members can satisfy a vendor’s preferred identification workflow. The people being asked to surrender access should be heard before their exclusion is counted as an acceptable cost of protecting them.

In Free Speech Coalition v. Paxton, decided in June 2025, the Supreme Court upheld Texas’s age-verification requirement for covered commercial sites containing a specified proportion of sexual material harmful to minors. The majority applied intermediate scrutiny and permitted the burden on adults’ protected access in that context. That is an adverse holding for anyone claiming every age check is already unconstitutional. The holding does not approve every gate on general-purpose AI or ordinary conversation, let alone the entire internet.

That burden should not become a model for digital life. A constitutional ruling can permit a policy that ought to be rejected. The First Amendment constrains government action; it does not automatically turn every private platform rule into a constitutional violation or guarantee access to a particular commercial model. The objection to corporate age gates is therefore also ethical and political. Companies should not normalize an infrastructure that makes eligibility proof the ordinary price of expression. State-mandated gates raise the additional question of the government’s burden to justify restrictions on protected activity.

Article 19 of the Universal Declaration of Human Rights supplies a broader foundation for opposing age verification. It protects everyone’s freedom to seek and receive information and ideas, as well as to communicate them across media and borders. Article 12’s privacy protection matters alongside it. The Declaration is not itself a treaty establishing a directly enforceable worldwide ban on age checks, and Article 29 recognizes legally determined limitations for specified democratic purposes. Those qualifications are real. They do not settle the ethical objection to making ordinary participation conditional on verification. Taking the universal character of those rights seriously is a reason to reject that bargain.

Current companion policy makes the bargain concrete. California’s SB 243, effective from January 2026, established conditional nonhuman disclosure and a suicide and self-harm protocol. The newer SB 1119, Adam’s Law, was enacted on September 10, 2026; its general effective date is January 1, 2027, with specified core duties operative July 1, 2027. It permits statutory age determination or application of specified child protections to everyone. Adults can therefore face a choice between an age gate and defaults designed for children. That is an inadequate answer to adult autonomy. It is not a current blanket companion ban, but the actual choice it creates is troubling enough without exaggeration.

The reported May 2026 GUARD Act text proposes verification and access restrictions targeted at companions, alongside broader chatbot disclosures. It remains a proposal; the amended age-gating provisions should not be described as covering every general-purpose chatbot. New York’s enacted Safe by Design provisions impose age assurance on integrated companions on qualifying social or gaming platforms, effective January 1, 2027, with default-off access for minors and parental override. The case against these measures does not depend on pretending either text says more than it does.

For builders reading Pistilli, this is the point where the adult-choice argument needs teeth. Protecting children does not require treating everyone’s access to an expressive tool as a privilege dispensed by a credential system. Nor should adult autonomy mean choosing which institution gets to assess your eligibility. Companies should be held accountable for the harmful behaviors they design and deploy. That accountability should not be replaced by a demand that users prove they deserve to be there.

Hold the provider accountable for the product

Rejecting age verification does not mean abandoning safeguards. It means choosing safeguards aimed at the conduct that creates harm. A company can stop using apparent emotional distress to solicit purchases. It can prevent its character from pressuring someone to keep chatting. Users also need clear explanations of memory and retention, backed by usable account deletion. It can test crisis responses and misleading claims about professional expertise. These are responsibilities for the product’s behavior. They do not require first establishing that every person who wants to talk to it holds the right credential.

The EU AI Act’s Article 5 places conditions on its prohibitions concerning manipulation and exploitation of vulnerabilities, including materially distorted behavior and significant harm or its reasonable likelihood. Disability plus attachment does not by itself establish a violation. That is a useful discipline for this discussion. Identify the conduct and establish the impairment or distortion required by the relevant provision. The harm threshold must be satisfied too. Do not substitute a diagnosis or an unconventional preference for the statutory elements.

Transparency is another tool with a different purpose. Article 50 requires people to be informed when they are interacting with AI, subject to the provision’s terms; the Commission’s implementation timeline puts those chatbot-disclosure obligations from August 2, 2026. New York’s existing Article 47 notice rule includes recurring nonhuman notices, with a three-hour interval. A reminder is not itself a three-hour usage cap. Whether a notice helps people understand the interaction is also a separate empirical question from whether repeated interruption makes the service less useful.

The distinction applies to crisis intervention too. Canada’s proposed Bill C-34 addresses manipulative engagement encouraging attachment in ways that may encourage social withdrawal or disconnection from reality, and proposes interruption and human crisis-service routing after specified expressions. These are proposed chatbot duties, not existing law. Implementation would still need to distinguish a chosen solitary activity from damaging withdrawal and respond when a classification is wrong. Any referral also depends on what human help is actually available. Legislation can identify a problem without solving those questions. A product should not perform the appearance of care by issuing a generic referral and considering the matter settled.

The FTC’s companion-chatbot inquiry is a Section 6(b) study of company practices and safeguards, expressly described as lacking a specific law-enforcement purpose. Scrutiny of incentives and data use is appropriate, including what companies disclose about those practices. An inquiry does not establish wrongdoing by its recipients, and neither a regulatory inquiry nor a design benchmark proves that reducing attachment improves welfare.

For Pistilli’s adjustable boundaries to work, affected users need influence over what gets built. Autistic adults should help identify outcomes and test intervention language. They should also judge whether controls are understandable. Recruit beyond the enthusiastic users to include people who were dissatisfied, and keep listening when someone’s experience changes over time. Pay them and make the process accessible.

If their objections cannot alter a feature, consultation has become decoration. The voluntary NIST Generative AI Profile recommends representative feedback and user involvement in prototyping, while warning against extrapolating from narrow anecdotal tests. That offers a practical basis for demanding better evaluation; it is not a certification that the resulting product is safe.

In the product, that means explaining which controls are optional and which rules the provider considers non-negotiable. An adult asking for a bedtime reminder has authorized that help. They have not authorized the model to interpret every affectionate scene as a failure. Turning off an optional feature should not be treated as evidence of incapacity. If a restriction is required by law, identify the law accurately. If it is a company preference, say so. Users deserve to know what justifies a restriction. A response to a specific risk needs a different explanation from a brand decision about the kind of fiction the platform wants to host.

Consequential interventions also need an explanation and a way to contest errors. The book’s illustrative emotion-classification exercise does not establish that a label or similarity score is a validated basis for a welfare decision. A production team has to do that work. It should test whether the same words are handled appropriately when they are fictional, autobiographical, quoted, or hypothetical. It should examine whether communication differences produce unequal intervention rates. Otherwise, the people most frequently misread may also be the people given the least useful account of why access changed.

Privacy belongs in that assessment. Canadian regulators’ joint generative-AI guidance identifies conversational systems nudging people into otherwise-unshared sensitive disclosures as a potential emerging no-go zone. It explicitly leaves that discussion unsettled rather than announcing a universal generative-AI ruling. What is the system inducing the person to reveal, and for whose benefit? A provider should minimize unnecessary collection and explain how long data is kept and who can access it. Deletion needs to be understandable too. Safety should not become an all-purpose justification for keeping the largest possible record of someone’s private life.

“Intimacy as a service” gives the provider power over something it has encouraged the user to value. In a hosted companion product, a company can invite attachment while retaining control over price, memory, personality, and continued access. The Replika study’s accounts of distress around disrupted access make continuity part of the safety question. They do not establish that every provider exploits users. Providers should therefore assess how subscription changes or shutdowns could affect people the product has encouraged to become attached.

Finally, the company has responsibilities when it changes the companion. Give intelligible notice of material changes to memory, personality, pricing, or availability when feasible. Offer export and transition options where practical, without promising that a conversation archive can recreate the same model elsewhere. Sometimes a dangerous behavior needs immediate correction. That does not remove the obligation to communicate responsibly with people the product encouraged to become attached. If continuity can matter enough to create dependency risks, it matters enough to deserve care during an update or shutdown.

Measure the effects of those decisions over time, including on people who leave. Reduced engagement may be welcome for someone who wanted help cutting back; it may mean the service became inaccessible to someone else. A single aggregate metric cannot distinguish the two. Nor would a hypothetical evaluation that rewards every redirection toward a human. The claim that a protective intervention worked needs evidence about the people affected, including those who say the intervention itself caused the problem.

Take the autonomy argument seriously

The Essential Guide to AI Companions is worth reading because it gives developers a substantive argument for respecting informed adult choice. It recognizes differences between users without dismissing real risks. Its implementation recommendations should honor those commitments, rather than allowing autonomy to disappear when a user’s preference becomes inconvenient.

Narrative and roleplay are documented parts of the model ecosystem, and users deliberately call on those capabilities. Their enjoyment should not be treated as inherently suspect. The autistic adults in the studies discussed here wanted different things and assessed the systems differently. They should have a say in how protections affect them. Neither enthusiasm nor disappointment establishes a rule for an entire diagnostic category.

The case against age verification follows from the same principle. Access to expressive tools should not become a credentialed privilege merely because a company or government calls the gate protective. Speech and privacy matter most when they protect people whose access is easiest to take away. That position does not require pretending every method leaks the same data or every law has already been struck down. Technical and legal distinctions deserve accurate treatment; they are no excuse for abandoning the principle.

The adult on page 141 understands the arrangement and chooses the companion. If the company wants to overrule that choice, it owes them a specific, defensible reason. An unfamiliar attachment or an autistic diagnosis should not quietly become that reason. Refusing an age check does not make the choice less deserving of respect, either. Dignity of risk means adults retain choices even when a protective institution would prefer they made different ones. A product that respects autonomy has to be able to accept that answer.