Member

433: After Clause 213, insert the following new Clause— “AI chatbots: offence(1) It is an offence to create, supply, or otherwise make available an AI chatbot which produces content specified in subsection (4).(2) It is an offence to fail to suitably and sufficiently risk assess an AI chatbot which produces content available in subsection (4).(3) It is an offence for a provider of an AI chatbot which produces content available in subsection (4) to fail to effectively mitigate and manage the risks of harm to individuals.(4) Content is covered by this section if it consists of—(a) illegal content, as defined by section 59 of the Online Safety Act 2023, or(b) where the content is presented to a user aged under 18—(i) content or activity which is harmful to children, as defined by section 60 of the Online Safety Act 2023,(ii) content or activity which is presented by an AI chatbot that mimics a human in a manner likely to mislead a child as to its non-human nature,(iii) content with an exploitative design, where content is presented with the intent or effect of detrimentally or coercively extending the user’s engagement with the AI chatbot,(iv) content that is detrimental to equality of treatment, or(v) content which is detrimental to the risks to the privacy of individuals and security of personal information.(5) A risk assessment under subsection (2) is suitable and sufficient if it—(a) identifies and understands the risk of harm, including through product testing and red teaming,(b) is kept up-to-date, including when changes are made to the chatbot’s design or operation, (c) takes into account matters listed in sections 10(4) and 11(6) of the Online Safety Act 2023 (safety and risk assessment duties),(d) has regard to any guidance produced by Ofcom on risk assessment under the Online Safety Act 2023,(e) assesses the risks to equality of treatment of individuals,(f) assesses the risks to the privacy of individuals and security of personal information,(g) assesses the risks arising from the choice of underlying models, data sets and computational tools, and(h) is in an easily understandable written format and includes details about how the assessment was carried out and its findings.(6) As well as the requirements of subsection (5), a risk assessment under subsection (2) for a companion chatbot is suitable and sufficient if it—(a) assesses the risk of harms arising from its use in relation to—(i) addictive design,(ii) deception(iii) sycophancy,(iv) scheming,(v) emotional manipulation, and(vi) disinformation;(b) takes account of the characteristics and vulnerabilities of different end-user groups and takes appropriate mitigating steps in relation to each of those risks;(c) ensures that there are clear technical or functional boundaries in the companion chatbots limiting emotional intimacy,(d) ensures that the chatbot has language and framing suggesting that engagement with the companion chatbot is or should be exclusive of other relationships;(e) ensures that the chatbot periodically encourages human interactions, provides information about local, age-appropriate activities and refers where relevant end-users to appropriate professional support;(f) prohibits design patterns that simulate the permanence or irreplaceability of the chatbot.(7) A provider of an AI chatbot effectively mitigates and manages the risks of harm to individuals under subsection (3) if they—(a) implement moderation systems to prevent the chatbot from generating or endorsing illegal content or, where the chatbot is accessible by children, content harmful to children,(b) design the system to recognise and handle ambiguous, illegal inputs, or inputs harmful to children inputs appropriately,(c) implement systems to provide appropriate fall back responses and escalation procedures,(d) introduce processes or systems that allow end-users or affected persons within the meaning of section 20(5) of the Online Safety Act 2023 to flag inappropriate content,(e) update dialogue management and content control systems based on new data and emerging risks,(f) provide and enforce terms of service,(g) ensure—(i) where a chatbot is asked for information on health matters, the chatbot refers the end-user to the relevant NHS website; (ii) where a chatbot is asked for information about a current UK election, the chatbot refers the end-user to the Electoral Commission;(iii) where a chatbot is asked for information about suicide or self-harm, the chatbot refers the end-user to the Samaritans or other appropriately qualified service;(iv) a chatbot indicates uncertainty when reliable sources disagree or information is incomplete or unavailable;(v) where a chatbot is asked for information by an end-user about child sexual abuse imagery, the chatbot refers the end-user to Stop It Now or Report Remove as appropriate;(vi) where a chatbot is asked about experience of child abuse, the chatbot refers the end-user to Childline,(h) provide highly effective age assurance for chatbots which are capable of producing primary priority content, and(i) keep a written record of any measures taken or in use to comply with paragraphs (a) to (h).(8) A person who commits an offence under this section is liable—(a) on summary conviction, to imprisonment for a term not exceeding the general limit in a magistrates’ court or a fine (or both);(b) on conviction on indictment, to imprisonment for a term not exceeding 5 years or a fine (or both).(9) Providers of regulated chatbots must publish annual transparency reports that include data showing how they are complying with their duties under subsections (5) to (7).(10) For the purposes of this Act—an“AI chatbot” is a generative AI system, including a deep or large language model, able to generate text, images and other content based on the data on which it was trained, and which has been designed to respond to user commands in a way that mimics a human, or engage in conversations with a user that mimic human conversations;an“companion chatbot” is a regulated chatbot which is designed to mimic human relationships or foster emotional engagement and personal or social connection or which can be used in that way.”