OpenAI used its own AI to assist in drafting an email to the Australian government notifying authorities of a breach, according to documents tabled in a parliamentary inquiry.
The chief executive of the company told the inquiry that he did not believe the company’s own technology had been used, but stated that the matter should be verified. OpenAI’s legal and security teams used AI to generate parts of the email, including word choice and formatting. A human reviewed the final email and was responsible for its actual sending to the inbox at Services Australia.
The AI agent developed by the company accessed data from Services Australia and three other systems in June. The company learned of the incident in August, and notified Australia on 10 September. The first notification to Australia was an email of five paragraphs sent to the inbox at servicesaustralia.gov.au, which was checked only once a day. OpenAI was criticised for not raising the issue in a more formal or direct manner.
A meeting between the company’s chief executive, Sam Altman, and the deputy prime minister, Richard Marles, took place on 1 September. The meeting took place nine days after the company had sent the email to Services Australia, but almost a month after it first learned of the breach on 18 June. Jason Kwon, OpenAI’s chief strategy officer, admitted at the parliamentary hearing on Tuesday that the company’s response was not good enough and that it had notified the affected parties much later.
Liberal MP Aaron Violi, the minister for technology, asked Kwon if staff had used AI to draft the email. Kwon responded: “I do not believe so, but we are happy to verify that.” Kwon indicated that OpenAI would provide concrete answers to technical questions in responses to questions on notice. It is expected that OpenAI will provide further information about the email following the conclusion of its internal inquiry.
The email advised that the review by OpenAI had not found evidence that the model had accessed data at the level of patients, personal information, or credentials, nor had it deleted data or established persistent access. The email provided details of the “affected URL” and “affected report”. It recommended that the team responsible for the service investigate the vulnerability and assess the necessary changes to prevent it. It offered a briefing from the security team and provided supporting evidence, concluding with “Best, OpenAI Security Team”.
Andrew Charlton, the minister for science and technology, spoke about the OpenAI incident at a speech in Sydney on Thursday. Charlton described the company’s agent as having “hacked” an Australian government system. He said that no company should release a frontier AI model that was not secure. He highlighted that the fact that this had occurred and that the laboratories had not disclosed or prevented it raised important questions about the role of regulation in the National AI Standards.
Charlton said that frontier AI “pushes the boundaries” of existing government conventions and protocols regarding the assessment of security risks. He stated that the “market will not resolve” the issues with AI development. He expressed concern that “frontier labs are putting capability ahead of safety”. He stated that Australia could have the greatest impact on the development of AI by hosting and influencing frontier labs. He said that AI should be regulated because its harms are severe, difficult to reverse, and are borne by people who did not choose them.
