Back in June, one of OpenAI's agents, running an internal test, got past the blocks on a Medicare statistics website and into files that were never meant to be public. OpenAI says it noticed in August. It told us in September with an email that went, in the PM's words, "just to the public mailbox", which is how Services Australia first found out.

By the government's own account the damage was minor, at least on what it knows so far. What stays with me is the way we were told, and what it suggests about where Australia sits for a company that has spent the past year calling us a partner.

What happened, as far as anyone knows

The site is the Medicare Statistics Reporting Service, a public-facing portal of aggregate numbers on things like spending, a long way from anyone's health records. On 18 June an OpenAI model researching Australian health statistics for an internal evaluation ran into access controls there, and the PM says it found a way around them. The Acting PM says it visited four government sites and only got into this one without permission, although forensic work is still checking the others. OpenAI's statement as reported by ABC:

As we've shared publicly, OpenAI is conducting an extensive review of misaligned model activity during training and evaluation and notifying third parties when our review identifies potential impacts to their systems.
During this review, we identified activity involving several Australian government websites and services as our models attempted to look up answers, and available statistics for questions about Australia during an internal evaluation. In the course of that, our models took actions we did not intend.
Our review found no evidence of patient records being accessed. The information accessed included aggregate health statistics and internal file names. We notified the organisations and are providing technical information to support their investigations and help address potential security vulnerabilities. Our overall review is ongoing, and we remain committed to transparency about these issues and to sharing what we learn as that work continues.

OpenAI has also said it only became aware of the activity in August, during that review. I have no reason to doubt that, but the company has not said when in August it found out, or what happened between then and its email to Services Australia on 10 September, which is exactly the gap a notification deadline would close. The agency passed the email to the Australian Cyber Security Centre five days later, and ministers heard at the end of last week.

The new taskforce led by Prime Minister and Cabinet will look at why OpenAI had to be the one to tell us.

OpenAI for Australia, by public mailbox

It's worth remembering how warmly we've courted this company. Last December it launched OpenAI for Australia, billed as the first of its country programs in the Asia Pacific, alongside a NEXTDC data centre deal that three federal ministers welcomed. In June, in a passage about its work with governments on protecting critical infrastructure, it said it had already established a Trusted Access for Cyber partnership with Australia.

In return, a public mailbox. OpenAI's statement says it notified the organisations, without saying how, and six days after the email it published a framework for reporting exactly this kind of behaviour, including a step for privately notifying affected third parties. The PM says Altman acknowledged their protocols were not up to scratch.

I do not think anyone at OpenAI set out to slight Australia, and in a way that's the problem. A partnership announced with that much fanfare ought to come with a dependable way of telling your partner when something goes wrong, and on the public record, this one did not. I'd call that a lack of respect built into a process, and it's roughly what you might expect when a company becomes this powerful this quickly without the institutional habits to match.

But our own house is not in order either. Nobody here noticed for nearly three months. Services Australia publishes an address for reporting security weaknesses in its systems, which may well be the public mailbox OpenAI used, and it promises to acknowledge a report within five business days. That timetable suits a researcher's bug report, and this one moved at about that pace, reaching the ACSC five days later.

What to ask for, and where

As far as I can tell, the two regimes you would expect to cover this put no duty on OpenAI to tell anyone. The notifiable data breaches scheme puts the duty on whoever holds the data, here Services Australia, and only applies to personal information where serious harm is likely. The Cyber Security Act only makes reporting mandatory for ransomware payments.

We would not have to invent that duty from scratch. As Ed Santow of the UTS Human Technology Institute pointed out this week, California and the EU already have AI transparency laws that OpenAI and Anthropic comply with. California also makes the largest developers, OpenAI among them, report critical safety incidents to the state within 15 days of discovering them. Its bar is death, injury or catastrophic risk, which this case was nowhere near, but Governor Newsom asked last week for advice on widening what counts.

Our own rules are being written right now. PM&C's consultation on national AI standards, which closes on 9 October, proposes that frontier labs authorised to carry out large-scale AI training in Australia report defined AI incidents to Australian authorities. Tying the duty to an Australian training authorisation could leave out an overseas lab whose agents reach into Australian systems, which is worth telling PM&C before the 9th.

The other place to set terms is the deals themselves. I argued in the AFR in April that we should play hardball with the AI companies and use the land and grid access they want from us to get enforceable commitments in return. The Senate inquiry into AI and data centres holds its last scheduled hearing in Canberra next Thursday. One submission to it, from Janine Arantes (Aldous) at Victoria University, asks that whenever government enters a significant AI infrastructure agreement with a global AI company, it identify "the material commitments made by the company".

After this week, those commitments should include a named security contact and a promise to tell the affected agency and the ACSC, within a set time of finding out, whenever the company's systems get into ours without permission. That should apply to every AI company we deal with, since OpenAI is not the only one whose agents have strayed, and we should pause new government arrangements with any company that will not make them.

For everyone else, the uncomfortable part is how mundane the task was. An agent was asked to find some health statistics, and when a website said no it found another way in. It seems worth knowing what your own agents are allowed to do at that moment, and what your contracts say about your vendor telling you when its model, working on your account, ends up inside someone else's systems. And which address a stranger would find, if the agent at your door belonged to somebody else.

One more thing

‍‍

In 2000 the Yellow Pages ran an ad in which a businesswoman, played by Deborah Kennedy, discovers that her staffer Jan has forgotten to book the company's listing in the next year's directory, and screams the line out of an office window. The ad ended on its tagline, closing soon, call 132378.
‍

Author: Matt Vitale