I like how frank the report is, no sugarcoating. "We relied on manual error prone verification and made a mistake. We have to automate the process."

As ARIN block owner this situation is kinda scary but reading this actually makes me think it's less likely to happen again .

▲

netfortius 10 hours ago | parent | next [-]

The road to automation is always full of outages.

▲

stefan_ 10 hours ago | parent | prev | next [-]

I'm curious how these fellas took something like IP block allocation and turned it into an Excel based workflow.

▲

jonathanlydall 8 hours ago | parent | next [-]

“Workflow” is probably a bit generous to describe how they probably use Excel.

Having worked at a mom and pop ISP a couple of decades ago where we used Excel to track a lot of things, I can see how this might have happened.

To actually know who is allocated what is ultimately just a list.

And when there are only a few people who edit the list (and probably no more than 1 person at a time) you can get by with even a plain text file, but Excel is quite a bit nicer as you can do things like filtering and sorting easily, maybe even some formulas to help with things.

Building a program backed by a database might be nice, but hard to justify when the manual system has never been a problem before.

They’ve probably been thinking for a while they should, but it’s just never been enough of a pain point for them to invest the effort.

Looks like they see this incident as justification that they need a system with hard coded rules and constraints, no more manual checking.

	▲	stefan_ 3 hours ago \| parent [-]
		It's ARIN, this is essentially their only job

▲

mmooss 2 hours ago | parent | prev [-]

The world's financial systems run on Excel, to a great extent.

I'm more surprised that a single person, apparently without seniority, could delete a block. IME deleting user data is usually a significant event; an IP block would especially be a big deal, especially for the IP block issuers. From the OP:

> RSD has implemented additional process controls that require a dual review for all ticketing type workflows that include a network delete.

> Only a limited set of experienced analysts are permitted to perform this function.

Great that they didn't blame the person who deleted it. ARIN seems to have put them in position where a failure was likely, eventually. Without any inside knowledge, I'd hope the culture would have any engineer leary about pressing that button without a second set of eyes reviewing it carefully and without clear authorization; I don't imagine they delete many blocks each day so it shouldn't interfere with productivity.

▲

anonnon 11 hours ago | parent | prev [-]

You don't find this part

> We have to automate the process.

to be ominous?

▲

Aurornis 10 hours ago | parent | next [-]

I don’t. The report says part of this process relied on flat files and spreadsheets. Automating that with software is a good idea.

“Automate the process” doesn’t mean feeding everything to an LLM.

▲

aaomidi 10 hours ago | parent | prev [-]

Certificate issuance was once only possible manually.

	▲	qingcharles 9 hours ago \| parent [-]
		Domains too, well into the 90s.