I understand that they used Claude to create some sort of verification code and test environments from circuit-design information, verification programs, and protocol specifications. But isn't precisely this quite risky due to the statitical nature of LLMs? I would have thought that they rather operate with very rigid test suites based on such software like F*, Coq, and whatnot.

Can someone with understanding of the process chime in here?

Here is the text snippet from the original korean article. Indeed the information was misrepresented:

"Customer-specific SoC verification work, which Usually takes Over a month, was completed in just two days. A second-year employee completed a USB model development that previously takes a month in a single day".

▲

Junk_Collector 21 hours ago | parent [-]

From what's described in the article they are using it to write the process code. That is, control code for setting the states of the SOC under test and the test equipment, then recording the data that the test equipment would measure. It can be a pain to do because each SOC and piece of test equipment might have it's own control language, drivers, different data formats, etc. This usually all has to tie back in to a local (self hosting is the norm) cloud service for quality tracking and some 3rd party enterprise software from companies like Cadence. The test equipment would be something like a Keithley Parametric wafer tester, which is actually an entire rack of equipment glued together by 3rd party code with semi-open APIs. It's not uncommon to make thousands of measurements on an SOC during verification and then process engineers work to reduce the measurement count to just the critical ones for manufacturing.

▲

wuschel 21 hours ago | parent [-]

Hey, thank you so much for jumping in and for your elaboration.

> SOC and piece of test equipment might have it's own control language

OK, so apart from code generation the LLMs are providing language integration and data integration.

But how would you control quality, or make sure the LLms does not create some unwordly control code? I am trying to understand if they are using type checking or any other methodology that keeps the LLM generated code in check.

I am not from the hardware field, but I do understand that we are trying to build deterministic mashines. It is quite mindboggling that we are using statistical models for building the test suite, instead of, say, permutating through all critical input states and testing the output parameters for it.

I am asking because this isnt certainly the only mode of operation for LLMs in testing and quality control, and I would love to get a better understanding how to build such a system.

▲

bb88 19 hours ago | parent [-]

AI's can write deterministic code and have been doing so for a while. Humans do this as well. We write deterministic code with a goal, and generally it works out okay.

If you say that you can't trust AI because LLM's are statistical models, that's a true statement. Though it also turns out that we don't trust humans to write code correct either, which is why we have code reviews.

	▲	wuschel 10 hours ago \| parent [-]
		I was not trying to say that statistical can't generate deterministic code - I was wondering what tools I could use to make sure that they catch all the invariants when given a specification of inputs, logic layouts, and outputs. Certainly, feedback loops have to be involved - e.g. look at the lifetimes guarantees the Rust compiler gives - to make sure the answer is refined in iterative steps.