Remix.run Logo
cbg0 6 hours ago

No word on model hallucinations in the blog post.

https://artificialanalysis.ai/models/gpt-6-astra?omniscience...

pelagicAustral 6 hours ago | parent | next [-]

Not like common law doesnt already have a lot of hallucination going on.

cobbzilla 5 hours ago | parent [-]

can “penumbras and emanations” compete with hallucinations?

ch4s3 5 hours ago | parent [-]

Common law is all about rummaging around in dead mens’ letters, LLMs are a natural fit.

alexfringes 5 hours ago | parent [-]

Genuinely wondering (aka not snarky): Has anyone found frontier models to provide useful research in the context of European civil law systems?

Your comment made me wonder if there are any halfway-acceptable model benchmarks for law tasks? Specifically I’d love to know how the frontier models’ abilities compare between common law vs. civil law systems. My guess would be that an AI in a common law context should have a clearer idea of how a specific case is interpreted/accepted by (common law) practitioners, whereas trying to rely on AI in a civil law context, like Germany, can be daunting. In a few Germany-specific recent examples, the models feel like they present only (maybe too stubbornly?) the “civil law”-based laws. All while negating much of AI’s research benefits because civil statutes are portrayed as being absolutely accurate, binding, and their enforcement (and thereby the legal reality) being uniformly applied. Am I making this interpretation up? If so, how can I prove myself wrong?

jll29 2 hours ago | parent [-]

I don't know of anything that is working yet, but I know some people working on it (intentionally vague).

Roman law, on which the legal systems of Germany, most of Europe, Turkey etc. are based relies more heavily on statutes than Anglo-American case law, but cases do play a role there, too. That's why a practically useful system also needs to have access to court decisions, commentary etc. - and while the statues are in the public domain, a lot of the other knowledge is owned by specialist publishers (such as: C.H. Beck, Nomos, Mohr Siebeck, De Gruyter, Otto Schmidt, C.F. Müller, Manz, Verlag Österreich, Schulthess, Stämpfli, Dalloz, LexisNexis, Lefebvre Dalloz, Giuffrè Francis Lefebvre, CEDAM, Giappichelli, Il Mulino, Aranzadi, Tirant lo Blanch, Marcial Pons, Dykinson, Tecnos, Lefebvre, Almedina, Gestlegal, AAFDL, Kluwer, Boom juridisch, Wolters Kluwer, Larcier-Intersentia, Anthemis, Sakkoulas, Nomiki Bibliothiki, Universul Juridic, Hamangiu).

Internationally, legal systems appear to undergo a sort of convergence, which means statutes increasingly matter in the U.S. and cases increasingly matter in European law (according to a law partner friend).

heaney-555 5 hours ago | parent | prev | next [-]

The benchmark you linked to shows GPT-6 Astra having the lowest hallucination rate of all tested models.

jdiff 5 hours ago | parent [-]

The blog post makes no mention. If the rate isn't 0% it should be mentioned for a field like this.

TomGarden 5 hours ago | parent | next [-]

Agreed. Hallucinations and reliability are the main hurdles to anything being 'agi' in my book

Tanjreeve 5 hours ago | parent [-]

The definition of AGI is whatever a frontier lab wrote a blog about doing in the last week.

simianwords 5 hours ago | parent | prev [-]

why this maximalism? There's nothing that has 0% hallucination including humans. lets use reasonable baselines.

drakythe 5 hours ago | parent [-]

Humans can be sanctioned and fined, and eventually disbarred if they continue to lie in court filings.

john_strinlai 4 hours ago | parent [-]

presumably that will remain true for the lawyers using the product

jdiff 3 hours ago | parent [-]

So those lawyers will likely want to know about the likeliness of it, strategies to work around it, and overall mitigations.

It might be helpful if the company launching this product included such information in their blog post instead of ignoring it in a field with such a high cost of getting it wrong.

john_strinlai 3 hours ago | parent [-]

agreed, but i dont think there are many (any?) product launch blogs that cover all the fine details and legal minutia of the product.

that all typically comes in pages of terms of service, purchase contracts, SLAs, conversations with your rep, knowledge base articles, and that sort of thing. not in the initial marketing post.

probably available by the "contact legal sales" link at the top of the post https://openai.com/business/contact-sales-legal/

jrflo 5 hours ago | parent | prev [-]

Does hallucination matter for this application? We've moved beyond raw recall being that important, it seems like for law specifically all relevant facts will be cited and checked easily by humans.

Implicated 5 hours ago | parent | next [-]

> it seems like for law specifically all relevant facts will be cited and checked easily by humans.

Don't be too sure about that. [0]

0: https://www.damiencharlotin.com/hallucinations/

jakevoytko 4 hours ago | parent | prev | next [-]

> all relevant facts will be cited and checked easily by humans

I've talked to a lawyer about how they handle this. They do indeed double-check everything, since it'd be embarrassing (or worse) to send hallucinated statements to opposing council or to the court. They still find the assembly a huge time saver

But based on stories in the news on the subject, not everyone has this same level of diligence

samtp 3 hours ago | parent | prev | next [-]

Sure but just like generating 100x more code, someone has to review it. So you are wasting everyone in court's time (defendants, prosecutors, judges, staff) by making them parse through what is quite often a bunch of hallucinated slop. Time that could be much better spent on parties who prepared and reviews their own arguments.

jordanpg 2 hours ago | parent | prev [-]

The lawyers I know are very fixated on the problem of hallucinated case citations which is amusing to me as a onetime programmer, since case citations have a well-defined syntax and would be relatively easy to check programmatically.

massysett 2 hours ago | parent [-]

Which makes it all the more bizarre that LLMs have this problem. Claude Code runs the code it generates through a compiler, why can’t an LLM run its product through a cite checker? I’ve seen LLMs fabricate citations.

heaney-555 an hour ago | parent [-]

A modern frontier model with an agentic harness will check its citations if the system instructions or prompt tell it to.

You can get an effectively-zero hallucination rate with the right setup already.