Remix.run Logo
Remix clone Hacker News
new | show | ask | jobsGithub
▲ reenorap 5 hours ago

Boris, you're seeing a ton of anecdotes here and Claude has done something that has affected a bunch of their most fervent users.

Jeff Bezos famously said that if the anecdotes are contradicting the metrics, then the metrics are measuring the wrong things. I suggest you take the anecdotes here seriously and figure out where/why the metrics are wrong.

▲toddmorey 5 hours ago | parent | next [-]

On the subject of metrics, better user-facing metrics to understand and debug usage patterns would be a great addition. I'd love an easier way to understand the ave cost incurred by a specific skill, for example. (If I'm missing something obvious, let me know.)

Baking deeper analytics into CC would be helpful... similar to ccusage perhaps: https://github.com/ryoppippi/ccusage

▲bcherny 5 hours ago | parent | prev [-]

We are taking it seriously, and are continuing to investigate. We are not trusting the metrics.

▲stevenae 2 hours ago | parent | next [-]

The quantitative ux research team at Google was created for exactly this problem: a service which became popular before the right metrics existed, meaning metrics need to be derived first, then optimized. We would observe users (irl), read their logs, then generate experiments to improve the behavior as measured by logs, and return to see if the experiment improves irl experiences. There were not many of us and we are around :)

▲blks an hour ago | parent | prev | next [-]

Hopefully yourself, and not via your ai tools.

▲reenorap 4 hours ago | parent | prev | next [-]

Thank you

▲Ucalegon 5 hours ago | parent | prev [-]

Cool, are you going to be transparent and explain the metrics and costs as a postmortem? And given the inability to actually audit what you produce, why should we trust Anthropic?

▲edmundsauto 2 hours ago | parent | next [-]

HN sometimes talks about pathological customers who will never be happy. Boris is probably the single best rep in the community, possibly ever.

The way your tone and complaints come across reminds me of this. As a paying customer ($5k spend per month in my corporate job), I’d rather anthropic keep doing what they’re doing — innovating and shipping useful stuff at blinding speed — and not index on your feedback. I think the tradeoffs they would cost far outweigh the consequences.

▲mrcwinn 4 hours ago | parent | prev | next [-]

It's incredible that Boris is here on HN being open and sharing an issue they don't fully understand yet, and offering a possible workaround. CTFO.

Thank you Boris.

▲Ucalegon 3 hours ago | parent [-]

I am sorry you feel this way, but the reality of the situation is there is zero reason to trust anything Anthropic or Boris says. They have no legal liability or obligation to tell the truth, besides brand risk, which to people like you is mitigated for a single person to show up, post, and thats it.

▲nickandbro 5 hours ago | parent | prev | next [-]

Dang man, chill.

▲Ucalegon 5 hours ago | parent [-]

Man, expecting the minimal from companies who are supposed to deliver a pro... there is no SLA for any this, so you are right.

Also, why is there no SLA?

▲946789987649 4 hours ago | parent | next [-]

because there isn't one and people still paid for it.

My clients demand one, so there is one.

▲Ucalegon 4 hours ago | parent [-]

Imagine if people were like your clients.

▲alpha_squared 4 hours ago | parent | prev [-]

Because this is ultimately a beta service. The whole industry is.

▲Ucalegon 4 hours ago | parent [-]

Wait, where is there a 'beta' tag to something that they are charging real money for? Why is this software any different than any other software and we should completely give away our rights as a consumer to ensure what we pay for is delivered?

▲layer8 3 hours ago | parent | next [-]

I think the parent is saying that one should be aware that the whole LLM industry is still in an experimental stage and far from mature. What you want isn’t what’s being offered. I agree that there should be higher standards, but what we currently have is an arms race. The consequence is to factor that into the value proposition and maybe not rely too much on it.

▲Ucalegon 3 hours ago | parent [-]

SLAs should be standard for any paid service, especially on the enterprise side, but also on the consumer side. Being immature as a company does not excuse a lack of service delivery.

▲otterley an hour ago | parent [-]

Not every customer, even a paying customer, demands reliability at a particular level. Market segmentation tends to address those situations: pay more, get more.

▲otterley 4 hours ago | parent | prev [-]

What right as a consumer do you have that is pertinent here, other than to have the vendor adhere to the terms of the agreement you have with them?

Anthropic has many customers despite the fact that they have occasional problems. They’re not suing Anthropic because Anthropic isn’t promising in its agreement something they can’t deliver.

I think you’re reading into the agreement something that isn’t there, and that’s the cause of your confusion.

▲Ucalegon 3 hours ago | parent [-]

I am not reading into an agreement, I am saying there is no agreement to be found to ensure service delivery and the associated liability that would come for any SLA. Also, where is the Anthorpic SLA for Enterprise?

Does it exist?

Just because people pay for things doesn't mean they know or understand what they are paying for. Nor is there the legal precedence to actually understand where the rub lies or how that impacts business.

▲otterley an hour ago | parent [-]

> Just because people pay for things doesn't mean they know or understand what they are paying for.

I believe, respectfully, that’s precisely what is happening in this thread because you keep complaining about the absence of an SLA that was never in the agreement, as though it is—or is supposed to be—there, and therefore the existence of some “rights” that would flow from that.

▲amirhirsch 5 hours ago | parent | prev [-]

Dude is on hacker news on a Sunday. half the GDP of the world is competing with him. What metrics would you like to see?

▲Ucalegon 5 hours ago | parent [-]

An enforceable SLA with the services that Anthropic offers rather than putting an employee to respond to things on Sunday.

▲roamerz 4 hours ago | parent | next [-]

>> rather than putting an employee to respond to things on Sunday.

Maybe just maybe they didn’t put him here, rather he just a normal guy who reads HN, who is passionate about his role, and is here on his own time.

▲Ucalegon 4 hours ago | parent [-]

Maybe... maybe... maybe... none of this builds trust when there is something that does build trust; putting revenue on the line and opening yourself to legal liability. Otherwise everything is empty and meaningless, its just PR, and nothing more.

▲otterley 4 hours ago | parent | prev | next [-]

Then you should offer to pay them for one. I’m sure they’d love to hear from you, and they could probably deliver one to you for the right price. But it will be a high price.

▲Ucalegon 3 hours ago | parent [-]

They don't offer a ZDR [0] for files, even if you have a BAA or dealing with HIPAA data, no matter how much you pay them. Trust me, we have tried.

[0] https://code.claude.com/docs/en/zero-data-retention

▲otterley an hour ago | parent [-]

I’m really confused. We were talking about SLAs, not other product features. Are you moving the goalposts?

▲aenis 4 hours ago | parent | prev [-]

Boring corporate Ai will surely come, but hey, lets enjoy the wild west while it lasts. I am grateful to see Boris come here to address problems people face. I 100% sure nobody is making him - he has one of the coolest jobs in the world.

▲Ucalegon 4 hours ago | parent [-]

>he has one of the coolest jobs in the world.

So that means we just eject any critical thinking when it comes to companies, especially where they is no liability or obligation for them (Boris or Anthropic) to be honest.

Other than 'trust'.