| ▲ | lmf4lol 2 days ago |
| We only look IF they ask us. Our users aren't engineers and often get stuck and even the LLM can't figure it out. Then - AND ONLY THEN - we jack into the sandbox and help them. For instance, one customer has 15(!) cronjobs. And they all used LLM calls to do stuff. So they ran out of tokens super quick and asked us for help because they didn' tknow what to do. Its not an engineering company, the owner has a sales background. So we went in and wrote scripts for them so that the cron calls don't need to use the LLM anymore. |
|
| ▲ | sksksk 2 days ago | parent | next [-] |
| That’s interesting, so in that scenario, you fixed it for that customer. How do you then ensure it’s fixed for customers in the future? Do you add instructions to agents to write scripts where possible? |
| |
| ▲ | lmf4lol 2 days ago | parent [-] | | Yes. Or we make a skill of it and offer it via our skills marketplace. Depends really on what the fix was. Sometimes its really a one-off issue, and sometimes it makes sense to roll the new "insights" out to everyone. | | |
| ▲ | sksksk 2 days ago | parent | next [-] | | Thanks! Another question I have, is from a product perspective, how do you decide how much of the internals you show to users (like token usage). I always thought that seems like (from a non technical users POV), like a implementation detail; and quite hard to map to actual usage. For example, one request might use lots of tokens, and another will barely use any; but it can be difficult for a non technical user to undertand the difference. | | | |
| ▲ | newswasboring a day ago | parent | prev [-] | | I am not a user of your service, but a "system diagnostics" skill is something I have been meaning to develop for my hermes instance. Things like checking if I have left any running servers, cleaning up consumed linux ISOs from jellyfin, check for any repeated tool failures etc. Also its fun to say "run system diagnostics". |
|
|
|
| ▲ | customguy a day ago | parent | prev | next [-] |
| > For instance, one customer has 15(!) cronjobs. But they didn't ask US for help. We didn't need to know a customer of yours is running 15 cronjobs. Even if this factoid itself may be harmless, it's a good thing to practice on how you talk about these things. I'm not saying you're doing anything wrong, I don't know that, but I totally get why how you talk about it rubs some people (including me) the wrong way. > Its not an engineering company, the owner has a sales background. Again, it's good if you have a close relationship with your customers, but we don't, and they don't have one with us. |
| |
| ▲ | lmf4lol a day ago | parent [-] | | Thanks for the constructive feedback. I think you may be right about this.
It thought it was anonymous enough in the way I phrased it. Just wanted to be honest. | | |
|
|
| ▲ | m4rtink 2 days ago | parent | prev [-] |
| Even just the possibility anyone other than the user can "take a look" makes a service like this unfit for purpose. |
| |
| ▲ | weird-eye-issue 2 days ago | parent [-] | | You do realize basically every service does this right? | | |
| ▲ | ryandrake a day ago | parent | next [-] | | At the company I work for, a human being gaining access to “look” at a user’s actual prod data (anything that might contain PII) is super rare and very, very difficult. And I’m not even in Europe. At least a dozen people need to approve the access, including people very senior in leadership, you need to describe exactly what data you need to see, what the business purpose is, what steps you are taking to anonymize the data (can you live with a redacted view?), exactly how long you need that access, how you are going to destroy any copies after that access is finished, and so on. This employee access, IF approved, gets logged and can be scrutinized later. And I have absolutely no doubt if you misused that access it would be noticed and you’d be fired (it has happened). I’m pretty confident that nobody at my company is going on the internet and posting “OMFG LOL our users have such crazy data u should see it LOL.” I always assumed most SeriousBusiness companies over a certain size, not run by 20 year olds, operate similarly. | | |
| ▲ | csallen a day ago | parent | next [-] | | If that's all true then it's almost certainly because you work at a large, mature, bureaucratic, unusually privacy-focused company. Verrrrry few companies work like this when they're small and new. I'd round down to ~0%. And I would bet that, if your company was ever a small startup, it didn't work like this back then, either. Also, this idea that it's bad or weird to anonymously discuss customers is strange. I've known cops, doctors, dentists, and all sorts of other people who routinely share stories about weird things that happen with people and customers at work, anonymously. This is the norm in the world and always has been. It's also the norm for the companies who own your data to look at it, when asked to. What do you think customer support even is? It's only really a small corner of the tech world that treats these things as strange and sacrilege. | |
| ▲ | m4rtink a day ago | parent | prev | next [-] | | Yeah, that is the correct behavior what I would expect in a business setting - its a basic decency, not to mention customers switching to competition if they find out their data is not protected. | |
| ▲ | weird-eye-issue a day ago | parent | prev | next [-] | | > anything that might contain PII Sounds like your software might require HIPAA compliance in which case everything that you just said makes sense and is likely required in that context. But for the average SaaS that doesn't need HIPAA compliance it doesn't work like that. | |
| ▲ | rcxdude a day ago | parent | prev [-] | | What do you do when you're providing tech support to a customer about an issue that is related to their data? Obviously there should be some process in place but a customer going 'hey could you look at this' is a perfectly valid reason to, you know, look at it. And it's not particularly unusual to discuss vague details of your customers anonymously. Even in the medical field, with very strict privacy rules, doctors will often publicly discuss encounters they have had with patients, and they only cross the line when they identify the patient. | | |
| ▲ | ryandrake a day ago | parent [-] | | That’s a good question, I honestly have no idea how customer support works. My guess is they have access to heavily audited systems that automate away these approvals, are limited to tools that can delete/reset but can’t read, and/or are given a filtered/redacted window to the user data they do actually need to see with human eyes. |
|
| |
| ▲ | m4rtink a day ago | parent | prev [-] | | Sure, which is why I think if one wants to run any agents with sensitive data, they need to be fully self hosted so that operators can't look into your data. Anything else is insane and will be used against you. | | |
| ▲ | weird-eye-issue a day ago | parent | next [-] | | > will be used against you I mean I'm all for privacy and security but this statement right here just comes across as pretty paranoid | | |
| ▲ | threecheese a day ago | parent [-] | | - A woman using Claude for her diary just had her information used against her.
- A nytimes civil suit against OpenAI just had user data handed over to lawyers.
- US federal agencies are buying ad marketplace data and joining it with other commercial and government data to extra-judicially identify persons We are giving an unprecedented amount of personal data to these tools at the same time that the technology is being leveraged against people at an unprecedented scope and scale. No US legal framework exists to protect you against this. You should take this risk seriously. |
| |
| ▲ | lmf4lol a day ago | parent | prev [-] | | Do you feel the same way about emails, cloud drives or SaaS like Salesforce or Jira? |
|
|
|
