| This statement stood out to me. > I have a similar thought about "there's no reason for software to be slow" BTW; in many cases, if you use AI to speed up some flow, you will add a bunch of messy code that will be overfit for the use case you tested on and will neither generalize, perf-wise, for all the use cases you should care about, nor will be maintainable upon future changes. LLMs have hardened hundreds of apps where I work. Maybe it's a language-specific thing. Most of the time, in Java, LLMs take into consideration tradeoffs (especially if they are part of the initial prompt) and actively explains them. It generally won't make messy workarounds, without a lot of extra prompting forcing it to do so. Ofc, if you have: - Functional tests - you'll be able to see it in the execution times of specific APIs - Stress testing - performance profiling is the most important output artifact |
| > Maybe it's a language-specific thing. Most of the time, in Java, LLMs take into consideration tradeoffs (especially if they are part of the initial prompt) and actively explains them. It generally won't make messy workarounds, without a lot of extra prompting forcing it to do so. Claude in its current Opus 5.5 form has made some wild reaches in response to reasonable-seeming prompts. For instance: "this error is getting swallowed, make sure the real termination message exposes the root cause" -> discovering that the issue is actually in a third party library that isn't exposing the original error except so through logs, so doing some hackery to inject a log-capturer that gets the stdout, buffers it, and looks for the message in order to re-throw based on that... And it can demonstrate that working, but it's added some fragile and/or unintended consequences like (a) string matching on a particular WARN log message of a particular version of the lib, and (b) buffering those logs such that there were now cases where they would never make it to the actual output stream before the program terminating. It's certainly capable of being used smarter than that, but it's also really easy to end up with messy fragile stuff if you get lazy about reviewing the output. |
| |
| ▲ | Supermancho a day ago | parent | next [-] | | > "this error is getting swallowed, make sure the real termination message exposes the root cause" Without judgement, I think this illustrates a fundamental objection the anti-AI crowd raises. The example prompt requires no understanding of what's happening, before suggesting action. Where I work, process requires that a PR review for a solution, has to have a corresponding explanation of why, or it's marked needs work | | |
| ▲ | majormajor a day ago | parent | next [-] | | I don't disagree. I think there's some nuance here though between "fully anti-AI" and "be careful with that power saw." That example I mentioned didn't make it to production. It got noticed and fixed more properly by the person who submitted the first prompt before it even went to PR. But a lot of times things in that same vein can be solved sufficiently with a "dumb" prompt like that, so a one-line prompt to get a PR drafted in the background for the dumb stuff can be a pretty good time saver. The art is not skimping on your attention to the bigger or more important stuff so much that all the time you save on first getting code written is lost by cleaning up stuff like that... | | |
| ▲ | ike2792 7 hours ago | parent [-] | | As a very amateur woodworker in addition to my day job, I have always thought the power tool analogy is a good one for LLMs. The tools don’t remove the need for skill and being very very careful with how you use them. You can work faster but do a lot more damage also. |
| |
| ▲ | rerdavies a day ago | parent | prev [-] | | I think you're misunderstanding where in the process that prompt got made. |
| |
| ▲ | Karrot_Kream a day ago | parent | prev | next [-] | | I used Opus 5.5 to figure out that a language model we were serving at work for internal use (we don't really serve models, we only use this internally because it processes sensitive data that we can't send to external endpoints) was being bottlenecked by CPU contention on tokenization, by having Opus run in an autoresearch style loop. Opus's first proposal for a fix was to spawn multiple Python processes and build a load balancer between them in Python. At first I asked it why it wasn't using haproxy and it said that haproxy would be an added dependency and that the overhead for this kind of thing in Python would be negligible. I knew this was idiotic but wanted to test this out, so had Opus run two experiments in parallel. Sure enough an haproxy load balancer was much, much better on HTTP load balancing than Python code was. This is the kind of thing that if you're unfamiliar with how networking code works, the models can autopilot into bad decisions that are verifiably correct (higher throughput than the original CPU-bottlenecked implementation) but much short of best practice. | | |
| ▲ | mike_hearn 17 hours ago | parent [-] | | Are you doing CPU inferencing? For small models you could check qxotic.ai which is a whole inferencing stack for many popular open models in Java, which can beat llama.cpp in speed in some cases. Its tokenizer toknroll is faster than tiktoken, which is what made me think of it for your use case. The advantage is that you don't have issues like the GIL or whatever, so there's no need for all the complexity of load balancing between multiple pythons. | | |
| ▲ | Karrot_Kream 7 hours ago | parent [-] | | Thanks for the pointers, I'm gonna look into them. I'm not doing CPU inferencing but I'm doing some tokenization in Python which is CPU-bound. In all honesty I started on this project to scale the model and did not build it myself so have been trying to read code and walk back the approach. I've been thinking about rewriting the Python glue in Go to avoid issues with the GIL. |
|
| |
| ▲ | Izkata 10 hours ago | parent | prev | next [-] | | > but it's added some fragile and/or unintended consequences In python, in a co-worker's code, it seems to significantly prefer "some_dict.get('key')" over "some_dict['key']". For the non-python-users, the first one will return None (equivalent to null) when the key is missing, while the second raises a KeyError. It was being used in a whole bunch of places that would cause problems (bad data instead of errors) if it did receive None. Some of these keys were actually optional and it does makes sense for those, but it looks like whichever model he was using just repeated the pattern everywhere instead of just where it was appropriate. | | |
| ▲ | saint-evan 9 hours ago | parent | next [-] | | I don't get it... What's the problem? I too prefer getting None as opposed to a key Error which would prolly necessitate a try..catch as opposed just checking for None. I'm having trouble reasoning up a situation where raising a keyError would make any sense over a None for data structures like dicts that could return a value or no value! Help? | | |
| ▲ | beecasthurlbow 9 hours ago | parent | next [-] | | KeyError localizes the issue to the lookup (ie “right here this key should be present and it’s not) vs later in the code where it can be hard to trace where the None value came from or how to handle it. | | |
| ▲ | saint-evan 8 hours ago | parent [-] | | Well... yeah but at the cost of halting your entire program? As opposed to logging? | | |
| ▲ | wfn 8 hours ago | parent [-] | | Think of it like an assert. Sometimes you do want some flow to terminate if a schema / data contract is breached, and not have this broken data contract propagate elsewhere / outwards. And/or, it is a perfectly valid pattern to indeed add try..catch, and contain (hence "localize") the issue there and then. If there is a way to handle this breach - you will need some kind of if check anyway. Catching by matching against the particular KeyError exception is in fact robust - and if you forgot to add an intended catch, then the exception will propagate upstack, yes - which is what you want - and this way you do not need to worry about forgetting an "if None" check. This sort of approach makes it structurally impossible to "forget" - if exception is not caught, it will abort the cannot-be-null breach continuing with program flow, possibly (e.g.) ending up persisted in some place it shouldn't be, causing problems later. | | |
|
| |
| ▲ | sirsinsalot 4 hours ago | parent | prev [-] | | Or you know, test if the key exists before using it. The key could exist and be None, representing a legitimate None at that key. Or the key might not exist and asking for it is a bug. Use a defaultdict if you must have all keys optional. Using get() everywhere is buggy because it masks illegal key use. |
| |
| ▲ | dirtbag__dad 4 hours ago | parent | prev | next [-] | | This to me feels more like a developer skills issue than a problem with the LLM. At my day job, we banned using untyped data structures like this entirely well before LLMs because no one could read code that passed them around. You could use a data structure that better articulates an optional field, like a dataclass with type hints or a pydantic model. Then you, your coworker, and the agent alike, aren’t guessing at whether the key’s value is optional. Another example in this vein is having many optional parameters on a function or api route, that drive behavior deeply nested in the application logic. Or maybe they were there for testing and never deleted. No one comes across this code next and understands it easily. How could you expect a LLM to understand it, either? I find it funny when people blame the LLM for dog shit code, but at the same time have a pollock painting of code they load into its context, and are like “put a person in the horse and carriage!” And the LLM is like “bro I just see dots and squiggles but you asked so boom!” | |
| ▲ | AgentOrange1234 7 hours ago | parent | prev [-] | | Oh man, you're giving me flashbacks to a vibe coded utter mess. Gets were a big offender. It seemed to have no discipline to commit to a data format, added all kinds of optional parameters everywhere, was also doing truly insane things like having contingency fallbacks in case some library it wanted to import wasn't where it expected, sometimes even importing things in the middle of a function. |
| |
| ▲ | mbrumlow 7 hours ago | parent | prev [-] | | It did what you asked. Had you first told it to figure out why the error was being lost,
You would have had a very different follow up prompt. |
|
