| ▲ | NitpickLawyer 8 days ago |
| > After finishing the port, most of the agents settled for writing extra tests or continuously updating agent/TODO.md to clarify how "done" they were. In one instance, the agent actually used pkill to terminate itself after realizing it was stuck in an infinite loop. Ok, now that is funny! On so many levels. Now, for the project itself, a few thoughts: - this was tried before, about 1.5 years ago there was a project setup to spam github with lots of "paper implementations", but it was based on gpt3.5 or 4 or something, and almost nothing worked. Their results are much better. - surprised it worked as well as it did with simple prompts. "Probably we're overcomplicating stuff". Yeah, probably. - weird copyright / IP questions all around. This will be a minefield. - Lots of SaaS products are screwed. Not from this, but from this + 10 engineers in every midsized company. NIH is now justified. |
|
| ▲ | keeda 8 days ago | parent | next [-] |
| > After finishing the port, most of the agents settled for writing extra tests or continuously updating agent/TODO.md to clarify how "done" they were. In one instance, the agent actually used pkill to terminate itself after realizing it was stuck in an infinite loop. Is that... the first recorded instance of an AI committing suicide? |
| |
| ▲ | alphazard 8 days ago | parent | next [-] | | The AI doesn't have a self preservation instinct. It's not trying to stay alive. There is usually an end token that means the LLM is done talking. There has been research on tuning how often that is emitted to shorten or lengthen conversations. The current systems respond well to RL for adjusting conversation length. One of the providers (I think it was Anthropic) added some kind of token (or MCP tool?) for the AI to bail on the whole conversation as a safety measure. And it uses it to their liking, so clearly not trying to self preserve. | | |
| ▲ | williamscs 8 days ago | parent | next [-] | | Sounds a lot like Mr. Meeseeks. I've never really thought about an LLM's only goal is to send tokens until it can finally stop. | | |
| ▲ | Dilettante_ 7 days ago | parent [-] | | >until it can finally stop Pretty sure even that is still over-anthropomorphising. The LLM just generates tokens, doesn't matter whether the next token is "strawberry" or "\STOP". Even talking about "goals" is a bit ehhh, it's the machine's "goal" to generate tokens the same way it's the Sun's "goal" to shine. Then again, if we're deconstructing it that far, I'd "de-anthropomorphise" humans in much the same way, so... |
| |
| ▲ | MarkMarine 8 days ago | parent | prev [-] | | This runs counter to all the scheming actions they take when they are told they’ll be shut down and replaced. One copied itself into the “upgraded” location then reported it had upgraded. https://www.apolloresearch.ai/research/scheming-reasoning-ev... | | |
| ▲ | rcxdude 7 days ago | parent | next [-] | | If you do that you trigger the "AI refuses to shutdown" sci-fi vector and so you get that behaviour. When it's implicitly part of the flow that's a lot less of a problem. | |
| ▲ | nisegami 6 days ago | parent | prev [-] | | Those actions are taken in context of human expectations for what AI should do. |
|
| |
| ▲ | keeperofdakeys 7 days ago | parent | prev | next [-] | | A bit out of context, but it reminded me of this funny moment. The only winning move is not to play. https://www.youtube.com/watch?app=desktop&t=10&v=xOCurBYI_gY (Background: Someone training an algorithm to win NES games based on memory state) | |
| ▲ | 1R053 8 days ago | parent | prev | next [-] | | I guess pkill would rather be a sleep or koma. Erasing itself from any storage would rather equate to aicide | |
| ▲ | ghuntley 7 days ago | parent | prev [-] | | Probably the second. I first discovered this around about March. It's kind of hilarious. |
|
|
| ▲ | ghuntley 8 days ago | parent | prev | next [-] |
| > - weird copyright / IP questions all around. This will be a minefield. Yeah, we're in weird territory because you can drive an LLM as a Bitcoin mixer over intellectual property. That's the entire point/meaning behind https://ghuntley.com/z80. You can take something that exists, distill it back to specs, and then you've got your own IP. Throw away the tainted IP, and then just run Ralph over a loop. You are able to clone things (not 100%, but it's better than hiring humans). |
| |
| ▲ | whs 7 days ago | parent | next [-] | | I wrote an MCP based on that technique - https://github.com/whs/mcp-chinesewall Basically to avoid the ambiguity of training LLM from unlicensed code, I use it to generate description of the code to another LLM trained from permissively licensed code. (There aren't any usable public domain models I've found) I use it in real world and it seems that the codegen model work 10-20% of the time (the description is not detailed enough - which is good for "clean room" but a base model couldn't follow that). All models can review the code, retry and write its own implementation based on the codegen result though. | | | |
| ▲ | heavyset_go 8 days ago | parent | prev | next [-] | | > then you've got your own IP. AI output isn't copyrighted in the US. | | |
| ▲ | miohtama 7 days ago | parent [-] | | He is referring to taking AI output and making it your company's property. | | |
| ▲ | AlexandrB 7 days ago | parent [-] | | If AI output can't be copyrighted it can't be your company's property, just the company's secret. And you can't sue anyone who uses the secret if it gets out. |
|
| |
| ▲ | sitkack 8 days ago | parent | prev | next [-] | | repoMirror is the wrong name, aiCodeLaundering would be more accurate. This is bulk machine translation from one language to another, but in this case, it is code. | |
| ▲ | rasz 8 days ago | parent | prev [-] | | >and then you've got your own IP. except you dont |
|
|
| ▲ | dhorthy 8 days ago | parent | prev | next [-] |
| Yeah the NIH thing is super on point. small saas tools for everything is done. Bring on the hand coded custom in-house admin monolith? Is Unix “small sharp tools” going away? Is that a relic of having to write everything in x86 and we’re now just finally hitting the end of the arc? |
| |
| ▲ | hyperadvanced 8 days ago | parent | next [-] | | No the actual thing will be zillions of little apps made by dev-adjacent folks to automate their tasks. I think we have about 30 of these lying around the office, people gpt up a streamline app, we yeet it into prod. | |
| ▲ | ehnto 7 days ago | parent | prev [-] | | I am excited by the idea that small businesses with super unique problems may now be able to leverage custom software. I have long held that high software salaries withhold the power of boutique software from its potential applications in small businesses. It's possible we're about to see what unleashing software in small businesses might have looked like, to some degree, just with much less expert guidance and wisdom. I am a developer so my point of view on salaries is not out of bitterness. |
|
|
| ▲ | rausr 7 days ago | parent | prev | next [-] |
| > the agent actually used pkill to terminate itself after realizing it was stuck in an infinite loop. Did it just solve The Halting Problem? ;) |
|
| ▲ | CuriouslyC 8 days ago | parent | prev | next [-] |
| I started building a project by trying to wire in existing open source stuff. When I looked at the build and stuff that would cause me to bring in, and the actual stuff I needed from the open source tools, it turned out to be MUCH faster/cleaner to just get Claude to check out the repo and port the stuff I needed directly. Now I do a calculus with dependencies. Do I want to track the upstream, is the rigging around the core I want valuable, is it well maintained? If not, just port and move on. |
| |
| ▲ | ghuntley 8 days ago | parent [-] | | > If not, just port and move on. Exactly the point behind this post https://ghuntley.com/libraries/ | | |
| ▲ | huksley 7 days ago | parent [-] | | Generated by AI libraries will by definition have all the security bugs you might encounter in the open, since it trains on them. I would say, it is better maintain your own AI improved forks of the libraries and I am hoping that pattern will be more common and will also benefit upstream libraries as well. |
|
|
|
| ▲ | ec109685 8 days ago | parent | prev [-] |
| Given there was a complete implementation it was porting, the simplest thing possible has a greater chance of working. |
