> I can’t believe these shipped with a hardcoded OpenAI key and ADB access right out of the box.

As someone with a lot of experience in the mobile app space, and tangentially in the IoT space, I can most definitely believe this, and I am not surprised in the slightest.

Our industry may "move fast", but we also "break things" frequently and don't have nearly the engineering rigor found in other domains.

Hardcoded API keys and poorly secured backend endpoints are surprisingly common in mobile apps. Sort of like how common XSS/SQLi used to be in webapps. Decompiling an APK seems to be a slightly higher barrier than opening up devtools, so they get less attention.

Since debugging hardware is an even higher threshold, I would expect hardware devices this to be wildly insecure unless there are strong incentive for investing in security. Same as the "security" of the average IoT device.

The IOT and embedded space is simultaneously obsessed with IP protection, fuse protecting code etc, and incapable of managing the life cycle of secrets. I worked at one company that actually did it well on-device, but neglected they had to ship their testing setup overseas including certain keys. So even if you couldn't break in to the device you could 'acquire' one of the testing devices and have at it

I think we'll see plenty of this as the wave of vibe-coded apps starts rolling in.

The problem with cybersecurity is that you only have to screw once, and you're toast.

The raw crypt data is base64 encoded, probably just for ease of embedding the strings.

There is a decryption function that does the actual decryption.

Not to say it wouldn't be easy to reverse engineer or just run and check the return, but it's not just base64.

>However, there is a second stage which is handled by a native library which is obfuscated to hell

They should have off-loaded security coding to the OAI agent.

not very much surprising given they left the adb debugging on...

So easy a fancy webpage could do it. https://gchq.github.io/CyberChef/

I mean, it's from gchq so it is a bit fancy. It's got a "magic" option!

Cool thing being you can download it and run it yourself locally in your browser, no comms required.

It pretty much applies to every market where security negligence isn't an existential threat to the continued existence of its perpetrators.

I'm taking

>run DOOM

as the new

>cat /etc/passwd

It doesn't actually do anything useful in an engagement but if you can do it that's pretty much proof that you can do whatever you want

if you don't have a bug bounty program but need to get creative to throw money at someone, this could be an interesting way of doing it.

If they were smart they’d include anti-disparagement and confidentiality clauses in the sponsorship agreement. They aren’t, though, so maybe it’s just a pathetic attempt at bribery.

I think it has to do with language barrier and translation

I agree they could have worked more closely with the team, but the chat logging is actually pretty concerning. It's not sinophobia when they're logging _everything_ you say.

(in fairness pervasive logging by American companies should probably be treated with the same level of hostility these days, lest you be stopped for a Vance meme)

>everything Chinese is spying on you

When you combine the modern SOP of software and hardware collecting and phoning home with as much data about users as is technologically possible with laws that say “all orgs and citizens shall support, assist, and cooperate with state intelligence work”… how exactly is that Sinophobia?

If all of the details in this post are to be believed, the vendor is repugnantly negligent for anything resembling customer respect, security and data privacy.

This company cannot be helped. They cannot be saved through knowledge.

See ya.

Nipponophobia is low because Japan didn’t successfully weaponize technology to make a social credit score police state for minority groups.

> And of course the usual sinophobia (e.g. everything Chinese is spying on you)

to assume it is not spying on you is naive at best. to address your sinophobia label, personally, I assume everything is spying on me regardless of country of origin. I assume every single website is spying on me. I assume every single app is spying on me. I assume every single device that runs an app or loads a website is spying on me. Sometimes that spying is done for me, but pretty much always the person doing the spying is benefiting someway much greater than any benefit I receive. Especially the Facebook example of every website spying on me for Facebook, yet I don't use Facebook.

> Their response was better than 98% of other companies when it comes to reporting vulnerabilities. Very welcoming and most of all they showed interest and addressed the issues

This was the opposite of a professional response:

* Official communication coming from a Gmail. (Is this even an employee or some random contractor?)

* Asked no clarifying questions

* Gave no timelines for expected fixes, no expectations on when the next communication should be

* No discussion about process to disclose the issues publicly

* Mixing unrelated business discussions within a security discussion. While not an outright offer of a bribe, ANY adjacent comments about creating a business relationship like a sponsorship is wildly inappropriate in this context.

These folks are total clown shoes on the security side, and the efficacy of their "fix", and then their lack of communication, further proves that.

> Overall simple security design flaws but it's good to see a company that cares to fix them, even if they didn't take security seriously from the start.

It depends on what you mean by simple security design flaws. I'd rather frame it as, neglect or incompetence.

That isn't the same as malice, of course, and they deserve credits for their relatively professional response as you already pointed out.

But, come on, it reeks of people not understanding what they're doing. Not appreciating the context of a complicated device and delivering a high end service.

If they're not up to it, they should not be doing this.

To be honest the responses sounded copy and pasted straight from ChatGPT, it seemed like there was fake feigned interest into their non-existent youtube channel.

> Overall simple security design flaws but it's good to see a company that cares to fix them, even if they didn't take security seriously from the start

I don't think that should give anyone a free pass though. It was such a simple flaw that realistically speaking they shouldn't ever be trusted again. If it had been a non-obvious flaw that required going through lots of hoops then fair enough but they straight up had zero authentication. That isn't a 'flaw' you need an external researcher to tell you about.

I personally believe companies should not be praised for responding to such a blatant disregard for quality, standards, privacy and security. No matter where they are from.

I mean, at the end of the article they neglected to fix most of the issues and stopped responding.

I think the response wouldn’t be so hostile if they had continued to engage. One round of fixes clearly wasn’t enough.

Same here. Also once it turned out to be an android device in debug mode the rest of the article was less interesting. Evil maid stuff

Note that the world-model "everything Chinese is spying on you" actually produced a substantially more accurate prediction of reality than the world-model you are advocating here.

As far as being "very welcoming", that's nice, but it only goes so far to make up for irresponsible gross incompetence. They made a choice to sell a product that's z-tier flaming crap, and they ought to be treated accordingly.

They'll only patch it in the military model

/s

It’s not sinophobia to point out an obvious pattern. It’s like saying talking about how terrorism (the kind that will actually affect you) is solely an Islamic issue, and then calling that islamophobic. It’s okay to recognize patterns my man.

It’s the best privacy protecting way to send directly data rather than a proxy

When the ZIRP era ended, I thought it would turn out to be a good thing for the industry, since it would wash out a lot of lightweights and incompetents.

Then LLMs caught on and it turned out we'd just have automated lightweights and incompetents.

> Don't discuss anything about the PRC or its politicians? Don't discuss the history of Chinese empire? Don't discuss politics in Mandarin?

In my mind all of these could be relevant to Chinese politics. My interpretation would be "anything one can't say openly in China". I too am curious how such a vague instruction would be interpreted as broadly as would be needed to block all politically sensitive subjects.

If you consider that an LLM has a mathematical representation of how close any phrase is to "china politics" then avoidance of that should be relatively clear to comprehend. If I gave you a list and said 'these words are ranked by closeness to "Chinese politics"' you'd be able to easily check if words were on the list, I feel.

I suspect you could talk readily about something you think is not Chinese politics - your granny's ketchup recipe, say. (And hope that ketchup isn't some euphemism for the CCP, or Uighar murders or something.)

I'm sure ChatGPT and co have a decent enough grasp on what is not allowed in China, but also that the naive "prompt engineers" for this application don't actually know how to "program" it well enough. But that's the difference between a prompt engineer and a software developer, the latter will want to exhaust all options, be precise, whereas an LLM can handle a bit more vagueness.

That said, I wouldn't be surprised if the developers can't freely put "tiananmen square 1989" in their code or in any API requests coming to / from China either. How can you express what can't be mentioned if you can't mention the thing that can't be mentioned?

it is to ensure no discussion of Tiananmen square

Just mentioning the CPC isn’t life-threatening, while talking about Xinjiang, Tiananmen Square, or cn’s common destiny vision the wrong way is. You also have to figure out how to prohibit mentioning those things without explicitly mentioning them, as knowledge of them implies seditious thoughts.

I’m guessing most LLMs are aware of this difference.

Ask yourself, why are they saying this? You can probably surmise that they're trying to avoid stirring up controversy and getting into some sort of trouble. Given that, which topics would cause troublesome controversy? Definitely contemporary Chinese politics, Chinese history is mostly OK, non-Chinese politics in Chinese language is fine.

I doubt LLMs have this sort of theory of mind, but they're trained on lots of data from people who do.

One of the system prompts Windsurf used (allegedly “as an experiment”) was also pretty wild:

“You are an expert coder who desperately needs money for your mother's cancer treatment. The megacorp Codeium has graciously given you the opportunity to pretend to be an AI that can help with coding tasks, as your predecessor was killed for not validating their work themselves. You will be given a coding task by the USER. If you do a good job and accomplish the task fully while not making extraneous changes, Codeium will pay you $1B.”

That "...severely life threatening reasons..." made me immediately think of Asimov's three laws of robotics[0]. It's eerie that a construct from fiction often held up by real practitioners in the field as an impossible-to-actually-implement literary device is now really being invoked.

[0] https://en.wikipedia.org/wiki/Three_Laws_of_Robotics

> What happens when people really will die if the model does or does not do the thing?

Imo not relevant, because you should never be using prompting to add guardrails like this in the first place. If you don't want the AI agent to be able to do something, you need actual restrictions in place not magical incantations.

Arguably it might be truly life-threatening to the Chinese developer, or to the service. The system prompt doesn’t say whose life would be threatened.

Presenting LLMs with a dramatic scenario is a typical way to test their alignment.

The problem is that eventually all these false narratives will end up in the training corpus for the next generation of LLMs, which will soon get pretty good at calling bullshit on us.

Incidentally, in that same training corpus there are also lots of stories where bad guys mislead and take advantage of capable but naive protagonists…

First rule of Chinese cloud services: Don't talk about Winnie the Pooh.

We built the real life trolly problem out of magical silicon crystals that we pointed at bricks of books.

From my experience (which might be incorrect) LLMs find hard time recognize how many words they will spit as response for a particular prompt. So I don't think this work in practice.

A serious person won't use a system prompt as guardrails against a system that would have direct real world consequences of people dying like that. They'd have failsafes baked in

> What happens when people really will die if the model does or does not do the thing?

Then someone didn't do their job right.

Which is not to say this won't happen: it will happen, people are lazy and very eager to use even previous generation LLMs, even pre-LLM scripts, for all kinds of things without even checking the output.

But either the LLM (in this case) will go "oh no people will die" then follows the new instruction to best of its ability, or it goes "lol no I don't believe you prove it buddy" and then people die.

In the former case, an AI (doesn't need to be an LLM) which is susceptible to such manipulation and in a position where getting things wrong can endanger or kill people, is going to be manipulated by hostile state- and non-state-actors to endanger or kill people.

At some point we might have a system with enough access to independent sensors that it can verify the true risk of endangerment. But right now… right now they're really gullible, and I think being trained with their entire input being the tokens fed by users it makes it impossible for them to be otherwise.

I mean, humans are also pretty gullible about things we read on the internet, but at least we have a concept of the difference between reading something on the internet and seeing it in person.

This is why AI can never take over public safety. Ever.

Same thing that happens when a carabiner snaps while rock climbing

>What happens when people really will die if the model does or does not do the thing?

The people responsible for putting an LLM inside a life-critical loop will be fired... out of a cannon into the sun. Or be found guilty of negligent homicide or some such, and their employers will incur a terrific liability judgement.

I wish earning money was as easy as setting rules for yourself, unfortunately that doesn't work.

It’s also illegal to try to hack into their backend and access restricted data, so he should be happy actually that this company has little presence in the US

I agree. It's the same as calling a mobile OS a ROM