Remix.run Logo
dangelosaurus 5 hours ago

Hey HN, Michael here, co-founder of Promptfoo and one of the people working on the Codex Security CLI at OpenAI.

Thanks for checking this out and for flagging the auth issues. We just open-sourced it, and there's still plenty for us to improve. Expect the product to evolve quickly.

If you try it, I'd really appreciate hearing what works well and what you think we should improve. Happy to answer questions here.

CLI docs: https://learn.chatgpt.com/docs/security/cli

EDIT: If you'd like to help make this better, we're hiring: https://openai.com/careers/full-stack-software-engineer-cybe...

orangelimesoda 3 hours ago | parent | next [-]

> Have experience shipping production full-stack products across modern web frontends and backend services.

I'm amazed that the requirements are so low (or at least this vague) for jobs at companies like these.

Has anyone else had the experience of going to an interview and feeling like you were never asked any qualifying questions?

All the questions were easy, your answers were straightforward, you "got them right", but then were not chosen?

I find on the other side, they're also left with dozens of people who "passed" and then it comes down to a pretty arbitrary decision on who gets hired (if we are talking external, no referral, etc.)

I wonder if they can make job descriptions highly specific to filter the shortlist faster and more effectively (to actually get a shortlist).

Anyway end rant. Cool job, hope you fill it.

jameshart 3 hours ago | parent | next [-]

This sounds like it’s just an app development role, not a security analysis position, so I’m not sure what your complaint is.

solid_fuel an hour ago | parent [-]

Professional app development requires an understanding of security.

corndoge 31 minutes ago | parent [-]

I think the evidence contradicts you at this point

Groxx 38 minutes ago | parent | prev | next [-]

Companies doing interviews like this don't want skilled individuals. They want a common, easily-replaced baseline human (currently), so they can easily fire and rehire to reset salaries.

And also the job needs might change, and anyone decent should be able to learn it and keep going. So why pay for specific expertise you might not need.

pertymcpert 2 hours ago | parent | prev [-]

If you didn't already know, jobs at highly competitive companies tend to have vague job requirements because they expect to be able to apply your raw intelligence to changing demands quickly. There's no point being hyper-specific about the exact software packages because that's not what they want. What they want is someone who, after talking to an interviewer for 30 minutes, leaves them with the thought "Wow, this person can do anything we need of them. They can probably tell us what we need too and take ownership of large projects. Hire!"

moscoe 2 hours ago | parent | next [-]

Agree. Being smart, competent, and high in conscientiousness is more important than any highly specific “qualification”. It’s not about checking a bunch of boxes. If you have a track record of getting shit done, you’ll have something to contribute.

_superposition_ 14 minutes ago | parent | prev [-]

This. Being too specific on the stack requirements is a red flag imo.

vladoh 4 hours ago | parent | prev | next [-]

This looks great, thanks for open-sourcing it!

How does it deal with the current guardrails 5.6 Sol has on finding vulnerabilities? When I use it in the Codex app it would sometimes say it found a vulnerability, but it cannot tell me what it is.

dangelosaurus 2 hours ago | parent [-]

Thanks! You've run into a real limitation: the CLI doesn't bypass the model's cybersecurity guardrails. If GPT-5.6 Sol finds a vulnerability but refuses to explain it, switching from the Codex app to the CLI won't automatically fix that.

For authorized defensive work, Trusted Access for Cyber (TAC1/Daybreak) can reduce refusals depending on the model and the account or organization where access is provisioned. It isn't a blanket bypass.

If you're an open-source maintainer, you can apply for conditional Codex Security access here:

https://openai.com/form/codex-for-oss/

For enterprise teams, the public Daybreak onboarding guide is here:

https://help.openai.com/en/articles/20001261-enterprise-dayb...

If you have an example of "found a vulnerability but won't tell me what it is," I'd love to take a look too. You can send it to use with /feedback (or message me).

Quai 3 hours ago | parent | prev | next [-]

I tried it, it started a scan but stopped after hitting the rate-limit of my account. It gave up after just a minute of retrying (rate limits are tokens per minute, so... :P).

It said "Partial output was kept at <...>", but I dont see a obvious way of picking it up in a new scan? (The failed run cost me ~$13)

dangelosaurus 3 hours ago | parent [-]

Yeah, you're right. A per-minute rate limit shouldn't kill a scan after a minute, and "partial output was kept" makes it sound like you can pick up where you left off. You can't yet, unfortunately. --max-cost can limit estimated spend, but we still need proper retries and resume. Sorry you spent $13 finding that out. Please send me an email and I'll help make it right.

gizmodo59 4 hours ago | parent | prev | next [-]

When would I use this over the plugin in codex? Which I think can be invoked from cli as well

dangelosaurus 4 hours ago | parent [-]

The plugin, including when invoked through the Codex CLI, is great for scanning the repo you're currently working in. The standalone Security CLI/SDK uses the same scanner, but is built for running security across many repos over time: org-wide scans, historical results, deduplication, false-positive tracking, budget controls, and CI integration.

We've been talking to hundreds of engineering and security teams, and their feedback is shaping what we build.

Like Promptfoo, our goal is practical tooling that fits into the workflows teams already have.

strictnein 4 hours ago | parent | prev | next [-]

Does it require hitting OpenAI's APIs or can one also stand up a local OpenAI compatible LLM endpoint?

dangelosaurus 4 hours ago | parent [-]

We are actively working on officially supporting this. Because it's open source it is pretty easy to point a coding agent at it now and switch out the model.

ignoramous 4 hours ago | parent [-]

Exciting! Is there any open GitHub issue we can track?

dangelosaurus 2 hours ago | parent [-]

Not yet, unfortunately. We only just opened the repo, and there isn't a public issue specifically tracking local or OpenAI-compatible endpoint support.

The issue tracker is here: https://github.com/openai/codex-security/issues

If you open an issue with the endpoint or model you want to use, I'd be happy to follow up there.

robotswantdata 4 hours ago | parent | prev | next [-]

Been watching your progress for a while, glad OpenAI have looked after you and the team and you still get to ship!

dangelosaurus 2 hours ago | parent [-]

Thank you, that means a lot. Being able to keep building practical, open-source security tooling was important to us.

Really glad we got to ship this, and there's still a lot we want to improve in Codex Security and in Promptfoo!

6thbit 3 hours ago | parent | prev | next [-]

How does it fare against its own codebase?

troupo 4 hours ago | parent | prev | next [-]

> co-founder of Promptfoo and one of the people working on the Codex Security CLI at OpenAI.

> Thanks for checking this out and for flagging the auth issues.

Offtopic, but this right here is why I don't believe any marketing around "great amazing models that one-shot everything and programmers are no longer needed".

You just have to look at what these labs routinely produce, and their own products.

Edit to respond to @simonw whose comment I saw before he retracted it ;)

This comment is tied directly to consistent continuous claims by the LLM labs. Their own products disprove their own claims, and it would indeed be nice if fewer people believed them :)

4 hours ago | parent [-]
[deleted]
waterTanuki an hour ago | parent | prev | next [-]

Why does this need an entirely separate repo instead of being a feature in the existing Codex project?

3 hours ago | parent | prev [-]
[deleted]