Timely! I spent a good chunk of yesterday investigating Bombadil. I'm a huge fan of PBT and a huge fan of temporal logic (though not linear temporal logic) and we're currently trying to level up our QA, so it seemed like a good match.

Unfortunately, I concluded that Bombadil is a toy. Not as in "Does some very nice things, but missing the features for enterprise adoption." I mean that in a very strong sense, as in: I could not figure out how to do anything real with it.

The place where this is most obvious is in its action generator type. You give it a function that generates actions, where each action is drawn from a small menu of choices like "scroll up this many pixel" or "click on this pixel." You cannot add new types of actions. If you want to click on a button, you need it to generate a sequence of actions to first scroll down enough, and then look up a pixel that's on that button.

Except that it selects actions randomly from your list, so you somehow need the action generator, when run the first time, to generate only the scroll action, and then have it, when run the second time, generate only the click action. If you are silly enough as to have an action generator that, you know, actually gives a list of reasonable actions, you'll get a tester that spends all its time going in circles clicking on the nav bar.

(Something in the docs claimed that actions are weighted, but I found nothing about an actual affordance for doing that. Having weights would make this go from basically impossible to somewhat impossible.)

(Edit: Found the weighted thing.)

I am terrified to imagine how to get Bombadil to fill out a form. At least that seems possible -- you can inspect the state of the web page to figure out what's already been filled out. But if you want the state to be based on anything not on the current page, like the action that you took on the previous page, or gasp the state on disk (as for an Electron app), that seems completely impossible. Action generators are based on the current state, and the state must be a pure function of the web page.

Its temporal logic has a cool time-bounded construct, but it's missing the U (until) operator. One of their few examples is "If you show the user a notification, it disappears within 5 seconds." But I want to say "When you click the Generate button, it says 'Generating...' up until it's finished generating." And I can't.

(Note: everything above is according to the docs. Hopefully everything I said is a limitation of the docs, not an actual limitation of the framework.)

I shared my comments with the author yesterday on LinkedIn, but he hasn't responded yet. Maybe I'll hear from him here.

I have a pretty positive opinion of Antithesis as a company and they seem to be investing seriously in it, and generally see it as a strong positive sign when someone knows what temporal logic is, so I have hopes for this framework. I am nonetheless disappointed that I can't use it now, especially because I was supposed to finish an internal GUI testing tool this week and my god I'm behind.

Hey, yeah, so actions are indeed independent and you can't express fixed sequences of them (e.g. fill out thus form by doing X, then Y, then Z). If you want to enforce certain sequentiality you'd need to precondition your generators. This is admittedly limited right now. I do want to add QoL things like `filter` on the generators to make that more ergonomic.

Also, as you note, you can't implement custom actions. And that's just something that I haven't gotten to yet. It'd be quite straightforward to add a plain function (toStringed) as one of the variants of the Action type and send that to the browser for eval. (Btw, we're taking contributions!)

Weighting is also important right now. There's no smart exploration in Bombadil yet, only blind random, so manual weighting becomes pretty crucial to have more effective tests (i.e. unlikely to run in circles). I'd like to both make the Bombadil "fuzzer" better at this, but eventually you might want to run Bombadil inside Antithesis instead to get a much better exploration, even in a single campaign.

The Until operator is also one of those things that I haven't gotten to yet. I actually didn't expect someone to hit this as a major limitation of the tool so quickly, which is why I've focused on other things. Surprising!

To add some more context, Bombadil is an OSS project with one developer (but we're hiring!) and it's 4 months old and marked experimental. I'm sorry you were disappointed by its current state, but it's very early days, so expect a lot of these things to improve. And your feedback will be taken into account. Thanks!