Remix.run Logo
▲ GIMP now has UI tests(gimp.org)
90 points by qtqtqt 4 hours ago | 81 comments
▲pnkzin 2 hours ago | parent | next [-]

The first try is most difficult, 13 tests is good start. Because before the first test, we have to make reasonable test setting.

▲sinoziqi 2 hours ago | parent | prev | next [-]

Having only 13 tests to start is actually a healthy sign for a GUI codebase this old. The hard part of desktop UI testing is almost never writing the 14th through 200th test -- it's building a deterministic harness that doesn't flake on window-manager focus races, font/DPI differences, or async redraw timing.

Once you have a small suite that reliably catches real regressions without false positives in CI, adding one regression test per fixed UI bug scales much better than trying to assert on every widget up front. Huge generated UI suites usually end up cementing accidental layout behavior and training contributors to ignore red builds.

▲basedpolymer an hour ago | parent | prev | next [-]

Pangram detects AI in the text at +40%. That's a bit disappointing coming from Gimp, which prohibits the use of AI.

It seems hypocritical and raises questions about whether the lead maintainers are following their own guidelines.

edit : typo

https://www.pangram.com/history/be1f7073-5b2c-4431-9c5f-dae7...

▲miksuko an hour ago | parent | next [-]

AI detectors are notoriously unreliable. As someone who's read too much slop in the past few years, this doesn't strike me as AI. There's typos like "coult" instead of "could", and clunky sentences like this:

> Furthermore such legacy tests directly hijacking core code come with their own problems, such as breaking all too often (and not because they detected a bug), all while not being able to detect whole categories of bugs.

▲alex43578 an hour ago | parent [-]

That’s not really a clunky sentence. While it might not be Pulitzer-worthy, it clearly conveys a set of interconnected issues. Arbitrarily splitting it into a few sentences wouldn’t make it better.

▲weird-eye-issue an hour ago | parent | prev | next [-]

"The article detects AI in the text at +40%"

Huh?

▲basedpolymer an hour ago | parent [-]

I don't know how, but my iOS keyboard has completely replaced “Pangram” with something else...

▲raincole an hour ago | parent | prev | next [-]

All the articles on HN that I suspected as AI-generated got ~100%. So a 40% means likely no AI in my dictionary.

▲basedpolymer an hour ago | parent [-]

I think the text was originally written entirely by AI (the structure is typical). Then it was partially rewritten.

Some clues that caught my attention on their own:

“Each test launches a real instance of GIMP and performs a small task. Then it reads the result from GIMP. If the result is incorrect, the test fails and tells us what went wrong.”

That’s classic Claude; just like what I read every day. His approach to humanizing everything

And Pangram IS conservative in its estimates of results. With a text this long, I have no doubt about its accuracy.

▲Nnnes 4 minutes ago | parent [-]

Where did you get that quote from? It definitely looks like an LLM rewrite of a human-written paragraph from the post

> Each test starts a real GIMP and does one small task. Then it reads the result from GIMP. If the result is not correct, the test fails and tells us what went wrong.

Your quote doesn't appear in your Pangram link or in the commit history for the blog https://gitlab.gnome.org/Infrastructure/gimp-web/-/commits/t...

▲cadamsdotcom an hour ago | parent | prev | next [-]

At the risk of being obvious, Pangram has heavy incentive to give you something to say about the Pangrammyness of text you plug into Pangram.

If it mostly said "nope! No AI here!" you'd walk away and wouldn't Pangram your comment with mentions of Pangram, which would be a missed opportunity for you to give free brandgram awareness to Pangram.

Pangram!

▲basedpolymer 42 minutes ago | parent [-]

Pangram is currently the most reliable tool for avoiding the flood of low-quality AI content on the internet. There are independent studies on its effectiveness.

I used it because I had strong suspicions about certain passages in this text. Otherwise, I would never have used my Pangram credits on this.

Anyone who regularly reads AI-generated text will quickly recognize the few AI quirks in the text.

Humans can do better. They DO better

▲cadamsdotcom 35 minutes ago | parent [-]

Thank you for your Pangram primer, I had no idea!

Pangram!

▲fragmede an hour ago | parent | prev [-]

Or maybe don't treat pangram as some sort of magic oracle on AI generated writing. Most recently they gave out a prize because their detector scored a poem from 2006 with the phase "the night we ate whale" repeated four thousand times, as 100% AI generated.

https://x.com/max_spero_/status/2122032840204226687

▲rafram 3 hours ago | parent | prev | next [-]

> We have 13 tests at this time.

Does GIMP have a very restrictive AI policy or something? I get not wanting to ship a gigantic pile of UI tests without thorough review, but basically any model released in the last six months would be able to write UI tests covering every single control in an afternoon.

▲deskglass 2 hours ago | parent | next [-]

Turns out you're right! No AI allowed https://gitlab.gnome.org/GNOME/gimp/-/blob/master/.gitlab/me...

▲einichi 2 hours ago | parent | next [-]

> for contributors without Developer role

▲lukeify an hour ago | parent [-]

From the blame PR:

> Of course, core developers don't need to check this monotomous checklist. This is only useful for the newcomers.

So this is saying core devs already know not to use AI, only new developers shouldn't.

It's quaint at this point. Embrace the future.

▲Xirdus an hour ago | parent [-]

The future seems to be blocking external contributors and running the equivalent AI agents yourself.

▲echelon an hour ago | parent | prev [-]

Meanwhile I'm watching a community of hundreds of agents reworking game engines and chat apps. And it's progressing fast.

You have to serve users, not hold code up as a sacred relic.

Our own DNA was vibe coded. It's a mess. Stochastic hill climbing can get pretty far, and it can reach surprising fitness.

▲fuussssip an hour ago | parent | next [-]

It's ok to have principles. Some folks have extremely valid reasons to hold a hard no stance on AI. It's not me, but I respect that position as principled, logical, and reasoned. Just a very different set of values from me.

▲lukeify an hour ago | parent | next [-]

At this point it's like the Amish. Sure I can respect their principles, but at the same time I find it bloody bizarre.

▲fuussssip 2 minutes ago | parent | next [-]

I liken it to my vegetarianism. I'm not morally against eating animals. I have no religious convictions about it either. I just look at factory farming and think "that seems like torturing animals at scale to me" and choose to not participate.

The people looking at AI and taking a "none for me thanks" stance are looking at externalities and saying they don't like them, and they dont like them enough to not participate.

▲Xirdus an hour ago | parent | prev [-]

But there's no denying Amish food is much healthier and higher quality than regular store bought products (with some exceptions I guess).

▲alex43578 37 minutes ago | parent [-]

But there’s a dozen other invisible tradeoffs. It’s not a simple binary. The Amish don’t have their own grapes year round, for instance. The Amish would need to dedicate a major percentage of their population to farming, given the drastically lower productivity.

Our food supply has perfectly healthy options. People choose to buy and eat junk, then blow up to 400lbs. You can gild the lily by arguing over organic vs Amish-grown in terms of micronutrients and traces of pesticides, but even an inorganic ear of corn is 10x healthier than what the average American eats.

▲HlessClaudesman an hour ago | parent | prev [-]

If you start with the assumption that AI is the devil, than rejecting the devil and all the fruit he offers is logical.

If you start with the assumption that AI is a tool, then finding uses for the tool is logical.

Seems to me that some people really love riding a purity spiral, and collectively they have chosen a new tool to fill the role of the impure thing to be cast out.

▲fuussssip 5 minutes ago | parent | next [-]

I think you aren't being thoughtful enough.

Someone can believe AI was trained on material in a way that violated the consent of authors and artists and musicians. Someone can see the value of the tool, but not want that value to be primarily captured by megacorps. One can believe that the tool is having a deleterious effect on students, workers, and artists of all kinds, as it's being deployed. One can't be concerned about the added environmental effects, the impact to world markets, or the shortages it causes to consumer hardware. One can be worried about the copyright implications (can AI produce copyrighted material?) and not want to risk losing their IP. One can believe that art is necessarily a human enterprise, and art produced mechanically is just images and sounds, but not art nor music.

None of these are "the tool is the devil". All of these are "the tool has externalities, and those externalities come with a cost I am not comfortable paying".

I'm a vegetarian. Not because I think eating meat is immoral, but because I believe factory farming causes undue suffering to animals and the people who raise and process those animals. So I make a choice for myself, in my own life, and don't moralize to others.

It's the same thing. Some folks are making choices for their own lives based on their values for those externalities. You value energy use differently than they do. You value art differently than they do. You value corporate control over communications differently than they do. Neither of you is right, but standing here and pretending an eminently justifiable difference of opinion is just purity signaling is boorish.

▲Xirdus an hour ago | parent | prev [-]

From what I've seen, it's less spiritualism and more "they're taking our jobs!" kind of fearmongering. Like all "taking our jobs" fears, whether it's logical or not depends purely on whether they're actually taking these jobs or not.

▲bigstrat2003 34 minutes ago | parent | prev | next [-]

> You have to serve users, not hold code up as a sacred relic.

They are serving users, by not allowing AI slop. Software developed by people and not an LLM is of much higher quality, and I'm glad they are taking a stand.

▲ngruhn an hour ago | parent | prev | next [-]

At least DNA has a reward signal to clean up tech dept. Everything is use-it-or-loose-it. Don't use your eyes because it's always dark? Bye bye. Claude would maintain that as a fallback for all eternity.

▲globular-toast an hour ago | parent | prev | next [-]

It wasn't vibe coded, it evolved over millions of years. And we ended up with our eating hole right next to our breathing hole because there's no way up back out of that "decision".

▲mepiethree an hour ago | parent | prev | next [-]

Does free open source software “have to” serve users? You clearly aren’t the user they’re trying to serve. That’s okay though; there’s something for everyone.

▲keybrd-intrrpt an hour ago | parent | prev [-]

> Our own DNA was vibe coded.

No, that was Reinforcement Learning

▲adastra22 an hour ago | parent [-]

Neither, if we really want to get technical. RL is directed whereas evolution is not.

▲samuria 2 hours ago | parent | prev | next [-]

It’s frustrating to create tests with models, not sure if it’s everyone’s experience, but every time I asked it to create tests, it makes the tests conform to the sometimes broken code. Rather than calling out what the logic/code is wrong, it will make the tests conform to whatever is already broken.

▲forgotmypw17 2 hours ago | parent | next [-]

I recommend iterating on your prompts.

Prod your agent first until it can explain back to you what's wrong with the tests.

Then ask your agent to write a better prompt for writing tests (and save it to an .md file)

Then reference that .md file and iterate on the tests again. It may take a couple of tries, but I've had good results with this strategy.

Ended up with a whole prompting framework.

▲doginasuit an hour ago | parent | next [-]

The problem I've had is even when it can explain back the issue, it still writes for itself awful instructions with extra fluff, strange points of focus, and borderline misinterpretations of what I said.

Maybe that's not a big deal, they can still be generally useful, but they seem important to commit with the project if they will be used going forward. I hate the idea of a massive amount of slop riding along with the code, and slop begets slop.

The idea I'm chasing now is writing the documents myself with the hope that enough examples and a document about writing documents will be enough to allow it to write clearly.

▲skydhash 2 hours ago | parent | prev [-]

Prodding an agent vs doing interesting things with my time? What should I choose?

To be fair, some people like to prod. But, for me, I'd rather take hours reading manuals than do trial and error prodding for 30 minutes (even if I knew it would be that short). Yes it's not optimal in terms of time, but it's better for my soul and well being.

▲HiPhish an hour ago | parent | next [-]

> Yes it's not optimal in terms of time, but it's better for my soul and well being.

It is also optimal in terms of time in the long run. The prodder will have to prod over and over again because he has learned nothing. You on the other hand will actually understand what is going on and will be able to either do it yourself or give precise instructions to the machine next time. Don't let yourself become a meat proxy.

▲0xcoffee 2 hours ago | parent | prev [-]

It's a good use of your time to ask the agent to reflect and update it's skills if you intervene to manually steer it. Why wouldn't you want it to learn from your guidance?

▲Waterluvian 2 hours ago | parent | prev | next [-]

It can’t magically know expectations. If the code is well documented or there’s decent associated documentation, I’ve found it has no issue writing tests.

▲hulitu 2 hours ago | parent | prev [-]

> Rather than calling out what the logic/code is wrong, it will make the tests conform to whatever is already broken.

It is the same like humans: if they say your code is wrong, you will hire someone else who doesn't find so many bugs. It doesn't look good.

▲locusofself 2 hours ago | parent | prev | next [-]

Not using AI to at least writes tests in this day and age is crazy work. I started a desktop application 2 months ago and it has almost 3,000 tests now, MANY of them UI tests.

▲HiPhish an hour ago | parent | next [-]

Using AI to write tests in this day and age is crazy work. The machine is great at writing tests, but the quality of the tests is awful. It will write redundant tests, it will write tests that assert too many thing in one, it will not write tests that need to be tested. It's the part where I have to clean up the most every time. Quantity is not quality, and you really don't want to skip on the quality of your quality assurance. I'm writing a React + Django full-stack application, so there should be more than enough training data out there.

▲timcambrant 2 hours ago | parent | prev | next [-]

AI is a bit too excellent at writing tests. The number of tests doesn't really convey any information on test coverage. At least I've found that agents that naively improve codebases will write tests for pretty much anything, including irrelevant behavior that has little value in ensuring stability, security or correct behavior. It would probably help to run a test review stage with a strong model, where agents attempt to correct and consolidate tests to guard behavior rather than ensuring that theoretical code paths are followed.

▲skydhash 2 hours ago | parent [-]

In my experience, designing tests cases to ensure that they're othogonal and verify your invariants (domain or technical) is very mentally taxing and not something you can truly delegate. The actual typing maybe, but testing is a lot of duplicate code and you can go far with just copy-pasting stuff around and building helpers.

▲aidenialislife 2 hours ago | parent [-]

Designing for testing helps as well. Pure functions for core logic, coding against interfaces etc.

▲ares623 2 hours ago | parent | prev [-]

Look, guys, you are literally allowed to fork and go ham on the code.

▲mepiethree an hour ago | parent [-]

Yeah if people can make a better and more popular Gimp in an afternoon, go do it.

Gimp has millions of downloads this month. They are doing something right.

▲nomilk 2 hours ago | parent | prev | next [-]

This raises questions of AI ethics; namely, is it conscionable to subject AI to GIMP’s UI?

▲hulitu 2 hours ago | parent | prev [-]

> any model released in the last six months would be able to write UI tests covering every single control in an afternoon.

Microsoft also seems to use "models" to "test" its programs. Look how shitty they are.

▲dllu 2 hours ago | parent | prev | next [-]

Funny how GIMP spawned the Gimp Toolkit, aka GTK, which is widely used in Linux, and yet GIMP's own UI seems to have long languished as a rather outdated design. Glad that they have tests, though, and hope they continue to improve it.

▲alexey-salmin 2 hours ago | parent [-]

Ah! I always thought it was Gnome Toolkit, didn't even bother to check. Thanks

▲yjftsjthsd-h an hour ago | parent | next [-]

Well it is, regardless of the name (I think "GTK" officially doesn't stand for anything now); to quote https://www.gtk.org/ -

> GTK is a free and open-source project maintained by GNOME and an active community of contributors.

and

> GTK and the GTK logo are trademarks of the GNOME Foundation.

▲veltas an hour ago | parent | prev [-]

It's GNOME Toolkit now, you're not wrong, but originally was GIMP.

▲joegibbs 2 hours ago | parent | prev | next [-]

They made a blog post to announce that they’d written thirteen unit tests? Sure, there’s value in handmade software and all that but GIMP is not really a particularly excellent piece of software - if anything was to benefit from unleashing the agents it would be GIMP.

▲raincole 2 hours ago | parent | next [-]

It'd be really interesting to look back the comparison between projects with strict non-AI policies (e.g. GIMP) and ArtCraft in 2027.

My personal prediction is that ArtCraft will blow all the old non-AI projects in similar fields out of the water and GIMP will be remembered as a thing of the past. But if I got proven wrong I'd be happy too: at least we will know the value of non-AI policies.

▲flexagoon 14 minutes ago | parent [-]

GIMP is far from the best FOSS image editor for many use cases. IMO for many simple editing tasks that people usually need unless they're professional photo editors, even Inkscape, a primarily vector editor, is a better photo editor than GIMP.

▲raincole 11 minutes ago | parent [-]

Inkscape is what made me believe ArtCraft would be the clear winner, btw. It's very hard to imagine using Inkscape ever again since I tried VectorCraft. (And I'm fully aware that some Path Effects can only be done quickly in Inkscape.)

▲qtqtqt 2 hours ago | parent | prev | next [-]

They must have set up a reusable harness, and it's the most important part.

▲Jolter 2 hours ago | parent | prev | next [-]

I’d go so far as to say that in general, adding tests for legacy software before a refactor or rewrite is the perfect time to put the agent to work.

▲plant-ian an hour ago | parent | prev | next [-]

It sounds like they are nearly end to end user interface (UI) tests, not just unit tests.

▲WD-42 2 hours ago | parent | prev | next [-]

Did you read the article? These aren’t unit tests. They are UI tests which are pretty hard to do well with desktop apps.

▲qtqtqt 2 hours ago | parent [-]

Especially given it isn't as easy as using Cypress.

▲slopinthebag 2 hours ago | parent | prev | next [-]

we forget, but before AI this would have been at least 6 months of work, at least 50 meetings, and at least 7 separate sprints.

▲altmanaltman 2 hours ago | parent | prev | next [-]

The blog article doesn't say they have written thirteen unit tests.

> *GIMP already has unit tests*. A unit test checks one small part of the code, for example how GIMP reads an XCF file. But our unit tests were not opening a real window. So it coult not tell us if a dialog opens, or if a menu item did its job.

> Note that GIMP *already had tests of GUI code*, usually interacting directly with the core code, and sometimes even simulating input devices’ events (pointer movements, clicks, drags…). But nothing can beat interacting with the real interface. Furthermore such legacy tests directly hijacking core code come with their own problems, such as breaking all too often (and not because they detected a bug), all while not being able to detect whole categories of bugs.

> UI tests check the program as a whole, in the same way as you use it.

▲hulitu 2 hours ago | parent | prev [-]

> if anything was to benefit from unleashing the agents

AI can write tests now ? For its own software ? Can it evaluate the results ? Correctly ? /s

▲aidenialislife 2 hours ago | parent [-]

Writing tests never was the hard part of the job anyway.

▲ModernTimes 2 hours ago | parent | prev | next [-]

Timely, as I removed Gimp from my Macbook today after successfully testing PhotoCraft [1] for a week (the AI Photoshop clone written in open-source rust which has had an absured number of updates since you last tried it).

It makes me so HAPPY to remove the dinosaur Gimp, and to run "Adobe" software for free (it's good enough for me, maybe good enough for you). I expect to lose Internet points for saying that.

[1] https://getartcraft.com/apps/photocraft

▲plasticbugs an hour ago | parent | next [-]

I have a long history with Gimp. I created Gimpshop... the Gimp maintainers did not appreciate that little stunt, I'll tell ya! :D

I have canceled my Adobe subscription as of last month and have fully moved over to Photocraft. Even if development of Photocraft stopped today, I would be able to do whatever I need to get done in Photoshop without paying Adobe or wrestling with Gimp. It's good enough for me as well. I had been using Adobe Photoshop for more than 30 years now and was bought in enough to spend a significant amount of effort making Gimp act like Photoshop. Bye Photoshop, bye Gimp, bye Gimpshop.

▲hackshack an hour ago | parent [-]

Same with VectorCraft. I spent over a decade looking for an Illustrator alternative. Muscle memory and specific workflows won out. VectorCraft is extremely close.

▲Aozora7 an hour ago | parent | prev [-]

Regular users would use a lot more open source software if more of it was like photocraft and less of it was like GIMP.

▲fishgoesblub an hour ago | parent | prev | next [-]

Great on GIMP for continuing on and attempting to improve things, you'd think the GIMP team killed people's dogs with some of these comments..

▲bigstrat2003 31 minutes ago | parent [-]

AI bros can't stand the thought that not everyone agrees with them about LLMs. It's like a religion to them, they feel compelled to get everyone else on board. Never mind that if they are right, it will become apparent to everyone without them having to preach.

▲Aozora7 3 minutes ago | parent | next [-]

It has to be a religion, it can't just be that I prefer the quality of AI-assisted software over projects with no-AI policies.

Besides, GIMP has always been unpopular long before AI.

▲Aozora7 5 minutes ago | parent | prev [-]

It has to be a religion, it can't just be that I prefer the quality of AI-assisted software over projects with no-AI policies.

▲iLoveOncall 8 minutes ago | parent | prev | next [-]

The tests should simply take a screenshot and send it to an LLM with the question "Is the UI absolutely horrible?". If the answer is yes then the UI is working as expected.

▲fuussssip an hour ago | parent | prev | next [-]

GIMP... didn't have ui tests before?

I'm surprised, tbh. I understand that UI is very much an afterthought in GIMP, but I figured it'd still at least have tests.

▲YuechenLi 2 hours ago | parent | prev | next [-]

Oh, haven't heard from GIMP in a long time, has there been significant UX improvement and have they carved out a niche like Krita yet?

▲aquafox an hour ago | parent | prev | next [-]

GIMP doesn't need UI tests, but a more intuitive UI.

▲snissn an hour ago | parent | prev [-]

"GIMP now has UI" would be big news