This is effectively what macOS does. It presents any end user GUI programs (or Terminal.app if it's a CLI program) to elect to kill by way of popup with the amount of memory the application is consuming and whether or not it is responsive.

Not a bad system, but macOS has a fundamental memory leak for the past few versions which causes even simple apps like Preview.app, your favorite browser (doesn't matter which one), etc. to 'leak' and bring up the choose-your-own-OOM-kill-target dialog.