> Home Assistant already knows which bins (trash cans, for the international readers) are due for collection. [...] What it can’t tell me is whether I’ve actually done anything about it.

My first gut reaction to this was: Have you tried using a vision-language model?

I used to have an elaborate Home Assistant setup with tons of zigbee sensors here and there telling me if front door is not closed properly, if the rice cooker is left on (via current monitoring), if the garage door is left open (with a magnetic contact sensor), if window is left open, if the soil is dry, etc. I had to keep changing batteries of all these things and manage flaky Zigbee connections in the past.

More recently, I've found the absolute most reliable, ~zero-maintainence solution to be an LLM. Home Assistant already has an LLM plugin, you can wire it up to a local LLM if you want, and you can just feed it an image and ask "Is this garage door open or closed" and I found that to be much better for my sanity than changing batteries every 6 months and dealing with flaky sensors sending me false alerts.

I even found it useful for "Did the cat vomit on the floor" (if so notify me and pause the cleaning robot schedule) and "Do any plants look like they need attention"

For TFA's use case, if you already have a security camera facing front, you can just feed it that image and ask "Are the garbage bins out"

Also, if you're worried about cameras sending home images to the cloud, there are non-cloud options. (I use Unifi cameras and Home Assistant gets a RTSP stream over LAN.)

Yah, this is how I've solved this in my Home Assistant setup. My doorbell camera can see where I keep my bins when they're not out. We have three types of bins- a trash bin, a recycle bin, and two yard waste bins. Recycling and yard waste are picked up on alternate weeks, and I often forget which bins need to go out.

I have an automation that will grab an image from that camera and send it to the Anthropic API with a prompt describing where the bins are, and the color of each bin. The prompt comes back with JSON indicating which bins it can see. If it can see e.g. the yard waste bin, and it's time to take the yard waste bin to the curb, it will send me a notification.

It also reminds me when it's time to return them to their "resting" position after pickup- unless I already have.

Home Assistant orchestrates this all. I also have it hooked up to a "Bindicator"- a white 3D printed bin with a color-changing LED inside, controlled by an ESP8266 using ESPHome. When it's time to move the bin it lights up the color of the bin that needs to be moved. If I haven't moved the bins back after a day it turns orange to remind me.

Claude configured the automations for me via MCP but it's not particularly complicated. It's pretty reliable, though sometimes just because of where the doorbell camera is it can't clearly see the bins. If it can't make a ruling on the bin location it just falls back to notifying me so I will sometimes get a reminder I don't need.

- I have to change batteries of zigbee devices maybe every three years or so

- how often do you have to change your cameras' batteries, or are you accepting ugly wiring for powering from mains?

- how many cameras do you have installed to cover your home?

- IP cameras are flaky, too

- constantly sending images of one's private space to a cloud LLM is Orwellian

- are you polling your LLM minutewise "Did the cat vomit on the floor"?

Kinda liking this idea :)

I’ve not played with vision models much. I did some experiments with OpenCV and Processing many years ago, but times have changed.

I think you just wrote me next weekend project ;)

My first-gut reaction was:

why are so few hacking their cognitive abilities (basic short-term memory)?

[dead]