Quick framing, since the post is long: I did robotic manipulation research at OpenAI from 2017–2020, and the tabletop setup back then cost roughly 10x this one and took a team to run. This project is me testing whether a single person can now do meaningful work on the same class of problems: starting with physical and software setup.

A few decisions I'm least settled on, and would love some pushback/feedback on:

- single arm vs. bimanual (I went single for cost/space, knowing it rules out things like folding cloth)

- not calibrating camera extrinsics/intrinsics for now

- RGB vs. RGB-D for from-scratch policies (ACT / Diffusion Policy)

And one I'm more confident about but expect disagreement on: not building on ROS 2 / LeRobot, and writing my own stack instead. Happy to get into the reasoning.

Ah nice I've tinkered a bit with robotics few years ago, e.g. https://twitter-archive.benetou.fr/utopiah/status/1760260544... and it's pretty empowering... but also showcases how amazing the human body is. Our dexterity, ability to sense, etc.

Reminds me of https://rodneybrooks.com/why-todays-humanoids-wont-learn-dex... which is basically a stark warning against the hype.

Very interesting article. I'm looking to get my first robot arm soon-ish, probably something in the SO-101 category. Can someone get reasonably far using recorded sessions compared to training in a simulated environment¹? Do you have any experience with third-party or DIY attachments for robot arms? I assume it's going to be more difficult for something like the Ufactory arm vs. the open models.

¹https://blog.comma.ai/mlsim/

Cool stuff. At my previous (startup / research) job I had to set up similar system (but with franka arm and multi view camera) alone because I was the only one with robotics background.

>I do not intend to calibrate the camera’s extrinsics or intrinsics for now.

Sensible choice, although I suggest it's good in the long run to do at early stage in your setup, especially if you intend to collect data for policy learning.

Debugging trained policies for visual manipulation task can be a headache and having as much context on variables that can change is a good practice.

My previous setup was in Japan, a earthquake prone place and I wasted some time after realizing the camera got misaligned due to earthquake. A simple solution is just to place an Aruco marker on the table that tracks the relative extrinsic position of camera, and add it as metadata to collected teleoperation dataset.

Love this, I’m playing around with the cheapo esp32+servos version of this, super fun.

Something I’m working on is a hardware CLI for agents to run experiments, with a “CICD” pipeline that validates everything and means I can delegate more of the experiments to the agents. I wonder if you have any thoughts on this?

The idea is to allow the coding agent to run the full loop of experiments and validations, with vision, audio, button pressing, speaking etc to interact in place of the human

How does Lerobot prevent « full control » and « understanding »? I thought this was an open source library.

I am not an official supporter of the library but am asking out of curiosity.

Hey this is cool! I am doing something similar myself with the SO101 arm robot from Robot Studio using a patchwork of my own code and LeRobot. Would love to collaborate with you if you are open to it. You can find me on Discord as `.avilay`. https://www.linkedin.com/posts/avilay_lerobot-huggingface-ro...

Would like to know your reasoning on not going with LeRobot.

As impressive as this setup may be, I'm still amazed at how slow this type of robot is, whether amateur or professional grade. I have no expertise in this field, but as an observer, the apparent progress in this area seems very limited. I guess my expectations are too high and my understanding of the problems to solve is too low.

Great article. I'll be following along. Would like to learn more about the robotics space.

- I've heard the advantage of ROS besides the architecture is the ecosystem (driver integrations, etc). Is that not an issue because the arm supports a Python SDK OOTB?

- Any issues you've been running into with this setup?

- How do you determine if a session recording is good enough for training? Is 50/100 samples really all you need?

ROS sucks, good move. Too complicated

This is really exciting. Incredible that you can do this for this budget at home. Unthinkable a couple years ago.