Remix.run Logo
Show HN: LemonSlice – Upgrade your voice agents to real-time video
35 points by lcolucci 3 hours ago | 50 comments

Hey HN, we're the co-founders of LemonSlice (try our HN playground here: https://lemonslice.com/hn). We train interactive avatar video models. Our API lets you upload a photo and immediately jump into a FaceTime-style call with that character. Here's a demo: https://www.loom.com/share/941577113141418e80d2834c83a5a0a9

Chatbots are everywhere and voice AI has taken off, but we believe video avatars will be the most common form factor for conversational AI. Most people would rather watch something than read it. The problem is that generating video in real-time is hard, and overcoming the uncanny valley is even harder.

We haven’t broken the uncanny valley yet. Nobody has. But we’re getting close and our photorealistic avatars are currently best-in-class (judge for yourself: https://lemonslice.com/try/taylor). Plus, we're the only avatar model that can do animals and heavily stylized cartoons. Try it: https://lemonslice.com/try/alien. Warning! Talking to this little guy may improve your mood.

Today we're releasing our new model* - Lemon Slice 2, a 20B-parameter diffusion transformer that generates infinite-length video at 20fps on a single GPU - and opening up our API.

How did we get a video diffusion model to run in real-time? There was no single trick, just a lot of them stacked together. The first big change was making our model causal. Standard video diffusion models are bidirectional (they look at frames both before and after the current one), which means you can't stream.

From there it was about fitting everything on one GPU. We switched from full to sliding window attention, which killed our memory bottleneck. We distilled from 40 denoising steps down to just a few - quality degraded less than we feared, especially after using GAN-based distillation (though tuning that adversarial loss to avoid mode collapse was its own adventure).

And the rest was inference work: modifying RoPE from complex to real (this one was cool!), precision tuning, fusing kernels, a special rolling KV cache, lots of other caching, and more. We kept shaving off milliseconds wherever we could and eventually got to real-time.

We set up a guest playground for HN so you can create and talk to characters without logging in: https://lemonslice.com/hn. For those who want to build with our API (we have a new LiveKit integration that we’re pumped about!), grab a coupon code in the HN playground for your first Pro month free ($100 value). See the docs: https://lemonslice.com/docs. Pricing is usage-based at $0.12-0.20/min for video generation.

Looking forward to your feedback!

EDIT: Tell us what characters you want to see in the comments and we can make them for you to talk to (e.g. Max Headroom)

*We did a Show HN last year for our V1 model: https://news.ycombinator.com/item?id=43785044. It was technically impressive but so bad compared to what we have today.

pickleballcourt 6 minutes ago | parent | next [-]

One thing I've learnt from movie production is actually what separates professional from amateur quality is in the audio itself. Have you thought about implementing personaplex from NVDIA or other voice models that can both talk and listen at the same time?

Currently the conversation still feels too STT-LLM-TTS that I think a lot of the voice agents suffer from (Seems like only Sesame and NVDIA so far have nailed the natural conversation flow). Still, crazy good work train your own diffusion models, I remember taking a look at the latest literature on diffusion and was mind blown by the advances in last years or so since u-net architecture days.

bennyp101 2 hours ago | parent | prev | next [-]

Heads up, your privacy policy[0] does not work in dark mode - I was going to comment saying it made no sense, then I highlighted the page and more text appeared :)

[0] https://lemonslice.com/privacy

sid-the-kid an hour ago | parent | next [-]

Fix deployed! This is why it's good to launch on hacker news. thanks for the tip.

bennyp101 an hour ago | parent [-]

Nice one - thanks :)

sid-the-kid 2 hours ago | parent | prev [-]

Good catch! Working on a fix now.

convivialdingo an hour ago | parent | prev | next [-]

That's super impressive! Definitely one of the best quality conversational agents I've tried syncing A/V and response times.

The text processing is running Qwen / Alibaba?

lcolucci an hour ago | parent | next [-]

Qwen is the default but you can pick any LLM in the web app (though not the HN playground)

sid-the-kid an hour ago | parent | prev [-]

Thank you! Yes, right now we are using Qwen for the LLM. They also released a super fast TTS model that we have not tried yet, which is supposed to be very fast.

wumms an hour ago | parent | prev | next [-]

You could add a Max Headroom to the hn link. You might reach real time by interspersing freeze frames, duplicates, or static.

sid-the-kid an hour ago | parent | next [-]

And, just like that, Max Headroom is back: https://lemonslice.com/try/agent_ccb102bdfc1fcb30

sid-the-kid an hour ago | parent | prev [-]

1) yes on Max Headroom. we are on it. 2) it already is real time...?

wumms an hour ago | parent [-]

Whoops! Mistook the "You're about to speak with an AI."-progress bar for processing delay.

sid-the-kid 32 minutes ago | parent [-]

Makes sense. The init should be about 10s. But, after that, it should be real time. TBH, this is probably a common confusion. So thanks for calling it out.

r0fl 2 hours ago | parent | prev | next [-]

Where’s the hn playground to grab a free month?

I have so many websites that would do well with this!

dang 41 minutes ago | parent | next [-]

(We've replaced the link to their homepage (https://lemonslice.com/) with the HN playground at the start of the text above)

lcolucci 14 minutes ago | parent [-]

Thanks Dan! The HN playground let's anyone try out for free without login

lcolucci 2 hours ago | parent | prev [-]

https://lemonslice.com/hn - There's a button for "Get 1st month free" in the Developer Quickstart

r0fl 2 hours ago | parent | prev | next [-]

Pricing is confusing

Video Agents Unlimited agents Up to 3 concurrent calls Creative Studio 1min long videos Up to 3 concurrent generations

Does that mean I can have a total of 1 minute of video calls? Or video calls can only be 1 minute long? Or does it mean I can have unlimited calls, 3 calls at a time all month long?

Can I have different avatars or only the same avatar x 3?

Can I record the avatar and make videos and post on social media?

lcolucci 2 hours ago | parent [-]

Sorry about the confusion. Video Agents and Creative Studio are two entirely different products. Video Agents = interactive video. Creative Studio = make a video and download it. If you're interested in real-time video calls, then Video Agents is the only pricing and feature set you should look at.

skandan 2 hours ago | parent | prev | next [-]

Wow this team is non-stop!!! Wild that this small crew is dropping hit after hit. Is there an open polymarket on who acquires them?

lcolucci an hour ago | parent [-]

haha thank you so much! The team is incredible - small but mighty

r0fl 2 hours ago | parent | prev | next [-]

Wow I can’t get enough of this site! This is literally all I’ve been playing with for like half an hour. Even moved a meeting!

My mind is blown! It feels like the first time I used my microphone to chat with ai

lcolucci an hour ago | parent | next [-]

This comment made my day! So happy you're liking it

sid-the-kid an hour ago | parent | prev [-]

glad we found somebody who likes it as much as us! BTW, biggest thing we are working to improve is speed of the response. I think we can make that much faster.

zvonimirs 3 hours ago | parent | prev | next [-]

We're launching a new AI assistant and I wanted to make it alive so I started to play around with LemonSlice and I loved it!! I wanted to make our assistant be like a coworker that can give it an ability to create Loom style videos. Here's what I created - https://drive.google.com/file/d/1nIpEvNkuXA0jeZVjHC8OjuJlT-3...

Anyway, big thumbs up for the LemonSlice team, I'm excited to see it progress. I can definitely see products start coming alive with tools like this.

sid-the-kid 2 hours ago | parent [-]

Very cool! Thanks for sharing. I love your use-case of turning an AI coding agent into more of an AI employee. Will be interesting to see if users can connect better with the product this way.

dreamdeadline 2 hours ago | parent | prev | next [-]

Cool! Do you plan to expose controls over the avatar’s movement, facial expressions, or emotional reactions so users can fine-tune interactions?

lcolucci 2 hours ago | parent | next [-]

Yes we do! Within the web app, there's a "action text prompt" section that allows you to control the overall actions of the character (e.g. "a fox talking with lots of arm motions"). We'll soon expose this in the API so you can control the characters movements dynamically (e.g. "now wave your hand")

sid-the-kid 2 hours ago | parent | prev [-]

Our text control is good, especially for emotions. For example, you can add the text prompt: "a person talking. they are angry", and agent will have an angry expression.

You can also control background motions (like ocean waves, or a waterfall or car driving).

We are actively training a model that has better text control over hand motions.

koakuma-chan 2 hours ago | parent | prev | next [-]

> You're probably thinking, how is this useful

I was thinking why the quality is so poor.

sid-the-kid an hour ago | parent [-]

curious what avatar you think is poor quality? Or, what you think is poor quality. i want to know :)

koakuma-chan an hour ago | parent [-]

Low res and low fps. Not sure if lipsync is poor, or if low fps makes it look poor. Voice sounds low quality, as if recorded on a bad mic, and doesn't feel like it matches the avatar.

sid-the-kid an hour ago | parent [-]

thanks for the feedback. that's helpful. Ya, some avatars have worse lip synch than others. It depends a little on how zoomed in you are.

I am double checking now to make 100% sure we return the original audio (and not the encoded/decoded audio).

We are working on high-res.

r0fl 2 hours ago | parent | prev | next [-]

Wow this is the most impressive thing I’ve seen on hacker news in years!!!!!

Take my money!!!!!!

lcolucci 2 hours ago | parent [-]

Wow thank you so much :) We're so proud of it!!

sid-the-kid 3 hours ago | parent | prev | next [-]

hey HN! one of the founders here. as of today, we are seeing informational avatars + roleplaying for training as the most common use cases. The roleplaying use-case was surprising to us. Think a nurse training to triage with AI patients. Or, SDRs practicing lead qualification with different kinds of clients.

buddycorp 2 hours ago | parent | prev | next [-]

I'm curious if I can plug in my own OpenAI realtime voice agents into this.

lcolucci 2 hours ago | parent | next [-]

Good question! Yes and to do this you'd want to use our "Self-Managed Pipeline": https://lemonslice.com/docs/self-managed/overview. You can combine any TTS, LLM and STT combination with LemonSlice as the avatar layer.

jfaat 2 hours ago | parent | prev | next [-]

I'm using an openAI realtime voice with livekit, and they said they have a livekit integration so it would probably be doable that way. I haven't used video in livekit though and I don't know how the plugins are setup for it

lcolucci 2 hours ago | parent [-]

Yes this is exactly right. Using the LiveKit integration you can add LemonSlice as an avatar layer on top of any voice provider

sid-the-kid 2 hours ago | parent | prev [-]

Good question. When using the API, you can bring any voice agent (or LLM). Our API takes in what the agent will say, and then streams back the video of the agent saying it.

For the fully hosted version, we are currently partnered with ElevenLabs.

shj2105 an hour ago | parent | prev | next [-]

Not working on mobile iOS

lcolucci an hour ago | parent [-]

what's not working for you?

benswerd 2 hours ago | parent | prev | next [-]

The last year vs this year is crazy

lcolucci an hour ago | parent | next [-]

Agreed. We were so excited about the results last year and they are SO BAD now by comparison. Hopefully we'll say the same thing again in the couple months

sid-the-kid 2 hours ago | parent | prev [-]

thanks! it just barley worked last year, but not much else. this year it's actually good. we got lucky: it's both new tech and turned out to be good quality.

ed_mercer 2 hours ago | parent | prev | next [-]

This looks super awesome!

sid-the-kid 2 hours ago | parent [-]

thank you! it's by far the thing I have worked on that I am most proud of.

marieschneegans 2 hours ago | parent | prev [-]

This is next-level!

lcolucci 2 hours ago | parent [-]

Thanks so much! We're super proud of it