Remix.run Logo
Springtime 17 hours ago

In an earlier video they made a couple years back about Disney's sodium vapor technique Paul Debevec suggested he was considering creating a dataset using a similar premise: filming enough perfectly masked references to be able to train models to achieve better keying. So it was interesting seeing Corridor tackle this by instead using synthetic data.

somat 17 hours ago | parent [-]

With regards to the sodium vapor process, an idea has been percolating in the back of my head ever since I saw that video. But I don't really have the budget to try it out.

theory: make the mask out of non-visable light

illuminate the backing screen in near Infra-Red light. (after a bit of thought I chose near-IR as opposed to near-UV for hopefully obvious reasons)

point two cameras at a splitting prism with a near IR pass filter(I have confirmed that such thing exists and is commercially available)

Leave the 90 degree(unaltered path) camera untouched, this is the visible camera.

Remove the IR filter from the 180 degree(filter path) camera, this is the mask camera.

Now you get a perfect non-color shifting mask(in theory), The splitting prism would hurt light intake. It might be worth it to try putting the cameras really close together , pointed same direction, no prism, and see if that is close enough.

overvale 17 hours ago | parent | next [-]

Debevec tried a version of this: https://arxiv.org/abs/2306.13702

dgently7 8 hours ago | parent [-]

im familiar with this work and specifically they tried replicating the sodium vapor style approach but what worked for poppins level isnt actually good enough for today. Specifically you still end up with light spill that contaminates the foreground, especially for things like the fresnel reflections on the side of a face. the magenta idea was to still do what is basically a color difference key, but increase the color separation between fg and bg by lighting the two with different opposite colored lights. then using a ml model to recover the original fg color.

wiml 16 hours ago | parent | prev | next [-]

This approach was used in the 1950s/60s with ultraviolet light (rather than IR) to create a traveling matte. I'm not sure why visible-light techniques won out. Easier to make sure that the illumination is set up correctly, maybe?

regularfry 6 hours ago | parent [-]

Or maybe they didn't want to blind the talent. UV isn't something you want to bathe your retinas in.

randyrand an hour ago | parent | prev | next [-]

Couldn't this be summarized as the Sodium Vapor technique but with near-IR? Or do I misunderstand something?

diacritical 17 hours ago | parent | prev | next [-]

Don't humans and other warm objects also radiate IR?

somat 17 hours ago | parent [-]

That is far-IR, thermal stuff, Near-IR, 700 nanometer-ish is right below red in human vision.

Camera sensors can pick up a little near-IR so they have have a filter to block it. If that filter was removed and a filter to block visable light was used in place you would have a camera that can only see non-visable light. Poorly, the camera was not engineered to operate in this bandwidth, but it might be good enough for a mask. A mask that does not interfere with any visible colors.

fc417fc802 15 hours ago | parent | next [-]

> Poorly, the camera was not engineered to operate in this bandwidth

At least for cheap sensors in phones and security cameras that engineering consists of installing an IR filter. They pick it up just fine but we often don't want them to.

Keep in mind that sensors are inherently monochrome. They use multiple input pixels per output pixel with various filters in order to determine information about color.

throwway120385 16 hours ago | parent | prev [-]

You can actually dimly perceive near-IR LEDs -- they'll glow slightly red in darkness.

adrian_b 3 hours ago | parent | next [-]

That depends on how "near" they are.

The sensitivity to red light decreases quickly at wavelengths greater than 650 nm, but light can still be perceived if it is strong enough, up to around 780 nm.

Many so-called near-IR LEDs may actually be somewhere around 750 nm, so they are still visible on a dark background, even if they are perceived as extremely dim.

On the other hand, there are many near infrared LEDs around 900 nm and those are really invisible. Near-infrared LEDs around 1300 nm or around 1550 nm are also completely invisible.

An invisible near-infrared laser beam could become visible due to double-photon absorption, but if a beam of such intensity as to cause double-photon absorption hits your retina, there are more serious things to worry about.

diacritical 15 hours ago | parent | prev [-]

I remember reading some people can perceive some near IR, but mostly that near-IR LEDs actually leak some red themselves due to imperfections in manufacturing or something?

actionfromafar 17 hours ago | parent | prev [-]

I'll do you one better, which requires no special cameras (most have IR filters) nor double cameras or prisms.

Shoot the scene in 48 or 96 fps. Sync the set lighting to odd frames. Every odd frame, the set lights are on. Every even frame, set lights are off.

For the backing screen, do the reverse. Even frames, the backing screen is on. Odd frames, backing screen is off.

There you go. Mask / normal shot / Mask normal shot / Mask ... you get the idea.

Of course, motion will cause normal image and mask go out of sync, but I bet that can be remedied by interpolating a new frame between every mask frame. Plus, when you mix it down to 24fps you can introduce as much motion blur and shutter angle "emulation" as you want.

ryandamm 16 hours ago | parent | next [-]

This is called “ghost frame” and already exists in Red cameras and virtual production wall tools like Disguise.

actionfromafar 15 hours ago | parent [-]

You need to basically timecode/genlock the greenscreen "illumination LEDs" to the camera so the greenscreen lights up only exactly at every other frame. Not sure if there exists any off the shelf solution which can do that but if not it can't be super hard to cobble together.

cma 14 hours ago | parent [-]

https://evermorestud.io/retroreflective-chromakey-experiment...

eichin 11 hours ago | parent | prev | next [-]

Somebody recently used a variation of this to get good video of welding - basically a camera synced with a very bright (strobe-ish) light, brighter than the weld itself, so you adjust the camera to the ludicrous-but-consistent brightness level and get details of the weld and the surroundings. https://www.youtube.com/watch?v=wSUxK8q4D0Q (Chronos "Helios", from early 2025)

shdudns 13 hours ago | parent | prev | next [-]

Two problems:

- It'll bleed on fast motion. Hair in the wind would just not work.

- Incandescent lights are out.

You could solve both by having two ghost frames shot very close to the real frame (no need to evenly space the frames, after-all) and using strobing a high powered laser.

You'd need very fast sensor or another one optically on the same position.

actionfromafar 5 hours ago | parent [-]

At some point higher fps solves it. Is 240 fps enough?

amluto 16 hours ago | parent | prev | next [-]

Surely this makes your actors feel sick? And wouldn’t it make your motion blur look dashed and also cause artifacts at the edge of the mask if there’s a lot of motion?

throwway120385 16 hours ago | parent | next [-]

You could strobe at some multiple of the sensor frame rate as long as your strobes are continuous through the integration period of the sensor and the lighting fades very quickly. This probably wouldn't work with incandescents but people strobe LEDs a lot to boost the instantaneous illumination without going past the continuous power rating in the datasheet.

amluto 14 hours ago | parent [-]

You mean do strobe, strobe, strobe, strobe, pause, pause, pause, pause? I bet that's at least as bad as holding the source on for the first four intervals and then off for the latter four intervals.

In any case, if you actually have a scene bright for 1/24th of a second and then dark for 1/24th of a second, repeating, you're well within photosensitive epilepsy range. Don't do that to your actors unless you've discussed it with them and with your insurance company first.

actionfromafar 13 hours ago | parent [-]

So, shoot at 240 fps and strobe set lights for 1/240s and backdrop for 1/240s.

wlesieutre 11 hours ago | parent [-]

And if you want a slower than 1/240th second shutter speed, no you don't

actionfromafar 6 hours ago | parent [-]

Or... you frame blend in Fusion or go full hog in Nuke.

( https://www.nukepedia.com/tools/gizmos/time/vectorframeblend... )

kibibu 16 hours ago | parent | prev | next [-]

Incandescent and fluorescent lights already flicker at your AC power frequency. Just gotta be higher than that

amluto 14 hours ago | parent | next [-]

No.

Incandescent lights flicker at twice your AC power frequency -- to a decent approximation, their power is proportional to V^2. But this is input power -- the cooling of the filament is slowish and the modulation depth is low. Most people aren't bothered by this.

Fluorescent lights with old or very crappy "magnetic" ballasts flicker at twice the mains frequency, with deep modulation. The effect on people varies from moderate to extremely unpleasant, and it's extra bad if anything is moving quickly (gyms, etc). There are even studies showing that office workers perform worse under such lighting even if they don't experience personally perceptible symptoms. The effect is so severe that people invented the "electronic ballast", which flickers at much, much higher frequency and avoids low-frequency components. Phew. (The light might still be a nasty color, but the temporal output is okay.)

"Driverless LEDs" are deeply modulated at twice the mains frequency. These are very nasty.

If you actually have a light that flickers at the AC power frequency (certain LED sources in a two-brightness diode-dimmed kitchen appliance fixture will do this, as will driverless LEDs with certain types of failures), then it's extra nasty.

There are plenty of people around who find (depending on the actual waveform) 60Hz flicker intolerable and 120Hz flicker extremely unpleasant. And there are plenty of people who can often perceive flicker under appropriate circumstances up to at least several hundred Hz and even into the low kHz with certain shapes of light sources. You can read up on IEEE 1789 to find a standard based on actual research on what lighting waveforms should look like.

The effect of 120 Hz flicker is bad enough that energy codes in some places (e.g. California) have started to require that LED sources minimize this flicker, but, sadly, it's poorly enforced.

kibibu 10 hours ago | parent | next [-]

Hey thanks for clearing this up. I had no idea that CFLs and fluorescent lights with electronic ballasts now flicker at ~ 20kHz.

SoftTalker 12 hours ago | parent | prev | next [-]

The fluorescent light strobing is why you often see fluorescent tubes in pairs. They will be wired in opposite phase to cancel the strobing.

tlb 2 hours ago | parent [-]

I think the total light output of each bulb in the pair is the same at all points in time, but the orange-blue gradient is reversed. So when one is orange at one end, the bulb beside it is blue at that end.

IIRC, the end that's negative looks orange, because the electrons emitted from the filament haven't gotten up to speed yet and can't ionize the mercury atoms at that end to the highest states.

If you didn't do this, you'd see 60 Hz strobing when you looked at one end.

toss1 11 hours ago | parent | prev [-]

Also, the human eye sees flicker much better at the periphery than in the central area. The Rod receptor cells respond more rapidly than the Cone color-sensitive cells, and the peripheral vision is also more tuned to quick motions (much advantage in having faster detection of peripheral motion, so positive selection evolutionary pressure).

15 hours ago | parent | prev | next [-]
[deleted]
joecool1029 15 hours ago | parent | prev [-]

phosphors and capacitors are a thing that mask that, so is high frequency switching way above this rate…

Anyway, an old HN submission I still use when buying light bulbs: https://news.ycombinator.com/item?id=14023196

actionfromafar 14 hours ago | parent | prev [-]

Feel sick? Possibly. People are more or less sensitive to imperceptable flicker.

Artifacts?

I bet that can be remedied by interpolating a new frame between every mask frame. Plus, when you mix it down to 24fps you can introduce as much motion blur and shutter angle "emulation" as you want.

Motion blur can also be very forgiving. You are more likely to notice artifacts in still or slow moving scenes and then the problem goes away.

dgently7 8 hours ago | parent | prev | next [-]

this is the approach that stop motion uses, except they get to keep the camera in the same place. its still not perfect because of spill from the background onto the foreground and requires additional masking and cleanup.

ErroneousBosh 14 hours ago | parent | prev | next [-]

Corridor Crew cover this in one of their VFX breakdowns where I can't remember the film but it was supposed to be filmed on a rapidly rotating platfom.

There were a large number of lights around it and each one was blinked on for an instant while the camera shot at an insanely high frame rate - something like 288 frames per second with twelve lights.

This meant that after the fact you could pick any one of the twelve frames for that 1/24th of a second, to choose the angle the light was hitting at.

huflungdung 11 hours ago | parent | prev [-]

[dead]