Interesting take and similar to something I've recently been working on. Have a look at janet-num.[1] Plain numeric C kernels under Janet.

Janet for orchestration, C for the frame loop.

[1] https://github.com/jennystats/janet-num