Interesting... is it overkill to use a multimodal model like this on just audio? Could the model size be decreased dramatically, or does the multimodality help the accuracy and speed in some way?

▲

pjc50 25 minutes ago | parent | next [-]

The benefit is that you can take input in one "mode" and extract output in another; that is, the input is audio mode and the output is text mode. The intermediate representation is the "embedding".

▲

sd9 2 hours ago | parent | prev | next [-]

Multimodality seems to just help with everything

▲

coldtea 3 hours ago | parent | prev [-]

Correctly transcribing requires context too, no?

	▲	ranger_danger 2 hours ago \| parent [-]
		I suppose it depends on the words in question... language knowledge and general world knowledge may not always both be necessary to transcribe accurately.