OK, so I get that you're trying an alternative on the metaphor that drag and drop uses. The metaphor is "pick up the item, move it, then drop it", but the implementation doesn't work well on mobile, so you made it modal. But for me scrolling feels like "pick up the item, move the rest of the world so the item above the target, then drop it". It feels very strange.

As an refinement, I would keep the modality but add back in the dragging paradigm. Use a typical "move item" scrubber control, but have that control switch the item to a floating modality, and give it "send back" and "drop here" buttons. The user can drag the item, scroll, interact with the page, and the item stays with them to get where they want to go. Here is a crude mockup: https://excalidraw.com/#json=9DwYqkWRhdEgEEqyzY7X8,fCdPzIzTm...