I don't have that experience with Nextcloud Memories.

Everything works well and it's comparably fast with Google Photos for me, and scrolling to specific dates works fine.

How long ago did you try it? I've only been using it for a few months so maybe it's improved over time.