You can pick which albums on your phone to upload to Immich. You and your wife could have separate users on the server too if you want that. I think you can probably share a user account, or share albums between users, but the syncing might get confusing if you both have an album with the same name. The only reason I can think of to not upload everything on your phone and try to share one or two albums is that it might get hard to search through many pictures, even with the AI.

As for not wanting most of your photos, Immich also includes AI search and facial recognition which both work really well. I can't remember if it detects near-duplicates, but I thought it did. I think you should play around with it before you leap into the giant project of making your own app.