There are several ways to make this rigorous.

An explicit epsilon-delta style proof is not that hard to produce, it's just a little messy. What you have to do is, for a given x and 0<ε, pick N large enough that you can bound the tail after x^N/N! onwards with a geometric series adding up to at most ε/2. Now pick n large enough enough that the sum of errors in the terms up to N is also bounded by ε/2. From that n on, (1+x/n)^n is within ε of the power series for e^x.