Yes he did start with a "small loan of a million dollars"!

Buffett started with 1.2 million dollars (105K in 1956) in investments from his family, i.e. his aunt, sister, and father in law.

It is very nearly impossible to get rich without starting with a huge chunk of money, Buffett is no exception.

Gates/Allen started Microsoft with $5000. Jobs started Apple by selling his VW bug.

You should read his whole bio.