There is an article by Daniel T. Willingham entitled "Is It True That some People Just Can't do Math?" that answers this particular question well for me. D.T.W. references many others in his Endnotes.
I think memorize is the wrong word. You don't memorize where to put your foot when you ride a bike. When you drive, you don't memorize whether the right foot or left foot is the break. Well.... maybe when you first start driving you do, but my point is, calculation is a skill, that you learn by using. Memorizing is for facts.