Physics and math demand a certain mode of thought. Lots of people think that doing well in those classes takes intelligence, and that's part of it. But they also require something else that is not always a good thing: comfort with abstraction, or stripping problems down to an idealized cartoon.
Those of us who excelled in these subjects can be a bit smug towards those who didn't, but replacing real life with a cartoon isn't always a good thing. In addition to hindering social relations, it can obscure important truths.
It's interesting to contrast Aristotle's view, for example--that objects in motion naturally come to rest--with Newton's--that they naturally keep moving. Thinking about familiar objects, you have to grant that Aristotle had a good point. Of course, he'll leave you flat: if you figure out how to include friction, Newton is going to get you a lot further--even to the moon. But beginning students are asked to commit to an abstract formalism that has a stylized and flawed relationship to the world they know.
Probability has a similar problem.
Most normal people, for example, expect that after a flipped coin shows a string of heads, tails is "due." Probability theory says otherwise: the coin doesn't "know" what happened before, so the chances on the next flip are still 50/50. Abstraction wins, intuition loses.
[Actually, Stanford researchers showed in 2007 (pdf here) showing that unless a coin is flipped perfectly, the results will not be 50/50: if the coin is spinning in its plane at all, its angular momentum will tend to keep it pointed the way it started out. But that's a minor issue.]
On the other hand, there are lots of cases where common intuition is "directionally correct." Take another staple of introductory probability courses: a bag full of different-colored balls. In this case, the probability won't stay the same unless you put each ball back in the bag after you choose it. If you keep it, choosing a ball of one color will increase the chances of a different color on the next pick, in keeping with intuition.
Of course, intuition doesn't get the answer with any precision, and it gets it completely wrong for the coin flip. To do it right, you need the abstract formalism. Still, it's easy to imagine that our brains are hard-wired with an estimating procedure that gets many real-world cases about right.
In other cases, our intuition is more flexible than slavish devotion to calculation. Suppose I start flipping a coin. It's not surprising to see heads the first time, and the second time. How about the third time? The tenth? If it keeps coming up heads, you will quickly suspect that there's a problem with you original assumption that the probability is 50%. Your natural thought processes will make this shift naturally, even if you might be hard pressed to calculate why. Probability theory is not going to help much when the assumptions are wrong.
It's true that people are notoriously bad at probability. ScienceBlogger Jason Rosenhouse has just devoted an entire book to one example, the "The Monty Hall Problem: The Remarkable Story of Math's Most Contentious Brain Teaser. (It was also discussed in 2008's The Drunkard's Walk, by Leonard Mlodinow, and in The Power of Logical Thinking, by Marilyn Vos Savant (1997), the Parade columnist who popularized it.)
The Daily Show's John Olliver amusingly explored how simple probability estimates (especially at around minute 3:20) help us misunderstand the chances that the Large Hadron Collider will destroy the world.
Still, our innate estimation skills developed to deal tolerably well with a wide variety of situations in which we had only a vague notion of the underlying principles. Highly contrived, predictable situations like the coin flip would have been the exception. Even though our intuition frequently fails in detail, it helped us survive a complex, murky world.