Well, being a well educated mathematician, you should have basic knowledge of statistics. Things to learn: basic probability theory (independent events, conditional probability, Bayes' Theorem), random variables (r.v.), expectation, convergence of r.v., maximum likelihood estimator, basic hypothesis testing, p-values are the basic. If you want to go a bit further, check out linear regression, linear model, residual, categorical predictors, logistic regression, ANOVA (analysis of variance) etc.
Probability:
- Bass R.F. Undergraduate Probability [.pdf] (FREE!)
- Bertsekas D.P. and Tsitsiklis J.N. Introduction to Probability - Leave chapter 5 and 6 to next stage. Solution here [.pdf].
- Boik R. J. Probability [.pdf] (FREE!)
- Gnedenko B.V. The Theory of Probability - First six chapters.
- Grinstead C.M. and Snell J.L.'s Introduction to Probability (FREE!) - Read first ten chapters.
- Lebanon G. Tutorial Notes (FREE!) - Read P1 - P14.
- Lipschutz S. and Lipson M. Schaum's Outline of Probability - Read chapter 1-6.
- Mosteller F. Fifty Challenging Problems in Probability with Solutions - Later stages topics included.
- Ross S. A First Course in Probability - First eight chapters.
- Santos D. Probability Lecture Notes [.pdf] (FREE!)
- Stirzaker D. Elementary Probability - Leave the last chapter.
Statistics:
- Abebe A., Daniels J., McKean J.W. and Kapenga J.A. Statistics and Data Analysis (FREE!)
- Berry D.A. Statistics: A Bayesian Perspective
- Boik R. J. Elementary Statistics (FREE!)
- Boik R. J. Statistics [.pdf] (FREE!)
- Bulmer M.G. Principles of Statistics
- DeCoster J. Introductory Statistics Notes [.pdf] (FREE!)
- Dekking F.M., Kraaikamp C., Lopuhaä H.P. and Meester L.E. A Modern Introduction to Probability and Statistics: Understanding Why and How
- El-Taha M. MBA604 Introduction Pobabiltity and Statistics Lecture Notes [.pdf] (FREE!)
- Gerstman B.B. StatPrimer (FREE!)
- Hodges J.L. and Lehmann E.L. Basic Concepts of Probability and Statistics - Non-calculus based text by two renowned statisticians.
- Lane D., Lu J., Peres C. and Zitek E. Online Statistics: An Interactive Multimedia Course of Study (FREE!)
- Larsen R.J. and Marx M.L. An Introduction to Mathematical Statistics and Its Applications
- Lavine M.L. Introduction to Statistical Thought (FREE!)
- McFadden D. Statistical Tools for Economists (FREE!)
- Ross S.M. Introduction to Probability and Statistics for Engineers and Scientists
- Simon J.L. Resampling: The New Statistics (FREE!)
- Spiegel M.R. Schaum's Outline of Statistics - As a reference. Chapters up to 12 are core, interested reader may read up to chapter 16 at this stage.
- Stockburger D.W. Introductory Statistics: Concepts, Models, and Applications (FREE!)
- Utts J.M. Seeing Through Statistics
- 山东大学数学学院:概率论与数理统计 [.rar] (FREE!)
Second course in Statistics:
- Janke S.J. and Tinsley F. Introduction to Linear Models and Statistical Inference - Examples illustrated with Minitab.
- Myers R.H. Classical and Modern Regression with Applications
- Miller J. and Haden P. Statistical Analysis with the General Linear Model (FREE!)
- Taylor J. Introduction to Applied Statistics (FREE!)
- Taylor J. Introduction to Regression Models and Analysis of Variance (FREE!)