Mathematics is widely considered to be a prerequisite for learning physics. However, it is quite naive to believe that learning basic math is sufficient to use mathematics as a reasoning tool to think about the physical world. The main reason is that using mathematics in physics is substantially different than in math. In this paper we show how the way physicists make use of some basic mathematical concepts (e.g. multiplication, division and functions) is specific to physics by identifying their historical genesis and contrasting with the way these concepts are usually taught in math lessons. We argue that the explicit acknowledgement of these differences has important didactical implications.