Azure VM limit access to specific IP?

My question hass to do with the Azure Virtual Machines. In the VM we can configure our end points, and allow access through on various ports etc. but is there any way that we can limit that access to a specific IP address? For example, I have RDP open on 3389, but is there a way to limit that access to only my offices external IP address, rather than it being open to everyone?
 
Thanks in advance!