Unable to SSH into a VM in a cloud service

I have 2 Linux VMs on the same cloud service (added to a load-balanced set). Assigned port 22 as public and private port to SSH into VM1. Azure doesn't allow me to specify port 22 again as the public port for VM2. It gave me a random port number.

But I'm not able to SSH into VM2 using that port. I'm only able to SSH using port 23. If I add another VM to the same cloud service, I'm not sure which port needs to be used to SSH.

Is there any way I can SSH into VM2 other than port 23?

Thanks