Removing VHD Disk fails in PowerShell after Removing a VM

I'm trying to script the removal of my Azure VM's and their associated Data Disk (VHD). I have the following PowerShell script:

...
# Remove both the VM and then the Data Disk VHD so it's not orphaned.
$vmToDelete = Get-AzureVM -ServiceName $serviceName -Name $vmName
$vmDisk = Get-AzureOSDisk -VM $vmToDelete
# Remove the VM first.
Remove-AzureVM -ServiceName $serviceName -Name $vmName

# THE FOLLOWING FAILS SINCE IT SEEMS TO THINK THE DISK IS STILL IN USE	
Remove-AzureDisk -DiskName $vmDisk.DiskName -DeleteVHD

When the script is run, it fails with the following error:

Remove-AzureDisk : HTTP Status Code: BadRequest - HTTP Error Message: A disk with name <DISKNAME>is currently in use by virtual machine <VMNAME> running within hosted service <SERVICENAME>, deployment <DEPLOYMENTNAME>.
Operation ID: 6fcd670056c547cfba5d930c392765b6
At C:\Workspaces\DeProvisionAzureTestServers.ps1:33 char:18+     Remove-AzureDisk <<<<  -DiskName $vmDisk.DiskName -DeleteVHD+ CategoryInfo          : CloseError: (:) [Remove-AzureDisk], CommunicationException+ FullyQualifiedErrorId : Microsoft.WindowsAzure.Management.ServiceManagement.IaaS.RemoveAzureDiskCommand

It seems I need to add about a 5 minute pause to the script between removing the VM and deleting the associated VHD. This seems hacky. Why can't the VHD be removed immediately after the VM is deleted? Is there a better way to accomplish this?

Thanks,
Richard