The key to the solution is the message that the container has an active disk artifact and the advice to remove it from the repository.
The procedure to remove the disk image from the repository is:
- Go to the preview site (manage.windowsazure.com).
- Click on Virtual Machines.
- Click on Disks.
- Click on the disk.
- Click on Delete Disk.
After that, the storage account can be deleted.
Note: this applies even if you've already deleted all of your Virtual Machines and it shows 0; there still will be artifacts under the disks tab.