Issue:
Our azure web sites started timing out during data querying operations. The sites didn’t start out this way, just got worse overtime. At first we thought it was related to bad query designs or amount of data, so we tweaked what we could but not much improvement.
It is all pointing to Azure having data disk I/O issues or network issues to high volume i/o operations.
Tests:
We created three different database servers to prove the issue, we wanted to make sure it wasn’t related to a bad storage account or VM size/hardware. We used the same database or catalog on all servers, and ran the same test query. Note we recreated stats, updated indexes and flushed sql cache before all tests. All VM inside of our own location/group.
Findings:
Test 1:
DBServer1 – Large VM running windows 2008r2, with sql 2008r2 standard – our original server to go live on, our own image.
DB1:data disk attached to existing storage account1, test query would takeover 7 minutes to complete and just under 4 minutes without flushing the cache.
DB1:attached to local c: drive, same test query would run under 11 seconds, 4 seconds without flushing the cache.
DBServer2 – Large VM running windows 2008r2 datacenter, with sql 2008r2 standard - Microsoft image.
DB1:data disk attached to existing storage account1, test query also tookover 7 minutes to complete and just under 3 minutes without flushing the cache.
DB1:attached to local c: drive, same test query would run under 11 seconds, 4 seconds without flushing the cache.
DBServer2 -Same but – Extra Large VM running windows 2008r2 datacenter, with sql 2008r2 standard - Microsoft image.
DB1:data disk attached to existing storage account1, test query also tookalmost 7 minutes to complete and just under 3 minutes without flushing the cache.
DB1:attached to local c: drive, same test query would run under 11 seconds, 4 seconds without flushing the cache.
DBServer3 – Large VM running windows 2008r2 datacenter, with sql 2008r2 standard - Microsoft image. New storageaccount2.
DB1:data disk attached to new storage account2, test query also tookalmost 2 minutes to complete and just under 40 seconds without flushing the cache. (big improvement, but nowhere close to local drive)
DB1:attached to local c: drive, same test query would run under 11 seconds, 4 seconds without flushing the cache.
Options at this point are: move all databases to local drives, but space issue is my concern, or move off Azure and back to our own rack.
Question:
Is this a known issue, and can it be fixed?
JLA