Bump - anyone got an idea how to monitor this?

Hi there,

Unfortunately there isn't a Cloudwatch Metric for the current IOPS Credit balance like there is with the CPU credits but I am definitely submitting a feature request through to our EBS/Cloudwatch teams for such a metric.

With regards to the Provisioned IOPS SSD Volumes the line "The ratio of IOPS provisioned to the volume size requested can be a maximum of 30" would mean that for your 10GB volume the maximum would be 300 IOPS.

Provisioned Iops < 30 x Volume Size 

Regards
Jaco

We just got burned by an extended IO burst, which turned into extended bad performance. (Not enough to drop the box, but crappy performance for some time)

I think most people would consider Running low/out of IOPS Credit an exceptional situation.

Where do I throw in another vote for a Cloudwatch Metric for current IOPS Credit balance?

Hi ivan_squixa,

Sorry to hear about the poor performance you experienced. Your vote has been added to our internal feature request.

In the meantime, you can check CloudWatch for the Read and Write operations performed within a given time period and compare that to the IO credits earned in the same period (3 IOPS per GB per second). You could base this on the following AWS CLI query (adjust to suit your needs):

aws cloudwatch get-metric-statistics --metric-name VolumeReadOps --start-time 2015-02-10T00:00:00 --end-time 2015-02-11T23:59:59 --period 86400 --namespace AWS/EBS --statistics Sum --dimensions Name=VolumeId,Value=vol-cfa9f0c4 --region eu-west-1

The above query sums the number of Read operations in a 24 hour period. You can adjust the query to count the number of Write operations (VolumeWriteOps) and then add the two results together to compare it to the IO credits earned during the same period.

Any GP2 volume will be able to provide up to 3,000 IOPS in bursts that can last up to 30 minutes. This is due to the Token Bucket model that is used: each GP2 SSD volume can accumulate up to 5.4 million IOPS credits ( 5400000 / (3000 * 60) = 30min ). When these credits are exhausted, the volume will be limited to its baseline IOPS performance (3 IOPS per GB per second). If you're using fewer than 3000 IOPS, your "above-baseline" performance will last longer than 30 minutes.

Obviously this method is not ideal - having a method of querying the IO Credit Balance directly would be a much easier and very useful addition to CloudWatch.

I hope this helps!

Kind regards,
Chris

Hi guys,

We just got burned by this too - IO EBS performance went through the floor and it would take 20-30 minutes to reboot due to (I guess) slow baseline performance on an 8 GB volume, and the need to wait for credit accumulation. Support finally came through with the explanation, but took them a while to sort it out.

Please add my vote for this metric to the total. Being able to see this would make using GP2 volume types much more predictable.

Many thanks,
-John

Hi John,

Thanks for sharing your experience. Your vote has been added.

Kind regards,
Chris

please add my vote for this very useful cloudwatch metric

We hear you all loud and clear... Feel free to continue adding +1's.

-Chris

+1 this would help troubleshoot weird performance issues more quickly.

I'm currently getting burned by this. Please add a vote for me!

+1 would avoid getting caught out by this as happened to us yesterday

+1

+1
Also, how do you get the volume id used by a RDS instance ?
This is required to apply ChrisP@AWS suggestion above.

+1

+1. Good feature. Also would be a nice option to buy credits if you went over your limit and were in a pinch. The analog of like putting dimes in the old public phones when you were over your minutes.

Paying for a little boost would be great.

If you're stuck in the hole today, I'd suggest creating a snapshot of the volume & recreate it. New volumes start with 5.4M I/O credits (3K IOPS for 30 mins). This is particularly effective for small volumes, which will are most prone to falling into the I/O hole, take the longest to climb out, and are the quickest to snapshot.

+1 for paying for the extra boost

Edited by: vharutyunyan on May 18, 2015 11:55 AM

Edited by: vharutyunyan on May 18, 2015 11:56 AM

+1

Ouch... again and again (got bitten today, took a farm down in minutes)

We've moved to alerting on any Read or Write OPs exceeding our provisioned OPs for a period.

It seems there is no volume metric for overall OPs, only ReadOps and WriteOps. Which is a shame because perf is limited on the combined value... and cloudwatch doesn't do metric addition.

+1 - it's extremely difficult to gauge whether you have enough IOPS because as ivan said the metrics are split, so you can't even set an alarm to be aware if you are overusing your credits. And it seems it takes hours to perform a modification once you are out of the credits, which is terrible. Makes me seriously consider going back to magnetic until this is easier to monitor.

+1 on this too from me.
We have also been bitten by this. We know what to look for now, but it was a shock going from magnetic to SSD and having an instance fall over for seemingly no reason - the issue turned out to be that we had run out of IOPS.
I know it's mentioned in the documentation but I feel a lot of support queries would be answered before they were even asked if it was made crystal clear that users should be mindful of running out of IOPS when using SSD EBS especially on boot volumes.

+1

Hello,

Thank you for expressing your interest in this feature. I have added your votes to our feature request portal.

Regards
Shiroy

Why isn't this request added yet????

+111111111111111111111!!!!!!!!!!!!!!!!!!!!!!