We are testing PostgreSQL on Graviton2 instances more specifically on c6g.8xlarge instance with 3TB of EBS space with GP2 volume.
Testing was performed using CentOS 7.9, CentOS 8.0 and Amazon Linux, PostgreSQL compiled with LSE enabled, and with LSE disabled, PostGis compiled manually.
The results with LSE were better (reaching 5%), very different from the benchmarks published out there, but that is not the focus of this thread nor the problem.
We ran a POC in production with two ASGs, one with 150 instances of c6g.8xlarge, and other ASG with 150 instances of c5.9xlarge instances.
In all instances c6g.8xlarge we reached a usage plate of 4 thousand IOPS even with 9 thousand IOPS available, causing increased CPU usage and consequently latency in response times, behavior expected when we reached the IOPS limit available for the instance, which was not the case.
We thought it was a problem with the instance used, but using the fio (Flexible I/O tester) we were able to use all the IOPS available for the instance. This way we rule out a problem with this type of instance and EBS.
We focused our tests using PostgreSQL, we created several tests using pgbench and we didn't achieve more than 4 thousand IOPS even having 9 thousand IOPS available.
These same tests using instances of type C5.9xlarge consume all the IOPS available for the instance.
What makes us arrive at this moment, where we think it could be a problem or configuration of the operating system but very remotely because using the fio we were able to use all the available capacity, or something from PostgreSQL?
