I am running MongoDB on an Ubuntu on a single AWS EC2 as m4.2xlarge. The disk is 1280GB io1 22000 IOPS on XFS file system, 73% full (it is only used for MongoDB). The MongoDB server version is 3.4.10.
I have a collection with 11.8 billion documents. I set in motion creation of a background index on this collection on a single Int64 field. It's been running for several days now with only 23% of the index complete. It started off fast but right now the rate is about 4.5k documents per second. If the rate stays, the indexing will be completed in 20+ more days. That's totally unacceptable. The document per second rate has been dropping since start of indexing.
The server is currently unaffected by other queries. Unfortunately, it is impossible for me to run a foreground index as there are other databases that are actively being used.
What can I do to speed this up? Any suggestions are welcome. Should I stop the index, beef up the EC2 instance and try again? I do not see that the CPU is working too hard. Should I change the filesystem? Will upgrade to MongoDB 3.6.2 affect indexing performance?
Is there anything in CloudWatch metrics for my disk that is eye-opening?