What is the user experiencing, in reality

Some of our active user pains regarding performance:

Check the p90 that corresponds to the user pain- track the 90% worst case scenario for performance

Use more measurement systems to see if we are constrained by CPU/Memory/Disk/Network/Contention

This will often require additional measurement, custom code that drills into the specific area.

The solution depends on the bottleneck, but the short answer is to remove work during the hot path. Use fewer of the constrained resource.

Cache, batch, eliminate redundancy, precompute.

The solution must be verified with measurements.

To prevent perf regressions, we should continue measuring.