perf event collection overhaul

[brayniac]

Perf events sampler needs more advanced scheduling. We may want to limit the number of event counters to avoid multiplexing on the PMUs. Extending support to per-cgroup as requested in #19 requires that we do something clever to avoid performance penalties. At the same time, we would want to be able to enforce that things like Cycles and Instructions are sampled across the same intervals so CPI calculations remain valid.

[sargun]

In regards to sampling cycles / instructions across the same interval, wouldn't putting them in the same perf event group solve that problem so the kernel schedules them together?

[brayniac]

@sargun - there's more to this than that. We might want to change the config to allow for explicit collection of CPI instead of tracking cycles and instructions separately. This would allow us to provide histograms of CPI from sub-minutely intervals across each minute.

The other aspect of this is how we can collect perf events with low impact to production workloads. We've seen high-impact collecting per-cgroup for even a single event and impact from collecting many host-level counters with perf subsystem handling multiplexing. I haven't dug into the perf multiplexing code yet to see if there's anything we could do better in terms of mapping events to PMUs. Assuming that's already optimized, we probably need to limit the duty cycle of collection for each event and essentially do sampling instead of counting.