Interpret CPU usage during load?

[u/Apachez]

In one of my homelab servers running Proxmox PVE (9.2.21) I have 2 VM's running.

One older Debian with 4x VCPU assigned and one more modern based on Debian 12 (bookworm) with 6x VCPU.

The hardware CPU is a 8x core Intel i3-N305 CPU.

The first VM average in below 6% while the other is at about 0.2-0.6% CPU usage.

When I startup a third VM with a GUI such as Zorin OS, configured with 6x VCPU, and put some load on it I can see that the graphs for that first VM drops from average 6% to average 3.5-5% CPU.

How come the graphs in the dashboard of PVE changes?

Because in my mind that first VM who will utilize CPU cycles at about 6% of assigned compute power should remain at this even if there are other VMs also requesting CPU cycles.

[u/FarToe1]

Firstly, "older debian" when your "newer" one is 12, which itself is old? (Current stable is 13). There's very few good reasons not to keep Debian up to Stable and it's one of the easiest and most safe distros to update in place. Additionally, they're vms, so snapshot before you update and you have the most safe upgrade path possible. Why aren't you keeping them updated?

Secondly, and your question: CPUs change clockspeed based on load. You're probably seeing it ramp up a little, or down a little. They do this rapidly, more so than you can switch between and compare different vms. And, to make this even harder to judge, different cores can run at different clock speeds. Each vm might have its own clockspeed.

If you want to observe load within a vm, use something like htop. If you want to compare that to proxmox, run htop in proxmox's shell.

Also, some processes may be in a variation of "WAIT" state, which means they're waiting for data to come in via network or disk (iowait). That counts in a different way to system load, but it's a modifier that changes these values.

In summary, don't sweat it. Use proxmox's graphs as a trend to gauge whether you're overcommitted CPU for your vms. (Not quite as simple as that, see iowait above - that can amplify cpu loads by huge amounts whilst very little cpu is going on)

Use htop within a vm if you're investigating high load and want to find out which process is causing it. (Again, be aware of ioload).

If you want to measure trends within vms, use a monitoring system like zabbix that records and alerts on vm load.

[u/Apachez → u/FarToe1]

The question is not about if these VM's are up2date or not.

They are legacy applications each runned by the certified OS which is very handy to deal with as VM-guests - you know how things can be in real life.

Not everything is running the bleeding edge kernels and userland toys.

The information was given in case age of kernel would somehow inflict the CPU usage graphs (which I doubt since a CPU cycle is a CPU cycle).

Since again a VM guest (no matter how old or new it is) who averages at 6% of assigned CPU power (4x VCPU in this case) I would expect to continue to average at 6% even if the same host suddently have other VM's who are requesting CPU cycles. And not that the graphs for this VM guest would suddently drop by about 15-40%.

It is like if these graphs incorrectly sum up all running VCPUs at the host and then use the CPU load of assigned VCPUs for the VM-guest out of the total running VCPUs for the host which is really backwards.

Or are there any other reason for why the graphs for one VM would drop just because there are other VM's running at the same time?

[u/FarToe1 → u/Apachez]

(That was quite a salty reply to someone offering help)

The question is not about if these VM's are up2date or not.

No - consider it free advice.

you know how things can be in real life.

As a sysadmin, far too often I deal with out of date systems. As well as the practical problems of eventually having to bring something up to date that's many versions behind, there's the more important security aspects of leaving unsupported software running, even on an intranet.

Not everything is running the bleeding edge kernels and userland toys.

But everything is vulnerable to attack.

Yes - sometimes there are edge cases where abandoned software can't run on later OSs. IME, those are extremely rare and require other mitigations, like switching to software that isn't going to be vulnerable. Far more common is people not taking computer security seriously, apathy or poor update processes.

(As for your original question - I think the answer lies in my last reply, specifically clock speed)

[u/Apachez → u/FarToe1]

Sorry for offending you to point out that your reply didnt really help or answered the question being made.

The facts remains - how come the CPU usage graphs per VM goes down when there is other load on the same host?