Discussion:
Performance degraded, Bookworm on Raspberry Pi 4B
(too old to reply)
Hank Barta
2022-03-14 21:40:01 UTC
Permalink
First of all, many many thanks for making Debian work on a Raspberry Pi 4B.
I bounce back and forth between Bullseye and Bookworm on a Pi 4B and have
noticed a performance degradation with Bookworm.

TL;DR
Install (current) Bookworm - performance good.
apt update/upgrade - performance tanks.
Revert 5.16 to 5.15 kernel - no improvement.

It seems that one of the 103 packages that get installed during the update
causes this to happen.

More detail at https://github.com/HankB/Pi-4B-bookworm-performance

Is there anything I can do to help track this down? (I hope the answer is
not "install the packages one at a time to see what causes this.")

Thanks!
--
Beautiful Sunny Winfield
Hank Barta
2022-03-15 02:40:01 UTC
Permalink
I started installing packages I thought were likely to cause the regression
(one or two at a time) and following installation of raspi-firmware the
problem appeared.
Post by Hank Barta
First of all, many many thanks for making Debian work on a Raspberry Pi
4B. I bounce back and forth between Bullseye and Bookworm on a Pi 4B and
have noticed a performance degradation with Bookworm.
TL;DR
Install (current) Bookworm - performance good.
apt update/upgrade - performance tanks.
Revert 5.16 to 5.15 kernel - no improvement.
It seems that one of the 103 packages that get installed during the update
causes this to happen.
More detail at https://github.com/HankB/Pi-4B-bookworm-performance
Is there anything I can do to help track this down? (I hope the answer is
not "install the packages one at a time to see what causes this.")
Thanks!
--
Beautiful Sunny Winfield
--
Beautiful Sunny Winfield
Gunnar Wolf
2022-03-15 04:30:01 UTC
Permalink
Post by Hank Barta
I started installing packages I thought were likely to cause the regression
(one or two at a time) and following installation of raspi-firmware the
problem appeared.
As the maintainer of raspi-firmware, I have to say:

Ouch.


raspi-firmware, as you know, is just a blob we cannot debug or do
anything about; it is a core component of our Raspberries, but... it
is non-free ☹

Given you have already pursued the situation to this detail, could you
share... between which versions do you see the regression?

FWIW, I cannot do much about it, other than report it as an issue in
https://github.com/raspberrypi/firmware/issues (which I will do) ;
taking a quick look at the reported issues, and found just the
following, although I don't know if they are in any way related to
your report:

https://github.com/raspberrypi/firmware/issues/1619

Greetings, and thanks for this report!
Diederik de Haas
2022-03-15 11:20:01 UTC
Permalink
Post by Gunnar Wolf
Given you have already pursued the situation to this detail, could you
share... between which versions do you see the regression?
raspi-firmware/testing 1.20220120+ds-1 arm64 [upgradable from: 1.20210805+ds-1]
Hank Barta
2022-03-15 12:40:01 UTC
Permalink
Some additional information that might be useful is that the system seems
unable to determine CPU frequencies. For example

***@up:~# cpupower frequency-info
analyzing CPU 0:
no or unknown cpufreq driver is active on this CPU
CPUs which run at the same hardware frequency: Not Available
CPUs which need to have their frequency coordinated by software: Not
Available
maximum transition latency: Cannot determine or is not supported.
Not Available
available cpufreq governors: Not Available
Unable to determine current policy
current CPU frequency: Unable to call hardware
current CPU frequency: Unable to call to kernel
***@up:~# ls /sys/devices/system/cpu/cpu0/
cpu_capacity crash_notes_size node0/ power/
subsystem/ uevent
crash_notes hotplug/ of_node/ regs/
topology/
***@up:~# ls -l /sys/devices/system/cpu/cpu0/
total 0
-r--r--r-- 1 root root 4096 Mar 15 12:22 cpu_capacity
-r-------- 1 root root 4096 Mar 15 12:22 crash_notes
-r-------- 1 root root 4096 Mar 15 12:22 crash_notes_size
drwxr-xr-x 2 root root 0 Mar 15 12:22 hotplug
lrwxrwxrwx 1 root root 0 Mar 15 12:22 node0 -> ../../node/node0
lrwxrwxrwx 1 root root 0 Mar 15 12:22 of_node ->
../../../../firmware/devicetree/base/cpus/***@0
drwxr-xr-x 2 root root 0 Mar 15 12:22 power
drwxr-xr-x 3 root root 0 Mar 15 12:22 regs
lrwxrwxrwx 1 root root 0 Jan 1 1970 subsystem -> ../../../../bus/cpu
drwxr-xr-x 2 root root 0 Mar 15 12:16 topology
-rw-r--r-- 1 root root 4096 Jan 1 1970 uevent
***@up:~#

Whereas on a Pi 4B running Bullseye without this issue

***@kweli:~$ cpupower frequency-info
analyzing CPU 0:
driver: cpufreq-dt
CPUs which run at the same hardware frequency: 0 1 2 3
CPUs which need to have their frequency coordinated by software: 0 1 2 3
maximum transition latency: Cannot determine or is not supported.
hardware limits: 600 MHz - 2.00 GHz
available frequency steps: 600 MHz, 700 MHz, 800 MHz, 900 MHz, 1000 MHz,
1.10 GHz, 1.20 GHz, 1.30 GHz, 1.40 GHz, 1.50 GHz, 1.60 GHz, 1.70 GHz, 1.80
GHz, 1.90 GHz, 2.00 GHz
available cpufreq governors: performance schedutil
current policy: frequency should be within 600 MHz and 2.00 GHz.
The governor "schedutil" may decide which speed to use
within this range.
current CPU frequency: Unable to call hardware
current CPU frequency: 600 MHz (asserted by call to kernel)
***@kweli:~$ cat /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq
2000000
***@kweli:~$

The second example shows a Pi that has been overclocked and in fact, I
discovered this issue when I tried to overclock a Pi 4B running Bookworm. I
also discovered that the system did not respond to the settings in
/boot/firmware/config.txt that are used to overclock. The following
settings seem to have no effect

arm_freq=2000
over_voltage=6
gpu_freq=750

best,
Post by Diederik de Haas
Post by Gunnar Wolf
Given you have already pursued the situation to this detail, could you
share... between which versions do you see the regression?
1.20210805+ds-1]
--
Beautiful Sunny Winfield
Diederik de Haas
2022-03-15 14:10:01 UTC
Permalink
Post by Hank Barta
Some additional information that might be useful is that the system seems
unable to determine CPU frequencies. For example
no or unknown cpufreq driver is active on this CPU
CPUs which run at the same hardware frequency: Not Available
CPUs which need to have their frequency coordinated by software: Not
Available
maximum transition latency: Cannot determine or is not supported.
Not Available
available cpufreq governors: Not Available
Unable to determine current policy
current CPU frequency: Unable to call hardware
current CPU frequency: Unable to call to kernel
I lack the knowledge to properly/fully understand the issue Gunnar linked to,
but it looks applicable ... still (despite the last msg indicating the issue
was fixed).
I don't know exactly the role that clocks play, but they seem quite important.
And when the value of a/multiple clock(s) return 0, it doesn't surprise me
that it would cause other things to fail, like the ones you reported above,
which in turn cause the CPU (f.e.) to be way slower then it can be.

Personally I think it's more useful if Hank reported the issue himself to
https://github.com/raspberrypi/firmware/ then let Gunnar do it on his behalf.

Having a bug in Debian's BTS (against raspi-firmware) regarding this issue
seems useful as well and with the 'forwarded' keyword it can be linked to the
upstream issue.

Cheers,
Diederik
Hank Barta
2022-03-15 15:40:01 UTC
Permalink
I tested this morning with an install on an SD card and the disk benchmarks
were not significantly different. The updated `raspi-firmware` did seem to
result in slightly slower performance (7188 events before and 6525 after)
but I attribute this to a low CPU clock rather than a problem with the SD
card driver. Details at
https://github.com/HankB/Pi-4B-bookworm-performance#2022-03-15-testing-with-sd-card
.

I'll be happy to file the bug reports.
https://github.com/raspberrypi/firmware/issues/1705

Debian BTS next.
Post by Diederik de Haas
Post by Hank Barta
Some additional information that might be useful is that the system seems
unable to determine CPU frequencies. For example
no or unknown cpufreq driver is active on this CPU
CPUs which run at the same hardware frequency: Not Available
CPUs which need to have their frequency coordinated by software: Not
Available
maximum transition latency: Cannot determine or is not supported.
Not Available
available cpufreq governors: Not Available
Unable to determine current policy
current CPU frequency: Unable to call hardware
current CPU frequency: Unable to call to kernel
I lack the knowledge to properly/fully understand the issue Gunnar linked to,
but it looks applicable ... still (despite the last msg indicating the issue
was fixed).
I don't know exactly the role that clocks play, but they seem quite important.
And when the value of a/multiple clock(s) return 0, it doesn't surprise me
that it would cause other things to fail, like the ones you reported above,
which in turn cause the CPU (f.e.) to be way slower then it can be.
Personally I think it's more useful if Hank reported the issue himself to
https://github.com/raspberrypi/firmware/ then let Gunnar do it on his behalf.
Having a bug in Debian's BTS (against raspi-firmware) regarding this issue
seems useful as well and with the 'forwarded' keyword it can be linked to the
upstream issue.
Cheers,
Diederik
--
Beautiful Sunny Winfield
Gunnar Wolf
2022-03-15 16:10:01 UTC
Permalink
Post by Diederik de Haas
Personally I think it's more useful if Hank reported the issue himself to
https://github.com/raspberrypi/firmware/ then let Gunnar do it on his behalf.
I agree with you, Diederik. I think Hank has a better grasp of the
problems he is reporting than me, and it would be better if he were
able to send this information to the upstream authors; of course, it
makes sense to link to this thread.

Hank, are you OK with this suggestion?
Gunnar Wolf
2022-03-15 17:00:01 UTC
Permalink
Package: raspi-firmware
Version: 1.20220120+ds-1
Severity: important
Tags: upstream

Hank Barta reported a performance regression under the Raspberry Pi
4B, after comparing Bullseye and Bookworm (Debian 11 and 12). I am
copying over to this bug report only part of the information; a
full(er) report is available in Hank's GitHub project
Pi-4B-bookworm-performance:

https://github.com/HankB/Pi-4B-bookworm-performance

The problem presented when running sysbench:

/--- bullseye ↓
| $ time sysbench --test=cpu --cpu-max-prime=20000 run
| (...)
| CPU speed:
| events per second: 584.72
|
| General statistics:
| total time: 10.0006s
| total number of events: 5850
| Latency (ms):
| min: 1.70
| avg: 1.71
| max: 2.99
| 95th percentile: 1.73
| sum: 9997.34
|
| Threads fairness:
| events (avg/stddev): 5850.0000/0.00
| execution time (avg/stddev): 9.9973/0.00
+--- bookworm ↓
| # time sysbench --test=cpu --cpu-max-prime=20000 run
| (...)
| CPU speed:
| events per second: 233.03
|
| General statistics:
| total time: 10.0038s
| total number of events: 2333
|
| Latency (ms):
| min: 4.28
| avg: 4.29
| max: 4.34
| 95th percentile: 4.33
| sum: 10001.67
|
| Threads fairness:
| events (avg/stddev): 2333.0000/0.00
| execution time (avg/stddev): 10.0017/0.00
\---

Hank did a thorough saerch, and found out this issue happens after the
firmware update (raspi-firmware/testing 1.20220120+ds-1 arm64
[upgradable from: 1.20210805+ds-1]).

Further information is available on the debian-arm mailing list
thread:

https://lists.debian.org/debian-arm/2022/03/threads.html#00003

I prompted Hank to report this issue to the Raspberrrypi Firmware
project in GitHub, which he did, but was outright dismissed:

https://github.com/raspberrypi/firmware/issues/1705

-- System Information:
Debian Release: bookworm/sid
APT prefers unstable
APT policy: (500, 'unstable')
Architecture: amd64 (x86_64)

Kernel: Linux 5.16.0-3-amd64 (SMP w/8 CPU threads; PREEMPT)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US:en
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

--

Loading...