Cyril Brulebois
2023-01-24 04:40:01 UTC
Package: hw-detect
Severity: important
X-Debbugs-Cc: debian-***@lists.debian.org
Hi,
[debian-arm@ in copy for the question regarding concatenateable
images near the end.]
I'm filing this as important for the time being, but this can be made
serious if required. Reminder: doing so might or might not block the
12.0 release (it could be usertagged can-defer).
Firmware support has long been a very difficult topic to navigate, for
mere users but also for more advanced users who might try and include
firmware files and/or packages onto external media, then wonder what
they did wrong because that still doesn't work.
With the 2022 General Resolution about non-free firmware, and with
non-free-firmware packages being allowed on official installation
images, Steve and I were hoping[1] to limit ourselves to a much more
streamlined process: use whatever's available on the official image, and
stop pretending to support loading firmware material from elsewhere.
1. https://lists.debian.org/debian-boot/2022/10/msg00044.html
Since then, the received feedback and my own testing triggered mixed
feelings:
- In that thread, some reason was given suggesting to retain the
ability to load firmware packages from external media;
- On IRC, while I was rubberducking firmare support in hw-detect,
the same person mentioned that the current implementation doesn't
suit their needs anyway, due to the way the search for firmware is
performed.
- My own tests show that the “Detect network hardware” step can be
stuck for a long time (e.g. 15+ seconds without anything on the
screen but a title, probably depends on the storage). While I initially
suspected a problem with my ongoing work, it turned out that both
“mountmedia” and “mountmedia driver” calls were responsible for most of
that time's being wasted.
The first call (“mountmedia”):
- tries to find “loose firmware files” under /media as mounted by
mountmedia, to copy them in the installer's context;
- lets those files be propagated to /target later on.
The second call (“mountmedia driver”):
- uses the same check_firmware function that's already called on
/firmware and /cdrom/firmware (where official installation images for
Bookworm will find non-free-firmware packages); this means packages
are unpacked in the installer environment, and queued for later
installation into /target;
- looks under both /media and /media/firmware (with /media having been
mounted by “mountmedia driver”).
Some use cases:
- Users might require some firmware packages that aren't available in
Debian yet;
- Users might want to stash a higher version of a firmware package that
is available in Debian already;
- <yours goes here.>
Some problems:
- Delays for users that have everything on their official installation
images.
- Insufficient lookup (something about stopping at the first partition
that's found or something, but I don't know the specifics, and I'm
not diving into what “mountmedia driver” does or should be doing).
- What happens in the “higher version” case? Currently we do not have
anything to detect/process multiple versions of a package, so we might
end up deploying a firmware package found on the installation image,
then deploying a different version of the same firmware package found
on external media; but then both are queued and which one gets
installed first and which one second depends on the order in which
their filenames come up in the for loop around `in-target dpkg -i`…
If we wanted to support that, we would need more code to only keep the
higher version.
I'm also not sure what our plan for e.g. ARM images (concatenateable
images) would be; I don't think we'll pull any firmware packages during a
d-i build, so they wouldn't end up under [2], but maybe it'd be feasible
to just concatenate some file/image containing non-free-firmware packages
(along with the associated metadata if relevant) as published by debian-cd
or something? Or would those concatenateable images need to load firmware
from external media anyway?
2. https://deb.debian.org/debian/dists/bookworm/main/installer-arm64/current/images/netboot/SD-card-images/
In any case, we could probably try and accommodate architecture-specific
issues… by wrapping required calls inside proper conditions, so that
special needs (on release archs or non-release archs) don't negatively
impact regular users.
In the meanwhile, I'll disable both mountmedia calls in the upcoming
upload:
- to reduce delays when official installation images are just
self-sufficient (that's why we had that GR in the first place!);
- to gather feedback/complaints from users for which those calls are
actually important, so that we can better assess the needs, and the
possible shortcomings in the historical implementation.
Cheers,
Severity: important
X-Debbugs-Cc: debian-***@lists.debian.org
Hi,
[debian-arm@ in copy for the question regarding concatenateable
images near the end.]
I'm filing this as important for the time being, but this can be made
serious if required. Reminder: doing so might or might not block the
12.0 release (it could be usertagged can-defer).
Firmware support has long been a very difficult topic to navigate, for
mere users but also for more advanced users who might try and include
firmware files and/or packages onto external media, then wonder what
they did wrong because that still doesn't work.
With the 2022 General Resolution about non-free firmware, and with
non-free-firmware packages being allowed on official installation
images, Steve and I were hoping[1] to limit ourselves to a much more
streamlined process: use whatever's available on the official image, and
stop pretending to support loading firmware material from elsewhere.
1. https://lists.debian.org/debian-boot/2022/10/msg00044.html
Since then, the received feedback and my own testing triggered mixed
feelings:
- In that thread, some reason was given suggesting to retain the
ability to load firmware packages from external media;
- On IRC, while I was rubberducking firmare support in hw-detect,
the same person mentioned that the current implementation doesn't
suit their needs anyway, due to the way the search for firmware is
performed.
- My own tests show that the “Detect network hardware” step can be
stuck for a long time (e.g. 15+ seconds without anything on the
screen but a title, probably depends on the storage). While I initially
suspected a problem with my ongoing work, it turned out that both
“mountmedia” and “mountmedia driver” calls were responsible for most of
that time's being wasted.
The first call (“mountmedia”):
- tries to find “loose firmware files” under /media as mounted by
mountmedia, to copy them in the installer's context;
- lets those files be propagated to /target later on.
The second call (“mountmedia driver”):
- uses the same check_firmware function that's already called on
/firmware and /cdrom/firmware (where official installation images for
Bookworm will find non-free-firmware packages); this means packages
are unpacked in the installer environment, and queued for later
installation into /target;
- looks under both /media and /media/firmware (with /media having been
mounted by “mountmedia driver”).
Some use cases:
- Users might require some firmware packages that aren't available in
Debian yet;
- Users might want to stash a higher version of a firmware package that
is available in Debian already;
- <yours goes here.>
Some problems:
- Delays for users that have everything on their official installation
images.
- Insufficient lookup (something about stopping at the first partition
that's found or something, but I don't know the specifics, and I'm
not diving into what “mountmedia driver” does or should be doing).
- What happens in the “higher version” case? Currently we do not have
anything to detect/process multiple versions of a package, so we might
end up deploying a firmware package found on the installation image,
then deploying a different version of the same firmware package found
on external media; but then both are queued and which one gets
installed first and which one second depends on the order in which
their filenames come up in the for loop around `in-target dpkg -i`…
If we wanted to support that, we would need more code to only keep the
higher version.
I'm also not sure what our plan for e.g. ARM images (concatenateable
images) would be; I don't think we'll pull any firmware packages during a
d-i build, so they wouldn't end up under [2], but maybe it'd be feasible
to just concatenate some file/image containing non-free-firmware packages
(along with the associated metadata if relevant) as published by debian-cd
or something? Or would those concatenateable images need to load firmware
from external media anyway?
2. https://deb.debian.org/debian/dists/bookworm/main/installer-arm64/current/images/netboot/SD-card-images/
In any case, we could probably try and accommodate architecture-specific
issues… by wrapping required calls inside proper conditions, so that
special needs (on release archs or non-release archs) don't negatively
impact regular users.
In the meanwhile, I'll disable both mountmedia calls in the upcoming
upload:
- to reduce delays when official installation images are just
self-sufficient (that's why we had that GR in the first place!);
- to gather feedback/complaints from users for which those calls are
actually important, so that we can better assess the needs, and the
possible shortcomings in the historical implementation.
Cheers,
--
Cyril Brulebois (***@debian.org) <https://debamax.com/>
D-I release manager -- Release
Cyril Brulebois (***@debian.org) <https://debamax.com/>
D-I release manager -- Release