Hi,
I have looked into the CMA setting issue a bit. This is what I have found so far.
The rpi4 needs CMA to be in ZONE_DMA (lower 1GB of memory) as this is the only area that the peripherals on the rpi4 can address.
The DT sets the allowed range to allocate the CMA from ( arch/arm/boot/dts/bcm2711.dtsi#L869 https://github.com/torvalds/linux/blob/master/arch/arm/boot/dts/bcm2711.dtsi#L869), but it seems to not work here. What does work is instead to set the offset manually. I replaced "cma=256MB" with "cma=256M@704M" and then it boots. Note that it has to be 256M instead of 256MB.
Removing the cma option on the command line was known as a workaround. Without that we would fall back to the build config of 64MB cma which was located at offset 0x38000000. This left 64MB at the end of ZONE_DMA, and I chose offset 704M so that those 64MB would still be free. Not sure if that is needed or not. The crashkernel needs to be in ZONE_DMA as well but it seems to be set to 0 size.
I have tested on 5.7 rc2 from rawhide.
This probably belongs in a bug report. What would be the correct place to file that? From what I can tell upstream has been tested with cma settings without problems (as long as the requested CMA size can fit in ZONE_DMA). From that it seems like fedora-specific issue. Not sure though.
Cheers, Thomas
Hi,
Thomas A's advice works on my RPi 4B and is prerequisite for using the Workstation 32 edition on it. Here is my cookbook:
https://dl.fedoraproject.org/pub/fedora-secondary/development/32/Workstation...
xzcat ./ | sudo dd status=progress bs=1M of=/dev/sdx && sync
EFI/fedora/grub.cfg EFI/fedora/grubenv cma=256M@704M
I end up with an apparenlty fully functional Fedora and after deactivation of animation effects using Tweaks, Gnome is not (very) sluggish and works fine for me.
Cheers, Thomas B
On Thu, Apr 23, 2020 at 5:46 PM Thomas H.P. Andersen phomes@gmail.com wrote:
Hi,
I have looked into the CMA setting issue a bit. This is what I have found so far.
The rpi4 needs CMA to be in ZONE_DMA (lower 1GB of memory) as this is the only area that the peripherals on the rpi4 can address.
The DT sets the allowed range to allocate the CMA from ( arch/arm/boot/dts/bcm2711.dtsi#L869 https://github.com/torvalds/linux/blob/master/arch/arm/boot/dts/bcm2711.dtsi#L869), but it seems to not work here. What does work is instead to set the offset manually. I replaced "cma=256MB" with "cma=256M@704M" and then it boots. Note that it has to be 256M instead of 256MB.
Removing the cma option on the command line was known as a workaround. Without that we would fall back to the build config of 64MB cma which was located at offset 0x38000000. This left 64MB at the end of ZONE_DMA, and I chose offset 704M so that those 64MB would still be free. Not sure if that is needed or not. The crashkernel needs to be in ZONE_DMA as well but it seems to be set to 0 size.
I have tested on 5.7 rc2 from rawhide.
This probably belongs in a bug report. What would be the correct place to file that? From what I can tell upstream has been tested with cma settings without problems (as long as the requested CMA size can fit in ZONE_DMA). From that it seems like fedora-specific issue. Not sure though.
Cheers, Thomas _______________________________________________ arm mailing list -- arm@lists.fedoraproject.org To unsubscribe send an email to arm-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/arm@lists.fedoraproject.org
On Mon, Apr 27, 2020 at 6:49 PM ng0177@gmail.com wrote:
Hi,
Thomas A's advice works on my RPi 4B and is prerequisite for using the Workstation 32 edition on it.
Thanks for testing it. It is still just a work-around and we need to find a real fix for this. It does hint to what real problem could be though. Debugging further is difficult as the usb controller is one of the things that break. I have a ordered a cable for serial connection and will continue to look into it when it arrives.
Here is my cookbook:
https://dl.fedoraproject.org/pub/fedora-secondary/development/32/Workstation...
xzcat ./ | sudo dd status=progress bs=1M of=/dev/sdx && sync
Consider taking a look at arm-image-installer. I does mostly the same but has some convenient extra features.
EFI/fedora/grub.cfg EFI/fedora/grubenv cma=256M@704M
I end up with an apparenlty fully functional Fedora and after deactivation of animation effects using Tweaks, Gnome is not (very) sluggish and works fine for me.
Cheers, Thomas B
On Thu, Apr 23, 2020 at 5:46 PM Thomas H.P. Andersen phomes@gmail.com wrote:
Hi,
I have looked into the CMA setting issue a bit. This is what I have found so far.
The rpi4 needs CMA to be in ZONE_DMA (lower 1GB of memory) as this is the only area that the peripherals on the rpi4 can address.
The DT sets the allowed range to allocate the CMA from ( arch/arm/boot/dts/bcm2711.dtsi#L869 https://github.com/torvalds/linux/blob/master/arch/arm/boot/dts/bcm2711.dtsi#L869), but it seems to not work here. What does work is instead to set the offset manually. I replaced "cma=256MB" with "cma=256M@704M" and then it boots. Note that it has to be 256M instead of 256MB.
Removing the cma option on the command line was known as a workaround. Without that we would fall back to the build config of 64MB cma which was located at offset 0x38000000. This left 64MB at the end of ZONE_DMA, and I chose offset 704M so that those 64MB would still be free. Not sure if that is needed or not. The crashkernel needs to be in ZONE_DMA as well but it seems to be set to 0 size.
I have tested on 5.7 rc2 from rawhide.
This probably belongs in a bug report. What would be the correct place to file that? From what I can tell upstream has been tested with cma settings without problems (as long as the requested CMA size can fit in ZONE_DMA). From that it seems like fedora-specific issue. Not sure though.
Cheers, Thomas _______________________________________________ arm mailing list -- arm@lists.fedoraproject.org To unsubscribe send an email to arm-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/arm@lists.fedoraproject.org
Hi,
"workaround" is a slight understatement. The RPi works and performs very well with it on Fedora 32!
Regards, Thomas B
On Tue, Apr 28, 2020 at 2:54 PM Thomas H.P. Andersen phomes@gmail.com wrote:
On Mon, Apr 27, 2020 at 6:49 PM ng0177@gmail.com wrote:
Hi,
Thomas A's advice works on my RPi 4B and is prerequisite for using the Workstation 32 edition on it.
Thanks for testing it. It is still just a work-around and we need to find a real fix for this. It does hint to what real problem could be though. Debugging further is difficult as the usb controller is one of the things that break. I have a ordered a cable for serial connection and will continue to look into it when it arrives.
Here is my cookbook:
https://dl.fedoraproject.org/pub/fedora-secondary/development/32/Workstation...
xzcat ./ | sudo dd status=progress bs=1M of=/dev/sdx && sync
Consider taking a look at arm-image-installer. I does mostly the same but has some convenient extra features.
EFI/fedora/grub.cfg EFI/fedora/grubenv cma=256M@704M
I end up with an apparenlty fully functional Fedora and after deactivation of animation effects using Tweaks, Gnome is not (very) sluggish and works fine for me.
Cheers, Thomas B
On Thu, Apr 23, 2020 at 5:46 PM Thomas H.P. Andersen phomes@gmail.com wrote:
Hi,
I have looked into the CMA setting issue a bit. This is what I have found so far.
The rpi4 needs CMA to be in ZONE_DMA (lower 1GB of memory) as this is the only area that the peripherals on the rpi4 can address.
The DT sets the allowed range to allocate the CMA from ( arch/arm/boot/dts/bcm2711.dtsi#L869 https://github.com/torvalds/linux/blob/master/arch/arm/boot/dts/bcm2711.dtsi#L869), but it seems to not work here. What does work is instead to set the offset manually. I replaced "cma=256MB" with "cma=256M@704M" and then it boots. Note that it has to be 256M instead of 256MB.
Removing the cma option on the command line was known as a workaround. Without that we would fall back to the build config of 64MB cma which was located at offset 0x38000000. This left 64MB at the end of ZONE_DMA, and I chose offset 704M so that those 64MB would still be free. Not sure if that is needed or not. The crashkernel needs to be in ZONE_DMA as well but it seems to be set to 0 size.
I have tested on 5.7 rc2 from rawhide.
This probably belongs in a bug report. What would be the correct place to file that? From what I can tell upstream has been tested with cma settings without problems (as long as the requested CMA size can fit in ZONE_DMA). From that it seems like fedora-specific issue. Not sure though.
Cheers, Thomas _______________________________________________ arm mailing list -- arm@lists.fedoraproject.org To unsubscribe send an email to arm-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/arm@lists.fedoraproject.org
"workaround" is a slight understatement. The RPi works and performs very well with it on Fedora 32!
Please keep the thread on topic. If you think it performs "very well" you've either got very low expectations or will be thrilled when it's got accelerated graphics. There's a LONG way to go!
On Tue, Apr 28, 2020 at 2:54 PM Thomas H.P. Andersen phomes@gmail.com wrote:
On Mon, Apr 27, 2020 at 6:49 PM ng0177@gmail.com wrote:
Hi,
Thomas A's advice works on my RPi 4B and is prerequisite for using the Workstation 32 edition on it.
Thanks for testing it. It is still just a work-around and we need to find a real fix for this. It does hint to what real problem could be though. Debugging further is difficult as the usb controller is one of the things that break. I have a ordered a cable for serial connection and will continue to look into it when it arrives.
Here is my cookbook:
https://dl.fedoraproject.org/pub/fedora-secondary/development/32/Workstation...
xzcat ./ | sudo dd status=progress bs=1M of=/dev/sdx && sync
Consider taking a look at arm-image-installer. I does mostly the same but has some convenient extra features.
EFI/fedora/grub.cfg EFI/fedora/grubenv cma=256M@704M
I end up with an apparenlty fully functional Fedora and after deactivation of animation effects using Tweaks, Gnome is not (very) sluggish and works fine for me.
Cheers, Thomas B
On Thu, Apr 23, 2020 at 5:46 PM Thomas H.P. Andersen phomes@gmail.com wrote:
Hi,
I have looked into the CMA setting issue a bit. This is what I have found so far.
The rpi4 needs CMA to be in ZONE_DMA (lower 1GB of memory) as this is the only area that the peripherals on the rpi4 can address.
The DT sets the allowed range to allocate the CMA from (arch/arm/boot/dts/bcm2711.dtsi#L869), but it seems to not work here. What does work is instead to set the offset manually. I replaced "cma=256MB" with "cma=256M@704M" and then it boots. Note that it has to be 256M instead of 256MB.
Removing the cma option on the command line was known as a workaround. Without that we would fall back to the build config of 64MB cma which was located at offset 0x38000000. This left 64MB at the end of ZONE_DMA, and I chose offset 704M so that those 64MB would still be free. Not sure if that is needed or not. The crashkernel needs to be in ZONE_DMA as well but it seems to be set to 0 size.
I have tested on 5.7 rc2 from rawhide.
This probably belongs in a bug report. What would be the correct place to file that? From what I can tell upstream has been tested with cma settings without problems (as long as the requested CMA size can fit in ZONE_DMA). From that it seems like fedora-specific issue. Not sure though.
Cheers, Thomas _______________________________________________ arm mailing list -- arm@lists.fedoraproject.org To unsubscribe send an email to arm-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/arm@lists.fedoraproject.org
arm mailing list -- arm@lists.fedoraproject.org To unsubscribe send an email to arm-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/arm@lists.fedoraproject.org
Hi Thomas,
I have looked into the CMA setting issue a bit. This is what I have found so far.
The rpi4 needs CMA to be in ZONE_DMA (lower 1GB of memory) as this is the only area that the peripherals on the rpi4 can address.
The DT sets the allowed range to allocate the CMA from (arch/arm/boot/dts/bcm2711.dtsi#L869), but it seems to not work here. What does work is instead to set the offset manually. I replaced "cma=256MB" with "cma=256M@704M" and then it boots. Note that it has to be 256M instead of 256MB.
Right, because of this it may be able to be set in config.txt, I seem to remember seeing this somewhere but as we don't support accelerated graphics on the RPi4 I've not looked. I don't believe the unaccelerated graphics uses the CMA so for the current situation you may be able to drop it.
If it's an option to set it in config.txt we need to work out if this is a general option that works for all the rpi models or if it's explicitly for the RPi4, if the later we really need to report and get the bug fixed because we aim to produce generic images which "just work" across all the rpi devices, anything else just makes it a support nightmare for people like myself that attempt to support it and it would be less work just not to support the RPi4 at all TBH.
Removing the cma option on the command line was known as a workaround. Without that we would fall back to the build config of 64MB cma which was located at offset 0x38000000. This left 64MB at the end of ZONE_DMA, and I chose offset 704M so that those 64MB would still be free. Not sure if that is needed or not. The crashkernel needs to be in ZONE_DMA as well but it seems to be set to 0 size.
I have tested on 5.7 rc2 from rawhide.
This probably belongs in a bug report. What would be the correct place to file that? From what I can tell upstream has been tested with cma settings without problems (as long as the requested CMA size can fit in ZONE_DMA). From that it seems like fedora-specific issue. Not sure though.
Not sure what you mean by "upstream" here, we use an almost pure upstream Linus kernel, if you mean the "Raspberry Pi Foundation" and their kernel, that's a downstream fork of Linus's kernel. They also have a lot of other patches and use a different desktop, GNOME from experience and working with them then RPi upstream GPU maintainer we worked out GNOME needed the 256Mb allocation but desktops like XFCE use a lot less (~192Mb if memory serves) and Raspbian uses a light desktop so I suspect they allocate a lot less.
Peter
On Tue, Apr 28, 2020 at 3:20 PM Peter Robinson pbrobinson@gmail.com wrote:
Hi Thomas,
I have looked into the CMA setting issue a bit. This is what I have
found so far.
The rpi4 needs CMA to be in ZONE_DMA (lower 1GB of memory) as this is
the only area that the peripherals on the rpi4 can address.
The DT sets the allowed range to allocate the CMA from
(arch/arm/boot/dts/bcm2711.dtsi#L869), but it seems to not work here. What does work is instead to set the offset manually. I replaced "cma=256MB" with "cma=256M@704M" and then it boots. Note that it has to be 256M instead of 256MB.
Right, because of this it may be able to be set in config.txt, I seem to remember seeing this somewhere but as we don't support accelerated graphics on the RPi4 I've not looked. I don't believe the unaccelerated graphics uses the CMA so for the current situation you may be able to drop it.
If it's an option to set it in config.txt we need to work out if this is a general option that works for all the rpi models or if it's explicitly for the RPi4, if the later we really need to report and get the bug fixed because we aim to produce generic images which "just work" across all the rpi devices, anything else just makes it a support nightmare for people like myself that attempt to support it and it would be less work just not to support the RPi4 at all TBH.
Removing the cma option on the command line was known as a workaround.
Without that we would fall back to the build config of 64MB cma which was located at offset 0x38000000. This left 64MB at the end of ZONE_DMA, and I chose offset 704M so that those 64MB would still be free. Not sure if that is needed or not. The crashkernel needs to be in ZONE_DMA as well but it seems to be set to 0 size.
I have tested on 5.7 rc2 from rawhide.
This probably belongs in a bug report. What would be the correct place
to file that? From what I can tell upstream has been tested with cma settings without problems (as long as the requested CMA size can fit in ZONE_DMA). From that it seems like fedora-specific issue. Not sure though.
Not sure what you mean by "upstream" here, we use an almost pure upstream Linus kernel, if you mean the "Raspberry Pi Foundation" and their kernel, that's a downstream fork of Linus's kernel. They also have a lot of other patches and use a different desktop, GNOME from experience and working with them then RPi upstream GPU maintainer we worked out GNOME needed the 256Mb allocation but desktops like XFCE use a lot less (~192Mb if memory serves) and Raspbian uses a light desktop so I suspect they allocate a lot less.
I meant upstream as in linus tree. I noticed some comments in the review of rpi4 ustreaming patches where various cma sizes were tested. Thus I suspected that it could be related to downstream patches. If we do not carry any relevant then perhaps the issue could related to setting the cma both in config.txt and on the commandline?
Raspbian uses 256MB via kernel commandline and the config.txt in /boot does not have any setting for cma. The cma starts at 0x1ec00000 so in the lower 1gb. But it is a 4.19 kernel + patches so not really useful to look at for this specific issue.
Peter
Hi Thomas,
I have looked into the CMA setting issue a bit. This is what I have found so far.
The rpi4 needs CMA to be in ZONE_DMA (lower 1GB of memory) as this is the only area that the peripherals on the rpi4 can address.
The DT sets the allowed range to allocate the CMA from (arch/arm/boot/dts/bcm2711.dtsi#L869), but it seems to not work here. What does work is instead to set the offset manually. I replaced "cma=256MB" with "cma=256M@704M" and then it boots. Note that it has to be 256M instead of 256MB.
Right, because of this it may be able to be set in config.txt, I seem to remember seeing this somewhere but as we don't support accelerated graphics on the RPi4 I've not looked. I don't believe the unaccelerated graphics uses the CMA so for the current situation you may be able to drop it.
If it's an option to set it in config.txt we need to work out if this is a general option that works for all the rpi models or if it's explicitly for the RPi4, if the later we really need to report and get the bug fixed because we aim to produce generic images which "just work" across all the rpi devices, anything else just makes it a support nightmare for people like myself that attempt to support it and it would be less work just not to support the RPi4 at all TBH.
Removing the cma option on the command line was known as a workaround. Without that we would fall back to the build config of 64MB cma which was located at offset 0x38000000. This left 64MB at the end of ZONE_DMA, and I chose offset 704M so that those 64MB would still be free. Not sure if that is needed or not. The crashkernel needs to be in ZONE_DMA as well but it seems to be set to 0 size.
I have tested on 5.7 rc2 from rawhide.
This probably belongs in a bug report. What would be the correct place to file that? From what I can tell upstream has been tested with cma settings without problems (as long as the requested CMA size can fit in ZONE_DMA). From that it seems like fedora-specific issue. Not sure though.
Not sure what you mean by "upstream" here, we use an almost pure upstream Linus kernel, if you mean the "Raspberry Pi Foundation" and their kernel, that's a downstream fork of Linus's kernel. They also have a lot of other patches and use a different desktop, GNOME from experience and working with them then RPi upstream GPU maintainer we worked out GNOME needed the 256Mb allocation but desktops like XFCE use a lot less (~192Mb if memory serves) and Raspbian uses a light desktop so I suspect they allocate a lot less.
I meant upstream as in linus tree. I noticed some comments in the review of rpi4 ustreaming patches where various cma sizes were tested. Thus I suspected that it could be related to downstream patches. If we do not carry any relevant then perhaps the issue could related to setting the cma both in config.txt and on the commandline?
Raspbian uses 256MB via kernel commandline and the config.txt in /boot does not have any setting for cma. The cma starts at 0x1ec00000 so in the lower 1gb. But it is a 4.19 kernel + patches so not really useful to look at for this specific issue.
Just as a follow up to this it looks like Raspbian has just (finally?) rebased to the 5.4 LTS kernel [1] in their downstream kernel and looking through the change they've done what I thought would be needed and provided a means of dealing with CMA via dt-overlays, I wasn't sure whether they would have done it via a config.txt cma=XX or a dt-overlay option, I've literally just looked at the diff but it looks like for F-33 we can investigate dealing with this in the config.txt rather than kernel command line.
I've literally just looked at the diff here and won't have time to investigate this until later this month, which actually is probably just fine to give a few more firmware releases to settle the rebase down, but if someone wants to start looking further do reach out.
Peter
[1] https://github.com/raspberrypi/firmware/commit/7eff9f6774bb43bfd61e749a0b45f... [2] https://github.com/raspberrypi/firmware/commit/7eff9f6774bb43bfd61e749a0b45f...
On Thu, May 7, 2020 at 11:23 AM Peter Robinson pbrobinson@gmail.com wrote:
Hi Thomas,
I have looked into the CMA setting issue a bit. This is what I have
found so far.
The rpi4 needs CMA to be in ZONE_DMA (lower 1GB of memory) as this is
the only area that the peripherals on the rpi4 can address.
The DT sets the allowed range to allocate the CMA from
(arch/arm/boot/dts/bcm2711.dtsi#L869), but it seems to not work here. What does work is instead to set the offset manually. I replaced "cma=256MB" with "cma=256M@704M" and then it boots. Note that it has to be 256M instead of 256MB.
Right, because of this it may be able to be set in config.txt, I seem to remember seeing this somewhere but as we don't support accelerated graphics on the RPi4 I've not looked. I don't believe the unaccelerated graphics uses the CMA so for the current situation you may be able to drop it.
If it's an option to set it in config.txt we need to work out if this is a general option that works for all the rpi models or if it's explicitly for the RPi4, if the later we really need to report and get the bug fixed because we aim to produce generic images which "just work" across all the rpi devices, anything else just makes it a support nightmare for people like myself that attempt to support it and it would be less work just not to support the RPi4 at all TBH.
Removing the cma option on the command line was known as a
workaround. Without that we would fall back to the build config of 64MB cma which was located at offset 0x38000000. This left 64MB at the end of ZONE_DMA, and I chose offset 704M so that those 64MB would still be free. Not sure if that is needed or not. The crashkernel needs to be in ZONE_DMA as well but it seems to be set to 0 size.
I have tested on 5.7 rc2 from rawhide.
This probably belongs in a bug report. What would be the correct
place to file that? From what I can tell upstream has been tested with cma settings without problems (as long as the requested CMA size can fit in ZONE_DMA). From that it seems like fedora-specific issue. Not sure though.
Not sure what you mean by "upstream" here, we use an almost pure upstream Linus kernel, if you mean the "Raspberry Pi Foundation" and their kernel, that's a downstream fork of Linus's kernel. They also have a lot of other patches and use a different desktop, GNOME from experience and working with them then RPi upstream GPU maintainer we worked out GNOME needed the 256Mb allocation but desktops like XFCE use a lot less (~192Mb if memory serves) and Raspbian uses a light desktop so I suspect they allocate a lot less.
I meant upstream as in linus tree. I noticed some comments in the review
of rpi4 ustreaming patches where various cma sizes were tested. Thus I suspected that it could be related to downstream patches. If we do not carry any relevant then perhaps the issue could related to setting the cma both in config.txt and on the commandline?
Raspbian uses 256MB via kernel commandline and the config.txt in /boot
does not have any setting for cma. The cma starts at 0x1ec00000 so in the lower 1gb. But it is a 4.19 kernel + patches so not really useful to look at for this specific issue.
Just as a follow up to this it looks like Raspbian has just (finally?) rebased to the 5.4 LTS kernel [1] in their downstream kernel and looking through the change they've done what I thought would be needed and provided a means of dealing with CMA via dt-overlays, I wasn't sure whether they would have done it via a config.txt cma=XX or a dt-overlay option, I've literally just looked at the diff but it looks like for F-33 we can investigate dealing with this in the config.txt rather than kernel command line.
I've literally just looked at the diff here and won't have time to investigate this until later this month, which actually is probably just fine to give a few more firmware releases to settle the rebase down, but if someone wants to start looking further do reach out.
That is quite a commit to dive into :) I will take a look at it.
For info here is what I have found since my last email:
I attached the serial console to check what happens when we hang at boot with the cma setting. We allocate the 256 MB at 0xEC000000. That again leaves 64MB at the end but at the end of the 4GB this time. With [3] the logic is that users should have full control of the cma allocation when using the setting on the commandline. The idea makes sense but it is unfortunate that specifying only a cma size also leads to ignoring the valid allocation range specified in dt for rpi4. The commit was introduced in 5.6 and thus is not in the raspberrypi kernel. I guess this will still be a problem then if we continue to set the cma on the commandline?
[3] https://github.com/torvalds/linux/commit/8c8c5a4994a306c217fd061cbfc5903399f...
Peter
[1] https://github.com/raspberrypi/firmware/commit/7eff9f6774bb43bfd61e749a0b45f... [2] https://github.com/raspberrypi/firmware/commit/7eff9f6774bb43bfd61e749a0b45f...
I have looked into the CMA setting issue a bit. This is what I have found so far.
The rpi4 needs CMA to be in ZONE_DMA (lower 1GB of memory) as this is the only area that the peripherals on the rpi4 can address.
The DT sets the allowed range to allocate the CMA from (arch/arm/boot/dts/bcm2711.dtsi#L869), but it seems to not work here. What does work is instead to set the offset manually. I replaced "cma=256MB" with "cma=256M@704M" and then it boots. Note that it has to be 256M instead of 256MB.
Right, because of this it may be able to be set in config.txt, I seem to remember seeing this somewhere but as we don't support accelerated graphics on the RPi4 I've not looked. I don't believe the unaccelerated graphics uses the CMA so for the current situation you may be able to drop it.
If it's an option to set it in config.txt we need to work out if this is a general option that works for all the rpi models or if it's explicitly for the RPi4, if the later we really need to report and get the bug fixed because we aim to produce generic images which "just work" across all the rpi devices, anything else just makes it a support nightmare for people like myself that attempt to support it and it would be less work just not to support the RPi4 at all TBH.
Removing the cma option on the command line was known as a workaround. Without that we would fall back to the build config of 64MB cma which was located at offset 0x38000000. This left 64MB at the end of ZONE_DMA, and I chose offset 704M so that those 64MB would still be free. Not sure if that is needed or not. The crashkernel needs to be in ZONE_DMA as well but it seems to be set to 0 size.
I have tested on 5.7 rc2 from rawhide.
This probably belongs in a bug report. What would be the correct place to file that? From what I can tell upstream has been tested with cma settings without problems (as long as the requested CMA size can fit in ZONE_DMA). From that it seems like fedora-specific issue. Not sure though.
Not sure what you mean by "upstream" here, we use an almost pure upstream Linus kernel, if you mean the "Raspberry Pi Foundation" and their kernel, that's a downstream fork of Linus's kernel. They also have a lot of other patches and use a different desktop, GNOME from experience and working with them then RPi upstream GPU maintainer we worked out GNOME needed the 256Mb allocation but desktops like XFCE use a lot less (~192Mb if memory serves) and Raspbian uses a light desktop so I suspect they allocate a lot less.
I meant upstream as in linus tree. I noticed some comments in the review of rpi4 ustreaming patches where various cma sizes were tested. Thus I suspected that it could be related to downstream patches. If we do not carry any relevant then perhaps the issue could related to setting the cma both in config.txt and on the commandline?
Raspbian uses 256MB via kernel commandline and the config.txt in /boot does not have any setting for cma. The cma starts at 0x1ec00000 so in the lower 1gb. But it is a 4.19 kernel + patches so not really useful to look at for this specific issue.
Just as a follow up to this it looks like Raspbian has just (finally?) rebased to the 5.4 LTS kernel [1] in their downstream kernel and looking through the change they've done what I thought would be needed and provided a means of dealing with CMA via dt-overlays, I wasn't sure whether they would have done it via a config.txt cma=XX or a dt-overlay option, I've literally just looked at the diff but it looks like for F-33 we can investigate dealing with this in the config.txt rather than kernel command line.
I've literally just looked at the diff here and won't have time to investigate this until later this month, which actually is probably just fine to give a few more firmware releases to settle the rebase down, but if someone wants to start looking further do reach out.
That is quite a commit to dive into :) I will take a look at it.
For info here is what I have found since my last email:
I attached the serial console to check what happens when we hang at boot with the cma setting. We allocate the 256 MB at 0xEC000000. That again leaves 64MB at the end but at the end of the 4GB this time. With [3] the logic is that users should have full control of the cma allocation when using the setting on the commandline. The idea makes sense but it is unfortunate that specifying only a cma size also leads to ignoring the valid allocation range specified in dt for rpi4. The commit was introduced in 5.6 and thus is not in the raspberrypi kernel. I guess this will still be a problem then if we continue to set the cma on the commandline?
Hence my explicit mention of the cma DT overlay option they provide and looking at how we can move it from the cmdline to config.txt ;-)
[3] https://github.com/torvalds/linux/commit/8c8c5a4994a306c217fd061cbfc5903399f...
Peter
[1] https://github.com/raspberrypi/firmware/commit/7eff9f6774bb43bfd61e749a0b45f... [2] https://github.com/raspberrypi/firmware/commit/7eff9f6774bb43bfd61e749a0b45f...
On Thu, May 7, 2020 at 4:10 PM Peter Robinson pbrobinson@gmail.com wrote:
I have looked into the CMA setting issue a bit. This is what I
have found so far.
The rpi4 needs CMA to be in ZONE_DMA (lower 1GB of memory) as this
is the only area that the peripherals on the rpi4 can address.
The DT sets the allowed range to allocate the CMA from
(arch/arm/boot/dts/bcm2711.dtsi#L869), but it seems to not work here. What does work is instead to set the offset manually. I replaced "cma=256MB" with "cma=256M@704M" and then it boots. Note that it has to be 256M instead of 256MB.
Right, because of this it may be able to be set in config.txt, I seem to remember seeing this somewhere but as we don't support accelerated graphics on the RPi4 I've not looked. I don't believe the unaccelerated graphics uses the CMA so for the current situation you may be able to drop it.
If it's an option to set it in config.txt we need to work out if this is a general option that works for all the rpi models or if it's explicitly for the RPi4, if the later we really need to report and
get
the bug fixed because we aim to produce generic images which "just work" across all the rpi devices, anything else just makes it a support nightmare for people like myself that attempt to support it and it would be less work just not to support the RPi4 at all TBH.
Removing the cma option on the command line was known as a
workaround. Without that we would fall back to the build config of 64MB cma which was located at offset 0x38000000. This left 64MB at the end of ZONE_DMA, and I chose offset 704M so that those 64MB would still be free. Not sure if that is needed or not. The crashkernel needs to be in ZONE_DMA as well but it seems to be set to 0 size.
I have tested on 5.7 rc2 from rawhide.
This probably belongs in a bug report. What would be the correct
place to file that? From what I can tell upstream has been tested with cma settings without problems (as long as the requested CMA size can fit in ZONE_DMA). From that it seems like fedora-specific issue. Not sure though.
Not sure what you mean by "upstream" here, we use an almost pure upstream Linus kernel, if you mean the "Raspberry Pi Foundation" and their kernel, that's a downstream fork of Linus's kernel. They also have a lot of other patches and use a different desktop, GNOME from experience and working with them then RPi upstream GPU maintainer we worked out GNOME needed the 256Mb allocation but desktops like XFCE use a lot less (~192Mb if memory serves) and Raspbian uses a light desktop so I suspect they allocate a lot less.
I meant upstream as in linus tree. I noticed some comments in the
review of rpi4 ustreaming patches where various cma sizes were tested. Thus I suspected that it could be related to downstream patches. If we do not carry any relevant then perhaps the issue could related to setting the cma both in config.txt and on the commandline?
Raspbian uses 256MB via kernel commandline and the config.txt in
/boot does not have any setting for cma. The cma starts at 0x1ec00000 so in the lower 1gb. But it is a 4.19 kernel + patches so not really useful to look at for this specific issue.
Just as a follow up to this it looks like Raspbian has just (finally?) rebased to the 5.4 LTS kernel [1] in their downstream kernel and looking through the change they've done what I thought would be needed and provided a means of dealing with CMA via dt-overlays, I wasn't sure whether they would have done it via a config.txt cma=XX or a dt-overlay option, I've literally just looked at the diff but it looks like for F-33 we can investigate dealing with this in the config.txt rather than kernel command line.
I've literally just looked at the diff here and won't have time to investigate this until later this month, which actually is probably just fine to give a few more firmware releases to settle the rebase down, but if someone wants to start looking further do reach out.
That is quite a commit to dive into :) I will take a look at it.
For info here is what I have found since my last email:
I attached the serial console to check what happens when we hang at boot
with the cma setting. We allocate the 256 MB at 0xEC000000. That again leaves 64MB at the end but at the end of the 4GB this time.
With [3] the logic is that users should have full control of the cma
allocation when using the setting on the commandline. The idea makes sense but it is unfortunate that specifying only a cma size also leads to ignoring the valid allocation range specified in dt for rpi4. The commit was introduced in 5.6 and thus is not in the raspberrypi kernel. I guess this will still be a problem then if we continue to set the cma on the commandline?
Hence my explicit mention of the cma DT overlay option they provide and looking at how we can move it from the cmdline to config.txt ;-)
Yep. I was not disagreeing on that :) I just wanted to write what I had found in case it was valuable or interesting for anyone else looking into this.
Two questions.
Is the cma=256MB default on the arm image only for the sake of the raspberry pi's? Can we just drop if this config.txt options works?
Is there an easy way (a script?) to get the firmware from master to my sd card or should I copy the files manually? Will that even work? I can test the new options over the weekend I think.
[3]
https://github.com/torvalds/linux/commit/8c8c5a4994a306c217fd061cbfc5903399f...
Peter
[1]
https://github.com/raspberrypi/firmware/commit/7eff9f6774bb43bfd61e749a0b45f...
[2]
https://github.com/raspberrypi/firmware/commit/7eff9f6774bb43bfd61e749a0b45f...
Further to those Fedora32 efforts described as below. I have been quite happily using the
EFI/fedora/grub.cfg EFI/fedora/grubenv cma=256M@704M
settings on my RPi4 w/ 4GB. Now that I purchased a new RPi4 w/ 8GB they seem not to work any more (I get onto Gnome Desktop but moving the move blanks the screen) because I think that this space needs to be reserved at another address.
Any advice? Thanks!
Regards, Thomas B
---------- Forwarded message --------- From: Thomas H.P. Andersen phomes@gmail.com Date: Thu, May 7, 2020 at 4:03 PM Subject: [fedora-arm] Re: CMA on raspberry pi 4 To: Peter Robinson pbrobinson@gmail.com Cc: Fedora List arm@lists.fedoraproject.org
On Thu, May 7, 2020 at 11:23 AM Peter Robinson pbrobinson@gmail.com wrote:
Hi Thomas,
I have looked into the CMA setting issue a bit. This is what I have
found so far.
The rpi4 needs CMA to be in ZONE_DMA (lower 1GB of memory) as this is
the only area that the peripherals on the rpi4 can address.
The DT sets the allowed range to allocate the CMA from
(arch/arm/boot/dts/bcm2711.dtsi#L869), but it seems to not work here. What does work is instead to set the offset manually. I replaced "cma=256MB" with "cma=256M@704M" and then it boots. Note that it has to be 256M instead of 256MB.
Right, because of this it may be able to be set in config.txt, I seem to remember seeing this somewhere but as we don't support accelerated graphics on the RPi4 I've not looked. I don't believe the unaccelerated graphics uses the CMA so for the current situation you may be able to drop it.
If it's an option to set it in config.txt we need to work out if this is a general option that works for all the rpi models or if it's explicitly for the RPi4, if the later we really need to report and get the bug fixed because we aim to produce generic images which "just work" across all the rpi devices, anything else just makes it a support nightmare for people like myself that attempt to support it and it would be less work just not to support the RPi4 at all TBH.
Removing the cma option on the command line was known as a
workaround. Without that we would fall back to the build config of 64MB cma which was located at offset 0x38000000. This left 64MB at the end of ZONE_DMA, and I chose offset 704M so that those 64MB would still be free. Not sure if that is needed or not. The crashkernel needs to be in ZONE_DMA as well but it seems to be set to 0 size.
I have tested on 5.7 rc2 from rawhide.
This probably belongs in a bug report. What would be the correct
place to file that? From what I can tell upstream has been tested with cma settings without problems (as long as the requested CMA size can fit in ZONE_DMA). From that it seems like fedora-specific issue. Not sure though.
Not sure what you mean by "upstream" here, we use an almost pure upstream Linus kernel, if you mean the "Raspberry Pi Foundation" and their kernel, that's a downstream fork of Linus's kernel. They also have a lot of other patches and use a different desktop, GNOME from experience and working with them then RPi upstream GPU maintainer we worked out GNOME needed the 256Mb allocation but desktops like XFCE use a lot less (~192Mb if memory serves) and Raspbian uses a light desktop so I suspect they allocate a lot less.
I meant upstream as in linus tree. I noticed some comments in the review
of rpi4 ustreaming patches where various cma sizes were tested. Thus I suspected that it could be related to downstream patches. If we do not carry any relevant then perhaps the issue could related to setting the cma both in config.txt and on the commandline?
Raspbian uses 256MB via kernel commandline and the config.txt in /boot
does not have any setting for cma. The cma starts at 0x1ec00000 so in the lower 1gb. But it is a 4.19 kernel + patches so not really useful to look at for this specific issue.
Just as a follow up to this it looks like Raspbian has just (finally?) rebased to the 5.4 LTS kernel [1] in their downstream kernel and looking through the change they've done what I thought would be needed and provided a means of dealing with CMA via dt-overlays, I wasn't sure whether they would have done it via a config.txt cma=XX or a dt-overlay option, I've literally just looked at the diff but it looks like for F-33 we can investigate dealing with this in the config.txt rather than kernel command line.
I've literally just looked at the diff here and won't have time to investigate this until later this month, which actually is probably just fine to give a few more firmware releases to settle the rebase down, but if someone wants to start looking further do reach out.
That is quite a commit to dive into :) I will take a look at it.
For info here is what I have found since my last email:
I attached the serial console to check what happens when we hang at boot with the cma setting. We allocate the 256 MB at 0xEC000000. That again leaves 64MB at the end but at the end of the 4GB this time. With [3] the logic is that users should have full control of the cma allocation when using the setting on the commandline. The idea makes sense but it is unfortunate that specifying only a cma size also leads to ignoring the valid allocation range specified in dt for rpi4. The commit was introduced in 5.6 and thus is not in the raspberrypi kernel. I guess this will still be a problem then if we continue to set the cma on the commandline?
[3] https://github.com/torvalds/linux/commit/8c8c5a4994a306c217fd061cbfc5903399f...
Peter
[1] https://github.com/raspberrypi/firmware/commit/7eff9f6774bb43bfd61e749a0b45f... [2] https://github.com/raspberrypi/firmware/commit/7eff9f6774bb43bfd61e749a0b45f...
_______________________________________________ arm mailing list -- arm@lists.fedoraproject.org To unsubscribe send an email to arm-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/arm@lists.fedoraproject.org
On Sun, Aug 2, 2020 at 9:20 PM ng0177@gmail.com wrote:
Further to those Fedora32 efforts described as below. I have been quite happily using the
EFI/fedora/grub.cfg EFI/fedora/grubenv cma=256M@704M
settings on my RPi4 w/ 4GB. Now that I purchased a new RPi4 w/ 8GB they seem not to work any more (I get onto Gnome Desktop but moving the move blanks the screen) because I think that this space needs to be reserved at another address.
Any advice? Thanks!
You'll likely need the 5.8 kernel, there's been massive issues around DMA/CMA upstream due to the "architecture" of the rpi4 with 4+gb of RAM, there was a fix that was very nearly reverted in 5.8 but luckily they found the issue at the last minute.
Regards, Thomas B
---------- Forwarded message --------- From: Thomas H.P. Andersen phomes@gmail.com Date: Thu, May 7, 2020 at 4:03 PM Subject: [fedora-arm] Re: CMA on raspberry pi 4 To: Peter Robinson pbrobinson@gmail.com Cc: Fedora List arm@lists.fedoraproject.org
On Thu, May 7, 2020 at 11:23 AM Peter Robinson pbrobinson@gmail.com wrote:
Hi Thomas,
I have looked into the CMA setting issue a bit. This is what I have found so far.
The rpi4 needs CMA to be in ZONE_DMA (lower 1GB of memory) as this is the only area that the peripherals on the rpi4 can address.
The DT sets the allowed range to allocate the CMA from (arch/arm/boot/dts/bcm2711.dtsi#L869), but it seems to not work here. What does work is instead to set the offset manually. I replaced "cma=256MB" with "cma=256M@704M" and then it boots. Note that it has to be 256M instead of 256MB.
Right, because of this it may be able to be set in config.txt, I seem to remember seeing this somewhere but as we don't support accelerated graphics on the RPi4 I've not looked. I don't believe the unaccelerated graphics uses the CMA so for the current situation you may be able to drop it.
If it's an option to set it in config.txt we need to work out if this is a general option that works for all the rpi models or if it's explicitly for the RPi4, if the later we really need to report and get the bug fixed because we aim to produce generic images which "just work" across all the rpi devices, anything else just makes it a support nightmare for people like myself that attempt to support it and it would be less work just not to support the RPi4 at all TBH.
Removing the cma option on the command line was known as a workaround. Without that we would fall back to the build config of 64MB cma which was located at offset 0x38000000. This left 64MB at the end of ZONE_DMA, and I chose offset 704M so that those 64MB would still be free. Not sure if that is needed or not. The crashkernel needs to be in ZONE_DMA as well but it seems to be set to 0 size.
I have tested on 5.7 rc2 from rawhide.
This probably belongs in a bug report. What would be the correct place to file that? From what I can tell upstream has been tested with cma settings without problems (as long as the requested CMA size can fit in ZONE_DMA). From that it seems like fedora-specific issue. Not sure though.
Not sure what you mean by "upstream" here, we use an almost pure upstream Linus kernel, if you mean the "Raspberry Pi Foundation" and their kernel, that's a downstream fork of Linus's kernel. They also have a lot of other patches and use a different desktop, GNOME from experience and working with them then RPi upstream GPU maintainer we worked out GNOME needed the 256Mb allocation but desktops like XFCE use a lot less (~192Mb if memory serves) and Raspbian uses a light desktop so I suspect they allocate a lot less.
I meant upstream as in linus tree. I noticed some comments in the review of rpi4 ustreaming patches where various cma sizes were tested. Thus I suspected that it could be related to downstream patches. If we do not carry any relevant then perhaps the issue could related to setting the cma both in config.txt and on the commandline?
Raspbian uses 256MB via kernel commandline and the config.txt in /boot does not have any setting for cma. The cma starts at 0x1ec00000 so in the lower 1gb. But it is a 4.19 kernel + patches so not really useful to look at for this specific issue.
Just as a follow up to this it looks like Raspbian has just (finally?) rebased to the 5.4 LTS kernel [1] in their downstream kernel and looking through the change they've done what I thought would be needed and provided a means of dealing with CMA via dt-overlays, I wasn't sure whether they would have done it via a config.txt cma=XX or a dt-overlay option, I've literally just looked at the diff but it looks like for F-33 we can investigate dealing with this in the config.txt rather than kernel command line.
I've literally just looked at the diff here and won't have time to investigate this until later this month, which actually is probably just fine to give a few more firmware releases to settle the rebase down, but if someone wants to start looking further do reach out.
That is quite a commit to dive into :) I will take a look at it.
For info here is what I have found since my last email:
I attached the serial console to check what happens when we hang at boot with the cma setting. We allocate the 256 MB at 0xEC000000. That again leaves 64MB at the end but at the end of the 4GB this time. With [3] the logic is that users should have full control of the cma allocation when using the setting on the commandline. The idea makes sense but it is unfortunate that specifying only a cma size also leads to ignoring the valid allocation range specified in dt for rpi4. The commit was introduced in 5.6 and thus is not in the raspberrypi kernel. I guess this will still be a problem then if we continue to set the cma on the commandline?
[3] https://github.com/torvalds/linux/commit/8c8c5a4994a306c217fd061cbfc5903399f...
Peter
[1] https://github.com/raspberrypi/firmware/commit/7eff9f6774bb43bfd61e749a0b45f... [2] https://github.com/raspberrypi/firmware/commit/7eff9f6774bb43bfd61e749a0b45f...
arm mailing list -- arm@lists.fedoraproject.org To unsubscribe send an email to arm-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/arm@lists.fedoraproject.org _______________________________________________ arm mailing list -- arm@lists.fedoraproject.org To unsubscribe send an email to arm-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/arm@lists.fedoraproject.org
I used today's rawhide image from https://dl.fedoraproject.org/pub/fedora/linux/releases/32/Workstation/aarch6... and get to the Welcome screen but then neither USB keyboard nor mouse work and I am stuck on my RPi4/8GB.
cma=256M@704M has no effect.
On Mon, Aug 3, 2020 at 9:28 AM Peter Robinson pbrobinson@gmail.com wrote:
On Sun, Aug 2, 2020 at 9:20 PM ng0177@gmail.com wrote:
Further to those Fedora32 efforts described as below. I have been quite
happily using the
EFI/fedora/grub.cfg EFI/fedora/grubenv cma=256M@704M
settings on my RPi4 w/ 4GB. Now that I purchased a new RPi4 w/ 8GB they
seem not to work any more (I get onto Gnome Desktop but moving the move blanks the screen) because I think that this space needs to be reserved at another address.
Any advice? Thanks!
You'll likely need the 5.8 kernel, there's been massive issues around DMA/CMA upstream due to the "architecture" of the rpi4 with 4+gb of RAM, there was a fix that was very nearly reverted in 5.8 but luckily they found the issue at the last minute.
Regards, Thomas B
---------- Forwarded message --------- From: Thomas H.P. Andersen phomes@gmail.com Date: Thu, May 7, 2020 at 4:03 PM Subject: [fedora-arm] Re: CMA on raspberry pi 4 To: Peter Robinson pbrobinson@gmail.com Cc: Fedora List arm@lists.fedoraproject.org
On Thu, May 7, 2020 at 11:23 AM Peter Robinson pbrobinson@gmail.com
wrote:
Hi Thomas,
I have looked into the CMA setting issue a bit. This is what I
have found so far.
The rpi4 needs CMA to be in ZONE_DMA (lower 1GB of memory) as this
is the only area that the peripherals on the rpi4 can address.
The DT sets the allowed range to allocate the CMA from
(arch/arm/boot/dts/bcm2711.dtsi#L869), but it seems to not work here. What does work is instead to set the offset manually. I replaced "cma=256MB" with "cma=256M@704M" and then it boots. Note that it has to be 256M instead of 256MB.
Right, because of this it may be able to be set in config.txt, I seem to remember seeing this somewhere but as we don't support accelerated graphics on the RPi4 I've not looked. I don't believe the unaccelerated graphics uses the CMA so for the current situation you may be able to drop it.
If it's an option to set it in config.txt we need to work out if this is a general option that works for all the rpi models or if it's explicitly for the RPi4, if the later we really need to report and
get
the bug fixed because we aim to produce generic images which "just work" across all the rpi devices, anything else just makes it a support nightmare for people like myself that attempt to support it and it would be less work just not to support the RPi4 at all TBH.
Removing the cma option on the command line was known as a
workaround. Without that we would fall back to the build config of 64MB cma which was located at offset 0x38000000. This left 64MB at the end of ZONE_DMA, and I chose offset 704M so that those 64MB would still be free. Not sure if that is needed or not. The crashkernel needs to be in ZONE_DMA as well but it seems to be set to 0 size.
I have tested on 5.7 rc2 from rawhide.
This probably belongs in a bug report. What would be the correct
place to file that? From what I can tell upstream has been tested with cma settings without problems (as long as the requested CMA size can fit in ZONE_DMA). From that it seems like fedora-specific issue. Not sure though.
Not sure what you mean by "upstream" here, we use an almost pure upstream Linus kernel, if you mean the "Raspberry Pi Foundation" and their kernel, that's a downstream fork of Linus's kernel. They also have a lot of other patches and use a different desktop, GNOME from experience and working with them then RPi upstream GPU maintainer we worked out GNOME needed the 256Mb allocation but desktops like XFCE use a lot less (~192Mb if memory serves) and Raspbian uses a light desktop so I suspect they allocate a lot less.
I meant upstream as in linus tree. I noticed some comments in the
review of rpi4 ustreaming patches where various cma sizes were tested. Thus I suspected that it could be related to downstream patches. If we do not carry any relevant then perhaps the issue could related to setting the cma both in config.txt and on the commandline?
Raspbian uses 256MB via kernel commandline and the config.txt in
/boot does not have any setting for cma. The cma starts at 0x1ec00000 so in the lower 1gb. But it is a 4.19 kernel + patches so not really useful to look at for this specific issue.
Just as a follow up to this it looks like Raspbian has just (finally?) rebased to the 5.4 LTS kernel [1] in their downstream kernel and looking through the change they've done what I thought would be needed and provided a means of dealing with CMA via dt-overlays, I wasn't sure whether they would have done it via a config.txt cma=XX or a dt-overlay option, I've literally just looked at the diff but it looks like for F-33 we can investigate dealing with this in the config.txt rather than kernel command line.
I've literally just looked at the diff here and won't have time to investigate this until later this month, which actually is probably just fine to give a few more firmware releases to settle the rebase down, but if someone wants to start looking further do reach out.
That is quite a commit to dive into :) I will take a look at it.
For info here is what I have found since my last email:
I attached the serial console to check what happens when we hang at boot
with the cma setting. We allocate the 256 MB at 0xEC000000. That again leaves 64MB at the end but at the end of the 4GB this time.
With [3] the logic is that users should have full control of the cma
allocation when using the setting on the commandline. The idea makes sense but it is unfortunate that specifying only a cma size also leads to ignoring the valid allocation range specified in dt for rpi4. The commit was introduced in 5.6 and thus is not in the raspberrypi kernel. I guess this will still be a problem then if we continue to set the cma on the commandline?
[3]
https://github.com/torvalds/linux/commit/8c8c5a4994a306c217fd061cbfc5903399f...
Peter
[1]
https://github.com/raspberrypi/firmware/commit/7eff9f6774bb43bfd61e749a0b45f...
[2]
https://github.com/raspberrypi/firmware/commit/7eff9f6774bb43bfd61e749a0b45f...
arm mailing list -- arm@lists.fedoraproject.org To unsubscribe send an email to arm-leave@lists.fedoraproject.org Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
https://lists.fedoraproject.org/archives/list/arm@lists.fedoraproject.org
arm mailing list -- arm@lists.fedoraproject.org To unsubscribe send an email to arm-leave@lists.fedoraproject.org Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
https://lists.fedoraproject.org/archives/list/arm@lists.fedoraproject.org
On Mon, Aug 3, 2020 at 10:57 AM ng0177@gmail.com wrote:
I used today's rawhide image from https://dl.fedoraproject.org/pub/fedora/linux/releases/32/Workstation/aarch6... and get to the Welcome screen but then neither USB keyboard nor mouse work and I am stuck on my RPi4/8GB.
I don't think the support for loading the USB firmware on the 8Gb model has made it upstream yet. In the 1/2/4gb models it was loaded by the Raspberry Pi firmware from SPI flash, in the 8Gb model they dropped the SPI flash and there has to be kernel patches to load it. That problem is unrelated to the CMA/DMA issue.
cma=256M@704M has no effect.
On Mon, Aug 3, 2020 at 9:28 AM Peter Robinson pbrobinson@gmail.com wrote:
On Sun, Aug 2, 2020 at 9:20 PM ng0177@gmail.com wrote:
Further to those Fedora32 efforts described as below. I have been quite happily using the
EFI/fedora/grub.cfg EFI/fedora/grubenv cma=256M@704M
settings on my RPi4 w/ 4GB. Now that I purchased a new RPi4 w/ 8GB they seem not to work any more (I get onto Gnome Desktop but moving the move blanks the screen) because I think that this space needs to be reserved at another address.
Any advice? Thanks!
You'll likely need the 5.8 kernel, there's been massive issues around DMA/CMA upstream due to the "architecture" of the rpi4 with 4+gb of RAM, there was a fix that was very nearly reverted in 5.8 but luckily they found the issue at the last minute.
Regards, Thomas B
---------- Forwarded message --------- From: Thomas H.P. Andersen phomes@gmail.com Date: Thu, May 7, 2020 at 4:03 PM Subject: [fedora-arm] Re: CMA on raspberry pi 4 To: Peter Robinson pbrobinson@gmail.com Cc: Fedora List arm@lists.fedoraproject.org
On Thu, May 7, 2020 at 11:23 AM Peter Robinson pbrobinson@gmail.com wrote:
Hi Thomas,
> I have looked into the CMA setting issue a bit. This is what I have found so far. > > The rpi4 needs CMA to be in ZONE_DMA (lower 1GB of memory) as this is the only area that the peripherals on the rpi4 can address. > > The DT sets the allowed range to allocate the CMA from (arch/arm/boot/dts/bcm2711.dtsi#L869), but it seems to not work here. What does work is instead to set the offset manually. I replaced "cma=256MB" with "cma=256M@704M" and then it boots. Note that it has to be 256M instead of 256MB.
Right, because of this it may be able to be set in config.txt, I seem to remember seeing this somewhere but as we don't support accelerated graphics on the RPi4 I've not looked. I don't believe the unaccelerated graphics uses the CMA so for the current situation you may be able to drop it.
If it's an option to set it in config.txt we need to work out if this is a general option that works for all the rpi models or if it's explicitly for the RPi4, if the later we really need to report and get the bug fixed because we aim to produce generic images which "just work" across all the rpi devices, anything else just makes it a support nightmare for people like myself that attempt to support it and it would be less work just not to support the RPi4 at all TBH.
> Removing the cma option on the command line was known as a workaround. Without that we would fall back to the build config of 64MB cma which was located at offset 0x38000000. This left 64MB at the end of ZONE_DMA, and I chose offset 704M so that those 64MB would still be free. Not sure if that is needed or not. The crashkernel needs to be in ZONE_DMA as well but it seems to be set to 0 size. > > I have tested on 5.7 rc2 from rawhide. > > This probably belongs in a bug report. What would be the correct place to file that? From what I can tell upstream has been tested with cma settings without problems (as long as the requested CMA size can fit in ZONE_DMA). From that it seems like fedora-specific issue. Not sure though.
Not sure what you mean by "upstream" here, we use an almost pure upstream Linus kernel, if you mean the "Raspberry Pi Foundation" and their kernel, that's a downstream fork of Linus's kernel. They also have a lot of other patches and use a different desktop, GNOME from experience and working with them then RPi upstream GPU maintainer we worked out GNOME needed the 256Mb allocation but desktops like XFCE use a lot less (~192Mb if memory serves) and Raspbian uses a light desktop so I suspect they allocate a lot less.
I meant upstream as in linus tree. I noticed some comments in the review of rpi4 ustreaming patches where various cma sizes were tested. Thus I suspected that it could be related to downstream patches. If we do not carry any relevant then perhaps the issue could related to setting the cma both in config.txt and on the commandline?
Raspbian uses 256MB via kernel commandline and the config.txt in /boot does not have any setting for cma. The cma starts at 0x1ec00000 so in the lower 1gb. But it is a 4.19 kernel + patches so not really useful to look at for this specific issue.
Just as a follow up to this it looks like Raspbian has just (finally?) rebased to the 5.4 LTS kernel [1] in their downstream kernel and looking through the change they've done what I thought would be needed and provided a means of dealing with CMA via dt-overlays, I wasn't sure whether they would have done it via a config.txt cma=XX or a dt-overlay option, I've literally just looked at the diff but it looks like for F-33 we can investigate dealing with this in the config.txt rather than kernel command line.
I've literally just looked at the diff here and won't have time to investigate this until later this month, which actually is probably just fine to give a few more firmware releases to settle the rebase down, but if someone wants to start looking further do reach out.
That is quite a commit to dive into :) I will take a look at it.
For info here is what I have found since my last email:
I attached the serial console to check what happens when we hang at boot with the cma setting. We allocate the 256 MB at 0xEC000000. That again leaves 64MB at the end but at the end of the 4GB this time. With [3] the logic is that users should have full control of the cma allocation when using the setting on the commandline. The idea makes sense but it is unfortunate that specifying only a cma size also leads to ignoring the valid allocation range specified in dt for rpi4. The commit was introduced in 5.6 and thus is not in the raspberrypi kernel. I guess this will still be a problem then if we continue to set the cma on the commandline?
[3] https://github.com/torvalds/linux/commit/8c8c5a4994a306c217fd061cbfc5903399f...
Peter
[1] https://github.com/raspberrypi/firmware/commit/7eff9f6774bb43bfd61e749a0b45f... [2] https://github.com/raspberrypi/firmware/commit/7eff9f6774bb43bfd61e749a0b45f...
arm mailing list -- arm@lists.fedoraproject.org To unsubscribe send an email to arm-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/arm@lists.fedoraproject.org _______________________________________________ arm mailing list -- arm@lists.fedoraproject.org To unsubscribe send an email to arm-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/arm@lists.fedoraproject.org
Hi,
Am 03.08.20 um 12:04 schrieb Peter Robinson:
On Mon, Aug 3, 2020 at 10:57 AM ng0177@gmail.com wrote:
I used today's rawhide image from https://dl.fedoraproject.org/pub/fedora/linux/releases/32/Workstation/aarch6... and get to the Welcome screen but then neither USB keyboard nor mouse work and I am stuck on my RPi4/8GB.
I don't think the support for loading the USB firmware on the 8Gb model has made it upstream yet. In the 1/2/4gb models it was loaded by the Raspberry Pi firmware from SPI flash, in the 8Gb model they dropped the SPI flash and there has to be kernel patches to load it. That problem is unrelated to the CMA/DMA issue.
the USB quirk is available in Linux 5.8 [1], but there is a related report [2].
It seems to be related to the board revision and possibly arm64.
A dmesg output might be helpful.
Regards Stefan
[1] - https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h...
Thanks. Just out of curiosity, how can I use "dmesg" w/o keyboard?
Appreciate, Thomas
On Mon, Aug 3, 2020 at 7:37 PM Stefan Wahren stefan.wahren@i2se.com wrote:
Hi,
Am 03.08.20 um 12:04 schrieb Peter Robinson:
On Mon, Aug 3, 2020 at 10:57 AM ng0177@gmail.com wrote:
I used today's rawhide image from
https://dl.fedoraproject.org/pub/fedora/linux/releases/32/Workstation/aarch6... and get to the Welcome screen but then neither USB keyboard nor mouse work and I am stuck on my RPi4/8GB.
I don't think the support for loading the USB firmware on the 8Gb model has made it upstream yet. In the 1/2/4gb models it was loaded by the Raspberry Pi firmware from SPI flash, in the 8Gb model they dropped the SPI flash and there has to be kernel patches to load it. That problem is unrelated to the CMA/DMA issue.
the USB quirk is available in Linux 5.8 [1], but there is a related report [2].
It seems to be related to the board revision and possibly arm64.
A dmesg output might be helpful.
Regards Stefan
[1] -
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h...
Am 04.08.20 um 21:46 schrieb ng0177@gmail.com:
Thanks. Just out of curiosity, how can I use "dmesg" w/o keyboard?
In general there are two options
1) connect via SSH (not sure this a real option because this service could be disabled per default)
2) connect via debug UART (requires serial to USB adapter)
An alternative solution would be try to mount the SD card on a PC and look for the kernel messages in /var/log
Regards
Appreciate, Thomas
On Mon, Aug 3, 2020 at 7:37 PM Stefan Wahren <stefan.wahren@i2se.com mailto:stefan.wahren@i2se.com> wrote:
Hi, Am 03.08.20 um 12:04 schrieb Peter Robinson: > On Mon, Aug 3, 2020 at 10:57 AM <ng0177@gmail.com <mailto:ng0177@gmail.com>> wrote: >> I used today's rawhide image from https://dl.fedoraproject.org/pub/fedora/linux/releases/32/Workstation/aarch64/images/ and get to the Welcome screen but then neither USB keyboard nor mouse work and I am stuck on my RPi4/8GB. > I don't think the support for loading the USB firmware on the 8Gb > model has made it upstream yet. In the 1/2/4gb models it was loaded by > the Raspberry Pi firmware from SPI flash, in the 8Gb model they > dropped the SPI flash and there has to be kernel patches to load it. > That problem is unrelated to the CMA/DMA issue. the USB quirk is available in Linux 5.8 [1], but there is a related report [2]. It seems to be related to the board revision and possibly arm64. A dmesg output might be helpful. Regards Stefan [1] - https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v5.8&id=c65822fef4adc0ba40c37a47337376ce75f7a7bc [2] - https://github.com/raspberrypi/linux/issues/3747
-
ng0177＠gmail.com -
Peter Robinson -
Stefan Wahren -
Thomas H.P. Andersen