NOOBS won't boot on SDXC (256 & 512GB) cards #540
Comments
ThomasFreedman
changed the title
|
Formatting any SD card >32GB with a standard tool like gparted will result in the format being exFAT, not FAT32, because "themz the rulez"! For such large size SD cards you need to use a special tool to force FAT32, but I forget the name of this app. So an easy way is to DD raspbian Lite onto your card as you did before. Then delete the files on the boot partition and replace them with the files from the noobs_lite.zip file (not full noobs) and boot it. NOOBS should be able to partition the rest of the drive ok following that. Why are you using such large cards with NOOBS? NOOBS is primarily used to get beginners started (you don't sound like a beginner) or to multi-boot several OSes. If you are in neither of these categories, you might find it easier to flash your chosen OS onto the SD card with Etcher. If you are using them to multi-boot many OSes, be aware that NOOBS can only install the OSes all at once at the initial installation. It is not so easy after that to change an OS, or reinstall it without re-doing the whole card. if that is your aim, you might want to look at PINN which gives you more features to backup/restore/replace individual OSes without affecting the other OSes. PINN is also available as an image you can flash with Etcher, which is another way to avoid this FAT32 formatting problem. |
|
'because "themz the rulez" ' Hmmm... sounds wrong, but NOT saying it is. With a GNU tools like gparted which I've used zillions of times over the years, AND which provides initialization for many different file types, it seems odd it would say fat32 and format to a different standard. I use NOOBS cuz it was expedient and convenient. Also the project I'm working on has many "NEWBIE" user types and there is much more documentation for such users with much less confusing jargon etc. I have noticed odd behavior that leads me to believe you are right about gparted formating to exFAT. When I fire up the Pi with a sdxc card formated with it there are strange delays. One for example is that df -h doesn't show full capacity right away, it progressively increases (and rather slowly at that) until it finally stabilizes at the proper amount for that size sd card. I hypothesized it was being resized in the background and when the resize was done the capacity would be right. Weird thing to counter your idea is I don't see that on a 64GB sd card; those seem to work identically to 32GB. Also, I can use the sd card duplicator utility in the non-light version installed with NOOBS to dupe a 32GB card onto a 64GB card and when it is booted resizing occurs and rather quickly, in fact prior to the OS actually being booted as I recall. Thanks for the tip / work around for installing NOOBS tho, that may be a good way to go. If I understand what you said properly, I should be able to install NOOBS normally if I insure the full capacity is formated to FAT32, if that's even possible. I also prefer to use more native tools rather than 3rd party tools like etcher. |
|
The biggest sd card I have used is 64GB which formatted as exFAT with the SDA formatting tool and wouldn't boot, so I used my etcher image and it then worked fine. So the rest is speculation on my behalf and passing on info from other people who have experienced this issue. You don't have to format the whole card as FAT32 for noobs (which can cause the exfat issue depending on the tool you use and the size of the card). You only need the first partition on the card to be fat32 so that it will boot. Noobs will partition the rest of the card for you. So you can just create a 64MB partition, format it as fat32, unzip noobs to it and boot it. Noobs will do the rest. That's essentially what my Etcher image does. |
|
@procount see https://github.com/raspberrypi/documentation/blob/master/installation/sdxc_formatting.md (which I mentioned on @ThomasFreedman 's other issue ). |
|
Ah yes. I guess that's the tool I was thinking of. |
|
Just did a bit of searching, and found https://gparted.org/features.php For those not aware, one of the problems with exFAT is that it contains Microsoft patents https://en.wikipedia.org/wiki/ExFAT#Other_implementations |
|
I updated my OP, as it was missing a crucial NOT. Glad there were no trolls ready to pounce on that mistake! :) I've been looking for a native Linux tool to format to FAT32, not exFAT for sdxc cards, but I can't find any. So it's good to know NOOBS will resize the base partition to fill the entire card. This may be another reason to use NOOBS, b/c using dd to copy an image to the card results in multiple partitions, at least for the latest Raspbian Lite version. I'm not sure how to create a partition table for a large sdxc card on Linux tho. I mean, I suspect the partition table is format specific, so on an sdxc card it would be exFAT, not FAT32, unless dd creates the partition table as well. I'll use gparted, format to FAT32, then create an EXT4 linux partition on it. Then, I'll dd a Raspian Lite image over to it and see if it works. If it does it tells me the dd writes a new partition table. If the img shows up under EXT4 then it doesn't. |
|
@ThomasFreedman I think you're getting very confused, and unfortunately the slight misinformation from @procount didn't help 😉 Regarding "themz the rulez" see https://www.sdcard.org/developers/overview/capacity/ Partition tables come in two flavours, MBR and GPT. More info at Wikipedia: The Pi can only boot from cards with an MBR partition table. The partition table isn't format-specific. An out-of-the-box SDHC card (i.e. one that's 32GB or smaller) will have an MBR partition table with a single FAT32 partition. An out-of-the-box SDXC card (i.e. one that's 64GB or larger) will have an MBR partition with a single exFAT partition. Gparted doesn't support creating exFAT partitions, see the link I posted earlier https://gparted.org/features.php It's fine to use it for making a large FAT32 partition. The bootloader in the Pi's ROM can only boot from FAT16 and FAT32 partitions - AFAIK if you tried to boot off an exFAT partiton then you wouldn't even get as far as the Pi's rainbow-square screen. The partition table is simply the first 512 bytes on a disk, so writing an image with NOOBS doesn't resize the base image to fill the entire card - it actually resizes the boot partition to be just-big-enough to contain all the NOOBS files; and then partitions up the rest of the card for installing the other OSes into. More info at https://github.com/raspberrypi/noobs/wiki/NOOBS-partitioning-explained So, hopefully that all answers more questions than it causes 🤣 And WRT your original problem: if you're getting errors displayed by NOOBS, then your card must have been formatted with a FAT32 partition (and an MBR partition table), otherwise your Pi wouldn't have even got that far. However I don't know why you're getting those errors. |
|
Apologies if I misled 😉 (@lurch - I'd nearly finished this post, when yours popped up saying the same things, but yours is better so I trashed mine 🥇 ) I think the problem with partitioning is due to the way the technology has advanced and yet still has to maintain backwards compatibility. So it creates artificial C,H,S references for an LBA accessed partition that must end on a cylinder boundary and on some other boundary (4K/MB or something?), even though C,H,S values have no relevance to an SD card. I think sometimes these constraints conflict and hence cause problems. Maybe this is the issue with 256/512GB cards 🤷♂️ |
|
Thanks everyone and especially you @lurch for the detailed info, including the note that the Pi can't boot GPT partitioned cards. Since now you've confirmed what I originally thought that gparted does support large FAT32 partitions (and the 0.25 version doesn't provide an exFAT format option. idk about latest 0.33 version) then I should be able to use that to resize the root partition to fill the entire remaining capacity so NOOBS or Lite won't need to. idk how it determines is a resize is required, but hopefully using gparted as noted above will prevent the auto resize and therefore I should see full capacity reported by df -h immediately. |
|
@ThomasFreedman I think you're still getting mixed up between "NOOBS", "boot" and "root" partitions, and what gets resized and when 😉 |
|
I believe I'm pretty clear on the distinctions, tho more familiar with Lite as it only has 2 partitions. I don't recall looking at a Lite sd card after it was booted with gparted. I'll certainly review the info. For one thing it may clear up confusion regarding how a MSDOS partition table can contain so many partitions that I see with NOOBS cards after booting. I know about extended partitions, but gparted should show those as extended and I don't think it did. |
|
I'll need to review gparted again with NOOBS and Lite installations both before and after booting to visualize the partitions. I reviewed the wiki on MBR, very detailed info there, most of which I am familiar with. Regarding the resizing operation performed by the Pi, that is only done for the root partition right? All I was saying was I could use gparted to do the resize so the Pi wouldn't need to. That would eliminate a bug in the Pi resize code as the cause of hanging problem for extra large sdxc cards, as well as seeing if it eliminated the delay in df -h showing full capacity. |
|
Here is an image from gparted of a NOOBS-style formatted SD card with 3 OSes installed. |
|
I think the resizing errors you are referring to are the initial reformatting of the SD card that NOOBS does BEFORE it installs any OSes, so it has nothing to do with boot or root partitions. |
|
NOOBS does not resize any OS partitions, it only resizes the initial partition 1 to make space for the extended partition. Thereafter it simply creates OS partitions of the correct size and copies the files into each partition. In fact, I remove code in any OS that resizes the root partition on installation for any OS that I convert, because this is simply not needed in a NOOBS/PINN installation. |
|
Ahhhh, I grok you now @procount. And thx for the gparted pic, I haven't confirmed it here yet. It will be a bit different if only 1 OS, but the structure (and the use of an extended partition for all other items / OSs) should be the same. So I will try to format 1 small partition to put NOOBS in, and leave the rest unpartitioned and see if that boots. If so it would seem to confirm your theory. It still doesn't explain the slow incremental rise of capacity shown by df tho. That is taking place in the root partition, while it's active ! I therefore don't think it is actually a resize of the partition, but something else. |
|
I created a MBR partition table and then created a 4GB FAT32 partition on it, leaving the rest unallocated. I copied NOOBS 3.0.0 over to it. When it booted in my new RPi3+ it took some time to resize the partition, but then NOOBS popped up. I tried to connect to the net with WiFi to my unsecured SSID which always works on the RPi3, but no go on the 3+. It finds the SSID just fine, but it leaves the username & password fields grayed out, which I thought was normal. However I waited 5 minutes and I finally just canceled the attempt and went on to install Raspbian. It's now showing the slide show, but after 10 minutes progress shows only "20MB of 4079MB written (3.0 MB/sec). Just changed to "Raspbian Full: creating filesystem (ext4)" Looks to me like it will work. It definitely confirms @procount's theory about why it was locking up. It appears the resize code fails in resizing the single huge FAT32 partition. I'll let it continue. It went back to slide show, but the status remains the same, no further progress shown, but green light is on solid so it may just take it awhile to format that space to ext4. We'll see... |
|
Hmmm, one thing I didn't think to suggest yet: it'll probably take a long time to run, but given how big your microSD cards are it might be worth running h2testw or F3 over them, to check they're genuine and that they're functioning correctly. |
|
Not so fast - it timed out with a popup saying: "Raspian Full: Error mounting filesystem". I pulled the card and put it into my linux system. I can't take a screenshot as the screen memory is protected by the OS for security reasons. The structure looks good, there are 2 partitions: RECOVERY, and the extended, which contains SETTINGS (ext4), boot (FAT32), and root (ext4). The root partition is 248.16GB, but with an error. This is most of the info screen on that ext4 partition: Filesystem volume name: root dumpe2fs 1.43.4 (31-Jan-2017) Unable to read the contents of this file system! |
|
That is a mighty old utility. I would prefer the tool by RMPrepUSB, who I've corresponded with quite a bit. He's a developer that knows his stuff and creates some great tools. I may chk it, but I'd be very surprised if the problem was the card. They came from Amazon and from different sellers, plus packaging was pristine and identical to other Samsung sd cards I've bought before. Wouldn't hurt to check to rule that out tho. The 256GB Plus card does appear to be defective. Even gparted can't format that partition without errors. I repeated the exact same procedure with the 512GB card and the installation was successful. The new RPi3+ board refuses to connect to my unsecured WiFi tho. I looked at the wpa_suplicant conf and it was setup correctly. I pulled the card out of the RPi3+ and put it in a RPi3, and it failed to connect as well during the initial setup. However after completing that initial setup I used the WiFi icon at the top right of the screen and it connected. Although I know the WiFi / Blutooth chip is different on the 3+, it still (apparently) uses the same I/F. |
If the NOOBS partition-resizing code is getting confused by such large SD cards, I guess it's possible that NOOBS is writing an 'invalid' MBR to the card, which might then cause problems with formatting tools? Might be worth totally blanking the card and writing a new MBR, just to be sure? |
|
And I think @procount has much more experience with WiFi configuration on NOOBS / PINN than I do... 😸 |
You probably know this already, but I'll say anyway....
Not really, I think @XECDesign did all that stuff. I just converted/added some additional wifi drivers for PINN (most of which I have to do again for the latest kernel, when I have some time). I've never tested with an unsecured SSID, although I did manage to connect it to an EAP wifi network by manually editing wpa_supplicant.conf Large SD cards shouldn't be a problem for the MBR. I've used PINN on a 256GB HDD (not an SD card) without problem, but I installed it with Etcher. So if the 256GB SD card has been found to be defective, and the 512GB card can be installed OK by creating a small FAT32 partition, can we now conclude that it's the initial resizing of the huge FAT32 partition down to a tiny one for NOOBS that is causing the issue? |
|
@ThomasFreedman - Are you willing to do some tests? Am I right in thinking that you are using the FULL version of NOOBS that includes Raspbian and LibreELec? If so, could you try formatting your whole 512GB card as FAT32 and trying to install the NOOBS-LITE version? It does not include any OSes and is a lot smaller, resulting in different NOOBS behaviour. Secondly, it would be interesting to know the size of partition that causes the resizing to fail (assuming my theory is correct). So could you repeat your creation of an initial FAT32 partition using gparted, but use initial sizes of say, 15GB, 30GB, 62GB, 126GB, 254GB and 510GB and then try to install the FULL NOOBS version, noting at which initial size the resizing failed? |
|
@procount Also worth noting that it might not be a specific size of partition that fails, but a combination of partition size and whole-disk size? ISTR there were issues before where somebody with a particular-size card had problems with the partition resizing? Might be something to do with the CHS-translation that you referred to earlier? 🤷♂️ |
|
I'll be happy to do that testing, but it will be Sunday at the earliest. I just got a text that a dear friend is in intensive care ward and I need to go see him. Tomorrow I have a long conf. call, and tonight I need to prepare for it. As for using Lite, I have installed that as well as I described above. Not sure if you wanted me to test Lite along with NOOBS, please explain how you want that done, as I use dd to copy the Lite to the SD card. And yes, the NOOBS version is 3.0.0 with 2 OSs LibreElec and Raspbian Full. |
|
I hope your friend recovers okay.
There's both Raspbian Lite and NOOBS Lite, so referring to just "Lite" will probably cause confusion... 😉 |
|
Ahh, I didn't know there was a NOOBS lite. I'll have to search for that. |
|
I think I tried full once, not sure, but you requested I focus on Lite so with only an exception or 2 that's what I did. I think I tried full when I started having these login issues, but it was unrelated to that. I was getting tired and managed to get into a bad sequence I believe. I did at least 3 trials where after rebooting and expecting to run a IPFS install script I've been working on, I couldn't login. Still perplexed about why adding init=/bin/sh no longer seems to work. The boot sequence just stops. I can plug & unplug usb keyboard / mouse and see it reflected on the console. With the Lite version it would be nice to know how to get to a root prompt. From all my googling I saw nothing that indicated the appendage no longer works. |
|
Also, holding shift didn't seem to work either. |
|
[continuing off-topic: PINN includes the ability to reset your password to whatever you want, without having to mess with init parameters. Adding In my post about testing, the first paragraph was about using the Lite version, but the second was actually all about testing with the FULL version. Apologies if I wrote it ambiguously. The reason for testing with both versions is that NOOBS does different things depending on the size of the files on /dev/mmcblk0p1. If they are small (<63MB like in the LITE version) NOOBS will copy them to RAM, recreate the partition table, and then copy the files back again. (So I was surprised with your results of using the Lite version with >256GB partition size). if the files are large (like in the NOOBS FULL version), it cannot copy them to RAM so it relies on parted resizing the partition. My theory is that parted will struggle to convert a very large FAT32 disk size down to a very small disk size. I don't know if it will automatically convert the FAT32 to a more practical FAT16 format? |
|
OK, I see. Thx for the details. I just found a (the?) bug in my IPFS script, and it was a Doh! moment of missing the extra > so I was truncating the file. I'll see what I can do for additional testing. Now that I understand better the distinction between the full and lite boot process, and how RAM is involved. You guys have a great day. Not sure where in the world you are, but here it sunny and springtime, and I intend to enjoy it! |
See https://www.raspberrypi.org/documentation/linux/usage/users.md Regarding the Also, the "hammering shift" is a NOOBS thing and not a Raspbian thing 😉 Myself and @procount are both based in (different areas of) the UK, and it's currently sunny but still a bit chilly 🌤️ |
|
I was confused about which OS adding init=/bin/sh was applicable to. I was using Raspbian Lite created with dd. I was adding init=/bin/sh to cmdline.txt. which doesn't work. I tried to install NOOBS (not NOOBS-Lite) with an initial partition size of 300GB which failed, as I expected. If the lite version can't handle booting, I didn't expect the full version to boot either. There is some type of firmware boot loader issue related to the partition / card size or their combo. As I said I see zero activity on the green LED, once the threshold is crossed. |
|
ping @ghollingworth - is there any kind of limitation in the bootrom or firmware that might prevent it booting off such massive FAT32 partitions? If so, do we need to need to mention this in the NOOBS documentation? |
|
Yes, there was a bug in the Pi 3 bootrom which used 16 bit integers for some multiplications which limits to 256G Fixed in Pi 3+ |
|
Ahh, interesting! @ghollingworth Given the beginner-focused modus operandi of NOOBS, I guess we cant't really expect people to create a smaller-sized partition on their card in order to boot NOOBS; so does it make sense to tell a small white lie and say that NOOBS will only work on micro SD cards larger than 256GB on the Pi Model 3B+ and Pi Model 3A+, but that NOOBS is compatible with cards up to 256GB on all other models of Pi? Has @XECDesign tested NOOBS on such large SD cards? |
Easier to test with USB hard drives, which had same problem in the past. |
|
Ahh, so mystery solved regarding booting sdxc cards > 256GB on the Pi3. Now that I have a 3+ I will test noobs-lite with an initial 512GB FAT32 (the entire sdxc card) and verify that works. Probly still has the problem it won't connect to my unsecured SSID, but that's irrelevant to this discussion. Not even sure how to boot the pi from an external USB device to test as @maxnet suggested, but not necessary since I have a 512GB sdxc card to use. |
https://www.raspberrypi.org/documentation/hardware/raspberrypi/bootmodes/msd.md 🙂 |
Documentation like that scares away users. If you have a Pi 3+ it will already boot from USB by default if there is no SD card present. (Not suggesting the TS does that. |
|
Thanks guys for the boot option link and info. I agree with @maxnet, it scares me away from trying to boot the Pi3 from USB if it permanently disables the ability to boot from the sd card, and it sounds like setting that bit (which is not a trivial matter to accomplish, especially for NOOBS level users) is irreversible. |
|
It does not disable the ability to boot from the SD card. It gives you more options by allowing you to boot from USB as well. But to do that you have to remove the SD card. If you have an SD card fitted, it should boot from that in preference. That bit is programmed in the 3B+ anyway, so no harm in programming it in the 3B to match. |
Pull Requests are always welcome... 😉 |
Trust me. But I do believe documentation is often too technical and detailed, and could use a simplified version, with only the instructions for the latest Pi model and not more background information than necessary. |
Would that be the Model 3B+, the Model 3A+, the ZeroWH or the CM3+ ? 😉 |
|
If there is a problem with docs, you can write up an issue on the github, or provide a PR. All issues and PR's are assessed and copy edited as necessary. So even bad English will be corrected if the change is deemed valid. Reported issues will require more time, as I/someone have to find the time to write new docs. |
|
I've just spotted raspberrypi/documentation#1136 😃 |
|
Please update this bug with some clarification of whether the current version of NOOBS can or cannot boot from a 256GB SD card. The bug in the SoC boot ROM which Gordon referred to will prevent all but BCM2837B0-based Pi's from booting from SD cards larger than 256GB. |
|
It's a bit unclear what's limited to 256GB on older Pis:
|
Paging @ghollingworth |
|
What Rasberry Pi model are you using? |
|
I tried it with the 3B & 3B+ I now have a Pi4, but have misplaced my 256GB SD card so haven't tested on the Pi4. |
|
Is are they both SDXC type SD cards? |
|
By definition, anything over 32GB is an SDXC card https://www.sdcard.org/developers/overview/capacity/ |
|
@lurch thanks for that 👍 |
NOOBS 3.0.0 - Used gparted 0.25,9 on Debian Linux (Stretch v 9.8, OS kernel v 4.14) to format the entire card with fat32. Copied all of the unzipped files downloaded from the Raspberry Pi website (https://www.raspberrypi.org/downloads/noobs/) to the root of the sd card. Booted in RPi3 (not +), but got an error on 256 stating it could not resize the fat32 partition. I answered that dialog and another progress bar appeared and never got beyond that.
On the 512GB sd card I just got a kernel panic and system froze.
This behavior was repeatable. Both sd cards were Samsung EVO. The 512GB card was green & white, class 3, write speed 90MB/s. The 256GB sd card is red and white, but had same specs.
I had no difficulty using either of those cards with Raspbian Lite, installed with dd on a linux desktop. The cards are fine.
When I put the sd cards back in the Linux system and review their structure it appears to be the same as a 32GB sd card, except the root partition size is very big. Gparted shows no errors.
The text was updated successfully, but these errors were encountered: