64

I can't find any reliable information about this. I don't have the full specification of the SD/MMC Card hardware.

Is it true? My high-level application doesn't need to be concerned with wear leveling when working with these cards?

EDIT

Could someone confirm that wear leveling is guaranteed by the SD specification? I want to be sure, because it looks like most vendors do it, but it is not required by the specification.

ivarec
  • 953
  • 2
  • 7
  • 9
  • 5
    Not all flash media manufacturers invest in wear leveling algorithms. Wear leveling is more prevalent in SSDs where the manufacturers lead the market because of algorithms. – Chetan Bhargava Feb 14 '13 at 05:15

7 Answers7

50

I work for a company that used to be a member of the SD association, we are familiar with the 2.0 (SDHC) spec. The SD card spec has NO entry for wear leveling. That is completely dependent on the SD manufacturer to handle that if they so choose. We have seen that some likely do, while others very much do not (beware the super cheap knock-off SD cards). SDXC may have changed that to include wear leveling, but I am unsure of that. Unfortunately the only way to really show that is to get your hands on the official spec. You can find it online most likely, but the SD association really wants you to pay for it.

As a side note, taking a 2GB card and writing it beginning to end over and over again averages about 10TB before the card is dead and no longer is writable. Also, SD cards will not let you know when data is bad, i.e. wont return an I/O error like a PC harddrive will. This might not be an issue for embedded designs as 10TB is a LOT of data, but it could be a factor for someone.

Kris Bahnsen
  • 1,758
  • 12
  • 17
  • 2
    I assume that even though it isn't in the spec, companies still want to do it to keep their reputation good. While super cheap knock-offs don't care about reputation, they sell to people who look to go cheap on everything. Does this sound reasonable? – Kellenjb Mar 05 '12 at 22:25
  • 1
    @Kellenjb It is a very reasonable assumption, and I personally would agree with you. However, because it is not in the spec, and it's very unlikely to gain access to any particular manufacturer's SD controller designs, it can't really be proven. We have done a lot of testing over the years, some major brands do perform much better than others. It could be differences in internal algorithms or complete lack of them. – Kris Bahnsen Mar 05 '12 at 22:36
  • 4
    Cheap knockoffs aren't going to care even if it IS in the spec. It's not immediately visible to the customer, it COULD be missing. We're talking about people who sell lead plugs as ICs. I haven't heard of a name-brand card that didn't do wear leveling, but I suppose it's possible. – darron Mar 05 '12 at 22:40
  • @KrisBahnsen can you elaborate on what the SD card actually returns when there is a non recoverable error? I am very interested in this. Is there no way at all in detecting this condition? – fred basset May 23 '14 at 20:11
  • 3
    @fredbasset, way to zombie the thread ;) Its been a while since I did any SD card work. However, IIRC SD cards themselves will physically return whatever they feel like. Some of the newer ones combined with decent drivers may be able to detect an IO error. However may of the older SD cards will just return bad data that was stored in NAND and call it good. Though, if a write is not able to complete, that will return an IO error. – Kris Bahnsen Jun 13 '14 at 05:50
  • The failure to indicate a bad data read is what I was afraid of. How do you definitively tell if your SD card will behave like this or not? It's a pity that such an unreliable technology has crept into Linux designs. Is using JFFS2 on raw nandflash a more reliable solution? – fred basset Jun 14 '14 at 18:19
  • 1
    @fredbasset Generally, if linux gets bad data, it has a fit. The company I work for has implemented a checksumming process directly on the block device that goes through a software layer. Data recovery and errors are now able to be reported to the kernel through that setup. Raw NAND is its own beast, but from what I've experienced it is more robust in terms of handling faults. Check out UBI and UBIFS, its basically JFFS3, really robust and offers some speedups. – Kris Bahnsen Jun 26 '14 at 23:37
26

It's true! MicroSD cards contain a NAND flash chip bonded to an (ARM) microcontroller encapsulated in some black plastic. http://www.bunniestudios.com/blog/?p=898 explains.

At the end of the follow-up post http://www.bunniestudios.com/blog/?p=918 Bunnie posits integrating the controller probably costs less than testing the flash ahead of time.

Quoting SanDisk's SD card product manual: "1.9 Wear Leveling. Wear leveling is an intrinsic part of the erase pooling functionality of the SD Card, using NAND memory." You can read the whole thing in the datasheet for a SanDisk brand card.

thirtythreeforty
  • 294
  • 2
  • 15
joeforker
  • 5,596
  • 10
  • 38
  • 59
  • 1
    Bunnie Huang is definitely credible enough for me! – vicatcu Mar 05 '12 at 18:09
  • That's good to know, but I'll wait for someone that can enlighten this question based on the formal specification of this technology. – ivarec Mar 05 '12 at 19:07
  • 2
    Digikey showed me [this useless SanDisk brochure](http://media.digikey.com/pdf/Data%20Sheets/M-Systems%20Inc%20PDFs/Memory%20Cards%20for%20Mobile%20Phones%20Brochure.pdf) when I tried to look up a datasheet. The [Wintec](http://www.wintecind.com/OEM/Flash/doc/Wintec_H6_Series_Industrial_Grade_SD.pdf) and [Swissbit](http://www.swissbit.com/images/stories/pdfs/flash/securedigital/S-220_data_sheet_SD-LxBN_Rev101.pdf) datasheets were better, and both mentioned wear leveling. Can you link to that Digikey-hosted datasheet? – Kevin Vermeer Mar 06 '12 at 18:36
  • Those are way better than the old SanDisk one. – joeforker Mar 06 '12 at 19:12
  • The Wintec wasn't great, but the new SanDisk one is the best of the bunch! Thanks for the edit. – Kevin Vermeer Mar 06 '12 at 20:09
  • Does it mean that in sandisk's uSD the wear leveling is implemented in the card itself and controller knows nothing about it ? – ransh Mar 12 '19 at 14:21
21

Yes, SD/MMC cards have controllers that do wear leveling. If they didn't, you could destroy one in a matter of minutes with the wrong write patterns.

That's actually a problem for some embedded projects. There's absolutely no way (apparently) to know what sectors might be wear leveled at any time, so a power cycle at the wrong time can destroy data anywhere on the card, no matter where you THINK you're writing. (don't ask how I know :) )

SD cards must be used with a system that guarantees a clean system shutdown (or at least that writes are allowed to complete), or data loss will (eventually) result.

EDIT

The problem is that the wear leveling process is entirely hidden. ANY sector on the disk could be moved at any time (swapped with the page written), and if power was to fail in the middle of that process that random sector could get corrupted.

While there ARE reasonably safe ways to implement this move, it's not in any spec so you can't trust that the card will do it. You could test one card, have it work, then the manufacturer could change the implementation without changing the part number and you're screwed.

From testing, my SD cards' controller does NOT do this in a safe manner at all.

I may look into a "high reliability" SD card I saw advertised specifically for power failure tolerance... but then you have to trust the manufacturer to do that correctly, and I don't. I really want direct control over page erases. I'm still trying to figure this one out.

darron
  • 3,491
  • 2
  • 29
  • 42
  • 4
    See my posted answer. Since wear leveling algorithms are up to the manufacturer, they might not be up to snuff. A good wear leveling algorithm will move the data first, then mark that data good, then garbage collect the original data. This might be an issue on some cards, maybe not others. SD is a very ugly design when you get to the bottom of it. – Kris Bahnsen Mar 05 '12 at 22:16
  • 2
    Yeah, it's a crap standard as far as I'm concerned. It's very annoying that such a ubiquitous memory storage standard would be so inherently unreliable. – darron Mar 05 '12 at 22:43
  • @darron, do you have some more information about this issue? This is a severe thing but I don't think it is widely known. – jpc Mar 06 '12 at 11:47
  • 2
    @jpc: Maybe I'll do a blog entry then. I'm getting the impression it's not considered often. I didn't realize the implications myself until it was too late. I fought with this, talked with my SD card manufacturer, etc. No solution other than to minimize write time. I now write to NAND flash, and copy to SD once a day as quickly as I can. There ARE SD cards designed to be "resistant" to random power failures, but I'm not sure even these are completely trustworthy. – darron Mar 06 '12 at 15:38
  • ... I also check and double check the SD write completed before deleting the data from NAND flash, but then all that guarantees is that yesterday's data is okay... not that the SD doesn't end up entirely corrupted at some point. Anyway, the massively smaller time window for writes makes an error extremely unlikely at this point... but it's still possible. – darron Mar 06 '12 at 15:41
  • @darron did you end up testing any industrial rated SD cards - would be very interested to hear your findings. I'm working on a new Linux board that uses uSD as it's main storage and am pretty worried about data corruption on sudden power downs. – fred basset Apr 27 '13 at 17:45
  • 2
    @fred basset Oddly, the specs on industrial cards seems to have changed since this answer was written. I *was* seeing microSD cards with guarantees on writes (journaled, I think)... now, they make no such claim. Talking to a manufacturer, they're using supercaps now... but only the compact flash cards have room for them. Doing things safely reduces the performance too much. They're saying to just somehow provide enough backup power to complete a write. Given how complex modern ARM SBCs are... it's really hard to do if the vendor's reference design doesn't do it. None that I know of do. – darron Apr 27 '13 at 20:24
  • 1
    @darron You are completely correct; all the ARM reference designs I've seen so far completely gloss over the SD card corruption problem and do not include a supercap etc. to allow time for a safe shutdown. How much more reliable is your writing to NAND Flash? E.g. what filesystem are you using there, and what happens if the system is writing to it and it loses power? – fred basset May 23 '14 at 20:18
  • 2
    @freq basset I'm using UBI & UBIFS. It seems to work well... I've had a lot of systems running for years now with no problems. – darron May 24 '14 at 16:32
  • I see this uSD datasheet https://www.alliedelec.com/m/d/04db416b291011446889dbd6129e2644.pdf mentions that wear leveling is implemented. I would assume from this that it is part of card and not controller, so when you said that controller implement wear leveling did you mean in the device or in the microcontroller that communicates with the sd card ? – ransh Mar 12 '19 at 14:24
12

Any type of SD card using any type of conventional NAND flash memory is going to have to use some type of sector virtualization, since no conventional NAND flash device can support erasure of individual 512-byte sectors, and no conventional NAND flash device of significant size would be able to yield performance that was within an order of magnitude of being even marginally acceptable if every attempt to write a sector required the device to copy all the sectors in that sector's erase block (even to RAM), then erase the block and write all the sectors back. Most sector-virtualization techniques are inherently somewhat wear-leveling. I would expect the biggest issue of variance between quality devices and knock-offs will be the extent to which a device actively tries to even out the leveling between blocks, versus simply using pseudo-random block allocation and hoping that will yield acceptably close-to-uniform results. In practice, I would expect that even random/hope-for-the-best allocation would be adequate in most cases.

supercat
  • 45,939
  • 2
  • 84
  • 143
  • It very well might be that the "wear leveling" some manufacturers implement is caused by the NAND interface itself and block to sector/sector to block virtualizations. – Kris Bahnsen Mar 06 '12 at 00:09
  • 1
    @KrisBahnsen: I would expect that manufacturers which claim wear leveling will actively monitor the relative wear of different blocks and age of the data therein, and if blocks with long-lived data are found to be in blocks with much less wear than average, the data from those blocks will be moved into blocks that had more wear (so as to, most likely, minimize future wear on those blocks). This could improve by a factor of 10 the useful life of a device that's 95% filled with data that never changes, while the remaining 5% of space is churned frequently. – supercat Mar 06 '12 at 21:25
  • I agree, if they are advertising it I imagine they would do some kind of real wear-leveling. I am a bit negative on the whole subject which I do apologize for; I've been bitten a few times by the quirks of SD. – Kris Bahnsen Mar 06 '12 at 22:02
  • 1
    @KrisBahnsen: The biggest problem I think is a virtualization model which assumes a linear bunch of consecutively-numbered 512-byte sectors. It may have been convenient to interface with DOS, but it doesn't really fit well with the hardware that exists, nor with what the host software really wants. Open up block numbers to 64 bits and allow them to be arbitrarily non-consecutive, and then arrange to have files always stored in logically-consecutive blocks. To delete a file, delete its range of blocks. – supercat Mar 06 '12 at 22:56
  • 2
    This is possibly the best answer. I suspect this is how all of them do it. If you think about it, there is no way to do deterministic wear leveling with any kind of truly sophisticated algorithm because the _wear history itself_ would have to be stored _somewhere on the same card_ and that part would fail first. Randomized leveling is the only practical possibility. I am not convinced SSD's are any smarter. They may simply have more replacement buffers, though it is certainly within possibility that they store very coarse (and hence infrequently updated) wear information. – Nimrod Jan 02 '16 at 09:08
  • @Nimrod: Wear leveling can work out very nicely if one has a reasonably-sized area of memory that can atomically be written in chunks smaller than a page; it's not necessary that erase operations be atomic--merely write operations. – supercat Jan 02 '16 at 17:21
  • @Nimrod: Even without the ability to write chunks smaller than a page, it's possible to have a group of three "root" blocks of memory which are used to hold the addresses of some main index blocks. Things may be arranged so that at any time when one of the root blocks is being erased, the other two will both identify it as such. The main index block wouldn't need to move very often, so the root blocks wouldn't need to be written to very often. On a part with 528 byte pages grouped in 1024-page blocks, if one uses three blocks (3072 pages) to keep track of the main-block locations... – supercat Jan 02 '16 at 17:25
  • ...one would only go through one erase cycle on the root block every 3072 changes to the main-block locations. If rather than identifying individual blocks, each address listed in the root block specified a range of three locations where the main block could be, and used a similar approach with those, and if the part had an endurance life of 10,000 write/erase cycles, then that level of indirection would be sufficient to maintain an atomically-updated 528-byte block which could be updated about 900,000,000,000 times while needing less than 30 random accesses to find the latest value. – supercat Jan 02 '16 at 17:32
  • Adding one more level of indirection would push things to the point where even if one were only willing to assume an endurance of 3,333 cycles one could still accommodate 1E21 changes to the main page but find the latest master-page content in less than 50 random reads. – supercat Jan 02 '16 at 17:37
  • If one were to construct a flash drive with a NAND flash chip and a smaller NOR flash chip, one could probably make things faster and more robust, since NOR flash chips allow programming of a byte at a time. Actually, I suspect things could be improved a lot even by adding some validity-related bits which would mark a page as one of fours states: (1) not touched since erase; (2) maybe partially written [should be considered unusable until next erase; (3) validly written; (4) expressly invalidated [obviously unusable until next erase]. – supercat Jan 02 '16 at 17:52
  • Flash drives can presently tag each page with an indication of what logical sector it represents, along with a sequence number; if pages are only read when no successor exists and erased when a successor does exist, the drive state will always be recoverable even if indices become corrupt, but if the system can't directly tell if a page has been superseded a corrupted index could break those invariants. BTW, I wish there were a standard means by which a flash drive could indicate that indices need to be rebuilt, and by which such an operation could be initiated, with the caveat that... – supercat Jan 02 '16 at 17:58
  • ...rebuilding indices on a large drive might take over an hour, and loss of power could necessitate starting over, but that until indices were rebuilt the drive might perform *very* slowly. – supercat Jan 02 '16 at 18:03
9

Sandisk have a white paper that explains the wear levelling logic in their cards, and goes on to give estimates of the card's life under a number of scenarios. Executive summary: unless you are hammering the card non-stop, it will last decades.

Michael Slade
  • 211
  • 2
  • 4
  • Link is dead and the content seems to have been expunged -- because it revealed too much, perhaps? – Cuadue Aug 12 '14 at 16:04
  • 7
    @Cuadue The link is accessible. Just in case it breaks, the page is archived now at https://web.archive.org/web/20150326122100/http://ugweb.cs.ualberta.ca/~c274/resources/hardware/SDcards/WPaperWearLevelv1.0.pdf – Lekensteyn Mar 26 '15 at 12:21
2

Its interesting to note that despite this a lot of devices do corrupt SD and microSD cards especially high density ones if the battery is low or the phone crashes/shuts down/etc. I suspect that the problem is inadequate regulation of the voltage supply as this is well known on some cards (cough Adt /cough) to result in the phenomenon of an unreadable card on certain external readers but works fine on the micro variety supplied by some computer shops.

I am in the process of recovering a card at the moment with this fault, strangely enough most of the data is recoverable but some sectors are not though this changes on each attempt. Could the wear leveling itself be at fault? (yes tried multiple readers, same fault!)

Conundrum
  • 106
  • 4
1

Also had some success "nuking" zombie cards, ie ones which won't complete a format or are read-only. Only works on a small percentage but they are much more sensitive than most "official" guidelines to the effect.

A test card thus treated lasted a full four months before failing again, if it hadn't been for the device it was used in supplying noisy voltage the lifetime might have been longer.

Conundrum5642Z
  • 21
  • 1
  • 1
    By "nuking" you mean putting it in a microwave? At what wattage and how long? – 0x6d64 Jan 30 '17 at 12:31
  • Uh, no :-) Also why do that when a glow discharge (RF) would have a similar effect and be a lot less hazardous than the method I used? Turns out that uSD cards are a lot more sensitive due to the lack of a metal casing on one side. – Conundrum5642Z Jan 31 '17 at 10:36
  • I can't confirm or deny that nuking refers to using a "gadget" 5642+HVPS for legal reasons. However it did work repeatedly, and the technique is now known as I published it on H-a-D etc. – Conundrum Feb 01 '18 at 07:48