Reformatting The Micro SD Card

[SteveFL]

I have read mixed comments about the formatting of Micro SD cards on dash cams. Some critics say they should never be formatted, (outside the initial format) unless they are showing signs of going bad. Some say that the card should be formatted every month. Some even say it should be formatted anytime you save a video from it.

Ok, which option do you choose and do you have another opinion?

 

[Keiman]

I format whenever I replace the card after taking it out of the camera to download stuff - however this may only be every 3 months as. I only do short journeys now days

 

[DT MI]

Good, bad or indifferent I format mine only when I do a F/W update and that's just to make sure that there's nothing in the new F/W that might conflict with the formatting done by the previous version. I've done this with 2 different G1W-H cams, 2 A118-c cams, 2 SG9665GC V1 cams, 1 SG9665GC v2 and a Blacksys CF-100. In over 2 years of dash cam use I've never had an issue that could even remotely been attributed to a 'problem' with the SD card not being formatted on a regular basis.

I also do not regularly reformat the SD cards in use in my dSLR and P&S cameras and that's been the case for well over 15 years now, again with no issues.

That said if I were to start experiencing issues my first step would be to check the power supply and the second would be a low-level format on the card just to eliminate that as a possible source of the issue.

 

[throrope]

I use a 64gb card formatted to fat32 in the daily driver. I'm leary of formatting this card in the dashcam and finding it will only access 32gb, which is " maximum" based on the standard windows configuration. I check for "file looping" and capacity every so often. No issues to date.

Our other two dashcams have 32 gb cards that get formatted when I pay attention.

 

[Rajagra]

Sometimes my 0806 gives a message like "unoptimum, format card?"
Perhaps the file structure gets fragmented and that could slow down writing a lot with this type of storage. CrystalDiskMark shows why, I googled an example below. Orderly, sequential writes here are 40 to 50 times faster than randomly writing to different areas.

Now the card shouldn't really be getting fragmented - when the camera needs to make space it will be deleting one old, big file, then overwriting the space with a new, big file. But clearly something undesirable is happening over time. So I'm in the camp that says reformat the card as often is convenient - after making certain nothing on it is needed.

The more problematic the camera (model), the more likely it is you need to do this.

 

[Module 79L]

I have read mixed comments about the formatting of Micro SD cards on dash cams. Some critics say they should never be formatted, (outside the initial format) unless they are showing signs of going bad. Some say that the card should be formatted every month. Some even say it should be formatted anytime you save a video from it.

Ok, which option do you choose and do you have another opinion?

I'd go with @DT MI's option. That's exactly what I do.

 

[H-R]

Formatting may not always be necessary but it will make it 'easier for the camera' to find the next free space to drop the next recording into. Over time, file fragmentation can become an issue, which not only slows down the writing process but also increases the potential for file corruptions.

A recording which fits in its entirety into one continuous section of the memory card will be faster to access for read and write. The same recording stored in many small segments of free memory will increase read/write steps and increase the risk of the FAT table becoming corrupted. In a freshly formatted SD card the FAT table will hold one single address for the corresponding stored recording file. Once badly fragmented, that same recording will occupy not only several sections within the memory space, but also requires several FAT table entries which point to each of the file fragments.

Below are some real life boot time measurements done with an ITB-100SPW dashcam, highlighting the increasing time overheads with each additional file being stored, each measured from power up to the first voice prompt.

No memory installed ("please insert SD card"): 19 secs
Wrong format or missing SETUP folder ("check SD card"): 22 secs
8GB installed: 36 secs
64 GB with 1250 recordings (20+ hrs): 70 secs
128 GB after a new (device) format: 50 secs
128 GB with 3062 recordings (50+ hrs): 200 secs

In the above example with 128GB, the memory was only written up to its maximum capacity, no looping had yet occured, which would add even more time overheads with each additional file fragmentation.

Short answer: format whenever you think about it, but ensure to keep important recordings on a separate storage media (computer etc.), just in case you need it some time down the track.

 

[DT MI]

Some of the comments in this, and other threads, about file fragmentation potentially causing issues got me to thinking (that in itself can be scary at times ).

Being an old I.T. guy there's no doubt that performance can suffer when a disk is badly fragmented but FAT integrity is a separate issue and is not related to performance to any significant degree so can really be discounted.

Performance of a physical HD (spinning platter) that is fragmented is caused primarily because the data cannot be read or written until the correct sector/cluster is under the read/write head and moving the head to the proper track. It's the wait time for that positioning to happen that causes the performance decline. That does not happen with solid state devices because there is no physical movement involved - although there is still a small performance loss as compared to accessing sequentially available sectors.

But what I was really thinking about is "How bad is the fragmentation on the SD card in my dash cam?" since it's been in the camera a long time and it's never been reformatted. Answer - None.

I just went out and pulled the 32GB card from one of my side cams, a SpyTec A118-C. It has not been formatted since I first started using it as a side cam - and before anyone asks it was so long ago I don't remember when it was.

Here is some general information about the card and it's use:
- Loop recording set at 5 minutes
- Number of video files on the card - 61
- File size for a 5 minute file - 585K
- Smallest file on the card - 132K
- Number of files smaller than 585K - 21 (or roughly 1/3 of the total files)

Since my driving patterns are relatively consistent I'm going to assume that the mix of 5 minute files and shorter files currently on the card is typical but will agree that it can fluctuate somewhat over a given period of time.

When I initially formatted the card the sequential number of the first video file would have been '_001'. As of right now the most current file has a sequential number of '_809'. (Since the file numbers go back to '_001' after it reaches '_999' (I've confirmed that this is the case) the last file number could be '_1809' or '_2809', etc. if that were the naming convention but minimally there have been at least 809 files written to the card.)

Since the card holds only 61 files it has been overwritten (or looped over entirely) at least 13 times - 809 divided by 61.

What does all this mean? In my mind the concern about file fragmentation is greatly exaggerated as it doesn't appear that there is any significant fragmentation occurring with dash cam use. The following 2 screen shots show the file list (sorted by size) and the fragmentation analysis if anyone is interested.

 

[jokiin]

But what I was really thinking about is "How bad is the fragmentation on the SD card in my dash cam?" since it's been in the camera a long time and it's never been reformatted. Answer - None.

to be honest this is what I would reasonably expect, I think people are so used to Windows where it just picks up and dumps files anywhere (particularly in older versions of Windows) which is nothing like the way the cameras work

 

[DT MI]

to be honest this is what I would reasonably expect, I think people are so used to Windows where it just picks up and dumps files anywhere (particularly in older versions of Windows) which is nothing like the way the cameras work

OK, but now I'm getting confused.

I really did expect there to be some fragmentation after that long a period of time without formatting - 0% seemed to me to be pretty much impossible. To do a double check I ran another fragmentation analysis using a different program - 'Piriform Defraggler' with completely different results. Even so, if the Piriform analysis is correct the worse case is 5 files have 3 fragments - nothing that will impact performance to any statistically significant degree.

Don't know if it's a bug in Windows Defrag analysis or what. See below:

 

[DT MI]

...Don't know if it's a bug in Windows Defrag analysis or what....

Ta-Da, I think I found the answer. I found this tidbit of information "In Windows 7, Microsoft had turned off defragmentation for Solid State Disks" in this article - http://www.thewindowsclub.com/disk-defragmenter-windows.

The machine I'm on is Win-7 so I suspect what's happening is the OS is recognizing the mSD card as a SSD and showing 0% fragmentation since the defrag function is disabled.

 

[H-R]

@DT MI:

Agreed, harddisks are much slower than solid state storage devices due to the mechanics and landing the head(s) on the right place of the platter.

You already found out in the meantime that Windows 7's Defragment Tool is not a reliable indicator of the fragmentation level on SD cards, so no further comments to this point.

However, your assumption that "FAT integrity is not performance related to any significant degree" is relative: every read/write cycle which the processing unit needs to spend time on to find the next available slot for dumping the next recording is an added overhead, which is in addition to the actual time needed to write the data into the memory cells. With a slower processor and a badly fragmented SD card this can be quite significant as a percentage of overall write performance.

But more important than speed gains is file integrity (you want to be sure that if the unthinkable happens your recording is really available as that important piece of evidence).

The more fragmentation is present, the higher the chances of FAT integrity failures with resulting file corruption, as every single read/write cycle to a memory cell (which after many overwrite cycles might have turned flakey) is an added risk, hence negatively impacting upon the overall storage reliability. This is why most camera manufacturers do recommend regular formatting, more as a precaution to aid in file integrity than purely for absolute performance requirements alone.

In your specific case with a relative small memory card (32GB) and relative large recording files of around 0.5GB each (not 585 KB as you accidentially mentioned but 585 MB each) does make for a rather swift FAT allocation as only very few files fit on the card (around 61 as you stated), so any performance hits are quite negligible.

However, in my specific case with a 128GB SD card and only 1 minute recordings at around 6Mbps the avg. recording file size is around 40 MB, resulting in upto 3000+ files on the card. Now it makes a lot more sense to do a format from time to time to reduce fragmentation overheads, which in this scenario are quite measureable indeed.

Taken from here (see bottom of page):
https://www.sdcard.org/developers/overview/speed_class/

Fragmentation and Speed
The memory of a card is divided into minimum memory units. The device writes data onto memory units where no data is already stored. As available memory becomes divided into smaller units through normal use, this leads to an increase in non-linear, or fragmented storage. The amount of fragmentation can reduce write speeds, so faster SD memory card speed standards help compensate for fragmentation. [end quote]

The impact of fragmentation is different for various scenarios, and as such is "relative" for each individual:
ONE hair on the head is relatively little, but ONE hair in the soup is relatively much...

 

[Rajagra]

Newer Windows built in defrag does not consider a file split into large chunks to be fragmented, because it has little effect on performance. That is why it cannot / will not perform a genuine defragmentation. You have to use third party apps instead.

I too used to believe that fragmentation didn't affect performance in solid state memory, because there's no head to move. But refer back to the Crystal Disk Mark image I posted. When SD cards have to read / write randomly, speed is affected EVEN MORE badly than with physical media.

I'll check the card I've been using for a few days and see if there's any fragmentation to speak of.

Sent from my tap-to-talk using Tapatalk

 

[Module 79L]

Newer Windows built in defrag does not consider a file split into large chunks to be fragmented, because it has little effect on performance. That is why it cannot / will not perform a genuine defragmentation. You have to use third party apps instead.

However, I found out that there were always some files that Defraggler can't defrag, but if I run Windows' defragmenter (Win 7) afterwards those files end up being defragmented, as a posterior analysis with Defraggler confirms. So, in a way, they complement one another.

 

[Rajagra]

The last defragger that did the job properly was probably PC Tools for DOS. It defragged everything, including every file and the directory structure itself. It consolidated file space too if you wanted, so all the empty space was one big chunk at the end.
Man it was a thing of beauty, I could watch it for hours on end shuffling those dots around on screen.
In fact I did, it took around two hours to do the full job on a 20MB drive!
But I admit Linux does a more intelligent thing, it leaves space between files for growth. Now that's the kind of thing that would be useful for dash cams, since the file sizes will be roughly but not exactly the same due to compression.

Sent from my tap-to-talk using Tapatalk

 

[Nigel]

Since almost all the files on the card get regularly deleted, the file system should easily be able to keep it defragmented anyway, it is when you get lots of small files spread around the disk that never get deleted that fragmentation starts to become a problem since the size of the free bits gradually gets smaller. My main dashcam card hasn't been formatted since last year.

The wear levelling on a modern flash memory card means that consecutive sectors are not physically adjacent anyway so there is little point defragmenting a memory card unless you are using a sector size that doesn't match the wear levelling size, which would be a bad idea, and you don't want to defragment it anyway because doing so will wear the card out and even when you have got it perfect, the wear levelling will instantly start reorganising it! You do want to format it with the correct sector size that matches the one used by the wear levelling, hopefully the cameras are getting that right?

 

[Rajagra]

This card has had at least 5 days of heavy use, probably overwriting 6 or more times:



Just one fragmented file, pretty damn impressive if you ask me.
Which makes it all the more confusing that I sometimes get a "Non-optimal, format?" warning.

Edit> Hold on, what gives...


Edit2> Auslogics Defrag was skipping fragments over 10MB by default. Disabling this gives the same 329 fragmented file result as Piriform.
P.S. That's every video file.

Nearly all the fragmented files have 3 fragments. This shouldn't be an actual problem, but maybe it is enough to trigger the camera's warning message?

 

[DT MI]

.......files of around 0.5GB each (not 585 KB as you accidentially mentioned but 585 MB each)...

Big Dinner + Good Wine = Bad Math (and a wicked headache this morning)

....
However, your assumption that "FAT integrity is not performance related to any significant degree" is relative: every read/write cycle which the processing unit needs to spend time on to find the next available slot for dumping the next recording is an added overhead, which is in addition to the actual time needed to write the data into the memory cells. With a slower processor and a badly fragmented SD card this can be quite significant as a percentage of overall write performance..
.
.
.
However, in my specific case with a 128GB SD card and only 1 minute recordings at around 6Mbps the avg. recording file size is around 40 MB, resulting in upto 3000+ files on the card. Now it makes a lot more sense to do a format from time to time to reduce fragmentation overheads, which in this scenario are quite measureable indeed.

Taken from here (see bottom of page):
[URL]https://www.sdcard.org/developers/overview/speed_class/[/URL]

[SIZE=5][B][I]Fragmentation and Speed[/I][/B][/SIZE]
[I]The memory of a card is divided into minimum memory units. The device writes data onto memory units where no data is already stored. As available memory becomes divided into smaller units through normal use, this leads to an increase in non-linear, or fragmented storage. The amount of fragmentation can reduce write speeds, so faster SD memory card speed standards help compensate for fragmentation. [end quote][/I]...

I'm going to have to wait until tomorrow to study and digest this (see previous comment ) but it kind of runs contrary to what I've learned/been taught over the years. Not disagreeing but really am not fit to pursue it today.

...But refer back to the Crystal Disk Mark image I posted. When SD cards have to read / write randomly, speed is affected EVEN MORE badly than with physical media....

Which makes no sense at all given there is no physical positioning involved. I use CDM myself but only for comparing different devices (internal vs external drives, etc.) performing the same process rather than the type of comparison you mentioned. Not knowing how CDM does what it does (and programming can make a huge difference) I'm personally leery of using it's result as being gospel truth.

 

[Rajagra]

Which makes no sense at all given there is no physical positioning involved.

Have you ever set up memory settings in a PC's BIOS (e.g. for overclocking)?
There are settings for RAS and CAS - Row Access Strobe and Column Access Strobe.
The address lines are split into parts (RAS and CAS) representing high and low parts. One can be changed much faster than the other. Sequential reads are fast because you only have to change one part of the address, but random reads are slower because you are going all over the place and having to change both parts.
Now in a PC's internal memory these differences are not that huge. But I think with flash memory a similar thing is happening, but it's much more extreme. The act of paging in the required section of storage adds a massive overhead.

There is certainly a more accurate explanation for what's happening, but I'm sure that's the gist of it, and I'm inclined to trust CDM's results.

 

[kamkar]

Big Dinner + Good Wine = Bad Math

I killed my wine collection in a matter of no time during a period with low income, and i found no matter if its red or white and cheap or not i always got massive hangovers when the bulk of my intake had been wine.
And i wasent really prone to hangovers back then.

Was fun going to tastings i came as the 20 YO or so punk i was in my casual clothe thats mainly jeans, and the other people at the tasting was rich looking suit people.

I think i will get me a bottle of the local Vat No°3 Kræn Kræmmer that Jim Murray just rated 93.5 out of 100 in his whiskey bible.
Should ensure a good new years evening for me any my friend that will properly also uncap a good bottle.

If you wonder WTF Thy is, thats a area of Northern Jutland.