Archive for June, 2014

How to Fragment Your File System

Here is a tiny little python script to generate file system fragmentation.

“But Tiiiiiiim! Tools are supposed to defragment your filesystem! Why would you ever want a script to fragment one?”

In one of the gadgets I’m working on, I had a need to evaluate disk (well, memory card) performance in real-world and worst-case scenarios. If you are sampling high-speed data with a puny microcontroller, you cannot afford your disk going into lalaland while your puny buffer RAM runneth over. While file fragmentation is – in theory – not a big deal for Flash media as it is for spinning rust drives (no mechanical heads to reposition), your filesystem driver still needs to grovel through the filesystem to find the next free block to write. In a typical implementation, writing to a FAT filesystem with a giant file in the middle of it could incur a significant write hiccup as the FS driver encounters the file and has to seek through its entire FAT chain, potentially fetching and parsing numerous sectors of allocation data before finally finding a free cluster for the next write. This script allows testing of such scenarios.

What it does:

Give it the name of a disk to fragment, and it will begin creating files full of junk data on the disk until it receives a write error (normally indicating the disk is full). It will then delete a random subset of these files, leaving free-space holes scattered throughout the disk. For most filesystems and OSes, the freespace will not be automatically consolidated, and will remain fragmented until the remaining files are deleted (or the disk formatted, etc.) or a defragmentation utility is run. You can then evaluate the performance of your (software, device, etc.) on this disk on a realistic simulation of a well-used drive.

Configuration options:

path – set this to the directory to generate the fragments in. On FAT filesystems, it is necessary to use a subdirectory and not the root directory, due to a limitation on the number of files that can be stored in the root directory.

filldensity – this value, ranging from 0.0 to 1.0, sets the percentage of junk files to remain at the end of operation. A higher value means more files left behind, i.e. less freespace gaps.

minfragsize, maxfragsize – this sets the size range of junk files to create. The size of each file created will be selected at random from this range.

CAVEATS:

I only tested this on a Windows PC, for reasonable file sizes (MB, not GB) and card sizes (a few GB). If your device’s size is measured in rooms, gigaquads or Libraries of Congress, it may not work, or your device may be obsolete by the time it finishes. The “junk” to make the junk files is stored in RAM out of laziness; you probably want to fix this if making multi-GB junkfiles.

This script was written to test a FAT-based device. Not all filesystems respond the same way to fragmentation, so YMMV on other filesystem types.

This emulates fragmentation only. Many other factors could affect your embedded Flash media performance, such as Flash cell wear (aka hot count, or total number of write/erase cycles performed), write amplification, operating temperature and/or voltage (depending on the memory technology and controller), phase of the moon, etc. This script does not emulate any of these other factors. On the bright side, it should be a more faithful test for other memory technologies, e.g. FRAM/MRAM, that are fast and relatively insensitive to cell wear, and will better reflect software delays due to filesystem parsing.

Notes To Myself: ‘Paste Plain Text’ keyboard shortcut/macro for Excel

Very common need: Copy some data into an Excel cell from an arbitrary other source (including another Excel sheet, or webpage, etc.). In the process, strip any external formatting, HTML tables, etc. with extreme vengeance and only paste the plain text.

Traditional way: Mouse fandango (Excel 2013: Home -> Paste -> Paste Special…->Text->OK) for every time you want to do this.

Better: Create “PastePlainEffingText” macro activated by a nice fast keyboard shortcut equivalent to Ctrl-V. Store this macro persistently in the Excel “Personal Workbook”, not the currently open document, so it is available in any open document.

Steps:
1) View -> Unhide -> Personal etc. (The ‘Personal’ workbook is hidden by default. Attempting to save a macro to it generates an extremely helpful message saying to use the ‘unhide’ option, without giving the option to just do this, nor telling you where this setting is.)
2) View -> Record Macro
3) Mousedance as above (Paste Special etc.)
4) View -> Stop Recording
5) Assign keyboard shortcut. I just assigned it to “Ctrl-B” since it’s right next to Ctrl-V. This means I can no longer Ctrl-B to make text bold, but for the once-a-year I’d actually want to do this, it’s a plenty acceptable tradeoff.
6) Optional: Re-hide the “Personal” workbook.

Caveats:

When assigning the keyboard shortcut, the “Ctrl” portion is mandatory and cannot be changed. Excel will automatically insert a ‘Shift’ in addition to this if you happened to type an uppercase letter in the sole letter box provided (they way keyboard shortcuts are usually represented in text). This is somewhat unintuitive and does not mean Excel is blocking you from overwriting an existing shortcut – just change the letter to lowercase and it’ll go away. There is no warning if you do overwrite an existing shortcut, so you’ll have to check on this yourself.

At the time of this writing, Excel does not allow writing an actual macro (code) in the Personal Workbook directly. You have to ‘Record Macro’ and physically do whatever action/mousedance to initially generate the equivalent code. But once this is done, you can edit the actual code. To write/paste an arbitrary code macro, you can probably just “Record Macro” some trivial dummy operation (paste some text, etc.) then just replace the autogenerated code with your own.

The equivalent code for this macro is:

Sub PastePlainEffingText()
'
' PastePlainEffingText Macro
' Strip formatting when pasting buffer contents
'
ActiveSheet.PasteSpecial Format:="Text", Link:=False, DisplayAsIcon:= _
False
End Sub

Notes To Myself: J-Link SWOViewer with Silabs/EnergyMicro EFM32 CPUs

The EFM32 SWO port operates from a 14MHz timebase regardless of the current CPU frequency. Autodetection of actual frequency is feasible, but irrelevant. Manually specify 14MHz for “CPUFreq” in SWO Viewer. The corresponding calculated SWOFreq should be 900KHz. Tested and working as of SWOViewer version 4.84f.