Writing an embedded file system
Once upon a time, Julian M. Bucknall wrote an interesting article for The Delphi Magazine. Well, he wrote more than one and they are all interesting and (that's the best part) he's still writing them, but I particularly liked that one.
The article name is Manage The Damage (it had to be Julian's Jimmy Somerville phase) and it was published in Issue 69 (May 2001). Yes, quite an old stuff - at least in computing terms. Inside, Julian described an implementation of an embedded file system i. e. a file system that rests inside another storage (a file in an ordinary file system, for example). That's such a useful concept that even Microsoft recognizes it (think OLE structured storage).
In short, embedded file system (or compound file, or structured storage) allows you to store many unconnected or loosely connected pieces of information inside one physical file in a structured way. For example, it can be used to store different parts of configuration settings for a large application, or it can be used for data acquisition to store many data packets inside one file, or it can be used as a simple database to store text fragments or code snippets.
Although I liked the idea behind Manage The Damage, I couldn't live with the implementation. It was too limited for my taste (embedded file system was limited to 32 MB) and implementation was a mess of pointers, which made the code almost nonportable to .NET.
And then, as all fairy tales start, I decided to write my own implementation.
Why not use OLE structured storage, you'd ask? Well, I like to know how my data is stored (Have you ever tried to look inside an OLE structure storage file with a hex editor? I did. Not a pretty sight.), I want the implementation to be fast and simple to use from Win32 and potentially .NET. Besides that, it sounded like an interesting challenge.
So how did I fare? Good, if I'm the one to answer. There were no pointers killed while writing the code, total size of the file system is limited only to 4 TB (4 GB for files stored inside the compound file) and file internals are easy to understand (well, at least to me ;) ).
The code was used in some commercial projects. Also, GExperts use it for snippet storage (CodeLibrarian.fs file). It seems to be stable and mostly bug-free and as such I'm releasing it to the public, with the usual string attached.
For the brave, here's the code and test suite: GpStructuredStorage.
If you're still interested, continue reading. I'll show few examples on how the GpStructuredStorage can be used.
A small how-to
Next fragment creates compound file and then reopens it for reading. Note that the compound file is implemented as an interface and doesn't need explicit destructor calls as such.
Instead of a file name, one can also send a TStream or descendant to the .Initialize method.
Now that we have the storage interface, we can create a file and then read it.
There is no need to create a /folder manually - every file access automagically creates all required folders.
Still, you are free to do it the old way.
Of course, there is also a FileExists function.
File enumeration is simplified to the max.
(To enumerate folders, one would use FolderNames instead of FileNames.)
Additional information on file or folder can be access via FileInfo property:
Currently, FileInfo only exports file's size (FileInfo.Size) and file attributes (FileInfo.Attribute).
Attributes offer you a way to store additional string info for each file and folder. Unlimited number of attributes can be stored and the only limitation is that both attribute name and value must be stringified.
storage.FileInfo['/folder/file.dat'].Attribute['author'] := 'Gp';
At the end, I must mention that it is also possible to Move and Delete files/folders and Compact (defragment) the file system.
If I have convinced you, go and use the stuff. If not, wait for the next episode.
Next in the embedded file system series
Coming soon to a feed near you.