TDM Rerun #15: Many Faces Of An Application

That all sounds easy, but how can we combine the windows (forms-based) aspect of an application with something completely different, for example an SvCom-based service application? The problem here is that the GUI part of an application uses forms while the SvCom service is based on another Application object, based on the SvCom_NTService unit. How can we combine the GUI Application.Initialize (where Application is an object in the Forms unit) with a service Application.Initialize (where Application is an object in the SvCom_NTService unit)? By fully qualifying each object, of course. - Many Faces Of An Application, The Delphi Magazine 107, July 2004 In the 2004 July issue I described an approach that allows the programmer to put multiple application front-ends inside one .exe file by manually tweaking the project’s .dpr file. This is the technique I’m still using in my programs. For example, most of the services I write can be configured by starting the exe with the /config switch. Links: article (PDF, 126 KB), source code (ZIP, 1 MB).

TDM Rerun #14: A Portable XML

The code unit, OmniXML.pas, which contains the XML representation interfaces, parser and writer, was written by a single programmer, Miha Remec (he is also the guy behind the www.omnixml.com website). He started writing it in 2000, because he was missing a native Delphi DOM parser, one that would represent the DOM the same way as it was designed. The best Delphi parser around at that time was OpenXML, but it used classes to represent XML elements, not interfaces. OmniXML uses interfaces, derived from the IXMLNode (as specified by the DOM). That also makes it almost completely compatible with the MSXML parser, which uses the same approach. - A Portable XML, The Delphi Magazine 105, May 2004 In the 2004 May issue I wrote about OmniXML, a native Delphi XML parser. I described the OmniXML approach and wrote few short pieces of code that demonstrated its use. Today, five years later, OmniXML is still strong and I’m still using it, as you can see in my Fluent XML series. Links: article (PDF, 45 KB), source code (ZIP, 795 KB).

TDM Rerun #13: Shared Events, Part 2: Redesign

Now we can already guess where the general sluggishness of the shared event system comes from. The trouble lies in the constant XML loading and saving. Most of the shared event system tables are quite static, but that doesn’t hold for the Event Queue table, where new entries are inserted, modified and deleted all the time. - Shared Events, Part 2: Redesign, The Delphi Magazine 102, February 2004 My second article on shared events architecture first addressed speed issues (original code was quite slow), then discussed internals of shared counters, shared linked lists and shared tables (all of which are used in the shared events system) and at the end returned to fine-tuning by fixing some remaining speed issues. As you can expect, the basis for the tuning was hard data from the profiler, not some wave-of-hand ideas about where the problem maybe lies. Links: article (PDF, 99 KB), source code (ZIP, 1,9 MB), current version.

TDM Rerun #12: Shared Events

Shared event system, as I nicknamed this approach, is implemented as a shared set of in-memory tables, which are accessed and manipulated by producers and listeners. The important part is that there is no dedicated server: housekeeping is distributed between the producers and listeners. - Shared Events, The Delphi Magazine 97, September 2003 Shared events mechanism was definitely the most complicated architecture based on shared memory I ever put together. The system allowed multiple programs (running on the same computer) to cooperate using an event-based system. First program would publish an event (or more events) and others would subscribe to those events. First program would then broadcast the event, which would trigger notifications in all subscribed programs. The best trick was that there was no privileged part – no server, service or manager. Publishers and consumers shared all the work – tables were created as needed, housekeeping was done on both sides and so on. Underlying architecture was largely redesigned after this article was published. Original source files are included only for historical reference. Links: article (PDF, 496 KB), source code (ZIP, 1,9 MB), current version.

TDM Rerun #11: Shared Pools

Sometimes we run into problems because of a Windows feature, and most of the time this is a very good feature, that a file mapping (an essential part of my shared memory implementation) cannot exist on its own. A file mapping, like mutexes, events, and other Windows primitives, must have an owner. If all processes associated with a given file mapping die, the file mapping will be destroyed. Because in my shared memory implementation this file mapping is backed with a page file, its contents won’t be preserved on an accessible part of the disk. - Shared Pools, The Delphi Magazine 95, July 2003 The shared pool was one of my more baroque creations. In fact, it was so complicated that it was never used in a deployed application. Basically, the article described an architecture to implement a pool of shared memory objects, which multiple Writers could use to send data to one Reader (typically sitting in another process). The system also handled cleanup when a Reader task died and other management details. You really should not be playing with this code. There are better solutions. Links: article (PDF, 193 KB), source code (ZIP, 1.9 MB)

TDM Rerun #10: Synchronisation Toolkit Revisited

The plan for this article was to present an implementation of a shared memory pool: a mechanism that allows multiple data producers to send data to one data manipulator. In fact, I already had all the code and half the article completed when I found out that my solution doesn't always work. Everything was OK when both ends of the pool (producer and manipulator) were implemented in the same application or in two 'normal' applications. But if I tried to change one of those apps into an NT service, the shared pool stopped working. - Synchronisation Toolkit Revisited, The Delphi Magazine 91, March 2003 My 10th TDM article returned to my pet theme - synchronisation and communication. First part described some aspects of the NT security model and second implemented shared memory-based queue, capable of transporting messages from a GUI application to a service and back. It looks like I'll have to return to that topic again - I got some reports recently that my approach doesn't (always?) work on Vista :( Links: article (PDF, 54KB), source code (ZIP, 2.2 MB), current GpSync unit

TDM Rerun #9: My Data Is Your Data

There is no multiprocess solution without data sharing. If you have to do something useful and need more than one application to do it, you need a way to share data between those applications. - My Data Is Your Data, The Delphi Magazine 88, December 2002 In TDM #88 my synchronisation & sharing theme continued with an article on shared memory. The article first demonstrated how shared memory can be implemented with file mapping Win32 API and then proceeded with the description of a wrapper that simplified this operation. This wrapper also added some useful extensions, as are text, stream and array access to the shared memory. GpSharedMemory is still alive and well and used in many projects. Since the 2002 article it gained resizable shared memory, shared stream support and shared linked list. Links: article (PDF, 60 KB), source code (ZIP, 50 KB), current GpSharedMemory unit

TDM Rerun #8: A Synchronisation Toolkit

Every programmer accumulates reusable code over the years and I am no exception. Sometimes, parts of that code seem generic enough to be worth documenting and presenting it to the public. Today, I'd like to put on display four classes belonging to the inter-process synchronisation part of my personal toolbox. - A Synchronisation Toolkit, The Delphi Magazine 86, October 2002 In October 2002 I presented my GpSync unit, which is still very actively updated and developed. At the time of writing, there were four reusable classes in this unit - a token, a group, a counted group and single-write-multiple-readers reimplementation copied (and translated from C to Delphi) from Advanced Windows by Jeffrey Richter. Since that first public occurrence, GpSync was extended with a flag, a counter, a message queue (shared-memory based, can be used to send messages from an interactive application to a service and back), and a circular buffer. To this day, it represents most common building blocks I'm using in multithreaded applications.   Links: article (PDF, 148 KB), source code (ZIP, 196 KB), current GpSync unit

TDM Rerun #7: File Sharing on Linux

In the Windows world we are used to file locking. It is something natural, something that permeates the operating system, something that we had to learn when we stopped using DOS. But for all practical purposes, Linux is Unix, and Unix was designed with different things in mind, one of which was openness and sharing. From the very old days, Unix has supported different kinds of file systems, including those that span a network. Remember, that was well before Windows, and file locking was something that was simply too expensive (in terms of time and network traffic) to be implemented. And that is why there is no locking on Linux. - File Sharing on Linux, The Delphi Magazine 84, August 2002 The year was 2002, Kylix was still alive and I was planning to write some software for the new platform. Sadly, that never happened - other (Windows) projects were more important and before I came back to Kylix, it was more or less dead. Still, this short excursion was responsible for my only Linux article. I took my  GpFileSync unit from TDB 68: Let's Cooperate and tried to adapt it to Linux. As it turned out, this was nearly an impossible mission. The ideas behind Unix file systems almost totally prevent its abuse in such direction. To make myself clear - it is certainly possible to use file system for locking, but there are big problems where you want to a) make the lock disappear if the program terminates abnormally, b) put the lock file on a networked location (NFS) or c) all of the above. I ended with three less than perfect solutions. First used a simple lock file, second was an adaptation of the first that worked with the NFS 2 volums (both were not able to auto-delete the file-mutex on program termination) and third used fcntl() call (auto-delete worked well, but NFS was not fully supported). As far as I know, none of them was ever tested in a real-world application. Links: article (PDF, 125 KB), source code (ZIP, 16 KB), current GpFileSync unit

TDM Rerun #6: GExperts 1.0

The greatest (in my view), and one of the oldest of the Delphi IDE expert suites is GExperts. It was created by Gerals Nunn (in 1997, if I am not mistaken), who also developed it for almost two years. Gerald has moved to other projects, but he kindly donated the GExperts source code to the online community. - GExperts 1.0, The Delphi Magazine 72, August 2001 I still stand by those words. GExperts is a great suite of experts and I still use it every day. I also wrote few experts that are included in the GExperts pack and few that are included only in my private build. Links: article (PDF, 95 KB), GExperts

TDM Rerun #5: SvCom 4.0

When you get into really serious Windows development, you may well need to create NT service applications. Borland's input is half-hearted: Delphi includes support for NT service application writing, but as you work with this framework, you'll soon discover its less-polished parts! - SvCom 4.0, The Delphi Magazine 69, May 2001 In the TDM #69 I reviewed SvCom, "an integrated package of classes and tools for development of service applications and use of NT Security framework." (as the current SvCom web page states). Good stuff, I'm still using it for service development. Links: article (PDF, 67 KB), SvCom

TDM Rerun #4: Let's Cooperate

Every serious Windows programmer who has developed at least one multi-threaded application knows about thread synchronisation. You know what I mean: critical sections, mutexes, semaphores, signals and so on. ... But to write truly distributed applications, running on different computers, we need new tools. - Let's Cooperate, The Delphi Magazine #68, April 2001 I don't remember what was the exact reason for this article, but obviously I was working on some multi-process multi-computer solution and I needed a way to synchronise those computers. Of course, once I started writing the article, it got completely out of hand and I finished with file system based mutex, critical section, group, event, and even with a message queue and single-writer-multiple-readers synchronisation primitive. There were also many state diagrams like the one of the right dispersed through the article. Poor readers. >:-( The plan was to make most of them work on Linux as well, but as I discovered in File Sharing on Linux (not yet published on my blog), but as I found later, this was nearly impossible to do as Linux implements file locking in completely different way. I never used those primitives much as TCP/IP communication became much more prominent in later years. Still, they can be very useful if you want to quickly put together a multi-computer demo or quick and dirty solution. The GpFileSync unit never made it to my web so I'm linking to the current version at the bottom of this article. Links: article (PDF, 117 KB), source code (ZIP, 47 KB), current GpFileSync unit

TDM Rerun #3: Time Is The Simplest Thing

'Time, I'll tell you about time. Time is the simplest thing.' - Time Is The Simplest Thing, The Delphi Magazine #65, January 2001 [quote from Clifford D. Simak's The Fisherman] Now this is one of my favorite articles. It deals with time and more specifically with time zones. As most of my articles this one also builds on real-world problems I encounter during my programming practice. In this case, it was motivated by a big coding flop which taught me an important lesson: "Never store local time in a database." [No, I still don't want to recall all the details. It was related to daylight savings time transitions, that's all I'm willing to tell.] Anyway, I fixed the problems in the abovementioned code and during that process I found out that there is almost no support for time zones built in Delphi and that Windows APIs on that area are really not that simple to use. So I built a time zone manipulation library that handles time zone information access and time zone conversions with daylight savings time support. This GpTimezone unit is still alive and I'm using it regularly - mostly the NowUTC function. The article also dealt with the (now happily forgotten) Swatch Time. Also included was a neat utility (if I'm allowed to say so) for timezone conversions, which I'm still using today without any changes to the original code.   Links: article (PDF, 134 KB), source code (ZIP, 162 KB), current GpTimezone unit

TDM Rerun #2: A Better Build Process

I had to live with the problem. Until the day when it occurred to me that I could very probably find the solution to my problem on the internet. So I connected to Deja News and searched. - A Better Build Process, The Delphi Magazine #49, September 1999 A life without Google? Was that really possible? I can hardly remember those times. The article was written in Delphi 4 times and deals with building Delphi applications from a command line. More specifically, it solves the problem of dynamically recreating project.res file, which is something that you have to do if you want to increment build number in the version info resource during the build process. (A utility to increment build number was included, too.) Also on display was a batch file that built my trusty GpProfile profiler and a "very complicated" tail utility. {$APPTYPE Console} program tail; uses SysUtils, Classes; var str : TStringList; i : integer; first: integer; begin str := TStringList.Create; try str.LoadFromFile(ParamStr(1)); first := str.Count-StrToIntDef(ParamStr(2),20); if first < 0 then first := 0; for i := first to str.Count-1 do Writeln(str[i]); finally str.Free; end; end. Not much has changed in the Delphi world - I would code it almost identical today. Links: article (PDF, 70 KB), source code (ZIP, 185 KB)

TDM Rerun #1: Safer Sockets

Instead of fixing the ScktComp unit it is probably better to create a derived class, override the Read method, paste code from ScktComp and make this small modification. Choose your way but beware: CancelIO is not available in Windows 95, only in 98 and NT 4. - Safer Sockets, The Delphi Magazine #44, April 1999 Those were the times ... Most of computers were still running Windows 95, the actual Delphi version was 4 and we were using TWinSocketStream for data transfer over Windows sockets. Oh, happy days. Not! I won't comment this article much. It describes how to fix a problem with an obsolete component that should never be used in modern software. I still remember how happy I was switching to ICS ... Still, I promised all my articles and I have to start with this one. This article represents another important milestone in my 'journalist' career - it was my first article written in English language and a start of very happy long-term relationship with The Delphi Magazine. Links: article (PDF, 49KB), source code (ZIP, 1 KB)

Reposting TDM Articles

Like some other authors (Hallvard Vassbotn, Bob Swart) I'll be republishing articles I wrote for The Delphi Magazine. [Chris, thanks for the permission!] Unlike Dr. Bob and Hallvard I won't have problems with choosing from my TDM articles - I published much less than them so I'll simply republish everything :) In the next months you can expect online versions of my articles: Safer Sockets [TDM #44] Describes an issue with TWinSocketStream and a presents a workaround. A Better Build Process [TDM #49] How to build Delphi programs from a command line; introduces some helper tools. Time Is The Simplest Thing [TDM #55] The art of timezones. Introduces GpTimezone unit, which is still alive and regularly updated. Let's Cooperate [TDM #68] My first take on steampunk process synchronisation - not via OS primitives but by using file system. Introducing GpFileSync unit, which somehow never made it to my web site. SvCom 4.0 [TDM #69] A review of SvCom service development package. Still alive and well, currently at version 7. GExperts 1.0 [TDM #72] A review of Delphi experts we all love. File Sharing On Linux [TDM #84] My only article on Kylix continued the Let's Cooperate theme. A Synchronisation Toolkit [TDM #86] Simple multi-process synchronisation and communication primitives. Introduces the GpSync unit, which is sill alive and regularly updated. My Data Is Your Data [TDM #88] Describes an easy to use wrapper around Windows' shared memory. Introduces GpSharedMemory unit. Synchronisation Toolkit Revisited [TDM #91] An upgrade of the GpSync unit, introducing message queue. Shared Pools [TDM #95] Shared memory pools. I never developed this concept any further. Shared Events Part 1: Rationale, Design, Implementation [TDM #97] Cross-process event dispatch using shared memory. Introduces GpSharedEvents unit. Shared Events Part 2: Redesign [TDM #102] Implementation details and a redesign. A Portable XML [TDM #105] A review of the OmniXML library. Still alive and regularly updated. Many Faces Of An Application [TDM #107] How to write an application that can be a service, Windows GUI and system tray icon monitor. Thread Pooling, The Practical Way [TDM #112] Describes Windows thread pool, various problems with it and some workarounds. Put It In A Tree [TDM #118] Discusses various approaches to creating a tree structure from a list of mail messages. To Manage Memory [TDM #126] An introduction to memory management algorithms and structures plus a short overview of the (at that time very young) FastMM memory manager. And that's all folks. I'll probably attack this list in chronological manner. If you have any preferences I may make an exception and change priorities so don't be afraid to ask for article you want to read first.