The Delphi Geek

TDM Rerun #8: A Synchronisation Toolkit

2008-05-01T15:48:00.001+02:00

Every programmer accumulates reusable code over the years and I am no exception. Sometimes, parts of that code seem generic enough to be worth documenting and presenting it to the public.

Today, I'd like to put on display four classes belonging to the inter-process synchronisation part of my personal toolbox.

- A Synchronisation Toolkit, The Delphi Magazine 86, October 2002

In October 2002 I presented my GpSync unit, which is still very actively updated and developed. At the time of writing, there were four reusable classes in this unit - a token, a group, a counted group and single-write-multiple-readers reimplementation copied (and translated from C to Delphi) from Advanced Windows by Jeffrey Richter.

Since that first public occurrence, GpSync was extended with a flag, a counter, a message queue (shared-memory based, can be used to send messages from an interactive application to a service and back), and a circular buffer. To this day, it represents most common building blocks I'm using in multithreaded applications.

Links: article (PDF, 148 KB), source code (ZIP, 196 KB), current GpSync unit

Enumerating XML nodes

2008-04-21T22:10:00.001+02:00

I just submitted an update to OmniXmlUtils.pas - a helper library for the Delphi implementation of the XML DOM model, the OmniXML. It allows you to walk over child nodes with a simple enumerator (but you already saw that coming, huh?).

In OmniXML (and in MS XML on which the OmniXML interface is based), you access child nodes via IXMLNodeList interface. Actually, when you access any subset of nodes (for example, when you call SelectNodes and pass it an XPath expression), you are using IXMLNodeList. More and more I was using it, more I hated its stupid interface.

  IXMLCustomList = interface
    function GetLength: Integer;
    function GetItem(const Index: Integer): IXMLNode;
    property Item[const Index: Integer]: IXMLNode read GetItem;
    property Length: Integer read GetLength;
  end;

  IXMLNodeList = interface(IXMLCustomList)
    procedure Reset;
    function NextNode: IXMLNode;
  end;

[I removed declarations of all functions that are not used when walking over the list.]

Basically, I have two options with the current design. I can use a standard for loop using Length and Item[] or I can use NextNode and while loop.

Or, of course, I can write an enumerator - and that's what I did. XMLEnumNodes takes an IXMLNodeList and wraps it in an enumerator. Even better - it takes an xml node or document and an XPath expression and runs SelectNodes automatically for me so I can write code like this:

for nodePassword in XMLEnumNodes(xmlConfig, '//*/PasswordHash') do
  SetTextChild(nodePassword, '(removed)');

Now that's what I call a nice code!

If you can't wait for tomorrow update to the OmniXML daily snapshot, you can download OmniXMLUtils 1.25 here.

TDM Rerun #7: File Sharing on Linux

2008-04-01T13:06:00.002+02:00

In the Windows world we are used to file locking. It is something natural, something that permeates the operating system, something that we had to learn when we stopped using DOS. But for all practical purposes, Linux is Unix, and Unix was designed with different things in mind, one of which was openness and sharing. From the very old days, Unix has supported different kinds of file systems, including those that span a network. Remember, that was well before Windows, and file locking was something that was simply too expensive (in terms of time and network traffic) to be implemented. And that is why there is no locking on Linux.

- File Sharing on Linux, The Delphi Magazine 84, August 2002

The year was 2002, Kylix was still alive and I was planning to write some software for the new platform. Sadly, that never happened - other (Windows) projects were more important and before I came back to Kylix, it was more or less dead.

Still, this short excursion was responsible for my only Linux article. I took my GpFileSync unit from TDB 68: Let's Cooperate and tried to adapt it to Linux. As it turned out, this was nearly an impossible mission. The ideas behind Unix file systems almost totally prevent its abuse in such direction. To make myself clear - it is certainly possible to use file system for locking, but there are big problems where you want to a) make the lock disappear if the program terminates abnormally, b) put the lock file on a networked location (NFS) or c) all of the above.

I ended with three less than perfect solutions. First used a simple lock file, second was an adaptation of the first that worked with the NFS 2 volums (both were not able to auto-delete the file-mutex on program termination) and third used fcntl() call (auto-delete worked well, but NFS was not fully supported). As far as I know, none of them was ever tested in a real-world application.

Links: article (PDF, 125 KB), source code (ZIP, 16 KB), current GpFileSync unit

GpLists emergency update

2008-03-27T08:43:00.001+01:00

TGpInt64List.Move was totally broken. Great thanks go to 'Intra' for noticing it.

Fixed now.

Walking the key-value container

2008-03-21T11:31:00.001+01:00

The recent discussion in comments to my latest articles (Fun with enumerators - Walking the (integer) list, On bad examples and smelly code) caused a shift in my perspective. I was always treating my TGpIntegerObjectList and TGpInt64ObjectList as lists with some baggage attached, but in practice I'm almost never using them that way. Most of the time I treat them as a background storage for key-value pairs.

What's the difference? Most of the time, I don't care about item indices. When I use my lists as containers, I never need to know where in the list some (Key, Value) pair is stored. OK, almost never. When I'm deleting from the list, I sometimes use the index, just for the performance purposes. And when I access the Value part, I have to find the index with IndexOf and then use it to reference the Objects property. There are probably other cases too - but that's something that you just have to take into account if you want to use a list as a storage media.

From time to time I'm extending lists in my GpLists unit with small wrappers that help me not to use list indices in some specific situation. Today, I used the new Walk enumerator in some code and asked myself: "Why does it have to return list index? Why couldn't it return a (Key, Value) pair?"

Good question. Why couldn't it? It turns out that it could.

A little enumerator that could

Let's set up some assumptions first. Assume that I have this pointless list.

il := TGpIntegerObjectList.Create;
il.AddObject(1, TGpString.Create('one'));
il.AddObject(2, TGpString.Create('two'));
il.AddObject(3, TGpString.Create('three'));

Further assume that I want to walk over it and display both number and text for each item. I can do this with a standard loop.

for idx := 0 to il.Count - 1 do
  Log('%d: %s', [il[idx], TGpString(il.Objects[idx]).Value]);

Or with my index-returning walker.

for idx in il.Walk do
  Log('%d: %s', [il[idx], TGpString(il.Objects[idx]).Value]);

But what I'd really like to do is.

for kv in il.WalkKV do
  Log('%d: %s', [kv.Key, TGpString(kv.Value).Value]);

Or even better.

for kv in il.WalkKV do
  Log('%d: %s', [kv.Key, kv.StrValue]);

In other words, I want to return not a single item, but a pair of items from the enumerator. Of course, Delphi expressions can only return a single result and not a tuple so we have to provide a wrapper for enumerated (Key, Value) pairs. We have to pack them in a record, class or interface.

Getting all self-destructive

My first idea was to return an interface to a key-value object from the enumerator.

  IGpKeyValue = interface
    function GetKey: int64;
    function GetValue: TObject;
    property Key: int64 read GetKey;
    property Value: TObject read GetValue;
  end; { IGpKeyValue }

  TGpKeyValue = class(TInterfacedObject, IGpKeyValue)
  private
    kvKey: int64;
    kvValue: TObject;
  protected
    function  GetKey: int64;
    function  GetValue: TObject;
  public
    constructor Create(key: int64; value: TObject);
    property Key: int64 read GetKey;
    property Value: TObject read GetValue;
  end; { TGpKeyValue }

  TGpIntegerObjectListWalkKVEnumerator = class
    //...
    function  GetCurrent: IGpKeyValue;
    property Current: IGpKeyValue read GetCurrent;
  end; { TGpIntegerObjectListWalkKVEnumerator }

function TGpIntegerObjectListWalkKVEnumerator.GetCurrent: IGpKeyValue;
var
  idx: integer;
begin
  idx := wkeListEnumerator.GetCurrent;
  Result := TGpKeyValue.Create(wkeListEnumerator.List[idx],
    TGpIntegerObjectList(wkeListEnumerator.List).Objects[idx]);
end; { TGpIntegerObjectListWalkKVEnumerator.GetCurrent }

That surely works fine, but guess what - it's incredibly slow. I wouldn't expect anything else - after all an object is allocated for every enumerated value, plus all that complications with interface reference counting...

I did some testing, of course. Thousand iterations over a list with 10.000 elements. Results are quite interesting. 5 milliseconds for a standard for loop. 50 milliseconds for my Walk enumerator. 5 seconds for interface-based WalkKV. Ouch! Let's return to the drawing board...

One allocation is enough

My next idea was to return not an interface, but an object. When you return an object, you actually return a pointer to the object data, which is quite fast. It would not help much if I would recreate this object every time the GetCurrent is called, but luckily there is no real reason to do that. I can create the object when enumerator is created and destroy it when enumerator is destroyed.

  TGpKeyValue = class
  private
    kvKey  : int64;
    kvValue: TObject;
  public
    property Key: int64 read kvKey write kvKey;
    property Value: TObject read kvValue write kvValue;
  end; { TGpKeyValue }

  TGpIntegerObjectListWalkKVEnumerator = class
  private
    wkeCurrentKV: TGpKeyValue;
    wkeListEnumerator: TGpIntegerListWalkEnumerator;
  public
    constructor Create(aList: TGpIntegerObjectList; idxFrom, idxTo: integer);
    destructor  Destroy; override;
    //...
    function  GetCurrent: TGpKeyValue;
    property Current: TGpKeyValue read GetCurrent;
  end; { TGpIntegerObjectListWalkKVEnumerator }

constructor TGpIntegerObjectListWalkKVEnumerator.Create(aList: TGpIntegerObjectList;
  idxFrom, idxTo: integer);
begin
  inherited Create;
  wkeListEnumerator := TGpIntegerListWalkEnumerator.Create(aList, idxFrom, idxTo);
  wkeCurrentKV := TGpKeyValue.Create;
end; { TGpIntegerObjectListWalkKVEnumerator.Create }

destructor TGpIntegerObjectListWalkKVEnumerator.Destroy;
begin
  FreeAndNil(wkeCurrentKV);
  FreeAndNil(wkeListEnumerator);
  inherited;
end; { TGpIntegerObjectListWalkKVEnumerator.Destroy }

function TGpIntegerObjectListWalkKVEnumerator.GetCurrent: TGpKeyValue;
var
  idx: integer;
begin
  idx := wkeListEnumerator.GetCurrent;
  wkeCurrentKV.Key := wkeListEnumerator.List[idx];
  wkeCurrentKV.Value := TGpIntegerObjectList(wkeListEnumerator.List).Objects[idx];
  Result := wkeCurrentKV;
end; { TGpIntegerObjectListWalkKVEnumerator.GetCurrent }

BTW, you can see another trick in this implementation - enumeration by delegation. I'm reusing my Walk enumerator to do the actual walking.

That works much faster than the interface-based version - 300 ms for my test case. It's still six times slower than the Walk enumerator, though, and it is not really obvious why the difference is so big. Leave that be, for a moment.

The third approach would be to use a record to store the current (Key, Value) pair. Then there is no allocation/deallocation at all, but resulting code is not faster. Record-based enumerator needs about 500 ms to run the test case.

This slowdown occurs because record is not returned as a pointer, but as a full copy. In the class-based scenario, GetCurrent returns a pointer to the TGpKeyValue object and that pointer is passed in a register. In the record version, GetCurrent returns not a pointer to the record, but the record itself - it copies full record to the stack so the caller can use this copy - and that is waaaay slower.

The speed difference

Let's return to that major speed difference between Walk and WalkKV. I looked at the code, but couldn't find any good reason. Then I turned to the CPU view and it was evident. The problem lied not in the enumerator, but in my poorly designed benchmarking code :(

You see, I was timing multiple repetitions of these three loops:

for idx := 1 to 10000 do 
  ;

for idx in il.Walk do
  ;

for kv in il.WalkKV do
  ;

Three empty loops, so I'm timing just the enumeration, yes? No!

First loop just runs from 1 to 10000. Trivial job and compiler will generate great code.

Second loop does the same, but with more overhead.

Third loop does much more than that. It also accesses il[] and il.Objects[] (inside the GetCurrent).

In reality, code inside the for statement in the first two cases would have to access il[] and il.Objects[] by itself. The code inside the third for statement has no need for that - it gets data neatly prepared in kv.Key and kv.Value.

I changed the loops to:

for idx := 0 to il.Count - 1 do begin
  il[idx];
  obj := il.Objects[idx];
end;

for idx in il.Walk do begin
  il[idx];
  obj := il.Objects[idx];
end;

for kv in il.WalkKV do begin
  kv.Key;
  obj := kv.Value;
end;

I'm just accessing and throwing the Items[]/Key information away and then I copy the Objects[]/Value information into the local variable. All loops are now running comparable jobs.

Results are now significantly different. Simple for loop needs 300 ms to run the test, Walk version needs 310 ms (really small difference) and WalkKV version needs 470 ms. It is still slower, but by less than the factor of two. If there would be a real code inside the for loop, the difference would be unnoticeable.

Morale? You should benchmark, but you should also check that you're benchmarking the right thing!

Syntactic sugar

The generic version of WalkKV only supports this kind of code:

for kv in il.WalkKV do
  Log('%d: %s', [kv.Key, TGpString(kv.Value).Value]);

But there's a really neat trick to change this into

for kv in il.WalkKV do
  Log('%d: %s', [kv.Key, kv.StrValue]);

without modifying the WalkKV itself. Can you guess it? Class helpers, of course.

You just have to declare a simple helper in the WalkKV consumer (no need to change the WalkKV itself) and you can use the StrValue instead of TGpString(kv.Value).Value.

  TGpKeyStrValue = class helper for TGpKeyValue
  public
    function StrValue: string; inline;
  end; { TGpKeyStrValue } 

function TGpKeyStrValue.StrValue: string;
begin
  Result := TGpString(Self.Value).Value;
end;

Conclusion: class helpers are great. Key-value walker is useful. Enumerator-induced performance loss is negligible. Enumerators are fun.

---

I've tagged this and all previous enumerator-related articles with the enumerators label so you can now access them all at once via this simple link.

On bad examples and smelly code

2008-03-16T23:00:00.001+01:00

I've received lots of criticism on my latest article on enumerators. For the convenience of my readers, here are some excerpts.

Removal loops should be iterated backwards to prevent index shuffling.

... while I can certainly see the usefulness in your Slice- and Walk-enumerators ... I don't think they're the best solution to the two examples you quoted.

Simplicity does not justify an example that fails to work correctly. A bad example is still a bad example, no matter how simple.

I'd also like to echo the other comments w.r.t your initial example. Reversing the direction of the iteration is the simplest wasy to simplify the exemplar task, not invoking enumerators and the such like.

They can be summarized in one sentence: "Your examples suck." The main rationale for that reasoning seems to be the accepted way to delete stuff from a TList/TObjectList/TAnyList - start at the end and proceed backwards.

Well, I have two things to say. [I apologize for responding in a new post, but I feel that this is an interesting topic by itself.]

1. Yes, my examples are simplified. Terribly so. Maybe they are even simplistic. At least some part of my readers seems to think so. But that's just the way I like them.

I'm not trying to enforce my ideas on my readers. I'm only displaying them to the public. That's why I tend to minimize all my examples. I'll show you the code, I'll show you the usage, but you should think about the solution and how it applies to your problems. Maybe you'll see that it can be useful and you'll adopt it. Maybe not. Maybe you'll remember it a year from now and suddenly see the light. Maybe you'll use it for a month or two and then decide that I'm a twit and that your previous ways were better. [In this case, please refer to the last paragraph in this post.] I don't really care. I'm giving my ideas out, I'm glad of every feedback and I'm happy if somebody else is using my code. I've received a lot from the Delphi community in past ten years and I like to give back as much as I can.

2. I don't like the 'traverse the list backwards and all will be well' approach to list deletion. I have a really strong aversion to it. It looks fine, it works fine but it smells really bad.

I was thinking a lot about why I dislike it so much and I think that the main reason is that it relies too much on the list internals. Admittedly, this is a very well documented behavior, which will never change in the future. Let's try again - it relies too much on the internals of the underlying structure. If at some point in the future you change the structure, this assumption may fail. OK, it is hard to imagine an underlying structure that would behave almost the same as the TList but would behave differently on Delete, but what if you just forgot to adapt this part of the code to the new structure? Unlikely, I agree.

People will complain that I can't give a strong reason for my fears and they will be correct. I don't have any good cause against this kind of list cleanup.

You see, if I learned something in my programming career, it's that some code should never be used. When you program, some things work out and some not. Some code works fine from the beginning and some causes problem after a problem and bug report after a bug report. A good programmer recognizes that and stops using the code/fragments/patterns that are likely to cause problems in the future. Somehow I have classified the 'downto deletion' in the same category. I don't remember why. That's why I'm not saying it is bad, only that it smells bad.

The other reason is the code readability. If find the code that explicitly states its intention more readable and more maintainable. Your point may, of course, differ.

---

May my ramblings not stop you from commenting. I'm happy when I receive comments that point to my errors, however justifiable they may be. Even if I don't agree with them, they make me rethink my strategies so at the end I'm even more sure in my approach.

Flame on!

Fun with enumerators - Walking the (integer) list

2008-03-14T12:25:00.001+01:00

Do you hate such code?

    idx := 0;
    while idx < il.Count do
      if ShouldBeRemoved(il[idx]) then
        il.Delete(idx)
      else
        Inc(idx);

If you do, read on!

TGpIntegerList [For the purpose of this blog entry, TGpIntegerList token is redefined to implicitly expands to "TGpIntegerList, TGpInt64List, and all classes derived from them."] included enumeration support since Delphi 2005 times. You could, for example, write

  il := TGpIntegerList.Create([1, 3, 5, 7]);
  try
    for i in il do
      Log('%d', [i]);
  finally FreeAndNil(il); end;

and get numbers 1, 3, 5, and 7 in the log.

Slicing ...

Recently, I've extended TGpIntegerList with two additional enumerators, both (of course) springing from practice. I've noticed that I still write lots of 'old school' for loops because I want to do something special with the first element. For example, take this simple code fragment that finds a maximum element in the list.

    max := il[0];
    for idx := 1 to il.Count - 1 do
      if il[idx] > max then
        max := il[idx];

To do such thing with for-each, you have to introduce a boolean flag.

    gotMax := false;
    for i in il do
      if not gotMax then begin
        max := i;
        gotMax := true;
      end
      else if i > max then
        max := i;

Ugly.

Much nicer implementation can be written with the new Slice() enumerator.

    max := il[0];
    for i in il.Slice(1) do
      if i > max then
        max := i;

Slice takes two parameters representing starting and ending index in the list and returns all values between them (indices are inclusive). Ending index can be omitted in which case it will default to Count-1 (i.e. to the end of the list).

Slice enumerator is implemented in a traditional manner - with the enumerator factory. For more details see my older articles and GpLists source code.

... And Walking

The second and much more interesting enumerator solves the problem shown in the introduction to this article. I noticed that I write lots of code that iterates over some list and removes some entries while keeping others intact. Typical scenario: removing timed-out clients from the server's list.

Something like that is impossible to do with the TGpIntegerList default enumerator. Firstly because this enumerator returns list elements and not list indices (and we typically have to access the object stored in the .Objects[] part of the list during the process) and secondly because the enumerator does not work correctly if enumerated list is modified behind its back. The second issue also prevents the use of standard for loop.

To solve both problems at once, I've introduced the Walk enumerator. It returns list indices instead of list elements and functions correctly if Add, Insert or Delete are used on the list while enumerator is active. (The second part is implemented by using TGpIntegerList internal notification mechanism, introduced just for this purpose.)

Now I can write:

    for idx in il.Walk do
      if ShouldBeRemoved(il[idx]) then
        il.Delete(idx);

Implementation? Check the source, Luke. It's not complicated.

Now if only I could add such enumerator to the TObjectList ...

Gp* update

2008-03-14T10:32:00.001+01:00

GpHugeFile 5.05

Delphi 2007 changed the implementation of CreateFmtHelp so that it clears the Win32 'last error'. Therefore, it was impossible to detect windows error in detection handler when EGpHugeFile help context was hcHFWindowsError. To circumvent the problem, all EGpHugeFile help contexts were changed to a negative value. HcHFWindowsError constant was removed. All Win32 API errors are passed in the HelpContext unchanged. IOW, if HelpContext > 0, it contains an Win32 API error code, otherwise it contains one of hcHF constants.
Added method TGpHugeFile.GetTime and property TGpHugeFile.FileTime.

GpStreamS 1.22

Added AppendToFile helper functions (two overloads).
Added ReadTag and WriteTag support for int64 and WideString data.
Added two overloaded SafeCreateFileStream versions returning exception message.
Added Append stream helper.
Added AutoDestroyWrappedStream property to the TGpStreamWindow class.
Added TGpBufferedStream class. At the moment, only reading is buffered while writing is implemented as a pass-through operation.
Made 'count' parameter to CopyStream optional, the same way as TStream.CopyFrom is implemented.
Check for < 0 position in TGpStreamWindow.Seek.
Fixed reading/writing of zero bytes in TGpStreamWindow.
Added bunch of 'inline' directives.

GpLists 1.35

Implemented TGpClassList, a TStringList-based list of classes. Class names must be unique. Useful when you need to implement a class registry to generate objects from class names.
Added Walk enumerator to the TGpIntegerList and TGpInt64List. This enumerator allows list modifications (Add, Insert, Delete) from the enumeration consumer. IOW, you can do this:
```
for idx in list.Walk do
  if SomeCondition(list[idx]) then
    list.Delete(idx);
```
Modified TGpCountedInt64List to store int64 counters.
Added property ItemCounter[] to TGpCountedIntegerList and TGpCountedInt64List.
Added Slice(from, to) enumerator to TGpIntegerList and TGpInt64List.
Add TGpObjectRingBuffer and TGpDoublyLinkedList enumerators. Both lock access to the list during the enumeration process if multithreaded mode is enabled.
Added method Contains to TGpIntegerList, TGpInt64List, TGpCountedStringList, TGpTMethodList.
When TGpIntegerObjectList or TGpInt64ObjectList was Sorted and with Duplicates set to dupIgnore, calling AddObject with item that was already in the list caused internal exception.
Enumerators changed to records with GetCurrent inlined, as suggested in More fun with Enumerators.

GpTimezone 1.22

Implemented DateLT, DateLE, DateGT, DateGE.

DSiWin32 1.36a

Added procedures DSiCenterRectInRect and DSiMakeRectFullyVisibleOnRect.
Added DSiTerminateProcessById procedure.
Added three performance counter helpers DSiPerfCounterToMS, DSiPerfCounterToUS, and DSiQueryPerfCounterAsUS.
Added function DSiTimeGetTime64.
Added function DSiGetProcessFileName.
Added function DSiEditShortcut.
Added function DSiInitFontToSystemDefault.
Added many SHGetSpecialFolderLocation folder constants.
Added ShGetSpecialFolderLocation flags CSIDL_FLAG_DONT_UNEXPAND and CSIDL_FLAG_DONT_VERIFY.
Added dynamically loaded API forwarder DSiSetSuspendState.
Added dynamically loaded API forwarders DSiEnumProcessModules, DSiGetModuleFileNameEx, and DSiGetProcessImageFileName.
Changed DSiIsAdmin to use big enough buffer for token data.
Changed DSiIsAdmin to ignore SE_GROUP_ENABLED attribute because function was sometimes incorrectly returning False.
Added parameter 'parameters' to DSiCreateShortcut and DSiGetShortcutInfo.
More stringent checking in DSiGetProcessWindow.

TDM Rerun #6: GExperts 1.0

2008-03-07T08:44:00.001+01:00

The greatest (in my view), and one of the oldest of the Delphi IDE expert suites is GExperts. It was created by Gerals Nunn (in 1997, if I am not mistaken), who also developed it for almost two years. Gerald has moved to other projects, but he kindly donated the GExperts source code to the online community.

- GExperts 1.0, The Delphi Magazine 72, August 2001

I still stand by those words. GExperts is a great suite of experts and I still use it every day. I also wrote few experts that are included in the GExperts pack and few that are included only in my private build.

Links: article (PDF, 95 KB), GExperts

Debugging With Lazy Breakpoints

2008-03-03T11:26:00.001+01:00

Sometimes you're hunting down a problem that occurs very rarely. When it does, an exception is triggered. If you're debugging, Delphi should pop up (it does) and allow you to look at the stack and inspect local variables (it doesn't - happens a lot to me). What can you do?

You could set a conditional breakpoint just before the problem occurs and test for the problem causing condition here. That would work on a short run, but if you're hunting the problem for many weeks, the breakpoint may move around or even get disabled. Delphi's breakpoint storage mechanism is somewhat unstable if you edit the code around the breakpoint a lot (and even more if you edit the code on multiple computers).

More stable solution is to do everything in code, from testing if debugger is running to pausing the program and breaking into the debugger. Following fragment illustrates the approach:

if DebugHook <> 0 then // test if debugger is running
  if problem = true then //test the condition
    asm int 3 end; //break into debugger

Not very obvious and full of arcane knowledge, but it works.

Paint It Black (And Red)

2008-03-01T21:03:00.001+01:00

Just in case you're not following Julian's blog, I'd like to bring your attention to his new series on red-black trees. Red-black trees are great data structure, but are notoriously hard to code correctly as the algorithm is full of weird edge cases.

I've always been a follower of his 'algorithms and data structures' articles in The Delphi Magazine and I'm sure his new series will be presented at the equally high standard. He's using C#, VB and .NET but I'm sure the series will be interesting for us Delphi users, too. [And he already gave us RB tree implementation in the EZDSL library - thanks, Julian!]

Of course, if you own his DADS book, you can read all about RB trees there.

TDM Rerun #5: SvCom 4.0

2008-02-26T14:24:00.001+01:00

When you get into really serious Windows development, you may well need to create NT service applications. Borland's input is half-hearted: Delphi includes support for NT service application writing, but as you work with this framework, you'll soon discover its less-polished parts!

- SvCom 4.0, The Delphi Magazine 69, May 2001

In the TDM #69 I reviewed SvCom, "an integrated package of classes and tools for development of service applications and use of NT Security framework." (as the current SvCom web page states). Good stuff, I'm still using it for service development.

Links: article (PDF, 67 KB), SvCom

TDM Rerun #4: Let's Cooperate

2008-02-11T14:06:00.001+01:00

Every serious Windows programmer who has developed at least one multi-threaded application knows about thread synchronisation. You know what I mean: critical sections, mutexes, semaphores, signals and so on. ... But to write truly distributed applications, running on different computers, we need new tools.

- Let's Cooperate, The Delphi Magazine #68, April 2001

I don't remember what was the exact reason for this article, but obviously I was working on some multi-process multi-computer solution and I needed a way to synchronise those computers. Of course, once I started writing the article, it got completely out of hand and I finished with file system based mutex, critical section, group, event, and even with a message queue and single-writer-multiple-readers synchronisation primitive. There were also many state diagrams like the one of the right dispersed through the article. Poor readers. >:-(

The plan was to make most of them work on Linux as well, but as I discovered in File Sharing on Linux (not yet published on my blog), but as I found later, this was nearly impossible to do as Linux implements file locking in completely different way.

I never used those primitives much as TCP/IP communication became much more prominent in later years. Still, they can be very useful if you want to quickly put together a multi-computer demo or quick and dirty solution.

The GpFileSync unit never made it to my web so I'm linking to the current version at the bottom of this article.

Links: article (PDF, 117 KB), source code (ZIP, 47 KB), current GpFileSync unit

TDM Rerun #3: Time Is The Simplest Thing

2008-02-06T11:54:00.001+01:00

'Time, I'll tell you about time. Time is the simplest thing.'

- Time Is The Simplest Thing, The Delphi Magazine #65, January 2001
[quote from Clifford D. Simak's The Fisherman]

Now this is one of my favorite articles. It deals with time and more specifically with time zones. As most of my articles this one also builds on real-world problems I encounter during my programming practice. In this case, it was motivated by a big coding flop which taught me an important lesson: "Never store local time in a database." [No, I still don't want to recall all the details. It was related to daylight savings time transitions, that's all I'm willing to tell.]

Anyway, I fixed the problems in the abovementioned code and during that process I found out that there is almost no support for time zones built in Delphi and that Windows APIs on that area are really not that simple to use. So I built a time zone manipulation library that handles time zone information access and time zone conversions with daylight savings time support. This GpTimezone unit is still alive and I'm using it regularly - mostly the NowUTC function.

The article also dealt with the (now happily forgotten) Swatch Time.

Also included was a neat utility (if I'm allowed to say so) for timezone conversions, which I'm still using today without any changes to the original code.

Links: article (PDF, 134 KB), source code (ZIP, 162 KB), current GpTimezone unit

TWinControl.Controls Enumerator, Revisited

2008-02-06T11:07:00.001+01:00

Fredrik Loftheim made an interesting observation to my previous article on TWinControl.Controls enumerator - namely that the enumerator factory doesn't have to be interface based. It can be replaced by a record. Source is almost the same as before, but we can skip the interface declaration. Generated code is simpler as the enumerator factory is simply allocated from the stack and automatically destroyed when it goes out of scope. No interface support and no reference counting is required. Simpler and faster, that's the way to go.

interface

type
  TControlEnumerator = record
  strict private
    ceClass : TClass;
    ceIndex : integer;
    ceParent: TWinControl;
  public
    constructor Create(parent: TWinControl; matchClass: TClass);
    function GetCurrent: TControl;
    function MoveNext: boolean;
    property Current: TControl read GetCurrent;
  end; { TControlEnumerator }

  TControlEnumeratorFactory = record
  strict private
    cefClass : TClass;
    cefParent: TWinControl;
  public
    constructor Create(parent: TWinControl; matchClass: TClass);
    function  GetEnumerator: TControlEnumerator;
  end; { TControlEnumeratorFactory }

function  EnumControls(parent: TWinControl; matchClass: TClass = nil): TControlEnumeratorFactory;

implementation

function EnumControls(parent: TWinControl; matchClass: TClass = nil): TControlEnumeratorFactory;
begin
  Result := TControlEnumeratorFactory.Create(parent, matchClass);
end; { EnumControls }

{ TControlEnumerator }

constructor TControlEnumerator.Create(parent: TWinControl; matchClass: TClass);
begin
  ceParent := parent;
  ceClass := matchClass;
  ceIndex := -1;
end; { TControlEnumerator.Create }

function TControlEnumerator.GetCurrent: TControl;
begin
  Result := ceParent.Controls[ceIndex];
end; { TControlEnumerator.GetCurrent }

function TControlEnumerator.MoveNext: boolean;
begin
  Result := false;
  while ceIndex < (ceParent.ControlCount - 1) do begin
    Inc(ceIndex);
    if (ceClass = nil) or (ceParent.Controls[ceIndex].InheritsFrom(ceClass)) then begin
      Result := true;
      break; //while
    end;
  end; //while
end; { TControlEnumerator.MoveNext }

{ TControlEnumeratorFactory }

constructor TControlEnumeratorFactory.Create(parent: TWinControl; matchClass: TClass);
begin
  cefParent := parent;
  cefClass := matchClass;
end; { TControlEnumeratorFactory.Create }

function TControlEnumeratorFactory.GetEnumerator: TControlEnumerator;
begin
  Result := TControlEnumerator.Create(cefParent, cefClass);
end; { TControlEnumeratorFactory.GetEnumerator }

The new version of the TWinControl.Controls enumerator plus some other stuff is available at http://gp.17slon.com/gp/files/gpvcl.zip.

TWinControl.Controls Enumerator

2008-02-01T15:42:00.001+01:00

I'm still having fun with enumerators. The latest I wrote is a filtering enumerator for TWinControl.Controls[] that allows me to write code like this:

var
  control: TControl;

for control in EnumControls(pnlAttributes, TSpeedButton) do
  TSpeedButton(control).Enabled := false;

I found it interesting that Borland built a Components[] enumerator in the VCL, but not a Controls[] enumerator.

The EnumControls interface is simple. It takes a starting point for enumeration and an optional class filter. By specifying the latter, you tell the enumerator that you're only interested in child controls of a specified class.

function EnumControls(parent: TWinControl; 
  matchClass: TClass = nil): IControlEnumeratorFactory;

As it is used in a for..in loop, EnumControl must return a class or interface that implements GetEnumerator function and that is exactly what it does. GetEnumerator, on the other hand, creates an instance of the TControlEnumerator class, which implements the actual enumeration.

type
  TControlEnumerator = class
  strict private
    ceClass : TClass;
    ceIndex : integer;
    ceParent: TWinControl;
  public
    constructor Create(parent: TWinControl; matchClass: TClass);
    function GetCurrent: TControl;
    function MoveNext: boolean;
    property Current: TControl read GetCurrent;
  end; { TControlEnumerator }

  IControlEnumeratorFactory = interface
    function GetEnumerator: TControlEnumerator;
  end; { IControlEnumeratorFactory }

On the implementation side, EnumControls creates an instance of the TControlEnumeratorFactory class, which implements the IControlEnumeratorFactory interface.

function EnumControls(parent: TWinControl; matchClass: TClass = nil): IControlEnumeratorFactory;
begin
  Result := TControlEnumeratorFactory.Create(parent, matchClass);
end; { EnumControls }

type
  TControlEnumeratorFactory = class(TInterfacedObject, IControlEnumeratorFactory)
  strict private
    cefClass: TClass;
    cefParent: TWinControl;
  public
    constructor Create(parent: TWinControl; matchClass: TClass);
    function  GetEnumerator: TControlEnumerator;
  end; { GetEnumerator }

TControlEnumeratorFactory just stores parent and matchClass parameters for later use in the GetEnumerator function.

constructor TControlEnumeratorFactory.Create(parent: TWinControl; matchClass: TClass);
begin
  cefParent := parent;
  cefClass := matchClass;
end; { TControlEnumeratorFactory.Create }

function TControlEnumeratorFactory.GetEnumerator: TControlEnumerator;
begin
  Result := TControlEnumerator.Create(cefParent, cefClass);
end; { TControlEnumeratorFactory.GetEnumerator }

GetEnumerator creates an instance of the TControlEnumerator class, which implements the actual enumeration in a pretty standard manner. Only the MoveNext method is slightly more complicated than usual because it must optionally check the matchClass parameter.

constructor TControlEnumerator.Create(parent: TWinControl; matchClass: TClass);
begin
  inherited Create;
  ceParent := parent;
  ceClass := matchClass;
  ceIndex := -1;
end; { TControlEnumerator.Create }

function TControlEnumerator.GetCurrent: TControl;
begin
  Result := ceParent.Controls[ceIndex];
end; { TControlEnumerator.GetCurrent }

function TControlEnumerator.MoveNext: boolean;
begin
  Result := false;
  while ceIndex < (ceParent.ControlCount - 1) do begin
    Inc(ceIndex);
    if (ceClass = nil) or (ceParent.Controls[ceIndex].InheritsFrom(ceClass)) then begin
      Result := true;
      break; //while
    end;
  end; //while
end; { TControlEnumerator.MoveNext }

It's instructive to see how all those parts play together when the code fragment from the beginning of this blog entry is executed.

for control in EnumControls(pnlAttributes, TSpeedButton) do
  TSpeedButton(control).Enabled := false;

EnumControls is called. It creates an instance of the IControlEnumeratorFactory. TControlEnumeratorFactory constructor stores parent and matchClass parameters into internal fields.
- TControlEnumeratorFactory.GetEnumerator is called. It creates an instance of the TControlEnumerator and passes it internal copies of parent and matchClass parameters.
  - TControlEnumerator's MoveNext and Current are used to enumerate over the parent's child controls.
- For..in loop terminates and compiler automatically destroys the object, created by the GetEnumerator method.
Sometime later, the method containing this for..in loop terminates and compiler automatically frees the IControlEnumeratorFactory instance created in first step.

The TWinControl.Controls enumerator plus some other stuff is available at http://gp.17slon.com/gp/files/gpvcl.zip.

TDM Rerun #2: A Better Build Process

2008-01-29T18:39:00.001+01:00

I had to live with the problem. Until the day when it occurred to me that I could very probably find the solution to my problem on the internet. So I connected to Deja News and searched.

- A Better Build Process, The Delphi Magazine #49, September 1999

A life without Google? Was that really possible? I can hardly remember those times.

The article was written in Delphi 4 times and deals with building Delphi applications from a command line. More specifically, it solves the problem of dynamically recreating project.res file, which is something that you have to do if you want to increment build number in the version info resource during the build process. (A utility to increment build number was included, too.)

Also on display was a batch file that built my trusty GpProfile profiler and a "very complicated" tail utility.

{$APPTYPE Console}
program tail;
uses
  SysUtils, Classes;
var
  str  : TStringList;
  i    : integer;
  first: integer;
begin
  str := TStringList.Create;
  try
    str.LoadFromFile(ParamStr(1));
    first := str.Count-StrToIntDef(ParamStr(2),20);
    if first < 0 then first := 0;
    for i := first to str.Count-1 do Writeln(str[i]);
  finally
    str.Free;
  end;
end.

Not much has changed in the Delphi world - I would code it almost identical today.

Links: article (PDF, 70 KB), source code (ZIP, 185 KB)

TDM Rerun #1: Safer Sockets

2008-01-28T13:25:00.001+01:00

Instead of fixing the ScktComp unit it is probably better to create a derived class, override the Read method, paste code from ScktComp and make this small modification. Choose your way but beware: CancelIO is not available in Windows 95, only in 98 and NT 4.

- Safer Sockets, The Delphi Magazine #44, April 1999

Those were the times ... Most of computers were still running Windows 95, the actual Delphi version was 4 and we were using TWinSocketStream for data transfer over Windows sockets. Oh, happy days. Not!

I won't comment this article much. It describes how to fix a problem with an obsolete component that should never be used in modern software. I still remember how happy I was switching to ICS ... Still, I promised all my articles and I have to start with this one.

This article represents another important milestone in my 'journalist' career - it was my first article written in English language and a start of very happy long-term relationship with The Delphi Magazine.

Links: article (PDF, 49KB), source code (ZIP, 1 KB)

Make a better Delphi

2008-01-24T19:01:00.001+01:00

Delphi is a good language and Delphi IDE is a good development environment and together they make a good pair, but admit it - Delphi language is missing some modern language constructs (parameterized types, for example) and Delphi IDE can be rough at the edges. We can't do much about the former, but Delphi IDE can be greatly enhanced by installing third party add-ons (and great thanks to Borland/CodeGear for enabling us to do so).

Recently I've noticed that I can't really function well on a bare-bones IDE. I'm heavily dependent on three IDE extensions - GExperts, DDevExtensions, and ModelMaker Code Explorer. I decided to dedicate some blog space to them and describe what they do and why I like them.

It would take too much space to describe every aspect of those three add-ons, so I decided to focus on a navigation in a broad sense - moving inside unit, between units, between code and design view etc. Even more, I won't cover every navigational helper in those three, I'll just focus on the features I'm using regularly.

Delphi IDE

Delphi IDE has some neat navigation tools built in. First, there are markers. You can drop ten markers by pressing <Ctrl><Shift>number, where number is a numeric key from 0 to 9 (you don't have to use numeric keyboard keys, standard keyboard numbers will work just fine). By pressing <Ctrl>number you'll be transferred to the line containing the marker. To delete a marker you just press <Ctrl><Shift>number while the caret is positioned in the line containing the marker.

Another neat feature of the Delphi IDE is the ability to jump from method implementation to its declaration and back. The shortcut for this operation is <Ctrl><Shift><UpArrow> or <Ctrl><Shift><DownArrow> (both work the same).

When you want to find where some identifier is used (in current unit and in the rest of the project), you can use find references (<Ctrl><Shift><Enter>) command. I'm mentioning it here because you can also use it to jump to the place where the identifier is used.

I'm also using the unit list (<Ctrl><F12>) and form list (<Shift><F12>) regularly. First shows the list of units in the project and second the list of forms. Just double-click the unit/form or select it and press <Enter> and it will open in the IDE.

GExperts

GExperts is a set of freeware experts that enhance Delphi IDE in various ways. Only small subset is connected to the navigation.

One of the most useful helpers is open file expert. Press the key (I have it configured to <Ctrl><Shift><Alt>O) and up pops a list of all files that are accessible from the current project. You can select the unit and press <Enter> to open it (alternatively you can double-click that unit) but before that you can - and that's even more important - start typing in the filter box and the expert will remove all non-matching units.

Another useful tool is procedure list expert. As the name suggests, it presents a list of methods in the current unit, which you can, of course, filter by typing. Double-click or <Enter> will beam you to the selected method.

GExperts also offers bunch of helpful editor experts. Locate matching delimiter (<Ctrl><Alt><LeftArrow>/<RightArrow>) jumps to a matching delimiter - from left to right bracket, from begin to end, from implementation to interface and so on. Locate previous/next identifier (<Ctrl><Alt><UpArrow>/<DownArrow>) locates previous/next occurrence of the identifier under the caret.

The last navigational help in the GExperts is hidden in the form designer pop-up menu. It is extended with the find component reference command. Right-click on a component or control and select this option and GExperts will bring up the editor, focused on the part where this component/control is declared. Or used for the first time in the code. Sometimes former and sometimes latter and I can never guess which one it will be.

DDevExtensions

DDevExtensions is developed by Andreas Hausladen, author of well-known DelphiSpeedUp. I'm mostly using it because it reconfigures the <Tab> key to indent a block of code and <Shift><Tab> to unindent it, but that's not a navigational extension so you can pretend you didn't hear that from me. It also implements smart home functionality. First time you press the <Home> key, caret will jump to the first column in the row. If you press the <Home> button again, caret will jump to the first non-blank character in that line. Useful.

DDevExtensions also redefines unit list and form list keyboard shortcuts so that display a list that can be filtered - similar to the GExeperts' open file expert.

ModelMaker Code Explorer

ModelMaker Code Explorer (or MMX as it is usually called between loving users) is one of the largest Delphi IDE add-ons on the market. Its most important functionality is refactoring - creating classes, fields, properties, and methods becomes lightingly fast with MMX - but it also offers some navigational functionality. MMX is not free like GExperts and DDevExtensions but I can assure you that it's worth every cent.

An important part of MMX interface is the Explorer window. It lists every class and method in the current unit, has movements synchronized with the editor and is filterable.

Similar to Delphi's find reference is find in module command (<Ctrl><Shift><Alt>F). It locates all occurrences of the identifier in the current module and uses tabbed interface to store old searches. Great advantage of MMX over Delphi is that it works in older IDEs and that it works on a non-compilable code.

MMX also offers extended up/down navigation. Delphi's shortcuts <Ctrl><Shift><UpArrow>/<DownArrow> are redefined to jump (in addition to the default behaviour) between propery declaration, its getter and setter, between constructors and destructors and between overloaded methods. Highly useful.

The last enhancement on my today's list is global history. MMX will automatically remember any method where you spent more than a few seconds. When you press the keyboard shortcut (I have it set to <Ctrl>` - the latter being the key left to the '0' key), MMX displays pop-up menu with most recently edited methods. Click one and you're instantly transferred there.

Other enhancements

That's all by me, I've exhausted my knowledge. Here's where you, dear reader, can help. If you know a trick that I didn't mention or a helpful expert that I don't have installed, do tell me about it in the comments. You'll help me and other Delphi programmers.

Reposting TDM Articles

2008-01-23T19:59:00.001+01:00

Like some other authors (Hallvard Vassbotn, Bob Swart) I'll be republishing articles I wrote for The Delphi Magazine. [Chris, thanks for the permission!]

Unlike Dr. Bob and Hallvard I won't have problems with choosing from my TDM articles - I published much less than them so I'll simply republish everything :)

In the next months you can expect online versions of my articles:

Safer Sockets [TDM #44]
Describes an issue with TWinSocketStream and a presents a workaround.
A Better Build Process [TDM #49]
How to build Delphi programs from a command line; introduces some helper tools.
Time Is The Simplest Thing [TDM #55]
The art of timezones. Introduces GpTimezone unit, which is still alive and regularly updated.
Let's Cooperate [TDM #68]
My first take on steampunk process synchronisation - not via OS primitives but by using file system. Introducing GpFileSync unit, which somehow never made it to my web site.
SvCom 4.0 [TDM #69]
A review of SvCom service development package. Still alive and well, currently at version 7.
GExperts 1.0 [TDM #72]
A review of Delphi experts we all love.
File Sharing On Linux [TDM #84]
My only article on Kylix continued the Let's Cooperate theme.
A Synchronisation Toolkit [TDM #86]
Simple multi-process synchronisation and communication primitives. Introduces the GpSync unit, which is sill alive and regularly updated.
My Data Is Your Data [TDM #88]
Describes an easy to use wrapper around Windows' shared memory. Introduces GpSharedMemory unit.
Synchronisation Toolkit Revisited [TDM #91]
An upgrade of the GpSync unit, introducing message queue.
Shared Pools [TDM #95]
Shared memory pools. I never developed this concept any further.
Shared Events Part 1: Rationale, Design, Implementation [TDM #97]
Cross-process event dispatch using shared memory. Introduces GpSharedEvents unit.
Shared Events Part 2: Redesign [TDM #102]
Implementation details and a redesign.
A Portable XML [TDM #105]
A review of the OmniXML library. Still alive and regularly updated.
Many Faces Of An Application [TDM #107]
How to write an application that can be a service, Windows GUI and system tray icon monitor.
Thread Pooling, The Practical Way [TDM #112]
Describes Windows thread pool, various problems with it and some workarounds.
Put It In A Tree [TDM #118]
Discusses various approaches to creating a tree structure from a list of mail messages.
To Manage Memory [TDM #126]
An introduction to memory management algorithms and structures plus a short overview of the (at that time very young) FastMM memory manager.

And that's all folks. I'll probably attack this list in chronological manner. If you have any preferences I may make an exception and change priorities so don't be afraid to ask for article you want to read first.

Disappearing Welcome Page

2008-01-14T22:10:00.001+01:00

One of the most annoying bugs in Delphi 2007 must be the disappearing Welcome Page. Sometimes, when you do Close All, IDE decides that Welcome Page contains highly sensitive material that is not intended for your eyes, and deletes the contents. Like this:

By trial and error (and much luck) I have found a way to restore it.

Step 1: Click into the edit box next to the 'house' icon. Text (bds:/default.htm) will be selected.

Step 2: Press <Backspace> to delete the text.

Step 3: Press <Enter>. Nothing will happen, but the trick won't work without this.

Step 4: Click the 'house' icon. Welcome Page should reload.

If that doesn't work, you can always close the welcome page and reopen it via View, Welcome Page menu. Is is, however, faster to try click-backspace-enter-click trick first.

TDataset Enumerator

2008-01-12T21:37:00.001+01:00

You already know that I love all things enumeratorish. Just recently I found a great example that combines three kinds of Delphi magic to allow the programmer to write very simple TDataset enumeration loops. ~~[And for the sake of my life I can't remember where I first saw the link to this code. Apologies to the original author.]~~ [Updated 2008-01-14: Jim McKeeth found my source. Apparently Nick Hodges was the first to blog about this enumerator. Thanks, Nick!]

This TDataset enumerator was written by Uwe Raabe and is freely available in the CodeGear's Code Central as item #25386.

The three trick he uses are:

Data helper is used to push enumerator support into TDataset.
Standard enumerator support to enumerate over dataset and return special descendant of the Variant type for each record.
Custom variant type to automagically access record fields by by adding .RecordFieldName to the variant variable. [I must admit I didn't even know this was possible to do in Delphi.]

All together those tricks allow you to write very simple TDataset iteration loops (code taken from the example in the DN #25386):

var
  Employee: Variant;
  S: string;

for Employee in QuEmployee do begin
  S := Trim(Format('%s %s', [Employee.First_Name, Employee.Last_Name]));
  if Employee.Hire_Date < EncodeDate(1991, 1, 1) then
    S := '*' + S;
end;

Excellent job!

Unicode Delphi

2008-01-10T00:05:00.001+01:00

The Oracle at Delphi (aka Allen Bauer) has posted some information on Unicode support in the next Delphi (codenamed Tiburon).

Firstly, let me say that I really appreciate that we'll finally get VCL Unicode and reference-counted wide string support in Delphi. I really really really appreciate that.

Secondly, I must say, and I must say this very loud, that I DON'T LIKE HOW IT WILL BE IMPLEMENTED! (And I apologize for yelling in public.)

Allen wrote

This new data type will be mapped to string which is currently an AnsiString underlying type depending on how it is declared. Any string declared using the [] operator (string[64]) will still be a ShortString which is a length prefixed AnsiChar array. The UnicodeString payload will match that of the OS, which is UTF16.

And

If your code must still operate on Ansi string data, then simply be more explicit about it. Change the declaration to AnsiString, AnsiChar, and PAnsiChar. This can be done today and will recompile unchanged in Tiburon.

I wrote my response in comments to the Allen's post and I'll restate it here.

This is very very bad. This will not work for our company and I presume for many of us working with legacy applications with millions of source lines. We have applications using 3rd party components and libraries (all with source code, thankfully). We have applications using libraries that can be compiled for Win32 and for DOS (with BP7 and plenty of IFDEFs, of course). And we have no illusions that they will continue to work if string suddenly becomes an UTF-16-holding type.

We would like to move to Unicode Delphi, but we would like to do it gradually, when starting work on new applications. That way, we can test all units, components and libraries as they are introduced in the new program and we can fix all problems that will occur.

So please, CodeGear, give as an 'Unicode application' option. It should be disabled for old applications and enabled for new ones. That would be a much better plan.

[If you agree with me, make sure to post a comment in Allen's blog stating your opinion. If you don't agree with me, make sure to post a comment in Allen's blog stating your opinion. He needs as much feedback as possible to make decitions that will be good for us.]