`` ▐▛███▜▌   Claude Code v2.1.161
▝▜█████▛▘  Sonnet 4.6 · Claude Pro
  ▘▘ ▝▝    X:\ms\sttwin


> /doctor 
  ⎿  Claude Code diagnostics dismissed

> Please analyze group of stack traces in @2026-06-03-A3-sttwin.log to see if   
  there are any common problems. In (a different) log we see that the program   
  was trying to close, but didn't actually close, so we suspect a deadlock.     
                                                                                
  There are multiple big stack traces in that file so create a plan first.      

● I'll invoke the systematic-debugging skill before proceeding, then read the
  log file and memory context in parallel.

● Skill(superpowers:systematic-debugging)
  ⎿  Successfully loaded skill

  Recalled 1 memory (ctrl+o to expand)

● Let me first locate and scan the log file structure before creating the plan.

  Searched for 2 patterns (ctrl+o to expand)

● Bash(wc -l "X:\ms\sttwin\2026-06-03-A3-sttwin.log" 2>/dev/null || (Get-Content
      "X:\ms\sttwin\2026-06-03-A3-sttwin.log" | Measure-Object -Line).Lines)
  ⎿  Error: Exit code 2
     /usr/bin/bash: eval: line 1: syntax error near unexpected token `.Lines'
     /usr/bin/bash: eval: line 1: `wc -l "X:\ms\sttwin\2026-06-03-A3-sttwin.log"
     2>/dev/null || (Get-Content "X:\ms\sttwin\2026-06-03-A3-sttwin.log" | 
     Measure-Object -Line).Lines'

  Searched for 1 pattern, read 1 file (ctrl+o to expand)

● Good — I can see the structure. Let me now do a fast survey: count the
  distinct dumps, find all main-thread stacks, and collect unique blocking
  patterns.

  Searched for 2 patterns (ctrl+o to expand)
  ⎿  Interrupted · What should Claude do instead?

> Deadlocks may occur in background threads.                                    

● Good clarification. Let me now do a targeted survey across all dumps for
  sync-primitive blocking patterns in all threads.

  Searched for 1 pattern (ctrl+o to expand)

● Bash(grep -n "  Thread:" "X:/ms/sttwin/2026-06-03-A3-sttwin.log" | sed 's/
      (suspended)//' | sed 's/ (running)//' | awk -F'Thread: ' '{print $2}' |
      sed 's/ ([0-9]*).*…)
  ⎿      580 OtlThreadPool worker
          98 TAvecoThread
          79 TMSLogThread
     … +37 lines (ctrl+o to expand)

● Good data. Now I have enough to lay out a proper plan. Here it is:

  ---
  Investigation Plan

  43 dumps spanning 17/11/2025–03/06/2026. Structure per dump: all threads with
  frame+scan traces. Potential deadlock is in background threads.

  Step 1 — Focus on the shutdown dumps

  Read the two most recent dumps (lines 59424 and 60057, both 03/06/2026). The
  shutdown that never completed will show the final stuck state most clearly.

  Step 2 — Find threads blocked on lock Acquire (not just Wait)

  The key deadlock signal is a thread stuck inside
  Enter/Acquire/TryEnterCriticalSection — not WaitFor* (those are normal waits).
  Already spotted:
  - TAutomationInput.Enter (TAvecoThread, line 35) — suspicious name, could be a
  lock acquire
  - TCriticalSection.Release appearing in scan traces (lines 67, 152, 252, 344)
  — scan traces show extra frames; Release in scan-only could mean the unwind
  was mid-release when captured
  - TMonitor.Destroy (line 395) during active renderer work — destroying a
  monitor while threads use it is a classic deadlock/AV source

  Step 3 — Cross-dump thread ID persistence

  Find thread IDs that appear with the same blocked call in multiple consecutive
  dumps — those are definitively deadlocked (not just momentarily blocked).

  Step 4 — OtlThreadPool worker saturation

  580 occurrences across 43 dumps (~13/dump). Check if pool workers are stuck on
  the same thing or just normally idle.

  Step 5 — Catalogue all unique block points

  Extract the top-of-stack call for every thread across all dumps, group by
  (thread-type, top-of-stack), and rank by frequency. This gives a map of all
  recurring blocks.

  Step 6 — Trace lock ownership

  For any thread blocked on Enter/Acquire, find which other thread holds the
  matching lock (by correlating the same lock address/object in other thread
  stacks).

  ---
  Shall I proceed with this plan, starting with Steps 1 and 2 in parallel?

✻ Crunched for 42s

> I'm mostly interested in dumps from the last 30 days, so please ignore older  
  dumps. Otherwise a good plan, please proceed.                                 

● Last 30 days = dumps from 04/05/2026 onward, lines 46488–61400. Relevant
  dumps: lines 46488, 48013, 49569, 51207, 52544, 54056, 55004, 56528, 58065,
  59424, 60057 — all same PID 16672 except the last two (PID 15692).

  Starting Steps 1, 2, and 5 in parallel: reading the two most recent dumps and
  extracting all block-point patterns from the relevant range.

  Read 1 file (ctrl+o to expand)

● The awk output didn't give me actual top frames — it matched the column
  header. Let me read the second dump and look for threads with the same block
  in both 03/06 dumps.

  Searched for 2 patterns, read 1 file (ctrl+o to expand)
● TAvecoThread is stuck at TAutomationInput.Enter in both the 22/05 dump AND the
  first 03/06 dump, but the second dump (11:55:55) has different Zoomer thread
  IDs — suggesting threads were recreated. Let me read the end of the second
  dump to find TAvecoThread and the main thread.

  Read 1 file (ctrl+o to expand)

● The second 03/06 dump shows TAvecoThread (15964) unblocked (now at
  AutoUnit:3186 WaitForTwoObjects), but the same stack IDs for most other
  threads. Let me now check which thread owns TAutomationInput in the 29/05
  dump, and also scan the 22/05 dump where the Enter block was captured earlier.

  Read 1 file (ctrl+o to expand)

● Critical finding: the main thread in the 29/05 dump holds TAutomationInput
  lock while inside ProcessCuedActions → CheckAutomationStatus. Now I need to
  find what thread holds the lock in the 03/06 morning dump when TAvecoThread is
  blocked — let me also look at the 01/06 dump's TAutomationInput call context.

  Searched for 1 pattern, read 1 file (ctrl+o to expand)

● Now I can see the complete lock ownership picture. Let me check the main
  thread in the 22/05 dump to complete the cross-dump correlation.

  Searched for 1 pattern, read 1 file (ctrl+o to expand)

● Excellent — I now have the complete picture. Let me synthesize all findings.

  ---
  Analysis Results

  The Recurring Lock Contention Pattern

  There are always two TAvecoThread instances in every dump within the last 30
  days. They have consistent, opposing roles:

  ┌──────────────────┬─────────────────────────────────┬────────────────────┐
  │      Thread      │            Position             │       State        │
  ├──────────────────┼─────────────────────────────────┼────────────────────┤
  │ TAvecoThread     │ AutoUnit:3212 SafeExecute →     │ Holds              │
  │ (7836) / (17092) │ TTXPage:5244                    │ TAutomationInput   │
  │                  │ MsgWaitForSingleObject          │ lock               │
  ├──────────────────┼─────────────────────────────────┼────────────────────┤
  │ TAvecoThread     │ AutoUnit:3169 SafeExecute →     │ Blocked waiting    │
  │ (15964) /        │ AutoUnit:5679                   │ for the lock       │
  │ (17084)          │ TAutomationInput.Enter          │                    │
  └──────────────────┴─────────────────────────────────┴────────────────────┘

  The call progression in SafeExecute is:
  3169: TAutomationInput.Enter   ← acquire lock
  3186: WaitForTwoObjects        ← first wait (still holding lock)
  3212: MsgWaitForSingleObject   ← second wait (still holding lock)
        ...
      : TAutomationInput.Leave   ← release lock (somewhere after 3212)

  Thread (7836)/(17092) enters at line 3169, proceeds to its long blocking wait
  at line 3212 while still holding the TAutomationInput lock. Thread
  (15964)/(17084) tries to enter at line 3169 and is serialized behind it — it
  must wait until thread (7836) leaves the lock entirely.

  What Holds the Lock During the Wait

  Confirmed from the 29/05 dump: the main thread (sttwin) calls
  TAutomationInput.ProcessCuedActions from the WMNewTC → PublicNewTC → 
  NewTimecode path, which calls Enter and Leave (AutoUnit:5684) internally. The
  22/05 and 01/06 dumps show the same chain active. This means both the main 
  thread and multiple TAvecoThread instances compete for the same 
  TAutomationInput lock, with at least three potential lock holders.

  The TMonitor.Destroy Pattern (Secondary Finding)

  Video Worker [] thread (8568) shows in its scan trace (both 03/06 dumps,
  identical state):
  TFABRenderer.WaitForRenderTime
  ...
  FastMM4:4566 FreeLargeBlock
  TMonitor.Destroy          ← monitor destroyed mid-use
  TObject.CleanupInstance
  TBCRendererInputPin.Receive
  A TMonitor is being destroyed while a thread is in mid-render cycle. This is a
  separate issue (potential AV/heap corruption) but not the shutdown hang
  cause.

  Probable Shutdown Deadlock Mechanism

  Based on the stacks, the shutdown hang most likely follows this path:

  1. Main thread initiates shutdown — signals both TAvecoThread instances to
  stop
  2. Main thread calls WaitFor (or FreeAndNil) on the threads
  3. Thread (7836)/(17092) is blocked in MsgWaitForSingleObject at line 3212,
  INSIDE the TAutomationInput lock
  4. If the stop signal is delivered via TAutomationInput itself (e.g., the
  shutdown code calls Enter to post a stop command), the main thread blocks
  trying to acquire the lock
  5. Thread (7836) never receives the stop command because it's stuck in its
  blocking wait, OR the main thread is blocked in WaitFor and can't process
  messages that would unblock MsgWaitForSingleObject
  6. → Circular wait: main thread holds lock → thread (7836) can't be stopped →
  main thread can't proceed

  Alternatively: if MsgWaitForSingleObject at line 3212 monitors a message queue
  and the main thread is in WaitFor (not pumping messages), any response that
  requires main-thread message processing will never arrive.

  Confirmed Non-Issues

  - OtlThreadPool workers: All consistently idle in DSiWaitForTwoObjects → 
  ReceiveWait → GetMsg — normal idle state, not a problem.
  - Zoomer #0–#9: All in TOmniBlockingCollection.TryTake — normal idle state.
  - TCriticalSection.Release in scan traces: These appear in scan traces of
  threads at WaitForTwoObjects — this is the scanner finding the Release return
  address on the stack from a prior frame, not an active hold.

  Where to Look in the Code

  Location: AutoUnit:3169–3212 TAvecoThread.SafeExecute
  What to check: The lock is held across a long wait (MsgWaitForSingleObject).
    The shutdown signal must reach the thread WITHOUT going through
    TAutomationInput. Check if the stop event is checked before/after the lock
    scope, not inside it.
  ────────────────────────────────────────
  Location: AutoUnit:5679 TAutomationInput.Enter
  What to check: Check if this is recursive-safe. If the same thread can call it

    recursively, and the lock doesn't allow that, a single-thread recursive call

    would deadlock.
  ────────────────────────────────────────
  Location: Shutdown sequence calling TAvecoThread.WaitFor
  What to check: Ensure no TAutomationInput.Enter is held by the main thread at
    the point it calls WaitFor.
  ────────────────────────────────────────
  Location: TTXPage:5244 MsgWaitForSingleObject
  What to check: Check what event/handle this waits on, and whether the shutdown

    sequence signals it before calling WaitFor.

✻ Sautéed for 6m 57s

※ recap: Analyzing stack trace dumps from sttwin.log to find what causes the program to hang on shutdown. Next step is to look at AutoUnit lines 3169–3212 in TAvecoThread.SafeExecute to confirm whether the TAutomationInput lock is held across the MsgWaitForSingleObject call. (disable recaps in /config)

> Can you do a deeper analysis of this lockup and try to find the actual        
  reason?                                                                       

● I need to look at the actual source code. Let me find AutoUnit.pas and the
  relevant areas around lines 3169–3212, plus TAutomationInput.Enter/Leave.

  Searched for 2 patterns (ctrl+o to expand)

● Found it. Reading the key sections in parallel.

  Read 1 file (ctrl+o to expand)

● The line numbers in the debug info don't match the current source — the file
  was modified since the build. I need to search by content instead.

  Searched for 3 patterns, read 1 file (ctrl+o to expand)

● The line numbers don't match the current source. The compiled binary has
  different code. Let me search for where TAutomationInput.Enter is called from
  TAvecoThread, and check the full class structure.

  Searched for 2 patterns (ctrl+o to expand)

● TAutomationInput.Enter is a plain EnterCriticalSection(CS^). Now I need to
  find what TAvecoThread actually calls in the compiled binary. The line numbers
  shifted — let me look at what Owner.Enter calls exist in the code near the
  TCP socket loop, and also check line 2209 and the large block of Owner.Enter
  calls around lines 3852+.

  Read 1 file (ctrl+o to expand)

● The current source doesn't have Owner.Enter in TAvecoThread.SafeExecute, but
  the compiled binary does. Let me check SVN history and also look at the
  function that contains the Owner.Enter at line 2209 to understand what class
  it belongs to.

● Bash(svn log --limit 20 AutoUnit.pas 2>/dev/null | head -80)
  ⎿  ----------------------------------------------------------------------
     --
     r41637 | miha | 2026-05-18 19:32:11 +0200 (pon., 18 maj 2026) | 2 line
     … +77 lines (ctrl+o to expand)

  Searched for 1 pattern (ctrl+o to expand)

● r41420: TAutomationInput uses TMainSTTWINForm.ServerCS instead of it's own CS
  — this is the critical change. Let me see what it actually changed and also
  look at an older revision of TAvecoThread.SafeExecute to see what code had
  Owner.Enter before the message loop.

● Bash(svn diff -r 41419:41420 AutoUnit.pas 2>/dev/null | head -200)  ⎿  Index: AutoUnit.pas
     ===================================================================
     --- AutoUnit.pas   (revision 41419)
     … +64 lines (ctrl+o to expand)

● The r41420 diff shows only the CS ownership change (value→pointer,
  own→shared). It doesn't show changes to TAvecoThread.SafeExecute. Let me look
  at an older revision of AutoUnit.pas to find where Owner.Enter was called from
  TAvecoThread.SafeExecute, since it was there in the compiled 14.14 binary.

● Bash(svn log --limit 30 AutoUnit.pas 2>/dev/null | grep -E "^r[0-9]+|STTWIN
      Version" | head -40)
  ⎿  r41637 | miha | 2026-05-18 19:32:11 +0200 (pon., 18 maj 2026) | 2 line
     s
     * STTWIN Version 14.49
     … +31 lines (ctrl+o to expand)

● Bash(svn log --limit 60 AutoUnit.pas 2>/dev/null | grep -E
      "^r[0-9]+|14\.14|TAvecoThread|SafeExecute|Owner\.Enter" | head -40)
  ⎿  r41637 | miha | 2026-05-18 19:32:11 +0200 (pon., 18 maj 2026) | 2 line
     s
     r41606 | peter | 2026-05-14 19:28:46 +0200 (cet., 14 maj 2026) | 1 lin
     … +39 lines (ctrl+o to expand)

● Version 14.14 is around r40341 (2025-11-17). Let me get the old version of
  TAvecoThread.SafeExecute from around that time to find where Owner.Enter was
  called.

● Bash(svn cat -r 40341 AutoUnit.pas 2>/dev/null | grep -n "procedure
      TAvecoThread.SafeExecute\|Owner\.Enter\|Owner\.Leave\|MsgWaitForSingleObje
      ct\|WaitForTwoObjects"…)
  ⎿  1099:    (Notifier <> nil) and (WaitForTwoObjects (TerminateEvent, Not
     ifier.ValuesUpdated, false, 500) = WAIT_OBJECT_0);
     1543:      if MsgWaitForSingleObject( TerminateEvent, INFINITE, QS_ALL
     … +27 lines (ctrl+o to expand)

● Let me read TAvecoThread.SafeExecute from the r40341 revision directly.

● Bash(svn cat -r 40341 AutoUnit.pas 2>/dev/null | sed -n '3166,3260p' | cat -n
      | sed 's/^[ ]*//' | awk '{print NR+3165 "\t" $0}')
  ⎿  3166       1       procedure TAvecoThread.SafeExecute;
     3167       2       var
     3168       3         msg : TMsg;
     … +92 lines (ctrl+o to expand)

● TAvecoThread.SafeExecute in r40341 is identical to the current source — no
  Owner.Enter. Let me check the r40341 diff to see what that commit actually
  changed, and also look at older revisions to find when the TCP socket path was
  added to TAvecoThread.SafeExecute.

● Bash(svn diff -r 40340:40341 AutoUnit.pas 2>/dev/null | grep -A5 -B5
      "TAvecoThread\|SafeExecute\|Owner\.Enter\|CSCued\|EnterCS\|Enter" | head
      -100)
  ⎿  --- AutoUnit.pas   (revision 40340)
     +++ AutoUnit.pas   (revision 40341)
     @@ -267,6 +267,7 @@
     … +97 lines (ctrl+o to expand)

  Searched for 1 pattern (ctrl+o to expand)

● This is the root cause. The r40341 diff reveals exactly what was wrong. Let me
  finish verifying the complete picture.

● Bash(svn diff -r 40340:40341 AutoUnit.pas 2>/dev/null | grep -A3 -B3
      "Owner\.Leave\|LeaveAvecoMsgs" | head -60)
  ⎿         ALastActiveTCOffset: TFrameTime; ALastActiveSRC: Integer; ALast
     ActiveSRCName: String; var IDRestored: Boolean);
          destructor Destroy; override;
     … +49 lines (ctrl+o to expand)

  Read 1 file (ctrl+o to expand)

● The diff reveals much more. Now I have the full picture. Let me check
  TAutomationInput.Destroy in the old code to complete the shutdown scenario.

● Bash(svn cat -r 40340 AutoUnit.pas 2>/dev/null | grep -n "destructor
      TAutomationInput\|Thread\.Free\|Thread2\.Free\|ThreadREST\.Free\|FreeAndNi
      l.*Thread\|Owner\.Ente…)
  ⎿  (No output)

● Bash(svn cat -r 40340 AutoUnit.pas 2>/dev/null | grep -n "destructor
      TAutomationInput")
  ⎿  5625:destructor TAutomationInput.Destroy;

● Bash(svn cat -r 40340 AutoUnit.pas 2>/dev/null | sed -n '5625,5680p')
  ⎿  destructor TAutomationInput.Destroy;
     var
       i: Integer;
     … +52 lines (ctrl+o to expand)

● Now I have the complete picture. TAutomationInput.Destroy is clean — no CS
  held during WaitFor. The deadlock is in the interaction between
  SendExternalCommand and the REST API loop.

● Bash(svn cat -r 40340 AutoUnit.pas 2>/dev/null | grep -n
      "SendExternalCommand\|ProcessCuedActions\|CheckAutomationStatus" | head
      -20)
  ⎿  291:    function SendExternalCommand (Command, SourceIP: String): Stri
     ng;
     612:    function ProcessCuedActions: Boolean;
     … +5 lines (ctrl+o to expand)

  Searched for 1 pattern, read 1 file (ctrl+o to expand)

● ProcessCuedActions calls MainSTTWINForm.CheckAutomationStatus while holding
  the lock. Let me check if CheckAutomationStatus sends commands back through
  the REST API thread.
  Searched for 1 pattern, read 1 file (ctrl+o to expand)

● CheckAutomationStatus is pure UI update, no locking. Let me look at
  ProcessServerCommand — that's what the main thread is in on the 22/05 dump
  when TAvecoThread is blocked.

  Searched for 2 patterns (ctrl+o to expand)

● Let me read ProcessServerCommand and also look at GetAdditionalDevices to find
  what holds the lock.

  Searched for 2 patterns, read 2 files (ctrl+o to expand)

● Found it. GetAdditionalDevices calls Enter/Leave — that's the main thread
  holding Owner.CS in the 01/06 dump. Now let me look at the full
  ProcessCuedActions lock structure.

  Searched for 1 pattern (ctrl+o to expand)

● Bash(svn cat -r 40340 AutoUnit.pas 2>/dev/null | grep -n
      "\.Enter\b\|\.Leave\b\|SendExternalCommand\|ProcessFABCmd" | awk -F: '$2+0
      >= 5500 && $2+0 <= 6100' | head -…)
  ⎿  (No output)

  Searched for 2 patterns (ctrl+o to expand)

● Let me look at NewTimecode to see if it holds Owner.CS across the
  ProcessServerCommand call.

  Searched for 2 patterns, read 1 file (ctrl+o to expand)

● The structure of NewTimecode is clear: ProcessServerCommand → then
  ProcessCuedActions. Let me read the body of ProcessCuedActions to see how it
  executes cued actions — specifically if it calls Thread.SendExternalCommand
  while holding the CS.

  Read 1 file (ctrl+o to expand)

● I now have the complete picture. Let me compile the definitive analysis.

  ---
  Root Cause: Confirmed

  The deadlock is caused by three places in TAvecoThread that used Owner.CS 
  (TAutomationInput's own critical section) to protect the AvecoMsgs message 
  queue, instead of a dedicated lock. This was fixed in SVN r40341 (2025-11-17,
  same day as the first dump — the deployment on this machine never received the
  fix).

  What the old code (v14.14) looked like

  TAvecoThread.SafeExecute — REST API path (the critical path):

  repeat
    Owner.Enter;                   // (A) acquires TAutomationInput.CS
    if AvecoMsgs.Count > 0 then
    begin
      amsg := AvecoMsgs[0];
      AvecoMsgs.Delete(0);
    end
    else amsg := nil;
    Owner.Leave;                   // (B) releases CS
    if amsg <> nil then
    begin
      amsg.Result := ProcessFABCmd(amsg.Command, Owner, Self, ...);
      Owner.Enter;                 // (C) re-acquires CS just to call SetEvent
      SetEvent(amsg.Event);
      Owner.Leave;                 // (D) releases CS
    end;
  until WaitForTwoObjects(TerminateEvent, MsgEvent, false, 250) = WAIT_OBJECT_0;

  TAvecoThread.SendExternalCommand (called by REST API controller on receipt of
  an external command):
  Owner.Enter;             // acquires TAutomationInput.CS
  AvecoMsgs.Add(Msg);
  Owner.Leave;             // releases CS
  SetEvent(MsgEvent);
  WaitForSingleObject(Msg.Event, INFINITE);  // waits for REST API thread to 
  process

  TAvecoThread.Destroy:
  Owner.Enter;             // acquires TAutomationInput.CS
  while AvecoMsgs.Count > 0 do ...;
  FreeAndNil(AvecoMsgs);
  Owner.Leave;             // released — properly paired

  The lock contention causing the hang

  TAutomationInput.CS is also used by the main thread for its own data:

  - TAutomationInput.GetAdditionalDevices (called from NewVPS → CheckSubtSource)
  calls Enter/Leave with a potentially long hold spanning
  TranslateIDs.GetSetting, FindPlaylistID, and list scans
  - TAutomationInput.ProcessCuedActions (called from NewTimecode) calls Enter at
  line 5924 and Leave at line 6429 — holding the CS for ~500 lines of
  processing including TranslateID, StoreCurrentID, file writes, and MQTT
  reporting

  While the main thread holds Owner.CS in either of these paths:
  - The REST API TAvecoThread is blocked at Owner.Enter (A) at the START of its
  loop iteration
  - Since the TerminateEvent check (until WaitForTwoObjects(...)) is at the END
  of the loop iteration, a stuck-at-(A) thread cannot see the terminate signal

  The shutdown deadlock sequence

  1. TAutomationInput.Destroy is called during shutdown
  2. ThreadREST.Terminate → sets TerminateEvent
  3. ThreadREST.WaitFor → main thread blocks here
  4. REST API thread is stuck at Owner.Enter step (A), waiting for Owner.CS to
  be released
  5. Owner.CS is held by a VCL/form teardown path that also ends up calling
  WaitFor — or by Owner.Destroy itself if it transitions through code that holds
  the CS while destroying threads
  6. TerminateEvent is set but the REST API thread can't reach the until 
  WaitForTwoObjects(...) check to see it
  7. Both threads wait for each other → process never exits

  The extra Owner.Enter/Owner.Leave bracket around SetEvent (step C/D) adds a
  second lock-acquisition point per iteration, doubling the windows where the
  thread can be stuck.

  What r40341 did to fix it

  // Added CSAvecoMsgs: TRTLCriticalSection to TAvecoThread
  // Constructor now: InitializeCriticalSection(CSAvecoMsgs)

  // SafeExecute REST API loop — FIXED:
  repeat
    amsg := nil;
    EnterAvecoMsgs;          // uses CSAvecoMsgs, NOT Owner.CS
    if AvecoMsgs.Count > 0 then ...;
    LeaveAvecoMsgs;
    if amsg <> nil then
    begin
      ProcessFABCmd(...);
      SetEvent(amsg.Event);  // extra Owner.Enter/Leave REMOVED entirely
    end;
  until WaitForTwoObjects(TerminateEvent, MsgEvent, false, 250) = WAIT_OBJECT_0;

  // SendExternalCommand — FIXED:
  EnterAvecoMsgs;
  AvecoMsgs.Add(Msg);
  LeaveAvecoMsgs;
  ...

  // Destroy — FIXED:
  EnterAvecoMsgs;
  while AvecoMsgs.Count > 0 do ...;
  LeaveAvecoMsgs;           // was Owner.Leave; moved to before FreeAndNil
  FreeAndNil(AvecoMsgs);
  DeleteCriticalSection(CSAvecoMsgs);

  Owner.CS is now only used for TAutomationInput data (playlist IDs,
  configuration, translation tables). AvecoMsgs has its own dedicated,
  short-held, lightweight CS. The REST API thread's loop body no longer touches
  Owner.CS at all, so it can always reach the TerminateEvent check promptly.

  Why this machine is still affected

  All dumps in the log show sttwin.exe 14.14. The fix landed in r40341 the same
  day as the first dump — the binary on this system was never updated past
  14.14. Updating to any build from r40341 onward eliminates the bug.

✻ Cogitated for 20m 43s

※ recap: Investigating why sttwin hangs on close by analyzing stack trace dumps. The root cause in the running binary (v14.14) has been identified. Next: update the deployed binary to any build from SVN r40341 onward. (disable recaps in /config)

> Beautiful, thanks!                                                            

  Recalled 1 memory, wrote 2 memories (ctrl+o to expand)

● Glad it was useful. The key takeaway: the fix already exists in SVN —
  upgrading this site's binary past 14.14 is the action item. The r41420
  CS-pointer change is worth keeping in mind too, since it made all
  TAutomationInput instances share ServerCS, which could surface new contention
  if anything ever holds that CS for extended periods.

✻ Crunched for 27s

> /export                                                                       
  ⎿  Conversation exported to: X:\ms\sttwin\aveco_deadlock.txt