-->

User-mode deadlocks arise when multiple threads, usually of a single application, have blocked each other's access to the same resource. However, multiple threads of multiple applications can also block each other's access to a global/shared resource, such as a global event, or semaphore. Application deadlock example. Some developers simulate cursors by using two or more connections from DB-Library™. One connection performs a select and the other connection performs updates or deletes on the same tables. This can create application deadlocks. For example: Connection A holds a shared lock on a page. JE is capable of notifying your application when it detects a deadlock. (For JE, this is handled in the same way as any lock conflict that a JE application might encounter.) See Managing Deadlocks and other Lock Conflicts for more information.

When a thread needs exclusive access to code or some other resource, it requests a lock. If it can, Windows responds by giving this lock to the thread. At this point, nothing else in the system can access the locked code. This happens all the time and is a normal part of any well-written multithreaded application. Although a particular code segment can only have one lock on it at a time, multiple code segments can each have their own lock.

A deadlock arises when two or more threads have requested locks on two or more resources, in an incompatible sequence. For instance, suppose that Thread One has acquired a lock on Resource A and then requests access to Resource B. Meanwhile, Thread Two has acquired a lock on Resource B and then requests access to Resource A. Neither thread can proceed until the other thread's lock is relinquished, and, therefore, neither thread can proceed.

User-mode deadlocks arise when multiple threads, usually of a single application, have blocked each other's access to the same resource. However, multiple threads of multiple applications can also block each other's access to a global/shared resource, such as a global event, or semaphore.

Kernel-mode deadlocks arise when multiple threads (from the same process or from distinct processes) have blocked each others' access to the same kernel resource.

The procedure used to debug a deadlock depends on whether the deadlock occurs in user mode or in kernel mode.

Debugging a User-Mode Deadlock

When a deadlock occurs in user mode, use the following procedure to debug it:

1. Issue the !ntsdexts.locks extension. In user mode, you can just type !locks at the debugger prompt; the ntsdexts prefix is assumed.

2. This extension displays all the critical sections associated with the current process, along with the ID for the owning thread and the lock count for each critical section. If a critical section has a lock count of zero, it is not locked. Use the ~ (Thread Status) command to see information about the threads that own the other critical sections.

3. Use the kb (Display Stack Backtrace) command for each of these threads to determine whether they are waiting on other critical sections.

4. Using the output of these kb commands, you can find the deadlock: two threads that are each waiting on a lock held by the other thread. In rare cases, a deadlock could be caused by more than two threads holding locks in a circular pattern, but most deadlocks involve only two threads.

Here is an illustration of this procedure. You begin with the !ntdexts.locks extension:

The first critical section displayed has no locks and, therefore, can be ignored.

The second critical section displayed has a lock count of 2 and is, therefore, a possible cause of a deadlock. The owning thread has a thread ID of 0xA3.

You can find this thread by listing all threads with the ~ (Thread Status) command, and looking for the thread with this ID:

In this display, the first item is the debugger's internal thread number. The second item (the Id field) contains two hexadecimal numbers separated by a decimal point. The number before the decimal point is the process ID; the number after the decimal point is the thread ID. In this example, you see that thread ID 0xA3 corresponds to thread number 4.

You then use the kb (Display Stack Backtrace) command to display the stack that corresponds to thread number 4:

Notice that this thread has a call to the WaitForCriticalSection function, which means that not only does it have a lock, it is waiting for code that is locked by something else. We can find out which critical section we are waiting on by looking at the first parameter of the call to WaitForCriticalSection. This is the first address under Args to Child: '24e750'. So this thread is waiting on the critical section at address 0x24E750. This was the third critical section listed by the !locks extension that you used earlier.

In other words, thread 4, which owns the second critical section, is waiting on the third critical section. Now turn your attention to the third critical section, which is also locked. The owning thread has thread ID 0xA9. Returning to the output of the ~ command that you saw previously, note that the thread with this ID is thread number 6. Display the stack backtrace for this thread:

This thread, too, is waiting for a critical section to be freed. In this case, it is waiting on the critical section at 0x68629100. This was the second critical section in the list generated earlier by the !locks extension.

This is the deadlock. Thread 4, which owns the second critical section, is waiting on the third critical section. Thread 6, which owns the third critical section, is waiting on the second critical section.

Having confirmed the nature of this deadlock, you can use the usual debugging techniques to analyze threads 4 and 6.

Debugging a Kernel-Mode Deadlock

There are several debugger extensions that are useful for debugging deadlocks in kernel mode:

• The !kdexts.locks extension displays information about all locks held on kernel resources and the threads holding these locks. (In kernel mode, you can just type !locks at the debugger prompt; the kdexts prefix is assumed.)

• The !qlocks extension displays the state of all queued spin locks.

• The !wdfkd.wdfspinlock extension displays information about a Kernel-Mode Driver Framework (KMDF) spin-lock object.

• The !deadlock extension is used in conjunction with Driver Verifier to detect inconsistent use of locks in your code that have the potential to cause deadlocks.

When a deadlock occurs in kernel mode, use the !kdexts.locks extension to list all the locks currently acquired by threads.

You can usually pinpoint the deadlock by finding one non-executing thread that holds an exclusive lock on a resource that is required by an executing thread. Most of the locks are shared.

A deadlock detection library can be used to track down async/await relateddeadlocks in your code with minimal overhead and effort.

Overview

The async/await pattern greatly simplified writing sequential code thatoperates asynchronous. A lot of libraries now use this pattern to allowcode to be more scalable and responsive. Despite of the simplified code,asynchronous code can still be tricky and a potential source of deadlocks.

Application That Can Deadlock

Most deadlocks are caused by mixing synchronous and asynchronous code andnot dealing with synchronization contexts. The best practice is to use asyncall the way, but this sometimes is not possible. Some examples:

• Constructors, destructors, disposal, event handlers require synchronouscode. It's often better to rethink your design and check if you can use analternative implementation, but sometimes you can't.

• .NET libraries (or third party libraries) assume your code to be synchronous.Starting a Windows service means overriding the OnStart and OnStop methodsthat are synchronous. If you need to call an asynchronous method insidethese methods, then the easiest option is to use Wait() or Result.

If you need to call Task.Wait()or Task<T>.Resultthen your current thread blocks. If one of the underlying tasks require that itneeds to continue on the same synchronization context, then the applicationwill deadlock.

A good starting point is to install the ConfigureAwait Resharper extension,so you don't forget to explicitly mark each await. If your code doesn't dependon the synchronization context then it's often best to callConfigureAwait(false) to prevent deadlocks. If your method is called by externalcode then be cautious when using ConfigureAwait(true) (or discarding it). Itmight deadlock if the caller uses a synchronization context and uses a blockingoperation.

These deadlocks are hard to debug and I have spent several hours debuggingsuch deadlocks for projects that I work on. To prevent even more work, I thoughtof a simple and lightweight library that I could use to find deadlocks earlierand more easy.

Example

Let's illustrate the functionality with an example. Suppose we have thefollowing program that has a potential deadlock:

If this code is run with a synchronization context (i.e. GUI application or anASP.NET request handler), then it will block. The Test method calls theasynchronous TestAsync method and will block until it completes. However, thedelayed task needs to complete on the thread that is now blocked. We haveencountered a deadlock situation.

These situations can be hard to find and to track down. An application may stopresponding or a service won't respond and it will slowly eat up your systemresources, because it will use more and more threads.

The DeadlockDetection library can help to track down these issues with minimaleffort and overhead. The only thing you need to do is to make sure you injecta custom synchronization context. This synchronization context will interceptall task continuations that have require continuation on the synchronizationcontext.

If the synchronization context is blocked, due to a Wait, Result or otherblocking operation then it will raise a DeadlockException when continuationis requested on the blocked thread.

The only thing that you need to do is to install theDeadlockDetectionSynchronizationContext on your current thread. It's bestto use the Enable.DeadlockDetection method for this purpose. If deadlockdetection is disabled (i.e. for production systems) then it makes sure ithas zero side-effects and no performance penalty.

So this code will now look like this:

In this example I have set the detection mode to the most strict mode, so itwill also raise an exception when a potential deadlock might occur. If you runthis code, then the DeadlockException will be thrown when the TestAsyncmethod starts awaiting.

Suppose that you are calling external code or libraries, then deadlocks willstill be detected. Only the calling code needs to be modified.

Caveat

The DeadlockException will be thrown whenever a continuation methods isscheduled while the synchronization context is blocked. It is possible that thesynchronization context is only blocked for a short duration (i.e. Sleep) andit is not actually a deadlock situation.

Although it might be possible that the application will not deadlock, thislibrary will still raise the DeadlockException. It might be considered a bug,but you might also rethink your design. Mixing blocking calls and asynchronouscalls on the same synchronization context is a bit awkward and reduces theasynchronous behavior of your application (and therefore reduce scalability),so you might even consider this a feature that you'll be notified of thesecircumstances :-)

Application That Can Deadlock Go

Settings

There are some global settings that can be set to change the behavior of thedeadlock detection library.

DeadlockDetection.GlobalSettings.DefaultDetectionMode

This setting specifies how deadlocks are detected:

• Disabled will disable the entire library and the code will run exactlyas if the library wasn't used at all. It won't affect performance orbehavior in any way. It's okay to leave the usings (without setting thedetection mode) in production code this way.
• OnlyActualDeadlocks will only raise exceptions when the deadlock detectionis explicitly enabled via the using block. Potential deadlocks will not bedetected.
• AlsoPotentialDeadlocks will raise exceptions when both actual or potentialdeadlocks are detected. This should only be enabled during development and/ortesting.

DeadlockDetection.GlobalSettings.GenerateStackTraces

This setting specifies if stack traces are generated during the detection of(potential) deadlocks. Although it is convenient to have the location of theblocking operation, the generation of stack traces might affect performance,so it is best to disable it when not running in development and/or testing.

Under the hood

The deadlock detection library uses existing facilities in theSynchronizationContext class to detect deadlocks. That's why I was able toimplement it without any overhead. It uses the following two techniques:

1. When a task is continued on the synchronization context, then the CLRwill use the SynchronizationContext.Post method to call the continuationmethod.
2. Synchronization contexts can opt-in to be notified when it blocks (viathe SynchronizationContext.SetWaitNotificationRequired method). Itcalls the SynchronizationContext.Wait method to block the thread.

The DeadlockDetectionSynchronizationContext opts-in for the wait notificationand sets the _isBlocked flag. If the continuation method is invoked via thePost method of our synchronization context, then it will check if the contextis currently blocked and if so, it throws the DeadlockException.

Integrating in your application

Enabling deadlock detection should be done at the level where thesynchronization context is known and won't change anymore. I will provideseveral examples on how to integrate it the least intrusive way and maximizethe effect.

First you start with installing it via NuGet

Integrating in ASP.NET applications

ASP.NET applications use a synchronization context that is unique for eachrequest. It provides access to the current HTTP context and can be used toflow the principal through the request handler code.

It's important to wrap the proper synchronization context, so to enable Thedeadlock detection it must be enabled in the middleware.

ASP.NET applications using a global.asax file the following lines of codeshould be added to the file:

This will enable deadlock detection for each request. When the request hasfinished, the synchronization context will be reset again.

Integration in WinForms applications

Integrating in a WinForms application is pretty easy and relativenon-intrusive. The typical Main method of a WinForm application looks likethis:

The deadlock detection synchronization context only needs to be installed onthe UI thread. The WinForms library automatically installs theWindowsFormsSynchronizationContext during the creation of the first Windowscontrol. Because the deadlock detection synchronization context needs to wrapthe synchronization context, we need to make sure the synchronization contextis created.

We can simply create a dummy control, which will install the propersynchronization context. We can then wrap it and run the actual application.The Main method will look something like this:

This will enable automatic deadlock detection on the GUI thread.

Integration in WPF applications

I am currently investigating how to inject the deadlock detectionsynchronization context in a non-intrusive way for WPF applications. thedifficulty is that for each message that is processed internally a newDispatcherSynchronizationContext is created. This makes it moredifficult to inject it globally in your application.

You can use the local detection method (described below) if you want todetect deadlocks in a part of your code.

What about other applications?

Normal console applications or Windows services don't have a synchronizationcontext, so they will never generate a deadlock situation. So i is not necessaryto have deadlock detection for these kind of applications.

Local detection

Application That Can Deadlock Get

If you don't want to enable the deadlock detection for your entire application,but only for a certain point (i.e. during debugging) you can simply wrap thesynchronization context.

Suppose you know you have a deadlock when clicking a certain button, then youcan use the following code:

Within the using block the deadlocks will be detected.