rusanu.com

Update: a version of this sample that accepts parameters is available in the post Passing Parameters to a Background Procedure

Recently an user on StackOverflow raised the question Execute a stored procedure from a windows form asynchronously and then disconnect?. This is a known problem, how to invoke a long running procedure on SQL Server without constraining the client to wait for the procedure execution to terminate. Most times I’ve seen this question raised in the context of web applications when waiting for a result means delaying the response to the client browser. On Web apps the time constraint is even more drastic, the developer often desires to launch the procedure and immediately return the page even when the execution lasts only few seconds. The application will retrieve the execution result later, usually via an Ajax call driven by the returned page script.

Frankly I was a bit surprised to see that the responses gravitated either around the SqlClient asynchronous methods (BeginExecute…) or around having a dedicated process with the sole pupose of maintaining the client connection alive for the duration of the long running procedure.

This problem is perfectly addressed by Service Broker Activation. Since I wanted to preserve the solution for further reference, I decided to put it in as a blog entry, with additional comments. For many of you Service Broker aficionados that read my blog regularly, this article is not innovative as is mostly a rehash of well known techniques I’ve been talking about on forums for many years now.

I’m going to use a table to store the result of the procedure execution. In this version I’ll keep things simple by not allowing for any parameters to be passed to the procedure, nor collecting any execution result set data. So the table will only contain the procedure start time, the execution finish time and any error that occurred during the procedure execution:

create table [AsyncExecResults] (
	[token] uniqueidentifier primary key
	, [submit_time] datetime not null
	, [start_time] datetime null
	, [finish_time] datetime null
	, [error_number]	int null
	, [error_message] nvarchar(2048) null);
go

Next we’re going to create the service and queue we need. I will use one single service for both roles (initiator and target) and I won’t create an explicit contract, relying instead on the predefined DEFAULT contract:

create queue [AsyncExecQueue];
go

create service [AsyncExecService] on queue [AsyncExecQueue] ([DEFAULT]);
GO

Next is the core of our asynchronous execution: the activated procedure. The procedure has to dequeue the message that specifies the user procedure, run the procedure and write the result in the results table. I will also deploy the error handling template I elaborated on my previous article Exception handling and nested transactions:

create procedure usp_AsyncExecActivated
as
begin
    set nocount on;
    declare @h uniqueidentifier
        , @messageTypeName sysname
        , @messageBody varbinary(max)
        , @xmlBody xml
        , @procedureName sysname
        , @startTime datetime
        , @finishTime datetime
        , @execErrorNumber int
        , @execErrorMessage nvarchar(2048)
        , @xactState smallint
        , @token uniqueidentifier;

    begin transaction;
    begin try;
        receive top(1) 
            @h = [conversation_handle]
            , @messageTypeName = [message_type_name]
            , @messageBody = [message_body]
            from [AsyncExecQueue];
        if (@h is not null)
        begin
            if (@messageTypeName = N'DEFAULT')
            begin
                -- The DEFAULT message type is a procedure invocation.
                -- Extract the name of the procedure from the message body.
                --
                select @xmlBody = CAST(@messageBody as xml);
                select @procedureName = @xmlBody.value(
                    '(//procedure/name)[1]'
                    , 'sysname');

                save transaction usp_AsyncExec_procedure;
                select @startTime = GETUTCDATE();
                begin try
                    exec @procedureName;
                end try
                begin catch
                -- This catch block tries to deal with failures of the procedure execution
                -- If possible it rolls back to the savepoint created earlier, allowing
                -- the activated procedure to continue. If the executed procedure 
                -- raises an error with severity 16 or higher, it will doom the transaction
                -- and thus rollback the RECEIVE. Such case will be a poison message,
                -- resulting in the queue disabling.
                --
                select @execErrorNumber = ERROR_NUMBER(),
                    @execErrorMessage = ERROR_MESSAGE(),
                    @xactState = XACT_STATE();
                if (@xactState = -1)
                begin
                    rollback;
                    raiserror(N'Unrecoverable error in procedure %s: %i: %s', 16, 10,
                        @procedureName, @execErrorNumber, @execErrorMessage);
                end
                else if (@xactState = 1)
                begin
                    rollback transaction usp_AsyncExec_procedure;
                end
                end catch

                select @finishTime = GETUTCDATE();
                select @token = [conversation_id] 
                    from sys.conversation_endpoints 
                    where [conversation_handle] = @h;
                if (@token is null)
                begin
                    raiserror(N'Internal consistency error: conversation not found', 16, 20);
                end
                update [AsyncExecResults] set
                    [start_time] = @starttime
                    , [finish_time] = @finishTime
                    , [error_number] = @execErrorNumber
                    , [error_message] = @execErrorMessage
                    where [token] = @token;
                if (0 = @@ROWCOUNT)
                begin
                    raiserror(N'Internal consistency error: token not found', 16, 30);
                end
                end conversation @h;
            end 
            else if (@messageTypeName = N'http://schemas.microsoft.com/SQL/ServiceBroker/EndDialog')
            begin
                end conversation @h;
            end
            else if (@messageTypeName = N'http://schemas.microsoft.com/SQL/ServiceBroker/Error')
            begin
                declare @errorNumber int
                    , @errorMessage nvarchar(4000);
                select @xmlBody = CAST(@messageBody as xml);
                with xmlnamespaces (DEFAULT N'http://schemas.microsoft.com/SQL/ServiceBroker/Error')
                select @errorNumber = @xmlBody.value ('(/Error/Code)[1]', 'INT'),
                    @errorMessage = @xmlBody.value ('(/Error/Description)[1]', 'NVARCHAR(4000)');
                -- Update the request with the received error
                select @token = [conversation_id] 
                    from sys.conversation_endpoints 
                    where [conversation_handle] = @h;
                update [AsyncExecResults] set
                    [error_number] = @errorNumber
                    , [error_message] = @errorMessage
                    where [token] = @token;
                end conversation @h;
             end
           else
           begin
                raiserror(N'Received unexpected message type: %s', 16, 50, @messageTypeName);
           end
        end
        commit;
    end try
    begin catch
        declare @error int
            , @message nvarchar(2048);
        select @error = ERROR_NUMBER()
            , @message = ERROR_MESSAGE()
            , @xactState = XACT_STATE();
        if (@xactState <> 0)
        begin
            rollback;
        end;
        raiserror(N'Error: %i, %s', 1, 60,  @error, @message) with log;
    end catch
end
go

To make the procedure activated we need to attach it to our service queue. This will ensure this procedure is run whenever a message arrives to our [AsyncExecService]:

alter queue [AsyncExecQueue]
    with activation (
    procedure_name = [usp_AsyncExecActivated]
    , max_queue_readers = 1
    , execute as owner
    , status = on);
go

And finaly the last piece of the puzzle: the procedure that submits the message to invoke the desired asyncronous executed procedure. This procedure resturns an output parameter ‘token’ than can be used to lookup the asynchronous execution result.

create procedure [usp_AsyncExecInvoke]
    @procedureName sysname
    , @token uniqueidentifier output
as
begin
    declare @h uniqueidentifier
	    , @xmlBody xml
        , @trancount int;
    set nocount on;

	set @trancount = @@trancount;
    if @trancount = 0
        begin transaction
    else
        save transaction usp_AsyncExecInvoke;
    begin try
        begin dialog conversation @h
            from service [AsyncExecService]
            to service N'AsyncExecService', 'current database'
            with encryption = off;
        select @token = [conversation_id]
            from sys.conversation_endpoints
            where [conversation_handle] = @h;
        select @xmlBody = (
            select @procedureName as [name]
            for xml path('procedure'), type);
        send on conversation @h (@xmlBody);
        insert into [AsyncExecResults]
            ([token], [submit_time])
            values
            (@token, getutcdate());
    if @trancount = 0
        commit;
    end try
    begin catch
        declare @error int
            , @message nvarchar(2048)
            , @xactState smallint;
        select @error = ERROR_NUMBER()
            , @message = ERROR_MESSAGE()
            , @xactState = XACT_STATE();
        if @xactState = -1
            rollback;
        if @xactState = 1 and @trancount = 0
            rollback
        if @xactState = 1 and @trancount > 0
            rollback transaction usp_my_procedure_name;

        raiserror(N'Error: %i, %s', 16, 1, @error, @message);
    end catch
end
go

To test our asynchronous execution infrastructure we create a test procedure and invoke it asynchronously. I will create two test procedures, one that simply waits for 5 seconds to simulate a ‘long’ running procedure and one that produces intentionally a primary key violation, to simulate a fault in the asynchronously executed procedure:

create procedure [usp_MyLongRunningProcedure]
as
begin
    waitfor delay '00:00:05';
end
go

create procedure [usp_MyFaultyProcedure]
as
begin
    set nocount on;
    declare @t table (id int primary key);
    insert into @t (id) values (1);
    insert into @t (id) values (1);
end
go


declare @token uniqueidentifier;
exec usp_AsyncExecInvoke N'usp_MyLongRunningProcedure', @token output;
select * from [AsyncExecResults] where [token] = @token;
go

declare @token uniqueidentifier;
exec usp_AsyncExecInvoke N'usp_MyFaultyProcedure', @token output;
select * from [AsyncExecResults] where [token] = @token;
go

waitfor delay '00:00:10';
select * from [AsyncExecResults];
go

Activation Context

If you check the start time of the second asynchronosuly executed procedure you will notice that it started right after the first one finished. This is because we declare a max_queue_readers value of 1 when we set up activation on the queue. This restricts that at most one activated procedure to run at any time, effectively serializing all the asynchronously executed procedures. Whether this is desired or not depends a lot on the actual usage scenario. The limit can be increased as necessary.

If you start playing around with this method of invoking procedures asynchronously you will notice that sometimes the asynchrnously executed procedure is misteriously denied access to other databases or to server scoped objects. When the same procedure is run manually from a query window in SSMS, it executes fine. This is caused by the EXECUTE AS context under which activation occurs. the details are explained in MSDN’s Extending Database Impersonation by Using EXECUTE AS and myself I had covered this subject repeatedly in this blog. The best solution is to simply turn the trustworthy bit on on the database where the activated procedure runs. When this is not desired, or not allowed by your hosting environment, the solution is to code sign the activated procedure: Signing an activated procedure.

Using Service Broker Activation to invoke procedures asynchronously may look daunting at beginning. It sure is significantly more complex than just calling BeginExecuteNonQuery. But what needs to be understood is that this is a reliable way to invoke the procedure. The client is free to disconnect as soon as it commited the call to usp_AsyncExecInvoke. The procedure invoked will run, even if the server is stopped and restarted, even if a mirroring or clustering failover occurs. The server may even crash and be completely rebuilt. As soon as the database is back online, that queue will activate and invoke the asynchronous execution. Such level of reliability is difficult, if not impossible, to guarantee by using a client process.

I just found that Microsoft has published a use case about the way MySpace is using Service Broker on their service as the core message delivery system for the Service Dispatcher. We’re talking here 440 SQL Server instances and over 1000 databases. Quote from the use case:

Service Broker has enabled MySpace to perform foreign key management across its 440 database servers, activating and deactivating accounts for its millions of users, with one-touch asynchronous efficiency. MySpace also uses Service Broker administratively to distribute new stored procedures and other updates across all 440 database servers through the Service Dispatcher infrastructure.

That is pretty impressive. I knew about the MySpace SSB adoption since the days when I was with the Service Broker team. You probably all know my mantra I repeat all the time “don’t use fire and forget, is a bad message exchange pattern and there are scenarios when the database may be taken offline”? Guess how I found out those ‘scenarios’… Anyway, I’m really glad that they also made public some performance numbers. Until now I could only quote the 5000 message per second I can push in my own test test environment. Well, looks like MySpace has some beefier hardware:

Stelzmuller: “When we went to the lab we brought our own workloads to ensure the quality of the testing. We needed to see if Service Broker could handle loads of 4,000 messages per second. Our testing found it could handle more than 18,000 messages a second.”

Sometimes you are faced with applications that seem to take ages to start up. Usually they freeze for about 30-40 seconds and then all of the sudden they come to live. This happens for both native and managed application and it sometimes manifest as an IIS/ASP/ASP.Net AppPool starting up slow on the first request. The very first thing I always suspect is code signing verification. When a signed module is checked the certificate verification engine may consider that the Certificate Revocation List (CRL) it posses is obsolete and attempt to download a new one. For this it connects to the internet. The problem occurs when the connectivity is either slow, or blocked for some reason. By default the verification engine will time out after 15 seconds and resume with the old, obsolete, CRL it has. The timeout can occur several times, adding up to start up times of even minutes. This occurs completely outside of the control of the application being started, its modules are not even properly wired up in memory so there is no question of application code yet running.

The information on this subject is scarce to say the least. Luckily there is an TechNet article that describes not only the process occuring, but also the controlling parameters: Certificate Revocation and Status Checking. To fix the problem on computers with poor internet conectivity, registry settings have to be modified in the HKLM\SOFTWARE\Microsoft\Cryptography\OID\EncodingType 0\CertDllCreateCertificateChainEngine\Config key:

ChainUrlRetrievalTimeoutMilliseconds

This is each individual CRL check call timeout. If is 0 or not present the default value of 15 seconds is used. Change this timeout to a reasonable value like 200 milliseconds.

ChainRevAccumulativeUrlRetrievalTimeoutMilliseconds

This is the aggregate CRL retrieval timeout. If set to 0 or not present the default value of 20 seconds is used. Change this timeout to a value like 500 milliseconds.

With these two changes the code signing verification engine will timeout the CRL refresh operation in 500 milliseconds. If the connectivity to the certificate authority site is bad, this will dramatically increase the application start up times for code signed applications.

Update

Andrei Ignat pointed out there is a class in the .Net Framework that deals exactly with this issue: NetworkChange Class. That is definitely a better approach than leveraging the native API through p-Invoke. The MSDN sample also shows how to check for network availability on each interface, when notified of a change.

Original post:

A common problem for applications that use the network stack is to detect when they are connected to the network. It would be ideal if the application was notified when you plugged in your network cable into your laptop, or when it came in range of a WiFi station and has joined a wireless network.

The Windows API has a function for registering for exactly such notifications: NotifyAddrChange. An application can register a wait handle with this function and the handle will be signaled by the operating system every time a change occurs in the IP address to interface mapping table. Technically is true that ‘IP address table change’ does mean strictly that ‘network connectivity has changed’, but in reality the overwhelming majority of IP address table changes can be associated with the network connectivity being gained or lost.

To leverage this API from managed code we have to use p-Invoke. I use a helper class that wraps the registration for notification and handling the event notification in a simple to use package:

    /// <summary>
    /// Raises an event each time a change occurs 
    /// on the system IP address table.
    /// </summary>
    public class NotifyAddressChanges: IDisposable
    {
        #region State fields
        /// <summary>
        /// The OVERLAPPED structure passed to NotifyAddrChange
        /// </summary>
        private NativeOverlapped _overlapped;

        /// <summary>
        /// File handle created  by NotifyAddrChange
        /// </summary>
        private IntPtr _handle;

        /// <summary>
        /// The event signaled by NotifyAddrChange
        /// </summary>
        private AutoResetEvent _eventChanged;

        /// <summary>
        /// The thread pool notification registration
        /// </summary>
        private RegisteredWaitHandle _registered;

        /// <summary>
        /// Submitted state tracking
        /// </summary>
        private volatile int _submitted;

        #endregion

        #region Win32 API declarations

        [DllImport("Iphlpapi.dll", SetLastError = true)]
        private static extern UInt32 NotifyAddrChange(
            ref IntPtr Handle, 
            ref NativeOverlapped overlapped);
        [DllImport("Iphlpapi.dll", SetLastError = true)]
        private static extern bool CancelIPChangeNotify(
            ref NativeOverlapped overlapped);
        #endregion

        #region Public usable methods and events

        /// <summary>
        /// The event raised when a change in the system
        /// IP address table occurs. 
        /// </summary>
        public event EventHandler AddressChanged;


        /// <summary>
        /// CTOR. Initializes the notifications event and submits it
        /// to the ThreadPool. Caller need to call Submit() to enable
        /// system IP address notifications to raise the AddressChange event.
        /// </summary>
        public NotifyAddressChanges()
        {
            _submitted = 0;
            _eventChanged = new AutoResetEvent(false);
            _overlapped.EventHandle = _eventChanged.SafeWaitHandle.DangerousGetHandle();

            _registered = ThreadPool.RegisterWaitForSingleObject(
                _eventChanged,
                new WaitOrTimerCallback(AddressChangedNotification),
                null,
                Timeout.Infinite,
                false);
        }

        /// <summary>
        /// Enables the system IP address notifications to raise the AddressChange event.
        /// </summary>
        public void Submit()
        {
            if (0 == Interlocked.CompareExchange(
                ref _submitted, 1, 0))
            {
                NotifyAddrChange(ref _handle, ref _overlapped);
            }
        }

        #endregion

        #region Callback and event raising

        /// <summary>
        /// IP address change notification callback.
        /// </summary>
        private void AddressChangedNotification(object state, bool timedOut)
        {
            if (1 == Interlocked.CompareExchange(
                ref _submitted, 0, 1))
            {
                EventHandler handler = AddressChanged;
                if (null != handler)
                {
                    try
                    {
                        handler(this, EventArgs.Empty);
                    }
                    catch (Exception)
                    {
                        // silent ignore
                    }
                }
                Submit();
            }
        }
        #endregion

        #region Dispose Logic
        private void Dispose(bool userInvoked)
        {
            if (null != _registered)
            {
                _registered.Unregister(_eventChanged);
            }
            if (1 == Interlocked.CompareExchange(
                ref _submitted, 0, 1))
            {
                CancelIPChangeNotify(ref _overlapped);
            }
        }

        ~NotifyAddressChanges()
        {
            Dispose(false);
        }


        public void Dispose()
        {
            Dispose(true);
            GC.SuppressFinalize(this);
        }

        #endregion
    }

To use this class, simply create an instance, add an event handler to the AddressChanged event and then call the Submit() method to start getting notifications. The AddressChanged event will be raised each time the system changes an IP address on any of its network interfaces. For example consider a simple form named Form1 to which I added a TextBox named textbox1 (in other words I created a new Windows forms application and I dropped a multiline text box on its surface, without bothering to rename either). Here is how I can hook up the IP address change event to display ‘changed’ on a new line in the text box:

    public partial class Form1 : Form
    {
        private NotifyAddressChanges _notification;

        public Form1()
        {
            InitializeComponent();

            _notification = new NotifyAddressChanges();
            _notification.AddressChanged += 
                new EventHandler(_notification_AddressChanged);
            _notification.Submit();
        }

        void _notification_AddressChanged(object sender, EventArgs e)
        {
            Invoke(new MethodInvoker(delegate ()
            {
                textBox1.AppendText("changed\r\n");
            }));
        }
    }

I can now start my test application and play with the network cable. As I unplug and plug back the cable, the system looses and re-gains the IP address from my base station DHCP server and it signals the NotifyAddrChange registered events. My form gets notified and it adds a “changed” line to the text box. I can also release and renew my DHCP license from the command line running ipconfig /release and ipconfig /renew.

Note that this notification occurs with any change, both on IP4 and IPv6 interfaces. You will be notified if a VPN is connected or disconnected, if a dial-up connection is established and so on and so forth. Being notified does not mean that network is available, nor that is unavailable. It simply means a change occurred. While I could add support to the AddressChanged event to indicate what change, that would needlessly complicate this simple class, because the number of possible changes is actually much bigger than one would expect (think again at all those VPN, dial-up, point-to-point and other connections). My goal with this class was to eliminate the need to periodically wake the application and check if the network is available every minute, based on a timer. I can simply wait for the AddressChanged notification and then I can try to see if I can establish a connection.

I am following the 37signals blog ever since I kinda randomly stumbled upon their Getting Real book. If you never heard about them, definitely check out the book, is a very common sense approach to managing product development in the age of Internets. On this post http://www.37signals.com/svn/posts/1798-jasons-talk-at-big-omaha-2009 I was really touched by one remark: Inspiration is perishable. The ideas you have can linger in your head for a long time, but the inspiration for it fades quickly. So don’t postpone it, by the time you get to it you’ll only deliver a pale image of the original idea. Do it when you’re pumped up and thrilled by it.

I reckon I’m a procrastinator deLuxe, but I have to agree. I know the difference between working at 2 am. and not feeling a bit tired when I’m excited about my work on one hand, and the damp feeling of exhaustion that drags you to watch some stupid TV show at 6 pm because I’m bored with the current project on the other hand.