Performance comparison of varchar(max) vs. varchar(N)

March 22nd, 2010

The question of comparing the MAX types (VARCHAR, NVARCHAR, VARBINARY) with their non-max counterparts is often asked, but the answer usually gravitate around the storage differences. But I’d like to address the point that these types have inherent, intrinsic performance differences that are not driven by different storage characteristics. In other words, simply comparing and manipulating variables and columns in T-SQL can yield different performance when VARCHAR(MAX) is used vs. VARCHAR(N).

Assignment

First comparing simple assignment, assign a value to a VARBINARY(8000) variable in a tight loop:

declare @x varchar(8000);
declare @startTime datetime;
declare @i int;

set @i = 0;
set @startTime = getutcdate();

while @i < 1000000
begin
  set @x = 'abc';
  set @i = @i + 1;
end

select datediff(ms, @startTime, getutcdate());

This script runs on my test server in 4.9 seconds on average. Simply changing the variable declaration to VARBINARY(MAX) changes the run time to an average of of 9.2 seconds.

Comparison

Next I measured the performance of a simple comparison:

declare @x varchar(8000);
declare @startTime datetime;
declare @i int;

set @i = 0;
set @startTime = getutcdate();
set @x = 'abc';


while @i < 1000000
begin
  declare @y bit;
  set @y = case when @x = 'abc' then 1 else 0 end;
  set @i = @i + 1;
end

select datediff(ms, @startTime, getutcdate());

Average run time for VARCHAR(8000): 5.8 seconds. For VARCHAR(MAX): 6.5 seconds.

Data Access

The next text does a simple string comparison of a VARCHAR(MAX) field vs. a VARCHAR(8000) field. The actual value stored is identical in both cases, a simple 3 letter string ‘abc’. The data access does not retrieve the string value, it simply compares its value against a WHERE clause predicate:

create table test (
  id int identity(1,1) primary key, 
  x varchar(max));
go

insert into test(x) values ('abc');
go

declare @x varchar(max);
declare @startTime datetime;
declare @i int;

set nocount on;
set @i = 0;
set @startTime = getutcdate();

while @i < 100000
begin
  declare @id int;
  select @id = id 
    from test 
    where id = 1
    and x = 'abc';
  set @i = @i + 1;
end

select datediff(ms, @startTime, getutcdate());

Loop execution time for VARCHAR(8000): 3.1 seconds. For VARCHAR(MAX): 3.6 seconds.

Furthermore, if we change the WHERE predicate to compare against a max type variable instead of the literal ‘abc‘ the loop time increases to 3.9 seconds.

Conclusion

The code path that handles the MAX types (varchar, nvarchar and varbinary) is different from the code path that handles their equivalent non-max length types. The non-max types can internally be represented as an ordinary pointer-and-length structure. But the max types cannot be stored internally as a contiguous memory area, since they can possibly grow up to 2Gb. So they have to be represented by a streaming interface, similar to COM’s IStream. This carries over to every operation that involves the max types, including simple assignment and comparison, since these operations are more complicated over a streaming interface. The biggest impact is visible in the code that allocates and assign max-type variables (my first test), but the impact is visible on every operation.

The impact is reasonable on most operations with the max type operations being about 10% slower. So is this something to worry about, or even worth blogging about, after all? Actually, in a recent project of mine I had to change a column type from varchar(max) to varchar(5000) to alleviate the impact of max-types performance. The code happened to be on a very very performance critical path and testing clearly showed the impact was not only measurable, was quite actually quite serious. The processing throughput increased by about 25% just from this minor change in my case.

Although much discussion exists around the storage of max types vs. the storage of non-max types and the similarities and differences between them (eg. in-row vs. out-of-row vs. row-overflow storage), there is also a more fundamental aspect that differentiate these types: SQL Server internal code differences. These differences manifest themselves even when the storage of the max type is optimized in-row, for small and very small values of the BLOB field.

select count(*);

October 26th, 2009

Quick trivia: what is the result of running SELECT COUNT(*);?

That’s right, no FROM clause, just COUNT(*). The answer may be a little bit surprising, is 1. When you query SELECT 1; the result is, as expected, 1. And SELECT 2; will return 2. So SELECT COUNT(2); returns, as expected, 1, after all it counts how many rows are in the result set. But SELECT COUNT(*); has a certain smell of voo-doo to it. Ok, is the * project operator, but project from… what exactly? It feels eerie, like a count is materialized out of the blue.

How about SELECT COUNT(*) [MyTable]. Well, that’s actually just a shortcut for SELECT COUNT(*) AS [MyTable], so it still returns 1 but in a column named MyTable. Now you understand why my heart missed a bit when I checked how I initialized a replication subscription and I forgot to type in FROM

Passing Parameters to a Background Procedure

August 18th, 2009

I have posted previously an example how to invoke a procedure asynchronously using service Broker activation. Several readers have inquired how to extend this mechanism to add parameters to the background launched procedure.

Passing parameters to a single well know procedure is easy: the parameters are be added to the message body and the activated procedure looks them up in the received XML, passing them to the called procedure. But is significantly more complex to create a generic mechanism that can pass parameters to any procedure. The problem is the type system, because the parameters have unknown types and the activated procedure has to pass proper typed parameters to the invoked procedure.

A generic solution should accept a variety of parameter types and should deal with the peculiarities of Transact-SQL parameters passing, namely the named parameters capabilities. Also the invocation wrapper usp_AsyncExecInvoke should directly accept the parameters for the desired background procedure. After considering several alternatives, I settled on the following approach:

  • Rely on the generic sql_variant data type available in SQL Server. The invocation wrapper usp_AsyncExecInvoke accept all the parameters as sql_variant.
  • Pass the background procedure parameter names explicitly to the invocation wrapper usp_AsyncExecInvoke.
  • Always used named parameters in the background procedure invocation.
  • Build a dynamic SQL batch to invoke the background procedure that deals with the required parameter type conversions.

Accepting Parameters

The invocation wrapper usp_AsyncExecInvoke has changed the signature to accept a variable number of parameters:

create procedure [usp_AsyncExecInvoke]
    @procedureName sysname
    , @p1 sql_variant = NULL, @n1 sysname = NULL
    , @p2 sql_variant = NULL, @n2 sysname = NULL
    , @p3 sql_variant = NULL, @n3 sysname = NULL
    , @p4 sql_variant = NULL, @n4 sysname = NULL
    , @p5 sql_variant = NULL, @n5 sysname = NULL
    , @token uniqueidentifier output

The new parameters are used to collect the desired parameter values and names to be passed to the background procedure. So sonsider you would like to make the following traditional, synchronous, call:

exec usp_withParam @id = 1.0
	, @name = N'Foo'
	, @bytes = 0xBAADF00D;

The equivalent asynchronous invocation would need to pass the parameter values (1.0, 'Foo' and 0xbaadf00d) as @p1, @p2 and @p3 respectively and the parameter names (@id, @name and @bytes) as @n1, @n2 and @n3:

exec usp_AsyncExecInvoke @procedureName = N'usp_withParam'
 , @p1 = 1.0, @n1 = N'@id'
 , @p2 = N'Foo', @n2='@name'
 , @p3 = 0xBAADF00D, @n3 = '@bytes'
 , @token = @token output;

The invocation wrapper will pack the parameter names, values and type information into the message body. The parameter type information is deduced at run time from the actual parameters, by using the SQL_VARIANT_PROPERTY system function. This is the XML message body corresponding to the invocation above (note how the varbinary parameter was encoded using base64 encoding):

<procedure>
  <name>usp_withParam</name>
  <parameters>
    <parameter Name="@id" BaseType="numeric" Precision="2" Scale="1" MaxLength="5">1.0</parameter>
    <parameter Name="@name" BaseType="nvarchar" Precision="0" Scale="0" MaxLength="6">Foo</parameter>
    <parameter Name="@bytes" BaseType="varbinary" 
          Precision="0" Scale="0" MaxLength="4">uq3wDQ==</parameter>
  </parameters>
</procedure>

Extracting the Parameters

The activated procedure job is now significantly more complex because of the task of extracting the parameter names, types and value and constructing a proper Transact-SQL statement to invoke the original procedure. A new helper procedure was added, usp_procedureInvokeHelper, that constructs a dynamic SQL batch from a invokation message body. Here is the dynamic SQL that results from our sample invocation:

declare @pt1 numeric(2,1)
declare @pt2 nvarchar(6)
declare @pt3 varbinary(4)
select @pt1=@x.value(N'(//procedure/parameters/parameter)[1]', N'numeric(2,1)');
select @pt2=@x.value(N'(//procedure/parameters/parameter)[2]', N'nvarchar(6)');
select @pt3=@x.value(N'(//procedure/parameters/parameter)[3]', N'varbinary(4)');
exec [usp_withParam]  @id=@pt1,@name=@pt2,@bytes=@pt3

Limitations

Because the invocation wrapper uses sql_variant typed parameters, all sql_variant restrictions apply to this sample. Most notably the wrapper will not accept XML type parameters nor large data types like varchar(max) or varbinary(max). It is possible to work around these limitations but it would complicate my sample beyond the point of being usefull as an example how to pass the parameters.

Because the error handling of the activated procedure will rollback in case of error, passing in a message that results in a procedure invocation error will cause the activation to rollback repeatedly and the poison message detection mechanism will kick in, deactivating the queue. A trivial example is if we omit the required @id parameter for our sample invocation.

The code

create table [AsyncExecResults] (
 [token] uniqueidentifier primary key
 , [submit_time] datetime not null
 , [start_time] datetime null
 , [finish_time] datetime null
 , [error_number] int null
 , [error_message] nvarchar(2048) null);
go

create queue [AsyncExecQueue];
go

create service [AsyncExecService] on queue [AsyncExecQueue] ([DEFAULT]);
GO

-- Dynamic SQL helper procedure
-- Extracts the parameters from the message body
-- Creates the invocation Transact-SQL batch
-- Invokes the dynmic SQL batch
create procedure [usp_procedureInvokeHelper] (@x xml)
as
begin
    set nocount on;

    declare @stmt nvarchar(max)
        , @stmtDeclarations nvarchar(max)
        , @stmtValues nvarchar(max)
        , @i int
        , @countParams int
        , @namedParams nvarchar(max)
        , @paramName sysname
        , @paramType sysname
        , @paramPrecision int
        , @paramScale int
        , @paramLength int
        , @paramTypeFull nvarchar(300)
        , @comma nchar(1)

    select @i = 0
        , @stmtDeclarations = N''
        , @stmtValues = N''
        , @namedParams = N''
        , @comma = N''

    declare crsParam cursor forward_only static read_only for
        select x.value(N'@Name', N'sysname')
            , x.value(N'@BaseType', N'sysname')
            , x.value(N'@Precision', N'int')
            , x.value(N'@Scale', N'int')
            , x.value(N'@MaxLength', N'int')
        from @x.nodes(N'//procedure/parameters/parameter') t(x);
    open crsParam;

    fetch next from crsParam into @paramName
        , @paramType
        , @paramPrecision
        , @paramScale
        , @paramLength;
    while (@@fetch_status = 0)
    begin
        select @i = @i + 1;

        select @paramTypeFull = @paramType +
            case
            when @paramType in (N'varchar'
                , N'nvarchar'
                , N'varbinary'
                , N'char'
                , N'nchar'
                , N'binary') then
                N'(' + cast(@paramLength as nvarchar(5)) + N')'
            when @paramType in (N'numeric') then
                N'(' + cast(@paramPrecision as nvarchar(10)) + N',' +
                cast(@paramScale as nvarchar(10))+ N')'
            else N''
            end;

        -- Some basic sanity check on the input XML
        if (@paramName is NULL
            or @paramType is NULL
            or @paramTypeFull is NULL
            or charindex(N'''', @paramName) > 0
            or charindex(N'''', @paramTypeFull) > 0)
            raiserror(N'Incorrect parameter attributes %i: %s:%s %i:%i:%i'
                , 16, 10, @i, @paramName, @paramType
                , @paramPrecision, @paramScale, @paramLength);

        select @stmtDeclarations = @stmtDeclarations + N'
declare @pt' + cast(@i as varchar(3)) + N' ' + @paramTypeFull
            , @stmtValues = @stmtValues + N'
select @pt' + cast(@i as varchar(3)) + N'=@x.value(
    N''(//procedure/parameters/parameter)[' + cast(@i as varchar(3)) 
                + N']'', N''' + @paramTypeFull + ''');'
            , @namedParams = @namedParams + @comma + @paramName
                + N'=@pt' + cast(@i as varchar(3));

        select @comma = N',';

        fetch next from crsParam into @paramName
            , @paramType
            , @paramPrecision
            , @paramScale
            , @paramLength;
    end

    close crsParam;
    deallocate crsParam;        

    select @stmt = @stmtDeclarations + @stmtValues + N'
exec ' + quotename(@x.value(N'(//procedure/name)[1]', N'sysname'));

    if (@namedParams != N'')
        select @stmt = @stmt + N' ' + @namedParams;

    exec sp_executesql @stmt, N'@x xml', @x;
end
go

create procedure usp_AsyncExecActivated
as
begin
    set nocount on;
    declare @h uniqueidentifier
        , @messageTypeName sysname
        , @messageBody varbinary(max)
        , @xmlBody xml
        , @startTime datetime
        , @finishTime datetime
        , @execErrorNumber int
        , @execErrorMessage nvarchar(2048)
        , @xactState smallint
        , @token uniqueidentifier;

    begin transaction;
    begin try;
        receive top(1)
            @h = [conversation_handle]
            , @messageTypeName = [message_type_name]
            , @messageBody = [message_body]
            from [AsyncExecQueue];
        if (@h is not null)
        begin
            if (@messageTypeName = N'DEFAULT')
            begin
                -- The DEFAULT message type is a procedure invocation.
                --
                select @xmlBody = CAST(@messageBody as xml);

                save transaction usp_AsyncExec_procedure;
                select @startTime = GETUTCDATE();
                begin try
                    exec [usp_procedureInvokeHelper] @xmlBody;
                end try
                begin catch
                -- This catch block tries to deal with failures of the procedure execution
                -- If possible it rolls back to the savepoint created earlier, allowing
                -- the activated procedure to continue. If the executed procedure
                -- raises an error with severity 16 or higher, it will doom the transaction
                -- and thus rollback the RECEIVE. Such case will be a poison message,
                -- resulting in the queue disabling.
                --
                select @execErrorNumber = ERROR_NUMBER(),
                    @execErrorMessage = ERROR_MESSAGE(),
                    @xactState = XACT_STATE();
                if (@xactState = -1)
                begin
                    rollback;
                    raiserror(N'Unrecoverable error in procedure: %i: %s', 16, 10,
                        @execErrorNumber, @execErrorMessage);
                end
                else if (@xactState = 1)
                begin
                    rollback transaction usp_AsyncExec_procedure;
                end
                end catch

                select @finishTime = GETUTCDATE();
                select @token = [conversation_id]
                    from sys.conversation_endpoints
                    where [conversation_handle] = @h;
                if (@token is null)
                begin
                    raiserror(N'Internal consistency error: conversation not found', 16, 20);
                end
                update [AsyncExecResults] set
                    [start_time] = @starttime
                    , [finish_time] = @finishTime
                    , [error_number] = @execErrorNumber
                    , [error_message] = @execErrorMessage
                    where [token] = @token;
                if (0 = @@ROWCOUNT)
                begin
                    raiserror(N'Internal consistency error: token not found', 16, 30);
                end
                end conversation @h;
            end
            else if (@messageTypeName = N'http://schemas.microsoft.com/SQL/ServiceBroker/EndDialog')
            begin
                end conversation @h;
            end
            else if (@messageTypeName = N'http://schemas.microsoft.com/SQL/ServiceBroker/Error')
            begin
                declare @errorNumber int
                    , @errorMessage nvarchar(4000);
                select @xmlBody = CAST(@messageBody as xml);
                with xmlnamespaces (DEFAULT N'http://schemas.microsoft.com/SQL/ServiceBroker/Error')
                select @errorNumber = @xmlBody.value ('(/Error/Code)[1]', 'INT'),
                    @errorMessage = @xmlBody.value ('(/Error/Description)[1]', 'NVARCHAR(4000)');
                -- Update the request with the received error
                select @token = [conversation_id] 
                    from sys.conversation_endpoints 
                    where [conversation_handle] = @h;
                update [AsyncExecResults] set
                    [error_number] = @errorNumber
                    , [error_message] = @errorMessage
                    where [token] = @token;
                end conversation @h;
             end
           else
           begin
                raiserror(N'Received unexpected message type: %s', 16, 50, @messageTypeName);
           end
        end
        commit;
    end try
    begin catch
        declare @error int
         , @message nvarchar(2048);
        select @error = ERROR_NUMBER()
            , @message = ERROR_MESSAGE()
            , @xactState = XACT_STATE();
        if (@xactState <> 0)
        begin
         rollback;
        end;
        raiserror(N'Error: %i, %s', 1, 60,  @error, @message) with log;
    end catch
end
go

alter queue [AsyncExecQueue]
    with activation (
    procedure_name = [usp_AsyncExecActivated]
    , max_queue_readers = 1
    , execute as owner
    , status = on);
go

-- Helper function to create the XML element 
-- for a passed in parameter
create function [dbo].[fn_DescribeSqlVariant] (
 @p sql_variant
 , @n sysname)
returns xml
with schemabinding
as
begin
 return (
 select @n as [@Name]
  , sql_variant_property(@p, 'BaseType') as [@BaseType]
  , sql_variant_property(@p, 'Precision') as [@Precision]
  , sql_variant_property(@p, 'Scale') as [@Scale]
  , sql_variant_property(@p, 'MaxLength') as [@MaxLength]
  , @p
  for xml path('parameter'), type)
end
GO

-- Invocation wrapper. Accepts arbitrary
-- named parameetrs to be passed to the
-- background procedure
create procedure [usp_AsyncExecInvoke]
    @procedureName sysname
    , @p1 sql_variant = NULL, @n1 sysname = NULL
    , @p2 sql_variant = NULL, @n2 sysname = NULL
    , @p3 sql_variant = NULL, @n3 sysname = NULL
    , @p4 sql_variant = NULL, @n4 sysname = NULL
    , @p5 sql_variant = NULL, @n5 sysname = NULL
    , @token uniqueidentifier output
as
begin
    declare @h uniqueidentifier
     , @xmlBody xml
        , @trancount int;
    set nocount on;

 set @trancount = @@trancount;
    if @trancount = 0
        begin transaction
    else
        save transaction usp_AsyncExecInvoke;
    begin try
        begin dialog conversation @h
            from service [AsyncExecService]
            to service N'AsyncExecService', 'current database'
            with encryption = off;
        select @token = [conversation_id]
            from sys.conversation_endpoints
            where [conversation_handle] = @h;

        select @xmlBody = (
            select @procedureName as [name]
            , (select * from (
                select [dbo].[fn_DescribeSqlVariant] (@p1, @n1) AS [*] 
                    WHERE @p1 IS NOT NULL
                union all select [dbo].[fn_DescribeSqlVariant] (@p2, @n2) AS [*] 
                    WHERE @p2 IS NOT NULL
                union all select [dbo].[fn_DescribeSqlVariant] (@p3, @n3) AS [*] 
                    WHERE @p3 IS NOT NULL
                union all select [dbo].[fn_DescribeSqlVariant] (@p4, @n4) AS [*] 
                    WHERE @p4 IS NOT NULL
                union all select [dbo].[fn_DescribeSqlVariant] (@p5, @n5) AS [*] 
                    WHERE @p5 IS NOT NULL
                ) as p for xml path(''), type
            ) as [parameters]
            for xml path('procedure'), type);
        send on conversation @h (@xmlBody);
        insert into [AsyncExecResults]
            ([token], [submit_time])
            values
            (@token, getutcdate());
    if @trancount = 0
        commit;
    end try
    begin catch
        declare @error int
            , @message nvarchar(2048)
            , @xactState smallint;
        select @error = ERROR_NUMBER()
            , @message = ERROR_MESSAGE()
            , @xactState = XACT_STATE();
        if @xactState = -1
            rollback;
        if @xactState = 1 and @trancount = 0
            rollback
        if @xactState = 1 and @trancount > 0
            rollback transaction usp_my_procedure_name;

        raiserror(N'Error: %i, %s', 16, 1, @error, @message);
    end catch
end
go

-- Sample invocation example
-- The usp_withParam will insert 
-- all the received parameters into this table
-- 
create table [withParam] (
    id numeric(4,1) NULL
    , name varchar(150) NULL
    , date datetime  NULL
    , value int NULL
    , bytes varbinary(max) NULL);
go

create procedure usp_withParam
 @id numeric(4,1)
 , @name varchar(150)
 , @date datetime = NULL
 , @value int = 0
 , @bytes varbinary(max) = NULL
as
begin
    insert into [withParam] (
        id
        , name
        , date
        , value
        , bytes)
     select @id as [id]
      , @name as [name]
      , @date as [date]
      , @value as [value]
      , @bytes as [bytes]
end
go

declare @token uniqueidentifier;

exec usp_AsyncExecInvoke @procedureName = N'usp_withParam'
 , @p1 = 1.0, @n1 = N'@id'
 , @p2 = N'Foo', @n2='@name'
 , @p3 = 0xBAADF00D, @n3 = '@bytes'
 , @token = @token output;

waitfor delay '00:00:05';

select * from AsyncExecResults;
select * from withParam;

go

Asynchronous procedure execution

August 5th, 2009

Update: a version of this sample that accepts parameters is available in the post Passing Parameters to a Background Procedure

Recently an user on StackOverflow raised the question Execute a stored procedure from a windows form asynchronously and then disconnect?. This is a known problem, how to invoke a long running procedure on SQL Server without constraining the client to wait for the procedure execution to terminate. Most times I’ve seen this question raised in the context of web applications when waiting for a result means delaying the response to the client browser. On Web apps the time constraint is even more drastic, the developer often desires to launch the procedure and immediately return the page even when the execution lasts only few seconds. The application will retrieve the execution result later, usually via an Ajax call driven by the returned page script.

Frankly I was a bit surprised to see that the responses gravitated either around the SqlClient asynchronous methods (BeginExecute…) or around having a dedicated process with the sole pupose of maintaining the client connection alive for the duration of the long running procedure.

This problem is perfectly addressed by Service Broker Activation. Since I wanted to preserve the solution for further reference, I decided to put it in as a blog entry, with additional comments. For many of you Service Broker aficionados that read my blog regularly, this article is not innovative as is mostly a rehash of well known techniques I’ve been talking about on forums for many years now.

I’m going to use a table to store the result of the procedure execution. In this version I’ll keep things simple by not allowing for any parameters to be passed to the procedure, nor collecting any execution result set data. So the table will only contain the procedure start time, the execution finish time and any error that occurred during the procedure execution:

create table [AsyncExecResults] (
	[token] uniqueidentifier primary key
	, [submit_time] datetime not null
	, [start_time] datetime null
	, [finish_time] datetime null
	, [error_number]	int null
	, [error_message] nvarchar(2048) null);
go

Next we’re going to create the service and queue we need. I will use one single service for both roles (initiator and target) and I won’t create an explicit contract, relying instead on the predefined DEFAULT contract:

create queue [AsyncExecQueue];
go

create service [AsyncExecService] on queue [AsyncExecQueue] ([DEFAULT]);
GO

Next is the core of our asynchronous execution: the activated procedure. The procedure has to dequeue the message that specifies the user procedure, run the procedure and write the result in the results table. I will also deploy the error handling template I elaborated on my previous article Exception handling and nested transactions:

create procedure usp_AsyncExecActivated
as
begin
    set nocount on;
    declare @h uniqueidentifier
        , @messageTypeName sysname
        , @messageBody varbinary(max)
        , @xmlBody xml
        , @procedureName sysname
        , @startTime datetime
        , @finishTime datetime
        , @execErrorNumber int
        , @execErrorMessage nvarchar(2048)
        , @xactState smallint
        , @token uniqueidentifier;

    begin transaction;
    begin try;
        receive top(1) 
            @h = [conversation_handle]
            , @messageTypeName = [message_type_name]
            , @messageBody = [message_body]
            from [AsyncExecQueue];
        if (@h is not null)
        begin
            if (@messageTypeName = N'DEFAULT')
            begin
                -- The DEFAULT message type is a procedure invocation.
                -- Extract the name of the procedure from the message body.
                --
                select @xmlBody = CAST(@messageBody as xml);
                select @procedureName = @xmlBody.value(
                    '(//procedure/name)[1]'
                    , 'sysname');

                save transaction usp_AsyncExec_procedure;
                select @startTime = GETUTCDATE();
                begin try
                    exec @procedureName;
                end try
                begin catch
                -- This catch block tries to deal with failures of the procedure execution
                -- If possible it rolls back to the savepoint created earlier, allowing
                -- the activated procedure to continue. If the executed procedure 
                -- raises an error with severity 16 or higher, it will doom the transaction
                -- and thus rollback the RECEIVE. Such case will be a poison message,
                -- resulting in the queue disabling.
                --
                select @execErrorNumber = ERROR_NUMBER(),
                    @execErrorMessage = ERROR_MESSAGE(),
                    @xactState = XACT_STATE();
                if (@xactState = -1)
                begin
                    rollback;
                    raiserror(N'Unrecoverable error in procedure %s: %i: %s', 16, 10,
                        @procedureName, @execErrorNumber, @execErrorMessage);
                end
                else if (@xactState = 1)
                begin
                    rollback transaction usp_AsyncExec_procedure;
                end
                end catch

                select @finishTime = GETUTCDATE();
                select @token = [conversation_id] 
                    from sys.conversation_endpoints 
                    where [conversation_handle] = @h;
                if (@token is null)
                begin
                    raiserror(N'Internal consistency error: conversation not found', 16, 20);
                end
                update [AsyncExecResults] set
                    [start_time] = @starttime
                    , [finish_time] = @finishTime
                    , [error_number] = @execErrorNumber
                    , [error_message] = @execErrorMessage
                    where [token] = @token;
                if (0 = @@ROWCOUNT)
                begin
                    raiserror(N'Internal consistency error: token not found', 16, 30);
                end
                end conversation @h;
            end 
            else if (@messageTypeName = N'http://schemas.microsoft.com/SQL/ServiceBroker/EndDialog')
            begin
                end conversation @h;
            end
            else if (@messageTypeName = N'http://schemas.microsoft.com/SQL/ServiceBroker/Error')
            begin
                declare @errorNumber int
                    , @errorMessage nvarchar(4000);
                select @xmlBody = CAST(@messageBody as xml);
                with xmlnamespaces (DEFAULT N'http://schemas.microsoft.com/SQL/ServiceBroker/Error')
                select @errorNumber = @xmlBody.value ('(/Error/Code)[1]', 'INT'),
                    @errorMessage = @xmlBody.value ('(/Error/Description)[1]', 'NVARCHAR(4000)');
                -- Update the request with the received error
                select @token = [conversation_id] 
                    from sys.conversation_endpoints 
                    where [conversation_handle] = @h;
                update [AsyncExecResults] set
                    [error_number] = @errorNumber
                    , [error_message] = @errorMessage
                    where [token] = @token;
                end conversation @h;
             end
           else
           begin
                raiserror(N'Received unexpected message type: %s', 16, 50, @messageTypeName);
           end
        end
        commit;
    end try
    begin catch
        declare @error int
            , @message nvarchar(2048);
        select @error = ERROR_NUMBER()
            , @message = ERROR_MESSAGE()
            , @xactState = XACT_STATE();
        if (@xactState <> 0)
        begin
            rollback;
        end;
        raiserror(N'Error: %i, %s', 1, 60,  @error, @message) with log;
    end catch
end
go

To make the procedure activated we need to attach it to our service queue. This will ensure this procedure is run whenever a message arrives to our [AsyncExecService]:

alter queue [AsyncExecQueue]
    with activation (
    procedure_name = [usp_AsyncExecActivated]
    , max_queue_readers = 1
    , execute as owner
    , status = on);
go

And finaly the last piece of the puzzle: the procedure that submits the message to invoke the desired asyncronous executed procedure. This procedure resturns an output parameter ‘token’ than can be used to lookup the asynchronous execution result.

create procedure [usp_AsyncExecInvoke]
    @procedureName sysname
    , @token uniqueidentifier output
as
begin
    declare @h uniqueidentifier
	    , @xmlBody xml
        , @trancount int;
    set nocount on;

	set @trancount = @@trancount;
    if @trancount = 0
        begin transaction
    else
        save transaction usp_AsyncExecInvoke;
    begin try
        begin dialog conversation @h
            from service [AsyncExecService]
            to service N'AsyncExecService', 'current database'
            with encryption = off;
        select @token = [conversation_id]
            from sys.conversation_endpoints
            where [conversation_handle] = @h;
        select @xmlBody = (
            select @procedureName as [name]
            for xml path('procedure'), type);
        send on conversation @h (@xmlBody);
        insert into [AsyncExecResults]
            ([token], [submit_time])
            values
            (@token, getutcdate());
    if @trancount = 0
        commit;
    end try
    begin catch
        declare @error int
            , @message nvarchar(2048)
            , @xactState smallint;
        select @error = ERROR_NUMBER()
            , @message = ERROR_MESSAGE()
            , @xactState = XACT_STATE();
        if @xactState = -1
            rollback;
        if @xactState = 1 and @trancount = 0
            rollback
        if @xactState = 1 and @trancount > 0
            rollback transaction usp_my_procedure_name;

        raiserror(N'Error: %i, %s', 16, 1, @error, @message);
    end catch
end
go

To test our asynchronous execution infrastructure we create a test procedure and invoke it asynchronously. I will create two test procedures, one that simply waits for 5 seconds to simulate a ‘long’ running procedure and one that produces intentionally a primary key violation, to simulate a fault in the asynchronously executed procedure:

create procedure [usp_MyLongRunningProcedure]
as
begin
    waitfor delay '00:00:05';
end
go

create procedure [usp_MyFaultyProcedure]
as
begin
    set nocount on;
    declare @t table (id int primary key);
    insert into @t (id) values (1);
    insert into @t (id) values (1);
end
go


declare @token uniqueidentifier;
exec usp_AsyncExecInvoke N'usp_MyLongRunningProcedure', @token output;
select * from [AsyncExecResults] where [token] = @token;
go

declare @token uniqueidentifier;
exec usp_AsyncExecInvoke N'usp_MyFaultyProcedure', @token output;
select * from [AsyncExecResults] where [token] = @token;
go

waitfor delay '00:00:10';
select * from [AsyncExecResults];
go

Activation Context

If you check the start time of the second asynchronosuly executed procedure you will notice that it started right after the first one finished. This is because we declare a max_queue_readers value of 1 when we set up activation on the queue. This restricts that at most one activated procedure to run at any time, effectively serializing all the asynchronously executed procedures. Whether this is desired or not depends a lot on the actual usage scenario. The limit can be increased as necessary.

If you start playing around with this method of invoking procedures asynchronously you will notice that sometimes the asynchrnously executed procedure is misteriously denied access to other databases or to server scoped objects. When the same procedure is run manually from a query window in SSMS, it executes fine. This is caused by the EXECUTE AS context under which activation occurs. the details are explained in MSDN’s Extending Database Impersonation by Using EXECUTE AS and myself I had covered this subject repeatedly in this blog. The best solution is to simply turn the trustworthy bit on on the database where the activated procedure runs. When this is not desired, or not allowed by your hosting environment, the solution is to code sign the activated procedure: Signing an activated procedure.

Using Service Broker Activation to invoke procedures asynchronously may look daunting at beginning. It sure is significantly more complex than just calling BeginExecuteNonQuery. But what needs to be understood is that this is a reliable way to invoke the procedure. The client is free to disconnect as soon as it commited the call to usp_AsyncExecInvoke. The procedure invoked will run, even if the server is stopped and restarted, even if a mirroring or clustering failover occurs. The server may even crash and be completely rebuilt. As soon as the database is back online, that queue will activate and invoke the asynchronous execution. Such level of reliability is difficult, if not impossible, to guarantee by using a client process.

Detect network connectivity changes in .Net applications

July 21st, 2009

Update

Andrei Ignat pointed out there is a class in the .Net Framework that deals exactly with this issue: NetworkChange Class. That is definitely a better approach than leveraging the native API through p-Invoke. The MSDN sample also shows how to check for network availability on each interface, when notified of a change.

Original post:

A common problem for applications that use the network stack is to detect when they are connected to the network. It would be ideal if the application was notified when you plugged in your network cable into your laptop, or when it came in range of a WiFi station and has joined a wireless network.

The Windows API has a function for registering for exactly such notifications: NotifyAddrChange. An application can register a wait handle with this function and the handle will be signaled by the operating system every time a change occurs in the IP address to interface mapping table. Technically is true that ‘IP address table change’ does mean strictly that ‘network connectivity has changed’, but in reality the overwhelming majority of IP address table changes can be associated with the network connectivity being gained or lost.

To leverage this API from managed code we have to use p-Invoke. I use a helper class that wraps the registration for notification and handling the event notification in a simple to use package:

    /// <summary>
    /// Raises an event each time a change occurs 
    /// on the system IP address table.
    /// </summary>
    public class NotifyAddressChangesIDisposable
    {
        #region State fields
        /// <summary>
        /// The OVERLAPPED structure passed to NotifyAddrChange
        /// </summary>
        private NativeOverlapped _overlapped;

        /// <summary>
        /// File handle created  by NotifyAddrChange
        /// </summary>
        private IntPtr _handle;

        /// <summary>
        /// The event signaled by NotifyAddrChange
        /// </summary>
        private AutoResetEvent _eventChanged;

        /// <summary>
        /// The thread pool notification registration
        /// </summary>
        private RegisteredWaitHandle _registered;

        /// <summary>
        /// Submitted state tracking
        /// </summary>
        private volatile int _submitted;

        #endregion

        #region Win32 API declarations

        [DllImport("Iphlpapi.dll", SetLastError = true)]
        private static extern UInt32 NotifyAddrChange(
            ref IntPtr Handle, 
            ref NativeOverlapped overlapped);
        [DllImport("Iphlpapi.dll", SetLastError = true)]
        private static extern bool CancelIPChangeNotify(
            ref NativeOverlapped overlapped);
        #endregion

        #region Public usable methods and events

        /// <summary>
        /// The event raised when a change in the system
        /// IP address table occurs. 
        /// </summary>
        public event EventHandler AddressChanged;


        /// <summary>
        /// CTOR. Initializes the notifications event and submits it
        /// to the ThreadPool. Caller need to call Submit() to enable
        /// system IP address notifications to raise the AddressChange event.
        /// </summary>
        public NotifyAddressChanges()
        {
            _submitted = 0;
            _eventChanged = new AutoResetEvent(false);
            _overlapped.EventHandle = _eventChanged.SafeWaitHandle.DangerousGetHandle();

            _registered = ThreadPool.RegisterWaitForSingleObject(
                _eventChanged,
                new WaitOrTimerCallback(AddressChangedNotification),
                null,
                Timeout.Infinite,
                false);
        }

        /// <summary>
        /// Enables the system IP address notifications to raise the AddressChange event.
        /// </summary>
        public void Submit()
        {
            if (0 == Interlocked.CompareExchange(
                ref _submitted, 1, 0))
            {
                NotifyAddrChange(ref _handle, ref _overlapped);
            }
        }

        #endregion

        #region Callback and event raising

        /// <summary>
        /// IP address change notification callback.
        /// </summary>
        private void AddressChangedNotification(object state, bool timedOut)
        {
            if (1 == Interlocked.CompareExchange(
                ref _submitted, 0, 1))
            {
                EventHandler handler = AddressChanged;
                if (null != handler)
                {
                    try
                    {
                        handler(thisEventArgs.Empty);
                    }
                    catch (Exception)
                    {
                        // silent ignore
                    }
                }
                Submit();
            }
        }
        #endregion

        #region Dispose Logic
        private void Dispose(bool userInvoked)
        {
            if (null != _registered)
            {
                _registered.Unregister(_eventChanged);
            }
            if (1 == Interlocked.CompareExchange(
                ref _submitted, 0, 1))
            {
                CancelIPChangeNotify(ref _overlapped);
            }
        }

        ~NotifyAddressChanges()
        {
            Dispose(false);
        }


        public void Dispose()
        {
            Dispose(true);
            GC.SuppressFinalize(this);
        }

        #endregion
    }

To use this class, simply create an instance, add an event handler to the AddressChanged event and then call the Submit() method to start getting notifications. The AddressChanged event will be raised each time the system changes an IP address on any of its network interfaces. For example consider a simple form named Form1 to which I added a TextBox named textbox1 (in other words I created a new Windows forms application and I dropped a multiline text box on its surface, without bothering to rename either). Here is how I can hook up the IP address change event to display ‘changed’ on a new line in the text box:

    public partial class Form1 : Form
    {
        private NotifyAddressChanges _notification;

        public Form1()
        {
            InitializeComponent();

            _notification = new NotifyAddressChanges();
            _notification.AddressChanged += 
                new EventHandler(_notification_AddressChanged);
            _notification.Submit();
        }

        void _notification_AddressChanged(object sender, EventArgs e)
        {
            Invoke(new MethodInvoker(delegate ()
            {
                textBox1.AppendText("changed\r\n");
            }));
        }
    }

I can now start my test application and play with the network cable. As I unplug and plug back the cable, the system looses and re-gains the IP address from my base station DHCP server and it signals the NotifyAddrChange registered events. My form gets notified and it adds a “changed” line to the text box. I can also release and renew my DHCP license from the command line running ipconfig /release and ipconfig /renew.

Note that this notification occurs with any change, both on IP4 and IPv6 interfaces. You will be notified if a VPN is connected or disconnected, if a dial-up connection is established and so on and so forth. Being notified does not mean that network is available, nor that is unavailable. It simply means a change occurred. While I could add support to the AddressChanged event to indicate what change, that would needlessly complicate this simple class, because the number of possible changes is actually much bigger than one would expect (think again at all those VPN, dial-up, point-to-point and other connections). My goal with this class was to eliminate the need to periodically wake the application and check if the network is available every minute, based on a timer. I can simply wait for the AddressChanged notification and then I can try to see if I can establish a connection.

Exception handling and nested transactions

June 11th, 2009

I wanted to use a template for writing procedures that behave as intuitively as possible in regard to nested transactions. My goals were:

  • The procedure template should wrap all the work done in the procedure in a transaction.
  • The procedures should be able to call each other and the calee should nest its transaction inside the outer caller transaction.
  • The procedure should only rollback its own work in case of exception, if possible.
  • The caller should be able to resume and continue even if the calee rolled back its work.

My solution is to use a either a transactions or a savepoint, depending on the value of @@TRANCOUNT at procedure start. The procedures start a new transaction if no transaction is pending. Otherwise they simply create a savepoint. On exit the procedure commits the transaction they started (if they started one), otherwise they simply exit. On exception, if the transaction is not doomed, the procedure either rolls back (if it started the transaction), or rolls back to the savepoint it created (if calee already provided a transaction).

Because of the use of savepoints this template does not work in all situations, since there are cases like distributed transactions, that cannot be mixed with savepoints. But for the average run of the mill procedures, this template has saved me very well.

create procedure [usp_my_procedure_name]
as
begin
	set nocount on;
	declare @trancount int;
	set @trancount = @@trancount;
	begin try
		if @trancount = 0
			begin transaction
		else
			save transaction usp_my_procedure_name;

		-- Do the actual work here
	
lbexit:
		if @trancount = 0	
			commit;
	end try
	begin catch
		declare @error int, @message varchar(4000), @xstate int;
		select @error = ERROR_NUMBER(), @message = ERROR_MESSAGE(), @xstate = XACT_STATE();
		if @xstate = -1
			rollback;
		if @xstate = 1 and @trancount = 0
			rollback
		if @xstate = 1 and @trancount > 0
			rollback transaction usp_my_procedure_name;

		raiserror ('usp_my_procedure_name: %d: %s', 16, 1, @error, @message) ;
	end catch	
end

Using XSLT to generate Performance Counters code

April 11th, 2009

Whenever I’m faced with a project in which I have to create a lot of tedious and repeating code I turn to the power of XML and XSLT. Rather than copy/paste the same code over and over again, just to end up with a refactoring and maintenance nightmare, I create an XML definition file and an XSLT transformation. I am then free to add new elements to the XML definition or to change the way the final code is generated from the XSLT transformation. This can be fully integrated with Visual Studio so that the code generation happens at project build time and the environment shows the generated code as a dependency of the XML definition file.

A few examples of how I’m using this code generation via XSLT are:

Data Access Layer
I know this will raise quite a few eyebrows, but I always write my own data access layer from scratch and is generated via XSLT.
Performance Counters
I create all my performance counters objects via XSLT generation, automating the process of defining/installing them and the access to emit and consume the counter values.
Wire Frames
In any project that has networking communication I access the wire format from classes generated via XSLT that take care of serialization and validation.

For example I’ll show how to create a class library that can be added to your project to expose Performance Counters from your application.

MSXSL.EXE

To start you’ll need to download the Command Line Transformation Utility (msxsl.exe). This is a command line tool that takes an XML and an XSLT file as input and produces the result transformation. As a side note I actually use the same technique on non-Windows platforms, but there of course I use the xsltproc utility.

msxsl.exe has to be placed in a convenient location from where it is accessible by Visual Studio. Download the file and place it in Visual Studio’s Common\IDE folder because the build environment has a macro for it: $DevEnvDir.

The Project

Create a new project in Visual Studio. Choose a C# Class Library type project and save it as “PerformanceCounters”. After the project is created, add to it two new files, from menu Project/Add New Item or press Ctrl+Shift+A. For the first file choose an XML file type and name it Counters.xml. For the second one choose an XSLT file type and name it Counters.xslt.

Next add the step that actually performs XSLT transformation to the project build process. Go to the project properties, either from menu Project/PerformcanceCounters properties... or right-click the project in the Solutions Explorer and select Properties from the context menu. Choose the Build Events pane in the right side navigation tab to get to the pre-build and post-build project events. Click the Edit pre-build... button and add the following command:

"$(DevEnvDir)msxsl.exe" "$(ProjectDir)Counters.xml" "$(ProjectDir)Counters.xslt"
-o "$(ProjectDir)CountersGenerated.cs"

This command will run the command line transformation tool (msxsl.exe) and generate the CountersGenerated.cs file each time the project is built, prior to the project being actually built.

Save and close the project properties. The last thing we need now is to add the generated CountersGenerated.cs file to our project. However, we do not want this to be treated as an ordinary source file, we want the IDE to recognize that is a generated file dependent of the Counters.xml file. The way I prefer to do this is to actually manually add it directly into the project definition file. Open the PerformanceCounters.csproj file in your editor, locate the <ItemGroup> containing the Program.cs and insert this snippet:

	  <Compile Include="CountersGenerated.cs">
		  <AutoGen>True</AutoGen>
		  <DependentUpon>Counters.xml</DependentUpon>
	  </Compile>

Save the manually edited .csproj file and open the project normally from Visual Studio. The CountersGenerated.cs file now shows up in the Project Explorer as a generated file depending on Counters.xml:

The warning triangle is shown because the file does not exist on the disk, but don't worry about it as it will be generated shortly.

Defining the Performance Counters

We can go ahead and define the Performance Counters. The counters covering related functionality are grouped together and the groups can be single instance or multiple instance types. I also like to create a manager class that controls the management of the counters taking care of things like deployment during application setup and loading the counters when application starts. So my choice is for a XML like manager/group/counter. Some attributes are needed for the code generation XSLT transformation to know what class names and namespaces to use and some attributes are Performance Counters specific, like the counter types and bases for average counters. For start we'll create a simple counter and a group:

<?xml version="1.0" encoding="utf-8" ?>
<manager xmlns="http://rusanu.com/Schemas/PerformanceCounters/v1.0"
		clrname="PerformanceCountersManager" namespace="PerformanceCounters">
	<group name="My Counters" clrname="MyCounters" type="SingleInstance">
		<description>Demo Counters for XSLT code generation project</description>
		<counter name="My Count" type="NumberOfItems32" 
                                      clrname="MyCount" hasbase="No">Demo Counter</counter>
	</group>
</manager>

This definition will create:

  • A manager class named PerformanceCounters.PerformanceCountersManager.
  • A performance counters category named My Group.
  • A C# class named MyGroup for the above performance counters category.
  • A performance counter named My Count.
  • A C# method named IncrementMyCount to increment the counter above.

Now we need to set in place the actual XSLT transformation that will generate code as we desire.

The XSTL transformation

We want our XSLT transformation to generate C# code, so we're going to start by a stylesheet that will generate the skeleton of a C# file: the using directives and some comment to warn the user that the file is a generated file and should not be manually modified. Also our XSLT transformation is directed to generate an text output as opposed to the default XML output:

<?xml version="1.0" encoding="UTF-8" ?>
<xsl:stylesheet version="1.0" 
	xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:pc="http://rusanu.com/Schemas/PerformanceCounters/v1.0">
	<xsl:output method="text" encoding="utf-8"/>
	<xsl:template match="/">
/* This file was automatically generated during project build.
Do not manually modify this file as updates will be lost.*/
using System;
using System.Diagnostics;

		<xsl:apply-templates/>
</xsl:template>
</xsl:stylesheet>

This skeleton XSLT can be used for any C# code generation, provided of course you modify the appropriate using directives.

Next we should add a template to generate our C# code. How one writes XSLT transformation templates is largely a matter of style and once you get the hang of it it quickly becomes a second nature. XSLT is a fully fledged programming language of its own and you can start thinking at templates in terms of procedures and functions. At least I know I do :) . Our XSLT template should do the following:

  • Identify the manager defined in the XML file and generate a C# class for it.
  • Generate a class for each performance counter category.
  • Generate a method for each individual performance counter value to read and increment the counter.
  • Hook up the manager class to provide methods for deploying and deleting the performance counter categories.
  • Hook up the manager class with an instance of each performance counter category class.
  • Hook up the manager to provide loading of the default instance of the performance counters category classes.

Note that the above refer mostly to the SingleInstance performance counter category types. Multiple Instance category are a bitmore complex and I will cover them in my next post.

<xsl:template match="/pc:manager">
namespace <xsl:value-of select="@namespace"/>
{
public partial class <xsl:value-of select="@clrname"/>
{
	<xsl:for-each select="./pc:group[@type='SingleInstance']">
	private static <xsl:value-of select="@clrname"/> __<xsl:value-of select="@clrname"/> = 
		new <xsl:value-of select="@clrname"/>();
	public static <xsl:value-of select="@clrname"/> Default<xsl:value-of select="@clrname"/> {
		get {return __<xsl:value-of select="@clrname"/>; } 
	}
	</xsl:for-each>

	public static void LoadSingleInstance(bool fReadOnly)
	{
		<xsl:for-each select="./pc:group[@type='SingleInstance']">
		__<xsl:value-of select="@clrname"/>.Load(fReadOnly);
		</xsl:for-each>
	}		
	
	public static void DeleteSingleInstance()
	{
		<xsl:for-each select="./pc:group[@type='SingleInstance']">
		<xsl:value-of select="@clrname"/>.Delete();
		</xsl:for-each>
	}
	
	public static void CreateSingleInstance()
	{
		<xsl:for-each select="./pc:group[@type='SingleInstance']">
		<xsl:value-of select="@clrname"/>.Create();
		</xsl:for-each>
	}	
}

<xsl:for-each select="./pc:group">
public partial class <xsl:value-of select="@clrname"/>
{
	private const string __description = 
		@"<xsl:value-of select="./pc:description/text()"/>";
	private const string __category = 
		@"<xsl:value-of select="@name"/>";
	
	<xsl:for-each select="./pc:counter">
 	private PerformanceCounter _<xsl:value-of select="@clrname"/>;
	<xsl:if test="@hasbase='Yes'">
	private PerformanceCounter _<xsl:value-of select="@clrname"/>Base;
	</xsl:if>

	public void Increment<xsl:value-of select="@clrname"/>(long value)
	{
		_<xsl:value-of select="@clrname"/>.IncrementBy(value);
		<xsl:if test="@hasbase='Yes'">
		_<xsl:value-of select="@clrname"/>Base.IncrementBy(1);
		</xsl:if>
	}
	
	public float <xsl:value-of select="@clrname"/> {
		get { return _<xsl:value-of select="@clrname"/>.NextValue(); } 
	}
	
	</xsl:for-each>

	<xsl:if test="@type='MultipleInstance'">
   internal void Load(bool fReadOnly, string instanceName) {
    <xsl:for-each select="./pc:counter">
      _<xsl:value-of select="@clrname"/> = new PerformanceCounter(
      __category,
      @"<xsl:value-of select="@name"/>",
      instanceName,
      fReadOnly);
      <xsl:if test="@hasbase='Yes'">
        _<xsl:value-of select="@clrname"/>Base = new PerformanceCounter(
        __category,
        @"<xsl:value-of select="@name"/>Base",
        instanceName,
        fReadOnly);
      </xsl:if>
    </xsl:for-each>
    }
	</xsl:if>

	<xsl:if test="@type='SingleInstance'">
    internal void Load (bool fReadOnly)
    {
    <xsl:for-each select="./pc:counter">
      _<xsl:value-of select="@clrname"/> = new PerformanceCounter(
      __category,
      @"<xsl:value-of select="@name"/>",
      fReadOnly);
      <xsl:if test="@hasbase='Yes'">
        _<xsl:value-of select="@clrname"/>Base = new PerformanceCounter(
        __category,
        @"<xsl:value-of select="@name"/>Base",
        fReadOnly);
      </xsl:if>    </xsl:for-each>
    }	
	</xsl:if>

   internal static bool Exists
    {
		get {return PerformanceCounterCategory.Exists(__category);}
    }

    internal static void Delete ()
    {
		PerformanceCounterCategory.Delete(__category);
    }

    public static void Create ()
    {
		CounterCreationDataCollection ccdc = new CounterCreationDataCollection();
		<xsl:for-each select="./pc:counter">
		ccdc.Add(new CounterCreationData (
			@"<xsl:value-of select="@name"/>",
			@"<xsl:value-of select="."/>",
			PerformanceCounterType.<xsl:value-of select="@type"/>));
		<xsl:if test="@hasbase='Yes'">
		ccdc.Add(new CounterCreationData (
			@"<xsl:value-of select="@name"/>Base",
			@"Base for <xsl:value-of select="."/>",
			PerformanceCounterType.AverageBase));
		</xsl:if>
		</xsl:for-each>
		PerformanceCounterCategory.Create(
			__category,
			__description,
			PerformanceCounterCategoryType.<xsl:value-of select="@type"/>,
			ccdc);
    }
}
 
</xsl:for-each>
}
</xsl:template>

The generate code

If we go ahead and build the project. It will run our XSLT transformation over the definition XML file and the result will be saved in the CountersGenerated.cs file:

/* This file was automatically generated during project build.
Do not manually modify this file as updates will be lost.*/
using System;
using System.Diagnostics;


namespace PerformanceCounters
{
public partial class PerformanceCountersManager
{

	private static MyCounters __MyCounters = 
		new MyCounters();
	public static MyCounters DefaultMyCounters {
		get {return __MyCounters; } 
	}


	public static void LoadSingleInstance(bool fReadOnly)
	{

		__MyCounters.Load(fReadOnly);

	}		

	public static void DeleteSingleInstance()
	{
		MyCounters.Delete();

	}

	public static void CreateSingleInstance()
	{
		MyCounters.Create();

	}	
}


public partial class MyCounters
{
	private const string __description = 
		@"Demo Counters for XSLT code generation project";
	private const string __category = 
		@"My Counters";


 	private PerformanceCounter _MyCount;


	public void IncrementMyCount(long value)
	{
		_MyCount.IncrementBy(value);

	}

	public float MyCount {
		get { return _MyCount.NextValue(); } 
	}


    internal void Load (bool fReadOnly)
    {
    
      _MyCount = new PerformanceCounter(
      __category,
      @"My Count",
      fReadOnly);
      
    }	


   internal static bool Exists
    {
		get {return PerformanceCounterCategory.Exists(__category);}
    }

    internal static void Delete ()
    {
		PerformanceCounterCategory.Delete(__category);
    }

    public static void Create ()
    {
		CounterCreationDataCollection ccdc = 
            new CounterCreationDataCollection();

		ccdc.Add(new CounterCreationData (
			@"My Count",
			@"Demo Counter",
			PerformanceCounterType.NumberOfItems32));

		PerformanceCounterCategory.Create(
			__category,
			__description,
			PerformanceCounterCategoryType.SingleInstance,
			ccdc);
    }
}
 

}

So what did we achieve? We could had written this code manually in about 5 minutes. But the big advantage comes as we add more counters to our XML definition file. We could expand it to have 10 categories with 15 counters each and it would still generate all the needed counters. And refactoring the code is a breeze. Don't like how the counters are created? Just change the XSLT stylesheet and rebuild the project, all the performance counters creation code wil match your new preference.

In a future post I will cover how to use the performance counters in your projects, how to work with multiple instance counters and how to deal with the pesky AvgTimer32 counter type.

Chained Updates

April 9th, 2008

One of the interesting features of the OUTPUT clauses introduced in SQL Server 2005 is that one can actualy chain DML statements into one complex statement that operates updates on several tables at once. Say we have a table with customer data and a process that has to bill each customer periodically. The ‘billing’ process consist of an update on the table (say extend the subscription date), but the billing has to be processed separately. Consider an example where the processing involves a Web call to a bank portal to charge a credit card and, like all HTTP calls, it has the potential to fail. So we have two tables like this:

-- The customer data. Each customer has to be billed
-- when the expiration_date is passed
--
create table [customer_data] (
      [customer_id] int identity (1,1) not null,
      [name] nchar(256) not null,
      [account_no] nchar(80) not null,
      [expiration_date] datetime not null,
      [status] nchar(25) not null,
      constraint [customer_data_pk] primary key ([customer_id]));
go

create index [customer_data_expiration_date] on [customer_data] ([expiration_date]);
go

-- The processing queue
--
create table [bills] (
      [bill_id] int identity(1,1) not null,
      [customer_id] int,
      [enqueue_date] datetime not null,
      [next_retry] datetime not null,
      constraint [bills_pk] primary key nonclustered ([bill_id]),
      constraint [bills_cdx] unique clustered ([next_retry], [bill_id]));
go

The ‘billing’ operation has to update the status and expiration_date on customer_data and insert the pending billing requests to the bills table. Using the OUTPUT clause of the UPDATE we can actually do both operations in one single step:

declare @now datetime;
set @now = GETDATE();
with cte_billable_customers as (
      select top(100)
                  [customer_id],
                  [status],
                  [expiration_date]
            from [customer_data] with (UPDLOCK)
            where [expiration_date] < @now
            order by [expiration_date])
update cte_billable_customers
      set [status] = 'Pending',
            [expiration_date] = DATEADD(month, 1, @now)
      output INSERTED.[customer_id],
            @now as [enqueue_date],
            @now as [next_retry]
      into [bills] (
            [customer_id],
            [enqueue_date],
            [next_retry]);

The TOP(100) has the role of restricting the processing into small batches to prevent a runaway update that might block the entire table.

So what advantages does such a construct offer? Personaly I like the fact that is one single statement instead of a pair of statements that uses an intermediary table variable or temporary table. The gives better isolation semantics for the locks acquired, it guarantees an atomic operation even if not called in a transaction context and it generates one single execution tree for the entire operation (both the update and the insert). It also has a certain coolnes factor.

As it is right now this is not much more other than a trivia alternate on your T-SQL bag'o'tricks. This sort of construct would potentially be much more powerful if further updates could be chained. That is if the INTO clause could have its own OUTPUT to be chained into an update, a delete or another insert. Potentialy building up chains of tens of table operations in one single statement. The would allow for entire procedures that today are declared in procedural fashion (statement after statement in T-SQL batches) to be writen in a SQL declarative fashion, leaving way for the optimizer to find shortcuts impossible to find today with separate operations. Unfortunately the language does not permit such constructs.

And now a piece of trivia: can you name the difference between an UPDLOCK and an XLOCK? I recently realized that I actualy forgot the difference. the answer is that UPDLOCK is asymmetric, it is compatible with S locks but S lock is not compatible with UPDLOCK. This kind of asymmetry helps in resolving the most common form of deadlock: a reader reads a row and then updates the same row (the so called 'read-write deadlock'). Because of its asymmetry the 'read' part can read the value while other readers are holding an S lock, but it prevents further readers from getting that row. When the update is applied, the UPDLOCK is upgraded to an XLOCK.

And this finally explains why my example contains an UPDLOCK hint in the SELECT statement: to prevent deadlocks between two concurent statements trying to process the customer_data table.

Resending messages

December 3rd, 2007

This article is a continuation of my two articles on using Service Broker as a pure data-push one-way communication channel: Reusing Conversations and Recycling Conversations. I originally did not plan for this third part, but I was more than once asked the same question: if a conversation is in error what happens to the messages that were sent but not yet delivered?

The short answer is that messages that are still pending are in the sender’s database sys.transmission_queue system table so when an Error message is received the sender can scan this table and resend each message that is still pending. An example of such procedure is not difficult to code:

-- Resends all pending messages in sys.transmission_queue
-- belonging to an old colversation on a new conversation
--
CREATE PROCEDURE usp_ResendPending (
      @oldConversation UNIQUEIDENTIFIER,
      @newConversation UNIQUEIDENTIFIER)
AS
BEGIN
      DECLARE @mt SYSNAME;
      DECLARE @body VARBINARY;
 
      -- Must declare a cursor to iterate over
      -- all the pending messages
      -- It is important to keep the message order
      -- and to keep the original message type
      DECLARE cursorPending CURSOR
            LOCAL
            FORWARD_ONLY
            READ_ONLY
            FOR SELECT [message_type_name],
                        [message_body]
                  FROM sys.[transmission_queue]
                  WHERE [conversation_handle] = @oldConversation
                  ORDER BY message_sequence_number;
      OPEN cursorPending;
     
      FETCH NEXT FROM cursorPending
            INTO @mt, @body;
      WHILE (@@FETCH_STATUS = 0)
      BEGIN
            -- Resend the message on the new conversation
            IF (@body IS NOT NULL)
            BEGIN
                  -- If the @body is not null it must be sent explicitly
                  SEND ON CONVERSATION (@newConversation)
                        MESSAGE TYPE @mt
                        (@body);
            END
            ELSE
            BEGIN
                  -- Messages with no body must *not* specify the body,
                  -- cannot send a NULL value argument
                  --
                  SEND ON CONVERSATION (@newConversation)
                        MESSAGE TYPE @mt;
            END
            FETCH NEXT FROM cursorPending
                  INTO @mt, @body;
      END
     
      CLOSE cursorPending;
      DEALLOCATE cursorPending;
END
GO

Now the interesting question here is not how to do this but when to do this. My previous article had this code that was dealing with error messages:

ELSE IF @messageTypeName = N'http://schemas.microsoft.com/SQL/ServiceBroker/Error'
BEGIN
      -- Insert your error handling here. Could send a notification or
      -- store the error in a table for further inspection
      -- We’re gonna log the error into the ERRORLOG and Eventvwr.exe
      --
      DECLARE @error INT;
      DECLARE @description NVARCHAR(4000);
      WITH XMLNAMESPACES ('http://schemas.microsoft.com/SQL/ServiceBroker/Error' AS ssb)
      SELECT
            @error = CAST(@messageBody AS XML).value(
					'(//ssb:Error/ssb:Code)[1]', 'INT'),
            @description = CAST(@messageBody AS XML).value(
					'(//ssb:Error/ssb:Description)[1]', 'NVARCHAR(4000)');
      RAISERROR(N'Received error Code:%i Description:"%s"', 
			16, 1, @error, @description) WITH LOG;
     
      -- Insert call to usp_ResendPending here
           
      -- END CONVERSATION has to be issued AFTER the call to usp_ResendPending
      END CONVERSATION @handle;
END

I’ve already highlighted the place where the call to the resending procedure should be inserted. Important note: the place where the END CONVERSATION is issued in code has moved from my earlier post because once a conversation is ended all pending messages in sys.transmission_queue are deleted. So now your sender is going to resend any pending message in case it receives an error. But what sort of errors may the sender receive? Now that is no longer a trivial problem.

Service Broker conversation errors

The message_body of a conversation error message contains the error code and error description. There are two types of errors one conversation can receive:

·         application errors that were sent by the other endpoint of the conversation using the END CONVERSATION … WITH ERROR … verb. The error code for these errors is a positive number.

·         system errors that were sent by the Service Broker infrastructure when an error condition has occurred. The error code for these errors is a negative number and its value is negated value of a SQL Server error code from sys.messages.

I cannot comment anything about the application error values because, of course, they are application specific and is probably you that defined them in the first place. But I can give details on the system error codes what they mean:

-8494     You do not have permission to access the service ‘%.*ls’.
This system error is sent to a conversation that is trying to reach a service that is denying access to the sender.

-8489     The dialog has exceeded the specified LIFETIME.
This system error is sent to a conversation when the lifetime specified during BEGIN DIALOG has expired. Note that all conversations have a lifetime, if you omitted it then the lifetime is the 231 seconds (that is roughly 68 years)

-8462     The remote conversation endpoint is either in a state where no more messages can be exchanged, or it has been dropped.
This system error is sent on a conversation that continued to send messages to a peer that has already closed it’s endpoint.

-8470     Remote service has been dropped.
This  system error is sent on each active conversation that was initiated from or targeted a service that was DROPed from the database

-8469     Remote service has been altered.
This  system error is sent on each active conversation that was initiated from or targeted a service that was ALTERed

-8487     The remote service contract has been dropped
This system error is sent on each active conversation that was initiated using the contract that is being DROPed from the database

-8490     Cannot find the remote service ‘%.*ls’ because it does not exist.
This system error is sent to a conversation that is trying to reach a service in a specific broker instance and the service does not exist. This can happen if the optional broker instance is specified during BEGIN DIALOG

-8425     The service contract ‘%.*ls’ is not found.
This system error is sent to a conversation that is using a contract not installed on the peer database

-8408     Target service ‘%.*ls’ does not support contract ‘%.*ls’.
This system error is sent to a conversation that is using a contract not bound to the target service

-8428     The message type “%.*ls” is not found.
This system error is sent to a conversation that is using a message type not found in the target database

-8498     The remote service has sent a message of type ‘%.*ls’ that is not part of the local contract.
This system error is sent to a conversation that is using a message not bound to the contract used

-8430     The message body failed the configured validation.
This system error is sent to a conversation that is using a message type validation (none, empty, well_formed_xml) that does not agree with the peer validation for the same message type.

-8457     The message received was sent by the initiator of the conversation, but the message type ‘%.*ls’ is marked SENT BY TARGET in the contract.
This system error is sent to a conversation that has sent a message from initiator but the message is marked as SENT BY TARGET in the contract

-8437     The message received was sent by a Target service, but the message type ‘%.*ls’ is marked SENT BY INITIATOR in the contract.
This system error is sent to a conversation that has sent a message from target but the message is marked as SENT BY INITIATOR in the contract

-8495     The conversation has already been acknowledged by another instance of this service.
This system error is sent to a conversation that is sending a reply to an initiator but the initiator is rejecting this peer because has already established conversation with another peer. This scenario can happen only in certain conditions when the load-balancing routes of Service Broker are used.

-9616     A message of type ‘%.*ls’ was received and failed XML validation.  %.*ls This occurred in the message with Conversation ID ‘%.*ls’, Initiator: %d, and Message sequence number: %I64d.
This system error is sent to a conversation that has sent a message type marked as conforming to a certain XML schema but the payload has failed to pass the XML validation for the said schema

-9719     The database for this conversation endpoint is attached or restored.
This system error is sent to any active conversation when the ALTER DATABASE … SET ERROR_BROKER_CONVERSATIONS is issued. The text of the error message comes from the fact that this ALTER statement is usually issued in conjunction with restore or attach operations.

-28052    Cannot decrypt session key while regenerating master key with FORCE option.
This system error is sent to a conversation when the database master key was regenerated with FORCE and the conversation sessions keys (encrypted with the database master key) are lost

Although the list of system errors is quite large (and I actually left out three more errors that I really think too much of a corner case to worth mention) we can distinguish three cases:

1.      Errors that we can recover from immediately and can continue sending messages, including resend any pending messages. These are 8489 LIFETIME expired, 8462 Peer has closed, 9719 attach/restore and 28052 Master key regenerate.

2.      Errors that are the result of a breach of contract between the two parties (initiator and target). There is no point in retrying in this case since these problems indicate either an application defect or a deployment configuration problem. The list of these errors is 8494 Access denied, 8490 Service does not exist and 8425, 8408, 8428, 8498, 8430, 8457, 8437, 9616 all of which are a form of service contract agreement violation. In case you wonder how can one hit errors like 8437 and why isn’t SQL Server itself enforcing the correct SENT BY TARGET/SENT BY INITIATOR when the SEND verb is issued this is because the message type and contract definition may be different between the initiator and target database.

3.      Error that are the result of a database DROP or ALTER operation and the retry result is undefined. These are 8470, 8487 and 8469.

The appropriate action for each case depends on you application semantics. For my example of reusing/recycling dialogs I would only take the action of resending pending messages for errors in the first case 8489, 8462, 9719 and 28052:

...
ELSE IF @messageTypeName = N'http://schemas.microsoft.com/SQL/ServiceBroker/Error'
BEGIN
      -- Insert your error handling here. Could send a notification or
      -- store the error in a table for further inspection
      -- We’re gonna log the error into the ERRORLOG and Eventvwr.exe
      --
      DECLARE @error INT;
      DECLARE @description NVARCHAR(4000);
      WITH XMLNAMESPACES ('http://schemas.microsoft.com/SQL/ServiceBroker/Error' AS ssb)
      SELECT
            @error = CAST(@messageBody AS XML).value(
					'(//ssb:Error/ssb:Code)[1]', 'INT'),
            @description = CAST(@messageBody AS XML).value(
					'(//ssb:Error/ssb:Description)[1]', 'NVARCHAR(4000)');
      RAISERROR(N'Received error Code:%i Description:"%s"', 
			16, 1, @error, @description) WITH LOG;
     
      IF (@error IN (-8489, -8462, -9719, -28052))
      BEGIN
            -- Insert call to usp_ResendPending here
      END
           
      -- END CONVERSATION has to be issued AFTER the call to usp_ResendPending
      END CONVERSATION @handle;
END

Resending on any of the other errors would most likely result immediately in an error being sent back because the condition that caused the first error would simply repeat itself, being SEND access denied or service contract agreement violation or whatever.

Idempotent Messages

Ultimately a warning for caution if you choose to resend pending messages. The fact that a message is pending in sys.transmission_queue is not a guarantee that the target has not received that message. It only indicates that an acknowledgment was not received back and/or the sender did not yet delete the pending message. So if you decide to add resending logic in your application be careful that the sender is now prone to receiving duplicate data as the same message payload is successfully delivered twice. One way to achieve this is to make the payload your messages carry idempotent, but how to achieve this is completely application specific, so I cannot give any generic advice on this (but hey, you could choose to use my consulting services and get a problem specific answer J).

Remove pooling for data changes from a WCF front end

November 1st, 2007

The question was asked on the forums at http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=2320807&SiteID=1: How to enable a WCF based application to benefit from SQL Server 2005 Query Notifications? Basically the front end clients should be notified when data is changed on the back-end server tables as possible with least possible pooling.

Deploying SqlDependency in the WCF service itself to leverage the server’s Query Notification is a relatively easy process, simply follow the SqlDependency usage guidelines for any application: start SqlDependency, then attach a SqlDependency object instance to your SqlCommand objects, use this object’s OnChange event to react to changes, and subscribe again once notified.

Read the rest of this entry »