Writing Service Broker Procedures

October 16th, 2006

It’s been a while since I posted an entry in this blog and this article is long overdue. There were a series of events that prevented me from posting this, not least impacting being the fact that I’ve opened a WoW account…

T-SQL RECEIVE. Fast.

A question often asked is how to write a typical activated procedure? This article will cover the ways to write a performant T-SQL procedure to process messages. I am not going to cover CLR procedures for now.

Read the rest of this entry »

The Mysterious Notification

June 17th, 2006

Update:: for a way to leverage SqlDependency and Query Notifications from LINQ, see SqlDependency based caching of LINQ Queries.

One of the most mysterious features shipped with SQL Server 2005 seems to be the various flavors of notifications on data change: SqlNotificationRequest, SqlDependency and SqlCacheDependency. I see confusion on how this features work, how to use them and how to troubleshoot problems. Why are there three flavors of apparently the same functionality? And how is Service Broker involved into all of these?

Query Notifications

In reality, there is only one feature in the SQL Server 2005 engine that delivers notifications on subscriptions for data changes: Query Notifications. Clients can submit a query requesting to be notified when data was modified in a manner that would change the query result and the server sends a notification when this change occurs. These requests are called ‘query notification subscriptions’. The list of notification subscriptions can be seen in the server level view sys.dm_qn_subscriptions:

 

select * from sys.dm_qn_subscriptions

 

Along with the query submitted for the notification, the client submits a service name and a broker instance. Each notification subscription begins a Service Broker dialog with this provided service and broker instance. When data is changed and the change would affect the result of the submitted query result, a message is sent on this dialog. By sending this message, the client is considered notified and the notification subscription is removed. If client desires to be notified on further changes, is supposed to subscribe again.

So we’ve seen that the notification is not delivered back to the client, but a Service Broker message is instead sent to the service the client provided in the subscription request. All normal rules for delivery, routing and dialog security apply to the dialog used to send this message. This means that the notification message can be sent to service hosted in the same database, in a different database or even on a remote machine. Also, there is no need for the client to be connected to receive the notification. It is perfectly acceptable for a client to submit a query for a notification subscription then disconnect and shutdown.

The client consumes the notification message just like any other Service Broker message: by RECEIVE-ing it from the service’s queue. The notification message will be of type [http://schemas.microsoft.com/SQL/Notifications/QueryNotification], an XML message type. This message is part of the [http://schemas.microsoft.com/SQL/Notifications/PostQueryNotification]contract, which means that the service that receives the notification messages must be bound to this contract. After the client receives the message, is supposed to end the conversation on which the message was received, using END CONVERSATION (make sure you don’t use the CLEANUP clause!).

 

The restrictions that apply to queries submitted for notifications are explained in BOL in the ‘Creating a Query for Notification’ topic:http://msdn2.microsoft.com/en-us/library/ms181122(SQL.90).aspx. Although the query can be a stored procedure, it must not contain any flow control statements (IF, WHILE, BEGIN TRY  etc).

Clients can submit notification subscription request by programming directly against the SQL Native Client, using the HTTP SOAP access to SQL Server or, most commonly, using the ADO.Net client components. AFAIK clients cannot subscribe using the OLEDB, ODBC or JDBC components. Updated: It is possible from OleDB and ODBC as well, see Working with Query Notifications.

The cost of a subscription

Query notification subscriptions have little cost in the SQL Server 2005 engine. Notification subscriptions are only metadata, and the effect of a subscription is to modify the query plans in a manner that allows the relevant data changes to be detected. The picture below shows how a plan for a INSERT INTO TEST … statement is modified when there is a query notification subscription active with the query SELECT * FROM TEST:

 

This plan shows that, while cheap, query notification subscriptions are not free. The cost associated with them is similar to the cost of having a secondary index on the data, or the cost of having an indexed view.

Server Restart

When the SQL Server 2005 is restarted, all query notification subscriptions are notified and ended.

KILL QUERY NOTIFICATION SUBSCRIPTION

This Transact-SQL statement can be used to administratively end query notification subscriptions. The BOL describe its usage: http://msdn2.microsoft.com/en-us/library/ms186764.aspx

SqlNotificationRequest

This is simplest ADO.Net component for subscribing to Query Notifications. This class is used directly to create a query notifications subscription. The usage is straightforward:

         Create a new SqlNotificationRequest object, passing in the appropriate Service Broker service name and broker instance

         Assign the newly created SqlNotificationRequest to the Notification property of a SqlCommand

         Execute the SqlCommand.

MSDN has a sample of this technique here: http://msdn2.microsoft.com/en-us/3ht3391b.aspx

Using the SqlNotificationRequest leaves the task of handling the notification message entirely to the client application. While this is the most flexible way of leveraging the Query Notification functionality, it requires knowledge and understanding of the way Service Broker delivers the messages and how to write a Service Broker application to process the notification messages.

SqlDependency

This component tries to make the task of handling the Query Notification subscription messages as straightforward as possible. Using the SqlDependency the application developer gets a CLR callback whenever data has changed. How this is achieved is very simple: the SqlDependency uses SqlNotification to subscribe to query notifications and then the SqlDependency infrastructure uses a just-in-time-created service and queue to receive the notification messages. It starts a background client thread that posts a WAITFOR(RECEIVE…) on this queue and whenever a notification message is delivered by Service Broker into this queue, the background thread will receive this message, inspects it and invoke the appropriate callback. The static methods Start and Stop on the SqlDependency class are starting and stopping this background thread, as well as create and drop the service and queue used by the SqlDependency infrastructure. This background thread is shared by all requests in one appdomain. This is important, because if each SqlDependency request would start its own listener, the back end server would quickly get swamped by all those requests issuing WAITFOR(RECEIVE…) statements, each blocking a server thread.

The MSDN contains a sample of how to use the SqlDependency here: http://msdn2.microsoft.com/en-us/a52dhwx7.aspx. Note how, similarly to the SqlNotification usage, the client is expected to subscribe again if it whishes to be further notified.

Abrupt client disconnects

If a client disconnects abruptly without stopping the SqlDependency infrastructure, the queue and service created by the SqlDependency.Start are left stranded. This is not a problem, because the SqlDependency infrastructure also deploys a stored proc associated with activation on this queue and sets up a dialog timer. When this timer fires, the procedure is activated and this procedure cleans up the queue, the service and the activated procedure itself. Due to the pending WAITFOR(RECEIVE…) permanently posted by the SqlDependency background thread, the activated procedure will launch only if the client has disconnected w/o cleaning up.

SqlCacheDependency

This is a component designed specifically for ASP.Net to cache data, using the SqlDependency whenever possible. I’m not going to dwell into how it works, simply because I don’t really know. But once one understands how the Query Notifications and SqlDependency class work, the SqlCacheDependency is just another use of this infrastructure, with nothing special to it.

Troubleshooting Query Notifications

While the SqlDependency infrastructure is a great help to developers, is it often used w/o properly understanding its functionality and I often see people totally lost when it comes to troubleshooting a problem. In fact, BOL has a dedicated chapter for this topic here: http://msdn2.microsoft.com/en-us/library/ms177469.aspx. The Profiler can show the Query Notification events that are reported when a new subscription is registered. Once a notification subscription is notified, the notification message is delivered using Service Broker and all of my comments related to troubleshooting dialogs apply to this message delivery as well. If the notification message is no delivered, the first place to look is the transmission_status column in the sys.trasnmission_queue view in the sender’s database.

Call a procedure in another database from an activated procedure

March 7th, 2006

In my previous post http://rusanu.com/2006/03/01/signing-an-activated-procedure/ I’ve showed how to use code signing to enable a server level privilege (view server state) when running an activated stored procedure. I’ll show now how to solve a very similar issue: call a stored procedure from another database. Why this is a problem is explained in this post: http://rusanu.com/2006/01/12/why-does-feature-not-work-under-activation/

So let’s say the ‘SessionsService’ from my previous post needs some new functionality: it has to audit the requests it’s receive. An audit infrastructure already exists in the [AuditInfrastructure] database, all is needed is to call the stored procedure [audit_record_request] in that database and the request will be audited. First create this ‘audit infrastructure’, which for our example will be very simple:

create database [AuditInfrastructure]
go

use
[AuditInfrastructure]
go

-- An audit table, simply stores the time of the request
--
create table audit_table (request_time datetime);
go

-- this is the audit procedure
--
create procedure audit_record_request
as
begin
set nocount on
insert into
audit_table (request_time) values (getdate());
end

So now all we have to do is to change the SessionService procedure to call this audit procedure:

USE [DemoActivationSigning];
GO

ALTER PROCEDURE
[SessionsServiceProcedure]
AS
BEGIN
SET NOCOUNT ON
;
DECLARE @dh UNIQUEIDENTIFIER;
DECLARE @mt SYSNAME;

BEGIN TRANSACTION;
WAITFOR (RECEIVE TOP (1) @dh = conversation_handle,
@mt = message_type_name
FROM [SessionsQueue]), TIMEOUT 1000;
WHILE (@dh IS NOT NULL)
BEGIN
If the message is a request message,
send back a response with the list of sessions

IF (@mt = N‘RequestSessions’)
BEGIN
Get the list of current sessions
and send it back as a response

DECLARE @response XML;
SELECT @response = (
SELECT * FROM sys.dm_exec_sessions
FOR XML PATH (session), TYPE);
SEND ON CONVERSATION @dh
MESSAGE TYPE [Sessions]
(@response);
-- This is the extra call to the audit procedure
EXECUTE [AuditInfrastructure]..[audit_record_request];
END
End the conversation and commit
END CONVERSATION
@dh;
COMMIT;

Try to loop once more if there are message

SELECT @dh = NULL;
BEGIN TRANSACTION;
WAITFOR (RECEIVE TOP (1) @dh = conversation_handle,
@mt = message_type_name
FROM [SessionsQueue]), TIMEOUT 1000;
END
COMMIT
;
END
GO

So now we send a request to the ‘SessionsService’

 

DECLARE @dh UNIQUEIDENTIFIER;

BEGIN DIALOG CONVERSATION @dh

      FROM SERVICE [Requests]

      TO SERVICE ‘SessionsService’

      ON CONTRACT [SessionsContract]

      WITH ENCRYPTION = OFF;

SEND ON CONVERSATION @dh

      MESSAGE TYPE [RequestSessions];

 

 

We expect a response back, but nothing happens. The ‘SessionsService’ has gone silent. If we look into the Event Viewer, will find some troublesome entries:

 

Event Type: Information
Event Source: MSSQLSERVER
Event Category: (2)
Event ID: 9724
Date:  3/7/2006
Time:  10:36:25 AM
User:  REDMOND\remusr
Computer: REMUSR10
Description:
The activated proc [dbo].[SessionsServiceProcedure] running on queue DemoActivationSigning.dbo.SessionsQueue output the following:  ‘The server principal “REDMOND\remusr” is not able to access the database “AuditInfrastructure” under the current security context.’

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: fc 25 00 00 0a 00 00 00   ü%……
0008: 09 00 00 00 52 00 45 00   ….R.E.
0010: 4d 00 55 00 53 00 52 00   M.U.S.R.
0018: 31 00 30 00 00 00 14 00   1.0…..
0020: 00 00 41 00 75 00 64 00   ..A.u.d.
0028: 69 00 74 00 49 00 6e 00   i.t.I.n.
0030: 66 00 72 00 61 00 73 00   f.r.a.s.
0038: 74 00 72 00 75 00 63 00   t.r.u.c.
0040: 74 00 75 00 72 00 65 00   t.u.r.e.
0048: 00 00
                     ..

 

 

Note: In case you wonder ‘so what happened to my request?.The activated stored procedure has thrown an error and it rolled back. Because Service Broker is a fully transactional, the dequeued request was rolled back and is again available for dequeue. The activation will kick in again, causing the same error and again the request to be rolled back. Then again activation will kick in and so on and so forth. Eventually (after 5 consecutive rollbacks) the Service Broker Poisson Message support will detect this situation and will disable the queue.

 

The problem is, as expected, the database impersonation context, as explained here. Same as in the case of server level privileges, the easiest fix is to mark the database trustworthy:

 

ALTER DATABASE [DemoActivationSigning] SET TRUSTWORTHY ON

 

If we cannot afford this due to security risks (marking the database trustworthy elevates the database dbo to a de-facto sysadmin), we must reside to code signing. The steps are these:

         alter the procedure to have an EXECUTE AS clause (otherwise the code signing infrastructure does not work)

         create a certificate with a private key in the [DemoActivationSigning] database

         sign the procedure

         drop the private key of the certificate

         copy the certificate into the [AuditInfrastructure] database (backup the certificate to a file and the create from that file)

         derive a user from the certificate in the [AuditInfrastructure]database

         grant the desired privileges to this user

 

Here is the code for these steps:

 

USE [DemoActivationSigning];

GO

 

— Create aprocedure that implements

— the [SessionsService] service

ALTER PROCEDURE [SessionsServiceProcedure]

      WITH EXECUTE AS OWNER

AS

BEGIN

      SET NOCOUNT ON;

      DECLARE @dh UNIQUEIDENTIFIER;

      DECLARE @mt SYSNAME;

 

      BEGIN TRANSACTION;

      WAITFOR (RECEIVE TOP (1) @dh = conversation_handle,

            @mt = message_type_name

            FROM [SessionsQueue]), TIMEOUT 1000;

      WHILE (@dh IS NOT NULL)

      BEGIN

            — If the message is a request message,

            — send back a response with the list of sessions

           

            IF (@mt = N‘RequestSessions’)

            BEGIN

                  — Get the list of current sessions

                  — and send it back as a response

                 

                  DECLARE @response XML;

                  SELECT @response = (

                        SELECT * FROM sys.dm_exec_sessions

                              FOR XML PATH (‘session’), TYPE);

                  SEND ON CONVERSATION @dh

                        MESSAGE TYPE [Sessions]

                        (@response);

                  EXECUTE [AuditInfrastructure]..[audit_record_request];

            END

            — End the conversation and commit

            END CONVERSATION @dh;

            COMMIT;

     

            — Try to loop once more if there are message

           

            SELECT @dh = NULL;

            BEGIN TRANSACTION;

            WAITFOR (RECEIVE TOP (1) @dh = conversation_handle,

                  @mt = message_type_name

                  FROM [SessionsQueue]), TIMEOUT 1000;

      END

      COMMIT;

END

GO

 

— Create a certificate with a private key

— to sign the procedure with. The password

— used is not important, we’ll drop the

— private key

CREATE CERTIFICATE [SessionsServiceProcedureAudit]

      ENCRYPTION BY PASSWORD = ‘Password#1234’

      WITH SUBJECT = ‘SessionsServiceProcedure Signing  for audit certificate’;

GO

 

— Sign the procedure with the certificate’s private key

ADD SIGNATURE TO OBJECT::[SessionsServiceProcedure]

      BY CERTIFICATE [SessionsServiceProcedureAudit]

            WITH PASSWORD = ‘Password#1234’;

GO

 

— Drop the private key. This way it cannot be

— used again to sign other procedures.

ALTER CERTIFICATE [SessionsServiceProcedureAudit]

      REMOVE PRIVATE KEY;

GO

 

— Copy the certificate in [master]

— We must backup to a file and create

— the certificate in [master] from this file

BACKUP CERTIFICATE [SessionsServiceProcedureAudit]

      TO FILE = ‘C:\SessionsServiceProcedureAudit.CER’;

GO

 

USE [AuditInfrastructure]

GO

 

CREATE CERTIFICATE [SessionsServiceProcedureAudit]

      FROM FILE = ‘C:\SessionsServiceProcedureAudit.CER’;

GO

 

CREATE USER [SessionsServiceProcedureAudit] FROM CERTIFICATE [SessionsServiceProcedureAudit];

G0

 

— ‘AUTHENTICATE’ permission is required for all other permissions to take effect

GRANT AUTHENTICATE TO [SessionsServiceProcedureAudit];

GRANT EXECUTE ON [audit_record_request] TO [SessionsServiceProcedureAudit];

GO

 

— Enable back the disabled ‘SessionsService’ queue

ALTER QUEUE [SessionsQueue] WITH STATUS = ON;

GO

 

— Check that the response is now sent back

WAITFOR (RECEIVE CAST(message_body AS XML) FROM [RequestsQueue]);

 

Notice that I did not sent another request. The existing request was still there, waiting for the procedure to be fixed so it can be processed correctly.

Signing an activated procedure

March 1st, 2006

In my previous entry http://rusanu.com/2006/01/12/why-does-feature-not-work-under-activation/ I mentioned that one way to enable an activated procedure to be trusted at the server level is to sign the stored procedure with a certificate and grant appropriate permissions to the signing certificate. In this entry I want to show the details how to do this.

Read the rest of this entry »