rusanu.com

SQL Server Clustered Columnstore Indexes at TechEd 2013

remus — Tue, 11 Jun 2013 13:31:05 +0000

Now that the TechEd 2013 presentations are up online on Channel9 you can check out Brian Mitchell’s session What’s New for Columnstore Indexes and Batch Mode Processing. Brian does a great job at presenting the new updatable clustered columnstore indexes and the enhancements done to the vectorized query execution (aka. batch mode). Besides the TechEd presentation there is also another excellent resource available online right now for your education on the topic: the SIGMOD 2013 paper Enhancements to SQL Server Column Stores. Besides the obvious updatability, this paper cites some more improvements that are available in clustered columnstores:

Improved Index Build
Sampling Support
Bookmark Support
Schema modification support
Support for short strings
Mixed Execution Mode
Hash Join support
Improvements in Bitmap filters
Archival support

It should be no surprise to anyone studying columnar storage that the updatable clustered columnstores coming with the next version of SQL Server are based on deltastores. I talked before about the SQL Server 2012 Columnstore internals and I explained why the highly compressed format that makes columnar storage so fast it also makes it basically impossible to update in-place. The technique of having a ‘deltastore’ which stores updates and, during scans, merge the updates with the columnar data is not new and is employed by several of the columnar storage solution vendors.

Deltastores

Deltastores are ordinary B-Trees that store uncompressed row groups of the clustered columnstore

Columnstores introduce a new unit of organization, a row-group. A row-group is a logical unit that groups up to 2²⁰ rows (about 1 million rows). In SQL Server 2012 the row-groups where implicit and there was catalog view to show them. As you can see in Brian’s presentation SQL Server 14 adds a new catalog view: sys.column_store_row_groups. This catalog view show the state of each row group for all columnstores (including non-clustered ones). Updatable clustered columnstores can show the row groups in COMPRESSED state or in OPEN/CLOSED state. The OPEN/CLOSED row groups are deltastores (yes, there could be multiple deltastores per columnstore, see the above mentioned SIGMOD 2013 paper). OPEN row groups are ready to accept more inserts while CLOSED row groups have filled up and are awaiting compression. The structure of a deltastore is explained in the SIGMOD paper:

A delta store contains the same columns as the corresponding column store index. The B-tree key is a unique integer row ID generated by the system (column stores do not have unique keys).

If you wonder what a unique integer row ID generated by the system actually is, remember how uniqueifier columns work. Deltastores are managed entirely by the engine, there is no DDL to control the creation and deletion of deltastores. The engine creates a new deltastore whenever it needs one to handle inserts, closes them when full (have 1 million rows) and a background process called the Tuple Mover compresses this closed deltastores into columnar storage format.

When handling deltastores the columnar storage advantages are dimmed. Deltastores are row mode storage so the entire row has to be read, not only the column(s) of interest. Segment elimination does not occur for deltastores since the deltastores do not have metadata about min and max values contained inside each column. Parallel scans will distribute the deltastores among the threads so that multiple deltastores are scanned in parallel, but there is no attempt for parallelism inside a single deltastores. At maximum size of 1 million rows they’re simply too small to justify the engineering complications of handling parallelism inside the deltastores. All this is explained in the SIGMOD paper:

Parallel scans assign each delta store to a single thread of execution. A single delta store is too small to justify scanning in parallel but multiple delta stores can be scanned in parallel. Scanning delta stores is slower than scanning data in columnar format because complete records have to be read and not just the columns needed by the query.

Tuple Mover

The background Tuple Mover process is responsible to compressing full deltastores. The Tuple Mover is an online operation, it does not prevent data reads from the deltastores being compressed. This is described in the SIGMOD paper:

The Tuple Mover reads one closed delta store at a time and starts building the corresponding compressed segments. During this time scans continue to see and read the delta store. When the Tuple Mover has finished compressing the delta store, the newly created segments are made visible and the delta store is made invisible in a single, atomic operation. New scans will see and scan the compressed format. The Tuple Mover then waits for all scans still operating in the delta store to drain after which the delta store is removed.

Concurrent deletes or updates are blocked while the Tuple Mover compresses a deltastore. Concurrent Inserts are not blocked by the Tuple Mover. As for the ‘background process’ part of the Tuple Mover the closest analogy is the Ghost Cleanup process. Veterans know that Ghost Cleanup is never tuned right for my job, is always either too aggressive for some users or too slow for others. Will Tuple Mover suffer from the same problems? I don’t expect it to, primarily because the unit of work is big. It takes time to accumulate 1 million rows in single row by row inserts.

The removal of compressed deltastores is very efficient as is basically a deallocation (as efficient as a DROP). This is important so that the Tuple Mover does not generate exaggerate logging during compression.

Delete Bitmaps

The deleted bitmap is another B-Tree associated with the clustered columnstore. There is only one deleted bitmap for the entire columnstore, it covers all the row-groups (all segments). Only compressed segments use a deleted bitmap. The DELETE operation is in effect an insert into the deleted bitmap, the row-group and tuple number of the deleted row is inserted into the deleted bitmap. Scans (reads) honor the deleted bitmap by filtering out any row (tuple) marked as deleted. It is recommendable that clustered columnstore indexes that have seen a large number of deletes to be rebuild in order to restore the ‘health’ of their segments by removing the deleted rows. UPDATE operations on clustered columnstores are always split updates, meaning the Query Optimizer will create a plan that contains a delete and an insert for any UPDATE.

Delete bitmaps do not cover deltastores. As B-Trees, the deltastores support direct delete so they implement the delete by removing the deleted row.

Bulk Insert

Large bulk insert operations do not insert rows into delta stores but convert batches of rows directly into columnar format. The operation buffers rows until a sufficient number of rows has accumulated, converts them into columnar format, and writes the resulting segments and dictionaries to disk. This is very efficient; it reduces the IO requirements and immediately produces the columnar format needed for fast scans. The downside is the large memory space needed to buffer rows.

For clustered columnstores bulk insert performance is critical, as is the bread and butter of the data warehousing ETL scenarios. You are going to have to give attention to your ETL pipeline and drive to achieve the optimal directly compressed format. This requires the bulk insert to be able to upload close to 1 million rows per partition in each batch. If the bulk insert uses too small batches then the result will be sub-optimal deltastores instead of the optimized compressed segments. With SSIS you will have to pay attention to the data flow buffer size as the default 10MB is way too small for achieving efficient columnstore bulk inserts. Of course there will be cases when there simply isn’t enough data to upload and in such cases the bulk insert will result in a deltastore instead of compressed columnstores. The deltastores will be left OPEN and subsequent bulk insert operations will reuse them and fill them up, close them and leave them for the Tuple Mover to compress them. While achieving the directly compressed format during bulk insert is desirable, you should not stretch your ETL and business logic out of the way just to achieve it. Going through the intermediate deltastore will remedy itself automatically once sufficient data accumulates.

INSERT … SELECT … targeting clustered columnstore tables is a bulk insert operation

Enjoy! SELECT … INTO has always been a bulk (and minimally logged) operation, but that can only create a row-mode heap. With INSERT … SELECT being now optimized for clustered columnstore indexes to use the bulk insert API, your ETL pipeline can construct the switch-in data directly into columnar storage format in one single pass, w/o having to resort to a rebuild. This is, again, described in the SIGMOD paper.

Trickle Insert

Trickle inserts are all inserts that do not come through the bulk insert API. INSERT INTO ... VALUES ... statements are trickle inserts. Trickle inserts are handled always by a deltastore and they can never create directly compressed data. UPDATE statements (as well as MERGE) result in split updates (delete and insert) and will insert in trickle mode.

Improved Index Build

Clustered columnstore index build is smart. It solve several problems in new ways:

Relevant Dictionaries: Columnar storage columns use a global dictionary, shared by all segments, and local dictionaries shared by specific segments. The more relevant the global dictionary (the more actual data values are covered by it) the better the data compression achieved, as the secondary dictionaries are smaller or even not needed. In SQL Server 2012 the global dictionary was just the first dictionary built, so it could contain skewed data resulting in poor relevance. With clustered columnstore index build the entire data is first sampled ina first stage, global dictionaries are built for all columns that requires them, and then the build proper starts. This creates much better global dictionaries and result in significant storage (compression) improvement.
Minimize blocking: This problem is not specific to clustered columnstore indexes: during an offline index rebuild, there is no reason to block reads. Yes, there is a risk of deadlock at the end of the index rebuild, but there are ways to solve that problem. With clustered columnstore indexes offline rebuild operations are semi-online, meaning reads are allowed but updates are blocked.
Workload Variation: Columnstore index build is very memory intensive. Traditional execution model determines the DOP at beginning of execution and then the query executes with the given DOP. With the improved columnstore build the actual execution DOP varies as the build progresses and the build process can actively and voluntarily reduce it’s DOP (by ‘parking’ execution threads) in order to adjust to low memory conditions.

Registry bloat after SQL Server 2012 SP1 installation

remus — Fri, 15 Feb 2013 12:55:09 +0000

SQL Server 2012 installation has the potential to leave an msiexec.exe installer process running after the installation finishes, as described in Windows Installer starts repeatedly after you install SQL Server 2012 SP1:

After you install SQL Server 2012 SP1 on a computer, the Windows Installer (Msiexec.exe) process is repeatedly started to repair certain assemblies. Additionally, the following events are logged in the Application log:
EventId: 1004
Source: MsiInstaller
Description: Detection of product ‘{A7037EB2-F953-4B12-B843-195F4D988DA1}’, feature ‘SQL_Tools_Ans’, Component ‘{0CECE655-2A0F-4593-AF4B-EFC31D622982}’ failed. The resource”does not exist.

EventId: 1001
Source: MsiInstaller
Description: Detection of product ‘{A7037EB2-F953-4B12-B843-195F4D988DA1}’, feature ‘SQL_Tools_Ans’ failed during request for component ‘{6E985C15-8B6D-413D-B456-4F624D9C11C2}’

When this issue occurs, you experience high CPU usage.
Cause

This issue occurs because the SQL Server 2012 components reference mismatched assemblies. This behavior causes native image generation to fail repeatedly on certain assemblies. Therefore, a repair operation is initiated on the installer package.

But the this problem has a much sinister side effect: it causes growth of the HKLM\Software registry hive. Except for the System hive, all the other registry hives are still restricted in size to a max of 2GB, see Registry Storage Space:

Views of the registry files are mapped in paged pool memory…The maximum size of a registry hive is 2 GB, except for the system hive.

As the runaway msiexec process bloats the Software registry hive your system may approach the maximum hive size and, even before that maximum is reached, the large hive will consume more and more of the very precious kernel paged pool memory. The system may start to exhibit erratic behavior, complaining about low ‘resources’, closing connections and other symptoms. For an example of how this erratic bahior may manifest, read Why the registry size can cause problems with your SQL 2012 AlwaysOn/Failover Cluster setup.

If you installed SQL Server 2012 SP1 recently follow the steps in KB2793634 for a fix

The biggest problem with the registry bloat erratic behavior is that it occurs long after the SQL Server 2012 SP1 installation and is quite difficult, even for expert users, to trace back the causality of the server erratic behavior to the SP1 installation. If you are uncertain if this issue is affecting you, run dir %SystemRoot%\system32\config and check the size of the SOFTWARE file, which is the storage of this registry hive. If you are indeed dealing with a bloated registry hive, visit KB2498915: How to Compress “Bloated” Registry Hives:

1) Boot from a WinPE disk.
2) Open regedit while booted in WinPe, load the bloated hive under HLKM. (e.g. HKLM\Bloated)
3) Once the bloated hive has been loaded, export the loaded hive as a “Registry Hive” file with a unique name. (e.g. %windir%\system32\config\compressedhive)
a) You can use dir from a command line to verify the old and new sizes of the registry hives.
4) Unload the bloated hive from regedit. (If you get an error here, close the registry editor. Then reopen the registry editor and try again.)
5) Rename the hives so that you will boot with the compressed hive.
e.g.
c:\windows\system32\config\ren software software.old
c:\windows\system32\config\ren compressedhive software

Update: At the time of writing this the SQL Server SP1 download page recommends installing the KB2793634. Kind folk that have been in the trenches and had to deal with this problem have also give me more feedback about the symptoms and solutions. I’ve been told that if the registry is already bloated due to the msiexec issue then applying the KB2793634 will not be enough. The fix only prevents further auto-restarts of msiexec, but it does not compress the registry. The following registry keys *may* be already huge:

HKLM\Software\Wow6432Node\Microsoft\.NETFramework\v2.0.50.27\NGENService
HKLM\SOFTWARE\Microsoft\.NETFramework\v2.0.50727\NGENService
HKLM\Software\Wow6432Node\Microsoft\.NETFramework\v4.0.30319\NGENService
HKLM\SOFTWARE\Microsoft\.NETFramework\v4.0.30319\NGENService

Symptoms of registry hive bloat include, but are not limited to:

Cluster service is crashing and “The Group Policy Client service failed the logon. Insufficient system resources exist to complete the requested service” are logged into the event viewer log.
It is impossible to perform additional installations, or you can do this just shortly after rebooting the server.
Users with domain accounts are unable to logon on the machine. When they log in a temporary profile is loaded.
In Server Manager the Roles and Features are not listed anymore.
Many actions are prevented with the error message ” Insufficient system resources exist to complete the requested service”.

The recommended action is to open a support case and contact CSS. If you are brave, can’t afford CSS support contracts, and you are convinced that the problems you’re experiencing is due to the SQL Server 2012 SP1 registry bload, you may try the following at your own risk:

Install the KB2793634 hot fix, as per the procedure described in the KB article.
From cmd prompt run %SystemRoot%\Microsoft.Net\Framework\v4.0.30319\ngen.exe executequeueditems and %SystemRoot%\Microsoft.Net\Framework64\v4.0.30319\ngen.exe executequeueditems
Compress the registry hive using a WinPE disk or from the Windows recovery mode (press F8 at the boot screen). Note that the Windows system on which you attempt to compress the hive must have the KB973817 installed: The Reg.exe utility does not compress a registry key when the utility saves a registry key to a hive file on a computer that is running Windows Server 2008, Windows Vista, Windows 7 or Windows Server 2008 R2.
After the compression procedure is complete, from a cmd prompt run SCF /SCANNOW. See KB929833 for more details.

In the case when the registry hive bloat prevents even the installation of the SP1 hotfix you may attempt increasing the hive size to 4GB, see RegistrySizeLimit. Editing this particular registry key to a wrong value can result in your system blue-screening during boot due to BAD_SYSTEM_CONFIG_INFO kernel panic. Each boot. Use at your own risk. The correct value to put is 0xFFFFFFFF.

If you plan to upgrade an RTM SQL Server 2012 instance to SP1 apply the slipstream SP1+CU2

If you have an RTM instance of SQL Server 2012 and plan to upgrade it to SP1 it is recommended to create a slipstream SP1+CU2 instalation and apply this instead. See Product Updates in SQL Server 2012 Installation, in SQL Server 2012 the slipstream functionality has been renamed to Product Updates:

The Product Update feature replaces the Slipstream functionality that was available in SQL Server 2008 PCU1. Therefore the command-line parameters, /PCUSource and /CUSource, associated with Slipstream functionality should no longer be used. The parameters will continue to work, but may be removed in a future release of SQL Server Setup. The /UpdateSource parameter combines the functionality of the Slipstream parameters.

Frankly I think the term ‘slipstream’ was less ambiguous than ‘product update’ but then who am I to complain…

How to enable Selective XML indexes in SQL Server 2012 SP1

remus — Mon, 11 Feb 2013 10:39:37 +0000

SQL Server 2012 SP1 has shipped a great enhancement to XML: Selective XML Indexes. When properly used these indexes can speed up the searching of XML columns tremendously, at little disk/size cost:

The selective XML index feature lets you promote only certain paths from the XML documents to index. At index creation time, these paths are evaluated, and the nodes that they point to are shredded and stored inside a relational table in SQL Server. This feature uses an efficient mapping algorithm developed by Microsoft Research in collaboration with the SQL Server product team. This algorithm maps the XML nodes to a single relational table, and achieves exceptional performance while requiring only modest storage space.

However, at the time of this post, the documentation on MSDN omits a crucial requirement: the database has to be enabled to support these new indexes. Because is an on-disk format change shipped in a service pack release the engine cannot use the new selective XML indexes unless explicitly allowed. And the database version must change to a version that the RTM SQL Server 2012 does not recognize so that the database is not accidentally attached/restored on an RTM instance of SQL Sever 2012 that would not comprehend the new Selective XML Index objects and would panic (undefined behavior). To enable Selective XML Indexes in the database you must run sp_db_selective_xml_index:

Enables and disables Selective XML Index functionality on a SQL Server database. If called without any parameters, the stored procedure returns 1 if the Selective XML Index is enabled on a particular database.

EXECUTE sys.sp_db_selective_xml_index @db_name = N'AdventureWorks2012' , @action = N'true'; GO

Be aware that once upgraded to support Selective XML indexes this database can no longer be attached or restored on an RTM instance of SQL Server 2012. This applies to log shipping, Database Mirroring and AlwaysOn relationships, which will break when this upgrade is performed. If all the partners in the log shipping, DBM session or AG are also upgraded to SQL Server 2012 SP1 you can re-enable the relationship after you enabled Selective XML Indexes.

How to enable and disable a queue using SMO

remus — Wed, 06 Feb 2013 12:22:19 +0000

The SMO object model for SQL Server ServiceQueue does allow one to enable or disable a queue, but the property that modifies the queue status is not intuitive, it is IsEnqueueEnabled:

Gets or sets the Boolean property that specifies whether the queue is enabled.

This property matches the catalog view column is_enqueue_enabled in sys.service_queues but bears little resemblance to the T-SQL statement used to enable or disable a queue: ALTER QUEUE ... WITH STATUS = {ON|OFF}

For example the following SMO code snippet:

            Server server = new Server("...");
            Database db = server.Databases["msdb"];
            ServiceQueue sq = new ServiceQueue(db.ServiceBroker, "foo");
            sq.Create();
            sq.IsEnqueueEnabled = false;
            sq.Alter();
            sq.IsEnqueueEnabled = true;
            sq.Alter();

generates the following T-SQL:

CREATE QUEUE [dbo].[foo];
ALTER QUEUE [dbo].[foo]  WITH STATUS = OFF ...;
ALTER QUEUE [dbo].[foo] WITH STATUS = ON ...;

SQL Server backup to URL

remus — Fri, 25 Jan 2013 12:24:15 +0000

With the SQL Server 2012 SP1 CU2 release a new important feature was added: ability to back up and restore a database straight from Azure Blob storage:

This feature released in SQL Server 2012 SP1 CU2, enables SQL Server backup and restore directly to the Windows Azure Blob service. This feature can be used to backup SQL Server databases on an on-premises instance or an instance of SQL Server running a hosted environment such as Windows Azure Virtual Machine. Backup to cloud offers benefits such as availability, limitless geo-replicated off-site storage, and ease of migration of data to and from the cloud.

The syntax for the new feature is straight forward. You must first create a credential for accessing your Azure Blob storage:

SQL Server requires Windows Azure account name and access key authentication to be stored in a SQL Server Credential. This information is used to authenticate to the Windows Azure account when it performs backup or restore operations.

CREATE CREDENTIAL mycredential WITH IDENTITY = 'mystorageaccount'
       ,SECRET = '' ;

BACKUP DATABASE AdventureWorks2012
      TO URL = 'https://mystorageaccount.blob.core.windows.net/mycontainer/db.bak'
      WITH CREDENTIAL = 'mycredential'
     ,STATS = 5;

RESTORE DATABASE AdventureWorks2012
     FROM URL = 'https://mystorageaccount.blob.core.windows.net/mycontainer/db.bak'
     WITH CREDENTIAL = 'mycredential'

Full backups, differential backups, log backups, filegroup backups, compressed backups are all supported. The only notable restrictions is that is not allowed to backup to two locations simultaneously. Here is the complete list of limitations:

The maximum backup size supported is 1 TB.

Backup to or restoring from the Windows Azure Blob storage service by using SQL Server Management Studio Backup or Restore wizard is not currently enabled

Creating a logical device name is not supported. So adding URL as a backup device using sp_dumpdevice or through SQL Server Management Studio is not supported

Appending to existing backup blobs is not supported. Backups to an existing Blob can only be overwritten by using the WITH FORMAT option

Backup to multiple blobs in a single backup operation is not supported

Specifying a block size with BACKUP is not supported.

Specifying a block size for restores might be required in certain scenarios

Specifying MAXTRANSFERSIZE is not supported.

Specifying backupset options – RETAINDAYS and EXPIREDATE are not supported.

SQL Server has a maximum limit of 259 characters for a backup device name. The BACKUP TO URL consumes 36 characters for the required elements used to specify the URL – ‘https://.blob.core.windows.net//.bak’, leaving 223 characters for account, container, and blob names put together.

I recommend going over Backup and Restore to Azure Blob Best Practices.

Case Sensitive collation sort order

remus — Fri, 23 Nov 2012 09:15:30 +0000

A recent inquiry from one of our front line CSS engineers had me look into how case sensitive collations decide the sort order. Consider a simple question like How should the values 'a 1', 'a 2', 'A 1' and 'A 2' sort?


create table [test] (
	[col] varchar(10)
		collate Latin1_General_CS_AS);
go

insert into [test] ([col]) values
	('a 1'),
	('a 2'),
	('A 1'),
	('A 2');
go

select [col]
from [test]
order by [col];

Here are two possible outputs:

Which one is correct? A programmer will chose the first order: 'a 1', 'a 2', 'A 1', 'A 2'. Because if one would implement a string comparison routine it would compare character by character until a difference is encountered, and 'a' sorts ahead of 'A'. But this answer is wrong. The correct sort order is 'a 1', 'A 1', 'a 2', 'A 2'! And if you ran the query in SQL Server you certainly got the second output. But look again at the sort order and focus on just the first character:

By default, the algorithm makes use of three fully-customizable levels. For the Latin script, these levels correspond roughly to: alphabetic ordering, diacritic ordering, case ordering

So, in a case sensitive collation, is 'a' ahead or after 'A' in the sort order? The images shows them actually interleaved, is 'a', 'A', 'a', 'A'. What’s going on? The answer is that collation sort order is a little more nuanced that just comparing characters until a difference is encountered. This is described in the Unicode Technical Standard #10: UNICODE COLLATION ALGORITHM. And yes, the same algorithm is applied for non-Unicode types (VARCHAR) too. The algorithm actually gives different weight to character differences and case differences, a difference in alphabetic order is more important than one in case order. To compare the sort order of two strings the algorithm is more like the following:

Compare every character in case insensitive, accent insensitive manner. If a difference is found, this decides the sort order. If no difference is found, continue.
Compare every character in case insensitive, accent sensitive manner. If a difference is found, this decides the sort order. If no difference is found, continue.
Compare every character in case sensitive manner (we already know from the step above there is no accent difference). If a difference is found, this decides the sort order. If no difference is found the strings are equal.

Needless to say the real algorithm does not need to traverse the strings 3 times, but the logic is equivalent to above. And remember that when the strings have different lengths then the comparison expands the shorter string with spaces and compares up to the length of the longest string. Combined with the case sensitivity rules this gives to a somewhat surprising result when using an inequality in a WHERE clause:


select [col]
from [test]
where [col] > 'A'
order by [col];

That’s right, we got back all 4 rows, including those that start with 'a'. This surprises some, but is the correct result. 'a 1' should be in the result, even though 'a' is < 'A'. If you follow the algorithm above: first we expand the shorter string with spaces, so the comparison is between 'a 1' and 'A '. Then we do the first pass comparison, which is only alphabetic order, case insensitive and accent insensitive, character by character: 'a' and 'A' are equal, ' ' and ' ' are equal, but '1' is > ' '. The comparison stops, we found a alphabetic order difference so 'a 1' > 'A ', the row qualifies and is included in the result. Ditto for 'a 2'.

Handling exceptions that occur during the RECEIVE statement in activated procedures

remus — Mon, 15 Oct 2012 11:43:18 +0000

The typical SQL Server activation procedure is contains a WHILE (1=1) loop and exit conditions based on checking @@ROWCOUNT. Error handling is done via a BEGIN TRY ... BEGIN CATCH block. This pattern is present in many Service Broker articles on the web, including this web site, in books and in Microsoft samples:

create procedure []
as
declare @dialog_handle uniqueidentifier
 , @message_type_name sysname
 , @message_body varbinary(max);
set nocount on;

while(1=1)
begin
 begin transaction;
 begin try;
  receive top(1)
   @dialog_handle = conversation_handle
   , @message_type_name = message_type_name
   , @message_body = message_body
  from [];
  if @@rowcount = 0
  begin
   rollback;
   break;
  end
  if @message_type_name = N''
  begin
   -- process the message here
                        ...
  end
  else if @message_type_name = N'http://schemas.microsoft.com/SQL/ServiceBroker/Error'
     or @message_type_name = N'http://schemas.microsoft.com/SQL/ServiceBroker/EndDialog'
  begin
   end conversation @dialog_handle;
  end
  commit transaction;
 end try
 begin catch
  declare @error_number int = ERROR_NUMBER()
   , @error_message nvarchar(4000) = ERROR_MESSAGE()
   , @xact_state int = XACT_STATE();
  if @xact_state = -1 or @xact_state = 1
  begin
   rollback;
  end
  -- log the error here
               ....
 end catch
end
go

This patter though contains a problem: it will handle very poorly a disabled queue, and hence it will handle very poorly poison messages.

Error 9617 The service queue “…” is currently disabled

Attempting to RECEIVE from a disabled queue will raise error 9617 but will not interrupt the batch. The above procedure error handling will handle the exception, eventually logging it somewhere, and then it will resume and loop again, hitting again error 9617. Your ERRORLOG file will likely grow quickly, the CPU will be busy due to the tight loop, all to no avail. Furthermore the constant executing RECEIVE and the activated procedure may even prevent you from re-enabling the queue as is being locked. The only way to stop such a runaway procedure is to KILL it.

Separate the exception handling of the RECEIVE statement

You can try several solutions. You could check for the queue state before issuing the RECEIVE. You could special case error 9617 in the CATCH block. But if error 9617 requires special handling, are you sure you have covered all the special cases? I would recommend a different approach. What happens here is that your error handling does not differentiate between an exception that occurred in the RECEIVE statement itself and an exception that occurred in the processing of the messages returned by RECEIVE. Differentiating this two cases would allow for separate handling: if the RECEIVE statement itself has raised an exception then is better to abandon and exit the loop. If the exception occurred in the message processing then, after the exception is handled based on business specific rules, it is OK to dequeue more messages and issue RECEIVE again:

create procedure []
as
declare @dialog_handle uniqueidentifier
 , @message_type_name sysname
 , @message_body varbinary(max)
 , @error_number int
 , @error_message nvarchar(4000)
 , @xact_state int;
set nocount on;

while(1=1)
begin
 set  @dialog_handle = null;
 begin transaction;
 begin try;
  receive top(1)
   @dialog_handle = conversation_handle
   , @message_type_name = message_type_name
   , @message_body = message_body
  from [];
 end try
begin catch
  -- this catch block handles errors in the RECEIVE statement
  set @error_number = ERROR_NUMBER();
  set @error_message = ERROR_MESSAGE();
  set @xact_state = XACT_STATE();
  if @xact_state = -1 or @xact_state = 1
  begin
   rollback;
  end
  -- log the error here
  ...
  break;
end catch
if @dialog_handle is null
begin
  rollback;
  break;
end
begin try
  if @message_type_name = N''
  begin
   -- process the message here
   ...
  end
  else if @message_type_name = N'http://schemas.microsoft.com/SQL/ServiceBroker/Error'
     or @message_type_name = N'http://schemas.microsoft.com/SQL/ServiceBroker/EndDialog'
  begin
   end conversation @dialog_handle;
  end
  commit transaction;
 end try
 begin catch
  -- this catch block handles errors in the message processing
  set @error_number = ERROR_NUMBER();
  set @error_message = ERROR_MESSAGE();
  set @xact_state = XACT_STATE();
  if @xact_state = -1 or @xact_state = 1
  begin
   rollback;
  end
  -- log the error here
 ...
 end catch
end
go

Why loop in the first place?

The idea of the loop inside the activated stored procedure is that once activated, a procedure should RECEIVE until it drains the message queue and only then exit. But here is the deal: the Queue Monitor that launched the activated procedure already loops for you! It will keep calling the activated procedure until the procedure issues a RECEIVE statement that returns no rows. So you can get rid of the WHILE (1=1) loop, which simplifies the procedure and make sit more robust. If we do so, we no longer need to distinguish between errors that occurred in the RECEIVE statement vs. errors that occur in the message processing:

create procedure []
as
declare @dialog_handle uniqueidentifier
 , @message_type_name sysname
 , @message_body varbinary(max)
 , @error_number int
 , @error_message nvarchar(4000)
 , @xact_state int;
set nocount on;
 begin transaction;
 begin try;
  receive top(1)
   @dialog_handle = conversation_handle
   , @message_type_name = message_type_name
   , @message_body = message_body
  from [];

  if @message_type_name = N''
  begin
   -- process the message here
   ...
  end
  else if @message_type_name = N'http://schemas.microsoft.com/SQL/ServiceBroker/Error'
     or @message_type_name = N'http://schemas.microsoft.com/SQL/ServiceBroker/EndDialog'
  begin
   end conversation @dialog_handle;
  end
  commit transaction;
 end try
 begin catch
  set @error_number = ERROR_NUMBER();
  set @error_message = ERROR_MESSAGE();
  set @xact_state = XACT_STATE();
  if @xact_state = -1 or @xact_state = 1
  begin
   rollback;
  end
  -- log the error here
 end catch
go

Notice there is no longer need for @@ROWCOUNT check, that is implicit in the testing of @message_type_name because it will be NULL if no message was received. There is only one BEGIN TRY ... BEGIN CATCH ... block, which makes for an easier to comprehend, test and present procedure. And with the WHILE (1=1) loop eliminated there are fewer chances for a bug causing an infinite loop, as it was the case with error 9617.

Deploying blog samples in production

Over time I learned that a lot of times readers deploy in production the code samples straight from the blogs they read. You have to keep in mind that a blog example is written first and foremost to illustrate the point of the blog post and not to handle a real life production environment. In my production code there must be error handling, exception logging, performance measurement instrumentation in place (with run-time knobs to enable or disable the instrumentation, or to dial the level of detail up or down) and so on and so forth. Also in a real production case I would try to leverage batch processing of messages as described in Writing Service Broker Procedures. When processing batch messages using a cursor the individual message processing is an excellent candidate for using a transaction savepoint, in the manner described in Exception Handling and Nested Transactions. Use your judgement and remember that this code here is a sample, only provided to guide you in the right direction.

How to shrink the SQL Server log

remus — Fri, 27 Jul 2012 07:52:41 +0000

I noticed that my database log file has grown to 200Gb. I tried to shrink it but is still 200Gb. How can I shrink the log and reduce the file size?

The problem is that even after you discover about DBCC SHRINKFILE and attempt to reduce the log size, the command seems not to work at all and leaves the log at the same size as before. What is happening?

If you look back at What is an LSN: Log Sequence Number you will see that LSNs are basically pointers (offsets) inside the log file. There is one level of indirection (the VLF sequence number) and then the rest of the LSN is basically an offset inside the Virtual Log File (the VLF). The log is always defined by the two LSNs: the head of the log (where new log records will be placed) and the tail of the log (what is the oldest log record of interest). Generating log activity (ie. any updates in the database) advance the head of the log LSN number. The tail of the log advances when the database log is being backed up (this is a simplification, more on it later).

Use DBCC LOGINFO to view the layout of the VFLs inside the log file

The image above shows as typical log file layout. To understand the layout of VLFs inside your LDF log file use DBCC LOGINFO. The head of the log moves ahead as transactions generate log records, occupying more of the free space ahead in the current VLF. If the current VLF fills up, a new empty VLF can be used. If there are no empty VLFs the system must grow the log LDF file and create a new VLF to allow for more log records to be written. This is when the physical LDF file actually increases and takes more space on disk:

Advancing the head of the log can cause LDF file growth when no free VLFs are available

In the image above some more transaction activity resulted in head of the log advancing forward. As the VLF2 filled up, the system had to allocate a new VLF by growing the physical log LDF file. The unused space in VLF1 cannot be used as long as there is even a single active LSN record in it, the What is an LSN: Log Sequence Number article explains why this is the case. If we now take a backup of the log truncation would occur. Truncation is described as ‘deleting the log records’ but no actual physical deletion needs to occur. It simply means the tail of the log will move forward: a database property that contains the LSN number of the tails of the log gets updated with the new LSN number repreesenting the new rail of the log position:

As the head of the log continues to advance due to normal database activity, it now has room to grow by re-using the free VLF 1. This will cause an active log wrap around: the log starts reusing free\ VLFs inside the LDF file and advancing the head of the log no longer causes physical LDF file to grow:

In this configuration if the head of the log continues to advance it has room to grow in the VLF 1. But, unless further truncation occurs, if it fills VLF 1 in order to continue to move ahead the head of the log will require yet again the LDF file to grow in size to accommodate a new free VLF:

If we truncate the log now the tail of the log would apparently follow the path taken by the head of the log and eventual catch up the current head of the log. In effect all that happens is that the database property that contains the LSN which si the current tail of the log gets updated with the new LSN. The apparent ‘path’ is just an effect of the nature of LSN structure.

In this state the LDF file contains 3 empty VLFs and only a small active log portion. However, the physical LDF file cannot be reduced in size, any attempt to shrink the file will fail. As a general rule a file (any file) can only be shrunk (reduce in size) by removing data at the end of the file (basically by reducing the file length). Is not possible to ‘delete’ from the beginning of a file or from the middle of a file. Unlike data MDF files, the system cannot afford to move records around in order to free space at the end of the file, remember that LSNs are basically an offset inside the file. The active portion of the log cannot be moved, because there all those records in the active portion of the log would suddenly become invalid.

If the last VLF in a log file is inactive, the log file can shrink

So if the LDF file can reduce its size (shrink) only be reducing its length and active VFLs cannot move it is clear what is the condition when the DBCC SHRINKFILE requires in order to succeed: the last VLF in the file must be free (inactive). Recursively apply this condition to the VLFs left after removing the last one and we have the answer to how much can the log shrink. And this is the explanation why almost always when you care about this information, you are in a bad state in which you cannot shrink the log file. Is just that the typical sequence of events leads exactly to this situation:

Full recovery model gets enabled on the database
No maintenance plan is in place and log backups are not taken
The large log LDF file is noticed
After a frantic search on internet, the solution of taking a log backup is found
Shrinking the log fails to yield any reduction in the log LDF file size

This sequence of actions causes the head of the log to advance forward, creating new VLFs as the log file expands. When the situation is detected and the log backups are taken, the tail of the log catches up as truncation (logical freeing) occurs. But in the end the head of the log is located in the last added VLF, which is exactly at the end of the log file. Since the last VLF is active, no shrink can occur, as explained above, despite a large portion of the log file being free (inactive). To make progress in this situation one must first cause the head of the log to move forward so that it fills the last VLF and then it wraps around and reuses the free VLF(s) at the beginning of the file. So, perhaps counter intuitively, you must actually generate more log activity in order to be able to shrink the log file. On an active database this log activity will occur naturally from ordinary use, but if there is no activity then you must cause some. For example, do some updates in a transaction and roll back. Repeat this until the head of the log has wrapped around and is located in the VLFs at the beginning of the file. As soon as this happens, take another log backup to cause truncation which will move the tail of the log forward, following the head of the log and wrapping around. Now the VLFs at the end of the file are inactive and DBCC SHRINKFILE can actually succeed:

DBCC LOGINFO

Here is an example output from running the DBCC LOGINFO command:

RecoveryUnitId FileId  FileSize  StartOffset  FSeqNo Status Parity CreateLSN
-------------- ----------------- ------------ ------ ------ ------ -----------------
0              2       253952    8192         242    0      128    0
0              2       278528    262144       243    0      128    0
0              2       311296    540672       241    0      64     30000000028200477
0              2       262144    851968       236    0      128    34000000025600489
0              2       262144    1114112      237    0      128    35000000010400007
0              2       262144    1376256      238    0      128    35000000023400481
0              2       262144    1638400      239    0      128    36000000013600418
0              2       262144    1900544      240    0      128    36000000036800482
0              2       262144    2162688      244    0      128    73000000037600584
0              2       262144    2424832      245    0      128    74000000009400581
0              2       262144    2686976      246    0      128    74000000034200584
0              2       327680    2949120      247    0      128    75000000013600585
0              2       327680    3276800      248    2      128    76000000001600581
(13 row(s) affected)

This example shows 13 VLFs in the file. 12 are inactive (status is 0) and only the very last one is active (status 2). The active VLF starts at offset 3276800 in the file and has a size of 327680. A quick sneak in sys.database_files reveals that the log file size is 440. The file size is in 8k pages (even for log, although the log file structure has nothing to do with the 8k data page of the MDF/NDF data files) so the file size is 3604480 bytes. Which matches exactly our last VLF ending (3276800 + 327680). So indeed the log file is 3.4 MB in size, from which 3.12 MB are inactive and the last 320 Kb are active (in use by the last VLF). Not that I would ever recommend shrinking a 3.4 MB log, but if we’d try to shrink this file it would yield no results because the very last VLF is active.

Tail of the log

I said before that the tail of the log is moved forward by log backups, but that is a simplification. Advancing the tail of the log is tad more complex because the tail is not a precise LSN: is the lowest LSN of any consumer that needs to look at the log records. Such consumers are:

Active Transactions: Any transaction that has not yet committed needs to retain all the log from when the transaction started, in case it has to rollback. Remember that rollback is implemented by reading back all the transaction log record and generating a compensating (undo) action for each.
Log Backup: Log backup needs to retain all the log generated until the next log backup runs, so it is copied into the backup file.
Transactional Replication: The transactional replication publisher agent has to read the log to identify changes that occurred and have to be distributed to the subscribers.
Database Mirroring, AlwaysOn: Both these technologies work by literally copying the log to the partners, so they require the log to be retained until is copied over.
Other: If you check the log_reuse_wait_desc column documentation in Factors That Can Delay Log Truncation you will see all the other processes that can retain the log tail, things like CHECKPOINT, an active backup operation, a database snapshot being created, a log scan (fn_dblog()) etc. All these will cause the tail of the log to stay in place (not progress forward).

The tail of the log will be the lowest LSN from any of the LSNs required by the processes above. Whenever the tail of the log advances forward the log is said to be truncated, as described in Transaction Log Truncation.

Inside the SQL Server 2012 Columnstore Indexes

remus — Tue, 29 May 2012 10:38:46 +0000

Columnar storage has established itself as the de-facto option for Business Intelligence (BI) storage. The traditional row-oriented storage of RDBMS was designed for fast single-row oriented OLTP workloads and it has problems handling the large volume range oriented analytical processing that characterizes BI workloads. But what is columnar storage and, more specifically, how does SQL Server 2012 implement columnar storage with the new COLUMNSTORE indexes?

Column Oriented Storage

The defining characteristic of columnar storage is the ability to read the values of a particular column of a table without having to read the values of all the other columns. In row-oriented storage this is impossible because the individual column values are physically stored grouped in rows on the pages and reading a page in order to read a column value must fetch the entire page in memory, thus automatically reading all the other columns in the row:

This image shows how in row oriented storage a query that only needs to read the values of column A needs to pay the penalty of reading the entire pages, including the unnecessary columns B, C, D and E, simply because the physical format of the record and page. If you’re not familiar with the row format I strongly recommend Paul Randal’s article Inside the Storage Engine: Anatomy of a Record.

By contrast the column oriented storage stores the data in a format that groups columns together on disk, so reading values from a single column incurs only the minimal IO needed to fetch in the column required by the query:

The most common question I hear often when columnar storage is how is a row stored?. In other words, if we store all the values of a column together, then how do we re-create a row? Ie. if the column Name has the values ‘John Doe’ and ‘Joe Public’, while the Date_of_Birth column contains values ’19650112′ and ’19680415′ then when was John born, and when was Joe? The answer is that that the position of the value in the column indicates to which row it belongs. So row 1 consists of the first value in Name column and first value in Date_of_Birth, while row 2 is second value in each column and so on.

By physically separating the values from individual columns into their own pages the engine can read only the columns needed. This reduces the IO for queries that:

read only a small subset of columns from all the columns of the table (no SELECT *)
read many rows (scans and range scans)

Columnar storage reads from disk only the data for the columns referenced by the query

Not surprisingly this type of queries is the typical BI query pattern. Note that a column qualifies as ‘read’ not only if is projected in the result set, but also if is referenced in the WHERE clause, or in a join, or anywhere else in the query. Typical BI queries aggregates fact values over large ranges defined by some dimension subsets (eg. “sum of sales in the NW region in Sep.- Nov. period”) and thus they tend to create exactly the queries that the columnar storage loves. The fact that BI workloads use a pattern of queries that references only few columns but many rows, combined with the ability of the columnar storage to read from disk only the columns actually referenced by the query is the first and foremost advantage of the column store technology.

An OLTP workload by contrast tends to read entire rows (all columns) and only one or very few rows at a time. Columnar storage loses its appeal in OLTP workloads and in fact the columnar storage can be significantly slower than row oriented storage for OLTP workloads.

Compression

Because column oriented storage groups values from the same column together there is a new beneficial side effect: data becomes more compressible. Compression can be deployed on row-oriented storage too, see Page Compression Implementation, but with column oriented storage the data is more homogenous, as it contains only values from a single column. Since compression ratio is subject to the data entropy (how homogenous the data is) it follows that columnar storage format is more compressible than the same data represented in row oriented storage format.

Columnstores target large datasets for which compression yields most benefits

Since the columnstore format is targeted explicitly at BI workloads and large datasets, it is justifiable to invest significantly more development effort into enhancing compression benefits: deploy several alternative compression algorithms for the engine to choose from, leverage operations directly in the compressed format w/o decompressing the data first. Row oriented storage have to strike a balance between compression benefits and its runtime cost in consideration of the typical OLTP workload.

Columnstores embrace compression fully and go to great lengths into achieving high compression ratio because in the expected BI workload the savings in IO from better compression more than offset the runtime CPU loss. The individual compression techniques vary between various columnar storage implementation, but your can make a safe bet that some common techniques are used by most implementations:

In addition domain specific compression is used, leveraging the knowledge of the types and values being stored. For example Wikipedia mentions the benefits of sorting the data in order to yield better compression:

To improve compression, sorting rows can also help. For example, using bitmap indexes, sorting can improve compression by an order of magnitude.[6] To maximize the compression benefits of the lexicographical order with respect to run-length encoding, it is best to use low-cardinality columns as the first sort keys.

Note that I am not saying to apply the advice from above mentioned Wikipedia article directly to SQL Server 2012. Specifically columnstore indexes do not require to ‘use low-cardinality keys first’, the order in which you specify keys in SQL Server 2012 columnstore indexes is irrelevant.

Batch Mode Processing

Iterating over hundreds of virtual function calls for each row processes is EXPENSIVE

Reading only the data for the columns referenced by the query and extensive use of compression are techniques used by all columnar storage engines. But SQL Server 2012 also brings something new: a completely new query execution engine optimized for BI queries. To understand why this was necessary and why it has a big performance impact one needs to understand first how query processing works on SQL Server. Paul White has a good series of introductory articles into query processing starting with Query Optimizer Deep Dive – Part 1 and I recommend these articles if you want a deeper introduction into SQL Server query processing. For our purposes it suffices to understand that a query execution consists of iterating over a tree of operators. Query execution is basically a loop that takes the query execution tree top operator and calls a method called GetNextRow until the methods returns end-of-file. This top operator in turn will have child operators on which it calls GetNextRow to produce the data, and these child operators have further child operators and so on and so forth, all the way to the bottom of the tree operators that implement actual physical access to the data. Whether is a simple scan, a seek, a nested loop join, a hash join, a sort, a where filter, any operator behaves the same. Which also implies that this GetNextRow call is a virtual function call. Sometimes these execution trees can be hundreds of operators deep, which means that there will be hundreds of virtual function calls in order to produce one row, followed by again hundreds of calls to produce the next row, then again for the next row and so on. For the analytical workload of OLTP this is not a big deal, since queries are supposed to produce only a few rows and iterate only over small range of the data (a direct seek or a small range scan). But analytical BI queries, even if they produce a small result set containing aggregates, need to scan extremely large sets of data and the cost of traversing deep execution sub-trees to retrieve rows one by one quickly adds up, specially when the calls are all going through a v-table indirection.

Batch mode execution amortizes the cost of calling the operators by requesting many rows in a single call

Enter the new batch mode operators. In batch mode instead of calling a function that returns one row at a time, the query operators call a function that returns many rows at a time: a batch. The benefits of doing so are not obvious for non-programmers, but believe you me, this can result in speed improvements of 2 and even 3 orders of magnitude.

But batch mode execution requires everybody to play the same tune: all operators have to implement the batch mode execution. Row mode operators cannot interact with batch mode operators nor the other way around. For best performance a query plan has to run in batch mode from top to bottom. Since not all operations were implemented in batch mode by the time SQL Server 2012 RTM shipped, sometimes there will be restrictions that prevent batch mode execution. The Query Optimizer can resort to a conversion operator from batch mode to row mode and it can create plans that have sub trees in batch mode even if the upper part of the tree is in row mode. This mixed mode can still yield significant performance improvements, provided that most data processing still occurs in batch mode (eg. aggregation occurs in batch mode and the result of aggregation is consumed in row mode). Eric Hanson has some very useful tricks for how to achieve batch mode processing in some edgier conditions at Ensuring Use of the Fast Batch Mode of Query Execution. Most tricks rely on splitting the query in two, a subquery that does the bulk of the work in batch mode and then project aggregated results into a an outer query that uses some non batch mode friendly operator like OUTER JOIN, NOT IT, IN, EXISTS, Scalar Aggregates or DISTRINCT aggregates.

SQL Server 2012 Columnar Storage

COLUMNSTORE indexes use the same xVelocity technology as PowerPivot

Prior to SQL Server 2012 the columnar storage technology offer from Microsoft was restricted to the BI analytical line of products: Power Pivot and Microsoft SQL Server Analysis Server. Both of these products use xVelocity (formerly known as Vertipac) to create highly compressed in-memory columnar databases. The COLUMNSTORE index of SQL Server 2012 uses the same technology, but adapted to the SQL Server product, specifically to the SQL Server storage and memory model. MOLAP servers store the entire columnar database as a single monolithic file and load it entirely in memory. Such an use pattern would not work along with the complex memory management of the SQL Server buffer pool. Besides, the storage options of SQL Server are capped at the maximum of 2GB size of a BLOB value and that would make for a really small columnar storage database. So COLUMNSTORE indexes instead use the xVelocity technology on smaller shards of the data called segments:

A segment represents a group of consecutive rows in the columnstore index. For example segment 1 will have rows from 0 to row number 1 million, segment 2 will have rows from 1000001 to 2000000, segment 3 from 2000001 and so on. Each column will have an entry for each segment in sys.column_store_segments. Note that at the time of writing this the MSDN entry says “Contains a row for each column in a columnstore index” which is incorrect, it should say “Contains a row for each column in each segment in a columnstore index”. I have talked before about how columnstore indexes store data in the LOB allocation unit of the table, see How to use columnstore indexes in SQL Server. Each column segment will be a BLOB value in this allocation unit. In effect you can think about sys.column_store_segments as having a VARINARY(MAX) column containing the actual segment data, with the amendment that the storage of this VARBINARY(MAX) column comes from the columnstore index LOB allocation unit and not from the sys.column_store_segments own LOB allocation unit. So it should be clear now what is the main limitation in a column segment: it cannot exceed 2GB in size, since this is the maximum size of a VARBINARY(MAX) column value. A column segment is uniformly encoded: for example if the column segment uses a dictionary encoding then all values in the segment are encoded using a dictionary encoding representation.

Dictionaries are always used to encode strings and may be used for non-string columns that have few distinct values

Besides column segments a columnstore consists of another data storage element: dictionaries. Dictionaries are widely used in columnar storage as a means to efficiently encode large data types, like strings. The values stores in the column segments will be just entry numbers in the dictionary, and the actual values are stored in the dictionary. This technique can yield very good compression for repeated values, but yields bad results if the values are all distinct (the required storage actually increases). This is what makes large columns (strings) with distinct values very poor candidates for columnstore indexes. Columnstore indexes contain separate dictionaries for each column and string columns contain two types of dictionaries:

Primary Dictionary: This is an global dictionary used by all segments of a column.
Secondary Dictionary: This is an overflow dictionary for entries that did not fit in the primary dictionaries. It can be shared by several segments of a column: the relation between dictionaries and column segments is one-to-many.

The image above illustrates the use of primary and secondary dictionaries. All segments of the column reference the primary dictionary. Segments 1,2 and 3 use the secondary dictionary with ID 1. Segment 4 does not need a secondary dictionary and segment 5 uses the secondary dictionary with ID 2.

Information about the dictionaries used by a columnstore can be found in sys.column_store_dictionaries catalog view. Again, at the time of writing the MSDN explanation on it is wrong, it should read “Contains a row for each dictionary in an xVelocity memory optimized columnstore index”. Information about which dictionary is used by each segment is available as the primary_dictionary_id and secondary_dictionary_id columns in sys.column_store_segments. Remember that:

Not all columns use dictionaries.
Non-string columns may use a primary dictionary.
A string column will always have a primary dictionary and some segments may use a secondary dictionary.

The storage of the dictionaries is very similar to the storage of the column segments: think of it as a VARBINARY(MAX) column in sys.column_store_dictionaries with the actual physical storage provided by the columnstore LOB allocation unit. The same 2Gb size limit limitation that applies to column segments applies to dictionaries.

Segment Elimination

The min and max value of the data in the segment is stored in metadata, which enables the query processing to skip entire segments

The sys.column_store_segments catalog view shows the columns min_data_id and max_data_id which are metadata about the min and max values stored in the column segment. The values in the catalog view cannot be interpreted directly because they have to be decoded based on the column segment encoding type and perhaps consider the dictionary used. Knowing that a column segment contains only values between the min and the max, query processing can decide to skip an entire segment. Since the min and the max are stored in the metadata it is not necessary to even load the segment from disk, saving both IO and processing CPU. Segment elimination thus works much like partition elimination works in classic row stores (BTrees and Heaps), but the columnstore segment elimination is actually better:

Columnstore segments min and max values are stored in metadata for every column. Partition elimination works only on predicates that reference the partitioning column, but segment elimination works on any predicates that reference any column.
Columnstore segments min and max values are from the actual data in the segment. Partition elimination works not on actual data values but on partition definition boundaries, which may be much larger than the actual values present in the partition. Thus segment elimination kicks in more often.
Segments are much smaller than partitions, so simple probabilities tell us that more small segments will be eliminated where a partition would had to scan the entire partition, even when much of the data does not qualify.

Where segment elimination comes to shine is time series. The nature of time series implies that segments will tend to have one column (the date column) with very narrow range of values, and each segment will have a different range of values: segment one will have sales from Jan. 1st, segment 2 will have from Jan. 1st and 2nd, segment 3 from Jan. 2nd and 3rd, segment 4 from Jan. 3rd only and so on. So queries that predicate a time range (which are the overwhelming type of queries on time series) will benefit massively from segment elimination, reading and scanning only a small subset of the data. But be aware that in order to get this benefit the columnstore index must be built in a way that results in the data being ordered by the time column. Remember that columnstore indexes have no key. The gritty details are described in Ensuring Your Data is Sorted or Nearly Sorted by Date to Benefit from Date Range Elimination. For more complex scenarios that involve multiple dimension you should consider aggregating the values into complex values, see Multi-Dimensional Clustering to Maximize the Benefit of Segment Elimination. As a side node I find interesting that similar technique of aggregating values into complex keys applies to such a different technology as SimpleDB, see SimpleDB Essentials for High Performance Users.

Of course if you have a partitioned columnstore index then the query can leverage both partition elimination and segment elimination.

How to use Columnstore Indexes

Columnstore indexes support only scan operations

If you remember one thing from this article is this: a columnstore index is not an index. In the traditional row storage an index is a b-tree structure which supports seek and range scan operations. Columnstores are much more like the row store heaps and supports only end-to-end scans. They do not have a key and do not offer any order of rows. Yet, on large data sets, they can offer exceptional performance for analytical workloads due to the factors mentioned above: read only for the columns referenced by the query, high compression, segment elimination and fast batch mode processing. I’ve seen dramatic examples of queries that on highly optimized row storage could not run below 17 seconds and a columnstore could run the same query in under 18 milliseconds! Performance improvements of factors of higher 10s and even 100s times are the norm you should be looking for with columnstores.

Add all columns to the columnstore index

Comparing with traditional row store indexing strategies, the columnstore indexing rules are straight forward: pick a large table that is usually the fact table in a star schema data warehouse and add a columnstore index that includes every column. Without key order, ASCENDING or DESCENDING clauses, fill factors, filtered indexes, INCLUDE columns or any other of the whistles and bells of row store indexes, the decision is much simpler. Consider the columnstore index as an alternative to the base table.

Of course the biggest columnstore draw back with the SQL Server 2012 release is the inability to update data. Once a columnstore index is added to a table the table becomes read-only. The best way to circumvent the problem is to use the ability to do fast partition switch-in operations, which allow for partitioned columnstore indexes to be updated. See How to update a table with a columnstore index.

1000 Consecutive days on StackOverflow

remus — Sun, 18 Mar 2012 00:29:42 +0000

Few days ago I noticed that my StackOverflow profile shows 995 consecutive visited days. So naturally I started thinking about what does it mean to come back every day for a thousand days in a row. Looking back at the post I wrote almost 3 years ago to the day: stackoverflow.com: how to execute well on a good idea I can say that not much has changed: StackOverflow (and now the entire StackExchange network) is first and foremost a great community. The technical execution and the social nurturing of the site makes for a low friction environment that invites and rewards contribution, and it keeps getting better and better.

rusanu.com

SQL Server Clustered Columnstore Indexes at TechEd 2013

Deltastores

Tuple Mover

Delete Bitmaps

Bulk Insert

Trickle Insert

Improved Index Build

Sampling Support

Bookmark Support

Schema modification support

Support for short strings

Mixed Execution Mode

Hash Join and Bitmap filters improvements

Archival support

Registry bloat after SQL Server 2012 SP1 installation

How to enable Selective XML indexes in SQL Server 2012 SP1

How to enable and disable a queue using SMO

SQL Server backup to URL

Case Sensitive collation sort order

Handling exceptions that occur during the RECEIVE statement in activated procedures

Error 9617 The service queue “…” is currently disabled

Why loop in the first place?

Deploying blog samples in production

How to shrink the SQL Server log

DBCC LOGINFO

Tail of the log

Further reading

Inside the SQL Server 2012 Columnstore Indexes

Column Oriented Storage

Compression

Batch Mode Processing

SQL Server 2012 Columnar Storage

Segment Elimination

How to use Columnstore Indexes

Recommended Reading

1000 Consecutive days on StackOverflow