Online non-NULL with values column add in SQL Server 2012
July 13th, 2011Prior to SQL Server 2012 when you add a new non-NULLable column with default values to an existing table a size-of data operation occurs: every row in the table is updated to add the default value of the new column. For small tables this is insignificant, but for large tables this can be so problematic as to completely prohibit the operation. But starting with SQL Server 2012 the operation is, in most cases, instantaneous: only the table metadata is changed, no rows are being updated.
Lets look at a simple example, we’ll create a table with some rows and then add a non-NULL column with default values. First create and populate the table:
create table test (
id int not null identity(1,1) primary key,
someValue int not null);
go
set nocount on;
insert into test (someValue) values (rand()*1000);
go 1000
We can inspect the physical structure of the table’s records using DBCC PAGE. First lets find the page that contains the first record of the table:
select %%physloc%%, * from test where id = 1;
In my case this returned 0xD900000001000000, which means slot 0 on page 0xD9 (aka. 217) of file 1, and my test database has the DB_ID 6. Hence the parameters to DBCC PAGE
dbcc traceon (3604,-1)
dbcc page(6,1,217,3)
Page @0x0000000170D5E000
m_pageId = (1:217) m_headerVersion = 1 m_type = 1
m_typeFlagBits = 0x0 m_level = 0 m_flagBits = 0x200
m_objId (AllocUnitId.idObj) = 84 m_indexId (AllocUnitId.idInd) = 256
Metadata: AllocUnitId = 72057594043432960
Metadata: PartitionId = 72057594039042048 Metadata: IndexId = 1
Metadata: ObjectId = 245575913 m_prevPage = (0:0) m_nextPage = (1:220)
pminlen = 12 m_slotCnt = 476 m_freeCnt = 4
m_freeData = 7236 m_reservedCnt = 0 m_lsn = (30:71:25)
m_xactReserved = 0 m_xdesId = (0:0) m_ghostRecCnt = 0
m_tornBits = 2135435720 DB Frag ID = 1
Allocation Status
GAM (1:2) = ALLOCATED SGAM (1:3) = ALLOCATED
PFS (1:1) = 0x60 MIXED_EXT ALLOCATED 0_PCT_FULL DIFF (1:6) = CHANGED
ML (1:7) = NOT MIN_LOGGED
Slot 0 Offset 0x60 Length 15
Record Type = PRIMARY_RECORD Record Attributes = NULL_BITMAP Record Size = 15
Memory Dump @0x000000000AEBA060
0000000000000000: 10000c00 01000000 34020000 020000†††††††††††††........4......
Slot 0 Column 1 Offset 0x4 Length 4 Length (physical) 4
id = 1
Slot 0 Column 2 Offset 0x8 Length 4 Length (physical) 4
someValue = 564
Note the last LSN that updated the page (30:71:25) and the size of the record in slot 0 (15 bytes). Now lets add a non-NULL column with default values:
alter table test add otherValue int not null default 42 with values;
We can select from the table and see that the table was changed and the rows have value 42 for the newly added column:
select top(2) * from test;
id someValue otherValue
----------- ----------- -----------
1 564 42
2 387 42
Yet if we inspect again the page, we can see that is unchanged:
dbcc traceon (3604,-1)
dbcc page(6,1,217,3)
Page @0x0000000170D5E000
m_pageId = (1:217) m_headerVersion = 1 m_type = 1
m_typeFlagBits = 0x0 m_level = 0 m_flagBits = 0x200
m_objId (AllocUnitId.idObj) = 84 m_indexId (AllocUnitId.idInd) = 256
Metadata: AllocUnitId = 72057594043432960
Metadata: PartitionId = 72057594039042048 Metadata: IndexId = 1
Metadata: ObjectId = 245575913 m_prevPage = (0:0) m_nextPage = (1:220)
pminlen = 12 m_slotCnt = 476 m_freeCnt = 4
m_freeData = 7236 m_reservedCnt = 0 m_lsn = (30:71:25)
m_xactReserved = 0 m_xdesId = (0:0) m_ghostRecCnt = 0
m_tornBits = 2135435720 DB Frag ID = 1
Allocation Status
GAM (1:2) = ALLOCATED SGAM (1:3) = ALLOCATED
PFS (1:1) = 0x60 MIXED_EXT ALLOCATED 0_PCT_FULL DIFF (1:6) = CHANGED
ML (1:7) = NOT MIN_LOGGED
Slot 0 Offset 0x60 Length 15
Record Type = PRIMARY_RECORD Record Attributes = NULL_BITMAP Record Size = 15
Memory Dump @0x000000000E83A060
0000000000000000: 10000c00 01000000 34020000 020000†††††††††††††........4......
Slot 0 Column 1 Offset 0x4 Length 4 Length (physical) 4
id = 1
Slot 0 Column 2 Offset 0x8 Length 4 Length (physical) 4
someValue = 564
Slot 0 Column 3 Offset 0x0 Length 4 Length (physical) 0
otherValue = 42
The page header is unchanged, the last LSN is still (30:71:25), proof that the page was not modified, and the physical record is unchanged and has the same size as before. Yet DBCC shows a Column 3 and its value 42! If you pay attention you’ll notice that the Column 3 though has an Offset 0x0 and a physical length of 0. Column 3 is somehow materialized out of thin air, as it does not physically exists in the record on this page. The ‘magic’ is that the table metadata has changed and it now contains a column with a ‘default’ value:
select pc.* from sys.system_internals_partitions p
join sys.system_internals_partition_columns pc on p.partition_id = pc.partition_id
where p.object_id = object_id('test');
Notice that sys.system_internals_partition_columns now has two new columns that are SQL Server 2012 specific: has_default and default_value. The column we added to the test table (the third row in the image above) has a default with value 42. This is how SQL Server 2012 knows how to show a value for Column 3 for this record, even though is physically missing on the page. With this ‘magic’ in place the ALTER TABLE will no longer have to update every row in the table and the operation is fast, metadata-only, no matter the number of rows in the table. This new behavior occurs automatically, no special syntax or setting is required, the engine will simply do the right thing. There is no penalty from having a missing value in a row. The ‘missing’ value can be queried, updated, indexed, exactly as if the update during ALTER TABLE really occurred. There is no measurable performance penalty from having a default value.
What happens when we update a row? The ‘default’ value is pushed into the row, even if the column was not modified. Consider this update:
update test set someValue = 565 where id = 1;
Although we did not touch the otherValue column, the row now was modified and it contains the materialized value:
dbcc page(6,1,217,3)
...
m_freeData = 7240 m_reservedCnt = 0 m_lsn = (31:271:2)
...
Slot 0 Offset 0x1c35 Length 19
Record Type = PRIMARY_RECORD Record Attributes = NULL_BITMAP Record Size = 19
Memory Dump @0x000000000AB8BC35
0000000000000000: 10001000 01000000 35020000 2a000000 030000††††........5...*......
Slot 0 Column 1 Offset 0x4 Length 4 Length (physical) 4
id = 1
Slot 0 Column 2 Offset 0x8 Length 4 Length (physical) 4
someValue = 565
Slot 0 Column 3 Offset 0xc Length 4 Length (physical) 4
otherValue = 42
KeyHashValue = (8194443284a0)
Slot 1 Offset 0x60 Length 15
Record Type = PRIMARY_RECORD Record Attributes = NULL_BITMAP Record Size = 15
Memory Dump @0x000000000AB8A060
0000000000000000: 10000c00 02000000 83010000 020000†††††††††††††..............
Slot 1 Column 1 Offset 0x4 Length 4 Length (physical) 4
id = 2
Slot 1 Column 2 Offset 0x8 Length 4 Length (physical) 4
someValue = 387
Slot 1 Column 3 Offset 0x0 Length 4 Length (physical) 0
otherValue = 42
Notice how the physical record has increased in size (19 bytes vs. 15), the record has the value 42 in it (the hex 2a000000) and the Column 3 now has a real offset and physical size. So the update has trully materialized the default value in the row image. I intentionally copied the output of DBCC PAGE for the next slot in the page, to show that the record with id=2 was unaffected, it continues to have a smaller size of 15 bytes and Column 3 has no physical length.
Default value vs. Default constraint
Is worth saying that the new SQL Server 2012 default column value is not the same as the default value constraint. The default value is captured when the ALTER TABLE statement is run and can never change. Only rows existing in the table at the time of running ALTER TABLE statement will have missing ‘default’ values. By contrast the default constraint can be dropped or modified and new rows inserted after the ALTER TABLE will always have a value present in row for the new column. Any REBUILD operation on the table (or on the clustered index) will materialize all the missing values as the rows are being copied from the old hobt to the new hobt. The new hobt columns (sys.system_internals_partition_columns) will loose the has_default and default_value attributes, in effect loosing any trace that this column was added online. A default constraint by contrast will be preserved as a table is rebuilt.
Restrictions
Not all data types and default values can be added online. BLOB values like varchar(max), nvarchar(max), varbinary(max) and XML cannot be added online (and frankly I see no valid data model that has a non-NULL BLOB with a default…). Types that cannot be converted to sql_variant cannot be added online, like hierarchy_id, geometry and geography or user CLR based UDTs. Default expressions that require a different value for each row, like NEWID or NEWSEQUENTIALID cannot be added online (the default expression has to be a runtime constant, not to be confused with a deterministic expression, see Conor vs. Runtime Constant Functions for more details). In the case when the newly added column increases the maximum possible row size over the 8060 bytes limit the column cannot be added online. And is an Enterprise Edition only feature. For all the cases above the behavior will revert to adding the column ‘offline’, by updating every row in the table during the ALTER TABLE statement, creating a size-of-data update. When such a situation occurs a new XEvent is fired, which contains the reason why a size-of-data update occurred: alter_table_update_data.