PCIE SSD Update – Optimizing SQL Backup Performance

In my last blog entry (http://blogs.msdn.com/b/microsoftbob/archive/2010/11/06/the-potential-of-ssd-for-millions-of-iops-on-increasingly-larger-databases.aspx), I shared the throughput numbers backing up from 2 Fusion-IO devices to a 4-spindle 15K SAS Raid-10 device.  I also shared the numbers of backing up to NUL device, and was surprised they weren’t better especially given the advertised throughput of 700 MB/Sec for a standard SLC Fusion-IO IoDrive. 

I did some searching and found out that there are several parameters that can have a profound impact on backup speed, especially related to SSDs.  The following article was especially helpful:

http://henkvandervalk.com/how-to-increase-sql-database-full-backup-speed-using-compression-and-solid-state-disks

Also, there is an article by the SQLCAT team that discusses how to tune backups:

http://sqlcat.com/technicalnotes/archive/2009/02/16/tuning-backup-compression-part-2.aspx

Armed with this, I did some trial and error to find optimal settings and retested both the backup to disk and the backup to NUL.  It turns out that for my scenarios using 6 backup files on the target media (SQL will stripe the backup to multiple files) along with a buffer count of 64 and a Block Size of 65536 (defaults to just 512) almost doubled the backup speed.  On the new test, the backup to disk completed in less than 2 minutes with a throughput of 618 MB/Sec contrasted with the prior 361 MB/Sec.  Changing the MaxTransferSize did not have measurable impact and made things worse when too large.

The Backup to Null device was an interesting experience.  The first time around I left the Compression option on, and achieved 765 GB/Sec with the changed parameters, but noticed that the CPU was bound for most of the backup, pegged at close to 98%.  Normally, Backup Compression improves throughput, but in this case the processors were not able to keep up.  Based on that, I did a Backup to NUL without compression and obtained a whopping 1.42 GB/Sec which matches to what the 2 IoDrives are capable of in tandem.  Here’s a screen snapshot for the skeptics:

image

BACKUP DATABASE [tp_v5] TO  — DISK = — ‘NUL:’
disk=’nul:’

WITH NOFORMAT, NOINIT,  NAME = N’tp_v5-Full Database Backup’, SKIP, NOREWIND, NOUNLOAD, NO_COMPRESSION, STATS = 10
— Magic:
,BUFFERCOUNT = 64
,BLOCKSIZE = 65536

10 percent processed.
20 percent processed.
30 percent processed.
40 percent processed.
50 percent processed.
60 percent processed.
70 percent processed.
80 percent processed.
90 percent processed.
Processed 41488 pages for database ‘tp_v5’, file ‘TradingOptimizer_v2’ on file 1.
Processed 160288 pages for database ‘tp_v5’, file ‘History09’ on file 1.
Processed 67584 pages for database ‘tp_v5’, file ‘History01’ on file 1.
Processed 63872 pages for database ‘tp_v5’, file ‘History02’ on file 1.
Processed 68424 pages for database ‘tp_v5’, file ‘History03’ on file 1.
Processed 60408 pages for database ‘tp_v5’, file ‘History04’ on file 1.
Processed 69936 pages for database ‘tp_v5’, file ‘History05’ on file 1.
Processed 85040 pages for database ‘tp_v5’, file ‘History06’ on file 1.
Processed 103368 pages for database ‘tp_v5’, file ‘History07’ on file 1.
Processed 132088 pages for database ‘tp_v5’, file ‘History08’ on file 1.
Processed 1225312 pages for database ‘tp_v5’, file ‘HistoryData2a’ on file 1.
Processed 580112 pages for database ‘tp_v5’, file ‘LoadData2’ on file 1.
Processed 408120 pages for database ‘tp_v5’, file ‘LoadData01’ on file 1.
Processed 288784 pages for database ‘tp_v5’, file ‘MiscData2’ on file 1.
Processed 233408 pages for database ‘tp_v5’, file ‘MiscData3’ on file 1.
Processed 154032 pages for database ‘tp_v5’, file ‘History10’ on file 1.
Processed 64 pages for database ‘tp_v5’, file ‘Load00’ on file 1.
Processed 66824 pages for database ‘tp_v5’, file ‘Load01’ on file 1.
Processed 173824 pages for database ‘tp_v5’, file ‘History09b’ on file 1.
Processed 188768 pages for database ‘tp_v5’, file ‘History09c’ on file 1.
Processed 205832 pages for database ‘tp_v5’, file ‘History09d’ on file 1.
Processed 195912 pages for database ‘tp_v5’, file ‘History10a’ on file 1.
Processed 209480 pages for database ‘tp_v5’, file ‘History10b’ on file 1.
Processed 206416 pages for database ‘tp_v5’, file ‘History10c’ on file 1.
Processed 100824 pages for database ‘tp_v5’, file ‘History10d’ on file 1.
Processed 160 pages for database ‘tp_v5’, file ‘Archive01’ on file 1.
Processed 65920 pages for database ‘tp_v5’, file ‘Load02’ on file 1.
Processed 77208 pages for database ‘tp_v5’, file ‘Load03’ on file 1.
Processed 73488 pages for database ‘tp_v5’, file ‘Load04’ on file 1.
Processed 72728 pages for database ‘tp_v5’, file ‘Load05’ on file 1.
Processed 80248 pages for database ‘tp_v5’, file ‘Load06’ on file 1.
Processed 78896 pages for database ‘tp_v5’, file ‘Load07’ on file 1.
Processed 80944 pages for database ‘tp_v5’, file ‘Load08’ on file 1.
Processed 90816 pages for database ‘tp_v5’, file ‘Load09’ on file 1.
Processed 95600 pages for database ‘tp_v5’, file ‘Load10’ on file 1.
Processed 85848 pages for database ‘tp_v5’, file ‘Load11’ on file 1.
Processed 96328 pages for database ‘tp_v5’, file ‘Load12’ on file 1.
Processed 82880 pages for database ‘tp_v5’, file ‘Load13’ on file 1.
Processed 72600 pages for database ‘tp_v5’, file ‘Load14’ on file 1.
Processed 89216 pages for database ‘tp_v5’, file ‘Load15’ on file 1.
Processed 83320 pages for database ‘tp_v5’, file ‘Load16’ on file 1.
Processed 83472 pages for database ‘tp_v5’, file ‘Load17’ on file 1.
Processed 85880 pages for database ‘tp_v5’, file ‘Load18’ on file 1.
Processed 238520 pages for database ‘tp_v5’, file ‘Load19’ on file 1.
Processed 224080 pages for database ‘tp_v5’, file ‘Load20’ on file 1.
Processed 169024 pages for database ‘tp_v5’, file ‘Load21’ on file 1.
Processed 176960 pages for database ‘tp_v5’, file ‘Load22’ on file 1.
Processed 80568 pages for database ‘tp_v5’, file ‘Load23’ on file 1.
Processed 16 pages for database ‘tp_v5’, file ‘Load24’ on file 1.
Processed 16 pages for database ‘tp_v5’, file ‘Sim42’ on file 1.
Processed 340576 pages for database ‘tp_v5’, file ‘Intraday01’ on file 1.
Processed 327112 pages for database ‘tp_v5’, file ‘Intraday’ on file 1.
100 percent processed.
Processed 1 pages for database ‘tp_v5’, file ‘TradingOptimizer_v2_log’ on file 1.
BACKUP DATABASE successfully processed 8042633 pages in 43.967 seconds (1429.096 MB/sec).

Here is what the backup command looked like and the final result for actually going to the Raid-10 devices

BACKUP DATABASE [tp_v5] TO  — DISK = — ‘NUL:’
disk=’h:\sqlbackup\folibackuptest1.bak’,
disk=’h:\sqlbackup\folibackuptest2.bak’,
disk=’h:\sqlbackup\folibackuptest3.bak’,
disk=’h:\sqlbackup\folibackuptest4.bak’,
disk=’h:\sqlbackup\folibackuptest5.bak’,
disk=’h:\sqlbackup\folibackuptest6.bak’

WITH NOFORMAT, NOINIT,  NAME = N’tp_v5-Full Database Backup’, SKIP, NOREWIND, NOUNLOAD, COMPRESSION, STATS = 10
— Magic:
,BUFFERCOUNT = 64
,BLOCKSIZE = 65536

And the final result:

BACKUP DATABASE successfully processed 8042633 pages in 101.513 seconds (618.965 MB/sec).

I was actually surprised to see that 4 146 SAS Drives could support this write throughput, but this is probably because the throughput is measured relative to the backup compression.  I.e., the source data was written at 618 MB/Sec, but the target throughput on the disk did not need to be that high as only about 25GB actually had to be written to disk for the backup of the 150GB database.  So, the actual combined write speed on the SAS drives was probably closer to about 225 MB/SEC

So, this makes me fairly certain that 4 Fusion-IO 1.28 TB MLC duo cards could read a SQL Database at a total of 6 GB/Sec and then write at a speed 6 GB/Sec to another server with 4 SLC duo cards (Backup compression typically shrinks by a factor of at least 5 so even a 5 TB database should fit on 4 320GB cards.  In that case, a Infiniband Network of 40Gb between the 2 servers is actually the bottleneck, as that probably won’t allow more than 5GB/Sec.

All this testing and blogging is getting me way behind on my day job and my research work, so you may not hear from me again for a while.  Hopefully you’ve found it useful.

I plan to create another blog for my research work (probably on Microsoft Live) related to my PhD and include that link as some of you have expressed interest in this. 

Also, some of you have wondered why I am only posting info related to the Fusion-io SSDs and not other devices.  The reason is that these are the only ones that I own and have personal experience with other than some Intel 2.5 SSD drives.   If another vendor wants to send me demo versions of their high-end products, I’ll be happy to try them out and let you know what I find (hint, hint…) Smile

Thanks for reading.

Posted in Uncategorized | Leave a comment

The potential of SSD for millions of IOPS on increasingly larger databases

NOTE: There is an update to this article at http://blogs.msdn.com/b/microsoftbob/archive/2010/11/08/pcie-ssd-update-optimizing-sql-backup-performance.aspx regarding the throughput I was able to achieve on my own configuration with the Fusion-io backup process.  Through some parameter changes on the backup command, was able to double the throughput of the backup to Raid-10 and more than triple speed to the Null device.

It seems that the computer industry is finally coming to the recognition that someday electro-mechanical disk storage will be obsolete. One of the obstacles to this is that solid-state devices up until recently were very limited in terms of capacity. So even though the devices are known to be many times faster than traditional storage, particularly in random access, many have thought that the limitation of storage space would prevent widespread adoption for large databases without prohibitive costs.

However we are now seeing SSD devices beginning to rival traditional platter-based storage devices in capacity while maintaining ever-improving performance metrics. A key enabler of this has been the refinement of multilevel cell technology to increase reliability, decreases in the NAND flash chip footprint, and addition of software based approaches to optimize both longevity and performance. The initial wave of enterprise-class SSDs were based on SLC, with MLC deemed too unreliable for enterprise applications. However, MLC technology is improving and many believe SLC technology will be obsoleted due to reduced costs of MLC per GB of storage and comparable performance.

A major player in the MLC market as well as SLC is Fusion-io. Fusion-io is now producing a 1.2 TB PCIE SSD IoDrive Duo based on MLC with 1.1 GB/s sequential write throughput (1.5 GB/sec on the 320 GB SLC version) and 1.5 GB/sec read throughput, a read latency of 2 microseconds, a write latency of 30 microseconds (26 microseconds on 320 GB SLC version), 185,000 Read IOPS (261,000 on 320GB SLC version), and 278,000 Write IOPS.  (http://www.fusionio.com/products/iodriveduo/).

Before getting too far into this and just in case somebody doesn’t make it to the bottom, I need to issue the following disclaimer:

DISCLAIMER: I am a Microsoft employee and not a representative of Fusion-io.  The numbers relating to performance, price, and all other metrics related to the product are estimates only.  I do not guarantee the accuracy of any of the numbers or charts or statements regarding the Fusion-io products.  My analysis should not be considered an endorsement of Fusion-io or it’s products and is provided for informational purposes only.

I am not a hardware guru, but hopefully the below estimates are hopefully close enough to give a rough idea of trade-offs between using PCIE SSD and traditional drives.  Manufacturer sites and testing sites show significant discrepancies on enterprise-class 15K SAS drives.  To be fair to the SAS drives, I’ve taken what appear to be the best numbers for the best drives but only for the 73GB size – that’s because it takes a lot of spindles to reach parity with a IoDrive so no sense in using large drives.  For Fusion-io, I am using their data sheet, which has been verified by some other sites to be realistic.  Here’s an example of a test that included HP using 5 IO Duos to achieve 1 Million IOPS.  http://www.eweek.com/c/a/Data-Storage/Fusionio-HP-Team-Up-for-1M-IOPs-8GBs-Throughput-Ben-362772/

Here’s another example of a test of a 320GB IoDrive: http://www.techeye.net/hardware/fusion-io-solid-state-drive-has-a-whole-lot-to-give

The big disadvantage of platter-based storage is that even the fastest enterprise-class SAS drives don’t provide more than 250 IOPS under the best conditions.  This means it would require about 900 SAS drives to match the averaged read/write IOPS of a single Fusion-io Duo drive. On the other hand, it only takes about 15 SAS 73GB SAS drives (sequential throughput increases with drive density, 73GB drives typically around 100 MB/sec) to match the sequential read/write throughput of a duo.

The other significant advantage of the Fusion SSD Device is that it is a PCIE 2.0 card which means that for the Intel platform, processors can leverage the storage device at the maximum Quick Path Interconnect (QPI) speed which is 6.4 for higher-class Intel XEON processors.  I’m sure AMD also has comparable numbers, but I am not proficient with their hardware.  The PCIE technology also gets around the SATA/SAS bus bottleneck which is 600MB/sec even for SATA II.  In order to get around that limitation, you have the added expense of a SAN with HBA, Fibrechannel, ISCSI, and even those technologies max out around 4GB/sec.

Below are charts of 1 Fusion-IO MLC 1.2 TB IoDrive Duo to various numbers of SAS 73K drives depending on the goal.  To simplify the chart, we average the throughput and IOPS between 50% read/write as read vs. write for these items are not dramatically different.  I could not find the price of the Fusion-io but guessing at around 20k based on the fact that a 640GB version costs nearly 12K.  Hopefully I get some feedback if I’m way off and will update this post.  I’m using a $300.00 price for a Dell 73GB 2.5 SAS drive.  Using 2.5 form factor versus 3.5 because 1U servers typically don’t support 3.5 form factor and the trend is toward the smaller form factor, especially considering the infrastructure savings for energy/space.  As we see the enormous numbers of drives required to match a single Fusion-io device realistically, this becomes more apparent.

Let’s start with the least impressive application of the IoDrive for maximizing sequential access throughput and storage.  If all you are after is sequential throughput and/or storage, going the Fusion route will cost you about 4 times as much, so you probably don’t want to get one of these devices just for that.  As you’ll see in the next chart, the big advantage of the Fusion-IO concerns random access.

clip_image002

The second chart shows the IOPS comparison if you wanted to get the same number of average IOPS from the SAS drives as that of the Fusion.  This is sort of a ridiculous comparison, since 900 drives is going to mean a significant infrastructure investment in terms of SAN and HBAs.  I ball-parked $1,000 for every 16 drives to cover controllers and storage arrays.  If you really want IOPS, traditional storage is going to be significantly more expensive from both an initial cost and an operational point of view.  The estimated initial price to achieve the IOPS of the Fusion-IO  IoDrive duo using traditional storage is over 16 times higher!

clip_image002[4]

The final chart shows the comparison of throughput, storage, read latency, write latency and IOPS spending the same amount on both storage technologies.

Here you can see the significant advantages at the same price point of the the IoDrive duo drive for IOPs and read latency, near equivalence for write latency, and advantages to the SAS drives for size and throughput.

clip_image002[6]

So, it really comes down to what type of access you are doing (sequential versus random), whether you need fault tolerance, and whether you need lots of storage or lots of speed.  My approach been to use the Fusion devices for my database files along with TempDB and use Raid-10 array of traditional drives for the log.  However, it appears that the storage capacities of flash is increasing at a faster rate than that of platter-based storage.  Therefore, it does appear that platter-based storage will eventually go the way of the dinosaur within a few years.

I am currently running a pair of Fusion-io SLC Cards in a Dell R610 along with 6 SAS drives on one of my servers at home as part of my non-Microsoft outside research work. Two of the SAS drives are mirrored for the OS and the other Four are in a Raid-10 configuration. The fusion drives are software-partitioned into a couple of drives (playing with software-based Raid-1 as well as Raid-0). The machine is a Dell R610 with dual Xeon Quad-core 2.66 processors 48GB of RAM with 24 GB reserved for SQL Server and the remaining memory utilized for several virtual machines. It contains among other things a database used to store stock histories including intraday data and simulations. Normally I backup the database across a 1GB/sec network to another server. I thought it would be interesting to see the statistics for backing up the database from the SSD devices to the 4 x SAS Raid-10 volume.

Below is a quick test of a database backup. Although the database is only around 150GB with compression, it is still much larger than the 24GB allocated to SQL available for caching. I managed to get 361MB per second on a R610 dual-Xeon quad-core with 48GB RAM. The limiting factor being the write speed of the RAID-10 receiving device which only had 4 spindles.

BACKUP DATABASE [tp_v5] TO DISK = N’H:\sqlbackup\folibackuptest.bak’ WITH NOFORMAT, NOINIT,  NAME = N’tp_v5-Full Database Backup’, SKIP, NOREWIND, NOUNLOAD, COMPRESSION, STATS = 10

GO

10 percent processed.
20 percent processed.
30 percent processed.
40 percent processed.
50 percent processed.
60 percent processed.
70 percent processed.
80 percent processed.
90 percent processed.
Processed 40944 pages for database ‘tp_v5’, file ‘TradingOptimizer_v2’ on file 1.
Processed 159088 pages for database ‘tp_v5’, file ‘History09’ on file 1.
Processed 66336 pages for database ‘tp_v5’, file ‘History01’ on file 1.
Processed 62792 pages for database ‘tp_v5’, file ‘History02’ on file 1.
Processed 67280 pages for database ‘tp_v5’, file ‘History03’ on file 1.
Processed 60064 pages for database ‘tp_v5’, file ‘History04’ on file 1.
Processed 69616 pages for database ‘tp_v5’, file ‘History05’ on file 1.
Processed 84704 pages for database ‘tp_v5’, file ‘History06’ on file 1.
Processed 102512 pages for database ‘tp_v5’, file ‘History07’ on file 1.
Processed 131456 pages for database ‘tp_v5’, file ‘History08’ on file 1.
Processed 1144248 pages for database ‘tp_v5’, file ‘HistoryData2a’ on file 1.
Processed 567240 pages for database ‘tp_v5’, file ‘LoadData2’ on file 1.
Processed 395288 pages for database ‘tp_v5’, file ‘LoadData01’ on file 1.
Processed 287648 pages for database ‘tp_v5’, file ‘MiscData2’ on file 1.
Processed 232264 pages for database ‘tp_v5’, file ‘MiscData3’ on file 1.
Processed 148624 pages for database ‘tp_v5’, file ‘History10’ on file 1.
Processed 64 pages for database ‘tp_v5’, file ‘Load00’ on file 1.
Processed 66824 pages for database ‘tp_v5’, file ‘Load01’ on file 1.
Processed 167816 pages for database ‘tp_v5’, file ‘History09b’ on file 1.
Processed 182648 pages for database ‘tp_v5’, file ‘History09c’ on file 1.
Processed 199664 pages for database ‘tp_v5’, file ‘History09d’ on file 1.
Processed 190000 pages for database ‘tp_v5’, file ‘History10a’ on file 1.
Processed 203648 pages for database ‘tp_v5’, file ‘History10b’ on file 1.
Processed 199712 pages for database ‘tp_v5’, file ‘History10c’ on file 1.
Processed 84376 pages for database ‘tp_v5’, file ‘History10d’ on file 1.
Processed 160 pages for database ‘tp_v5’, file ‘Archive01’ on file 1.
Processed 65920 pages for database ‘tp_v5’, file ‘Load02’ on file 1.
Processed 77208 pages for database ‘tp_v5’, file ‘Load03’ on file 1.
Processed 73488 pages for database ‘tp_v5’, file ‘Load04’ on file 1.
Processed 72728 pages for database ‘tp_v5’, file ‘Load05’ on file 1.
Processed 80248 pages for database ‘tp_v5’, file ‘Load06’ on file 1.
Processed 78896 pages for database ‘tp_v5’, file ‘Load07’ on file 1.
Processed 80944 pages for database ‘tp_v5’, file ‘Load08’ on file 1.
Processed 90816 pages for database ‘tp_v5’, file ‘Load09’ on file 1.
Processed 95600 pages for database ‘tp_v5’, file ‘Load10’ on file 1.
Processed 85848 pages for database ‘tp_v5’, file ‘Load11’ on file 1.
Processed 96328 pages for database ‘tp_v5’, file ‘Load12’ on file 1.
Processed 82880 pages for database ‘tp_v5’, file ‘Load13’ on file 1.
Processed 72600 pages for database ‘tp_v5’, file ‘Load14’ on file 1.
Processed 89216 pages for database ‘tp_v5’, file ‘Load15’ on file 1.
Processed 83320 pages for database ‘tp_v5’, file ‘Load16’ on file 1.
Processed 83472 pages for database ‘tp_v5’, file ‘Load17’ on file 1.
Processed 85880 pages for database ‘tp_v5’, file ‘Load18’ on file 1.
Processed 238520 pages for database ‘tp_v5’, file ‘Load19’ on file 1.
Processed 224080 pages for database ‘tp_v5’, file ‘Load20’ on file 1.
Processed 169024 pages for database ‘tp_v5’, file ‘Load21’ on file 1.
Processed 176960 pages for database ‘tp_v5’, file ‘Load22’ on file 1.
Processed 29040 pages for database ‘tp_v5’, file ‘Load23’ on file 1.
Processed 16 pages for database ‘tp_v5’, file ‘Load24’ on file 1.
Processed 16 pages for database ‘tp_v5’, file ‘Sim42’ on file 1.
Processed 334904 pages for database ‘tp_v5’, file ‘Intraday01’ on file 1.
Processed 321640 pages for database ‘tp_v5’, file ‘Intraday’ on file 1.
100 percent processed.
Processed 1 pages for database ‘tp_v5’, file ‘TradingOptimizer_v2_log’ on file 1.
BACKUP DATABASE successfully processed 7804609 pages in 169.215 seconds (360.331 MB/sec).

To confirm that the Raid-10 was the bottleneck, I ran a backup to null device. 
BACKUP DATABASE [tp_v5] TO  DISK = ‘NUL:’
—  N’g:\sqlbackup\folibackuptest.bak’
WITH NOFORMAT, NOINIT,  NAME = N’tp_v5-Full Database Backup’, SKIP, NOREWIND, NOUNLOAD, COMPRESSION, STATS = 10
GO

10 percent processed.
… (No sense boring you again with SQL output)
100 percent processed.
Processed 1 pages for database ‘tp_v5’, file ‘TradingOptimizer_v2_log’ on file 1.
BACKUP DATABASE successfully processed 8042633 pages in 148.289 seconds (423.720 MB/sec).

This got the speed to 423 MB/sec.  I was surprised it wasn’t higher, don’t know if this was a quark of the backup to NULL device or not, maybe slowed down because the transaction log is also on the Raid-10 device.  I’m still not sure why it wasn’t faster.  Hopefully I can figure that out.  However, even at that speed, if we project the performance out using 4 Fusion-io IoDrive Duo cards (http://www.fusionio.com/products/iodriveduo/) on a 4U Dell R900 in Raid-0, we should get 6 times the throughput as long as we have an equivalently equipped receiving server also with 4 duos and a fast network (i.e. Infiniband) between.  This would give us over 3 GB/sec backup speed which is in the upper range of highest publicized database backup throughputs publically recorded for a production SQL Server database.

NOTE: See my update at http://blogs.msdn.com/b/microsoftbob/archive/2010/11/08/pcie-ssd-update-optimizing-sql-backup-performance.aspx for latest figures.  Was able to get the backup to disk speed over 618 MB/Sec and backup to Null to 1.42 GB/SEC

One argument against using PCIE-SSD is that it cannot be used in a cluster to store the database because the PCIE-SSD storage cannot be failed over.  However, consider the following excerpt from http://www.storagesearch.com/infiniband.html

Fusion-io SSDs deployed in nuclear safety program

Editor:- June 15, 2010 – today it was announced that Lawrence Livermore National Laboratory has deployed over 100TB of Fusion-io’s dual 320GB enterprise MLC ioMemory modules deployed in ioSAN carrier cards that connect over Infiniband in a testbed project to develop high-performance simulation computing capabilities needed to ensure the safety, security and reliability of the nation’s aging nuclear deterrent without underground testing.
With the addition of Fusion-Powered I/O, the Hyperion testbed will deliver over 40 million IOPS and 320GB/s of bandwidth from just 80 1U appliances (2 racks) compared to 43 racks and more power if implemented in an HDD based system.

So, it does not appear we are that far away from Fusion-IO modules being usable within the context of a SAN.  Also, there is the option of allowing TempDB to be stored on local storage rather than requiring it to be on SAN storage for cluster support.  Storing TempDB on shared-storage does not add any more fault tolerance than on local storage since it is recreated whenever a SQL instance starts up, which is what happens on Failover.  There are a few folks pushing for support of this (including yours truly) as it seems more of an installation setup problem, than an actual operational problem and can have a significant impact for applications that make heavy use of complex queries which can only be resolved in TempDB.  There is a way to get around this and there are some people uding TempDB on local storage including RamDisks or SSDs on SQL Clusters, but since this is not supported, you’ll have to Bing for them and I can’t provide links.

SSD is going to have an impact, especially in PCIE or Infiniband flavors that allow the speed of flash memory to be leveraged.  I have posted earlier about how some queries in SQL Server are dramatically improved by using more random-type plans (seeks versus scans) when using SSD.  Relational database products including SQL Server will need to get “smarter” when it comes to understanding what the underlying disk operating system is capable of in order to correctly optimize query plans.

There are may good articles beyond the ones already referenced here.  Fusion-io has made these devices available to a few select individuals in the SQL community who have blogged about their experiences. Here are a few:

http://www.brentozar.com/archive/2010/03/fusion-io-iodrive-review-fusionio/

http://www.sqlskills.com/BLOGS/PAUL/post/New-hardware-to-play-with-Fusion-io-SSDs.aspx

http://www.mysqlperformanceblog.com/2010/01/11/fusionio-320gb-mlc-benchmarks/

DISCLAIMER: I am a Microsoft employee and not a representative of Fusion-io.  The numbers relating to performance, price, and all other metrics related to the product are estimates only.  I do not guarantee the accuracy of any of the numbers or charts or statements regarding the Fusion-io products.  My analysis should not be considered an endorsement of Fusion-io or it’s products and is provided for informational purposes only.

Posted in Uncategorized | Leave a comment

Back again, Dealing with Storage-aligned Index oddities

I’m back after being on leave for past several weeks in case you wonder why I haven’t posted.  To quote Joy Gresham from the move ShadowLands “I wasn’t dead, I was just in America (fill-in-the-blank)” This was one of my favorite movies that had so many wonderful quotes (http://www.imdb.com/title/tt0108101/quotes and http://www.moviequotes.com/repository.cgi?pg=3&tt=96601)-

Some of you may notice that my academic-research related content has been removed.  I am moving all of that stuff to a non-Microsoft blog since it really isn’t part of my day-to-day work with Microsoft as I am a consultant and not a Microsoft researcher. I’m going to keep this blog focused on my current Microsoft work involving SQL Server tuning and best practices, since after all it is a blog provided to me by Microsoft.

On to my post for today.  I think there is some confusion about how storage-alignment works when it comes to non-clustered indexes (at least I was).  There are some ramifications regarding performance and the types of plans chosen.  There is a significant advantage if you can keep all your indexes associated with a partitioned table storage-aligned.  It greatly simplifies and the automation of archival and sliding window scheme.  It can also help with contention issues.  In one of my applications where I was trying to parallelize my application to use different threads to perform simulation on a partitioning key (pardon the example from my research work…), I ran into deadlocks and all sorts of locking problems even though each thread was granulized to a filter to a different partition and I had set the locking escalation for partition rather than table.   The problem was the non-clustered indexes that were not storage aligned were getting escalated to range locks for sequential scans.

One of the reasons, I was using non-partitioned indexes was to help with performance as there were certain operations that were much slower when the partitioning column was associated with the index, Certain types of queries including join activity can be more optimal when using surrogates in other tables without the added weight of the partitioned column being propagated and included in all the joins.

I must confess to a previous fundamental misunderstanding of how partition alignment really works.  I was thinking that all indexes that were storage-aligned to a partitioned table were actually using the partitioned column as a hidden first segment on the alternate indexes to ensure perfect storage alignment.  However, that is not the case, if an index is created storage aligned, a separate b-tree is created at the index root level that got each partition range (not each partition value, that would be a nightmare).  In other words, if you have 100 different partitions based on a partitioning key and storage align on it (even without specifying the partition column as part of the aligned index), you will end up with 100 individual b-trees – 1 corresponding to each partition – each one will use an include column to store the partition. 

So, this architecture resolves one of my concerns, since seeks should be able to operate optimally even without filtering on the partitioned column, albeit with the overhead of visiting one b-tree per partition.   Storage alignment doesn’t mean alignment at the row level or even page level, but simply at the partition level.  For a great explanation of all this, see

http://blogs.msdn.com/b/hanspo/archive/2009/08/21/inside-of-table-and-index-partitioning-in-microsoft-sql-server.aspx

There’s another good article at http://download.microsoft.com/download/D/B/D/DBDE7972-1EB9-470A-BA18-58849DB3EB3B/PartTableAndIndexStrat.docx

To quote from the Hanspo:

“Partitioning an index transforms the single B-tree of a non-partitioned index into several smaller B-trees, one for each partition defined by the partition function, where each B-tree only consists of index rows where the partitioning column falls into the partition’s range. Therefore, the partitioning column always has to be part of a partitioned index.

Splitting a partitioned index into several independent B-trees obviously also has consequences for index seeks. As long as the partitioning column is one of the search predicates, the optimizer knows which of the trees to choose for the index seek. But if the filter criteria do not include the partitioning column, the optimizer would not know where to start the index seek or scan when using a partitioned index. E.g. if we create the following composite partitioned clustered index “CREATE CLUSTERED INDEX IX_Tab ON Tab (a, b)”, where table Tab is being partitioned on column b, the query “SELECT * FROM Tab WHERE a = @x” cannot be performed as a simple clustered index seek, like in the non-partitioned case, but each of the index trees would have to be traversed separately to find the qualifying records. As this example shows, for partitioned tables the rule, that the table rows are sorted by the clustered index keys, does not necessarily hold true for the whole table, but only for single partitions. Therefore, it’s always good practice to include the partitioning column in your search predicates, even if it does not affect the result set.”

From this, obviously it is more optimal if we partition on a column that is often used to filter, but do we need to redesign data structures to work more optimally with partitioning by replicating partitioned columns onto joined tables or have to use non-storage aligned indexes to get around the performance issues when filtering is not available?  Not necessarily. 

There are some tricks.  There is some behavior that seems baffling as far as query plans chosen for certain operations.  One example, is the case of a select max(non-partitioned index column) from a storage-aligned non-clustered index; Even with just a few partitions, the query optimizer seems to prefer an index scan rather than seeking on each of the b-trees.  The index scan may make sense if there are lots of little partitions, but in the case of a few big partitions it is sub-optimal, particularly when considering the near-zero latency of Solid State storage to support fast random-seeks.

Recently a colleague of mine ran into this issue on our SQL Master blog and since I’ve been dealing with it, decided to try some of the suggestions that came up and play with this some more.  Among the suggestion was the technique of a CTE with a group-by to force seeks within each partition.  An example of this is:

with localmax
as (select MAX(equityintradayid) as maxId from dbo.EquityIntraday group by MarketDate)select MAX(maxId)
from localmax

However in my scenario, I got a less than optimal query plan still doing an index scan and didn’t try making the efforts to clean it up.

The thread is still continuing and not all options have been explored, but at this point, my experience is that getting the max value of a non-clustered index column that is storage-aligned is one of the few things handled better the old fashion-way, rather than relying on the query optimizer. 

Using a simple select max(id) from a table with 79 million rows with a table partitioned on a date field and only 18 active partitions generated by default an index scan and took over 13 seconds of CPU and over 2 seconds elapsed on a dual XEON-Quad-core with 2 Fusion-IO SLC cards.  However, changing this to use a custom function improved the performance by 1000 times.  This highlights the significance of SSD storage for random access.  I have not tested this on a non-Fusion device.  My guess is that the performance would still be much better using the binary-search method, but the improvement is probably not nearly as dramatic.

Below contrasts the difference in results:

image

So, what does the function look like that provides the improvement.  Actually very simple, and easy to adapt to other scenarios.  Note, I have only tested this with a couple of tables, so have not tested all the boundary conditions, in case you try this and it gets stuck in an endless loop or returns the wrong result.

In a few days, I plan to post a stored procedure that receives a table and column name which will then perform the same type of logic so that this can be handled generically.

CREATE FUNCTION [dbo].[udf_GetTopIntradayId]

(

)

RETURNS INT

AS

BEGIN

    -- Bob Leithiser - Sample Only.  Disclaimer: 

    /*    Works so far, but boundary conditions not completely tested.

        May go into an endless loop or return wrong result.

    */    

    -- Uses binary search to find the top EquityIntradayId 

-- Typicall much faster than simply doing a select max for

    -- the scenario of finding a max value on an non-clustered

    -- storage-aligned index where no filtering is done on 

    -- the partitioned column.  To use for another table, 

    -- do global search/replace of the table name and desired

    -- column.

 

    DECLARE @EquityIntradayId INT = NULL

    -- Decrement down until we don't get a match

    DECLARE @Iteration FLOAT = 31.00

    WHILE @Iteration > 0

    BEGIN

        DECLARE @HighRange INT = POWER(2.00,@Iteration)-1.00

        DECLARE @LowRange INT = POWER(2.00,@Iteration-1.00)

        

        IF NOT EXISTS (SELECT TOP 1 0 FROM dbo.EquityIntraday h

                        WHERE h.EquityIntradayId > @LowRange)

            SET @Iteration = @Iteration - 1

        ELSE

            BREAK

    END

    

    -- Search the range

    SET @HighRange = (@HighRange + @LowRange) / 2.00

    WHILE @LowRange <> @HighRange 

    BEGIN

        IF EXISTS (SELECT TOP 1 0 FROM dbo.EquityIntraday 

                    WHERE EquityIntradayId >= (@LowRange + @HighRange) / 2.00)

            SET @LowRange = (@HighRange + @LowRange) / 2.00

        ELSE

            SET @HighRange = (@HighRange + @LowRange) / 2.00

    END 

    RETURN @HighRange  

    

END

That’s all I have for now.  Thanks for reading.  Remember I am very slow to respond on this blog, too much going on in my life, so this is sort of a 1-way brain-dump for me to contribute to the community, rather than in interactive blog.   If you are interested in my work (Microsoft or Academic-related), you can look for Bob Leithiser on linked-in and send me a request.  My thanks to Bob Duffy, Dirk Gubbels, Robert Davis, Assaf Fraenkel for much of the material for this post.  These guys are all Microsoft Certified SQL Masters and the type of people you want on your team for serious SQL Server stuff.

 

Posted in Uncategorized | Leave a comment

Indexes and SSDs and Random versus Sequential Access

If you’re one of the dozen or so people who follow my blog, you’re aware that I’m using SSDs for my doctoral research project involving simulations to find solutions to complex data correlation problems.  I finally have started to get down to the level of really looking at various query plans.  What I’ve found is that SQL often makes the decision to use a sequential scan plan rather than an index seek as the selectivity for a table decreases.

Decreasing the level of selectivity makes a lot of sense for traditional storage devices due to the latency costs associated with index seeks.  However, SSDs are a game changer as the access speed go down to near-zilch.  Recently, I have uploaded several months of intraday rows all the way back to January, 2009.  In my end-of-day history table, I recently added a field which I compute from the intraday data in order to speed up correlation processing for multiple queries that need to compare the intraday trading patterns.  The field will be maintained as part of a daily job that does a bunch of aggregations (you can find some earlier posts on the subject of why I sometimes chose to do this type of stuff for my database, rather than relying on queries or using indexed views).

In any case, There are 21,462,159 15 minute intraday rows in my database since January 4, 2010.  Along with this there are 2,086,605 daily history rows since January, 2010.  There are typically about 30 intraday 15 minute bars per equity per day.  Not all of the daily history rows have associated intraday rows.

My first attempt at updating the Intraday data was as follows:

update dbo.EquityHistory
set DayIntervalCount = (select COUNT(*) from dbo.EquityIntraday ei
where ei.MarketDate = EquityHistory.MarketDate
and ei.PeriodLength=15
and ei.TradingSymbol = dbo.EquityHistory.TradingSymbol)
from dbo.equityhistory
where EquityHistory.MarketDate > ‘20100101’

After 15 minutes, and not very impressive throughput on the SSDs, I cancelled the query.  The sub-plan that was used to feed the rows into the update is shown in the below graphic:

image

Notice, the Index Scan rather than seek on the Intraday rows, despite the fact that there is an index on Intraday containing the symbol, period length, and marketdate – all the fields involved in the search predicate.  However, the 20 million rows represent about 25 % of the total number of rows in the Intraday table – enough for SQL to say forget the index and decide to read the table sequentially.

My intuition was that if the index was used, this should happen quickly, to try this out, I experimented with lowering the granularity of the update such that a smaller percentage of rows were selected from Intraday.  Based on this, I found the maximum range I could use in order to get a scan on Intraday was about 6 trading days.  So, changing the query to:

update dbo.EquityHistory
set DayIntervalCount = (select COUNT(*) from dbo.EquityIntraday ei
where ei.MarketDate = EquityHistory.MarketDate
and ei.PeriodLength=15
and ei.TradingSymbol = dbo.EquityHistory.TradingSymbol)
from dbo.equityhistory
where EquityHistory.MarketDate between ‘20100101’  and ‘20100110’’

gave me the below query sub-plan.

image

 

Executing this update for the 6 days only took 5 seconds.  So, doing the math, there are about 144 trading days since January 1.  Divide 6 into 144 gives us 24.  24 times 5 = 120 seconds.  Based on that, in theory at least ,the update should only take around 2 minutes using this plan.

At this stage, I could have used the sp_create_plan_guide something like below, but took the quick and dirty way and just put the hints in the query.

EXEC sp_create_plan_guide
    @name = N’Update_History_IntradayCounts_FromIntraday’,
    @stmt = N’update dbo.EquityHistory
set DayIntervalCount = (select COUNT(*) from dbo.EquityIntraday ei
where ei.MarketDate = EquityHistory.MarketDate
and ei.PeriodLength=15
and ei.TradingSymbol = dbo.EquityHistory.TradingSymbol)
from dbo.equityhistory
where EquityHistory.MarketDate > ”20100101” ‘,
    @type = N’SQL’,
    @module_or_batch = NULL,
    @params = NULL,
    @hints = N’OPTION (TABLE HINT ( ei, INDEX( IX_EquityIntraday_DateSymbolPeriodLengthInterval ), FORCESEEK ) )’;
GO

So, what was the result of using the hint to force a seek on the index instead of a scan for retrieving the 20 million rows to update 2 million aggregation rows?

image

Update time comparison on SSD Storage

Using query hint: 2:14
Sequential Access: 10:26*
Benefit Ratio: 5 to 1

*First time, the sequential took over 15 minutes before I cancelled it, second time tested, it finished much faster, probably due to data caching.

As you can see, changing the query plan was effective.  So, you might ask: Would changing the query plan have the same benefit for a database not on SSD – is this just a SSD-issue?  I just happen to have a copy of the database on non-SSD storage.  So, here are the update times using both methods on that database:

Update time on non-SSD storage

Using query hint: 4:50
Sequential Access:  17:29
Benefit Ratio: 4 to 1

So, we see that the query hint on non-SSD was also more effective, but, it was more effective on SSD (5 to 1 benefit versus 4 to 1 benefit)

There are a couple of implications for this related to SSDs:

1) For large operations that end up using sequential access, you may need to revisit the query plan and tweak them to be seek-based to get the best performance.

2) SQL Server is a cost-based optimizer, not results-oriented.  SQL doesn’t factor in what type of storage your rows are located on, it will use the same cost calculation techniques whether the data is on SSD, on a RAM device, on a high-speed 15k SAS drive, or on a 5400 RPM laptop drive.  It also does not factor in database caching – i.e. with a large amount of memory, the data may already be in memory, making random access almost as fast as sequential access.  This type of issue makes the argument for a metrics-based optimizer for SQL – one that actually looks as the results of using different query plans and factors this in when determining a plan, rather than simply the estimates of performance.

 

Posted in Uncategorized | Leave a comment

SSIS – Chunking number of inserts with a For Each container

My main customer utilizes SSIS for some SQL and Oracle integration requirements.  One challenge they’ve had is timeouts from the Oracle side.  We’ve tried everything to eliminate the issue in terms of extending timeout values within SSIS, but the problem appears to be that for very large data transfers into Oracle, Oracle simply gets tired of waiting and the connection gets closed.  One of the developers working on this project has found a way around this by transferring data in smaller chunks to Oracle.  He asked me about a way to automate this in SSIS so that the records are loaded in smaller chunks.  I pointed him to the FOR LOOP construct and recommended creating a second table with an identity column that could be used as a means to ensure that there was a key value associated with the record range.

That’s about all I did, the rest of this is his work.  My thanks to Robert Skinner, HP for sharing the finished package with me.  I’ve modified the package to remove the application-specific data/fields and reduced down to just a simple table with a couple of fields, so we can focus on the approach and not get bogged down in the schema.  For our example, we will use a flat file for import and just a SQL database for output, the destination connectors can be changed as needed to support exporting to another OLE DB destination such as Oracle without changing the actual components.  Let’s walk through the design and implementation.

There are 3 main steps to the process:

  1. Ensure that a staging table exists and is prepared to store the imported rows that will need to be exported to Oracle.  The table needs to be empty prior to importing.   The staging table has the following attributes:
    1. All of the columns from the imported table
    2. A primary identity key.
  2. Load the imported data into the staging table.
  3. Prepare the destination (in this scenario, this requires truncating the data in the destination).
  4. Cycle through the staging table selecting the range of keys associated with the “chunking” number and copying to the destination table.  For example, if only 50,000 rows are being loaded per insertion, each selection would be the next 50,000 rows – for example: 1 – 50,000, 50,0001 to 100,000, etc.

Here is a screen snapshot of the package.

image

Before loading the import data, we want to have a clean table to load into and make sure the identity range is reset.  The easiest way to do this is with a TRUNCATE table if it already exists, otherwise simply create it.   Robert actually uses the same logic as is generated by scripting using the SQL Server tasks/script that will check for the table, truncate it if empty, and if not actually create the table.   For our scenario, we just truncate the 2 tables in the first 2 steps using a OLE DB Source with a SQL Command:

  • Truncate Table Quote_Staging
  • Truncate Table Quote

Next, we load from the flat file into the Staging Table.  This will result in all of the original rows being assigned to a sequential identifier.  Since we truncated the table, we are guaranteed that each number is sequential.   Note the use of the Row Count component, this will capture the total number of rows in the flat file that get loaded into the staging table.

image

Here are the first few rows after the load step. 

StagingId Symbol Date Open High Low Close Volume
1 A 00:00.0 16.52 17.07 16.21 16.9 2790900
2 AA 00:00.0 9.03 9.28 8.92 9.14 28107000
3 AAI 00:00.0 7.5 7.52 7.08 7.39 5395300
4 AAP 00:00.0 44.19 44.39 42.5 42.63 1818700
5 AAR 00:00.0 11.68 11.95 11.61 11.92 7700
6 AAV 00:00.0 2.89 2.96 2.86 2.93 703600
7 AB 00:00.0 16.11 16.62 15.88 16.29 700000
8 ABA 00:00.0 25.87 26.05 25.87 25.97 2800
9 ABB 00:00.0 14.77 15.18 14.69 15 4501200
10 ABC 00:00.0 35.41 36.06 34.15 34.58 3436700
11 ABD 00:00.0 2.19 2.29 2.04 2.22 452400
12 ABG 00:00.0 8.2 8.29 7.57 7.8 563900
13 ABK 00:00.0 1 1.02 0.95 1.01 4878800
14 ABM 00:00.0 17.06 17.72 16.9 17.45 253800
15 ABN-E 00:00.0 7.41 7.55 7.35 7.37 47300
16 ABN-F 00:00.0 8 8.24 7.25 8.08 14100
17 ABN-G 00:00.0 7.39 7.65 7.26 7.51 105400
18 ABR 00:00.0 1.35 1.69 1.35 1.48 372000
19 ABT 00:00.0 43.57 43.9 42.88 43.09 9309500
20 ABV 00:00.0 54.12 54.73 53.61 53.81 772900

Let’s look more closely at how the FOR LOOP works with the script task.

image

The For Loop Properties controls the initialization, evaluation, and assignment.  For this scenario, we have the following variables:

  • RowCountLast: The identity of the last row loaded.
  • RowCountIncrement: The amount of rows to load in each chunk.
  • RowCountTotal: The total number of rows in the table
  • SqlCommand: Contains the SQL command to execute on each iteration of the loop to “chunk” the data rows.  Initially this should be set to just “Select * from the staging table – this is “Select * from Quote_Staging” for our example.

The script is used to build the SQL command variable that selects on the range identified for each iteration of the for loop.  For example, on iteration 1, using an increment of 500, the SQL command would be set to query all of the rows between 0 and 500.  On the next iteration, the last row would be set to 500 and the SQL Command altered to load from 501 – 1000.  This continues until the number of rows loaded exceeds the total row count.  Each iteration of the for loop establishes a new connection and only load the desired number of rows.  Although, this is normally not optimal for most loading, this works around the Oracle issue.  This is also useful if there are a huge number of rows, and monitoring of the process at a lower granularity is needed than for the whole table.

Here is the script.  Note that this script is highly reusable, the only item that needs to be changed for another package would be the name of the table and the identity column.  To make this even more reusable, variables could be created. 

Code Snippet
  1. ‘ Microsoft SQL Server Integration Services Script Task
  2. ‘ Write scripts using Microsoft Visual Basic 2008.
  3. ‘ The ScriptMain is the entry point class of the script.
  4.  
  5. Imports System
  6. Imports System.Data
  7. Imports System.Math
  8. Imports Microsoft.SqlServer.Dts.Runtime
  9.  
  10. <System.AddIn.AddIn(“ScriptMain”, Version:=“1.0”, Publisher:=“”, Description:=“”)> _
  11. <System.CLSCompliantAttribute(False)> _
  12. Partial Public Class ScriptMain
  13.     Inherits Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
  14.  
  15.     Enum ScriptResults
  16.         Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success
  17.         Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
  18.     End Enum
  19.    
  20.  
  21.     ‘ The execution engine calls this method when the task executes.
  22.     ‘ To access the object model, use the Dts object. Connections, variables, events,
  23.     ‘ and logging features are available as static members of the Dts class.
  24.     ‘ Before returning from this method, set the value of Dts.TaskResult to indicate success or failure.
  25.    
  26.     ‘ To open Code and Text Editor Help, press F1.
  27.     ‘ To open Object Browser, press Ctrl+Alt+J.
  28.  
  29.     Public Sub Main()
  30.  
  31.         Dim RowCountLast As Integer
  32.         RowCountLast = CType(ReadVariable(“RowCountLast”), Integer)
  33.  
  34.         Dim RowCountIncrement As Integer
  35.         RowCountLast += RowCountIncrement
  36.  
  37.         RowCountIncrement = CType(ReadVariable(“RowCountIncrement”), Integer)
  38.  
  39.         Dim SqlCommand As String
  40.         SqlCommand = “Select * from Quote_Staging where StagingId >= “ & RowCountLast & ” and  StagingId < “ & RowCountLast + RowCountIncrement
  41.  
  42.         WriteVariable(“SqlCommand”, SqlCommand)
  43.  
  44.     End Sub ‘ Main
  45.     Private Function ReadVariable(ByVal varName As String) As Object
  46.         Dim result As Object
  47.         Try
  48.             Dim vars As Variables
  49.             Dts.VariableDispenser.LockForRead(varName)
  50.             Dts.VariableDispenser.GetVariables(vars)
  51.             Try
  52.                 result = vars(varName).Value
  53.             Catch ex As Exception
  54.                 Throw ex
  55.             Finally
  56.                 vars.Unlock()
  57.             End Try
  58.         Catch ex As Exception
  59.             Throw ex
  60.         End Try
  61.         Return result
  62.     End Function
  63.     Private Sub WriteVariable(ByVal varName As String, ByVal varValue As Object)
  64.         Try
  65.             Dim vars As Variables
  66.             Dts.VariableDispenser.LockForWrite(varName)
  67.             Dts.VariableDispenser.GetVariables(vars)
  68.             Try
  69.                 vars(varName).Value = varValue
  70.             Catch ex As Exception
  71.                 Throw ex
  72.             Finally
  73.                 vars.Unlock()
  74.             End Try
  75.         Catch ex As Exception
  76.             Throw ex
  77.         End Try
  78.     End Sub
  79.  
  80.  
  81. End Class

The data flow task in the for loop simply uses the SQL Command variable for the source and the final table for the destination.

image

image

Attached is a zip file containing a sample database schema, flat file with import data, and SSIS package.

My thanks again to Robert Skinner for providing me a sample SSIS package for using this including the script.

Posted in Uncategorized | Leave a comment

Using Checksum to create random data sets

Here’s a quick tip if you don’t want to try to use random and figure out seed, etc.  If you’ve got a lot of columns in a table, it is pretty likely that the checksum for all those data values are going to be random across your primary keys.

I recently needed to create a 5% random sample of my equities, so just ran the following query:

select TradingSymbol
from dbo.EquitySymbol
where EquityType=0
and CHECKSUM(*) % 20 = 0

That got me 367 out of about 7300 possible rows.  If you’re doing random sampling, I suggest the following table structure.

image

Then, you can create a random sample by inserting a sample ID into Data Sample and then insert from your selection into the DataSampleValue table:

insert into olap.DataSample select ‘Random Sample 5% for EquityType=0’
select @@IDENTITY

insert into olap.DataSampleEquity
select 18, TradingSymbol
from dbo.EquitySymbol
where EquityType=0
and CHECKSUM(*) % 20 = 0

select * from olap.DataSample

 

(No column name) DataValue
18 AAII
18 AAON
18 ABC
18 ADGF
18 ADLS
18 ADP
18 AEC-B
18 AFCB
18 AFFM
18 AIMC
18 AIR
18 AKT
18 ALLB
18 ALZM
18
Posted in Uncategorized | Leave a comment

Managing Index Fragmentation and Compression

For today, I recycle bits from a couple of excellent authors in the SQL community.  The first script is for optimizing compression.  I “borrowed” this from Paul Nielson, author of the SQL Bible.  You can visit his SQL Server bible site at http://www.sqlserverbible.com/.   It includes a link for sample scripts, which is where I pulled this from.  I’m also including a script for defragmenting indexes, which I borrowed from http://sqlfool.com/2010/04/index-defrag-script-v4-0/.  These scripts can be used together.  What I am doing is on a monthly basis running the compression script to make adjustments to my compression options based on data, and then doing the defragmentation weekly.

Based on my testing and testing on several other sites, compression should be the norm rather than the exception.  Unless you have heavy OLTP processing, most of the time the savings in memory, cache, and ultimately disk I/O because more data is stored in less space is greater than the CPU costs.  Along the same lines, backup compression should almost always be used as it will run faster than non-compressed for both backup and restore purposes.  They really got compression right in SQL 2008.

Below is the script almost as originally written, but with a couple of modifications.  Paul’s version had only 1 threshold, rather than separate for page and row, which I’ve added.  In his version, he uses page compression if it is more effective than row and row compression if it is more effective than page.   However almost always page is better than row because it incorporates row.  I’ve decided to take a slightly different approach and allow a threshold for row and another for page.  Page requires more CPU processing than row, so the idea is that we only want page if the compression benefits are significantly better than row.   On the other hand, row has a lower CPU cost.  By default, I am using 15 and 25%.  By default, I’m employing page compression if it has a benefit of 25% savings and row if it has a benefit of 15%, you can pass in different values if you want to be more conservative or aggressive. 

First the script for database compression setting from Paul Nielson with my changes noted in the comments:


.csharpcode, .csharpcode pre
{
font-size: small;
color: black;
font-family: consolas, “Courier New”, courier, monospace;
background-color: #ffffff;
/*white-space: pre;*/
}
.csharpcode pre { margin: 0em; }
.csharpcode .rem { color: #008000; }
.csharpcode .kwrd { color: #0000ff; }
.csharpcode .str { color: #006080; }
.csharpcode .op { color: #0000c0; }
.csharpcode .preproc { color: #cc6633; }
.csharpcode .asp { background-color: #ffff00; }
.csharpcode .html { color: #800000; }
.csharpcode .attr { color: #ff0000; }
.csharpcode .alt
{
background-color: #f4f4f4;
width: 100%;
margin: 0em;
}
.csharpcode .lnum { color: #606060; }

/****** Object:  StoredProcedure [dbo].[db_compression]    Script Date: 06/06/2010 21:05:22 ******/
SET ANSI_NULLS ON
GO

SET QUOTED_IDENTIFIER ON
GO



CREATE PROC [dbo].[db_compression] (
  @minCompression float = .15, -- e.g. .25 for minimum of 25% compression
  @minPageCompression float = .25 ,-- threshold to use page compression
  @ReportModeOnly bit = 0
  )
as
set nocount on 

-- Paul Nielsen
-- www.SQLServerBible.com 
-- March 13, 2008

/*
Mod by Bob L, June, 2010 - Receive parameters for minCompression and minPageCompression
Instead of checking to see if page or row is more effective, simply use the compression 
matching the threshold.  Passing in a higher page-level threshold than row eliminates 
the possibility of choosing the less effective compression type.  Normally page compression
will yield a higher return since it is includes row compression.  However, it has performance
penalty, so the page compression threshhold should be significant - using 25% as the default.

*/
/*
  sets compression for all objects and indexs in the database needing adjustment
  if estimated gain is equal to or greater than mincompression parameter
    then enables row or page compression whichever is greater gain
  if row and page have same gain
    then enables row compression
  if estimated gain is less than mincompression parameter 
    then compression is set to none
    
*/

-- to do:
-- [ ] transaction error from insert...Exec sp_estimate_data_compression_savings
-- [ ] filter objects to only those eligible for compression 


  IF @minCompression is null SET @minCompression = .15
  IF @minPageCompression is null SET @minPageCompression = .25

  CREATE TABLE #ObjEst (
    PK int identity not null primary key,
    object_name varchar(250),
    schema_name varchar(250),
    index_id INT,
    partition_number int,
    size_with_current_compression_setting bigint,
    size_with_requested_compression_setting bigint,
    sample_size_with_current_compression_setting bigint,
    sample_size_with_requested_compresison_setting bigint
    )
    
  CREATE TABLE #dbEstimate (
    PK int identity not null primary key,
    schema_name varchar(250),
    object_name varchar(250),
    index_id INT,
    ixName VARCHAR(255),
    ixType VARCHAR(50),
    partition_number int,
    data_compression_desc VARCHAR(50),
    None_Size INT,
    Row_Size INT,
    Page_Size INT
    )
  
  INSERT INTO #dbEstimate (
    schema_name, 
    object_name, 
    index_id, 
    ixName, 
    ixType, 
    partition_number, 
    data_compression_desc)
      select 
        S.name, 
        o.name, 
        I.index_id, 
        I.name, 
        I.type_desc, 
        P.partition_number, 
        P.data_compression_desc
        from sys.schemas as S
          join sys.objects as O
            on S.schema_id = O.schema_id 
          join sys.indexes as I
              on o.object_id = I.object_id 
            join sys.partitions as P
              on I.object_id = P.object_id
              and I.index_id= p.index_id
           where O.TYPE = 'U' 
           
 -- Determine Compression Estimates 
  DECLARE
    @PK INT,
    @Schema varchar(150),
    @object varchar(150),
    @DAD varchar(25),
    @partNO int,
    @indexID int,
    @ixName VARCHAR(250),
    @SQL nVARCHAR(max),
    @ixType VARCHAR(50), 
    @Recommended_Compression VARCHAR(10)
    
 
  DECLARE cCompress CURSOR FAST_FORWARD
    FOR 
      select schema_name, object_name, index_id, partition_number, data_compression_desc
        FROM #dbEstimate
   
  OPEN cCompress
  
  FETCH cCompress INTO @Schema, @object, @indexID, @partNO, @DAD  -- prime the cursor
 
  WHILE @@Fetch_Status = 0 
    BEGIN
        
    IF @DAD = 'none'
      BEGIN 
            -- estimate Page compression
            INSERT #ObjEst (
                object_name,
                schema_name,
                index_id,partition_number,
                size_with_current_compression_setting,
                size_with_requested_compression_setting,
                sample_size_with_current_compression_setting,
                sample_size_with_requested_compresison_setting )
              EXEC sp_estimate_data_compression_savings
                @Schema_name = @Schema,
                @object_name = @object,
                @index_id = @indexID,
                @partition_number = @partNO,
                @data_compression = 'page'
                
             UPDATE #dbEstimate
                SET none_size = O.size_with_current_compression_setting,
                    page_size = O.size_with_requested_compression_setting
                FROM #dbEstimate D
                  JOIN #ObjEst O
                    ON  D.Schema_name = O.Schema_Name
                    and D.Object_name = O.object_name
                    and D.index_id = O.index_id
                    and D.partition_number = O.partition_number  
                    
             DELETE #ObjEst  
             
             -- estimate Row compression
            INSERT #ObjEst (
                object_name,
                schema_name,
                index_id,partition_number,
                size_with_current_compression_setting,
                size_with_requested_compression_setting,
                sample_size_with_current_compression_setting,
                sample_size_with_requested_compresison_setting )
              EXEC sp_estimate_data_compression_savings
                @Schema_name = @Schema,
                @object_name = @object,
                @index_id = @indexID,
                @partition_number = @partNO,
                @data_compression = 'row'
                
             UPDATE #dbEstimate
                SET row_size = O.size_with_requested_compression_setting
                FROM #dbEstimate D
                  JOIN #ObjEst O
                    ON  D.Schema_name = O.Schema_Name
                    and D.Object_name = O.object_name
                    and D.index_id = O.index_id
                    and D.partition_number = O.partition_number  
                    
             DELETE #ObjEst       
        END -- none compression estimate      
 
    IF @DAD = 'row'
      BEGIN 
            -- estimate Page compression
            INSERT #ObjEst (
                object_name,schema_name,
                index_id,partition_number,
                size_with_current_compression_setting,
                size_with_requested_compression_setting,
                sample_size_with_current_compression_setting,
                sample_size_with_requested_compresison_setting )
              EXEC sp_estimate_data_compression_savings
                @Schema_name = @Schema,
                @object_name = @object,
                @index_id = @indexID,
                @partition_number = @partNO,
                @data_compression = 'page'
                
             UPDATE #dbEstimate
                SET row_size = O.size_with_current_compression_setting,
                    page_size = O.size_with_requested_compression_setting
                FROM #dbEstimate D
                  JOIN #ObjEst O
                    ON  D.Schema_name = O.Schema_Name
                    and D.Object_name = O.object_name
                    and D.index_id = O.index_id
                    and D.partition_number = O.partition_number  
                    
             DELETE #ObjEst  
             
             -- estimate None compression
            INSERT #ObjEst (
                object_name,schema_name,
                index_id,partition_number,
                size_with_current_compression_setting,
                size_with_requested_compression_setting,
                sample_size_with_current_compression_setting,
                sample_size_with_requested_compresison_setting )
              EXEC sp_estimate_data_compression_savings
                @Schema_name = @Schema,
                @object_name = @object,
                @index_id = @indexID,
                @partition_number = @partNO,
                @data_compression = 'none'
                
             UPDATE #dbEstimate
                SET none_size = O.size_with_requested_compression_setting
                FROM #dbEstimate D
                  JOIN #ObjEst O
                    ON  D.Schema_name = O.Schema_Name
                    and D.Object_name = O.object_name
                    and D.index_id = O.index_id
                    and D.partition_number = O.partition_number  
                    
             DELETE #ObjEst       
        END -- row compression estimate     
      
    IF @DAD = 'page'
      BEGIN 
            -- estimate Row compression
            INSERT #ObjEst (
                object_name,
                schema_name,
                index_id,
                partition_number,
                size_with_current_compression_setting,
                size_with_requested_compression_setting,
                sample_size_with_current_compression_setting,
                sample_size_with_requested_compresison_setting )
              EXEC sp_estimate_data_compression_savings
                @Schema_name = @Schema,
                @object_name = @object,
                @index_id = @indexID,
                @partition_number = @partNO,
                @data_compression = 'row'
                
             UPDATE #dbEstimate
                SET page_size = O.size_with_current_compression_setting,
                    row_size = O.size_with_requested_compression_setting
                FROM #dbEstimate D
                  JOIN #ObjEst O
                    ON  D.Schema_name = O.Schema_Name
                    and D.Object_name = O.object_name
                    and D.index_id = O.index_id
                    and D.partition_number = O.partition_number  
                    
             DELETE #ObjEst  
             
             -- estimate None compression
            INSERT #ObjEst (
                object_name,
                schema_name,
                index_id,partition_number,
                size_with_current_compression_setting,
                size_with_requested_compression_setting,
                sample_size_with_current_compression_setting,
                sample_size_with_requested_compresison_setting )
              EXEC sp_estimate_data_compression_savings
                @Schema_name = @Schema,
                @object_name = @object,
                @index_id = @indexID,
                @partition_number = @partNO,
                @data_compression = 'none'
                
             UPDATE #dbEstimate
                SET none_size = O.size_with_requested_compression_setting
                FROM #dbEstimate D
                  JOIN #ObjEst O
                    ON  D.Schema_name = O.Schema_Name
                    and D.Object_name = O.object_name
                    and D.index_id = O.index_id
                    and D.partition_number = O.partition_number  
                    
             DELETE #ObjEst       
        END -- page compression estimate 
          
       FETCH cCompress INTO @Schema, @object, @indexID, @partNO, @DAD 
    END

  CLOSE cCompress
  DEALLOCATE cCompress
  
   
 -- set the compression 
 DECLARE cCompress CURSOR FAST_FORWARD
    FOR 
      select schema_name, object_name, partition_number, ixName, ixType,  
         Case 
           when (1-(cast(Row_Size as float) / none_Size)) >= @minCompression 
           -- and (Row_Size <= Page_Size) 
           then 'Row' 
           when (1-(cast(page_Size as float) / none_Size)) >= @minPageCompression 
           -- and (Page_Size <= row_Size) 
           then 'Page' 
           else 'None' 
         end as Recommended_Compression
       from #dbEstimate 
       where None_Size <> 0
       and (Case 
           when (1-(cast(page_Size as float) / none_Size)) >= @minPageCompression 
and (Page_Size <= row_Size) then 'Page' when (1-(cast(Row_Size as float) / none_Size)) >= @minCompression then 'Row'
-- and (Row_Size <= Page_Size) then 'Row' -- Bob L, June, 2010 - Don't check row vs page since we already decided to use else 'None' end <> data_compression_desc) OPEN cCompress FETCH cCompress INTO @Schema, @object, @partNO, @ixName, @ixType, @Recommended_Compression
-- prime the cursor WHILE @@Fetch_Status = 0 BEGIN IF @ixType = 'Clustered' or @ixType='heap' set @SQL = 'ALTER TABLE ' + @schema + '.' + @object + ' Rebuild with (data_compression = ' +
@Recommended_Compression + ')' else set @SQL = 'ALTER INDEX ' + @ixName + ' on ' + @schema + '.' + @object + ' Rebuild with (data_compression = ' + @Recommended_Compression + ')' print @SQL IF @ReportModeOnly = 0 BEGIN EXEC sp_executesql @SQL END FETCH cCompress INTO @Schema, @object, @partNO, @ixName, @ixType, @Recommended_Compression
-- prime the cursor END CLOSE cCompress DEALLOCATE cCompress RETURN GO

Now for the script for defragmenting indexes from Michelle Ufford at http://sqlfool.com/2010/04/index-defrag-script-v4-0/

/* Scroll down to the see notes, disclaimers, and licensing information */

 

DECLARE @indexDefragLog_rename      VARCHAR(128)

    , @indexDefragExclusion_rename  VARCHAR(128)

    , @indexDefragStatus_rename     VARCHAR(128);

 

SELECT @indexDefragLog_rename       = 'dba_indexDefragLog_obsolete_' + CONVERT(VARCHAR(10), GETDATE(), 112)

    , @indexDefragExclusion_rename  = 'dba_indexDefragExclusion_obsolete_' + CONVERT(VARCHAR(10), GETDATE(), 112)

    , @indexDefragStatus_rename     = 'dba_indexDefragStatus_obsolete_' + CONVERT(VARCHAR(10), GETDATE(), 112);

 

IF Exists(SELECT [OBJECT_ID] FROM sys.tables WHERE [name] = 'dba_indexDefragLog')

    EXECUTE SP_RENAME dba_indexDefragLog, @indexDefragLog_rename;

 

IF Exists(SELECT [OBJECT_ID] FROM sys.tables WHERE [name] = 'dba_indexDefragExclusion')

    EXECUTE SP_RENAME dba_indexDefragExclusion, @indexDefragExclusion_rename;

 

IF Exists(SELECT [OBJECT_ID] FROM sys.tables WHERE [name] = 'dba_indexDefragStatus')

    EXECUTE SP_RENAME dba_indexDefragStatus, @indexDefragStatus_rename;

Go

 

CREATE TABLE dbo.dba_indexDefragLog

(

      indexDefrag_id    INT IDENTITY(1,1)   Not Null

    , databaseID        INT                 Not Null

    , databaseName      NVARCHAR(128)       Not Null

    , objectID          INT                 Not Null

    , objectName        NVARCHAR(128)       Not Null

    , indexID           INT                 Not Null

    , indexName         NVARCHAR(128)       Not Null

    , partitionNumber   SMALLINT            Not Null

    , fragmentation     FLOAT               Not Null

    , page_count        INT                 Not Null

    , dateTimeStart     DATETIME            Not Null

    , dateTimeEnd       DATETIME            Null

    , durationSeconds   INT                 Null

    , sqlStatement      VARCHAR(4000)       Null

    , errorMessage      VARCHAR(1000)       Null

 

    CONSTRAINT PK_indexDefragLog_v40

        PRIMARY KEY CLUSTERED (indexDefrag_id)

);

 

PRINT 'dba_indexDefragLog Table Created';

 

CREATE TABLE dbo.dba_indexDefragExclusion

(

      databaseID        INT                 Not Null

    , databaseName      NVARCHAR(128)       Not Null

    , objectID          INT                 Not Null

    , objectName        NVARCHAR(128)       Not Null

    , indexID           INT                 Not Null

    , indexName         NVARCHAR(128)       Not Null

    , exclusionMask     INT                 Not Null

        /* 1=Sunday, 2=Monday, 4=Tuesday, 8=Wednesday, 16=Thursday, 32=Friday, 64=Saturday */

 

    CONSTRAINT PK_indexDefragExclusion_v40

        PRIMARY KEY CLUSTERED (databaseID, objectID, indexID)

);

 

PRINT 'dba_indexDefragExclusion Table Created';

 

CREATE TABLE dbo.dba_indexDefragStatus

(

      databaseID        INT

    , databaseName      NVARCHAR(128)

    , objectID          INT

    , indexID           INT

    , partitionNumber   SMALLINT

    , fragmentation     FLOAT

    , page_count        INT

    , range_scan_count  BIGINT

    , schemaName        NVARCHAR(128)   Null

    , objectName        NVARCHAR(128)   Null

    , indexName         NVARCHAR(128)   Null

    , scanDate          DATETIME        

    , defragDate        DATETIME        Null

    , printStatus       BIT             DEFAULT(0)

    , exclusionMask     INT             DEFAULT(0)

 

    CONSTRAINT PK_indexDefragStatus_v40

        PRIMARY KEY CLUSTERED(databaseID, objectID, indexID, partitionNumber)

);

 

PRINT 'dba_indexDefragStatus Table Created';

 

IF OBJECTPROPERTY(OBJECT_ID('dbo.dba_indexDefrag_sp'), N'IsProcedure') = 1

BEGIN

    DROP PROCEDURE dbo.dba_indexDefrag_sp;

    PRINT 'Procedure dba_indexDefrag_sp dropped';

END;

Go

 

CREATE PROCEDURE dbo.dba_indexDefrag_sp

 

    /* Declare Parameters */

      @minFragmentation     FLOAT           = 10.0  

        /* in percent, will not defrag if fragmentation less than specified */

    , @rebuildThreshold     FLOAT           = 30.0  

        /* in percent, greater than @rebuildThreshold will result in rebuild instead of reorg */

    , @executeSQL           BIT             = 1     

        /* 1 = execute; 0 = print command only */

    , @defragOrderColumn    NVARCHAR(20)    = 'range_scan_count'

        /* Valid options are: range_scan_count, fragmentation, page_count */

    , @defragSortOrder      NVARCHAR(4)     = 'DESC'

        /* Valid options are: ASC, DESC */

    , @timeLimit            INT             = 720 /* defaulted to 12 hours */

        /* Optional time limitation; expressed in minutes */

    , @DATABASE             VARCHAR(128)    = Null

        /* Option to specify a database name; null will return all */

    , @tableName            VARCHAR(4000)   = Null  -- databaseName.schema.tableName

        /* Option to specify a table name; null will return all */

    , @forceRescan          BIT             = 0

        /* Whether or not to force a rescan of indexes; 1 = force, 0 = use existing scan, if available */

    , @scanMode             VARCHAR(10)     = N'LIMITED'

        /* Options are LIMITED, SAMPLED, and DETAILED */

    , @minPageCount         INT             = 8 

        /*  MS recommends > 1 extent (8 pages) */

    , @maxPageCount         INT             = Null

        /* NULL = no limit */

    , @excludeMaxPartition  BIT             = 0

        /* 1 = exclude right-most populated partition; 0 = do not exclude; see notes for caveats */

    , @onlineRebuild        BIT             = 1     

        /* 1 = online rebuild; 0 = offline rebuild; only in Enterprise */

    , @sortInTempDB         BIT             = 1

        /* 1 = perform sort operation in TempDB; 0 = perform sort operation in the index's database */

    , @maxDopRestriction    TINYINT         = Null

        /* Option to restrict the number of processors for the operation; only in Enterprise */

    , @printCommands        BIT             = 0     

        /* 1 = print commands; 0 = do not print commands */

    , @printFragmentation   BIT             = 0

        /* 1 = print fragmentation prior to defrag; 

           0 = do not print */

    , @defragDelay          CHAR(8)         = '00:00:05'

        /* time to wait between defrag commands */

    , @debugMode            BIT             = 0

        /* display some useful comments to help determine if/where issues occur */

 

AS

/*********************************************************************************

    Name:       dba_indexDefrag_sp

 

    Author:     Michelle Ufford, http://sqlfool.com

 

    Purpose:    Defrags one or more indexes for one or more databases

 

    Notes:

 

    CAUTION: TRANSACTION LOG SIZE SHOULD BE MONITORED CLOSELY WHEN DEFRAGMENTING.

             DO NOT RUN UNATTENDED ON LARGE DATABASES DURING BUSINESS HOURS.

 

      @minFragmentation     defaulted to 10%, will not defrag if fragmentation 

                            is less than that

 

      @rebuildThreshold     defaulted to 30% as recommended by Microsoft in BOL;

                            greater than 30% will result in rebuild instead

 

      @executeSQL           1 = execute the SQL generated by this proc; 

                            0 = print command only

 

      @defragOrderColumn    Defines how to prioritize the order of defrags.  Only

                            used if @executeSQL = 1.  

                            Valid options are: 

                            range_scan_count = count of range and table scans on the

                                               index; in general, this is what benefits 

                                               the most from defragmentation

                            fragmentation    = amount of fragmentation in the index;

                                               the higher the number, the worse it is

                            page_count       = number of pages in the index; affects

                                               how long it takes to defrag an index

 

      @defragSortOrder      The sort order of the ORDER BY clause.

                            Valid options are ASC (ascending) or DESC (descending).

 

      @timeLimit            Optional, limits how much time can be spent performing 

                            index defrags; expressed in minutes.

 

                            NOTE: The time limit is checked BEFORE an index defrag

                                  is begun, thus a long index defrag can exceed the

                                  time limitation.

 

      @database             Optional, specify specific database name to defrag;

                            If not specified, all non-system databases will

                            be defragged.

 

      @tableName            Specify if you only want to defrag indexes for a 

                            specific table, format = databaseName.schema.tableName;

                            if not specified, all tables will be defragged.

 

      @forceRescan          Whether or not to force a rescan of indexes.  If set

                            to 0, a rescan will not occur until all indexes have

                            been defragged.  This can span multiple executions.

                            1 = force a rescan

                            0 = use previous scan, if there are indexes left to defrag

 

      @scanMode             Specifies which scan mode to use to determine

                            fragmentation levels.  Options are:

                            LIMITED - scans the parent level; quickest mode,

                                      recommended for most cases.

                            SAMPLED - samples 1% of all data pages; if less than

                                      10k pages, performs a DETAILED scan.

                            DETAILED - scans all data pages.  Use great care with

                                       this mode, as it can cause performance issues.

 

      @minPageCount         Specifies how many pages must exist in an index in order 

                            to be considered for a defrag.  Defaulted to 8 pages, as 

                            Microsoft recommends only defragging indexes with more 

                            than 1 extent (8 pages).  

 

                            NOTE: The @minPageCount will restrict the indexes that

                            are stored in dba_indexDefragStatus table.

 

      @maxPageCount         Specifies the maximum number of pages that can exist in 

                            an index and still be considered for a defrag.  Useful

                            for scheduling small indexes during business hours and

                            large indexes for non-business hours.

 

                            NOTE: The @maxPageCount will restrict the indexes that

                            are defragged during the current operation; it will not

                            prevent indexes from being stored in the 

                            dba_indexDefragStatus table.  This way, a single scan

                            can support multiple page count thresholds.

 

      @excludeMaxPartition  If an index is partitioned, this option specifies whether

                            to exclude the right-most populated partition.  Typically,

                            this is the partition that is currently being written to in

                            a sliding-window scenario.  Enabling this feature may reduce

                            contention.  This may not be applicable in other types of 

                            partitioning scenarios.  Non-partitioned indexes are 

                            unaffected by this option.

                            1 = exclude right-most populated partition

                            0 = do not exclude

 

      @onlineRebuild        1 = online rebuild; 

                            0 = offline rebuild

 

      @sortInTempDB         Specifies whether to defrag the index in TEMPDB or in the

                            database the index belongs to.  Enabling this option may

                            result in faster defrags and prevent database file size 

                            inflation.

                            1 = perform sort operation in TempDB

                            0 = perform sort operation in the index's database 

 

      @maxDopRestriction    Option to specify a processor limit for index rebuilds

 

      @printCommands        1 = print commands to screen; 

                            0 = do not print commands

 

      @printFragmentation   1 = print fragmentation to screen;

                            0 = do not print fragmentation

 

      @defragDelay          Time to wait between defrag commands; gives the

                            server a little time to catch up 

 

      @debugMode            1 = display debug comments; helps with troubleshooting

                            0 = do not display debug comments

 

    Called by:  SQL Agent Job or DBA

 

    ----------------------------------------------------------------------------

    DISCLAIMER: 

    This code and information are provided "AS IS" without warranty of any kind,

    either expressed or implied, including but not limited to the implied 

    warranties or merchantability and/or fitness for a particular purpose.

    ----------------------------------------------------------------------------

    LICENSE: 

    This index defrag script is free to download and use for personal, educational, 

    and internal corporate purposes, provided that this header is preserved. 

    Redistribution or sale of this index defrag script, in whole or in part, is 

    prohibited without the author's express written consent.

    ----------------------------------------------------------------------------

    Date        Initials    Version Description

    ----------------------------------------------------------------------------

    2007-12-18  MFU         1.0     Initial Release

    2008-10-17  MFU         1.1     Added @defragDelay, CIX_temp_indexDefragList

    2008-11-17  MFU         1.2     Added page_count to log table

                                    , added @printFragmentation option

    2009-03-17  MFU         2.0     Provided support for centralized execution

                                    , consolidated Enterprise & Standard versions

                                    , added @debugMode, @maxDopRestriction

                                    , modified LOB and partition logic  

    2009-06-18  MFU         3.0     Fixed bug in LOB logic, added @scanMode option

                                    , added support for stat rebuilds (@rebuildStats)

                                    , support model and msdb defrag

                                    , added columns to the dba_indexDefragLog table

                                    , modified logging to show "in progress" defrags

                                    , added defrag exclusion list (scheduling)

    2009-08-28  MFU         3.1     Fixed read_only bug for database lists

    2010-04-20  MFU         4.0     Added time limit option

                                    , added static table with rescan logic

                                    , added parameters for page count & SORT_IN_TEMPDB

                                    , added try/catch logic and additional debug options

                                    , added options for defrag prioritization

                                    , fixed bug for indexes with allow_page_lock = off

                                    , added option to exclude right-most partition

                                    , removed @rebuildStats option

                                    , refer to http://sqlfool.com for full release notes

*********************************************************************************

    Example of how to call this script:

 

        Exec dbo.dba_indexDefrag_sp

              @executeSQL           = 1

            , @printCommands        = 1

            , @debugMode            = 1

            , @printFragmentation   = 1

            , @forceRescan          = 1

            , @maxDopRestriction    = 1

            , @minPageCount         = 8

            , @maxPageCount         = Null

            , @minFragmentation     = 1

            , @rebuildThreshold     = 30

            , @defragDelay          = '00:00:05'

            , @defragOrderColumn    = 'page_count'

            , @defragSortOrder      = 'DESC'

            , @excludeMaxPartition  = 1

            , @timeLimit            = Null;

*********************************************************************************/                                                                

SET NOCOUNT ON;

SET XACT_Abort ON;

SET Quoted_Identifier ON;

 

BEGIN

 

    BEGIN Try

 

        /* Just a little validation... */

        IF @minFragmentation IS Null 

            Or @minFragmentation Not Between 0.00 And 100.0

                SET @minFragmentation = 10.0;

 

        IF @rebuildThreshold IS Null

            Or @rebuildThreshold Not Between 0.00 And 100.0

                SET @rebuildThreshold = 30.0;

 

        IF @defragDelay Not Like '00:[0-5][0-9]:[0-5][0-9]'

            SET @defragDelay = '00:00:05';

 

        IF @defragOrderColumn IS Null

            Or @defragOrderColumn Not In ('range_scan_count', 'fragmentation', 'page_count')

                SET @defragOrderColumn = 'range_scan_count';

 

        IF @defragSortOrder IS Null

            Or @defragSortOrder Not In ('ASC', 'DESC')

                SET @defragSortOrder = 'DESC';

 

        IF @scanMode Not In ('LIMITED', 'SAMPLED', 'DETAILED')

            SET @scanMode = 'LIMITED';

 

        IF @debugMode IS Null

            SET @debugMode = 0;

 

        IF @forceRescan IS Null

            SET @forceRescan = 0;

 

        IF @sortInTempDB IS Null

            SET @sortInTempDB = 1;

 

 

        IF @debugMode = 1 RAISERROR('Undusting the cogs and starting up...', 0, 42) WITH NoWait;

 

        /* Declare our variables */

        DECLARE   @objectID                 INT

                , @databaseID               INT

                , @databaseName             NVARCHAR(128)

                , @indexID                  INT

                , @partitionCount           BIGINT

                , @schemaName               NVARCHAR(128)

                , @objectName               NVARCHAR(128)

                , @indexName                NVARCHAR(128)

                , @partitionNumber          SMALLINT

                , @fragmentation            FLOAT

                , @pageCount                INT

                , @sqlCommand               NVARCHAR(4000)

                , @rebuildCommand           NVARCHAR(200)

                , @dateTimeStart            DATETIME

                , @dateTimeEnd              DATETIME

                , @containsLOB              BIT

                , @editionCheck             BIT

                , @debugMessage             NVARCHAR(4000)

                , @updateSQL                NVARCHAR(4000)

                , @partitionSQL             NVARCHAR(4000)

                , @partitionSQL_Param       NVARCHAR(1000)

                , @LOB_SQL                  NVARCHAR(4000)

                , @LOB_SQL_Param            NVARCHAR(1000)

                , @indexDefrag_id           INT

                , @startDateTime            DATETIME

                , @endDateTime              DATETIME

                , @getIndexSQL              NVARCHAR(4000)

                , @getIndexSQL_Param        NVARCHAR(4000)

                , @allowPageLockSQL         NVARCHAR(4000)

                , @allowPageLockSQL_Param   NVARCHAR(4000)

                , @allowPageLocks           INT

                , @excludeMaxPartitionSQL   NVARCHAR(4000);

 

        /* Initialize our variables */

        SELECT @startDateTime = GETDATE()

            , @endDateTime = DATEADD(MINUTE, @timeLimit, GETDATE());

 

        /* Create our temporary tables */

        CREATE TABLE #databaseList

        (

              databaseID        INT

            , databaseName      VARCHAR(128)

            , scanStatus        BIT

        );

 

        CREATE TABLE #processor 

        (

              [INDEX]           INT

            , Name              VARCHAR(128)

            , Internal_Value    INT

            , Character_Value   INT

        );

 

        CREATE TABLE #maxPartitionList

        (

              databaseID        INT

            , objectID          INT

            , indexID           INT

            , maxPartition      INT

        );

 

        IF @debugMode = 1 RAISERROR('Beginning validation...', 0, 42) WITH NoWait;

 

        /* Make sure we're not exceeding the number of processors we have available */

        INSERT INTO #processor

        EXECUTE XP_MSVER 'ProcessorCount';

 

        IF @maxDopRestriction IS Not Null And @maxDopRestriction > (SELECT Internal_Value FROM #processor)

            SELECT @maxDopRestriction = Internal_Value

            FROM #processor;

 

        /* Check our server version; 1804890536 = Enterprise, 610778273 = Enterprise Evaluation, -2117995310 = Developer */

        IF (SELECT SERVERPROPERTY('EditionID')) In (1804890536, 610778273, -2117995310) 

            SET @editionCheck = 1 -- supports online rebuilds

        ELSE

            SET @editionCheck = 0; -- does not support online rebuilds

 

        /* Output the parameters we're working with */

        IF @debugMode = 1 

        BEGIN

 

            SELECT @debugMessage = 'Your selected parameters are... 

            Defrag indexes with fragmentation greater than ' + CAST(@minFragmentation AS VARCHAR(10)) + ';

            Rebuild indexes with fragmentation greater than ' + CAST(@rebuildThreshold AS VARCHAR(10)) + ';

            You' + CASE WHEN @executeSQL = 1 THEN ' DO' ELSE ' DO NOT' END + ' want the commands to be executed automatically; 

            You want to defrag indexes in ' + @defragSortOrder + ' order of the ' + UPPER(@defragOrderColumn) + ' value;

            You have' + CASE WHEN @timeLimit IS Null THEN ' not specified a time limit;' ELSE ' specified a time limit of ' 

                + CAST(@timeLimit AS VARCHAR(10)) END + ' minutes;

            ' + CASE WHEN @DATABASE IS Null THEN 'ALL databases' ELSE 'The ' + @DATABASE + ' database' END + ' will be defragged;

            ' + CASE WHEN @tableName IS Null THEN 'ALL tables' ELSE 'The ' + @tableName + ' table' END + ' will be defragged;

            We' + CASE WHEN Exists(SELECT TOP 1 * FROM dbo.dba_indexDefragStatus WHERE defragDate IS Null)

                And @forceRescan <> 1 THEN ' WILL NOT' ELSE ' WILL' END + ' be rescanning indexes;

            The scan will be performed in ' + @scanMode + ' mode;

            You want to limit defrags to indexes with' + CASE WHEN @maxPageCount IS Null THEN ' more than ' 

                + CAST(@minPageCount AS VARCHAR(10)) ELSE

                ' between ' + CAST(@minPageCount AS VARCHAR(10))

                + ' and ' + CAST(@maxPageCount AS VARCHAR(10)) END + ' pages;

            Indexes will be defragged' + CASE WHEN @editionCheck = 0 Or @onlineRebuild = 0 THEN ' OFFLINE;' ELSE ' ONLINE;' END + '

            Indexes will be sorted in' + CASE WHEN @sortInTempDB = 0 THEN ' the DATABASE' ELSE ' TEMPDB;' END + '

            Defrag operations will utilize ' + CASE WHEN @editionCheck = 0 Or @maxDopRestriction IS Null 

                THEN 'system defaults for processors;' 

                ELSE CAST(@maxDopRestriction AS VARCHAR(2)) + ' processors;' END + '

            You' + CASE WHEN @printCommands = 1 THEN ' DO' ELSE ' DO NOT' END + ' want to print the ALTER INDEX commands; 

            You' + CASE WHEN @printFragmentation = 1 THEN ' DO' ELSE ' DO NOT' END + ' want to output fragmentation levels; 

            You want to wait ' + @defragDelay + ' (hh:mm:ss) between defragging indexes;

            You want to run in' + CASE WHEN @debugMode = 1 THEN ' DEBUG' ELSE ' SILENT' END + ' mode.';

 

            RAISERROR(@debugMessage, 0, 42) WITH NoWait;

 

        END;

 

        IF @debugMode = 1 RAISERROR('Grabbing a list of our databases...', 0, 42) WITH NoWait;

 

        /* Retrieve the list of databases to investigate */

        INSERT INTO #databaseList

        SELECT database_id

            , name

            , 0 -- not scanned yet for fragmentation

        FROM sys.databases

        WHERE name = IsNull(@DATABASE, name)

            And [name] Not In ('master', 'tempdb')-- exclude system databases

            And [STATE] = 0 -- state must be ONLINE

            And is_read_only = 0;  -- cannot be read_only

 

        /* Check to see if we have indexes in need of defrag; otherwise, re-scan the database(s) */

        IF Not Exists(SELECT TOP 1 * FROM dbo.dba_indexDefragStatus WHERE defragDate IS Null)

            Or @forceRescan = 1

        BEGIN

 

            /* Truncate our list of indexes to prepare for a new scan */

            TRUNCATE TABLE dbo.dba_indexDefragStatus;

 

            IF @debugMode = 1 RAISERROR('Looping through our list of databases and checking for fragmentation...', 0, 42) WITH NoWait;

 

            /* Loop through our list of databases */

            WHILE (SELECT COUNT(*) FROM #databaseList WHERE scanStatus = 0) > 0

            BEGIN

 

                SELECT TOP 1 @databaseID = databaseID

                FROM #databaseList

                WHERE scanStatus = 0;

 

                SELECT @debugMessage = '  working on ' + DB_NAME(@databaseID) + '...';

 

                IF @debugMode = 1

                    RAISERROR(@debugMessage, 0, 42) WITH NoWait;

 

               /* Determine which indexes to defrag using our user-defined parameters */

                INSERT INTO dbo.dba_indexDefragStatus

                (

                      databaseID

                    , databaseName

                    , objectID

                    , indexID

                    , partitionNumber

                    , fragmentation

                    , page_count

                    , range_scan_count

                    , scanDate

                )

                SELECT

                      ps.database_id AS 'databaseID'

                    , QUOTENAME(DB_NAME(ps.database_id)) AS 'databaseName'

                    , ps.OBJECT_ID AS 'objectID'

                    , ps.index_id AS 'indexID'

                    , ps.partition_number AS 'partitionNumber'

                    , SUM(ps.avg_fragmentation_in_percent) AS 'fragmentation'

                    , SUM(ps.page_count) AS 'page_count'

                    , os.range_scan_count

                    , GETDATE() AS 'scanDate'

                FROM sys.dm_db_index_physical_stats(@databaseID, OBJECT_ID(@tableName), Null , Null, @scanMode) AS ps

                Join sys.dm_db_index_operational_stats(@databaseID, OBJECT_ID(@tableName), Null , Null) AS os

                    ON ps.database_id = os.database_id

                    And ps.OBJECT_ID = os.OBJECT_ID

                    and ps.index_id = os.index_id

                    And ps.partition_number = os.partition_number

                WHERE avg_fragmentation_in_percent >= @minFragmentation 

                    And ps.index_id > 0 -- ignore heaps

                    And ps.page_count > @minPageCount 

                    And ps.index_level = 0 -- leaf-level nodes only, supports @scanMode

                GROUP BY ps.database_id 

                    , QUOTENAME(DB_NAME(ps.database_id)) 

                    , ps.OBJECT_ID 

                    , ps.index_id 

                    , ps.partition_number 

                    , os.range_scan_count

                OPTION (MaxDop 2);

 

                /* Do we want to exclude right-most populated partition of our partitioned indexes? */

                IF @excludeMaxPartition = 1

                BEGIN

 

                    SET @excludeMaxPartitionSQL = '

                        Select ' + CAST(@databaseID AS VARCHAR(10)) + ' As [databaseID]

                            , [object_id]

                            , index_id

                            , Max(partition_number) As [maxPartition]

                        From ' + DB_NAME(@databaseID) + '.sys.partitions

                        Where partition_number > 1

                            And [rows] > 0

                        Group By object_id

                            , index_id;';

 

                    INSERT INTO #maxPartitionList

                    EXECUTE SP_EXECUTESQL @excludeMaxPartitionSQL;

 

                END;

 

                /* Keep track of which databases have already been scanned */

                UPDATE #databaseList

                SET scanStatus = 1

                WHERE databaseID = @databaseID;

 

            END

 

            /* We don't want to defrag the right-most populated partition, so

               delete any records for partitioned indexes where partition = Max(partition) */

            IF @excludeMaxPartition = 1

            BEGIN

 

                DELETE ids

                FROM dbo.dba_indexDefragStatus AS ids

                Join #maxPartitionList AS mpl

                    ON ids.databaseID = mpl.databaseID

                    And ids.objectID = mpl.objectID

                    And ids.indexID = mpl.indexID

                    And ids.partitionNumber = mpl.maxPartition;

 

            END;

 

            /* Update our exclusion mask for any index that has a restriction on the days it can be defragged */

            UPDATE ids

            SET ids.exclusionMask = ide.exclusionMask

            FROM dbo.dba_indexDefragStatus AS ids

            Join dbo.dba_indexDefragExclusion AS ide

                ON ids.databaseID = ide.databaseID

                And ids.objectID = ide.objectID

                And ids.indexID = ide.indexID;

 

        END

 

        SELECT @debugMessage = 'Looping through our list... there are ' + CAST(COUNT(*) AS VARCHAR(10)) + ' indexes to defrag!'

        FROM dbo.dba_indexDefragStatus

        WHERE defragDate IS Null

            And page_count Between @minPageCount And IsNull(@maxPageCount, page_count);

 

        IF @debugMode = 1 RAISERROR(@debugMessage, 0, 42) WITH NoWait;

 

        /* Begin our loop for defragging */

        WHILE (SELECT COUNT(*) 

               FROM dbo.dba_indexDefragStatus 

               WHERE (

                           (@executeSQL = 1 And defragDate IS Null) 

                        Or (@executeSQL = 0 And defragDate IS Null And printStatus = 0)

                     )

                And exclusionMask & POWER(2, DATEPART(weekday, GETDATE())-1) = 0

                And page_count Between @minPageCount And IsNull(@maxPageCount, page_count)) > 0

        BEGIN

 

            /* Check to see if we need to exit our loop because of our time limit */        

            IF IsNull(@endDateTime, GETDATE()) < GETDATE()

            BEGIN

                RAISERROR('Our time limit has been exceeded!', 11, 42) WITH NoWait;

            END;

 

            IF @debugMode = 1 RAISERROR('  Picking an index to beat into shape...', 0, 42) WITH NoWait;

 

            /* Grab the index with the highest priority, based on the values submitted; 

               Look at the exclusion mask to ensure it can be defragged today */

            SET @getIndexSQL = N'

            Select Top 1 

                  @objectID_Out         = objectID

                , @indexID_Out          = indexID

                , @databaseID_Out       = databaseID

                , @databaseName_Out     = databaseName

                , @fragmentation_Out    = fragmentation

                , @partitionNumber_Out  = partitionNumber

                , @pageCount_Out        = page_count

            From dbo.dba_indexDefragStatus

            Where defragDate Is Null ' 

                + CASE WHEN @executeSQL = 0 THEN 'And printStatus = 0' ELSE '' END + '

                And exclusionMask & Power(2, DatePart(weekday, GetDate())-1) = 0

                And page_count Between @p_minPageCount and IsNull(@p_maxPageCount, page_count)

            Order By + ' + @defragOrderColumn + ' ' + @defragSortOrder;

 

            SET @getIndexSQL_Param = N'@objectID_Out        int OutPut

                                     , @indexID_Out         int OutPut

                                     , @databaseID_Out      int OutPut

                                     , @databaseName_Out    nvarchar(128) OutPut

                                     , @fragmentation_Out   int OutPut

                                     , @partitionNumber_Out int OutPut

                                     , @pageCount_Out       int OutPut

                                     , @p_minPageCount      int

                                     , @p_maxPageCount      int';

 

            EXECUTE SP_EXECUTESQL @getIndexSQL

                , @getIndexSQL_Param

                , @p_minPageCount       = @minPageCount

                , @p_maxPageCount       = @maxPageCount

                , @objectID_Out         = @objectID OUTPUT

                , @indexID_Out          = @indexID OUTPUT

                , @databaseID_Out       = @databaseID OUTPUT

                , @databaseName_Out     = @databaseName OUTPUT

                , @fragmentation_Out    = @fragmentation OUTPUT

                , @partitionNumber_Out  = @partitionNumber OUTPUT

                , @pageCount_Out        = @pageCount OUTPUT;

 

            IF @debugMode = 1 RAISERROR('  Looking up the specifics for our index...', 0, 42) WITH NoWait;

 

            /* Look up index information */

            SELECT @updateSQL = N'Update ids

                Set schemaName = QuoteName(s.name)

                    , objectName = QuoteName(o.name)

                    , indexName = QuoteName(i.name)

                From dbo.dba_indexDefragStatus As ids

                Inner Join ' + @databaseName + '.sys.objects As o

                    On ids.objectID = o.object_id

                Inner Join ' + @databaseName + '.sys.indexes As i

                    On o.object_id = i.object_id

                    And ids.indexID = i.index_id

                Inner Join ' + @databaseName + '.sys.schemas As s

                    On o.schema_id = s.schema_id

                Where o.object_id = ' + CAST(@objectID AS VARCHAR(10)) + '

                    And i.index_id = ' + CAST(@indexID AS VARCHAR(10)) + '

                    And i.type > 0

                    And ids.databaseID = ' + CAST(@databaseID AS VARCHAR(10));

 

            EXECUTE SP_EXECUTESQL @updateSQL;

 

            /* Grab our object names */

            SELECT @objectName  = objectName

                , @schemaName   = schemaName

                , @indexName    = indexName

            FROM dbo.dba_indexDefragStatus

            WHERE objectID = @objectID

                And indexID = @indexID

                And databaseID = @databaseID;

 

            IF @debugMode = 1 RAISERROR('  Grabbing the partition count...', 0, 42) WITH NoWait;

 

            /* Determine if the index is partitioned */

            SELECT @partitionSQL = 'Select @partitionCount_OUT = Count(*)

                                        From ' + @databaseName + '.sys.partitions

                                        Where object_id = ' + CAST(@objectID AS VARCHAR(10)) + '

                                            And index_id = ' + CAST(@indexID AS VARCHAR(10)) + ';'

                , @partitionSQL_Param = '@partitionCount_OUT int OutPut';

 

            EXECUTE SP_EXECUTESQL @partitionSQL, @partitionSQL_Param, @partitionCount_OUT = @partitionCount OUTPUT;

 

            IF @debugMode = 1 RAISERROR('  Seeing if there are any LOBs to be handled...', 0, 42) WITH NoWait;

 

            /* Determine if the table contains LOBs */

            SELECT @LOB_SQL = ' Select @containsLOB_OUT = Count(*)

                                From ' + @databaseName + '.sys.columns With (NoLock) 

                                Where [object_id] = ' + CAST(@objectID AS VARCHAR(10)) + '

                                   And (system_type_id In (34, 35, 99)

                                            Or max_length = -1);'

                                /*  system_type_id --> 34 = image, 35 = text, 99 = ntext

                                    max_length = -1 --> varbinary(max), varchar(max), nvarchar(max), xml */

                    , @LOB_SQL_Param = '@containsLOB_OUT int OutPut';

 

            EXECUTE SP_EXECUTESQL @LOB_SQL, @LOB_SQL_Param, @containsLOB_OUT = @containsLOB OUTPUT;

 

            IF @debugMode = 1 RAISERROR('  Checking for indexes that do not allow page locks...', 0, 42) WITH NoWait;

 

            /* Determine if page locks are allowed; for those indexes, we need to always rebuild */

            SELECT @allowPageLockSQL = 'Select @allowPageLocks_OUT = Count(*)

                                        From ' + @databaseName + '.sys.indexes

                                        Where object_id = ' + CAST(@objectID AS VARCHAR(10)) + '

                                            And index_id = ' + CAST(@indexID AS VARCHAR(10)) + '

                                            And Allow_Page_Locks = 0;'

                , @allowPageLockSQL_Param = '@allowPageLocks_OUT int OutPut';

 

            EXECUTE SP_EXECUTESQL @allowPageLockSQL, @allowPageLockSQL_Param, @allowPageLocks_OUT = @allowPageLocks OUTPUT;

 

            IF @debugMode = 1 RAISERROR('  Building our SQL statements...', 0, 42) WITH NoWait;

 

            /* If there's not a lot of fragmentation, or if we have a LOB, we should reorganize */

            IF (@fragmentation < @rebuildThreshold Or @containsLOB >= 1 Or @partitionCount > 1)

                And @allowPageLocks = 0

            BEGIN

 

                SET @sqlCommand = N'Alter Index ' + @indexName + N' On ' + @databaseName + N'.' 

                                    + @schemaName + N'.' + @objectName + N' ReOrganize';

 

                /* If our index is partitioned, we should always reorganize */

                IF @partitionCount > 1

                    SET @sqlCommand = @sqlCommand + N' Partition = ' 

                                    + CAST(@partitionNumber AS NVARCHAR(10));

 

            END

            /* If the index is heavily fragmented and doesn't contain any partitions or LOB's, 

               or if the index does not allow page locks, rebuild it */

            ELSE IF (@fragmentation >= @rebuildThreshold Or @allowPageLocks <> 0)

                And IsNull(@containsLOB, 0) != 1 And @partitionCount <= 1

            BEGIN

 

                /* Set online rebuild options; requires Enterprise Edition */

                IF @onlineRebuild = 1 And @editionCheck = 1 

                    SET @rebuildCommand = N' Rebuild With (Online = On';

                ELSE

                    SET @rebuildCommand = N' Rebuild With (Online = Off';

 

                /* Set sort operation preferences */

                IF @sortInTempDB = 1 

                    SET @rebuildCommand = @rebuildCommand + N', Sort_In_TempDB = On';

                ELSE

                    SET @rebuildCommand = @rebuildCommand + N', Sort_In_TempDB = Off';

 

                /* Set processor restriction options; requires Enterprise Edition */

                IF @maxDopRestriction IS Not Null And @editionCheck = 1

                    SET @rebuildCommand = @rebuildCommand + N', MaxDop = ' + CAST(@maxDopRestriction AS VARCHAR(2)) + N')';

                ELSE

                    SET @rebuildCommand = @rebuildCommand + N')';

 

                SET @sqlCommand = N'Alter Index ' + @indexName + N' On ' + @databaseName + N'.'

                                + @schemaName + N'.' + @objectName + @rebuildCommand;

 

            END

            ELSE

                /* Print an error message if any indexes happen to not meet the criteria above */

                IF @printCommands = 1 Or @debugMode = 1

                    RAISERROR('We are unable to defrag this index.', 0, 42) WITH NoWait;

 

            /* Are we executing the SQL?  If so, do it */

            IF @executeSQL = 1

            BEGIN

 

                SET @debugMessage = 'Executing: ' + @sqlCommand;

 

                /* Print the commands we're executing if specified to do so */

                IF @printCommands = 1 Or @debugMode = 1

                    RAISERROR(@debugMessage, 0, 42) WITH NoWait;

 

                /* Grab the time for logging purposes */

                SET @dateTimeStart  = GETDATE();

 

                /* Log our actions */

                INSERT INTO dbo.dba_indexDefragLog

                (

                      databaseID

                    , databaseName

                    , objectID

                    , objectName

                    , indexID

                    , indexName

                    , partitionNumber

                    , fragmentation

                    , page_count

                    , dateTimeStart

                    , sqlStatement

                )

                SELECT

                      @databaseID

                    , @databaseName

                    , @objectID

                    , @objectName

                    , @indexID

                    , @indexName

                    , @partitionNumber

                    , @fragmentation

                    , @pageCount

                    , @dateTimeStart

                    , @sqlCommand;

 

                SET @indexDefrag_id = SCOPE_IDENTITY();

 

                /* Wrap our execution attempt in a try/catch and log any errors that occur */

                BEGIN Try

 

                    /* Execute our defrag! */

                    EXECUTE SP_EXECUTESQL @sqlCommand;

                    SET @dateTimeEnd = GETDATE();

 

                    /* Update our log with our completion time */

                    UPDATE dbo.dba_indexDefragLog

                    SET dateTimeEnd = @dateTimeEnd

                        , durationSeconds = DATEDIFF(SECOND, @dateTimeStart, @dateTimeEnd)

                    WHERE indexDefrag_id = @indexDefrag_id;

 

                END Try

                BEGIN Catch

 

                    /* Update our log with our error message */

                    UPDATE dbo.dba_indexDefragLog

                    SET dateTimeEnd = GETDATE()

                        , durationSeconds = -1

                        , errorMessage = Error_Message()

                    WHERE indexDefrag_id = @indexDefrag_id;

 

                    IF @debugMode = 1 

                        RAISERROR('  An error has occurred executing this command! Please review the dba_indexDefragLog table for details.'

                            , 0, 42) WITH NoWait;

 

                END Catch

 

                /* Just a little breather for the server */

                WAITFOR Delay @defragDelay;

 

                UPDATE dbo.dba_indexDefragStatus

                SET defragDate = GETDATE()

                    , printStatus = 1

                WHERE databaseID       = @databaseID

                  And objectID         = @objectID

                  And indexID          = @indexID

                  And partitionNumber  = @partitionNumber;

 

            END

            ELSE

            /* Looks like we're not executing, just printing the commands */

            BEGIN

                IF @debugMode = 1 RAISERROR('  Printing SQL statements...', 0, 42) WITH NoWait;

 

                IF @printCommands = 1 Or @debugMode = 1 

                    PRINT IsNull(@sqlCommand, 'error!');

 

                UPDATE dbo.dba_indexDefragStatus

                SET printStatus = 1

                WHERE databaseID       = @databaseID

                  And objectID         = @objectID

                  And indexID          = @indexID

                  And partitionNumber  = @partitionNumber;

            END

 

        END

 

        /* Do we want to output our fragmentation results? */

        IF @printFragmentation = 1

        BEGIN

 

            IF @debugMode = 1 RAISERROR('  Displaying a summary of our action...', 0, 42) WITH NoWait;

 

            SELECT databaseID

                , databaseName

                , objectID

                , objectName

                , indexID

                , indexName

                , partitionNumber

                , fragmentation

                , page_count

                , range_scan_count

            FROM dbo.dba_indexDefragStatus

            WHERE defragDate >= @startDateTime

            ORDER BY defragDate;

 

        END;

 

    END Try

    BEGIN Catch

 

        SET @debugMessage = Error_Message() + ' (Line Number: ' + CAST(Error_Line() AS VARCHAR(10)) + ')';

        PRINT @debugMessage;

 

    END Catch;

 

    /* When everything is said and done, make sure to get rid of our temp table */

    DROP TABLE #databaseList;

    DROP TABLE #processor;

    DROP TABLE #maxPartitionList;

 

    IF @debugMode = 1 RAISERROR('DONE!  Thank you for taking care of your indexes!  :)', 0, 42) WITH NoWait;

 

    SET NOCOUNT OFF;

    RETURN 0

END

.csharpcode, .csharpcode pre
{
font-size: small;
color: black;
font-family: consolas, “Courier New”, courier, monospace;
background-color: #ffffff;
/*white-space: pre;*/
}
.csharpcode pre { margin: 0em; }
.csharpcode .rem { color: #008000; }
.csharpcode .kwrd { color: #0000ff; }
.csharpcode .str { color: #006080; }
.csharpcode .op { color: #0000c0; }
.csharpcode .preproc { color: #cc6633; }
.csharpcode .asp { background-color: #ffff00; }
.csharpcode .html { color: #800000; }
.csharpcode .attr { color: #ff0000; }
.csharpcode .alt
{
background-color: #f4f4f4;
width: 100%;
margin: 0em;
}
.csharpcode .lnum { color: #606060; }

 

 

.csharpcode, .csharpcode pre
{
font-size: small;
color: black;
font-family: consolas, “Courier New”, courier, monospace;
background-color: #ffffff;
/*white-space: pre;*/
}
.csharpcode pre { margin: 0em; }
.csharpcode .rem { color: #008000; }
.csharpcode .kwrd { color: #0000ff; }
.csharpcode .str { color: #006080; }
.csharpcode .op { color: #0000c0; }
.csharpcode .preproc { color: #cc6633; }
.csharpcode .asp { background-color: #ffff00; }
.csharpcode .html { color: #800000; }
.csharpcode .attr { color: #ff0000; }
.csharpcode .alt
{
background-color: #f4f4f4;
width: 100%;
margin: 0em;
}
.csharpcode .lnum { color: #606060; }

 

.csharpcode, .csharpcode pre
{
font-size: small;
color: black;
font-family: consolas, “Courier New”, courier, monospace;
background-color: #ffffff;
/*white-space: pre;*/
}
.csharpcode pre { margin: 0em; }
.csharpcode .rem { color: #008000; }
.csharpcode .kwrd { color: #0000ff; }
.csharpcode .str { color: #006080; }
.csharpcode .op { color: #0000c0; }
.csharpcode .preproc { color: #cc6633; }
.csharpcode .asp { background-color: #ffff00; }
.csharpcode .html { color: #800000; }
.csharpcode .attr { color: #ff0000; }
.csharpcode .alt
{
background-color: #f4f4f4;
width: 100%;
margin: 0em;
}
.csharpcode .lnum { color: #606060; }

Posted in Uncategorized | Leave a comment

Using the SQL Scripting via the GUI for Admin Changes

If you’re like me, you like GUIs even for SQL Server, they make it easier to do things quickly without having to remember a lot of syntax.  But GUIs within Management Studio ultimately have to fire off TSQL to perform the changes.  And not knowing exactly what it is generating, short of running a profiler trace can be a little un-nerving when making changes on large mission-critical databases, especially when the GUI appears to hang.

Based on this, I’ve decided never to implement any important changes through the GUI.  That is probably common sense to most DBAs.  However, I find the GUI very useful for generating the SQL.  On virtually all of the dialogues within SQL Management Studio 2008, there is the little script button in the left-hand corner.  So, I typically use this to get my SQL and then just cancel the changes.

This gives you not only the advantage of being able to see what TSQL will actually execute when you hit the OK button, but the ability to save the script for later use, and more importantly gives you control over the execution, so you can cancel it rather than having the GUI timeout after a while on a long command or worse simply hang for ever with no recourse other than a forced exit from management studio.

I have found that in many cases, the SQL that is generated may not actually be what you thought you were going to get when you performed the function via the UI.  An example is changing auto-grow.  I am loading several million rows of data from an Intraday service this weekend into a new database and forgot to set the autogrow.  After trying a couple of times to do this via the UI only to be terminated by nasty messages about lock timeouts.  I could understand some contention due probably to the fact that about 6,500 rows per second are being inserted/updated as we speak (These are fully-logged upserts on an indexed/non-partitioned-yet partitioned table, not bulk-inserts – Did I tell you that I love SSDs…), but why would changing the autogrowth increment run into contention? 

Based on this, I decided to take a look at what TSQL was being generated:

Alas, here is the code generated just by changing JUST AUTOGROW increment and NOTHING else:

USE [TP_Public]
GO
DBCC SHRINKFILE (N’TP_Pub_IDFG03′ , 4500)
GO
USE [TP_Public]
GO
DBCC SHRINKFILE (N’TP_Pub_IDFG04′ , 4500)
GO
USE [master]
GO
ALTER DATABASE [TP_Public] MODIFY FILE ( NAME = N’TP_Pub_IDFG03′, FILEGROWTH = 1536000KB )
GO
ALTER DATABASE [TP_Public] MODIFY FILE ( NAME = N’TP_Pub_IDFG04′, FILEGROWTH = 1536000KB )
GO

Oops, where did that DBCC SHRINKFILE come from?

I just set AUTOGROWTH higher.  I wasn’t trying to shrink my database files…

Apparently the dialogue wants to lock in the size of the file that it is currently at and then do the AUTOGROW, maybe this is to make sure that if you specify a percent increase instead of a size increase, that it is accurate.  I don’t know what the engineers were thinking, but that would explain the lock timeout.

So, I just took off the DBCC SHRINKFILEs and ran just the ALTER DATABASE command and no worries, the auto growth increment was fixed quickly without impacting the load in process.

Posted in Uncategorized | Leave a comment

I love SSD

Been working with 2 PCIE Fusion cards testing out performance for my SQL Server simulation database.  I also have a raid set of 3 Intel X25-E SSDs.  This is a technology to take seriously.  The folks at Fusion have been most helpful.  I will be putting together more results over next couple of weeks, but the below spreadsheet outlines the case for PCIE based cards.  I am using 1 Fusion IODrive 320GB in a Dell 690, one SLC 160GB drive in a Dell R610 and have another 160GB SLC on the way to put into the R610.  Would love to try out a duo, but out of budget.   I was able to install Windows Server 2008 and fire up the O/S in about 4 minutes onto a VM under Hyper-V using the card.  I plan to do it again with 2 cards raid-0 together and put it on UTube – should finish in less than 2 minutes. 

These cards are a great option for 1U Servers.  The Dell 1U Server is fairly limited as far as card slots.  To run 2 cards, I had to remove the extra network card to make room for a second Fusion, since there is also a Perc Controller card in the system.  So, I had to choose between network redundancy and storage performance/redundancy in order to also have a fast onboard storage controller.  A better option is probably to use DUO cards for these servers or get them without the Perc controller and just use onboard for bootup/OS.  In any case, 1 Fusion SLC is going to outperform 6 SSDs for most operations, even if on paper the 6 SSDs together might look better because the controller becomes a bottleneck, even with SATA III, especially if you want to have RAID protection.  The nice thing about the Fusion card is that a RAID-5 protocol is built into the interface as parity chips are used for write operations.  For a production system, I would still probably want to RAID-1 2 cards together, to guard against a complete card failure.

For now, I have results in the attached spreadsheet from my old Dell 690 workstation showing IOMeter results for standard drive, multiple Intel SLC SSDs, and a Fusion 320 card running in “improved performance mode”.  As you can see when reaching higher queue depths, a single Fusion IO is roughly twice as quick as 3 SLC SSDs raided together and dozens of times faster than standard hard drives and about half as slow as an in-memory RAM disk.  In my stock simulation database, I am now loading intraday data, and have several million rows.  The other day, I added a new computed persisted field to one of the tables and it took less than 2 minutes to update over 3 million rows. 

That’s all I have time for now, will post more hopefully over next couple of weeks including results of using 2 Fusion-IO 160GB SLC cards in a R610 and some comparison differences with SQL Server running on the Fusion cards versus standard storage and the Intel SSDs.

Posted in Uncategorized | Leave a comment

Using a relative day index to enable correlative queries

I’m back after 2 months of non-posting – and with my usual nonsensical titles.  Today, we will have some real fun with a technique that helps in analyzing historical data.  We will use my favorite example, the stock market.  We’ll come up with some very interesting results by analyzing past history.  After reading this, you may rethink your investing strategy if you are a “buy-and-holder”…

One of my favorite things is

Posted in Uncategorized | Leave a comment