MySpace Uses SQL Server Service Broker to Protect Integrity of 1 Petabyte of Data

July 26th, 2009

I just found that Microsoft has published a use case about the way MySpace is using Service Broker on their service as the core message delivery system for the Service Dispatcher. We’re talking here 440 SQL Server instances and over 1000 databases. Quote from the use case:

Service Broker has enabled MySpace to perform foreign key management across its 440 database servers, activating and deactivating accounts for its millions of users, with one-touch asynchronous efficiency. MySpace also uses Service Broker administratively to distribute new stored procedures and other updates across all 440 database servers through the Service Dispatcher infrastructure.

That is pretty impressive. I knew about the MySpace SSB adoption since the days when I was with the Service Broker team. You probably all know my mantra I repeat all the time “don’t use fire and forget, is a bad message exchange pattern and there are scenarios when the database may be taken offline”? Guess how I found out those ‘scenarios’… Anyway, I’m really glad that they also made public some performance numbers. Until now I could only quote the 5000 message per second I can push in my own test test environment. Well, looks like MySpace has some beefier hardware:

Stelzmuller: “When we went to the lab we brought our own workloads to ensure the quality of the testing. We needed to see if Service Broker could handle loads of 4,000 messages per second. Our testing found it could handle more than 18,000 messages a second.”

Fix slow application startup due to code sign validation

July 24th, 2009

Sometimes you are faced with applications that seem to take ages to start up. Usually they freeze for about 30-40 seconds and then all of the sudden they come to live. This happens for both native and managed application and it sometimes manifest as an IIS/ASP/ASP.Net AppPool starting up slow on the first request. The very first thing I always suspect is code signing verification. When a signed module is checked the certificate verification engine may consider that the Certificate Revocation List (CRL) it posses is obsolete and attempt to download a new one. For this it connects to the internet. The problem occurs when the connectivity is either slow, or blocked for some reason. By default the verification engine will time out after 15 seconds and resume with the old, obsolete, CRL it has. The timeout can occur several times, adding up to start up times of even minutes. This occurs completely outside of the control of the application being started, its modules are not even properly wired up in memory so there is no question of application code yet running.

The information on this subject is scarce to say the least. Luckily there is an TechNet article that describes not only the process occuring, but also the controlling parameters: Certificate Revocation and Status Checking. To fix the problem on computers with poor internet conectivity, registry settings have to be modified in the HKLM\SOFTWARE\Microsoft\Cryptography\OID\EncodingType 0\CertDllCreateCertificateChainEngine\Config key:

ChainUrlRetrievalTimeoutMilliseconds
This is each individual CRL check call timeout. If is 0 or not present the default value of 15 seconds is used. Change this timeout to a reasonable value like 200 milliseconds.
ChainRevAccumulativeUrlRetrievalTimeoutMilliseconds
This is the aggregate CRL retrieval timeout. If set to 0 or not present the default value of 20 seconds is used. Change this timeout to a value like 500 milliseconds.

With these two changes the code signing verification engine will timeout the CRL refresh operation in 500 milliseconds. If the connectivity to the certificate authority site is bad, this will dramatically increase the application start up times for code signed applications.

Detect network connectivity changes in .Net applications

July 21st, 2009

Update

Andrei Ignat pointed out there is a class in the .Net Framework that deals exactly with this issue: NetworkChange Class. That is definitely a better approach than leveraging the native API through p-Invoke. The MSDN sample also shows how to check for network availability on each interface, when notified of a change.

Original post:

A common problem for applications that use the network stack is to detect when they are connected to the network. It would be ideal if the application was notified when you plugged in your network cable into your laptop, or when it came in range of a WiFi station and has joined a wireless network.

The Windows API has a function for registering for exactly such notifications: NotifyAddrChange. An application can register a wait handle with this function and the handle will be signaled by the operating system every time a change occurs in the IP address to interface mapping table. Technically is true that ‘IP address table change’ does mean strictly that ‘network connectivity has changed’, but in reality the overwhelming majority of IP address table changes can be associated with the network connectivity being gained or lost.

To leverage this API from managed code we have to use p-Invoke. I use a helper class that wraps the registration for notification and handling the event notification in a simple to use package:

    /// <summary>
    /// Raises an event each time a change occurs 
    /// on the system IP address table.
    /// </summary>
    public class NotifyAddressChangesIDisposable
    {
        #region State fields
        /// <summary>
        /// The OVERLAPPED structure passed to NotifyAddrChange
        /// </summary>
        private NativeOverlapped _overlapped;

        /// <summary>
        /// File handle created  by NotifyAddrChange
        /// </summary>
        private IntPtr _handle;

        /// <summary>
        /// The event signaled by NotifyAddrChange
        /// </summary>
        private AutoResetEvent _eventChanged;

        /// <summary>
        /// The thread pool notification registration
        /// </summary>
        private RegisteredWaitHandle _registered;

        /// <summary>
        /// Submitted state tracking
        /// </summary>
        private volatile int _submitted;

        #endregion

        #region Win32 API declarations

        [DllImport("Iphlpapi.dll", SetLastError = true)]
        private static extern UInt32 NotifyAddrChange(
            ref IntPtr Handle, 
            ref NativeOverlapped overlapped);
        [DllImport("Iphlpapi.dll", SetLastError = true)]
        private static extern bool CancelIPChangeNotify(
            ref NativeOverlapped overlapped);
        #endregion

        #region Public usable methods and events

        /// <summary>
        /// The event raised when a change in the system
        /// IP address table occurs. 
        /// </summary>
        public event EventHandler AddressChanged;


        /// <summary>
        /// CTOR. Initializes the notifications event and submits it
        /// to the ThreadPool. Caller need to call Submit() to enable
        /// system IP address notifications to raise the AddressChange event.
        /// </summary>
        public NotifyAddressChanges()
        {
            _submitted = 0;
            _eventChanged = new AutoResetEvent(false);
            _overlapped.EventHandle = _eventChanged.SafeWaitHandle.DangerousGetHandle();

            _registered = ThreadPool.RegisterWaitForSingleObject(
                _eventChanged,
                new WaitOrTimerCallback(AddressChangedNotification),
                null,
                Timeout.Infinite,
                false);
        }

        /// <summary>
        /// Enables the system IP address notifications to raise the AddressChange event.
        /// </summary>
        public void Submit()
        {
            if (0 == Interlocked.CompareExchange(
                ref _submitted, 1, 0))
            {
                NotifyAddrChange(ref _handle, ref _overlapped);
            }
        }

        #endregion

        #region Callback and event raising

        /// <summary>
        /// IP address change notification callback.
        /// </summary>
        private void AddressChangedNotification(object state, bool timedOut)
        {
            if (1 == Interlocked.CompareExchange(
                ref _submitted, 0, 1))
            {
                EventHandler handler = AddressChanged;
                if (null != handler)
                {
                    try
                    {
                        handler(thisEventArgs.Empty);
                    }
                    catch (Exception)
                    {
                        // silent ignore
                    }
                }
                Submit();
            }
        }
        #endregion

        #region Dispose Logic
        private void Dispose(bool userInvoked)
        {
            if (null != _registered)
            {
                _registered.Unregister(_eventChanged);
            }
            if (1 == Interlocked.CompareExchange(
                ref _submitted, 0, 1))
            {
                CancelIPChangeNotify(ref _overlapped);
            }
        }

        ~NotifyAddressChanges()
        {
            Dispose(false);
        }


        public void Dispose()
        {
            Dispose(true);
            GC.SuppressFinalize(this);
        }

        #endregion
    }

To use this class, simply create an instance, add an event handler to the AddressChanged event and then call the Submit() method to start getting notifications. The AddressChanged event will be raised each time the system changes an IP address on any of its network interfaces. For example consider a simple form named Form1 to which I added a TextBox named textbox1 (in other words I created a new Windows forms application and I dropped a multiline text box on its surface, without bothering to rename either). Here is how I can hook up the IP address change event to display ‘changed’ on a new line in the text box:

    public partial class Form1 : Form
    {
        private NotifyAddressChanges _notification;

        public Form1()
        {
            InitializeComponent();

            _notification = new NotifyAddressChanges();
            _notification.AddressChanged += 
                new EventHandler(_notification_AddressChanged);
            _notification.Submit();
        }

        void _notification_AddressChanged(object sender, EventArgs e)
        {
            Invoke(new MethodInvoker(delegate ()
            {
                textBox1.AppendText("changed\r\n");
            }));
        }
    }

I can now start my test application and play with the network cable. As I unplug and plug back the cable, the system looses and re-gains the IP address from my base station DHCP server and it signals the NotifyAddrChange registered events. My form gets notified and it adds a “changed” line to the text box. I can also release and renew my DHCP license from the command line running ipconfig /release and ipconfig /renew.

Note that this notification occurs with any change, both on IP4 and IPv6 interfaces. You will be notified if a VPN is connected or disconnected, if a dial-up connection is established and so on and so forth. Being notified does not mean that network is available, nor that is unavailable. It simply means a change occurred. While I could add support to the AddressChanged event to indicate what change, that would needlessly complicate this simple class, because the number of possible changes is actually much bigger than one would expect (think again at all those VPN, dial-up, point-to-point and other connections). My goal with this class was to eliminate the need to periodically wake the application and check if the network is available every minute, based on a timer. I can simply wait for the AddressChanged notification and then I can try to see if I can establish a connection.

Inspiration is perishable

July 8th, 2009

I am following the 37signals blog ever since I kinda randomly stumbled upon their Getting Real book. If you never heard about them, definitely check out the book, is a very common sense approach to managing product development in the age of Internets. On this post http://www.37signals.com/svn/posts/1798-jasons-talk-at-big-omaha-2009 I was really touched by one remark: Inspiration is perishable. The ideas you have can linger in your head for a long time, but the inspiration for it fades quickly. So don’t postpone it, by the time you get to it you’ll only deliver a pale image of the original idea. Do it when you’re pumped up and thrilled by it.

I reckon I’m a procrastinator deLuxe, but I have to agree. I know the difference between working at 2 am. and not feeling a bit tired when I’m excited about my work on one hand, and the damp feeling of exhaustion that drags you to watch some stupid TV show at 6 pm because I’m bored with the current project on the other hand.

Laptop battery bulged out of its tray

July 7th, 2009

That is how my 17” Mac book Pro battery looks like after today it bulged out of its tray. Seems to be a fairly common problem with this batch.