Should I virtualise my Domain Controllers ?

Now that’s a difficult question. If you asked me “Can I virtualise my Domain Controllers” then that’s a different question to which the answer is “Of course, its fully supported depending on your virtualisation platform and the version of Windows being used but if you’re on the latest Hyper-V and the latest Windows then its fine”. The question “Should I virtualise my Domain Controllers ?” recognises that you can but that you have a choice as to whether you do or not and, as with any IT decision, you should research, size and plan. What I’d like to talk about today is two items to consider when thinking of virtualising domain controllers.

The first is around synchronisation of system clocks. As mentioned in a previous article windows Servers use time synchronisation to ensure against replay attacks and thus increase the security of Kerberos authentication within an Active Directory environment. However, virtual platforms such as VMWare or Hyper-V also allow you to synchronise a virtual machines clock with the physical host. What this means though is that, if the server host is showing a different time from the root PDC Emulator then any virtualised domain member server or domain controller will set its clock against the domain and then set its clock against the physical host and then against the domain and then against the physical host and so on ad nauseum. This can cause five issues:

  1. If there is more than the amount of “difference” between the DC clock and other domain controller clocks then the server will not be able to synchronise
  2. Similarly, as the DC clock will different from those of clients, clients will fail authentication against this domain controller.
  3. This constant re-synchronisation will cause clock “flapping” so that any events or logs written will have events recorded in an incorrect order. This is an issue not only for domain controllers but also for other servers such as SQL or Exchange where they record the time of records being changed or messages arriving.
  4. If you run an environment where accurate times are important then this will into be possible with “flapping” clocks. For example, if you require staff to “clock in” and penalise them for late arrival then your solution will be at risk if your clock cannot keep accurate time.

So, by all means virtualise your domain controllers but don’t allow them to synchronise their clocks with the physical host. In Hyper-V this behaviour can be disabled by opening the Hyper-V Manager Console. selecting the virtual machine and clicking on Settings in the Actions pane for that virtual machine. Under the Management node select Integration Services and clear the Time Synchronization check box.

 

Click to enlarge

Click to enlarge

Click on Apply and that virtual machine will now synchronise its clock solely based on the settings within its operating system.

The second item to consider before virtualising your domain controllers concerns “snapshotting”. Snapshots allow you to take a point in time view of a server and then record differences to the virtual disk of that server over time. In this way you can “roll back” a virtual machine to the point the snap shot was taken by removing the changes made. However, this gives an issue when we consider domain controllers.

When a change is made on a Domain Controller it updates its own Update Sequence Number (USN) and, when a synchronisation is due with other domain controllers, issue the update to them. These USN’s are maintained per Domain Controller and a certain change may register on DC1 as 12345 and hold the USN of 7657622 on the far older DC2. You can see the USN on a particular Domain Controller by looking at the highestCommittedUSN value using ADSIEdit to connect to the RootDSE default naming context.

Click to enlarge

Click to enlarge

DC1 would look like above and DC2 would have the USN below, for example.
Click to enlarge

Click to enlarge

Now, it’s a basic premise that the USN on a domain controller should only ever get bigger, and not smaller. After all, transactions can’t just disappear. Indeed, domain controllers use this USN to keep track of the updates they have received from each other. The last USN received from each replicating partner is stored in a High Watermark Vector Table on each DC. In this way, the receiving domain controller knows which was the last change it received form a replicating partner. When it next wants to replicate it sends its high watermark value to the DC it wants to replicate from (the source domain controller). The source DC then uses the information in the high watermark value to determine which objects to replicate back to the target Domain Controller. This can be represented by the following table:

Step DC USN High Watermark Value Action
1 DC1 100 200 Initial Value
DC2 200 100
2 DC1 108 200 Changes made on DC1 (New user created for example)
DC2 200 100
3 DC1 108 200 DC2 requests changes, synchronises and updates it high watermark value for DC1
DC2 200 108
4 DC1 127 200 Further changes are made on DC1
DC2 200 108
5 DC1 127 200 Only changes 109 to 127 are synchronised to DC2
DC2 200 127

 

So far so good. So, what’s the issue. The issue is that if we had taken a snapshot of DC1 at, say, step 3 and rolled back then the following would happen.

Step DC USN High Watermark Value Action
1 DC1 100 200 Initial Value
DC2 200 100
2 DC1 108 200 Changes made on DC1 (New user created for example)
DC2 200 100
3 DC1 108 200 DC2 requests changes, synchronises and updates it high watermark value for DC1
DC2 200 108
4 DC1 127 200 Further changes are made on DC1
DC2 200 108
5 DC1 127 200 Only changes 109 to 127 are synchronised to DC2
DC2 200 127
6 DC1 108 200 Active Directory database “restored” on DC1
DC2 200 127
7 DC1 119 200 Further updates made on DC1 raising its USN past the old value of 127
DC2 200 127
8 DC1 147 200 DC2 requests changes past 127 – DC1 send changes 128 to 147 – the “new” changes in the range 109 to 127 are lost and never synchronised
DC2 200 127

 

So, by restoring Active Directory from a snapshot we would run the risk of losing updates IF Active Directory allowed us to do this. Fortunately the clever guys at Microsoft have worked this out and from Windows 2003 SP1 this is not likely to happen because AD will recognise that the USN’s have become out of sequence and will refuse to allow DC1 to synchronise. You will know if this has happened to you not only because your domain will not synchronise properly but you will see an event similar to the below logged in the event viewer on the “restored” Domain Controller.

Click to enlarge

Click to enlarge

 As you can see, the only solution for this is to forcibly demote the domain controller and start again. Of course, the situation is even worse if ALL domain controllers are snapshotted and then restored. It’s perfectly possible that you can end up without an operating Active Directory environment ! So, the original question was “Should I virtualise my Domain Controllers ?” and I say that this is a decision that you have to make yourself and the risk you want to assume. However, I would suggest that a best practice is to:

  • Never synchronise Domain Controller clocks with the virtualisation host
  • Never snapshot domain controllers
  • Always have at least one (and preferably two) physical domain controllers in case you have to force demote all virtualised domain controllers

If you follow the above advice I believe the risks in virutalising DC’s are relatively low.

One Response to “Should I virtualise my Domain Controllers ?”

  1. Kedai says:

    Nice information, thanks…
    But Plz fix your image, I can not open in my chrome browser…

Leave a Reply