The Imperfect Lab: Syncing AD to Azure AD

Today I decided to ease myself into my next steps and build out a member server to sync AD to. I reused some previous PowerShell to deploy a member server and join it to my domain. It is possible to run the sync services on an existing domain controller, but as a best practice I don’t like to install one-off applications on my domain controllers. I like to keep them identical, thus the need for different member server to perform the sync role.

I had previously uploaded the Microsoft Azure AD Sync Services (aka AADSync) application to my Azure file share, but you can find it at http://aka.ms/azureadsync. You will want to install and run the Microsoft Azure AD Connection Tool. Please note that Microsoft Azure AD Sync Services is DIFFERENT from Windows Azure Active Directory Sync (aka DirSync)

Once the Sync Server is built, you will want to kick off the installation of the application, but not before you’d made some adjustments to your Azure Directory. In the Portal, I went to my directory and created a new user account to be my Azure AD Administrator (newuser@imperfectlab.com) and made it a Global Administrator. You will also need to go through the sign-in process to set a non-temporary password.

Once you have this account, you simply need to throw the switch under “Directory Integration -> Directory Sync” from Inactive to Active. Once the setting is saved, the “Last Sync” field will say “never synced”. Now go over to your sync server and run that connection tool.

You’ll need the account and credentials you created for the new Azure AD Admin and some information about your domain. For the addition of the forest, you’ll need your domain name and the username and password of a enterprise domain admin from your local domain. This will be different than the account your created directly in Azure AD.

Leave the User Matching page at the defaults but select “Password Synchronization” from the Optional Features. Finally, review your configuration screen and verify that “Synchronize Now” is checked and click finish. At this point, your users should sync into Azure AD and after a few minutes you’ll see a list of them in the portal.

If you want to make any changes to the settings of your AD Sync, like adding in a feature, simply rerun the tool after disabling the Azure AD Sync Task in Task Scheduler. The task will be re-enabled automatically when you finish the wizard again.

If you want to force a sync for Azure AD Sync Services for any reason, the default location of the command line tool is:

c:\program files\microsoft azure ad sync\bin\directorysyncclientcmd [initial|delta]

Happy Syncing!

The Imperfect Lab: Adding A Custom Domain

This will be a super short post, because this task is super easy!

My lab in Azure wouldn’t be complete without its own custom domain. Honestly, this is one of those “just pop over to the Portal” tasks because it only takes a few clicks, particularly if you are only doing it once. But you won’t be able to complete in a hurry, because your registrar will update the public DNS entries on their own sweet time and that update is needed to complete the process.

By the way, if you really want to do this without the Portal, you can find information on installing the right PowerShell modules and the commands here. (http://msdn.microsoft.com/en-us/library/azure/jj151815.aspx) If you are going to managing multiple tenants over time, PowerShell will likely be the best way to go.

Anyway, when you are in the Portal, click “Active Directory” in the navigation. Select the domain directory you want to add a custom domain to. In this case, I wanted to create a new Azure Directory for the Imperfect Lab, so I clicked “New” and then went to APP SERVICES -> ACTIVE DIRECTORY -> DIRECTORY -> CUSTOM CREATE.

I named my directory “ImperfectLab” and picked my region. The domain name for the directory is now “Imperfectlab.onmicrosoft.com”. Since I don’t want to be using the “onmicrosoft.com” moniker for very long, I need to add my recently purchased domain. You actually have to a own (or at least control) the domain you want to add because it’s requirement to add a TXT or MX record to your public DNS.

Click into the directory you want to use and go to the “Domains” section. On the bottom action bar, click “Add”. Then type in the FQDN for your “real” domain, in my case “imperfectlab.com”. You be given the information to create either at TXT or MX record that needs to be added to your DNS records managed by your registrar.

My registrar doesn’t accept the @ symbol for the parent zone, but leaving that field blank worked fine. You have to add the record, wait for the external DNS to update and then return to the portal to verify it.

Once verified, you can create (or sync) users into your Azure Active Directory using either your “user@domain.onmicrosoft.com” UPN or your “user@domain.com” UPN.

The Imperfect Lab: Fleshing Out Active Directory

Having a domain with no users isn’t any fun. So my next task for the Imperfect Lab was to create a few accounts to act as my users for provisioning access and eventually syncing with Azure Active Directory.

You can do a lot with some basic PowerShell to create OUs and User Accounts. Here are a few basic lines that would create something in my lab domain:

New-ADOrganizationalUnit –Name “DOGS” –Path “DC=imperfectlab, DC=Com”

New-ADUser -Name “Lizbeth Tiburon” -Path “OU=DOGS,dc=imperfectlab,dc=com” -AccountPassword $newPassword -Department “Career Changed” -SamAccountName “LTibu” -Surname “Tiburon” -GivenName “Lizbeth” -DisplayName “Lizbeth Tiburon”

Those lines would create a OU and then a user account in the new OU. But what if you wanted to create more users at once? I could simply duplicate the 2nd line, but figured there had to a relatively easy way to get data straight from a CSV file.

I did some looking around online and since no good Internet search goes unpunished, I found this: https://gallery.technet.microsoft.com/scriptcenter/PowerShell-Create-Active-7e6a3978#contentby @mwashamtx. Honestly, this a great script that I couldn’t have written by myself at this point, but I was able to tweak it enough to do my bidding.

I changed the paths (to reflect the drive letter and file location I set up using Azure Files), removed a lot of the fields the script used to populate account attributes and edited the CSV file to match. I uploaded my CSV file to my Azure file share. I left the script writer’s five character SAM account name creation as is and ran it remotely via PS-Session on my domain controller. The DC tapped the CSV file in my Azure File share and wrote the log to that same location. The script does some great error handling, which was really helpful for troubleshooting. Mission accomplished!

And for those of you who are curious about the user created in that line above, Lizbeth is a dog who didn’t complete the training to become a guide dog.

The Imperfect Lab: More DCs and Static IPs

When I was last working in my Imperfect Lab, I added another server to the existing cloud service and decided to make it a domain controller. When you set up domain controllers (cloud or on-premises) a few things become really important – IP Addresses and DNS.

By default, Azure will provide DNS services from the fabric if you don’t specify your own DNS. You would think there is some PowerShell to do that directly, but surprisingly there isn’t. You can set the DNS for each network using the Management Portal or by exporting the network configuration file and updating it. I just used the portal and made sure that my ImperfectNet listed the IP address for both servers that would act as domain controllers.

If you don’t set a domain controller as the DNS server, all the VMs that come up inside your virtual network will look to an Azure fabric DNS server and won’t be able to authenticate to your domain. Since this is a crucial to AD function, I also wanted to make sure that the VMs that were acting as domain controllers had static internal IP addresses.

Now, these addresses aren’t really “static” on the OS. They are more like DHCP reservations handed out from the fabric manager. But the end result is the same – VMs that have the correct IP address, regardless of the order they are started.

To do this with PowerShell, you first need to have the VMs in the Stopped (Deallocated) state. This way the addresses are free to assign. If the VM is already running, the address is allocated already, thus can’t be assigned. You can double check that an address is free with:

Test-AzureStaticVNetIP –VNetName ImperfectNet –IPAddress 192.168.1.5

To set the static address, I used:

Get-AzureVM -ServiceName ImperfectCore -Name DC-Cloud1 | Set-AzureStaticVNetIP -IPAddress “192.168.1.4” | Update-AzureVM

Take note of the use of quotes around the IP address in that last line. It matters. I don’t know why. Just trust that I wasted a lot of time on your behalf for that knowledge.

Then to finally kick off the addition of my second domain controller in this domain, I used:

Install-ADDSDomainController -Credential (Get-Credential) -DatabasePath ‘C:\Windows\NTDS’ -DomainName ‘imperfectlab.com’ -InstallDns:$true -LogPath ‘C:\Windows\NTDS’ -NoGlobalCatalog:$false -SiteName ‘ImperfectNet’ -SysvolPath ‘C:\Windows\SYSVOL’ -NoRebootOnCompletion:$true -Force:$true -Verbose

One note about the paths used for the logs and SYSVOL… I’ve left them on C:\ for convenience, but for production, you will want to set up your DCs in Azure with an additional disk where you direct those files to go. Read more about the reason behind that best practice here.

Also, if this Domain Controller happens to connect back to an on-premises domain. Be sure to make the proper changes to you AD Sites and Services to ensure proper site topology.

Update (12/26/14): For easy access to code snippets, you can find them here.

Throwback Thursday: Sessions from TechEd Houston

Today is my final installment of highlights from TechEd Houston! Below are some of my session picks from the last day of the conference.

TWC: Hacker’s Perspective on Your Windows Infrastructure: Mandatory Check List (DCIM-B366)
Windows 8 Security Internals (WIN-B350)
Real-World Windows 8.1 Deployment Note from the Field (WIN-B358)
Providing SaaS Single Sign-on with Microsoft Azure Active Directory (PCIT-B326)
Delivering Disaster Recovery Solutions Using Windows Server 2012 R2, Microsoft System Center 2012 R2 and Microsoft Azure (DCIM-B421)
How IPv6 Impacts Private Cloud Deployments (DCIM-B373)
Windows Server 2003 End of Life Migration Planning for Your Workloads (DCIM-B376)
Upgrading Active Directory the Safe Way: Using Virtualization Technologies (PCIT-B341)

For my lists of sessions from the other days, you can find them here: Monday, Tuesday and Wednesday.

Replication Warnings? – It could be just one Attribute.

Active Directory can be a funny beast. This week, I noticed a reoccuring replication error that didn’t seem to be sorting itself within a reasonable time frame. I was seeing NTDS Replication Warning 1083, referencing a specific user account:

Event Type: Warning
Event Source: NTDS Replication
Event Category: Replication
Event ID: 1083
Date:  10/3/2011
Time:  11:45:00 AM
User:  NT AUTHORITY\ANONYMOUS LOGON
Computer: DC1
Description:
Active Directory could not update the following object with changes received from the domain controller at the following network address because Active Directory was busy processing information.

Object:
CN=Joe Smith,OU=Accounts,DC=mydomain,DC=org
Network address:
a5b5b72d-c74b-486a-9dfa-f6516f37b38b._msdcs.caclo.org

Following it was the informational event 1955 about a write conflict:

Event Type: Information
Event Source: NTDS Replication
Event Category: Replication
Event ID: 1955
Date:  10/3/2011
Time:  11:45:00 AM
User:  NT AUTHORITY\ANONYMOUS LOGON
Computer: DC1
Description:
Active Directory encountered a write conflict when applying replicated changes to the following object.

Object:
CN=Joe Smith,OU=Accounts,DC=mydomain,DC=org
Time in seconds: 0

After some research I tried the following troubleshooting steps:

1) Moved the offending user to a different OU temporarily to see if the problem resolved. This essentially “tickles” AD into replicating that particular user. I recieved the same messages, but the user’s CN had been updated to the new OU.
2) Used the LDP tool to see if there was duplicate entries for this user somehow, but only one instance was found.
3) Used repadmin to look at the time stamps of various attributes on the account, particular one with a time stamp close to the time that the replication warnings started appearing in the event log.

Repadmin was where I had the most luck. You’ll want to run the following command for Windows 2003 SP2 DCs:

repadmin /showobjmeta DC1 “CN=Joe Smith,OU=Accounts,DC=mydomain,DC=org”

This will return a list of attributes with timestamps. In my case it was the attribute related to the last password change, which was the only one that had a timestamp of the same date when the errors began. I reset the password on the account to “tickle” that particular attribute and the replication completed without any complaint.

Some anticodotal stories on the Internet indicate that this attribute can cause trouble if replication occurs while an account happens to be locked out. In this case, the account was for a consultant who didn’t log in very often, so the locked account went unnoticed for some time, causing the replication issue.

Inside MDOP: AGPM 4.0

In case you missed the PacITPros meeting on December 7th, you missed out on some interesting vendor and technical presentations. In addition to a presentation from BlueCat Networks and Hurricane Electric, I did a short demo of one of the MDOP tools – the Advanced Group Policy Manager 4.0.

This tool hooks right into the existing Group Policy Manager snap-in you know and love in your MMC and with the use of a designated archive server, extends the functionality to include better search features and change management. No matter the size of your organization or the number of IT staff you share group policy tasks with, you can benefit from this tool. Even if you are the only person who does anything with group policies, this tool will make your life easier.

First, the change control features take away much of the pain of keeping track of what was changed when and potentially by who. Policies that are controlled by the system must be checked in and out for adjustments, which automatically creates a history record capturing the state of a policy at any given time. These records can easily be reviewed for corporate compliance and policies can even be rolled back to previous states.

With new roles created within the tool, non-admininstrators (even regular domain users) can be granted the ability to review or edit policies… leaving the actual deployment and linking of the GPOs to system administrators.

The abililty to search and filter your view of policies is much improved. You can search by name, state (checked in, checked out), even by variables such being updated “last month” or “last week”.

Finally, you can easily import and export policies, even across forests. No more manual recreation of the perfect policy just because you want to use it in your test lab environment or in another forest.

Finally, keep in mind that APGM 4.0 adds support for Windows Server 2008 R2 and Windows 7, as well as runs on Windows Server 2008 and Vista. If you are supporting an environment with older versions of Windows Server, consider version 2.5 or 3.0 of the tool. Not of all of the features are included, but if you are looking specifically for the change management aspects, those older versions may work for you until you upgrade your servers.

Out of the six tools in the Microsoft Desktop Optimization Pack, APGM isn’t one I’d overlook.

The Post-Mortem of a Domain Death

The past few days have been busy as we’ve been performing the tasks to remove our failed domain controller and domain from our Windows 2003 Active Directory forest. Now that everything is working normally and I can check off that long-standing IT project of “remove child domain” from my task list, I’d like to share a few things we’ve learned.

NTDSUTIL will prompt you several times when it comes to removing the last DC in a domain using the steps in KB 216498. It will even hint that since you are removing the DC in the domain, that you are also removing the domain itself. But you are not. You must take additional steps in NTDSUTIL to remove the orphaned domain, see KB 230306 to finish up.
How do you know you have an orphaned domain? Check AD Domains and Trusts. If you still see a domain in your tree that you can’t view the properties of, you aren’t done yet. Also, if your workstations still show the domain as a logon option in the GINA, get back to work.
You might remember to clean up your DNS, but don’t forget to also clean up WINS. WINS resolution can haunt you and keep your workstations and applications busy looking for something that isn’t there anymore.
Watch your Group Policy links. If you’ve cross-linked policies from the child domain to your forest root or other domains, workstations will indicate USERENV errors referencing the missing domain. Policies from other domains won’t show up in your “Group Policy Objects” container the GPMC. You’ll need to expand all your other OUs in the GPMC to find any policy links that report an error.
If you are using a version of Exchange that has the infamous Recipient Update Service, remove the service entry that handles the missing domain. You’ll see repeated MSExchangeAL Events 8213, 8250, 8260 and 8026 on your mail server otherwise.

I’ve used NTDSUTIL in the lab and in production several times to remove failed domain controllers, but removing an orphaned domain happens far less frequently. While the majority of our Microsoft applications handled the existence of references to the orphaned domain with grace until we completed the clean up, one of our third party applications, ImageRight, was far more sensitive about it.

We found that a combination of the WINS resolution and the orphaned trust relationship distracted the application enough that it was slow to operate, sometimes refused to load at all, and hung on particular actions. If you happen to be an ImageRight customer who uses the Active Directory integration features, keep in mind that it likes all the AD ducks to be in a row.

While we had a little a bit of pain getting to this point, I’m really happy that our AD forest is neater and cleaner because of it. It’ll make it much easier to tackle other upgrade projects on the horizon for Active Directory and Exchange.

Virtualized Domain Controllers? I’ll pass, thanks.

There are a lot of good arguments for virtualizing DCs. You should have several of them for redundancy, but depending on the number of employees and general work load, DCs tend to be underutilized and it can be hard to warrant having a whole physical server for each one. But after loosing a second domain controller after doing essentially some basic VM maintenance, I’m not sold.

You may remember a previous post of mine from the summer of 2009 about NTDS Error 2103, when the DCs in a small child domain were virtualized. I had agreed to virtualize both DCs from that domain as the domain was not supporting any user accounts and had less than a half dozen servers as members. One did not convert well and we decided to just leave the remaining DC as the sole one standing for that domain after vetting out the risks. There are several “rules” to follow when virtualizing DCs, particularly not restoring snapshots of them and not putting yourself in the situation where your VM host machine need to authenticate to DCs that can’t start up until your host authenticates.

Fast forward about 16 months, to now. Our system administrator who handles the majority of our ESX management was migrating many of our VMs to our newly installed SAN. He reported that he shut down the DC normally, moved the VM and then started it back up a few hours later after all the server files had been copied over. The few servers that use that DC were working properly and everything looked good.

But alas, a few weeks later, the server reported a USN rollback condition. Replication and netlogon services stopped. I checked the logs to see if I could figure out the cause, but only saw things that added to the confusion. The DC was mysteriously missing logs from between the time of the VM relocation and the time of the NTDS error. And the forest domain controllers had logs indicating it had been silent for nearly 2 weeks. At this point, I can only speculate what went bad.

We slapped a bandage on the server by restarting netlogon so those few servers could authenticate, but without replication happening properly, the server will simply choke up again. And after the tombstone lifetime passes, the forest domain will consider it a lost cause. It’s essentially a zombie.

So begins our finally steps to decommission that child domain. I have no interest in restoring that domain from backup, since removing that domain has been an operations project that has been bumped for a long time. Now our hand has been forced and the plan is simple. Change a couple service accounts, move 2 servers to join the forest root domain and then NTDSUTIL that DC into nothingness.

As for our two forest root domain controllers? I’ll throw my body in front of their metal cases for a long time to come.

Round Up: September Events and Windows Resources

As the summer draws to a close, here are a few thing that have caught my eye. Check them out and see for yourself.

First up, a comprehensive article on Redmondmag.com by Greg Shields on “Avoiding Active Directory Disasters“, particularly with Windows 2000 and 2003 versions of AD. It might just be time to do a little end of summer health check for an important component of your network.

PacITPros will be holding their monthly meeting tonight starting at 6pm. It’s not to late to RSVP and get in on the action. Compellent will be presenting on their data storage solutions and there will also be a presentation on the latest and greatest with Microsoft’s OCS.

On 9/30, don’t miss out on the Springboard Series Virtual Roundtable on migrating from IE6 to IE8 on Windows 7, starting at 9am. This virtual, interactive discussion will cover migration strategies, standards, and support for organizations moving from Internet Explorer 6 to Internet Explorer 8.

Finally, check in with the folks at TechNet on 9/22 when they host an in-person event on Deploying Windows 7 at the Microsoft downtown office in San Francisco. Learn more or register at TechNet Events Presents: Deploying Windows 7. This event will also be held in Irvine, CA on 9/20 and in Los Angeles on 9/21.