NTDS Error 2103

This week one of my domain controllers developed a curious problem. I don’t like curious problems, especially ones that rear their heads after the server reboots.


The error was an NTDS General event 2103, which indicates that the AD database “was restored using an unsupported procedure and Net Logon service has been paused”. Research and KB Article 875495 lists event 2103 and 3 other events related to a condition known as USN Rollback.

This DC is running Windows 2003 SP2, so based on the article, I should be seeing at least the more serious NTDS Replication 2095 event as well, due to a hotfix in SP1 that made the error logging somewhat more verbose. But I’m not. This makes it more curious. Am I in a rollback state or not?

KB 8759495 also lists some possible causes of this state, some of which are possible in a virtual environment – the case for this DC. It points me to another KB Article 888794 which lists out a bunch of considerations for hosting DCs as VMs. However our environment met all the requirements, including one related to write caching on disks, as our host machine has battery backed disk caching. So I rule out that we actively caused a potential rollback.

Repadmin has a switch (/showutdvec) that can be used to determine USN status by displaying the up-to-dateness vector USN for all DCs that replicate a common naming context. If the direct replication partners have a higher USN for the DC in question than that DC has for itself, that’s considered evidence of a USN rollback. My DC did not have this problem, as it had a USN higher than it’s partners. So at this point I couldn’t confirm or deny a true USN rollback issue, however it seemed the the DC “thought” it was having this problem. Maybe I could figure out why the DC was in this limbo.

So I returned to the original article to look for specific causes. One line reads, “Starting an AD domain controller whose AD database file was restored (copied) into place by using an imaging program such as Norton Ghost.”

Thinking back, the conversion of this DC from physical to virtual did not go as smoothly as I would have hoped. I remembered I had to resolve some issue where I was getting an error in the logs related to the directory database file not being where the OS expected it, even though the path on the server hadn’t changed during the conversion. It was odd at the time, but the posted fix seemed to clear the issue and I’d moved on.

I’m guessing that perhaps that was the start of my issues – maybe the P2V process made the OS think the database was different copy even though it wasn’t. The result was that the server thought it was rolled back, but the USNs never reflected a problem. So I decided it was better to be safe than sorry and assume this “limbo” condition was not how I wanted to leave things.

The resolution for USN rollback is a forced removal of the domain controller from AD. Since this is a DC in a child domain that’s being phased out, very few changes happen to that domain so I wasn’t concerned about possibly loosing changes that may have been made on that DC. It was only the FSMO holder for one role which was easily seized by the other DC.

My decision now is to decided between bringing up a replacement DC for this domain next week or just run one DC for the time being and try to speed up the remaining tasks that need to be done before we can removed the child domain all together.

But that’s for another day!

Event Log Auditing

My company has had a policy for checking all server event logs at least weekly for as long as I can remember. Honestly I’m happy to review server logs on a regular basis, as I’ve caught a variety of small problems before they’ve become big problems by doing it. The bigger issue is creating a trail of some sort that proved that it was done to make our auditors happy.

Last fall I went looking for some software that would help with the whole process. We’d settled on NetPro’s LogAdmin because we were purchasing some of their other products and LogAdmin seemed like it would do the trick. A combination of factors led to us not getting it installed properly or in a timely manner – my time being pulled by a variety of “more pressing” projects, the purchase of NetPro by Quest Software, my lack of experience with SQL installations, misinformation about what IIS requirements were needed to support the software, and then the subsequent “end of life” announcement for LogAdmin by Quest.

I feel like a spent a lifetime on the phone and sending emails, but we got our LogAdmin licenses converted to the equivialent Quest product, InTrust. So finally after 2 days of scheduled phone support, some growing pains of installing SQL 2005 on Server 2008 and the software requirement of disabling UAC, the InTrust product is installed and I’ve had some basic training on configuration.

Since we didn’t originally look at this product, I feel like I’ve been flying blind. The support tech I was working with was great but concentrated his demos on the security logs, where I need reports and alerts for ALL the logs in Windows. I’m hoping I’ll have some time next week to RTFM and concentrate on setting up the agent, filters and reports on a server or two to get more comfortable.

Windows 7 RC on my Samsung NC10

Finally got around to installing Windows 7 on my Samsung NC10 netbook – not that I haven’t been dying to since TechEd in Los Angeles, I just hadn’t the time until the other night. It probably helped that PacITPros was having a Windows 7 Loadfest meeting and I wanted to have it ready to go.

The install was pretty quick and although I had backed up all my personal files, it was great that everything that was on XP was convienently backed up to the “windows.old” folder. (It made reinstalling iTunes extra fast since I didn’t have to reload my music from backup DVDs.)

I love the way it looks and it didn’t have any trouble finding drivers for all the basics – wireless network card, bluetooth mouse, built-in video. Had to use some Vista drivers from the manufacturer for some of the Samsung specific things, like the special function keys, battery manager, etc. Found a great blog article by Ade Miller about installing Window 7 on the Samsung, which was really helpful in the driver search.

The biggest issue so far has been with the free version of AVG anti-virus, which was severely slowing down the boot. I’m trying out the free version of Avaste and that seems to be working well so far. Now I need actually start using it do my regular work.

SAS70 II and the Cloud

Spam has become a bit of a sore subject at work. We’ve been using what was Sybari Antigen (now a Microsoft product) on our Exchange server for years. However, it’s just not managing all our spam issues at an acceptable rate any more. It certainly blocks a lot, but about 15-20 messages are still getting through to my Outlook client every day, where only about 80% of those are being caught with Outlook junk mail filtering.

And since I’m a BlackBerry user, it means that those 15-20 messages are delivered to my mobile device regardless of what my Outlook client does with them at my desktop. So I’ve started a search for another solution.

Our first decision was “appliance” vs “SaaS”. From a network admin perspective, there is a lot to like about moving anti-spam services into the cloud. I liked the idea of offloading spam traffic to an outside network, thus only having my network support legitimate mail delivery. I liked not having yet another box to plug in and wire into my LAN racks. I liked being about pay monthly/annually for exactly the services we were using. And I liked the possibility of being able to add on some email archiving and discovery services at a later time.

So I compared a few services, kicked my results up to management and was ready to sign up. But there was one roadblock – SAS70 II certification. As a company that does fall under HIPAA, SAS70 certification was something I was asking about while I was researching vendors, but now it was time to prove that certification to our auditors.

SAS70 II certification involves a variety of areas: Physical Security, Environmental Protection, Computer Operations, Information Security, Data Communications, Customer Access Controls and DR/Business Continuity Assurances. Many of the vendor we were considering were using data centers of major telecommunications companies/ISPs and while those companies were certified for themselves, that certification doesn’t necessary mean that the anti-spam vendor (a client of theirs) was also fully certified – especially in the areas outside of physical security and environmental protection. SAS70 certification is not transitive, so to speak.

Ultimately, our auditor recommended that we NOT use services based in the cloud for our email, because there was a chance (either by later using them for archiving or by them quarantining a legitimate message, etc), that they could be storing our company data. This was a risk my company was not willing to take.

This isn’t to say that their aren’t SaaS vendors who are SAS70 certified. But my company is a little spooked by the whole “cloud computing” idea right now. So it’s back to drawing board for me, this time looking at appliances.

The Joys of SPNs

Last week I was at TechEd in LA and spent some of my time listening to Mark Minasi talk about Kerberos in Active Directory. He spent some time talking about SPNs (Service Principal Names). The takeaways were:

a) They needed some love and care when creating them manually.
b) That you really didn’t want duplicates of them lying around in AD.

I’ve dealt with SPNs on occassion when dealing with delegating connections between SQL servers for one of our in-house applications and always remember it being a confusing process that I never spent enough time to seriously understand. So I nodded to myself in agreement with the speaker and moved on.

Then I came back to work. One of our on-going projects required serveral of our company SQL servers to be moved from one domain to another. Our DBA was responsible for planning this work, since he’d ultimately be the one fielding the support calls if things went bad. And he decided to work on this yesterday, tapping me to help out with anything that fell into the realm of Active Directory.

The key troublemaker in all of this? SPNs.

It can be tricky to find old SPNs when you aren’t really sure where to look, and since we were never really sure of what we’d done in the past, knowing where to look was a factor. It’s also hard to tell if you are doing things correctly using the SetSPN Tool that comes with Server 2003, as it lacks some of the improved features of the Server 2008 version. Also, we had a lot of moving parts involved – changing the domain membership of the servers, changing the service account that runs the SQL Services on each server and the additional issue of forgetting to check that DNS was updated properly.

The big helper in all of this? ADSIEdit.

Once we realized where we were supposed to look (at the service accounts, not the servers themselves), it made adding the new SPNs and remove the duplicates really easy. And now I really think I understand how SPNs work – instead of my previous attempts of just mucking around and getting lucky.

Customer Focus Design for Window Server with PacITPros

On May 5th, I helped organize a special event for the Pacific IT Pros user group in San Francisco. Customer Focused Design is a process used by Microsoft to collect feedback about features and requirements that need improvement in future product development.
The goal of this event was to provide Microsoft with feedback related to the future of the Windows Server operating system. The Customer Focused Design team was very appreciative of the time PacITPros spent brainstorming together to during the session. They saw a lot of really good ideas and value come out of the session. Overall, the three groups provided over 300 individual requirements and close to 50 high level requirements where improvements could be made.
That information was distilled into the following series of slides:Group 1 (Kevin Lane) – 15 high level requirements, with 97 individual sticky requirements.Group 2 (Robert DeLuca) – 18 high level requirements, with 54 individual sticky requirements.Group 3 (Pat Fetty) – 16 high level requirements, with 174 individual sticky requirements.
The slides highlight the following information:
“Customer Importance” – this provides the prioritization of the requirements that were generated.
“Current Ability” – This is the PacITPros ranking of Microsoft’s ability to deliver this requirement right now based on the technology Microsoft provides in Windows Server 2008 and Windows Server 2008R2. The ranking numbers are:
1 = Microsoft doesn’t deliver this at all
2-3 = you can do this with significant workarounds and/or 3rd party solutions
4-7 = Mircosoft delivers this with minimal workarounds or other applications
8-9 = Microsoft delivers this with no workarounds
10 = Microsoft couldn’t do this any better
“Improvement Pareto” – The requirements and the ability rankings are calculated together to determine the improvement areas needed for focus. Areas with high importance but low ability are areas that Microsoft needs to put some work into. Areas that are low mean that Microsoft needs less investment and effort to deliver what is needed.
Kudos to all the PacITPros members who participated. This was a hands-on way to have our voices heard directly by Microsoft.

The Kindle – A Quick Little Review

I’ve had my Kindle for all of 4 hours and I think it’s really cool.

The screen is really easy to read, it’s simple to navigate and pretty darn straight forward to use. The wireless connection makes it really handy to download books, search wikipedia.com and it has a built in dictionary so you can look up words on the fly.

I bought it because I’m really sick of carting books around and not reading when I have time to read simply because I don’t have something interesting handy. I’ve downloaded a bunch of sample chapters of some books I’ve wanted to buy and imported a few PDFs of books I already own. I’ve been using a free software download to do the file conversion – results vary depending on the complexity of the PDF, of course. Documents that are primarily text converted pretty nicely. The big study guide for my Microsoft exam is so-so. You can also email documents in various formats to it directly and Amazon will do the conversion for you and then deliver it automatically.

I know some people has DRM issues with the whole thing. I’m not too concerned. Sure, if you buy a book from Amazon it’s in the Amazon format, but its available to transfer to other Kindle devices registered to your account (like a family member) and you can delete and re-load then as often as you want.

You can also download a lot of free books from manybooks.net and Feedbooks provides a downloadable index of their books that you can link to directly from the Kindle and download the books on the fly. Lots of classics, etc.

And seriously, having an easy way to read those crazy Microsoft white papers I feel like I’m always printing. It’s totally worth it.

When Things Work.

This morning, I’ve been at the office. I needed to make a key change with our imaging system that affects the user’s logons, so it’s one of those things you can’t do during the business day.

And due to the additional security features we have turned on for the system, sometimes regular changes to the system actually break things. I don’t really like broken things, thus have given myself the entire weekend to fix anything that could have potentially gone wrong.

But it worked. Just like the documentation was supposed to. I appreciate that the tech just sent me their internal documentation, instead of making me rely on them to hand me information only when things start going wrong. Plus I didn’t have to make one of them actively work on the weekend and I end up understanding the system better overall because I was doing the work myself.

I did have a tech available via email – but that was more for moral support. He would have only jumped on if things went badly and we had to roll back the changes. But I hate rolling things back – I really like to just fix the problem and keep moving forward.

The Internet Life

Do you remember life before the Internet? I barely do.

It’s hard to imagine that there are some people I know best (or only) via the WWW. I was chatting with a friend the other evening and he commented on how he’s got a better handle on my personality via IM than he ever had from dealing with me in person.

Maybe that’s because when we see each other in person it’s always related to our jobs and there really isn’t much time to talk about anything other than technology. Not that we don’t spend most of our online time talking about technology and work too, but the other night we got onto other topics that probably would never come up when we happen to be in the same room.

It all got me thinking about how I interact with people via some kind of online chat – I’m in the age range where it’s not the most used medium for our generation, but I spent so many years developing relationships with remote co-workers that were exclusively via IM maybe I’m just more comfortable expressing myself that way.

There are a lot of people on my chat client list that I don’t regularly chat with anymore, but it’s nice to see them there throughout the day. I remember when one of my old colleagues switched jobs and wasn’t able to connect to IM from his office. I went through weeks of being really unnerved by the fact that he wasn’t showing up on my list. It was like some put up a wall between us that I couldn’t figure a way around.

Then there is another tech friend of mine who I know is pretty much accessible anytime. He’s not always on IM, but if he’s got a phone signal he’ll usually get back to an email or a text. I sent him tech question yesterday knowing full well that was skiing in Canada. I wouldn’t have minded if he waited until he returned to get back to me – but sure enough he replied within 10 minutes. Is it better for him that he’s that totally connected or is it a pain in the ass?

Maybe because we both are in an industry where we are available to be paged or alerted by our office servers when they are in need, we don’t mind being available to real people, too.