Workaround for ImageRight and Remote Desktop Display Bug

Many months ago, I wrote about using ImageRight via Server 2008 RemoteApp. There’s been a problem with accessing some of the drop down menus and as we’ve upgraded ImageRight a few times since then, I’ve been hoping the issue will just go away.

But after moving to 5.4 and having more and more users accessing ImageRight via RemoteApp, the issue really need to be addressed. Turns out there is a quick and easy workaround. Hold down the CTRL key when you select the drop down menus. Simple and effective.

Now, go forth and be Merry for the Holidays!

Customizing Distribution Group Management in Exchange 2010

One of the things I allowed certain end-users to do via Outlook was manage some of their own distribution lists. With a small office and a small IT staff, constantly changing distribution list membership was an easy thing to just delegate back to the people who really “owned” those lists. In Exchange 2003, it was an easy process to delegate that ability to end-users by making them the “manager” of the list.

Shortly after the migration to Exchange 2010, I started getting reports that the distribution lists could no longer be changed by the designated list managers. Exchange 2010 RBAC roles include a role called “MyDistributionGroups” that grants the ability for end-users to view and modify distribution groups. However, it also grants the right to create new distribution lists, which was not something I wanted for non-IT staffers.

I found this great blog post, Allowing End-Users to Manage Distribution Group Membership, in Exchange 2010 by Mike Pfeiffer on how to create a custom locked-down role for distribution group management using PowerShell. Written in early 2010, it’s still get lots of great comments and usage – it certainly made my day easier!

Exchange 2010: Database Stores, Not Quite Ready When You Are

Once I had my Exchange 2010 server up and running, I had a need to create a new store. Unfortunately, things didn’t look so great when the store wouldn’t mount after I created the store in the GUI console. There were even some fine error messages in the logs letting me know that Exchange was unable to mount the store. If you search the Web for answers to this problem, you’ll find all sorts of potential solutions and ideas.

Turns out the thing that worked best for me was some patience. Exchange 2010 is deeply ingrained in Active Directory and Active Directory does things at it’s own pace. Sometimes immediately, sometimes in 5 minutes and sometimes in fifteen.

So go ahead and read all those links you found in the great WWW and then after about 5 minutes, go back and try to mount that database again. Chances are, it’ll work just fine.

The Next Rev: ImageRight 5.4

My office is a week or so post-upgrade to ImageRight 5.4 from version 5.2. While this version integrates some post-5.2 hotfixes to resolve some annotation and image display issues, it comes with it’s own post-5.4 hotfixes that need to be installed after the primary installation. If you deploy the desktop client with Group Policy, you’ll need to create an MSI file and a third policy to fully deploy the software and hotfixes automatically.

While the desktop client hasn’t changed much from a user standpoint, there were some security additions and tweaks that are important to know about.

Alphabetizing Lists and Annotations – In previous versions, many of the lists that users interacted with were sorted by creation date. This was less than ideal when selecting from a long list of private annotations or selecting from the document type tree drop-down. Those list displays are now alphabetized.
Read/Write permissions added to File Notes – while this is a great addition as a security feature, it’s turned on by default post-upgrade with the result being that users can’t see or add any file notes. I needed to make a support call to find the odd place that permission change was located. (The Security properties of the “Storage Types” container in the EMC.)
Annotations Limited to Specific File Types – There is a feature in version 5 where you can filter or limit on what file types an annotation is available for use. When migrating from version 4 to 5.2 the system defaulted all the private annotations to be available on all file types (which was the behavior in previous version), but didn’t automatically check the “include all file types” option box. In version 5.4, the check box status is enforced, which may make private annotations seem to disappear for the end users.
New Permissions for “Desktop – Modify Document Date” – also defaulted to not having any permission set in 5.4, users will need this permission added to change a document date. Also new is some functionality to track the date and time a document is received (“Desktop – Modify Receive Date and Time”), you may or may not want to let users change that.

Also, if you do any automated processes where you are using the FUP tool for updating file information, it’s not working correctly. Hopefully, that one is resolved quickly. We don’t use it often, but when we do we tend to have a lot of files that need a change and a manual process would be tedious.

Overall, ImageRight 5.4 brought several new features and welcome changes to the document management product, with a relatively easy upgrade process from 5.2.

From BlackBerry to Windows Phone

Last week, I landed myself brand-new Samsung Focus Flash phone with Windows 7.5. I had debated about going with the older Samsung Focus model in mid-October, but figured it was probably worth the wait for some new hardware too.

Having a physical keyboard on the BlackBerry was hard to give up, but outside of the lack of real keys, I pretty much love everything about it right now. One of the big factors in deciding on what phone to select was my ability to hold it and have a reasonable chance of being able to type with one hand. The Focus Flash is the only phone in the Samsung Focus line that is the same width as the BlackBerry Bold 9700. If I wanted a tablet, I would have bought one that has a screen larger than 4.3 inches.

As someone who spends a good amount of time using Twitter and dabbling in the other popular social media sites (depending on where my friends are), the People Hub has got to be the best idea since sliced bread. Being able to group certain friends and family members and highlighting a tile for that group on home page is fantastic. Even after a week, I feel less like I have to constantly watch my Twitter stream or check Facebook because I can easily view the postings from the people I care about the most. The native integration for interacting with Facebook and Twitter lack some of the more robust features, but it certainly good enough for the majority of my social media interactions.

The live tiles on the home page are great for highlight the next appointment and the latest status updates from the People Hub. Not having to open the calendar to see my next appointment is a nice bonus. Plus having a miniature “digital picture frame” that highlights my favorite photos is a fun feature.

I know many iPhone lovers may find faults in some of current features in the Windows Phone. There isn’t the extensive catalog of apps yet and some of the ones that exist lack some of the more refined functions that more mature apps for iPhone and BlackBerry have. But I think it’s only a matter of time before those app offerings catch up. And I have a list of things that I do miss from the BlackBerry – battery life being one of them and I’m developing a wish-list of things I hope to see change or become available in 2012, but that’s a post of its own!

Exchange 2010 and External Relays (Migration – Part 3)

The “Receive” Connector is a funny thing in Exchange 2010. The receive connectors on my system seem to double as “Send” connectors depending on who’s doing the sending. Once my new server was up and running, it was a no brainer to make a proper “Send” connector so the server could access the Internet to deliver mail to external parties. I was also able to quickly bring up “Receive” connector to collect mail from our Barracuda appliance.

Then I started tackling the servers within our organization that send alerts and reports via email. I added their network addresses to the same connector I used for the Barracuda device, since they are all on the same network.

All the devices seemed happy until I ran across one that needed to send messages to external recipients. Turns out that on Exchange 2003, I was using the same connector for both internal and external relaying without issue, but Exchange 2010 is a little pickier from a security standpoint (a good thing) and I had to create a special receive connector to handle external relaying.

So why are we using “receive” connectors to relay external mail? The receive connectors collect mail coming to the Exchange 2010 server which are then sent out using the Internet send connector. So while all your devices are sending mail, the Exchange server is both receiving it and sending it.
Of course, I wouldn’t be writing a post about External Relays if there wasn’t something special about them.

When creating an external relay you want to be sure to un-check all the security mechanisms from the Authentication tab, since it’s likely you are relaying mail for things like your UPS which might be “phoning home” with updates to a support provider or copier/scanners that might need to send a scanned items to an outside party – all types of devices that likely won’t have a mechanism to authenticate to your mail server.

You also need to set your “Permission Groups” to Anonymous, but the configuration doesn’t end there. Be sure to kick off this little extra PowerShell as well.

Get-ReceiveConnector “External Relay” | Add-ADPermission -User “NT AUTHORITY\ANONYMOUS LOGON” -ExtendedRights “ms-Exch-SMTP-Accept-Any-Recipient”

Now that this relay is pretty wide open, so lock down which IP addresses from your network are allowed to use it so that its well controlled. If you need some screenshots for the configuration, check out this post from the Lazy Network Admin.
http://www.lazynetworkadmin.com/knowledgebase-mainmenu-6/2-windows/149-exchange-2010-configure-anonymous-relay-to-external-domains

Playing IT Fast and Loose

It’s been a long time since I’ve been at work from dusk ’til dawn. I not saying that I’m the reason we have such fabulous uptime, there are a lot of factors that play into it. We’ve got a well rounded NetOps team, we try to buy decent hardware, we work to keep everything backed up and we don’t screw with things when they are working. And we’ve been lucky for a long time.

It also helps that our business model doesn’t require selling things to the public or answering to many external “customers”. Which puts us in the interesting position where its almost okay if we are down for a day or two, as long as we can get things back to pretty close to where they were before they went down. That also sets up to make some very interesting decisions come budget time. They aren’t necessarily “wrong”, but they can end up being awkward at times.

For example, we’ve been working over the last two years to virtualize our infrastructure. This makes lots of sense for us – our office space requirements are shrinking and our servers aren’t heavily utilized individually, yet we tend to need lots of individual servers due to our line of business. When our virtualization project finally got rolling, we opted to us a small array of SAN devices from Lefthand (now HP). We’ve always used Compaq/HP equipment, we’ve been very happy with the dependability of the physical hardware. Hard drives are considered consumables and we do expect failures of those from time to time, but whole systems really biting the dust? Not so much.

Because of all the factors I’ve mentioned, we made the decision to NOT mirror our SAN array. Or do any network RAID. (That’s right, you can pause for a moment while the IT gods strike me down.) We opted for using all the space we could for data and weighed that against the odds of a failure that would destroyed the data on a SAN, rendering entire RAID 0 array useless.

Early this week, we came really close. We had a motherboard fail on one of the SANs, taking down our entire VM infrastructure. This included everything except the VoIP phone system and two major applications that have not yet been virtualized. We were down for about 18 hours total, which included one business day.

Granted, we spent the majority of our downtime waiting for parts from HP and planning for the ultimate worst – restoring everything from backup. While we may think highly of HP hardware overall, we don’t think very highly of their 4-hour response windows on Sunday nights. Ultimately, over 99% of the data on the SAN survived the hardware failure and the VMs popped back into action as soon as the SAN came back online. We only had to restore one non-production server from backup after the motherboard replacement.

Today, our upper management complemented us on how we handled the issue and was pleased with how quickly we got everything working again.

Do I recommend not having redundancy on your critical systems? Nope.

But if your company management fully understands and agrees to the risks related to certain budgeting decisions, then as a IT Pro your job is to simply do the best you can with what you have and clearly define the potential results of certain failure scenarios.

Still, I’m thinking it might be a good time to hit Vegas, because Lady Luck was certainly on our side.

Migrating to Exchange 2010 (Part 2) – Certificates

Depending on your installation of Exchange 2010 and what internal and external services you want to provide, you’ll likely need a new SSL certificate from a 3rd party provider. You probably already have a basic mail.company.com certificate, but that’s just not going to cut it anymore.

If youl’ll be supporting mailboxes on a previous version of Exchange or providing access to supporting Outlook Anywhere, you’ll likely need additional host names on your certificate, like legacy.company.com and autodiscover.company.com. This will require a SAN (Subject Alternate Name) certificate.

Exchange supports different URLs for internal and external access and after a typical installation, your internal URLs will be set to the FQDN of the server name (server.company.com) and external URLs will be set to whatever host name you specify during the install of the CAS server, like mail.company.com.

In order for us to get a shiny new SAN certificate, we had to revoke our existing mail.company.com while we were waiting for the new certificate to be issued. This would cause some temporary certificate problems with anyone who tried to use Outlook Web Access, but since this was a weekend project and I already declared the entire weekend as a maintenance window I wasn’t too concerned about it.

Meanwhile, I moved all my users mailboxes to the new server. All the Outlook clients were happy with the server’s self-signed certificate, which was great, since our 3rd party certificate provider took a few days to finish issuing the new cert. Once the new certificate came, I loaded it onto the mail server and authorized it for IIS to use.

My OWA certificate errors disappeared, but shortly there after we started getting reports of Outlook 2007 complaining about the certificate having a different name than what it was expecting. This was because we didn’t include the server name as part of the certificate, but all the internal URLs referenced the FQDN of the server’s real name.

Some of the internal URLs can be change in the Exchange Management Console, but there are a few that are easily overlooked since you can only change them using PowerShell, particularly the URLs for Autodiscover and EWS (Exchange Web Service).

Set-ClientAccessServer -Identity CAS_Server_Name -AutodiscoverServiceInternalUri https://mail.company.com/autodiscover/autodiscover.xml

Set-WebServicesVirtualDirectory -Identity “CAS_Server_Name\EWS (Default Web Site)” -InternalUrl https://mail.company.com/ews/exchange.asmx

Then be sure to recycle your MSExchangeAutodiscoverAppPool in IIS. You can read more about this issue in Microsoft’s KB 940726.

Migrating to Exchange 2010 (Part 1)

Ah, upgrades and migrations. Nothing every happens the same way it does in the lab! First off though, I do have to say that my upgrade/migration from Exchange 2003 to Exchange 2010 SP1 was successful and relatively transparent to my end users. Of course, we have a pretty small office and only one server, so there were not a lot of moving parts.

Before working in production, I did two lab-based migrations using some older copies of my Active Directory and Exchange servers – probably a tad too old, since I ran into totally different troubleshooting hurdles in production. Also, there were several things I couldn’t completely test in our lab environment, like our BlackBerry BES implementation or inbound and outbound mail connectors. But hey, I love flying by the seat of my pants.

One of the benefits of being late to Exchange 2010 was that there was lots of information on the Internet when I went search for solutions and nothing was insurmountable.
My primary source of guidance was the Microsoft Exchange Deployment Assistant, which is an online checklist of steps to follow. It asks a few questions about your environment and the produces a “customized” checklist. I have a few caveats about it though.

It assumes you are installing the various Exchange server roles on different machines or at different times. Since I was using the “typical” installation process my CAS, Hub and Mailbox roles were being installed together.
You must check off the completed steps in order. Sure, you can skip around and follow the instructions however you want, but if you like crossing things off a list as you go along and something early in the list is delayed, you can’t check of any of the later tasks. For example, “Adding digital certificates on the CAS” is something that is listed very early in the checklist. I had to wait several days for my new SAN certificate to be issued but that didn’t prevent me from moving forward with my migration. However, I couldn’t play along with with the checklist.

These are small gripes and if you are a stickler for documentation, you can print, email or copy/paste the instructions from the deployment assistant into your own project plan.

In the lab, the typical installation went along with out a hitch. However, I was not blessed with such luck in production. The CAS and Hub Transport roles installed fine, but the installation choked on the Mailbox role with the following error.

Couldn’t resolve the user or group “mydomain.local/Microsoft Exchange Security Groups/Discovery Management.” If the user or group is a foreign forest principal, you must have either a two-way trust or an outgoing trust.
I found the solution in several places, but it was very nicely documented here on Peter Schmidt’s blog.

Just to clarify, you are deleting the “DiscoverySearchMailbox” user from Active Directory, rerunning your install for the mailbox role and then rerunning “setup /prepareAD” to recreate the user you deleted. Interestingly, I can’t see the Discovery Search Mailbox in my Recipient Configuration in production, but I can in my test lab. (Odd… maybe one day I’ll figure that out.)

At this point, Exchange 2010 is humming along right next my Exchange 2003 server and everything is happy and still working the way it did before, mostly because we have a Barracuda appliance that collects our inbound mail and delivers it to the Exchange 2003 server, so really nothing had changed.

I created a Receive Connector for the Barracuda, updated the Barracuda to deliver mail the Exchange 2010 server, then created my new Send Connector as per the Deployment Assistant and removed the Send Connector on the Exchange 2003 server. Once I verified that inbound and outbound mail was still flowing it was time to take a breather and regroup for the next round.

Coming up – Getting BlackBerry BES to work again, fixing certificate errors with Outlook 2007, creating an external relay for some legacy devices on my network and figuring out why I couldn’t mount an new database after I created it. Stay tuned.

Replication Warnings? – It could be just one Attribute.

Active Directory can be a funny beast. This week, I noticed a reoccuring replication error that didn’t seem to be sorting itself within a reasonable time frame. I was seeing NTDS Replication Warning 1083, referencing a specific user account:

Event Type: Warning
Event Source: NTDS Replication
Event Category: Replication
Event ID: 1083
Date:  10/3/2011
Time:  11:45:00 AM
User:  NT AUTHORITY\ANONYMOUS LOGON
Computer: DC1
Description:
Active Directory could not update the following object with changes received from the domain controller at the following network address because Active Directory was busy processing information.

Object:
CN=Joe Smith,OU=Accounts,DC=mydomain,DC=org
Network address:
a5b5b72d-c74b-486a-9dfa-f6516f37b38b._msdcs.caclo.org

Following it was the informational event 1955 about a write conflict:

Event Type: Information
Event Source: NTDS Replication
Event Category: Replication
Event ID: 1955
Date:  10/3/2011
Time:  11:45:00 AM
User:  NT AUTHORITY\ANONYMOUS LOGON
Computer: DC1
Description:
Active Directory encountered a write conflict when applying replicated changes to the following object.

Object:
CN=Joe Smith,OU=Accounts,DC=mydomain,DC=org
Time in seconds: 0

After some research I tried the following troubleshooting steps:

1) Moved the offending user to a different OU temporarily to see if the problem resolved. This essentially “tickles” AD into replicating that particular user. I recieved the same messages, but the user’s CN had been updated to the new OU.
2) Used the LDP tool to see if there was duplicate entries for this user somehow, but only one instance was found.
3) Used repadmin to look at the time stamps of various attributes on the account, particular one with a time stamp close to the time that the replication warnings started appearing in the event log.

Repadmin was where I had the most luck. You’ll want to run the following command for Windows 2003 SP2 DCs:

repadmin /showobjmeta DC1 “CN=Joe Smith,OU=Accounts,DC=mydomain,DC=org”

This will return a list of attributes with timestamps. In my case it was the attribute related to the last password change, which was the only one that had a timestamp of the same date when the errors began. I reset the password on the account to “tickle” that particular attribute and the replication completed without any complaint.

Some anticodotal stories on the Internet indicate that this attribute can cause trouble if replication occurs while an account happens to be locked out. In this case, the account was for a consultant who didn’t log in very often, so the locked account went unnoticed for some time, causing the replication issue.