64-bit ImageRight support? – The "drivers" are in control.

The disaster recovery testing is touching more areas then I even though possible related to what options we can consider in our production and emergency environments. It’s bringing to light how interconnected software has become, and how those connections can sneak up on you, even when one is dealing with them everyday.

A basic premise of our recovery plan is to provide access to our recovered systems remotely, until we can make office space and desktop systems accessible to everyone. In order to keep things “simple” and provide the quickest possible up time, the plan calls for using Windows Terminal Services (aka “Remote Desktop Services” in 2008 R2) technology.

Due to the improvements in the offerings available directly by Microsoft related to remote access and the relatively small number of applications we need to make available, we determined that bringing terminal services up initially would be faster than recreating our Citrix environment during an emergency.

In conjunction with this (and the fact that we have only a small amount of remote use in production) we are currently planning to reduce licensing costs by only providing access using Microsoft products. Windows Server 2008 (and now R2) has many of the features we were looking to Citrix for in the past. While it’s possible for us to meet most of our needs with Server 2008, we’d much prefer to use 2008 R2.

While I was at the Vertafore Conference, one of my goals was to find out their schedule for 64-bit support. As one of our main enterprise applications, its important that it’s available on our remote access solution. Since I was unable to run the software on my 64-bit Windows 7 computer, I wanted know how far they were from addressing that.

Turns out, it all comes down to third-party drivers for peripherals. ImageRight works with several popular hardware vendors when it comes to scanners, including Kodak, Canon and Fujitsu. This allows customers to take advantage of more of the built-in scanner features that come with the hardware, instead of writing a generic scanner driver that could reduce the functionality native to the device. They also use the drivers to provide desktop features that allow end users to import documents directly from their PC.

Because of this, 64-bit support for the ImageRight software is directly related to how quickly scanner vendors make 64-bit drivers available. ImageRight claims that the makers of these key peripheral devices are complaining that Microsoft didn’t give them enough warning between Windows Server 2008 and the release of Server 2008 R2 regarding the official “death” of the 32-bit version of the OS to provide 64-bit drivers for all their devices.

ImageRight is planning to have support for 64-bit operating systems by the end of this year. We aren’t planning on a widespread upgrading of desktop hardware to 64-bit any time soon and will be able to wait without too much suffering. However, it does alter our plans for our remote access changes in the next 3-6 months. A disappointment for sure.

Also, the delay doesn’t help existing ImageRight clients or upcoming new ones that hope to run (or at least begin to test) an important software product on the most current hardware and operating systems available. An interesting domino effect that ends in needing to reconsider what I’ll be using for remote access during my recovery testing this month.

Windows 7 Tidbits

I found some interesting Windows 7 things online recently and wanted to share.

  • Check out the agenda for the Windows 7 Online Summit being held on October 7th. I’m hoping I’ll be able to keep people from interrupting me at work for a few hours that day.

  • Finally, if you haven’t had a chance to demo Windows 7 and you aren’t a TechNet/MSDN subscriber or a volume license customer, the Enterprise version is now available as a 90-day trial, from the TechNet Springboard.

Britian’s Blast from the Past

The National Museum of Computing in Bletchley Park is pushing aside the mothballs and reassembling a computer they’ve had in storage since 1973. Check out the Harwell WITCH story on Wired.com’s Gagdet Lab.

If you are looking for something more modern, take a look at the story behind the new Wired.com office kegerator, dubbed the Beer Robot. The exterior design was done by my husband – I’m not ashamed to flaunt his design skills.

Disaster Recovery Testing – Epic Fail #1

As I’ve mentioned before, my big project for this month is disaster recovery testing. A few things have changed since our last comprehensive test of our backup practices and we are long overdue. Because of this, I expect many “failures” along the way that will need to be remedied. I expect our network documentation to be lacking, I expect to be missing current versions of software in our disaster kit. I know for a fact that we don’t have detailed recovery instructions for several new enterprise systems. This is why we test – to find and fix these shortcomings.

This week, at the beginning stages of the testing we encountered our first “failure”. We’ve dubbed it “Epic Failure #1” and its all about those backup tapes.

A while back our outside auditor wanted us to password protect our tapes. We were running Symantec Backup Exec 10d at the time and were happy to comply. The password was promptly documented with our other important passwords. Our backup administrator successfully tested restores. Smiles all around.

We faithfully run backups daily. We run assorted restores every month to save lost Word documents, quickly migrate large file structures between servers, and to correct data corruption issues. We’ve had good luck with with the integrity of our tapes. More smiles.

Earlier this week, I load up the first tape I need to restore in my DR lab. I typed the password to catalog the tape and it tells me I have it wrong. I typed it again, because it’s not an easy password and perhaps I had made a mistake. The error message appears, my smile did not.

After poking in the Backup Exec databases on production and comparing existing XML catalog files from a tape known to work with the password, we conclude that our regular daily backup jobs simply have a different password. Or at least the password hash is completely different, yet this difference is repeated across the password protected backup jobs on all our production backup media servers. Frown.

After testing a series of tapes from different points in time from different servers, we came the the following disturbing conclusion: The migration of our Backup Exec software from 10d to 12.5, which also required us to install version 11 as part of the upgrade path, mangled the password hashes on the pre-existing job settings. Or uses a different algorithm, or something similar with the same result.

Any tapes with backup jobs that came from the 10d version of the software use the known password without issue. And any new jobs that are created without a password (since 12.5 doesn’t support media passwords anymore) are also fine. Tapes that have the “mystery password” on them are only readable by a media server that has the tape cataloged already, in this case the server that created it. So while they are useless in a full disaster scenario, they work for any current restorations we need in production. We upgraded Backup Exec just a few months ago, so the overall damage is limited to a specific time frame.

Correcting this issue required our backup administrator to create new jobs without password protection. Backup Exec 12.5 doesn’t support that type of media protection anymore (it was removed in version 11) so there is no obvious way to remove the password from the original job. Once we have some fresh, reliable backups off-site I can continue with the disaster testing. We’ll also have to look into testing the new tape encryption features in the current version of Backup Exec and see if we can use those to meet our audit requirements.

The lesson learned here was that even though the backup tapes were tested after the software upgrade, they should have been tested on a completely different media server. While our “routine” restore tasks showed our tapes had good data, it didn’t prove they would still save us in a severe disaster scenario.

Why Certify? Or Not?

Last night, I presented a brief overview of current Microsoft certifications at the PacITPros meeting. One of the questions that came up was how to determine the ROI of getting certified. Right now, I’m in the early stages of updating my messaging certification from Exchange 2003 to Exchange 2007. My office pays for exam fees, so I like to take advantage of that when I can. But why certify at all?

For me, it’s not a “bottom line” calculation. I do it as a motivator to keep learning. The nature of the business where I work means we tend to deal with a lot of dated software and don’t always have a need to upgrade to the latest or greatest of anything. We usually run about 3 years behind, particularly with Microsoft software, though that has been changing. If I wasn’t personally interested in staying current, I could easily let my skills lag behind.

Getting a certification in a specific technology gives me something tangible to work towards. By using some extra lab equipment at the office and making time to read, I can have a little fun and stay up to date on technology that will eventually get deployed in production.

Certification isn’t a perfect science. I know that the exams aren’t always in line with real production situations, but they have been improving over the years. And I know there are people on the ends of the spectrum -those that have great skills or experience with no certifications and others with limited experience and a series of letters after their name. I aim for balance. I stick with the topics and products that are in line with what I work with regularly so I can be confident that taking the time to study is going to provide value.

Right now, getting a certification doesn’t end in extra bonuses or a higher salary grade. But maybe one day it will be the item that stands out on my resume when compared to others with similar experience. Or show that I have the ability to set a goal and follow through. Or perhaps I’ll just enjoy challenging myself – certainly no harm in that!

PacITPros – Certifications, BranchCache and Office 2010

PacITPros will be having the September meeting tomorrow night at 6:30pm.

There is quite the line up of topics – Ed Horley, Microsoft MVP in Enterprise Security, will be presenting on Windows Server 2008 R2 and Windows 7 as a “Better Together” story. Specifically, what items are available only from Windows S2K8R2 with Windows 7 and how they would be compelling to use.

I’ll be doing a short presentation of what’s new with Microsoft certification tracks (specifically info about the MCITP and MCTS certifications) and Kathy Jacobs, Microsoft MVP in OneNote, will be doing an overview of some of the cool new features in Office 2010. Plus with the VMWare conference going on right down the street, there is sure to be a lot of chatter about what’s going on over at the Moscone Center.

Looking Forward with ImageRight

Enjoyed my first full day at the Vertafore Connections Conference. I’ve gotten a chance to chat with some of the great tech support staff that I’ve worked with over the last year and reconnect with some people who have been on-site at my office for installations and training in the past.

I’m looking forward to a few of the new features in version 5.2 and hope we’ll be able to upgrade to that as soon as possible. The integration with Outlook 2007 is pretty slick, especially for people who aren’t the heaviest users of ImageRight.

There has also been a lot of discussion about virtualization and disaster recovery related to the components of the ImageRight backend. Even though ImageRight isn’t “officially supported” in a virtualization environment, quite a few organizations have had success with some or all components of it. We are running our testing version on VMWare and haven’t had any issues thus far. If we have more success with the disaster recovery testing project over the next few weeks, it might be something to consider as we look at ways to make improvements in our server infrastructure.

Vertafore Connection (IUG09) Conference

Today I’m heading out to Altanta to take part in the Vertafore Connections Conference (previously known as the ImageRight User Group Conference). This will be the 2nd chance I’ve had to participate in this conference and I’m looking forward to seeing what’s new in version 5 of ImageRight, learn how I can take advantage of virtualization to reduce the footprint of servers we have for the application, and get to meet some of the great support team that I only know only through email and phone conversations.

I’ll also be following Vertafore Connections on Twitter (hashag #VconX) while I’m there.

Disaster Recovery – But for Real

This past week I’ve been doing the preliminary work (installing servers mostly) to get ready for our scheduled disaster recovery test. I expect that we’ll learn a lot about our existing plan and systems documentation and will be looking to make some changes that will make any need for a large recovery faster and more effective.

Meanwhile, I’m managing some real disaster recovery, but on a smaller scale. A few weeks ago I posted about the need to upgrade our ImageRight installation to resolve a bug that could cause some data loss. The ImageRight support staff worked hard to run the preliminary discovery/fixing of the image files and database, followed by performing the upgrade.

Not long after, I got an email from someone in another department asking me to “find” the annotations added to a invoice that seemed to have gone missing. She was assuming that since some temporary help had worked on the document, a user error had been made and a “copy without annotations” had been introduced. I could recover the annotations by looking through the deleted pages and at previous versions of those pages.

However, what I found was a bit unexpected. I found a history of changes being made to the document, but no actual annotations visible. Curious.

So I opened a support ticket. After several remote sessions and research, the ImageRight team was “nearly positive” (they need more testing to confirm) that the process run before our last upgrade to correct the potential data loss, actually introduced a different kind of data loss. The result is that the database knows about the affected annotations happening, but the physical files that represent the annotated versions had been replaced with non-annotated versions.

We do have the logs from the original process, so it was just a matter of ImageRight Support parsing that data to generate a list of files that were changed. Now we begin the task of recovering those files from tape.

Our Sr. DBA had been working on side project that loads all our backup catalogs into a database so we have a comprehensive reference from all backup servers to identify what tapes to recall when people ask for recoveries. That project is proving its worth this time around, since we need to locate and restore over 1000 files. He also needs to cross referencing them to the individual documents accessible via the desktop client so we can do a visual comparison to any special cases and to provide a record of which documents were affected in a format that’s understandable to everyone else, in case additional concerns come up after we repair the damage.

Our current plan is to have this resolved by the end of next weekend, but this is something that needs careful handling since we don’t want end users to have any doubt about the integrity of the system, which I still have total confidence in once we sort this out. Thus, I’m happy to spend the extra time making sure no other issues are introduced.

Plus I need some time to find our DBA some really nice coffee for his efforts.

Dusting off the Disaster Recovery Plan

This week, I started testing our department’s disaster recovery plan. The goal is to use the contents of our existing “disaster recovery box” that we keep off-site combined with our current backup tapes to restore some key parts of our infrastructure.

Success or failure will be measured by what road bumps we encounter and most importantly, our ability to work around them using only the resources in the box. If I have to go “outside the box” for some critical piece of software or some undocumented configuration detail it would be a black mark in our preparations that needs to be remedied.

Our testing scenario includes the domain, Exchange, the document imaging system, the financial system, the primary file server and the time card application. We are also going to provide remote access to restored applications so staff from other departments can test out the results and give us feedback on changes that could improve the end-user experience during this type of event. As an added bonus, we’ll be able to try out Server 2008 R2 Remote Desktop Services.

In the last 6 months we started using VMWare ESX to consolidate some of our servers in production, but none of the machines needed for this scenario are virtual yet. I will be doing “classic” restores where the OS has to be installed before restoring our data from backup tapes. However, we are using VMWare to host several of the machines in the disaster lab, so I will be able to save time by cloning my first installation of Windows Server a few extra times before installing specific applications.

Depending on how this project goes, I’d like to see us take more advantage of virtualization within our disaster recovery planning and maybe start looking into backup solutions that are easier and faster than tape.