Guess What Broke Down Again

Early this morning, my MS rep called me, wanting mostly to close the ticket. I bet his manager is breathing down his neck to get the account closed... We've already spent nearly 8 hours on the phone together. I finally let him talk me into it.

About 15 minutes later, I found out that the backup from the night before failed. ~sigh. Nothing sucks more than realizing that you just paid Microsoft $259 to waste two days of work. I'm still troubleshooting it now myself, while I've been in contact with my MS rep (via email only, apparently he doesn't feel like calling me back anymore) trying to get the back ups running again.

Damn you Exchange, you suck.

BTW: I also found this guy who actually did hit the 16 GB limit. Thank God that didn't happen in my Exchange environment... But it seems the two of us are having "post maintenance" troubleshooting. I'm glad that at least my users are able to send and receive easily, but without having decent backups, we definitely having a ticking time-bomb that is known as Microsoft Exchange Server.

[update]
I just finished some basic management of Exchange, I ran eseuitl /k (checksum checks) which turned out well, as well as eseutil /d, which defraged the database just fine. I then ran NTBackup and was able to backup the database... However, Veritas backups still don't work. ~sigh. I might look into troubleshooting that, but I'm not so sure that I'm as worried as I was earlier.

Exchange Sucks

I have no clue why Microsoft doesn't really design a better concept for Exchange. It works, and it works really well, until something goes wrong and undetected for a few months.

The repairs I ran with my Microsoft Support guy did get the job done... We did eseutil /k on the priv1.edb and priv1.stm files. Priv1.edb was find. No errors. Priv1.stm: a few thousand errors. My tech literally said "Oh My God" twice as the errors scrolled up the screen.

He then informed me that deleting the STM file and recreating it would be the best option for this situation. Repairing the file just wasn't as likely to succeed.

Unfortunately, both the database and our server were in bad shape. The server has very little storage space available on it. It was so low in fact, that we had to do much of our database management on an external USB hard drive and to expedite our troubleshooting, we also used the tape drive to back up the MDBData directory.

Once we got everything done, which we started at 5 PM and ended at 9:30, we made a backup of the MSExchangeIS and proceeded on to basically conclude the ticket. People were able to access their storage, the database was able to be backed up again, and I was even able to do an offline defrag of the database. Things were looking good and we were done long before I thought we would be.

I left the building after speaking with my "on-site boss." Neither of us got home until well after 11pm. I then spoke with Don, my "boss/mentor" down at CMC. We talked on the phone until 2 AM about the service call and what needed to be covered a few other things that needed to be done on site to be sure that we had everything under control. I was finally able to get into bed at 3AM.

I woke up at 7:30 to turn on my cell phone, just incase something happened on site, I went right back to bed.

At 8:30 AM I received a call from my on-site boss. There were problems. Apparently, after the work done on Monday night, now ever one on site who connected to Exchange had lost all of their external emails. They had emails from the local domain, but anything outside of that, including emails that have been saved for years were lost. It caused quite a stir come Tuesday morning. Fortunately, the most affected people were the IT staff.

I spent some time on the phone with Microsoft technicians and eventually we decided that further work was needed on the database. I worked with a new MS Technician for over an hour, before I got a call from the tech that I worked with the night before. We ended up deciding to switch the contact from my new support technician back to my original and continue work. I talked with my support tech and we decided that we needed to back restore the database to an alternative location and then we needed to get a second server up and running and mount the database on that guy to retrieve the data if at all possible.

Getting a decent computer, installing Windows 2000 Server and MS Exchange 2000 SP3 w/ Roll-up was troublesome enough. It was worse when I got back on site and received a call from my support tech at 4:30 and was informed that "Oops, we can't actually do that. It won't work unless everything is really similar." Great. 4 hours wasted. At least I got half an hour of sleep while the server OS was installed by Joel and Mitchell.

So, my support tech and I decided that it would be best to take our database offline, mount the old one and use ExMerge to pull out the six mailboxes that we needed. The process should be relatively quick compared to Monday's work.

Assuming, that is, if it works... Which it didn't.

ExMerge got one mailbox out of six. It was the largest mailbox too, at nearly 2 GB. At least that was something.

We then went and loaded up client machines and pulled their mailboxes off right from outlook instead of trying to do it through exchange. A much slower process, but at least it is much more likely to get the correct data. It seemed that it had. We remounted the "good" exchange database and all we needed to do was import the data back into Exchange.

Right then and there, my tech seemed really interested in getting me off the phone. We had pulled the mailboxes off of the server to back them up and we had reconnected the good exchange database, but we had not restored anything to Exchange.

I managed to talk him into helping me restore a single mailbox and keeping the trouble ticket open for another 24 hours before he closes it. And he quickly ushered me off of the phone. I can understand, the two of us have had busy days, but it was a little crappy there at the end... And I was not exactly pleased about the whole "Make a server by 4pm!" thing either.

~sigh. This is why I cannot stand Exchange.

Finally, The Horrors of Exchange, part 1

Last week, Wednesday, September 26, 2007, I was assigned to an on-site contract for a local Owosso business. On day 1, I focused on getting oriented with the network there, mostly just familiarizing myself with the 5 on site servers and the one off site server.

The network is interesting, it is international, two exchange servers, and a whole kaboodle of other traits of the system that give it a personality all its own.

Standard IT policy for managing a network, be it from day one administration or having a new administrator come into a network, the first place we look is at the domain controllers and fire up the event viewer to find out what errors the system is having.

I noticed several errors labeled as "MSExchange..." This is one of my nightmares. Exchange is really sensitive, and can be easy to break or lose data if it isn't dealt with according to Microsoft policy. I'm not sure why Microsoft made Exchange in such a sensitive way, but they must have had a reason for it, seeing as how they've been doing it for over a decade now [Microsoft Exchange Server, Wikipedia]. One of the interesting design concepts of Exchange is how messages are stored on the server. Exchange drops messages into one of several databases: priv1.edb priv1.stm pub1.edb pub1.stm or into an ever-increasing amount of log files. But one of the annoying features of Exchange is that it requires everything to be 100% operational for most of it's own functions to work correctly.

For example, while on site, I found that the Exchange wasn't backing up correctly each night and it had been doing this for several months. After doing some tinkering, I also noticed that it was running out of space. The MS Exchange database --the database is priv1.edb + priv1.stm, was 15.9 GB, while Microsoft has set maximum capacity on the Exchange database to be 16 GB. Email doesn't exactly come in at an extreme speed for the network, so it wasn't slated as a major thing, so we scheduled Exchange Database management for the upcoming Saturday, which would give me plenty of time to check to see what commands I need to run on the database and what contingencies I should be prepared to deal with (cough-disaster recovery-cough).

One day went by without major events, but then on Friday, I arrived on site to find that the Exchange had crashed and was re-enabled. It had crashed, according to event viewer, because the hard drive had gone down to less than 10 MB of free space (thankfully, Exchange was hosted on "D drive" instead of on C). However, when I fired up My Computer to see how much space was presently left on the drive, and found it to be under 512 KB, I immediately reported to my on-site boss that the Exchange server was going to go down any minute, due to the drive being filled up again and I was going to have to shut Exchange down to prevent it. He quickly remote into the server to take a look at the data, and noticed that the drive had 2 MB free and was growing?? And then several people immediately stated that they had suddenly lost email... Yup, Exchange had crashed again.

Now, this isn't a huge issue, Exchange is designed to protect itself in situations like this, by shutting down and displaying error messages so we avoid data loss. However, now that the drive had less than 10 MB free, we were in a bit of a bind. Exchange needs to be defraged (that would be the command eseutil /d) --the defrag command deletes information in the database that has been deleted by users or mailboxes that have been deleted by administrators, which should free up a decent bit of space in the database. The recommended procedure by Microsoft is that we take a backup of Exchange and then run the defrag command. While it is very common for the defrag command to cause problems with the database, it is still a possibility, especially if we have undetected hard ware problems.

So started the second fiasco. Backing up Exchange. Now, we have known for a while that the Veritas Backup Exec backups have been failing on that server for a while now, but they were backing up the entire server, and while it was getting an error while backing up Exchange, it wasn't clear if the Exchange errors were causing the backups to fail, or if it was more related to a tape/tape drive issue. So, I decided to just do an NTBackup.exe backup and save it to an external drive. It was redundant enough for our situation.

The backup started, and 2.5 hours later it had not displayed an error message and switched to data verification. 2.5 hours after that (a total of 5 hours after starting the backup), it returned "Backup failed." I checked the logs, and it said "\Mailsore (%servername%)" failed. The database may be corrupt or inaccessible. The file will not restore correctly. This was a becoming a major stressor for me. I have to do a defrag on the database, which a backup prior-to doing the defrag is highly recommended by Microsoft, yet I am unable to do the backup due to a corrupt database.

To keep this story from getting horribly long, I eventually was able to just run Defrag on priv1.edb, pub1.edb, and pub1.stm, which freed up nearly 6 GB of space. However, I was not able to defrag priv1.stm, which seems to be corrupt. I spent a lot of time with other variations of the eseutil command, but wasn't able to get the database back. I could use eseutil /p [database] /i to rebuild the indexes between priv1.edb and priv1.stm, however, the P switch will not only repair the database, but it will also delete anything it determines to be corrupt. Which, frequently will cause other problems with the database. But, there is one nice thing, even if it does delete files that it shouldn't, we can use the *.log files to replay transactions in exchange to get it back up and running with the correct information.

Once again however, my hopes were dashed. To save space, some log files had been deleted. There is no guarantee that we'll be able to use the log files to get the database up and running again. Hopes dashed. And upon further research, I'm suck between a rock and a hard place. There is no other options available. The only options I've got are to break the RAID array and tinker with one half of the mirrored image to see what happens (no guarantee of success, and it is possible to damage the RAID array in the process), or I can call Microsoft and start a support ticket. My boss at CyberMedics suggests the latter, after talking with the guys on site, we'll make the final decision come Monday.

More details as the story progresses.