Exchange Sucks

I have no clue why Microsoft doesn't really design a better concept for Exchange. It works, and it works really well, until something goes wrong and undetected for a few months.

The repairs I ran with my Microsoft Support guy did get the job done... We did eseutil /k on the priv1.edb and priv1.stm files. Priv1.edb was find. No errors. Priv1.stm: a few thousand errors. My tech literally said "Oh My God" twice as the errors scrolled up the screen.

He then informed me that deleting the STM file and recreating it would be the best option for this situation. Repairing the file just wasn't as likely to succeed.

Unfortunately, both the database and our server were in bad shape. The server has very little storage space available on it. It was so low in fact, that we had to do much of our database management on an external USB hard drive and to expedite our troubleshooting, we also used the tape drive to back up the MDBData directory.

Once we got everything done, which we started at 5 PM and ended at 9:30, we made a backup of the MSExchangeIS and proceeded on to basically conclude the ticket. People were able to access their storage, the database was able to be backed up again, and I was even able to do an offline defrag of the database. Things were looking good and we were done long before I thought we would be.

I left the building after speaking with my "on-site boss." Neither of us got home until well after 11pm. I then spoke with Don, my "boss/mentor" down at CMC. We talked on the phone until 2 AM about the service call and what needed to be covered a few other things that needed to be done on site to be sure that we had everything under control. I was finally able to get into bed at 3AM.

I woke up at 7:30 to turn on my cell phone, just incase something happened on site, I went right back to bed.

At 8:30 AM I received a call from my on-site boss. There were problems. Apparently, after the work done on Monday night, now ever one on site who connected to Exchange had lost all of their external emails. They had emails from the local domain, but anything outside of that, including emails that have been saved for years were lost. It caused quite a stir come Tuesday morning. Fortunately, the most affected people were the IT staff.

I spent some time on the phone with Microsoft technicians and eventually we decided that further work was needed on the database. I worked with a new MS Technician for over an hour, before I got a call from the tech that I worked with the night before. We ended up deciding to switch the contact from my new support technician back to my original and continue work. I talked with my support tech and we decided that we needed to back restore the database to an alternative location and then we needed to get a second server up and running and mount the database on that guy to retrieve the data if at all possible.

Getting a decent computer, installing Windows 2000 Server and MS Exchange 2000 SP3 w/ Roll-up was troublesome enough. It was worse when I got back on site and received a call from my support tech at 4:30 and was informed that "Oops, we can't actually do that. It won't work unless everything is really similar." Great. 4 hours wasted. At least I got half an hour of sleep while the server OS was installed by Joel and Mitchell.

So, my support tech and I decided that it would be best to take our database offline, mount the old one and use ExMerge to pull out the six mailboxes that we needed. The process should be relatively quick compared to Monday's work.

Assuming, that is, if it works... Which it didn't.

ExMerge got one mailbox out of six. It was the largest mailbox too, at nearly 2 GB. At least that was something.

We then went and loaded up client machines and pulled their mailboxes off right from outlook instead of trying to do it through exchange. A much slower process, but at least it is much more likely to get the correct data. It seemed that it had. We remounted the "good" exchange database and all we needed to do was import the data back into Exchange.

Right then and there, my tech seemed really interested in getting me off the phone. We had pulled the mailboxes off of the server to back them up and we had reconnected the good exchange database, but we had not restored anything to Exchange.

I managed to talk him into helping me restore a single mailbox and keeping the trouble ticket open for another 24 hours before he closes it. And he quickly ushered me off of the phone. I can understand, the two of us have had busy days, but it was a little crappy there at the end... And I was not exactly pleased about the whole "Make a server by 4pm!" thing either.

~sigh. This is why I cannot stand Exchange.