A Public Letter

Today I want to take a minute to write a letter to someone. This person has been posting items about CyberMedics and me on the community forms where I live, posts that I believe are not true.

Dear Poster,

If your intention was to hurt someone's feelings, you should know you have. Your posts on the forms have really hit a key with me and will be taken to heart. I am sorry that your experiences with Don, Joel, and myself have given you cause to be so angered with us; to the point where you don't feel comfortable talking with us anymore.

I respect your rights to post about your experiences at CyberMedics. I can only ask that you post only the truth.

Respectfully,

Daniel,
CyberMedics Computers,
Departments of PC Repair, On Site Support, and Accounting
117 E. Main St.
Owosso, MI 48867
989.729.2667 x115
dan [at] saycomputer [dotcom]


However, if these posts were from a customer, I feel that I must also write a response:

Dear Poster,

I am sorry that you feel that my service to you was less than what you expected. Policy at CyberMedics is to make ethical and logical decisions on the repair process of every computer and to keep the customer informed to the highest level of detail regarding the repair process. If I have failed you in any of these aspects I express my apologies.

As an IT professional it is not only unethical, but could also jeopardize my career if I install the wrong operating system on a computer. That being said, we are required to install the operating system listed on the product code on the side of your computer if we cannot repair the operating system on your system. If I did not make this clear, I take full responsibility for the situation.

Information is the most critical part of any computer system. In your post, you stated that a back up that you paid for was not performed and your information was lost. This is a most serious issue for a PC repair technician. I can only ask that you please stop by CyberMedics, preferably with your receipt or work order, and I will offer a full refund for your data back up and I will personally speak with the owners of the shop to see if I can offer you a full refund for all labour performed on your system.

If you do have any questions or would like to speak with me in person, please feel free to contact me in person at CyberMedics, 117 E. Main St., Owosso, Mi, 48867 or on the phone at 989.729.2667 x115. I am frequently out on service calls, but if you leave me a message, I'll return your call within two business days.

Once again, I offer my apology for any inconvenience you have had.

Daniel,
CyberMedics Computers,
Departments of PC Repair, On Site Support, and Accounting
117 E. Main St.
Owosso, MI 48867
989.729.2667 x115
dan [at] saycomputer [dotcom]

Keepon!

Razr Hell

I cannot emphasize how much I cannot stand this phone anymore. At first, it was beautiful. It had a better camera and was quite stylish to use. Best yet: it was easy to sync up with my MacBook via Bluetooth. Excellent.

That is, until I started playing around with my own mp3 ringtones.

I quickly found out that to get a ringtone onto my razr (a V3m from Sprint, USA - CDMA) phone, all I had to do was upload it to the \audio\ directory and then set the phone to use that as it's ring tone.

Excellent! In a test, I uploaded "My World Down" from the Animation Beck OST. However, I did that a year or so ago now... I've since got a few complaints that the ringtone sounds like a failing HDD and my co-workers freak out each time they hear it. I was getting a little tired of it too, and the ring tone was the full song (around 3 minutes) and over 2 MB, taking up a lot of space on the phone.

So, I decided to delete the mp3 from my razr. Simple enough, right? Just go in through bluetooth and delete the file like I do with the photos.

I keep getting an error, "Operation failed" each time I try to delete the file. There also seems to be no area of the phone itself where I can delete the ringtone. It doesn't show up in the "Ringtones" section of the phone, but it is listed as a ringtone. But, in the "assign a ringtone" section, I cannot delete ringtones. Frustrated, I tried connecting the phone via USB. No luck at all.

After doing some google searching, it seems that the Razr phones will do this to a lot of people. Requiring one to set the phone to silent, load it up as a USB disk drive, go into the "audio" subdirectory, delete two *.db files, and restart before deleting the MP3 can be done. --Getting it to load as a disk drive isn't bad, but I'll also have to use a special "razr explorer" app to view *all* the files on the phone. ~sigh. Okay, whatever... I'll follow the directions and do this.

Well, it didn't turn out so easy. First, the Mac directions won't work on a Razr V3m Sprint CDMA phone. Okay, what's next: using "hack the razr" to do it via USB in Windows XP. No prob, fire up XP in Parallels and we'll be all set. Parallels for some reason tonight freaks out my Mac however. Eating RAM and CPU like nothing else.

So, I'll just grab the link from Firefox and email it to myself and reboot the Mac into Windows XP (boot camp) and we'll go from there. Well, that's when I found out Firefox had freaked out after Parallels used up so much of the system resources. Took some time, but Firefox eventually came back to me. I quickly grabbed the link, pasted it into an email and sent it off, and restarted the Mac into XP.

Once in XP, I found the site really well laid out. Including video examples. Nice! I did exactly what they said, and loaded my phone into the system. Took a while to do everything just so, but it was going well. Drivers successfully loaded all I had to do was plug the phone in and wait for Windows to see it.

Apparently, Windows can't load the correct device drivers and it displays a message, "An error occured... Yadda yadda yadda... Device might not work." Great. Well, Hack the Razr says that I can get the most recent drivers from MotoDev, okay, no problem.

Registration required. Crap. At least it's free. Okay, do that. Signing away my identity again (*garbage in, garbage out*) and logged in. Downloaded the drivers, only to find that the device still won't work. Great, I can't even get the first and most critical step done. Grrrrr.

By now, I've spent over an hour and a half working on this dang phone, just to DELETE A SINGLE RINGTONE!!! I was so enraged, I was willing to just stomp on the phone to be done with it. ~sigh... This happens, lets try Bluetooth in Windows, maybe that'll have better luck... No dice. Bluetooth in Windows is even more shitty than Bluetooth in Mac OS X. Okay, well, the *.db files are probably hidden files, so how about using Onyx on OS X to view hidden files and connect via Bluetooth to delete the files. No dice once again. Too frustrated to continue on, I shut the phone and Mac off and decide that I'll just have to live with the ringtone until I get up the desire to get peeved by it again.

Guess What Broke Down Again

Early this morning, my MS rep called me, wanting mostly to close the ticket. I bet his manager is breathing down his neck to get the account closed... We've already spent nearly 8 hours on the phone together. I finally let him talk me into it.

About 15 minutes later, I found out that the backup from the night before failed. ~sigh. Nothing sucks more than realizing that you just paid Microsoft $259 to waste two days of work. I'm still troubleshooting it now myself, while I've been in contact with my MS rep (via email only, apparently he doesn't feel like calling me back anymore) trying to get the back ups running again.

Damn you Exchange, you suck.

BTW: I also found this guy who actually did hit the 16 GB limit. Thank God that didn't happen in my Exchange environment... But it seems the two of us are having "post maintenance" troubleshooting. I'm glad that at least my users are able to send and receive easily, but without having decent backups, we definitely having a ticking time-bomb that is known as Microsoft Exchange Server.

[update]
I just finished some basic management of Exchange, I ran eseuitl /k (checksum checks) which turned out well, as well as eseutil /d, which defraged the database just fine. I then ran NTBackup and was able to backup the database... However, Veritas backups still don't work. ~sigh. I might look into troubleshooting that, but I'm not so sure that I'm as worried as I was earlier.

Exchange Sucks

I have no clue why Microsoft doesn't really design a better concept for Exchange. It works, and it works really well, until something goes wrong and undetected for a few months.

The repairs I ran with my Microsoft Support guy did get the job done... We did eseutil /k on the priv1.edb and priv1.stm files. Priv1.edb was find. No errors. Priv1.stm: a few thousand errors. My tech literally said "Oh My God" twice as the errors scrolled up the screen.

He then informed me that deleting the STM file and recreating it would be the best option for this situation. Repairing the file just wasn't as likely to succeed.

Unfortunately, both the database and our server were in bad shape. The server has very little storage space available on it. It was so low in fact, that we had to do much of our database management on an external USB hard drive and to expedite our troubleshooting, we also used the tape drive to back up the MDBData directory.

Once we got everything done, which we started at 5 PM and ended at 9:30, we made a backup of the MSExchangeIS and proceeded on to basically conclude the ticket. People were able to access their storage, the database was able to be backed up again, and I was even able to do an offline defrag of the database. Things were looking good and we were done long before I thought we would be.

I left the building after speaking with my "on-site boss." Neither of us got home until well after 11pm. I then spoke with Don, my "boss/mentor" down at CMC. We talked on the phone until 2 AM about the service call and what needed to be covered a few other things that needed to be done on site to be sure that we had everything under control. I was finally able to get into bed at 3AM.

I woke up at 7:30 to turn on my cell phone, just incase something happened on site, I went right back to bed.

At 8:30 AM I received a call from my on-site boss. There were problems. Apparently, after the work done on Monday night, now ever one on site who connected to Exchange had lost all of their external emails. They had emails from the local domain, but anything outside of that, including emails that have been saved for years were lost. It caused quite a stir come Tuesday morning. Fortunately, the most affected people were the IT staff.

I spent some time on the phone with Microsoft technicians and eventually we decided that further work was needed on the database. I worked with a new MS Technician for over an hour, before I got a call from the tech that I worked with the night before. We ended up deciding to switch the contact from my new support technician back to my original and continue work. I talked with my support tech and we decided that we needed to back restore the database to an alternative location and then we needed to get a second server up and running and mount the database on that guy to retrieve the data if at all possible.

Getting a decent computer, installing Windows 2000 Server and MS Exchange 2000 SP3 w/ Roll-up was troublesome enough. It was worse when I got back on site and received a call from my support tech at 4:30 and was informed that "Oops, we can't actually do that. It won't work unless everything is really similar." Great. 4 hours wasted. At least I got half an hour of sleep while the server OS was installed by Joel and Mitchell.

So, my support tech and I decided that it would be best to take our database offline, mount the old one and use ExMerge to pull out the six mailboxes that we needed. The process should be relatively quick compared to Monday's work.

Assuming, that is, if it works... Which it didn't.

ExMerge got one mailbox out of six. It was the largest mailbox too, at nearly 2 GB. At least that was something.

We then went and loaded up client machines and pulled their mailboxes off right from outlook instead of trying to do it through exchange. A much slower process, but at least it is much more likely to get the correct data. It seemed that it had. We remounted the "good" exchange database and all we needed to do was import the data back into Exchange.

Right then and there, my tech seemed really interested in getting me off the phone. We had pulled the mailboxes off of the server to back them up and we had reconnected the good exchange database, but we had not restored anything to Exchange.

I managed to talk him into helping me restore a single mailbox and keeping the trouble ticket open for another 24 hours before he closes it. And he quickly ushered me off of the phone. I can understand, the two of us have had busy days, but it was a little crappy there at the end... And I was not exactly pleased about the whole "Make a server by 4pm!" thing either.

~sigh. This is why I cannot stand Exchange.

Finally, The Horrors of Exchange, part 1

Last week, Wednesday, September 26, 2007, I was assigned to an on-site contract for a local Owosso business. On day 1, I focused on getting oriented with the network there, mostly just familiarizing myself with the 5 on site servers and the one off site server.

The network is interesting, it is international, two exchange servers, and a whole kaboodle of other traits of the system that give it a personality all its own.

Standard IT policy for managing a network, be it from day one administration or having a new administrator come into a network, the first place we look is at the domain controllers and fire up the event viewer to find out what errors the system is having.

I noticed several errors labeled as "MSExchange..." This is one of my nightmares. Exchange is really sensitive, and can be easy to break or lose data if it isn't dealt with according to Microsoft policy. I'm not sure why Microsoft made Exchange in such a sensitive way, but they must have had a reason for it, seeing as how they've been doing it for over a decade now [Microsoft Exchange Server, Wikipedia]. One of the interesting design concepts of Exchange is how messages are stored on the server. Exchange drops messages into one of several databases: priv1.edb priv1.stm pub1.edb pub1.stm or into an ever-increasing amount of log files. But one of the annoying features of Exchange is that it requires everything to be 100% operational for most of it's own functions to work correctly.

For example, while on site, I found that the Exchange wasn't backing up correctly each night and it had been doing this for several months. After doing some tinkering, I also noticed that it was running out of space. The MS Exchange database --the database is priv1.edb + priv1.stm, was 15.9 GB, while Microsoft has set maximum capacity on the Exchange database to be 16 GB. Email doesn't exactly come in at an extreme speed for the network, so it wasn't slated as a major thing, so we scheduled Exchange Database management for the upcoming Saturday, which would give me plenty of time to check to see what commands I need to run on the database and what contingencies I should be prepared to deal with (cough-disaster recovery-cough).

One day went by without major events, but then on Friday, I arrived on site to find that the Exchange had crashed and was re-enabled. It had crashed, according to event viewer, because the hard drive had gone down to less than 10 MB of free space (thankfully, Exchange was hosted on "D drive" instead of on C). However, when I fired up My Computer to see how much space was presently left on the drive, and found it to be under 512 KB, I immediately reported to my on-site boss that the Exchange server was going to go down any minute, due to the drive being filled up again and I was going to have to shut Exchange down to prevent it. He quickly remote into the server to take a look at the data, and noticed that the drive had 2 MB free and was growing?? And then several people immediately stated that they had suddenly lost email... Yup, Exchange had crashed again.

Now, this isn't a huge issue, Exchange is designed to protect itself in situations like this, by shutting down and displaying error messages so we avoid data loss. However, now that the drive had less than 10 MB free, we were in a bit of a bind. Exchange needs to be defraged (that would be the command eseutil /d) --the defrag command deletes information in the database that has been deleted by users or mailboxes that have been deleted by administrators, which should free up a decent bit of space in the database. The recommended procedure by Microsoft is that we take a backup of Exchange and then run the defrag command. While it is very common for the defrag command to cause problems with the database, it is still a possibility, especially if we have undetected hard ware problems.

So started the second fiasco. Backing up Exchange. Now, we have known for a while that the Veritas Backup Exec backups have been failing on that server for a while now, but they were backing up the entire server, and while it was getting an error while backing up Exchange, it wasn't clear if the Exchange errors were causing the backups to fail, or if it was more related to a tape/tape drive issue. So, I decided to just do an NTBackup.exe backup and save it to an external drive. It was redundant enough for our situation.

The backup started, and 2.5 hours later it had not displayed an error message and switched to data verification. 2.5 hours after that (a total of 5 hours after starting the backup), it returned "Backup failed." I checked the logs, and it said "\Mailsore (%servername%)" failed. The database may be corrupt or inaccessible. The file will not restore correctly. This was a becoming a major stressor for me. I have to do a defrag on the database, which a backup prior-to doing the defrag is highly recommended by Microsoft, yet I am unable to do the backup due to a corrupt database.

To keep this story from getting horribly long, I eventually was able to just run Defrag on priv1.edb, pub1.edb, and pub1.stm, which freed up nearly 6 GB of space. However, I was not able to defrag priv1.stm, which seems to be corrupt. I spent a lot of time with other variations of the eseutil command, but wasn't able to get the database back. I could use eseutil /p [database] /i to rebuild the indexes between priv1.edb and priv1.stm, however, the P switch will not only repair the database, but it will also delete anything it determines to be corrupt. Which, frequently will cause other problems with the database. But, there is one nice thing, even if it does delete files that it shouldn't, we can use the *.log files to replay transactions in exchange to get it back up and running with the correct information.

Once again however, my hopes were dashed. To save space, some log files had been deleted. There is no guarantee that we'll be able to use the log files to get the database up and running again. Hopes dashed. And upon further research, I'm suck between a rock and a hard place. There is no other options available. The only options I've got are to break the RAID array and tinker with one half of the mirrored image to see what happens (no guarantee of success, and it is possible to damage the RAID array in the process), or I can call Microsoft and start a support ticket. My boss at CyberMedics suggests the latter, after talking with the guys on site, we'll make the final decision come Monday.

More details as the story progresses.

The 'k' is for krazY!

Krazy 3-D Sudoku!

Now That's Comedy

And yet, it's sooooo true!

Sorry the weekend article has been postponed. Upcoming (sometime soon) will be an article about the horrors of Exchange Server... You'll see why once I post it.