Just switch that wire....

A few weeks ago, I was working on site with a business planning on doing a minor change-over. Scheduled service was for Friday at 1pm. Due to another business having serious issues, I wasn't able to get on site until 3:30, at which time the in-house technician had already started some basic adjustments, nothing serious though.

Our goals were pretty simple. Migrate from the DSL internet connection to the T1 per the instructions of the ISP, adjust the backups on the server to be sure that they were working correctly, and ensure that the antivirus software was working correctly throughout the network, including the satellite offices (via remote desktop) several of which are located in Canada. Estimated time on site: 3 to 6 hours.

Well, our biggest problem was that due to the previous net admin's total lack of documentation, we were running into this with high-hopes while also being a little blind. We totally didn't think about the router configuration. A Cisco 800 SOHO router.

The switch over directions from the ISP were pretty much "just switch over the ethernet cable from the DSL to the T1 and you'll be all set." Well, we didn't have a crossover cable, we just had a standard ethernet. Doesn't work when connecting between the Cisco 800 SOHO and the Cisco 1700 T1 router.

So, after a few hours of troubleshooting, we go that situated with the help of the local ISP. We then realized that the configuration of the Cisco 800 SOHO wasn't correct, but we didn't have the password to it... Here's where the trouble started coming in...

We got the Cisco Console cable from the ISP and instructions on how to reset the password on the router without reseting the configuration. A few simple commands.

Even with our best efforts and constant contact with the ISP, the router configuration was lost. It was now coming up on 2 AM, I had been on site for over 10 hours and was losing the ability to think straight, so we decided to call it a night and resume troubleshooting the following day.

Day two's priorities were different. Get the router up and running. Everything else is secondary. So that's what we did. I contacted the ISP and had them out on site to take a look at the Cisco 800 SOHO and figure out what was wrong and if we could restore the system back into a running state.

The ISP guy informed me that we needed to get the config file from the other network site, to get the configuration details. So at 5 PM, we made the final decision to head deep into Canada and get done what needed to be done.

The drive was going to take 4 hours. Quite a haul. And I don't have a passport, so it was a little risky, for the most part the problems would arise when coming back though, and the priority would be to just go over there and get the config file and email it back to the ISP. Simple.

So, that's what our goals for "the Canada trip" were. As everything else, it didn't go smoothly.

Canadian Boarder Crossing Services stopped us for nearly an hour, (apparently, they don't like Americans that are going into the country to work) but they eventually let us in. We headed on to site an hour behind schedule and got to work right away.

The work was simple... Meet up with one of the Canadian employees to let us in the building, adjust set up a computer near the router, directly access the router, copy the config and send it to the local (American) ISP and have them adjust what needed to be adjusted. We would then test the config to be sure that everything was working and then go home. Total estimated time on site: 45 to 60 minutes.

Once again, things didn't go as planned. Once we got the computer hooked up, we found it to be incredibly slow. And once we started working on the router, we found the data transmission to be faulty. After talking with the American ISP guys to see what they thought, they told us to replace the computer with a different one to check to see if it was a problem with the COM port on the PC. So we did that (it wasn't easy to set up these computers in the networking closet either, it would take around 15 to 20 minutes just to plug everything in). The second computer was no better. But we lived with it. Once again the communication wasn't working correctly though. So we called the American ISP again to see what they thought...

Well, there's several areas of configuration for the HyperTerminal configuration. First, we had to figure out which COM port was used. After some troubleshooting, we figured it must be COM3, it was the only port that displayed anything when we tried to connect to it. But, the port would only display gibberish. After talking with the ISP again, they suggested playing around with the Bit Rate settings. Over an hour of troubleshooting later, we finally found the correct bit rate. We made some setting adjustments and hacked the password on the router (very carefully) and copied the router configuration. All we had to do then was to email it to the American ISP.

That's when we realized that the internet was down. We spent a good fifteen minutes or so troubleshooting that, when we finally gave up and did some resets to the router to get it back online. We finally were able to send the email to the American ISP. Me and the In-House IT guy snacked on some chocolates that the office had laying around. We talked a little about what else we needed to get done and what we needed to test before heading back home.

It didn't take long before the American ISP got back to us and told us that the router should be back online and running as it should be. We did some basic checks and tested the ability to communicate over the VPN and called it good at 5 AM, where we decided to head back to the US.

Once at the boarder at 8 AM, the US Department of Homeland Security didn't take too kindly to us not having passports, but let us back into the country anyway, after inspecting the vehicle. We didn't get back to Owosso until 10 AM, 24 hours after I got up to go to work. I was glad to be home and be free of tinkering with an international network.

There were some problems come Monday, but nothing major and nothing that we couldn't get fixed remotely. By Wednesday, everything was fully operational.