Updated Remedy Servers

This post is related to For Review.

In my previous post, I detailed how our Remedy ARS Server 7.0.01 patch 003 with MidTier (MT) 7.0.01 patch 010 was generating ARWARN 66 upon logon to the system. Well, our team team implemented ARS Server 7.0.01 patch 011. This rectified the issue, however, it generated a few issues with the default intrinsic permissions in our Remedy forms. In the end, we did have some forms that loaded without any data. Fortunately it was none of our major forms.

A few things to keep in mind when updating is dealing with the multilicense file, which we had some issues with, causing our server to go into Demo mode for about 20 minutes (during our down time window). And planning to go into Admin Only mode for the server... We didn't do this last one, usually our upgrades are pretty smooth, so we let the help desk continue to use Remedy and switch over to our Hot-Standby server. However, this time, when our main server came back online after the upgrade, our system took the services off of the Hot-Standby and placed them onto the main server, which was in Demo mode. ~sigh.

We switched it back over quite quickly, but we were really worried there for a minute.

For Review...

It has been a while since I last really got in-depth with a post. For the past eighteen months, I have been working with one of the (few) growing mid-size (now large) IT companies in the Detroit Metro area. I now work with a ticket tracking system utilizing BMC Action Request System (Remedy) version 7.0.01 (and an older server running Remedy 5). It's a much different work environment than I'm used to, seeing as now I'm doing a lost of administration and some development for this system which supports supports coming up on 2000 support users and also has a directory of a 100,000 people and tracking for nearly 2 million incidents.

Since we deal with complicated contracts and HIPPA regulated information, I am very cautious about what information I provide, but there are a few things that we are working with presently that we just have not been able to solve.

On to the issues...
The Details
Our ARS (7.0.01 patch 003) also runs Mid-Tier 7.0.01 (patch 010). Recently, we began tracking of users logging into our system upon loading of the default form (one of two customized forms that is set as default by the system administrators) and we have set a Temp field which contains a list of the groups (by ID number - eg: "101, 210, 6027" and not "Help Desk, Software Support, and AS400 Development" - this runs a search of our "notification records" form, not of the default BMC Remedy User form's "Groups." This is set so we can list just the support groups, and none of the permissions. We also now don't run into accidental over-flow of information, for example users who are a member of the "Help Desk Triage" group cannot see tickets for the "Help Desk" group in the "Tickets assigned to your other groups" table on our primary default form (the original cause of this problem was the command $GROUPS$ LIKE "something%", which is enough of a wild-card to cause these issues, but not something that we can really process in a way that is significantly different.

Not long after these two new concepts were deployed, we began getting the following message from using logging into the Mid Tier: "ARWARN 66 - The query matched more than the maximum number of entries specified for retrieval." and in most situations, the error message will display twice.

The Conditions
The only time this message occurs is when a user has logged into the Mid Tier (web-version of Remedy) and is a non-Administrator user. The message only occurs on one of the default forms (as far as we have been able to detect). It affects users system-wide, regardless of if they are logging in via the web locally or remotely, it even occurs in Firefox, IE, and doesn't seem to be cross-platform (Windows, Mac, Linux) sensitive. The warning message does not occur on the Windows User Tool (WUT) which most users have installed 7.0.01 patch 10, nor does it occur in our Citrix implementation of Remedy (which are also based off of patch 10).

One of our active links has required us to use the special run-if qualification: $CLIENT-TYPE$ != 9 (or whatever it is for the Mid-Tier) allowing us to write a second active link for $CLIENT-TYPE$ = 9. The workflow is associated with the support group listing detailed above.

The default form that loads has several tabs, most of which contains between 1 and 4 tables worth of data that runs some type of query. When right clicking on the table and selecting "Refresh" the error does not occur. Only when loading the form itself. NOTE: There are several hidden tables which are inaccessible to regular users (including my test ID) which I cannot refresh with our current implementation.

Troubleshooting
Note: Only our PRODUCTION system currently connects to the Mid Tier. All testing has been preformed on our production system, with the specialized workflow limited down to execute only for our test user.

We have tried setting and clearing the limits in the form AR System User Preferences. No change

We have attempted to modify the active links and filters that are associated with the two bits of code above which would "deflect" my test user logon from executing any of the recently implemented code that seems to generate the error.

We have tested the limitations of the server by running high-result searches, which up to 10,000 entries have returned successfully (albeit a little slowly).

And for those ARS Admins concerned about the MT's Cache, we are flushing it after implementing new test code and before logging into the MT again.

We have checked arerror.log and found nothing major in there.

We have ran Active Link and Filter logging from both the WUT and the MT. We have also ran Filter logging on the server. Nothing shows "ARWARN 66" but the last bit of code that executes in all three situations before the ARWARN displays is the code for tracking users logging into the system.

FWIW: We are running in a Windows Server 2003 environment with MS SQL 2005. Some of our code was pushed out from ARS Admin Tool 7.0.01 patch 10 into our patch 3 server. We have resaved the primary default form from a downgraded (patch 3) version of the Admin tool. We have not changed the code on any of the tables.

Situation Analysis
After working with one of our vendors, we are working toward upgrading the server itself from 7.0.01 patch 003 to patch 010 due to a BMC ARS Software bug that causes ARWARN 66 when a filter has executed with an "Create New Request" or "Modify first result" action in it. Unfortunately, due to scheduling conflicts, we keep having to push this off.

However, we have somewhere around two-thousand filters in our system, none of which have caused this issue in the past. Many of which execute only on other forms, but upon loading of those other forms, even now, we still do not get this warning.

I am curious to find out if anyone else has had the ARWARN 66 mysteriously pop-up on their ARS Servers and what you have done to fix it or what troubleshooting steps you took to pin-down the issue.

iPod Insanity

See Update Below

I am really in a strange situation here... I've got my good ol' iPod Nano (gen 1 baby!), where I keep a lot of my tech Podcasts in a Playlist so that I can easily manage just one show at a time.

Today however, I realized that in the Windows Weekly podcast playlist shows that episode 75 is still the first on the list, but I'm up-to-date with the podcast and should have just the most recently episodes (episodes 100+) on there. So I did some checking, and it looks like the iPod has Episode 75 listed in the playlist, but nowhere else on the iPod.

I don't want to just delete it from the playlist, I want to remove it completely from the iPod. The file has long since been removed from my Mac. So the question is, where is the data file at on the iPod?

My first thought, "Oh, it must have been classified incorrectly, and is hanging out in the Music folder." WRONG. I searched there, to no avail. I even checked the Audiobook area just incase, but I also had no luck with that.

Then I thought, "Well, it's gotta be on the iPod somewhere, so if I do a 'Get Info' command on the file, I should have some rudimentary information about where the file is located at on the iPod." WRONG. Get Info did not list a single thing about the actual location of the audio file. Just gave me an indication that the file was playable on the iPod and had several variables that I could set.

That's when I realize that I might be able to push the episode from being a "Podcast" to being "Music" or something. I tried that out as well from the Get Info prompt, but had no luck. I can change the Genera, but the media type is greyed out.

I tried moving the song from the playlist to the iPod in general, but also had no luck. I tried moving it to my Music on iTunes and to the Mac desktop, but I had no luck in this at all. The only thing I can do (or so it seems) is move it between playlists on the iPod or delete the entry from the playlist (and I would rather not have the space wasted on an entire episode like that).

Googling so far has not returned any useful results. I'm not even sure what to be looking for, just trying to come up with some solution.

I'm going to keep looking, but I've got a looming fear that I may have a decision to make, either format the iPod to remove the episode, or just delete the file from the playlist and not worry about the extra space that it is taking up.

***Update
Not wanting to give up so easily, I fired up the one sure way to view the contents of the *true* file system, and not these virtual folders with hidden objects that Finder gives you, I'd use Terminal!

I enabled my root user (for just a bit, I promise) and did

AkiraBook:~indigo$ su
Password:
AkiraBook:/Users/indigo root# cd /Volumes/Saji-chan/iPod_Control/Music/F03
AkiraBook:/Volumes/Saji-chan/iPod_Control/Music/F03

From here, I did an ls -sk to show me the files and size in Kb. I looked though F00 to F05, but there was only one file that was close, WDXV.mp3 that was 35700-some Kb, which is really close to the 35.1 Mb that I was looking for. Not wanting to be unsure, I copied the file to the desktop and opened it. Sure enough, it opened in iTunes and said that it was "Windows Weekly 75."

I did a quick rm -v command, and it got rid of the file. The iPod now has an additional 35 MB of free space, and I promptly removed episode 75 from the Windows Weekly playlist. SUCCESS!

so long, old friend

Last night, I came to the conclusion that my old iPod with Video (5G) had finally died out, it was my first iPod, my first HDD-based MP3 player, and the first to be over 1GB in capacity (my iPod was 30 GB).

The symptoms of impending failure were very minor. For about two years now, I've occasionally got the "You need to restore this iPod" message at start up or as I connected the device to iTunes, but the messages were really rare, and when they did occur, disconnecting the iPod and doing a reset (hold down "Select" and "Menu" for 6+ seconds) cleared up the issue. Occasionally, the iTunes database would be corrupted, and I would have to restore the iPod. Very annoying, but not the worst thing that could happen.

That is, until yesterday, when I synched up some video podcasts. I was getting behind on them and decided that using the iPod 5G would be the best option to get caught up -- the battery on my nano is starting to go anyway, requiring it to be on the charger more and more these days than. However, upon trying to play the video podcasts on the iPod 5G, the video would start although it would eventually have some type of processing problem and skip to the next episode or continue playing the sound while not updating the video. Sometimes, it would just give up and out-right freeze, requiring a reset of the device to get it up and running again. At first, I thought this was just symptoms of a poorly encoded video file stressing the iPod's CPU and causing it to crash and require a reset.

Unfortunately these frequent resets only seemed to make things worse over the evening. After two or three of those force-resets, the device powered back on and said that it was corrupted and needed to be connected to iTunes to be restored.

The restore went fine, but when I attempted to synch for the first time (a manual synch of a few dozen audio podcasts), the iPod crashed again and needed to be reset, and once again, restored.

But, this time, the restore function didn't work out so well. It seemed to freeze up as the iPod System Software was being installed, and even though the device wasn't disconnected from the Macintosh, the Mac said that it had been improperly removed. Retrying the restore didn't work either, the device was getting worse, freezing right after being connected via USB and by this time, it was crashing iTunes as well.

It seemed I was out of luck. None of the diagnostics said that there was anything wrong with the iPod, yet it seems to be having more and more difficulty starting up. It even seems to have trouble getting past the Apple logo and into diag mode now, which deepened my concern that the iPod was in serious hardware failure.

I'm pretty sure that it is hard drive failure that is getting to the device, but the Apple system diagnostics (upon reset, press and hold "Select" and "Previous" as the iPod is starting up), I found nothing wrong listed in the system diagnostics for the HDD... But, the diagnostics only list the SMART data for the drive, it doesn't actually do any sector by sector analysis.

I did do some research on running SprinRite on the iPod, but it looks like if I am to do that, I'll need to order special drive converters, disassemble the iPod and then attach the iPod's HDD to my computer (via IDE) and run SpinRite that way. I was hoping for an easier fix, such as being able to run SpinRite via USB, but it seems "via usb" isn't an option.

My only other recourse available last night was to run Disk Utility on the device, which returned no problems. The only remaining thing I can think of to try would be some of the Mac drive diagnostic utilities, which may have something in there that can work on a USB drive - I haven't gone though this option as of yet.

For the interim, I am more than able to just get by with my iPod nano (1G). I'll be looking forward to getting the latest iPod nano once the battery in this one starts to go so I can start watching videos on my iPod again, yet at the same time, I really will miss my nano 1G. Perhaps a trip to the Apple store is warranted after work sometime this week.

The eMac Horror

Today, Steve (one of my co-workers at CyberMedics moved his eMac from home down to the store. It's a G4 700 Mhz system, not the latest eMac, but still a nice little machine, and quite speedy when booting into Mac OS X 10.4.

The only issue is that this beautiful, although, oddly designed:

system kept booting into Open Firmware (Windows users, think BIOS). Occasionally, the eMac, wouldn't even boot up! He was not in the greatest mood over it, especially since recently the system was functioning normally.

So, since the issue of the eMac booting into Open Firmware at each power on is a symptom of bad memory, he decided to take the back side of the shell off and take a look at the innards, revealing an image similar to this:

Needless to say, doing a minor memory adjustment is not cool on an eMac, but it needed to be checked. As Steve was tinkering with the eMac, he found what was really causing the trouble: a small part on the back of the power button had broken off, leaving the eMac thinking that the power button was being held down each time the unit was powered on. The part requires soldering, but is way too small for our utilities at CMC, so Steve is just going ahead and ordering a new power button. But man, was that eMac ever troublesome!

Vista Vista bo Bista banana fana foe Fista, mi my moe Mi-sta... Vista.

What do you do when your state-of-the-art OS on a high-end laptop won't connect to the Internet? Ya call CyberMedics.

We just had a laptop come in, exceptionally new laptop, with Vista Home Premium pre-installed on it. The computer was able to get online for a while and then it wasn't able to. Our "Triage" diagnostic (performed by yours-truly) came up with a preliminary report that Norton Internet Security or the built in Windows/Internet Explorer security settings were set so high that they prevented the laptop from getting online. Triage showed that the laptop was able to get online, and was able to ping the router and DNS server, but it wasn't able to ping any domains, use IE7, or utilize any third-party programs to access the Internet.

After working with the laptop for a while, I was confused to say the least. I lowered much of the security related to connecting to the Internet, including Norton Internet Security. I even went through some of the automated diagnostic software included with Windows to check the system over (which seemed to work well, but in turn, revealed nothing of consequence). However, at one reboot of the system, MySpace IM was able to get online, however no other applications were. Thinking there must be a software issue going on, I restarted the computer in safe mode.

It was the first time I attempted to boot Vista into safe mode. I managed to hit the wrong key the first time, which brought up the Windows Boot Loader, much like Grub in Linux, upon hitting the correct key on the Grub screen will allow the user to pass different startup options to adjust how the OS boots. Now, it seems that Windows has a similar feature. I quickly showed by boss who was quite surprised and pleased with this new feature of Vista.

Not to become distracted by a new feature, I moved right along to get the system into Safe Mode. The first thing I noticed about the new "Advanced Startup Options" menu was that there was a "reverse print" title bar at the top that said "Advanced Startup Options," anyone who has used the Linux "man pages" knows what this looks like, the centered black text with the white "bar-like" background stretching from one side of the screen to the other.

There was also a new option, "Repair your computer" [link]. This looks like the Windows XP repair installation that has so many times saved a system that has crossed my bench at CyberMedics. If it is, and I am having trouble finding much details on the "repair your computer" option in the Vista ASO, then it would simplify matters from XP, which always had to be run from the CD after going through the preliminary section of the Windows XP installer. By the way, one should take note that the "Repair your computer" options is not always on the ASO screen, it seems that the option is only available if the system has the necessary system files stored on the computer itself.

The last thing that I noticed about the new ASO screen is that as you move between options, a section at the bottom (in bright white text) displays a description of what the option will do. This should help decrease the number of people that select Debugging or Directory Services Restore Mode... Neither of which are really needed on the average home user's computer.

Oh, and that customer's computer, after entering safe mode, the Internet was working correctly. 10 to 1 odds: the Internet isn't working correctly in Vista because of a conflict in Norton Internet Security.