Recently at the request of our vendor (HP), I went and had HP Systems Insight Manager updated to 5.3 about 3 weeks. Now this isn't "bleeding edge" and so I didn't feel it was a big deal to jump up to it already. Well over the past 3 weeks I've been getting this new alert coming from our SIM system and I just figured "Oh.. something new" and didn't fret about it too much.

Event Name: (SNMP) Remote Insight/ Integrated LightsOut Interface Error (9006)

URL: https://HPSIM:2381/

Event originator: HPSIM

Event Severity: Major

Event received: 12-Mar-2009, 17:31:02

Event description: Remote Insight/ Integrated Lights-Out Interface Error. The host OS has detected an error in the Remote Insight/ Integrated Lights-Out interface. The firmware is not responding.

As you can tell from my lazy comments, this became a pretty big deal as iLO was completely toasty on these boxes where SIM was generating this alert. I contacted HP and it was a pretty entertaining exchange.

HP Support: Are you running HP SIM 5.3?

Me: Yes

HP Support: Do you have iLO version 1.60 or 1.61?

Me: Beats me. The iLOs are not responding so I have zero way to tell.

HP Support: I think I know your problem.

Me: Well don't leave me hanging here.

HP Support: I'm reading an email alert that came through while we were talking. There's a hotfix for HP SIM 5.3 and a recommendation to go up to firmware 1.70 for the iLO.

Me: Oh great. *sigh*

So now I get to physically pull power cords to fix iLOs everywhere. (Yes this means an OS reboot too.)

* Hey HP. Put a button onto the servers to "reboot the iLO".