Recently at the request of our vendor (HP), I went and had HP Systems Insight Manager updated to 5.3 about 3 weeks. Now this isn't "bleeding edge" and so I didn't feel it was a big deal to jump up to it already. Well over the past 3 weeks I've been getting this new alert coming from our SIM system and I just figured "Oh.. something new" and didn't fret about it too much.
Event Name: (SNMP) Remote Insight/ Integrated LightsOut Interface Error (9006)
Event originator: HPSIM
Event Severity: Major
Event received: 12-Mar-2009, 17:31:02
Event description: Remote Insight/ Integrated Lights-Out Interface Error. The host OS has detected an error in the Remote Insight/ Integrated Lights-Out interface. The firmware is not responding.
As you can tell from my lazy comments, this became a pretty big deal as iLO was completely toasty on these boxes where SIM was generating this alert. I contacted HP and it was a pretty entertaining exchange.
HP Support: Are you running HP SIM 5.3?
HP Support: Do you have iLO version 1.60 or 1.61?
Me: Beats me. The iLOs are not responding so I have zero way to tell.
HP Support: I think I know your problem.
Well don't leave me hanging here.
HP Support: I'm reading an email alert that came through while we were talking. There's a hotfix for HP SIM 5.3 and a recommendation to go up to firmware 1.70 for the iLO.
Me: Oh great. *sigh*
So now I get to physically pull power cords to fix iLOs everywhere. (Yes this means an OS reboot too.)
* Hey HP. Put a button onto the servers to "reboot the iLO".