script to remote monitor solaris servers

I have 2 Solaris servers that are mirror images of each other, one being live and the other the backup. I need to have server 2 continually check if server 1 is up, and if it isn't, take-on the identity (IP and hostname) of server 1 and continue working.

I was thinking or hoping I could do this in scripts. Maybe having server 2 ping server 1, and if server 2 doesn't get a response from server 1, become server 1.

I don't know how to do that though, and I'm not a heavy scripter.

If anyone has an idea about how I can accomplist this, please let me know.

Thanks

Sun sells a commercial product to do just this. You set up a private "heartbeat" network (either on the same physical link or a separate one) so that either of the two servers can take on the identity of the other one.

Writing it yourself isn't too bad - testing it is. Do you already have an rdist/rsync script to keep the file systems in sync? That would be the first step. Solaris comes with rdist which is very similar conceptually with the rsync in Linux.

After that, you'll want to look at the ifconfig command so that you can add an IP address when you detect a failure. I say add because it will be much simpler to manage - it won't be a permanent (survives a reboot) unless you also write it to disk.

Your biggest issue may be ARP caching. ARP, the address resolution protocol maps an IP address on your network to an Ethernet address. This information is cached on any machine that ever talked to one of your servers. It has a limited lifespan (TTL or time to live) but it is non-zero. Different O/S's handle this differently. The arp cache on Solaris can get quite big. I'm not sure about Windows.

Basically the issue will be that anything that was talking to the failed server may need some amount of time before it recognizes the second host. If both of the server machines were behind some sort of router or hub that could also be rebooted as part of the fail over process this would speed things up.

Thank you very much.

How can I tell if the server is still alive though? I was thinking about pinging the machine but if it is done in a script, I would have to test some sort of return code from the ping to see if its alive.

Any ideas?

As with almost all Unix programs ping exits with a zero if it is able to successfully ping the host and non-zero if not. A very simple script portion to test that would be something like:



Note that I used bash syntax - other shells are similar but maddeningly different enough to cause problems.

Thank you very much for your help. What is the commercial product that does this?