We had a scenario here, which we needed to distribute load between few servers, buying a load balancer was not an option (or at least we wanted to experiment with DNS Load balancing) and it worked great.

Afterall, A load balancer is just another point of break in case your load balancer crash!

Please note that this load balancing won’t check status of servers and randomly return IP addresses (but your visitor won’t see a crashed server!)

Another side effect is that DNS load balancer won’t keep same visitor on same server on each page refresh or link change, so you need a shared session for all servers.

PHP’s default session uses file system and isn’t made for sharing between servers, so you’ll need a shared session manager. memcached is a great option here. sync all your server PHP sessions by connecting them to memcached of a server.

Next step, you need to make sure your database is replicated as master-master and file system is synced (you can use lsyncd for file system sync)
If you have more than 2 servers, you might need to use MySQL 5.7 above, because they offer multi-source master replication, so your slave server can connect to 2 masters at same time.
Before MySQL 5.7, multi-master replication was done using round robin replication which causes big problem in case one of servers go offline.

Now, lets see what will happen when you add multiple A records to your DNS.
so, lets say we have this record:

A   www.hitsteps.com   123.123.123.123

We change it to:

A   www.hitsteps.com   123.123.123.123
A   www.hitsteps.com   123.123.123.124

Where 123.123.123.123 is your first server’s IP address and 123.123.123.124 is your second server IP address.

And it is done!

So here is what happens in our experience. Browser randomly choose one of these IP addresses. since file system and session and database is cached, visitor will always see correct page regardless of which server he is connected to.
It is really random and we cannot

Now, if one of your servers crash and IP address become unavailable, browser will choose another available IP address, therefor your visitors won’t see any downtime. Once crashed server come back online, browser will start serving pages using this server again.

Update: Starting 9th November 2016, Cloudflare will no longer support API v1 which we’ve used in this post. You may look for migrating to v4 and example scripts such as lyoshenka’s great work 😉
Howver you can read our explaination below to figure out how we achieve Failover using CloudFlare service

We use Cloudflare for our service and we use it to have server switchover or failover in case our main server go down, and it been working perfectly ever since!

Wouldn’t it be perfect to have a DNS failover, saving costs by avoiding a hardware and also reduce risk of having main load balancer go down? It is possible to have DNS Load balancing by having two IP Addresses point to same record A, but we go for it in another article.

In this article we will cover how to have CloudFlare perform automatic IP failover for your servers, so in case main server go down, backup server kick in.

Please note for this to work seamlessly, you need your backup server be exactly same as main server. You can achieve this by having MySQL replicated and file system synced using lsyncd. You can also have your backup server show a temporarily message.

If you were to use a real DNS to set IP, it would take hours for new IP to apply to DNS servers around globe, and your visitors would still get pointed to crashed server but not with Cloudflare. new DNS will kick in less than few minutes (1min approx, and max 5mins)

You will need to run monit server in your backup server, ideally you can run monit on a separate server which only have monit server on it. You can get a simple and budget friendly vps linux cloud server from vpsdime and have it check status of your main server.

Let’s cut it short and get to instructions!

Please get your API key from Cloudflare. You can get it from Account tab in cloudflare. Cloudflare is free!

Then you need to shell scripts, one to change IP to backup, and one to change IP to main. Here we have provided both scripts for you. You can download them HERE and set appropriate configs into both files (IP addresses, cloudflare key, email address…)

You need to upload files in your backup server, in this example we use folder /hitsteps/cloudflare/

type in ssh console:

mkdir /hitsteps/
mkdir /hitsteps/cloudflare/

and then upload configured files there.
Make sure you give execute permission to this two files:

chmod +x /hitsteps/cloudflare/*.sh

Now, you need to install monit in your backup server:

yum install monit (or apt-get install monit)

and configure your monit to check your main server status by editing /etc/monit or /etc/monitrc and add following lines to end of it, edit 0.0.0.0 with your main server IP address and change alert email address to your address:

check host 0.0.0.0 with address 0.0.0.0
alert your@email.com
if failed icmp type echo count 3 with timeout 3 seconds for 2 cycles then exec "/bin/bash -c /hitsteps/cloudflare/tobackup.sh"
else if succeeded for 2 cycles then exec "/bin/bash -c /hitsteps/cloudflare/tomain.sh"
if failed url http://0.0.0.0/ with timeout 20 seconds and retry 2 for 2 cycles then exec "/bin/bash -c /hitsteps/cloudflare/tobackup.sh"
#else if succeeded for 2 cycles then exec "/bin/bash -c /hitsteps/cloudflare/tomain.sh"

Now you need to restart monit, and make sure it run on startup:

service monit restart
chkconfig monit on

Guess what? It is all ready! You’ve just setup an auto IP Fail Over using Cloudflare.

Now, monit will check status of your main server by pinging it, if ping failed for 3 times on 3 cycles, then it will fire tobackup.sh file, same will happen if it cannot access webserver for 2 times.
Once server is accessible, it will run tomain.sh.

tobackup.sh will connect to cloudflare API and change your domain IP addresses to backup server. this change usually apply in less than 1 minute by our experience. monit will keep monitor your main server status and change IP back once it is accessible again.

As a sidenote, your cloudflare folder which contain scripts should have write permission because it will write logs into it. you also need to have following apps installed in your linux box for tomain.sh and tobackup.sh parse cloudflare API correctly: bc, cut, curl, sed, head, python

yum install python bc cut curl sed head

Do you have any suggestion to enhance this script? or any feedback? Please do let us know!

SSD make difference!

We’ve been hearing that SSD hard disks are new generation and therefor, all database intensive servers have to move to new SSD servers. We gave it a shot and we never regret it!

In hitsteps, we analyze thousands of hits per seconds. this hits then categorized and processed by a background processor, and archived somewhere along with millions of other hits!

Given searching a visitor or looking back into archive of a visitor is an essential feature of hitsteps, we always had to struggle with performance of database searches, specially when searching for some visitors who visited months ago.

MySQL databases need RAM to hold all indexes of data, they also need a fast disk in case data are not in RAM. so we did the test in a mirrored server:

Our search parameter was a simple query of “Searching for a visitor for who visited hitsteps dashboard who are from Canada since 6 months ago”

HDD server took 45 seconds to return query (almost near to timeout in most apache web server setups). SSD server took only 6 seconds to return query. Going to page number 2 in HDD took 15 seconds, in SSD just a wee 2 seconds.

It worth mentioning all other simple queries take less than 0.2 seconds in SSD servers and 0.5 in HDD servers.

Please do note that this times are closed source and relative to our script and codes. It might be different with other Applications, codes and scripts you might use in your server. RAM also play a major role here which is more appearance if your server is using HDD.