Load Balancer with “LVS + Keepalived + DSR”

> to Japanese Pages
1. Summary
In this post, I will explain the effectiveness of the load balancer solution by “LVS + Keepalived + DSR” design technology and explain how to build it.

2. Introduction
The load balancer solution by “LVS + Keepalived + DSR” is a mature technology but I have posted this solution because I was asked by my friends.

For highly scalable projects, the topic of the load balancer is an agenda at least once in the system performance meeting.
I have done a lot of such experiences.
And we will have the opportunity to hear negative opinions about the performance of the software load balancer.
In such a case, the name of a hardware load balancer like BIG-IP sometimes comes up to the topic of that agenda.
However, we can not miss the fact that a load balancer using “LVS + Keepalived + DSR” design technology runs at 100% SLA and 10% load factor in our project receiving 1 million accesses per day.
This demonstrates that this design technology is one of the effective load balancer solutions in cloud hosting without load balancer PaaS or on premises.
Such a result is brought about by using the communication method called Direct Server Return (DSR).
The dramatic load reduction of the load balancer is realized by the feature of “returning it directly to the client without going through communication from the lower node” of the DSR.
In addition, this solution is not affected by various hardware related problems (failure, deterioration, support contract, support quality, end of product support, etc.).

In this post, I will explain how to build “LVS + Keepalived + DSR” design.
In addition, in this post, I will not specifically conduct benchmarks such as “DSR VS. Not DSR”.

3. Environment
In this post, I will explain the solution based on the following assumptions.
CentOS 7
Keepalived
ipvsadm
Firewalld

In this post, I will explain the solution based on the following system configuration diagram.



4. Install
First, we install the “Keeplived” on the Load Balancer 1.
$ sudo yum -y install keepalived

Next, we install the “Keeplived” on the Load Balancer 2.
$ sudo yum -y install keepalived

Next, we install the “ipvsadm” on the Load Balancer 1.
$ sudo yum -y install ipvsadm

Next, we install the “ipvsadm” on the Load Balancer 2.
$ sudo yum -y install ipvsadm


5. Configuration
Next, we configure the “firewalld” on the Web Server 1.
We startup the “firewalld” and enable it.
$ sudo systemctl start firewalld
$ sudo systemctl enable firewalld
$ sudo systemctl status firewalld

We configure the “firewalld.”
$ sudo firewall-cmd --set-default-zone=internal
$ sudo firewall-cmd --add-port=22/tcp --zone=internal
$ sudo firewall-cmd --add-port=22/tcp --zone=internal --permanent
$ sudo firewall-cmd --add-port=80/tcp --zone=internal
$ sudo firewall-cmd --add-port=80/tcp --zone=internal --permanent
$ sudo firewall-cmd --add-port=443/tcp --zone=internal
$ sudo firewall-cmd --add-port=443/tcp --zone=internal --permanent
$ sudo firewall-cmd --direct --add-rule ipv4 nat PREROUTING 0 -d 10.0.0.3 -j REDIRECT
$ sudo firewall-cmd --permanent --direct --add-rule ipv4 nat PREROUTING 0 -d 10.0.0.3 -j REDIRECT
$ sudo firewall-cmd --direct --add-rule ipv4 nat PREROUTING 0 -d 10.0.0.5 -j REDIRECT
$ sudo firewall-cmd --permanent --direct --add-rule ipv4 nat PREROUTING 0 -d 10.0.0.5 -j REDIRECT

We reload the “firewalld” and confirm the configuration.
$ sudo firewall-cmd --reload
$ sudo firewall-cmd --list-all-zone
$ sudo firewall-cmd --direct --get-rule ipv4 nat PREROUTING

We use the “telnet” command to verify the communication of the Web Server 1.
$ sudo telnet 10.0.0.3 80

Next, we configure the “firewalld” on the Web Server 2.
We startup the “firewalld” and enable it.
$ sudo systemctl start firewalld
$ sudo systemctl enable firewalld
$ sudo systemctl status firewalld

We configure the “firewalld.”
$ sudo firewall-cmd --set-default-zone=internal
$ sudo firewall-cmd --add-port=22/tcp --zone=internal
$ sudo firewall-cmd --add-port=22/tcp --zone=internal --permanent
$ sudo firewall-cmd --add-port=80/tcp --zone=internal
$ sudo firewall-cmd --add-port=80/tcp --zone=internal --permanent
$ sudo firewall-cmd --add-port=443/tcp --zone=internal
$ sudo firewall-cmd --add-port=443/tcp --zone=internal --permanent
$ sudo firewall-cmd --direct --add-rule ipv4 nat PREROUTING 0 -d 10.0.0.4 -j REDIRECT
$ sudo firewall-cmd --permanent --direct --add-rule ipv4 nat PREROUTING 0 -d 10.0.0.4 -j REDIRECT
$ sudo firewall-cmd --direct --add-rule ipv4 nat PREROUTING 0 -d 10.0.0.5 -j REDIRECT
$ sudo firewall-cmd --permanent --direct --add-rule ipv4 nat PREROUTING 0 -d 10.0.0.5 -j REDIRECT

We reload the “firewalld” and confirm the configuration.
$ sudo firewall-cmd --reload
$ sudo firewall-cmd --list-all-zone
$ sudo firewall-cmd --direct --get-rule ipv4 nat PREROUTING

We use the “telnet” command to verify the communication of the Web Server 2.
$ sudo telnet 10.0.0.4 80

Next, we configure the “Keepalived” on the Load Balancer 1.
$ sudo cp -a /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.org
$ sudo vim /etc/keepalived/keepalived.conf

; Common Configuration Block
global_defs {
    notification_email {
        alert@example.com
    }
    notification_email_from lb1@example.com
    smtp_server mail.example.com
    smtp_connect_timeout 30
    router_id lb1.example.com
}

; Master Configureation Block
vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 1
    priority 101
    nopreempt
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass foo
    }
    virtual_ipaddress {
        10.0.0.5/24 dev eth0
    }
}

; Virtual Server Configureation Block
virtusl_server 10.0.0.5 80 {
    delay_loop 6
    lvs_sched rr
    lvs_method DR
    persistence_timeout 50
    protocol TCP
    sorry_server 10.0.0.254 80
    real_server 10.0.0.3 80 {
        weight 1
        inhibit_on_failure
        HTTP_GET {
            url {
                path /
                status_code 200
            }
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }
    real_server 10.0.0.4 80 {
        weight 1
        inhibit_on_failure
        HTTP_GET {
            url {
                path /
                status_code 200
            }
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }
}

$ sudo systemctl start keepalived

In case of failback prohibition, you should disable automatic startup of “Keepalived”.
$ :sudo systemctl enable keepalived
$ sudo systemctl status keepalived
$ sudo ip addr

Next, we configure the “Keepalived” on the Load Balancer 2.
$ sudo cp -a /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.org
$ sudo vim /etc/keepalived/keepalived.conf

; Common Configuration Block
global_defs {
    notification_email {
        admin@example.com
    }
    notification_email_from lb2@example.com
    smtp_server mail.example.com
    smtp_connect_timeout 30
    router_id lb2.example.com
}

; Backup Configureation Block
vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 1
    priority 100
    nopreempt
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass foo
    }
    virtual_ipaddress {
        10.0.0.5/24 dev eth0
    }
}

; Virtual Server Configureation Block
virtusl_server 10.0.0.5 80 {
    delay_loop 6
    lvs_sched rr
    lvs_method DR
    persistence_timeout 50
    protocol TCP
    sorry_server 10.0.0.254 80
    real_server 10.0.0.3 80 {
        weight 1
        inhibit_on_failure
        HTTP_GET {
            url {
                path /
                status_code 200
            }
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }
    real_server 10.0.0.4 80 {
        weight 1
        inhibit_on_failure
        HTTP_GET {
            url {
                path /
                status_code 200
            }
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }
}

$ sudo systemctl start keepalived

In case of failback prohibition, you should disable automatic startup of “Keepalived”.
$ :sudo systemctl enable keepalived
$ sudo systemctl status keepalived
$ sudo ip addr

Next, we change the kernel parameters on the Load Balancer 1.
$ sudo vim /etc/sysctl.conf

# Enable Packet Transfer between Interfaces
net.ipv4.ip_forward = 1

# Do not discard packets from networks that do not belong to the interface.
net.ipv4.conf.all.rp_filter = 0

We reflect the setting of the kernel parameters.
$ sudo sysctl -p
net.ipv4.ip_forward = 1
net.ipv4.conf.all.rp_filter = 0

We startup the “ipvsadm.”
$ sudo touch /etc/sysconfig/ipvsadm
$ sudo systemctl start ipvsadm

In case of failback prohibition, you should disable automatic startup of “ipvsadm”.
$ :sudo systemctl enable ipvsadm
$ sudo systemctl status ipvsadm

Next, we change the kernel parameters on the Load Balancer 2.
$ sudo vim /etc/sysctl.conf

# Enable Packet Transfer between Interfaces
net.ipv4.ip_forward = 1

# Do not discard packets from networks that do not belong to the interface.
net.ipv4.conf.all.rp_filter = 0

We reflect the setting of the kernel parameters.
$ sudo sysctl -p
net.ipv4.ip_forward = 1
net.ipv4.conf.all.rp_filter = 0

We startup the “ipvsadm.”
$ sudo touch /etc/sysconfig/ipvsadm
$ sudo systemctl start ipvsadm

In case of failback prohibition, you should disable automatic startup of “ipvsadm”.
$ :sudo systemctl enable ipvsadm
$ sudo systemctl status ipvsadm

We will use the “ipvsadm” command to check the LVS communication settings on the Load Balancer 1.
$ sudo ipvsadm -Ln

We will use the “ipvsadm” command to check the LVS communication settings on the Load Balancer 2.
$ sudo ipvsadm -Ln


6. Conclusion
In this way, we can improve performance degradation against high load, which is a weak point of software load balancer, with the DSR technology.