Ansible – HAProxy
In this previous post I setup KeepAliveD using a fictitious tenant using Ansible. In this post I will be building upon that same configuration and creating the HAProxy setup.
Below is the vars/tenant_1.yml file that contains the specific tenant variables that will be used.
---
tenant_name: tenant_1
config_forward_rules_allow_spec: 'false' # set to true to configure specific firewall rules under forward_rules_allow_spec
tenant_subnets:
- { tenant_subnet: '192.168.70.0/24' } # Web
- { tenant_subnet: '192.168.71.0/24' } # App
- { tenant_subnet: '192.168.72.0/24' } # DB
tenant_vips:
- 10.10.10.100
- 10.10.10.101
- 10.10.10.102
- 10.10.10.103
- 10.10.10.104
###### Firewall Setup ######
###### Note.....Firewall rules are by default dropped in default setup ######
# Port specific rules - used to add firewall rules that should be port based rules **Note...SSH is allowed by default so there is no need to specify rules for SSH.
# Another note is that when configuring load balancer setup further down...The ports that will be load balanced will be added to the firewall by default as well.
# This section will normally be used to allow a specific port to a host|subnet etc. which does not require load balancing.
# Make sure to change config_forward_rules_allow_spec at the top to true to ensure the below firewall rules are applied.
forward_rules_allow_spec:
- { protocol: 'tcp', port: '8080', source: '0.0.0.0/0', destination: '192.168.70.0/24' }
# Generic non port specific rules to allow. Ex. allowing a host|subnet to communicate from|to one another on every port available.
forward_rules_allow_gen:
- { source: '192.168.70.0/24', destination: '0.0.0.0/0' }
- { source: '192.168.71.0/24', destination: '0.0.0.0/0' }
- { source: '192.168.72.0/24', destination: '0.0.0.0/0' }
# Define specific rules below to be dropped by firewall. Ex. explicitly denying communications between hosts|subnets.
forward_rules_out_drop:
- { source: '192.168.70.0/24', destination: '192.168.71.0/24' }
- { source: '192.168.70.0/24', destination: '192.168.72.0/24' }
- { source: '192.168.71.0/24', destination: '192.168.70.0/24' }
- { source: '192.168.71.0/24', destination: '192.168.72.0/24' }
- { source: '192.168.72.0/24', destination: '192.168.70.0/24' }
- { source: '192.168.72.0/24', destination: '192.168.71.0/24' }
###### Load Balancer Setup ######
# Allocate vips from tenant_vips assigned above....these will be used as variables for lb_details and lb_defs. You may choose to just enter IP addresses though.
web_vip: '10.10.10.100'
db_vip: '10.10.10.101'
app_vip: '10.10.10.102'
balance_method: 'roundrobin' # Set to one of the below types to configure load balancing method
#leastconn - The server with the lowest number of connections receives the connection
#roundrobin - Each server is used in turns, according to their weights.
#source - Source IP hashed and divided by total weight of servers designates which server will receive the request
# Defines the load balancing group setup
lb_details:
- { name: 'web', protocol: 'tcp', listen_port: '80', tenant_vip: '{{ web_vip }}', balance_type: '{{ balance_method }}' }
- { name: 'db', protocol: 'tcp', listen_port: '3306', tenant_vip: '{{ db_vip }}', balance_type: '{{ balance_method }}' }
- { name: 'rabbitmq-mgmt', protocol: 'tcp', listen_port: '15672', tenant_vip: '{{ app_vip }}', balance_type: '{{ balance_method }}' }
- { name: 'redis', protocol: 'tcp', listen_port: '6379', tenant_vip: '{{ app_vip }}', balance_type: '{{ balance_method }}' }
- { name: 'rabbitmq', protocol: 'tcp', listen_port: '5672', tenant_vip: '{{ app_vip }}', balance_type: '{{ balance_method }}' }
# Defines the load balancing servers within the load balancing group
lb_defs:
- { lb_def_name: 'web', protocol: 'tcp', listen_port: '80', tenant_vip: '{{ web_vip }}', lb_group: 'web', server: 'ans-cloud-web01', backend_port: '80' }
- { lb_def_name: 'web', protocol: 'tcp', listen_port: '80', tenant_vip: '{{ web_vip }}', lb_group: 'web', server: 'ans-cloud-web02', backend_port: '80' }
- { lb_def_name: 'web', protocol: 'tcp', listen_port: '80', tenant_vip: '{{ web_vip }}', lb_group: 'web', server: 'ans-cloud-web03', backend_port: '80' }
- { lb_def_name: 'db', protocol: 'tcp', listen_port: '3306', tenant_vip: '{{ db_vip }}', lb_group: 'db', server: 'ans-cloud-db01', backend_port: '3306' }
- { lb_def_name: 'db', protocol: 'tcp', listen_port: '3306', tenant_vip: '{{ db_vip }}', lb_group: 'db', server: 'ans-cloud-db02', backend_port: '3306' }
- { lb_def_name: 'rabbitmq-mgmt', protocol: 'tcp', listen_port: '15672', tenant_vip: '{{ app_vip }}', lb_group: 'rabbitmq-mgmt', server: 'ans-cloud-app01', backend_port: '15672' }
- { lb_def_name: 'rabbitmq-mgmt', protocol: 'tcp', listen_port: '15672', tenant_vip: '{{ app_vip }}', lb_group: 'rabbitmq-mgmt', server: 'ans-cloud-app02', backend_port: '15672' }
- { lb_def_name: 'rabbitmq-mgmt', protocol: 'tcp', listen_port: '15672', tenant_vip: '{{ app_vip }}', lb_group: 'rabbitmq-mgmt', server: 'ans-cloud-app03', backend_port: '15672' }
- { lb_def_name: 'redis', protocol: 'tcp', listen_port: '6379', tenant_vip: '{{ app_vip }}', lb_group: 'redis', server: 'ans-cloud-app01', backend_port: '6379' }
- { lb_def_name: 'redis', protocol: 'tcp', listen_port: '6379', tenant_vip: '{{ app_vip }}', lb_group: 'redis', server: 'ans-cloud-app02', backend_port: '6379' }
- { lb_def_name: 'redis', protocol: 'tcp', listen_port: '6379', tenant_vip: '{{ app_vip }}', lb_group: 'redis', server: 'ans-cloud-app03', backend_port: '6379' }
- { lb_def_name: 'rabbitmq', protocol: 'tcp', listen_port: '5672', tenant_vip: '{{ app_vip }}', lb_group: 'rabbitmq', server: 'ans-cloud-app01', backend_port: '5672' }
- { lb_def_name: 'rabbitmq', protocol: 'tcp', listen_port: '5672', tenant_vip: '{{ app_vip }}', lb_group: 'rabbitmq', server: 'ans-cloud-app02', backend_port: '5672' }
- { lb_def_name: 'rabbitmq', protocol: 'tcp', listen_port: '5672', tenant_vip: '{{ app_vip }}', lb_group: 'rabbitmq', server: 'ans-cloud-app03', backend_port: '5672' }
Below is the haproxy.cfg.j2 template that I will use.
# {{ ansible_managed }}
global
# log logstash local0 #Change logstash to your naming
log /dev/log local0
log /dev/log local1 notice
# log-send-hostname
chroot /var/lib/haproxy
user haproxy
group haproxy
daemon
maxconn 40000
spread-checks 3
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL).
ssl-default-bind-ciphers kEECDH+aRSA+AES:kRSA+AES:+AES256:RC4-SHA:!kEDH:!LOW:!EXP:!MD5:!aNULL:!eNULL
ssl-default-bind-options no-sslv3
defaults
log global
mode tcp
maxconn 40000
option httplog
option dontlognull
option redispatch
option tcp-smart-accept
option tcp-smart-connect
retries 3
timeout queue 5000
timeout connect 50000
timeout client 50000
timeout server 50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
userlist STATSUSERS
group admin users admin
user admin insecure-password admin
listen admin_page 0.0.0.0:9090
mode http
stats enable
stats refresh 60s
stats uri /
acl AuthOkay_ReadOnly http_auth(STATSUSERS)
acl AuthOkay_Admin http_auth_group(STATSUSERS) admin
stats http-request auth realm admin_page unless AuthOkay_ReadOnly
{% for lb_group_def in lb_details %}
listen {{ tenant_name }}_{{ lb_group_def.name }}-{{ lb_group_def.tenant_vip }}:{{ lb_group_def.listen_port }} {{ lb_group_def.tenant_vip }}:{{ lb_group_def.listen_port }}
mode tcp
option tcpka
option tcplog
#balance leastconn - The server with the lowest number of connections receives the connection
#balance roundrobin - Each server is used in turns, according to their weights.
#balance source - Source IP hashed and divided by total weight of servers designates which server will receive the request
balance {{ lb_group_def.balance_type }}
default-server inter 2s downinter 5s rise 3 fall 2 slowstart 60s maxconn 256 maxqueue 128 weight 100
{% for item in lb_defs %}
{% if item.lb_group == lb_group_def.name %}
server {{ item.server }} {{ item.server }}:{{ item.backend_port }} check
{% endif %}
{% endfor %}
{% endfor %}
And below is what we end up with as an haproxy.cfg for our setup by using the variables and template above.
# Ansible managed: /home/administrator/ansible/projects/ans-cloud-rt01/templates/etc/haproxy/haproxy.cfg.j2 modified on 2015-04-10 12:58:14 by administrator on ansible
global
# log logstash local0 #Change logstash to your naming
log /dev/log local0
log /dev/log local1 notice
# log-send-hostname
chroot /var/lib/haproxy
user haproxy
group haproxy
daemon
maxconn 40000
spread-checks 3
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL).
ssl-default-bind-ciphers kEECDH+aRSA+AES:kRSA+AES:+AES256:RC4-SHA:!kEDH:!LOW:!EXP:!MD5:!aNULL:!eNULL
ssl-default-bind-options no-sslv3
defaults
log global
mode tcp
maxconn 40000
option httplog
option dontlognull
option redispatch
option tcp-smart-accept
option tcp-smart-connect
retries 3
timeout queue 5000
timeout connect 50000
timeout client 50000
timeout server 50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
userlist STATSUSERS
group admin users admin
user admin insecure-password admin
listen admin_page 0.0.0.0:9090
mode http
stats enable
stats refresh 60s
stats uri /
acl AuthOkay_ReadOnly http_auth(STATSUSERS)
acl AuthOkay_Admin http_auth_group(STATSUSERS) admin
stats http-request auth realm admin_page unless AuthOkay_ReadOnly
listen tenant_1_web-10.10.10.100:80 10.10.10.100:80
mode tcp
option tcpka
option tcplog
#balance leastconn - The server with the lowest number of connections receives the connection
#balance roundrobin - Each server is used in turns, according to their weights.
#balance source - Source IP hashed and divided by total weight of servers designates which server will receive the request
balance roundrobin
default-server inter 2s downinter 5s rise 3 fall 2 slowstart 60s maxconn 256 maxqueue 128 weight 100
server ans-cloud-web01 ans-cloud-web01:80 check
server ans-cloud-web02 ans-cloud-web02:80 check
server ans-cloud-web03 ans-cloud-web03:80 check
listen tenant_1_db-10.10.10.101:3306 10.10.10.101:3306
mode tcp
option tcpka
option tcplog
#balance leastconn - The server with the lowest number of connections receives the connection
#balance roundrobin - Each server is used in turns, according to their weights.
#balance source - Source IP hashed and divided by total weight of servers designates which server will receive the request
balance roundrobin
default-server inter 2s downinter 5s rise 3 fall 2 slowstart 60s maxconn 256 maxqueue 128 weight 100
server ans-cloud-db01 ans-cloud-db01:3306 check
server ans-cloud-db02 ans-cloud-db02:3306 check
listen tenant_1_rabbitmq-mgmt-10.10.10.102:15672 10.10.10.102:15672
mode tcp
option tcpka
option tcplog
#balance leastconn - The server with the lowest number of connections receives the connection
#balance roundrobin - Each server is used in turns, according to their weights.
#balance source - Source IP hashed and divided by total weight of servers designates which server will receive the request
balance roundrobin
default-server inter 2s downinter 5s rise 3 fall 2 slowstart 60s maxconn 256 maxqueue 128 weight 100
server ans-cloud-app01 ans-cloud-app01:15672 check
server ans-cloud-app02 ans-cloud-app02:15672 check
server ans-cloud-app03 ans-cloud-app03:15672 check
listen tenant_1_redis-10.10.10.102:6379 10.10.10.102:6379
mode tcp
option tcpka
option tcplog
#balance leastconn - The server with the lowest number of connections receives the connection
#balance roundrobin - Each server is used in turns, according to their weights.
#balance source - Source IP hashed and divided by total weight of servers designates which server will receive the request
balance roundrobin
default-server inter 2s downinter 5s rise 3 fall 2 slowstart 60s maxconn 256 maxqueue 128 weight 100
server ans-cloud-app01 ans-cloud-app01:6379 check
server ans-cloud-app02 ans-cloud-app02:6379 check
server ans-cloud-app03 ans-cloud-app03:6379 check
listen tenant_1_rabbitmq-10.10.10.102:5672 10.10.10.102:5672
mode tcp
option tcpka
option tcplog
#balance leastconn - The server with the lowest number of connections receives the connection
#balance roundrobin - Each server is used in turns, according to their weights.
#balance source - Source IP hashed and divided by total weight of servers designates which server will receive the request
balance roundrobin
default-server inter 2s downinter 5s rise 3 fall 2 slowstart 60s maxconn 256 maxqueue 128 weight 100
server ans-cloud-app01 ans-cloud-app01:5672 check
server ans-cloud-app02 ans-cloud-app02:5672 check
server ans-cloud-app03 ans-cloud-app03:5672 check
Below is the group_vars code used across the different Ansible posts based on tenant setup
