Managing HAProxy backend servers

Summary

This procedure shows how to exclude (or include) a HAProxy backend server. This may be used for maintenance purposes or in case of a misbehaving server that should be excluded for production.

HAProxy keeps an internal admin state and operating state for each backend server. The default admin state is ready, however, it may be changed for each backend server using the disabled server option.

HAProxy exposes an API through the stat sockets that allows administrators to monitor or, if enabled, operate HAProxy internal state. This procedure uses it.

HAproxy Operating tools

HAProxy monitoring should be done using the HAProxy stats interface through a milkcheck procedure.

milkcheck haproxy <status|enable|disable>

This milkcheck procedure uses the HAProxy stats interface to monitor and manage the server states. Internally it uses show server states and set server ... state ... commands.

For enable and disable actions, you must provide a server variable.

For status action, you can provide a server variable to only show states for this particular server

This procedure manages all instances of HAProxy (puppet, ldap or http).

Actions

disable

Sets the admin state of the server provided in the server variable to MAINT. In READY state, no traffic will be forwarded to this server.

enable

Sets the admin state of the server provided in the server variable to READY. In READY state, traffic will be forwarded to this server if it is successfully checked.

HAProxy Procedure

To disable a server: * Get the HAserver name of the server, this should be the DNS name of the service IP, ie. puppet-i0-svc1 or ldap-svc2. It is referenced as <server> in the following examples.

• First, make sure that the cluster is in it’s intended state by checking it:

  # milkcheck haproxy status
  haproxy.http      [  OK  ]
  haproxy.ldap      [  OK  ]
  haproxy.puppet    [  OK  ]
  haproxy           [  OK  ]
  
  SUMMARY - 3 actions (0 failed)
  + Success on all services
  lb[1-2]
  

• Second, disable the server:

  # milkcheck haproxy -D server=<server> disable
  haproxy.http      [    OK   ]
  haproxy.ldap      [    OK   ]
  haproxy.puppet    [    OK   ]
  haproxy           [    OK   ]
  
  SUMMARY - 3 actions (0 failed)
  + Success on all services
  lb[1-2]
  

• And finally, check the cluster state once more:

  # milkcheck haproxy status
  haproxy.http      [    OK   ]
  haproxy.ldap      [    OK   ]
  status haproxy.puppet ran in 0.97s
  > lb[1-2]: # Format
  > lb[1-2]: # backend_name/frontend_name: operational_state/administrative_state
  > lb[1-2]:
  > lb[1-2]: all_puppet_masters/puppet-i0-svc1: STOPPED/MAINT
  > lb[1-2]: i0_puppet_masters/puppet-i0-svc1: STOPPED/MAINT
  > lb[1-2]: i10_puppet_masters/puppet-i0-svc1: STOPPED/MAINT
  > lb[1-2]: i20_puppet_masters/puppet-i0-svc1: STOPPED/MAINT
  > lb[1-2]: i24_puppet_masters/puppet-i0-svc1: STOPPED/MAINT
  > lb[1-2]: i28_puppet_masters/puppet-i0-svc1: STOPPED/MAINT
  > lb[1-2]: exited with 2
  haproxy.puppet    [  ERROR  ]
  haproxy           [DEP_ERROR]
  
  
  SUMMARY - 3 actions (1 failed)
  + haproxy.puppet.status
      Target: lb[1-2]
      Command:
        [...]