Managing HAProxy backend servers 
--------------------------------

Summary
^^^^^^^

This procedure shows how to exclude (or include) a HAProxy backend server. This may be used for maintenance purposes or in case of a misbehaving server that should be excluded for production.

HAProxy keeps an internal *admin state* and *operating state* for each backend server. The default *admin state* is *ready*, however, it may be changed for each backend server using the ``disabled`` server option.

HAProxy exposes an API through the **stat** sockets that allows administrators to monitor or, if enabled, operate HAProxy internal state. This procedure uses it.
 
HAproxy Operating tools
^^^^^^^^^^^^^^^^^^^^^^^

HAProxy monitoring should be done using the HAProxy **stats** interface through a **milkcheck** procedure.

``milkcheck haproxy <status|enable|disable>``
  This milkcheck procedure uses the HAProxy **stats** interface to monitor and manage the server states.
  Internally it uses ``show server states`` and ``set server ... state ...`` commands.
  
  For ``enable`` and ``disable`` actions, you must provide a ``server`` variable.

  For ``status`` action, you can provide a ``server`` variable to only show states for this particular server

  This procedure manages all instances of HAProxy (puppet, ldap or http).

  .. rubric:: Actions

  ``disable``
    Sets the admin state of the server provided in the ``server`` variable to ``MAINT``.
    In ``READY`` state, no traffic will be forwarded to this server.

  ``enable``
    Sets the admin state of the server provided in the ``server`` variable to ``READY``.
    In ``READY`` state, traffic will be forwarded to this server if it is successfully checked.


HAProxy Procedure
^^^^^^^^^^^^^^^^^

To disable a server:
* Get the HAserver name of the server, this should be the DNS name of the service IP, ie. ``puppet-i0-svc1`` or ``ldap-svc2``. It is referenced as ``<server>`` in the following examples.

* First, make sure that the cluster is in it's intended state by checking it:

  .. code-block:: console 
		
     # milkcheck haproxy status
     haproxy.http      [  OK  ]
     haproxy.ldap      [  OK  ]
     haproxy.puppet    [  OK  ]
     haproxy           [  OK  ]
     
     SUMMARY - 3 actions (0 failed)
     + Success on all services
     lb[1-2]

* Second, disable the server:

  .. code-block:: console 
		
     # milkcheck haproxy -D server=<server> disable
     haproxy.http      [    OK   ]
     haproxy.ldap      [    OK   ]
     haproxy.puppet    [    OK   ]
     haproxy           [    OK   ]
     
     SUMMARY - 3 actions (0 failed)
     + Success on all services
     lb[1-2]

* And finally, check the cluster state once more:

  .. code-block:: console 
		
     # milkcheck haproxy status
     haproxy.http      [    OK   ]
     haproxy.ldap      [    OK   ]
     status haproxy.puppet ran in 0.97s
     > lb[1-2]: # Format
     > lb[1-2]: # backend_name/frontend_name: operational_state/administrative_state
     > lb[1-2]:
     > lb[1-2]: all_puppet_masters/puppet-i0-svc1: STOPPED/MAINT
     > lb[1-2]: i0_puppet_masters/puppet-i0-svc1: STOPPED/MAINT
     > lb[1-2]: i10_puppet_masters/puppet-i0-svc1: STOPPED/MAINT
     > lb[1-2]: i20_puppet_masters/puppet-i0-svc1: STOPPED/MAINT
     > lb[1-2]: i24_puppet_masters/puppet-i0-svc1: STOPPED/MAINT
     > lb[1-2]: i28_puppet_masters/puppet-i0-svc1: STOPPED/MAINT
     > lb[1-2]: exited with 2
     haproxy.puppet    [  ERROR  ]
     haproxy           [DEP_ERROR]
     
     
     SUMMARY - 3 actions (1 failed)
     + haproxy.puppet.status
         Target: lb[1-2]
	 Command:
	   [...]