General overview
=================
 - etcd services (worth checking both ports)
    etcdctl3 --endpoints="192.168.213.1:2379" member list       - doesn't check health only reports members
    oc get cs                                                   - only etcd (other services will fail on Openshift)
 - All nodes and pods are fine and running and all pvc are bound
    oc get nodes
    oc get pods --all-namespaces -o wide                        - Check also that no pods stuck in Terminating/Pending status for a long time
    oc get pvc --all-namespaces -o wide
 - API health check
    curl -k https://apiserver.kube-service-catalog.svc/healthz
 - Docker status (at each node)
    docker info
    * Enough Data and Metadata Space is available 
    * The number of resident images is in check (>500-1000 - bad, >2000-3000 - terrible)

Nodes
=====
 - All systemd services are running
    * Communication with docker daemon is actually working
 - Replicas of mandatory pods (GlusterFS, Router) are running on all nodes

Storage
=======
 - Heketi status 
    heketi-cli -s  http://heketi-storage.glusterfs.svc.cluster.local:8080 --user admin --secret "$(oc get secret heketi-storage-admin-secret -n glusterfs -o jsonpath='{.data.key}' | base64 -d)" topology info
 - Status of Gluster Volume (and its bricks which with heketi fails often)
    gluster volume info
    ./gluster.sh info all_heketi
 - Check available storage space on system partition and  LVM volumes (docker, heketi, ands)
    Run 'df -h' and 'lvdisplay' on each node
 - Check status of hardware raids
    /opt/MegaRAID/storcli/storcli64 /c0/v0 show all

Networking
==========
 - Check that correct upstream name servers are listed for both DNSMasq (host) and SkyDNS (pods).
 If not fix and restart 'origin-node' and 'dnsmasq' (it happens that DNSMasq is just stuck).
    * '/etc/dnsmasq.d/origin-upstream-dns.conf'
    * '/etc/origin/node/resolv.conf'

 - Check that both internal and external addresses are resolvable from all hosts.
    * I.e. we should be able to resolve 'google.com'
    * And we should be able to resolve 'heketi-storage.glusterfs.svc.cluster.local'
    
 - Check that keepalived service is up and the corresponding ip's are really assigned to one
 of the nodes (vagrant provisioner would remove keepalived tracked ips, but keepalived will
 continue running without noticing it)
 
 - Ensure, we don't have override of cluster_name to first master (which we do during the
 provisioning of OpenShift plays)

 - Sometimes OpenShift fails to clean-up after terminated pod properly (this problem is particularly
 triggered on the systems with huge number of resident docker images). This causes rogue network 
 interfaces to  remain in OpenVSwitch fabric. This can be determined by errors like:
    could not open network device vethb9de241f (No such device)
 reported by 'ovs-vsctl show' or present in the log '/var/log/openvswitch/ovs-vswitchd.log' 
 which may quickly grow over 100MB quickly. If number of rogue interfaces grows too much,
 the pod scheduling gets even worse (compared to delays caused only be docker images) and 
 will start time-out on the affected node. 
  * The work-around is to delete rogue interfaces with 
    ovs-vsctl del-port br0 <iface>
 This does not solve the problem, however. The new interfaces will get abandoned by OpenShift.

ADEI
====
 - MySQL replication is working
 - No caching pods or maintenance pods are hung (for whatever reason)
    * Check no ADEI pods stuck in Deleting/Pending status
    * Check logs of 'cacher' and 'maintenace' scripts and ensure none is stuck on ages old time-stamp (unless we re-caching something huge)
    * Ensure were is no old pending scripts in '/adei/tmp/adminscripts'
 Possible reasons:
    * Stale 'flock' locks (could be found out by analyzing backtraces in correspond /proc/<pid>/stack)
    * Hunged connections to MySQL (could be found out by executing 'SHOW PROCESSLIST' on MySQL servers)