OpenShift Platform ------------------ The OpenShift web frontend is running at https://kaas.kit.edu:8443 However, I find it simpler to use command line tool 'oc' which - On RedHat platforms the package is called 'origin-clients' and is installed from OpenShift repository available as package 'centos-release-openshift-origin'. - For other distribut check here (we are running version 3.7) https://docs.openshift.org/latest/cli_reference/get_started_cli.html#installing-the-cli Basically, it is also a good documentation to start using it. https://docs.openshift.com/container-platform/3.7/dev_guide/index.html Infrastructure -------------- - We have 3 servers running with names ipekatrin[1-3].ipe.kit.edu. This is internal names. The external access is provided using 2 virtual ping-poing ip's katrin[1-2].ipe.kit.edu. By default they are assigned to both master servers of the cluster, but will migrate both to a single surviving server if one of the masters die. This is enabled by keepalived daemon and ensures load-balancing and high-availability. The domain name 'kaas.kit.edu' is resolved to both ips in round-robin fashion. - By default, the executed service have have names in the form '.kaas.kit.edu'. For instance, you can test adei-katrin.kaas.kit.edu - This is a ADEI service running on the new platform adas-autogen.kaas.kit.edu - Sample ADEI with generated data katrin.kaas.kit.edu - Is the placehorder for futre katrin router etc. - OpenVPN connection with KATRIN virtual network is running on master servers. Non-masters route the traffic trough the masters using keepalived IP. So, katrin network should be transparently visible from any pod in the cluster. Users ----- I have configured a few user accounts using ADEI and UFO passwords. Furthermore, to avoid a mess of conteiners, I have created a number of projects with appropriate administrators. kaas (csa, kopmann) - This is a routing service (basically Apache mod_rewrite) to set redirects from http://katrin.kit.edu/* katrin (katrin) - Katrin database adei (csa) - All ADEI setups bora (ntj) - BORA web (kopmann) - Various web sites, etc. mon (csa) - Monitoring test (*) - Project for testing If needed, I can create more projects/users. Just let me know. Storage ------- I have created a couple of gluster volumes for different purpose: katrin_data: - For katrin data files datastore - Other non-katrin large data files openshift - 3 times replicated volume for configuration, sources, and other important small files temporary - Logs, temporary files, etc. Again, to not mess data from the different projects, on each volume there are subfolders for all projects. Furthermore, I have tried to add a bit of protection and assigned each project a range of group ids. The subfolders can only be read by appropriate group. I also pre-created correpsonding PersistentVolume (pv) and PersistentVolumeClaims (pvc): 'katrin', 'data', ... There is a special pvc called 'host'. This is to save data on the local raid array bypassing gluster (i.e. on each OpenShift node the content of the folder will be different). WARNING: Gluster supports dynamic provisioning using Heketi. It is installed and worked. However, heketi is far from being of production quality. I think it is OK to use it for some temporary data if you want, but I would suggest to use pre-created volumes for important data. - Curently, I don't plan to provide access to the servers itself. The storage should be managed from the OpenShift pods solely. I made a sample 'manager' pod equipped with scp, lftp, curl, etc. It mounts all default storage. You need to start it and, then, you also can connect interactively either using both web interace and console app. oc -n katrin scale dc/kaas-manager --replicas 1 oc -n katrin rsh dc/kaas-manager Just an example, build your own configuration with required set of packages. Databases --------- Gluster works fine if you mostly read data or if you perform mostly sequential writes. It plays very bad with 'databases' and similar loads. I guess it should not be issue for Katrin database as it is relatively small (AFAIK) and do not perform many writes. For something, like ADEI the gluster is not viable option to back MySQL server. There are several options to handle volumes for appliations performing a large amount of small random writes: - If High Availability (HA) is not important, just pin a pod to a certain node and use 'host' pvc. - For databases, either Master/Slave replication can be enabled (you will still need to pin node and use 'host' pvc). The Galera cluster can be installed for multi-master replication. It is configured using StatefulSet feature of OpenShift. I have not tested recovery throughly, but it is working, quite performant, and masters are synchronized without problems. - For non-database applications, the Gluster block storage may be used. The block storage is not shared between multiple pods, but private to a specific pod. So, it is possible to avoid certain amount of locking and context switches. So, performance is significantly beter. I was even able to run ADEI database on top of such device. Though it is still singificnatly slower than native host performance. There is again heketi-based provisioner, but it works even worse when one providing standard Gluster volumes. So, I suggest to ask me to create block devices manually if necessary. Otherall, if you have data intensive workload, we can discuss the best approach.