diff options
Diffstat (limited to 'roles/openshift_logging')
-rw-r--r-- | roles/openshift_logging/README.md | 18 | ||||
-rw-r--r-- | roles/openshift_logging/defaults/main.yml | 36 | ||||
-rw-r--r-- | roles/openshift_logging/filter_plugins/openshift_logging.py | 19 | ||||
-rw-r--r-- | roles/openshift_logging/filter_plugins/test | 34 | ||||
-rw-r--r-- | roles/openshift_logging/library/openshift_logging_facts.py | 2 | ||||
-rw-r--r-- | roles/openshift_logging/tasks/install_logging.yaml | 9 |
6 files changed, 93 insertions, 25 deletions
diff --git a/roles/openshift_logging/README.md b/roles/openshift_logging/README.md index 69eb9283d..280d7d24c 100644 --- a/roles/openshift_logging/README.md +++ b/roles/openshift_logging/README.md @@ -44,23 +44,23 @@ When `openshift_logging_install_logging` is set to `False` the `openshift_loggin - `openshift_logging_curator_run_timezone`: The timezone that Curator uses for figuring out its run time. Defaults to 'UTC'. - `openshift_logging_curator_script_log_level`: The script log level for Curator. Defaults to 'INFO'. - `openshift_logging_curator_log_level`: The log level for the Curator process. Defaults to 'ERROR'. -- `openshift_logging_curator_cpu_limit`: The amount of CPU to allocate to Curator. Default is '100m'. +- `openshift_logging_curator_cpu_request`: The minimum amount of CPU to allocate to Curator. Default is '100m'. - `openshift_logging_curator_memory_limit`: The amount of memory to allocate to Curator. Unset if not specified. - `openshift_logging_curator_nodeselector`: A map of labels (e.g. {"node":"infra","region":"west"} to select the nodes where the curator pod will land. - `openshift_logging_image_pull_secret`: The name of an existing pull secret to link to the logging service accounts - `openshift_logging_kibana_hostname`: The Kibana hostname. Defaults to 'kibana.example.com'. -- `openshift_logging_kibana_cpu_limit`: The amount of CPU to allocate to Kibana or unset if not specified. +- `openshift_logging_kibana_cpu_request`: The minimum amount of CPU to allocate to Kibana or unset if not specified. - `openshift_logging_kibana_memory_limit`: The amount of memory to allocate to Kibana or unset if not specified. - `openshift_logging_kibana_proxy_debug`: When "True", set the Kibana Proxy log level to DEBUG. Defaults to 'false'. -- `openshift_logging_kibana_proxy_cpu_limit`: The amount of CPU to allocate to Kibana proxy or unset if not specified. +- `openshift_logging_kibana_proxy_cpu_request`: The minimum amount of CPU to allocate to Kibana proxy or unset if not specified. - `openshift_logging_kibana_proxy_memory_limit`: The amount of memory to allocate to Kibana proxy or unset if not specified. - `openshift_logging_kibana_replica_count`: The number of replicas Kibana should be scaled up to. Defaults to 1. - `openshift_logging_kibana_nodeselector`: A map of labels (e.g. {"node":"infra","region":"west"} to select the nodes where the pod will land. - `openshift_logging_kibana_edge_term_policy`: Insecure Edge Termination Policy. Defaults to Redirect. - `openshift_logging_fluentd_nodeselector`: The node selector that the Fluentd daemonset uses to determine where to deploy to. Defaults to '"logging-infra-fluentd": "true"'. -- `openshift_logging_fluentd_cpu_limit`: The CPU limit for Fluentd pods. Defaults to '100m'. +- `openshift_logging_fluentd_cpu_request`: The minimum amount of CPU to allocate for Fluentd collector pods. Defaults to '100m'. - `openshift_logging_fluentd_memory_limit`: The memory limit for Fluentd pods. Defaults to '512Mi'. - `openshift_logging_fluentd_use_journal`: *DEPRECATED - DO NOT USE* Fluentd will automatically detect whether or not Docker is using the journald log driver. - `openshift_logging_fluentd_journal_read_from_head`: If empty, Fluentd will use its internal default, which is false. @@ -80,7 +80,7 @@ When `openshift_logging_install_logging` is set to `False` the `openshift_loggin - `openshift_logging_es_client_key`: The location of the client key Fluentd uses for openshift_logging_es_host. Defaults to '/etc/fluent/keys/key'. - `openshift_logging_es_cluster_size`: The number of ES cluster members. Defaults to '1'. -- `openshift_logging_es_cpu_limit`: The amount of CPU limit for the ES cluster. Unused if not set +- `openshift_logging_es_cpu_request`: The minimum amount of CPU to allocate for an ES pod cluster member. Defaults to 1 CPU. - `openshift_logging_es_memory_limit`: The amount of RAM that should be assigned to ES. Defaults to '8Gi'. - `openshift_logging_es_log_appenders`: The list of rootLogger appenders for ES logs which can be: 'file', 'console'. Defaults to 'file'. - `openshift_logging_es_pv_selector`: A key/value map added to a PVC in order to select specific PVs. Defaults to 'None'. @@ -107,7 +107,7 @@ same as above for their non-ops counterparts, but apply to the OPS cluster insta - `openshift_logging_es_ops_client_cert`: /etc/fluent/keys/cert - `openshift_logging_es_ops_client_key`: /etc/fluent/keys/key - `openshift_logging_es_ops_cluster_size`: 1 -- `openshift_logging_es_ops_cpu_limit`: The amount of CPU limit for the ES cluster. Unused if not set +- `openshift_logging_es_ops_cpu_request`: The minimum amount of CPU to allocate for an ES ops pod cluster member. Defaults to 1 CPU. - `openshift_logging_es_ops_memory_limit`: 8Gi - `openshift_logging_es_ops_pvc_dynamic`: False - `openshift_logging_es_ops_pvc_size`: "" @@ -115,9 +115,9 @@ same as above for their non-ops counterparts, but apply to the OPS cluster insta - `openshift_logging_es_ops_recover_after_time`: 5m - `openshift_logging_es_ops_storage_group`: 65534 - `openshift_logging_kibana_ops_hostname`: The Operations Kibana hostname. Defaults to 'kibana-ops.example.com'. -- `openshift_logging_kibana_ops_cpu_limit`: The amount of CPU to allocate to Kibana or unset if not specified. +- `openshift_logging_kibana_ops_cpu_request`: The minimum amount of CPU to allocate to Kibana or unset if not specified. - `openshift_logging_kibana_ops_memory_limit`: The amount of memory to allocate to Kibana or unset if not specified. -- `openshift_logging_kibana_ops_proxy_cpu_limit`: The amount of CPU to allocate to Kibana proxy or unset if not specified. +- `openshift_logging_kibana_ops_proxy_cpu_request`: The minimum amount of CPU to allocate to Kibana proxy or unset if not specified. - `openshift_logging_kibana_ops_proxy_memory_limit`: The amount of memory to allocate to Kibana proxy or unset if not specified. - `openshift_logging_kibana_ops_replica_count`: The number of replicas Kibana ops should be scaled up to. Defaults to 1. @@ -176,7 +176,7 @@ Elasticsearch OPS too, if using an OPS cluster: clients will use to connect to mux, and will be used in the TLS server cert subject. - `openshift_logging_mux_port`: 24284 -- `openshift_logging_mux_cpu_limit`: 100m +- `openshift_logging_mux_cpu_request`: 100m - `openshift_logging_mux_memory_limit`: 512Mi - `openshift_logging_mux_default_namespaces`: Default `["mux-undefined"]` - the first value in the list is the namespace to use for undefined projects, diff --git a/roles/openshift_logging/defaults/main.yml b/roles/openshift_logging/defaults/main.yml index 6e7e2557f..626732d16 100644 --- a/roles/openshift_logging/defaults/main.yml +++ b/roles/openshift_logging/defaults/main.yml @@ -18,20 +18,24 @@ openshift_logging_curator_run_minute: 0 openshift_logging_curator_run_timezone: UTC openshift_logging_curator_script_log_level: INFO openshift_logging_curator_log_level: ERROR -openshift_logging_curator_cpu_limit: 100m -openshift_logging_curator_memory_limit: null +openshift_logging_curator_cpu_limit: null +openshift_logging_curator_memory_limit: 256Mi +openshift_logging_curator_cpu_request: 100m openshift_logging_curator_nodeselector: {} -openshift_logging_curator_ops_cpu_limit: 100m -openshift_logging_curator_ops_memory_limit: null +openshift_logging_curator_ops_cpu_limit: null +openshift_logging_curator_ops_memory_limit: 256Mi +openshift_logging_curator_ops_cpu_request: 100m openshift_logging_curator_ops_nodeselector: {} openshift_logging_kibana_hostname: "{{ 'kibana.' ~ (openshift_master_default_subdomain | default('router.default.svc.cluster.local', true)) }}" openshift_logging_kibana_cpu_limit: null openshift_logging_kibana_memory_limit: 736Mi +openshift_logging_kibana_cpu_request: 100m openshift_logging_kibana_proxy_debug: false openshift_logging_kibana_proxy_cpu_limit: null openshift_logging_kibana_proxy_memory_limit: 256Mi +openshift_logging_kibana_proxy_cpu_request: 100m openshift_logging_kibana_replica_count: 1 openshift_logging_kibana_edge_term_policy: Redirect @@ -53,9 +57,11 @@ openshift_logging_kibana_ca: "" openshift_logging_kibana_ops_hostname: "{{ 'kibana-ops.' ~ (openshift_master_default_subdomain | default('router.default.svc.cluster.local', true)) }}" openshift_logging_kibana_ops_cpu_limit: null openshift_logging_kibana_ops_memory_limit: 736Mi +openshift_logging_kibana_ops_cpu_request: 100m openshift_logging_kibana_ops_proxy_debug: false openshift_logging_kibana_ops_proxy_cpu_limit: null openshift_logging_kibana_ops_proxy_memory_limit: 256Mi +openshift_logging_kibana_ops_proxy_cpu_request: 100m openshift_logging_kibana_ops_replica_count: 1 #The absolute path on the control node to the cert file to use @@ -71,13 +77,14 @@ openshift_logging_kibana_ops_key: "" openshift_logging_kibana_ops_ca: "" openshift_logging_fluentd_nodeselector: {'logging-infra-fluentd': 'true'} -openshift_logging_fluentd_cpu_limit: 100m +openshift_logging_fluentd_cpu_limit: null openshift_logging_fluentd_memory_limit: 512Mi +openshift_logging_fluentd_cpu_request: 100m openshift_logging_fluentd_journal_source: "" openshift_logging_fluentd_journal_read_from_head: "" openshift_logging_fluentd_hosts: ['--all'] -openshift_logging_fluentd_buffer_queue_limit: 1024 -openshift_logging_fluentd_buffer_size_limit: 1m +openshift_logging_fluentd_buffer_queue_limit: 32 +openshift_logging_fluentd_buffer_size_limit: 8m openshift_logging_es_host: logging-es openshift_logging_es_port: 9200 @@ -85,7 +92,8 @@ openshift_logging_es_ca: /etc/fluent/keys/ca openshift_logging_es_client_cert: /etc/fluent/keys/cert openshift_logging_es_client_key: /etc/fluent/keys/key openshift_logging_es_cluster_size: 1 -openshift_logging_es_cpu_limit: 1000m +openshift_logging_es_cpu_limit: null +openshift_logging_es_cpu_request: "1" # the logging appenders for the root loggers to write ES logs. Valid values: 'file', 'console' openshift_logging_es_log_appenders: ['file'] openshift_logging_es_memory_limit: "8Gi" @@ -98,8 +106,6 @@ openshift_logging_es_storage_group: "65534" openshift_logging_es_nodeselector: {} # openshift_logging_es_config is a hash to be merged into the defaults for the elasticsearch.yaml openshift_logging_es_config: {} -openshift_logging_es_number_of_shards: 1 -openshift_logging_es_number_of_replicas: 0 # for exposing es to external (outside of the cluster) clients openshift_logging_es_allow_external: False @@ -126,8 +132,9 @@ openshift_logging_es_ops_ca: /etc/fluent/keys/ca openshift_logging_es_ops_client_cert: /etc/fluent/keys/cert openshift_logging_es_ops_client_key: /etc/fluent/keys/key openshift_logging_es_ops_cluster_size: "{{ openshift_logging_elasticsearch_ops_cluster_size | default(1) }}" -openshift_logging_es_ops_cpu_limit: 1000m -openshift_logging_es_ops_memory_limit: "8Gi" +openshift_logging_es_ops_cpu_limit: null +openshift_logging_es_ops_memory_limit: 8Gi +openshift_logging_es_ops_cpu_request: "1" openshift_logging_es_ops_pv_selector: "{{ openshift_loggingops_storage_labels | default('') }}" openshift_logging_es_ops_pvc_dynamic: "{{ openshift_logging_elasticsearch_ops_pvc_dynamic | default(False) }}" openshift_logging_es_ops_pvc_size: "{{ openshift_logging_elasticsearch_ops_pvc_size | default('') }}" @@ -160,8 +167,9 @@ openshift_logging_mux_allow_external: False openshift_logging_use_mux: "{{ openshift_logging_mux_allow_external | default(False) }}" openshift_logging_mux_hostname: "{{ 'mux.' ~ (openshift_master_default_subdomain | default('router.default.svc.cluster.local', true)) }}" openshift_logging_mux_port: 24284 -openshift_logging_mux_cpu_limit: 500m -openshift_logging_mux_memory_limit: 1Gi +openshift_logging_mux_cpu_limit: null +openshift_logging_mux_memory_limit: 512Mi +openshift_logging_mux_cpu_request: 100m # the namespace to use for undefined projects should come first, followed by any # additional namespaces to create by default - users will typically not need to set this openshift_logging_mux_default_namespaces: ["mux-undefined"] diff --git a/roles/openshift_logging/filter_plugins/openshift_logging.py b/roles/openshift_logging/filter_plugins/openshift_logging.py index 330e7e59a..959573635 100644 --- a/roles/openshift_logging/filter_plugins/openshift_logging.py +++ b/roles/openshift_logging/filter_plugins/openshift_logging.py @@ -17,6 +17,22 @@ def es_storage(os_logging_facts, dc_name, pvc_claim, root='elasticsearch'): return dict(kind='emptydir') +def walk(source, path, default, delimiter='.'): + '''Walk the sourch hash given the path and return the value or default if not found''' + if not isinstance(source, dict): + raise RuntimeError('The source is not a walkable dict: {} path: {}'.format(source, path)) + keys = path.split(delimiter) + max_depth = len(keys) + cur_depth = 0 + while cur_depth < max_depth: + if keys[cur_depth] in source: + source = source[keys[cur_depth]] + cur_depth = cur_depth + 1 + else: + return default + return source + + def random_word(source_alpha, length): ''' Returns a random word given the source of characters to pick from and resulting length ''' return ''.join(random.choice(source_alpha) for i in range(length)) @@ -73,5 +89,6 @@ class FilterModule(object): 'map_from_pairs': map_from_pairs, 'es_storage': es_storage, 'serviceaccount_name': serviceaccount_name, - 'serviceaccount_namespace': serviceaccount_namespace + 'serviceaccount_namespace': serviceaccount_namespace, + 'walk': walk } diff --git a/roles/openshift_logging/filter_plugins/test b/roles/openshift_logging/filter_plugins/test new file mode 100644 index 000000000..3ad956cca --- /dev/null +++ b/roles/openshift_logging/filter_plugins/test @@ -0,0 +1,34 @@ +import unittest +from openshift_logging import walk + +class TestFilterMethods(unittest.TestCase): + + + def test_walk_find_key(self): + source = {'foo': {'bar.xyz': 'myvalue'}} + self.assertEquals(walk(source,'foo#bar.xyz', 123, delimiter='#'), 'myvalue') + + + def test_walk_return_default(self): + source = {'foo': {'bar.xyz': 'myvalue'}} + self.assertEquals(walk(source,'foo#bar.abc', 123, delimiter='#'), 123) + + + def test_walk_limit_max_depth(self): + source = {'foo': {'bar.xyz': 'myvalue'}} + self.assertEquals(walk(source,'foo#bar.abc#dontfindme', 123, delimiter='#'), 123) + + def test_complex_hash(self): + source = { + 'elasticsearch': { + 'configmaps': { + 'logging-elasticsearch': { + 'elasticsearch.yml': "a string value" + } + } + } + } + self.assertEquals(walk(source,'elasticsearch#configmaps#logging-elasticsearch#elasticsearch.yml', 123, delimiter='#'), "a string value") + +if __name__ == '__main__': + unittest.main() diff --git a/roles/openshift_logging/library/openshift_logging_facts.py b/roles/openshift_logging/library/openshift_logging_facts.py index f10df8da5..98d0d1c4f 100644 --- a/roles/openshift_logging/library/openshift_logging_facts.py +++ b/roles/openshift_logging/library/openshift_logging_facts.py @@ -207,7 +207,7 @@ class OpenshiftLoggingFacts(OCBaseCommand): def facts_for_configmaps(self, namespace): ''' Gathers facts for configmaps in logging namespace ''' self.default_keys_for("configmaps") - a_list = self.oc_command("get", "configmaps", namespace=namespace, add_options=["-l", LOGGING_SELECTOR]) + a_list = self.oc_command("get", "configmaps", namespace=namespace) if len(a_list["items"]) == 0: return for item in a_list["items"]: diff --git a/roles/openshift_logging/tasks/install_logging.yaml b/roles/openshift_logging/tasks/install_logging.yaml index 21fd79c28..76627acf2 100644 --- a/roles/openshift_logging/tasks/install_logging.yaml +++ b/roles/openshift_logging/tasks/install_logging.yaml @@ -78,6 +78,7 @@ openshift_logging_elasticsearch_nodeselector: "{{ openshift_logging_es_nodeselector if item.0.nodeSelector | default(None) is none else item.0.nodeSelector }}" openshift_logging_elasticsearch_storage_group: "{{ [openshift_logging_es_storage_group] if item.0.storageGroups | default([]) | length == 0 else item.0.storageGroups }}" _es_containers: "{{item.0.containers}}" + _es_configmap: "{{ openshift_logging_facts | walk('elasticsearch#configmaps#logging-elasticsearch#elasticsearch.yml', '{}', delimiter='#') | from_yaml }}" with_together: - "{{ openshift_logging_facts.elasticsearch.deploymentconfigs.values() }}" @@ -133,6 +134,7 @@ openshift_logging_elasticsearch_pvc_pv_selector: "{{ openshift_logging_es_ops_pv_selector }}" openshift_logging_elasticsearch_memory_limit: "{{ openshift_logging_es_ops_memory_limit }}" openshift_logging_elasticsearch_cpu_limit: "{{ openshift_logging_es_ops_cpu_limit }}" + openshift_logging_elasticsearch_cpu_request: "{{ openshift_logging_es_ops_cpu_request }}" openshift_logging_elasticsearch_nodeselector: "{{ openshift_logging_es_ops_nodeselector if item.0.nodeSelector | default(None) is none else item.0.nodeSelector }}" openshift_logging_elasticsearch_storage_group: "{{ [openshift_logging_es_ops_storage_group] if item.0.storageGroups | default([]) | length == 0 else item.0.storageGroups }}" openshift_logging_es_key: "{{ openshift_logging_es_ops_key }}" @@ -141,7 +143,10 @@ openshift_logging_es_hostname: "{{ openshift_logging_es_ops_hostname }}" openshift_logging_es_edge_term_policy: "{{ openshift_logging_es_ops_edge_term_policy | default('') }}" openshift_logging_es_allow_external: "{{ openshift_logging_es_ops_allow_external }}" + openshift_logging_es_number_of_shards: "{{ openshift_logging_es_ops_number_of_shards | default(None) }}" + openshift_logging_es_number_of_replicas: "{{ openshift_logging_es_ops_number_of_replicas | default(None) }}" _es_containers: "{{item.0.containers}}" + _es_configmap: "{{ openshift_logging_facts | walk('elasticsearch_ops#configmaps#logging-elasticsearch-ops#elasticsearch.yml', '{}', delimiter='#') | from_yaml }}" with_together: - "{{ openshift_logging_facts.elasticsearch_ops.deploymentconfigs.values() }}" @@ -167,6 +172,7 @@ openshift_logging_elasticsearch_pvc_pv_selector: "{{ openshift_logging_es_ops_pv_selector }}" openshift_logging_elasticsearch_memory_limit: "{{ openshift_logging_es_ops_memory_limit }}" openshift_logging_elasticsearch_cpu_limit: "{{ openshift_logging_es_ops_cpu_limit }}" + openshift_logging_elasticsearch_cpu_request: "{{ openshift_logging_es_ops_cpu_request }}" openshift_logging_elasticsearch_nodeselector: "{{ openshift_logging_es_ops_nodeselector }}" openshift_logging_es_key: "{{ openshift_logging_es_ops_key }}" openshift_logging_es_cert: "{{ openshift_logging_es_ops_cert }}" @@ -207,11 +213,13 @@ openshift_logging_kibana_es_port: "{{ openshift_logging_es_ops_port }}" openshift_logging_kibana_nodeselector: "{{ openshift_logging_kibana_ops_nodeselector }}" openshift_logging_kibana_cpu_limit: "{{ openshift_logging_kibana_ops_cpu_limit }}" + openshift_logging_kibana_cpu_request: "{{ openshift_logging_kibana_ops_cpu_request }}" openshift_logging_kibana_memory_limit: "{{ openshift_logging_kibana_ops_memory_limit }}" openshift_logging_kibana_hostname: "{{ openshift_logging_kibana_ops_hostname }}" openshift_logging_kibana_replicas: "{{ openshift_logging_kibana_ops_replica_count }}" openshift_logging_kibana_proxy_debug: "{{ openshift_logging_kibana_ops_proxy_debug }}" openshift_logging_kibana_proxy_cpu_limit: "{{ openshift_logging_kibana_ops_proxy_cpu_limit }}" + openshift_logging_kibana_proxy_cpu_request: "{{ openshift_logging_kibana_ops_proxy_cpu_request }}" openshift_logging_kibana_proxy_memory_limit: "{{ openshift_logging_kibana_ops_proxy_memory_limit }}" openshift_logging_kibana_cert: "{{ openshift_logging_kibana_ops_cert }}" openshift_logging_kibana_key: "{{ openshift_logging_kibana_ops_key }}" @@ -243,6 +251,7 @@ openshift_logging_curator_master_url: "{{ openshift_logging_master_url }}" openshift_logging_curator_image_pull_secret: "{{ openshift_logging_image_pull_secret }}" openshift_logging_curator_cpu_limit: "{{ openshift_logging_curator_ops_cpu_limit }}" + openshift_logging_curator_cpu_request: "{{ openshift_logging_curator_ops_cpu_request }}" openshift_logging_curator_memory_limit: "{{ openshift_logging_curator_ops_memory_limit }}" openshift_logging_curator_nodeselector: "{{ openshift_logging_curator_ops_nodeselector }}" when: |