Persist DNS configuration for nodes for openstack provider

* Firstly, provision a Heat stack with given public resolvers.
* After the DNS node configured as an authoritative server,
  switch the Heat stack's Neutron subnet to that resolver
  (private_dns_server) the way it to become the first entry pushed
  into the  hosts /etc/resolv.conf. It will be serving the cluster
  domain requests for OpenShift nodes and workloads.
* Drop post-provision /etc/reslov.conf nameserver hacks as not
  needed anymore.
* Fix dns floating IPs output and add the priv IPs output as well.
* Update docs, clarify localhost vs servers requirements, add
  required Network Manager setup step.
* Use post-provision task names instead of comments.

Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>

7 files changed, 108 insertions, 73 deletions

diff --git a/playbooks/provisioning/openstack/README.md b/playbooks/provisioning/openstack/README.md
index 57b72c7f3..972ef705d 100644
--- a/playbooks/provisioning/openstack/README.md
+++ b/playbooks/provisioning/openstack/README.md
@@ -5,13 +5,19 @@ OpenStack resources (servers, networking, volumes, security groups,
 etc.). The result is an environment ready for openshift-ansible.
 
 
-## Dependencies
+## Dependencies for localhost (ansible control/admin node)
 
 * [Ansible 2.3](https://pypi.python.org/pypi/ansible)
 * [jinja2](http://jinja.pocoo.org/docs/2.9/)
 * [shade](https://pypi.python.org/pypi/shade)
-* python-dns
+* python-dns / [dnspython](https://pypi.python.org/pypi/dnspython)
+* Become (sudo) is not required.
 
+## Dependencies for OpenStack hosted cluster nodes (servers)
+
+There are no additional dependencies for the cluster nodes. Required
+configuration steps are done by Heat given a specific user data config
+that normally should not be changed.
 
 ## What does it do
 
@@ -42,12 +48,27 @@ etc.). The result is an environment ready for openshift-ansible.
 Pay special attention to the values in the first paragraph -- these
 will depend on your OpenStack environment.
 
-The `env_id` and `openstack_dns_domain` will form the DNS domain all
+The `env_id` and `public_dns_domain` will form the cluster's DNS domain all
 your servers will be under. With the default values, this will be
-`openshift.example.com`.
-
-`openstack_nameservers` is a list of DNS servers accessible from all
-the created Nova servers. These will be serve as your DNS forwarders.
+`openshift.example.com`. For workloads, the default subdomain is 'apps'.
+That sudomain can be set as well by the `openshift_app_domain` variable in
+the inventory.
+
+The `public_dns_nameservers` is a list of DNS servers accessible from all
+the created Nova servers. These will be serving as your DNS forwarders for
+external FQDNs that do not belong to the cluster's DNS domain and its subdomains.
+
+The `openshift_use_dnsmasq` controls either dnsmasq is deployed or not.
+By default, dnsmasq is deployed and comes as the hosts' /etc/resolv.conf file
+first nameserver entry that points to the local host instance of the dnsmasq
+daemon that in turn proxies DNS requests to the authoritative DNS server.
+When Network Manager is enabled for provisioned cluster nodes, which is
+normally the case, you should not change the defaults and always deploy dnsmasq.
+
+Note that the authoritative DNS server is configured on post provsision
+steps, and the Neutron subnet for the Heat stack is updated to point to that
+server in the end. So the provisioned servers will start using it natively
+as a default nameserver that comes from the NetworkManager and cloud-init.
 
 `openstack_ssh_key` is a Nova keypair -- you can see your keypairs with
 `openstack keypair list`.
@@ -136,8 +157,8 @@ Once it succeeds, you can install openshift by running:
     ansible-playbook --become --user openshift --private-key ~/.ssh/openshift -i inventory/ openshift-ansible/playbooks/byo/openshift-node/network_manager.yml
     ansible-playbook --become --user openshift --private-key ~/.ssh/openshift -i inventory/ openshift-ansible/playbooks/byo/config.yml
 
-Note, the `network_manager.yml` is only required if you're deploying OpenShift
-origin.
+Note, the `network_manager.yml` step is mandatory and is required for persisting
+the hosts' DNS configs.
 
 ## License
 
diff --git a/playbooks/provisioning/openstack/post-provision-openstack.yml b/playbooks/provisioning/openstack/post-provision-openstack.yml
index 918f9e065..412ccd221 100644
--- a/playbooks/provisioning/openstack/post-provision-openstack.yml
+++ b/playbooks/provisioning/openstack/post-provision-openstack.yml
@@ -1,6 +1,6 @@
 ---
-# Assign hostnames
-- hosts: cluster_hosts
+- name: Assign hostnames
+  hosts: cluster_hosts
   gather_facts: False
   become: true
   pre_tasks:
@@ -8,8 +8,8 @@
   roles:
   - role: hostnames
 
-# Subscribe DNS Host to allow for configuration below
-- hosts: dns
+- name: Subscribe DNS Host to allow for configuration below
+  hosts: dns
   gather_facts: False
   become: true
   roles:
@@ -17,15 +17,15 @@
     when: hostvars.localhost.rhsm_register
     tags: 'subscription-manager'
 
-# Determine which DNS server(s) to use for our generated records
-- hosts: localhost
+- name: Determine which DNS server(s) to use for our generated records
+  hosts: localhost
   gather_facts: False
   become: False
   roles:
   - dns-server-detect
 
-# Build the DNS Server Views and Configure DNS Server(s)
-- hosts: dns
+- name: Build the DNS Server Views and Configure DNS Server(s)
+  hosts: dns
   gather_facts: False
   become: true
   pre_tasks:
@@ -35,8 +35,8 @@
   roles:
   - role: dns-server
 
-# Build and process DNS Records
-- hosts: localhost
+- name: Build and process DNS Records
+  hosts: localhost
   gather_facts: False
   become: False
   pre_tasks:
@@ -46,18 +46,22 @@
   roles:
   - role: dns
 
-# OpenShift Pre-Requisites
-- hosts: OSEv3
+- name: Switch the stack subnet to the configured private DNS server
+  hosts: localhost
+  gather_facts: False
+  become: False
+  vars_files:
+  - stack_params.yaml
+  tasks:
+  - include_role:
+      name: openstack-stack
+      tasks_from: subnet_update_dns_servers
+
+- name: OpenShift Pre-Requisites
+  hosts: OSEv3
   gather_facts: False
   become: true
   tasks:
-  - name: "Edit /etc/resolv.conf on masters/nodes"
-    lineinfile:
-      state: present
-      dest: /etc/resolv.conf
-      regexp: "nameserver {{ hostvars['localhost'].private_dns_server }}"
-      line: "nameserver {{ hostvars['localhost'].private_dns_server }}"
-      insertafter: search*
   - name: "Include DNS configuration to ensure proper name resolution"
     lineinfile:
       state: present
diff --git a/playbooks/provisioning/openstack/provision-openstack.yml b/playbooks/provisioning/openstack/provision-openstack.yml
index 5d521432b..0c673af2f 100644
--- a/playbooks/provisioning/openstack/provision-openstack.yml
+++ b/playbooks/provisioning/openstack/provision-openstack.yml
@@ -2,33 +2,12 @@
 - hosts: localhost
   gather_facts: True
   become: False
+  vars_files:
+    - stack_params.yaml
   pre_tasks:
-  - include: pre_tasks.yml
+    - include: pre_tasks.yml
   roles:
-  - role: openstack-stack
-    stack_name: "{{ env_id }}.{{ public_dns_domain }}"
-    dns_domain: "{{ public_dns_domain }}"
-    dns_nameservers: "{{ public_dns_nameservers }}"
-    subnet_prefix: "{{ openstack_subnet_prefix }}"
-    ssh_public_key: "{{ openstack_ssh_public_key }}"
-    openstack_image: "{{ openstack_default_image_name }}"
-    lb_flavor: "{{ openstack_default_flavor | default('m1.small') }}"
-    etcd_flavor: "{{ openstack_default_flavor | default('m1.small') }}"
-    master_flavor: "{{ openstack_default_flavor | default('m1.medium') }}"
-    node_flavor: "{{ openstack_default_flavor | default('m1.medium') }}"
-    infra_flavor: "{{ openstack_default_flavor | default('m1.medium') }}"
-    dns_flavor: "{{ openstack_default_flavor | default('m1.small') }}"
-    external_network: "{{ openstack_external_network_name }}"
-    num_etcd: "{{ openstack_num_etcd | default(0) }}"
-    num_masters: "{{ openstack_num_masters }}"
-    num_nodes: "{{ openstack_num_nodes }}"
-    num_infra: "{{ openstack_num_infra }}"
-    num_dns: "{{ openstack_num_dns | default(1) }}"
-    nodes_to_remove: "{{ openstack_nodes_to_remove | default([]) |  to_yaml }}"
-    master_volume_size: "{{ docker_volume_size }}"
-    app_volume_size: "{{ docker_volume_size }}"
-    infra_volume_size: "{{ docker_volume_size }}"
-
+    - role: openstack-stack
 
 - name: Refresh Server inventory
   hosts: localhost
@@ -36,21 +15,21 @@
   become: False
   gather_facts: False
   tasks:
-  - meta: refresh_inventory
+    - meta: refresh_inventory
 
 - hosts: cluster_hosts
   name: Wait for the the nodes to come up
   become: False
   gather_facts: False
   tasks:
-  - wait_for_connection:
+    - wait_for_connection:
 
 - hosts: cluster_hosts
   gather_facts: True
   tasks:
-  - name: Debug hostvar
-    debug:
-      msg: "{{ hostvars[inventory_hostname] }}"
-      verbosity: 2
+    - name: Debug hostvar
+      debug:
+        msg: "{{ hostvars[inventory_hostname] }}"
+        verbosity: 2
 
 - include: post-provision-openstack.yml
diff --git a/playbooks/provisioning/openstack/stack_params.yaml b/playbooks/provisioning/openstack/stack_params.yaml
new file mode 100644
index 000000000..9c0b09b45
--- /dev/null
+++ b/playbooks/provisioning/openstack/stack_params.yaml
@@ -0,0 +1,23 @@
+---
+stack_name: "{{ env_id }}.{{ public_dns_domain }}"
+dns_domain: "{{ public_dns_domain }}"
+dns_nameservers: "{{ public_dns_nameservers }}"
+subnet_prefix: "{{ openstack_subnet_prefix }}"
+ssh_public_key: "{{ openstack_ssh_public_key }}"
+openstack_image: "{{ openstack_default_image_name }}"
+lb_flavor: "{{ openstack_default_flavor | default('m1.small') }}"
+etcd_flavor: "{{ openstack_default_flavor | default('m1.small') }}"
+master_flavor: "{{ openstack_default_flavor | default('m1.medium') }}"
+node_flavor: "{{ openstack_default_flavor | default('m1.medium') }}"
+infra_flavor: "{{ openstack_default_flavor | default('m1.medium') }}"
+dns_flavor: "{{ openstack_default_flavor | default('m1.small') }}"
+external_network: "{{ openstack_external_network_name }}"
+num_etcd: "{{ openstack_num_etcd | default(0) }}"
+num_masters: "{{ openstack_num_masters }}"
+num_nodes: "{{ openstack_num_nodes }}"
+num_infra: "{{ openstack_num_infra }}"
+num_dns: "{{ openstack_num_dns | default(1) }}"
+master_volume_size: "{{ docker_volume_size }}"
+app_volume_size: "{{ docker_volume_size }}"
+infra_volume_size: "{{ docker_volume_size }}"
+nodes_to_remove: "{{ openstack_nodes_to_remove | default([]) |  to_yaml }}"
diff --git a/roles/openstack-stack/tasks/main.yml b/roles/openstack-stack/tasks/main.yml
index 71c7bbe0d..a53e6350b 100644
--- a/roles/openstack-stack/tasks/main.yml
+++ b/roles/openstack-stack/tasks/main.yml
@@ -35,6 +35,11 @@
     template: "{{ stack_template_path }}"
     wait: yes
 
+# NOTE(bogdando) OS::Neutron::Subnet doesn't support live updates for
+# dns_nameservers, so we can't do that for the "create stack" task.
+- include: subnet_update_dns_servers.yaml
+  when: private_dns_server is defined
+
 - name: cleanup temp files
   file:
     path: "{{ stack_template_pre.path }}"
diff --git a/roles/openstack-stack/tasks/subnet_update_dns_servers.yaml b/roles/openstack-stack/tasks/subnet_update_dns_servers.yaml
new file mode 100644
index 000000000..be4f07b97
--- /dev/null
+++ b/roles/openstack-stack/tasks/subnet_update_dns_servers.yaml
@@ -0,0 +1,8 @@
+---
+- name: Live update the subnet's DNS servers
+  os_subnet:
+    name: openshift-ansible-{{ stack_name }}-subnet
+    network_name: openshift-ansible-{{ stack_name }}-net
+    state: present
+    use_default_subnetpool: yes
+    dns_nameservers: "{{ [private_dns_server|default(public_dns_nameservers[0])]|union(public_dns_nameservers)|unique }}"
diff --git a/roles/openstack-stack/templates/heat_stack.yaml.j2 b/roles/openstack-stack/templates/heat_stack.yaml.j2
index 00a46896c..8bf76b57c 100644
--- a/roles/openstack-stack/templates/heat_stack.yaml.j2
+++ b/roles/openstack-stack/templates/heat_stack.yaml.j2
@@ -61,18 +61,13 @@ outputs:
         - dns
         - name
 
-  dns_floating_ip:
-    description: Floating IP of the DNS
-    value:
-      get_attr:
-        - dns
-        - addresses
-        - str_replace:
-            template: openshift-ansible-cluster_id-net
-            params:
-              cluster_id: {{ stack_name }}
-        - 1
-        - addr
+  dns_floating_ips:
+    description: Floating IPs of the DNS
+    value: { get_attr: [ dns, floating_ip ] }
+
+  dns_private_ips:
+    description: Private IPs of the DNS
+    value: { get_attr: [ dns, private_ip ] }
 
 resources:
 
@@ -111,9 +106,9 @@ resources:
               params:
                 subnet_24_prefix: {{ subnet_prefix }}
       dns_nameservers:
-      {% for nameserver in dns_nameservers %}
+{% for nameserver in dns_nameservers %}
         - {{ nameserver }}
-      {% endfor %}
+{% endfor %}
 
   router:
     type: OS::Neutron::Router