FailedConsole Output

Skipping 92 KB.. Full Log
119096
    Duration: 35.323 ms
     Changes:   
----------
          ID: salt_update_certificates
    Function: cmd.wait
        Name: update-ca-certificates
      Result: True
     Comment: 
     Started: 13:09:13.156204
    Duration: 0.874 ms
     Changes:   
----------
          ID: salt_minion_service_restart
    Function: cmd.run
        Name: while true; do salt-call saltutil.running|grep fun: && continue; salt-call --local service.restart salt-minion; break; done
      Result: True
     Comment: Command "while true; do salt-call saltutil.running|grep fun: && continue; salt-call --local service.restart salt-minion; break; done" run
     Started: 13:09:13.164755
    Duration: 46.307 ms
     Changes:   
              ----------
              pid:
                  2590
              retcode:
                  None
              stderr:
              stdout:
                  None

Summary for local
-------------
Succeeded: 66 (changed=54)
Failed:     0
-------------
Total states run:     66
Total run time:   13.662 s
[INFO    ] Could not determine init system from command line: (/bin/tini -- /entrypoint.sh)
[INFO    ] Could not determine init system from command line: (/bin/tini -- /entrypoint.sh)
[INFO    ] Executing command ['service', 'salt-minion', 'restart'] in directory '/home/ubuntu'
local:
    True
[wait_for] OK: salt-call state.sls reclass,linux.network,salt.minion               exclude='[{id: reclass_packages}, {id: /etc/reclass/reclass-config.yml}]'

[wait_for] Waiting for cmd to return success: salt -C 'E@^(mas01).*' saltutil.refresh_pillar
[wait_for] Attempt 1/3 for: salt -C 'E@^(mas01).*' saltutil.refresh_pillar
mas01.mcp-odl-noha.local:
    True
[wait_for] OK: salt -C 'E@^(mas01).*' saltutil.refresh_pillar

[wait_for] Waiting for cmd to return success: (for n in mas01; do salt -C ${n}.* test.ping || exit; done)
[wait_for] Attempt 1/3 for: (for n in mas01; do salt -C ${n}.* test.ping || exit; done)
mas01.mcp-odl-noha.local:
    True
[wait_for] OK: (for n in mas01; do salt -C ${n}.* test.ping || exit; done)

[wait_for] Waiting for cmd to return success: salt -C 'E@^(mas01).*' saltutil.sync_all
[wait_for] Attempt 1/3 for: salt -C 'E@^(mas01).*' saltutil.sync_all
mas01.mcp-odl-noha.local:
    ----------
    beacons:
    clouds:
    engines:
        - engines.architect
        - engines.saltgraph
    grains:
        - grains.ceilometer_policy
        - grains.ceph
        - grains.cinder_policy
        - grains.docker_swarm
        - grains.glance_policy
        - grains.heat_policy
        - grains.jenkins_plugins
        - grains.keystone_policy
        - grains.kubernetes
        - grains.neutron_policy
        - grains.nova_policy
        - grains.ssh_fingerprints
    log_handlers:
    modules:
        - modules._modules.gnocchiv1.__init__
        - modules._modules.gnocchiv1.archive_policy
        - modules._modules.gnocchiv1.common
        - modules.architect
        - modules.artifactory
        - modules.auditd
        - modules.avinetworks
        - modules.barbicanv1.__init__
        - modules.barbicanv1.acl
        - modules.barbicanv1.common
        - modules.barbicanv1.secrets
        - modules.ceph_ng
        - modules.cfgdrive
        - modules.cinderng
        - modules.cinderv3.__init__
        - modules.cinderv3.arg_converter
        - modules.cinderv3.common
        - modules.cinderv3.lists
        - modules.cinderv3.services
        - modules.cinderv3.volume
        - modules.cinderv3.volume_actions
        - modules.cinderv3.volume_types
        - modules.cinderv3.volumes
        - modules.configdrive
        - modules.contrail
        - modules.contrail_health
        - modules.creds
        - modules.devops_utils
        - modules.dockerng_service
        - modules.encode_json
        - modules.gerrit
        - modules.gitlab
        - modules.glanceng
        - modules.glancev2.__init__
        - modules.glancev2.common
        - modules.glancev2.image
        - modules.glancev2.task
        - modules.gnocchiv1.__init__
        - modules.gnocchiv1.archive_policy
        - modules.gnocchiv1.common
        - modules.gnocchiv1.gnocchiv1.__init__
        - modules.gnocchiv1.gnocchiv1.archive_policy
        - modules.gnocchiv1.gnocchiv1.common
        - modules.heat
        - modules.heatv1.__init__
        - modules.heatv1.common
        - modules.heatv1.services
        - modules.heatv1.stack
        - modules.heka_alarming
        - modules.helm
        - modules.iptables_extra
        - modules.ironicng
        - modules.ironicv1.__init__
        - modules.ironicv1.chassis
        - modules.ironicv1.common
        - modules.ironicv1.drivers
        - modules.ironicv1.nodes
        - modules.ironicv1.ports
        - modules.ironicv1.volumes
        - modules.jenkins_common
        - modules.keystone_policy
        - modules.keystoneng
        - modules.keystonev3.__init__
        - modules.keystonev3.arg_converter
        - modules.keystonev3.common
        - modules.keystonev3.domains
        - modules.keystonev3.endpoints
        - modules.keystonev3.groups
        - modules.keystonev3.lists
        - modules.keystonev3.projects
        - modules.keystonev3.regions
        - modules.keystonev3.roles
        - modules.keystonev3.services
        - modules.keystonev3.users
        - modules.linux_hosts
        - modules.linux_netlink
        - modules.maas
        - modules.maas_client
        - modules.maasng
        - modules.manilang.__init__
        - modules.manilang.common
        - modules.manilang.share_types
        - modules.manilang.shares
        - modules.modelschema
        - modules.modelutils
        - modules.multipart
        - modules.nagios_alarming
        - modules.net_checks
        - modules.netutils
        - modules.neutronng
        - modules.neutronv2.__init__
        - modules.neutronv2.agents
        - modules.neutronv2.arg_converter
        - modules.neutronv2.auto_alloc
        - modules.neutronv2.common
        - modules.neutronv2.lists
        - modules.neutronv2.networks
        - modules.neutronv2.ports
        - modules.neutronv2.rbac
        - modules.neutronv2.routers
        - modules.neutronv2.subnetpools
        - modules.neutronv2.subnets
        - modules.novang
        - modules.novav21.__init__
        - modules.novav21.aggregates
        - modules.novav21.common
        - modules.novav21.flavors
        - modules.novav21.keypairs
        - modules.novav21.quotas
        - modules.novav21.servers
        - modules.novav21.services
        - modules.octaviav2.__init__
        - modules.octaviav2.common
        - modules.octaviav2.loadbalancers
        - modules.ovs_config
        - modules.reclass
        - modules.rsyslog_util
        - modules.rundeck
        - modules.saltkey
        - modules.saltresource
        - modules.seedng
        - modules.testing.__init__
        - modules.testing.credentials
        - modules.testing.django
        - modules.testing.django_client_proxy
        - modules.utils
        - modules.virtng
    output:
    proxymodules:
    renderers:
    returners:
        - returners.postgres_graph_db
    sdb:
    states:
        - states._states.gnocchiv1
        - states.artifactory
        - states.avinetworks
        - states.barbicanv1
        - states.cinderng
        - states.cinderv3
        - states.contrail
        - states.contrail_health
        - states.debmirror
        - states.dockerng_service
        - states.gerrit
        - states.gitlab
        - states.glanceng
        - states.glancev2
        - states.gnocchiv1
        - states.grafana3_dashboard
        - states.grafana3_datasource
        - states.heat
        - states.heatv1
        - states.helm_release
        - states.helm_repos
        - states.httpng
        - states.ironicng
        - states.ironicv1
        - states.jenkins_approval
        - states.jenkins_artifactory
        - states.jenkins_credential
        - states.jenkins_gerrit
        - states.jenkins_globalenvprop
        - states.jenkins_jira
        - states.jenkins_job
        - states.jenkins_lib
        - states.jenkins_node
        - states.jenkins_plugin
        - states.jenkins_security
        - states.jenkins_slack
        - states.jenkins_smtp
        - states.jenkins_theme
        - states.jenkins_throttle_category
        - states.jenkins_user
        - states.jenkins_view
        - states.keystone_policy
        - states.keystoneng
        - states.keystonev3
        - states.kibana_object
        - states.maasng
        - states.manilang
        - states.neutronng
        - states.neutronv2
        - states.novang
        - states.novav21
        - states.octaviav2
        - states.ovs_config
        - states.powerdns_mysql
        - states.reclass
        - states.rundeck_project
        - states.rundeck_scm
        - states.rundeck_secret
    utils:
[wait_for] OK: salt -C 'E@^(mas01).*' saltutil.sync_all
[wait_for] OK: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i /var/lib/opnfv/mcp.rsa ubuntu@10.20.0.2 sudo             CI_DEBUG=false ERASE_ENV=0             /root/fuel/mcp/config/states/virtual_init
[STATE] Applying state: maas
(B
[wait_for] Waiting for cmd to return success: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i /var/lib/opnfv/mcp.rsa ubuntu@10.20.0.2 sudo             CI_DEBUG=false ERASE_ENV=0             /root/fuel/mcp/config/states/maas
[wait_for] Attempt 1/5 for: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i /var/lib/opnfv/mcp.rsa ubuntu@10.20.0.2 sudo             CI_DEBUG=false ERASE_ENV=0             /root/fuel/mcp/config/states/maas
Warning: Permanently added '10.20.0.2' (ECDSA) to the list of known hosts.

[wait_for] Waiting for cmd to return success: salt --out yaml -C 'mas01*' service.status maas-fixup | fgrep -q 'false'
[wait_for] Attempt 1/60 for: salt --out yaml -C 'mas01*' service.status maas-fixup | fgrep -q 'false'
No response
[wait_for] Attempt 2/60 for: salt --out yaml -C 'mas01*' service.status maas-fixup | fgrep -q 'false'
No response
[wait_for] Attempt 3/60 for: salt --out yaml -C 'mas01*' service.status maas-fixup | fgrep -q 'false'
No response
[wait_for] Attempt 4/60 for: salt --out yaml -C 'mas01*' service.status maas-fixup | fgrep -q 'false'
No response
[wait_for] Attempt 5/60 for: salt --out yaml -C 'mas01*' service.status maas-fixup | fgrep -q 'false'
No response
[wait_for] Attempt 6/60 for: salt --out yaml -C 'mas01*' service.status maas-fixup | fgrep -q 'false'
No response
[wait_for] Attempt 7/60 for: salt --out yaml -C 'mas01*' service.status maas-fixup | fgrep -q 'false'
No response
[wait_for] Attempt 8/60 for: salt --out yaml -C 'mas01*' service.status maas-fixup | fgrep -q 'false'
No response
[wait_for] Attempt 9/60 for: salt --out yaml -C 'mas01*' service.status maas-fixup | fgrep -q 'false'
No response
[wait_for] Attempt 10/60 for: salt --out yaml -C 'mas01*' service.status maas-fixup | fgrep -q 'false'
No response
[wait_for] Attempt 11/60 for: salt --out yaml -C 'mas01*' service.status maas-fixup | fgrep -q 'false'
[wait_for] OK: salt --out yaml -C 'mas01*' service.status maas-fixup | fgrep -q 'false'

[wait_for] Waiting for cmd to return success: salt -C 'mas01*' state.apply salt,iptables
[wait_for] Attempt 1/10 for: salt -C 'mas01*' state.apply salt,iptables
mas01.mcp-odl-noha.local:
----------
          ID: salt_minion_packages
    Function: pkg.installed
        Name: salt-minion
      Result: True
     Comment: All specified packages are already installed
     Started: 13:13:10.064427
    Duration: 348.809 ms
     Changes:   
----------
          ID: salt_minion_dependency_packages
    Function: pkg.installed
      Result: True
     Comment: All specified packages are already installed
     Started: 13:13:10.413572
    Duration: 28.816 ms
     Changes:   
----------
          ID: /etc/salt/minion.d/minion.conf
    Function: file.managed
      Result: True
     Comment: File /etc/salt/minion.d/minion.conf updated
     Started: 13:13:10.524148
    Duration: 829.576 ms
     Changes:   
              ----------
              diff:
                  New file
----------
          ID: salt_minion_systemd_override
    Function: file.managed
        Name: /etc/systemd/system/salt-minion.service.d/50-restarts.conf
      Result: True
     Comment: File /etc/systemd/system/salt-minion.service.d/50-restarts.conf updated
     Started: 13:13:11.370793
    Duration: 201.804 ms
     Changes:   
              ----------
              diff:
                  New file
              mode:
                  0644
----------
          ID: salt_minion_systemd_reload
    Function: module.wait
        Name: service.systemctl_reload
      Result: True
     Comment: 
     Started: 13:13:11.573163
    Duration: 0.565 ms
     Changes:   
----------
          ID: salt_minion_service
    Function: service.running
        Name: salt-minion
      Result: True
     Comment: The service salt-minion is already running
     Started: 13:13:11.574542
    Duration: 423.354 ms
     Changes:   
----------
          ID: salt_minion_sync_all
    Function: module.run
        Name: saltutil.sync_all
      Result: True
     Comment: State was not run because none of the onchanges reqs changed
     Started: 13:13:11.999670
    Duration: 0.059 ms
     Changes:   
----------
          ID: salt_minion_grains_dir
    Function: file.directory
        Name: /etc/salt/grains.d
      Result: True
     Comment: Directory /etc/salt/grains.d updated
     Started: 13:13:12.000106
    Duration: 2.705 ms
     Changes:   
              ----------
              /etc/salt/grains.d:
                  New Dir
----------
          ID: salt_minion_grains_files
    Function: file.managed
        Name: /etc/salt/grains
      Result: True
     Comment: Empty file
     Started: 13:13:12.003756
    Duration: 1.624 ms
     Changes:   
              ----------
              new:
                  file /etc/salt/grains created
----------
          ID: salt_minion_grains_files
    Function: file.managed
        Name: /etc/salt/grains.d/placeholder
      Result: True
     Comment: Empty file
     Started: 13:13:12.005910
    Duration: 1.369 ms
     Changes:   
              ----------
              new:
                  file /etc/salt/grains.d/placeholder created
----------
          ID: salt_minion_grain_linux_sphinx
    Function: file.managed
        Name: /etc/salt/grains.d/sphinx
      Result: True
     Comment: File /etc/salt/grains.d/sphinx updated
     Started: 13:13:12.007789
    Duration: 2.746 ms
     Changes:   
              ----------
              diff:
                  New file
----------
          ID: salt_minion_grain_linux_sphinx_validity_check
    Function: cmd.wait
        Name: python -c "import yaml; stream = file('/etc/salt/grains.d/sphinx', 'r'); yaml.load(stream); stream.close()"
      Result: True
     Comment: Command "python -c "import yaml; stream = file('/etc/salt/grains.d/sphinx', 'r'); yaml.load(stream); stream.close()"" run
     Started: 13:13:12.029513
    Duration: 484.437 ms
     Changes:   
              ----------
              pid:
                  3352
              retcode:
                  0
              stderr:
              stdout:
----------
          ID: salt_minion_grain_linux_dns_records
    Function: file.managed
        Name: /etc/salt/grains.d/dns_records
      Result: True
     Comment: File /etc/salt/grains.d/dns_records updated
     Started: 13:13:12.514926
    Duration: 8.391 ms
     Changes:   
              ----------
              diff:
                  New file
----------
          ID: salt_minion_grain_linux_dns_records_validity_check
    Function: cmd.wait
        Name: python -c "import yaml; stream = file('/etc/salt/grains.d/dns_records', 'r'); yaml.load(stream); stream.close()"
      Result: True
     Comment: Command "python -c "import yaml; stream = file('/etc/salt/grains.d/dns_records', 'r'); yaml.load(stream); stream.close()"" run
     Started: 13:13:12.525652
    Duration: 495.874 ms
     Changes:   
              ----------
              pid:
                  3355
              retcode:
                  0
              stderr:
              stdout:
----------
          ID: salt_minion_grain_salt_salt
    Function: file.managed
        Name: /etc/salt/grains.d/salt
      Result: True
     Comment: File /etc/salt/grains.d/salt updated
     Started: 13:13:13.022373
    Duration: 11.829 ms
     Changes:   
              ----------
              diff:
                  New file
----------
          ID: salt_minion_grain_salt_salt_validity_check
    Function: cmd.wait
        Name: python -c "import yaml; stream = file('/etc/salt/grains.d/salt', 'r'); yaml.load(stream); stream.close()"
      Result: True
     Comment: Command "python -c "import yaml; stream = file('/etc/salt/grains.d/salt', 'r'); yaml.load(stream); stream.close()"" run
     Started: 13:13:13.036343
    Duration: 288.091 ms
     Changes:   
              ----------
              pid:
                  3357
              retcode:
                  0
              stderr:
              stdout:
----------
          ID: salt_minion_grains_file
    Function: cmd.wait
        Name: cat /etc/salt/grains.d/* > /etc/salt/grains
      Result: True
     Comment: Command "cat /etc/salt/grains.d/* > /etc/salt/grains" run
     Started: 13:13:13.328846
    Duration: 33.29 ms
     Changes:   
              ----------
              pid:
                  3359
              retcode:
                  0
              stderr:
              stdout:
----------
          ID: salt_minion_grains_publish
    Function: module.wait
        Name: mine.update
      Result: True
     Comment: Module function mine.update executed
     Started: 13:13:13.364607
    Duration: 1.561 ms
     Changes:   
              ----------
              ret:
                  None
----------
          ID: salt_ca_certificates_packages
    Function: pkg.installed
        Name: ca-certificates
      Result: True
     Comment: All specified packages are already installed
     Started: 13:13:13.366362
    Duration: 13.817 ms
     Changes:   
----------
          ID: salt_update_certificates
    Function: cmd.wait
        Name: update-ca-certificates
      Result: True
     Comment: 
     Started: 13:13:13.381228
    Duration: 0.648 ms
     Changes:   
----------
          ID: iptables_packages_v4
    Function: pkg.installed
        Name: iptables
      Result: True
     Comment: All specified packages are already installed
     Started: 13:13:13.382060
    Duration: 16.37 ms
     Changes:   
----------
          ID: iptables_packages_v4
    Function: pkg.installed
        Name: iptables-persistent
      Result: True
     Comment: All specified packages are already installed
     Started: 13:13:13.398701
    Duration: 35.302 ms
     Changes:   
----------
          ID: iptables_modules_v4_load
    Function: kmod.present
      Result: True
     Comment: Loaded kernel modules iptable_filter, ip_tables
     Started: 13:13:13.438424
    Duration: 530.11 ms
     Changes:   
              ----------
              ip_tables:
                  loaded
              iptable_filter:
                  loaded
----------
          ID: /etc/iptables/rules.v4
    Function: file.managed
      Result: True
     Comment: File /etc/iptables/rules.v4 updated
     Started: 13:13:13.969431
    Duration: 574.147 ms
     Changes:   
              ----------
              diff:
                  New file
              mode:
                  0640
----------
          ID: iptables_services_v4_start
    Function: cmd.run
        Name: find /usr/share/netfilter-persistent/plugins.d/[0-9]*-ip4tables -exec {} start \;
      Result: True
     Comment: onlyif execution failed
     Started: 13:13:14.544981
    Duration: 164.426 ms
     Changes:   
----------
          ID: netfilter-persistent
    Function: service.running
      Result: True
     Comment: Service restarted
     Started: 13:13:14.867447
    Duration: 339.1 ms
     Changes:   
              ----------
              netfilter-persistent:
                  True
----------
          ID: iptables_tables_cleanup_v4
    Function: module.wait
        Name: iptables_extra.remove_stale_tables
      Result: True
     Comment: Module function iptables_extra.remove_stale_tables executed
     Started: 13:13:15.208717
    Duration: 43.782 ms
     Changes:   
              ----------
              ret:
                  None
----------
          ID: iptables_services_v6_stop
    Function: cmd.run
        Name: find /usr/share/netfilter-persistent/plugins.d/[0-9]*-ip6tables -exec {} flush \;
      Result: True
     Comment: onlyif execution failed
     Started: 13:13:15.252952
    Duration: 59.128 ms
     Changes:   
----------
          ID: /etc/iptables/rules.v6
    Function: file.absent
      Result: True
     Comment: File /etc/iptables/rules.v6 is not present
     Started: 13:13:15.318393
    Duration: 1.296 ms
     Changes:   
----------
          ID: iptables_tables_flush_v6
    Function: module.wait
        Name: iptables_extra.flush_all
      Result: True
     Comment: 
     Started: 13:13:15.320693
    Duration: 0.647 ms
     Changes:   
----------
          ID: salt_minion_service_restart
    Function: cmd.run
        Name: while true; do salt-call saltutil.running|grep fun: && continue; salt-call --local service.restart salt-minion; break; done
      Result: True
     Comment: Command "while true; do salt-call saltutil.running|grep fun: && continue; salt-call --local service.restart salt-minion; break; done" run
     Started: 13:13:15.322486
    Duration: 33.025 ms
     Changes:   
              ----------
              pid:
                  3393
              retcode:
                  None
              stderr:
              stdout:
                  None

Summary for mas01.mcp-odl-noha.local
-------------
Succeeded: 31 (changed=18)
Failed:     0
-------------
Total states run:     31
Total run time:    4.977 s
[wait_for] OK: salt -C 'mas01*' state.apply salt,iptables
mas01.mcp-odl-noha.local:
----------
          ID: maas_cluster_packages
    Function: pkg.installed
        Name: maas-rack-controller
      Result: True
     Comment: All specified packages are already installed
     Started: 13:13:30.877877
    Duration: 302.614 ms
     Changes:   
----------
          ID: maas_cluster_packages
    Function: pkg.installed
        Name: ipmitool
      Result: True
     Comment: All specified packages are already installed
     Started: 13:13:31.185258
    Duration: 13.269 ms
     Changes:   
----------
          ID: /etc/maas/rackd.conf
    Function: file.line
      Result: True
     Comment: Changes were made
     Started: 13:13:31.204413
    Duration: 13.673 ms
     Changes:   
              ----------
              diff:
                  --- 
                  +++ 
                  @@ -1,2 +1,2 @@
                   cluster_uuid: d5154d6f-523e-4e67-8d72-f8723f98e15d
                  -maas_url: http://localhost:5240/MAAS
                  +maas_url: http://10.1.0.3:5240/MAAS
----------
          ID: /etc/maas/rackd_conf
    Function: file.managed
        Name: /etc/maas/rackd.conf
      Result: True
     Comment: File /etc/maas/rackd.conf exists with proper permissions. No changes made.
     Started: 13:13:31.218349
    Duration: 1.971 ms
     Changes:   
----------
          ID: maas_cluster_services
    Function: service.running
        Name: maas-rackd
      Result: True
     Comment: Service restarted
     Started: 13:13:31.494020
    Duration: 1779.878 ms
     Changes:   
              ----------
              maas-rackd:
                  True

Summary for mas01.mcp-odl-noha.local
------------
Succeeded: 5 (changed=2)
Failed:    0
------------
Total states run:     5
Total run time:   2.111 s

[wait_for] Waiting for cmd to return success: salt -C 'mas01*' state.apply maas.region
[wait_for] Attempt 1/10 for: salt -C 'mas01*' state.apply maas.region
mas01.mcp-odl-noha.local:
----------
          ID: maas_region_packages
    Function: pkg.installed
        Name: maas-region-controller
      Result: True
     Comment: All specified packages are already installed
     Started: 13:13:39.770399
    Duration: 217.827 ms
     Changes:   
----------
          ID: maas_region_packages
    Function: pkg.installed
        Name: python-oauth
      Result: True
     Comment: All specified packages are already installed
     Started: 13:13:39.988707
    Duration: 21.348 ms
     Changes:   
----------
          ID: /etc/maas/regiond.conf
    Function: file.replace
      Result: True
     Comment: Changes were made
     Started: 13:13:40.017266
    Duration: 30.018 ms
     Changes:   
              ----------
              diff:
                  --- 
                  +++ 
                  @@ -3,4 +3,4 @@
                   database_pass: 33CtprBUvtGY
                   database_port: 5432
                   database_user: maas
                  -maas_url: http://10.20.0.3:5240/MAAS
                  +maas_url: http://10.1.0.3:5240/MAAS
----------
          ID: /usr/lib/python3/dist-packages/provisioningserver/templates/proxy/maas-proxy.conf.template
    Function: file.managed
      Result: True
     Comment: File /usr/lib/python3/dist-packages/provisioningserver/templates/proxy/maas-proxy.conf.template updated
     Started: 13:13:40.047905
    Duration: 242.07 ms
     Changes:   
              ----------
              diff:
                  --- 
                  +++ 
                  @@ -1,3 +1,5 @@
                  +
                  +
                   # DO NOT EDIT.  This file is automatically created by MAAS.
                   # Last updated at {{modified}}.
                   
                  @@ -10,6 +12,7 @@
                   acl localnet src {{cidr}}
                   {{endfor}}
                   acl SSL_ports port 443
                  +acl SSL_ports port 50000        # jenkins slave
                   acl Safe_ports port 80          # http
                   acl Safe_ports port 21          # ftp
                   acl Safe_ports port 443         # https
                  @@ -23,11 +26,7 @@
                   http_access allow localhost
                   http_access deny all
                   http_port 3128 transparent
                  -{{if not maas_proxy_port}}
                   http_port 8000
                  -{{else}}
                  -http_port {{maas_proxy_port}}
                  -{{endif}}
                   refresh_pattern ^ftp:           1440    20%     10080
                   refresh_pattern -i (/cgi-bin/|\?) 0     0%      0
                   refresh_pattern \/Release(|\.gpg)$                        0    0%    0 refresh-ims
                  @@ -61,7 +60,5 @@
                   cache_log /var/log/maas/proxy/cache.log
                   cache_store_log /var/log/maas/proxy/store.log
                   {{endif}}
                  -{{if upstream_peer_proxy}}
                  -cache_peer {{upstream_proxy_address}} parent {{upstream_proxy_port}} 0 no-query default
                  -never_direct allow all
                  -{{endif}}
                  +
                  +
----------
          ID: maas_timeout_commissioning
    Function: file.replace
        Name: /usr/lib/python3/dist-packages/maasserver/node_status.py
      Result: True
     Comment: Changes were made
     Started: 13:13:40.290542
    Duration: 10.522 ms
     Changes:   
              ----------
              diff:
                  --- 
                  +++ 
                  @@ -284,7 +284,7 @@
                   }
                   
                   NODE_FAILURE_MONITORED_STATUS_TIMEOUTS = {
                  -    NODE_STATUS.COMMISSIONING: 20,
                  +    NODE_STATUS.COMMISSIONING: 10,
                       NODE_STATUS.DEPLOYING: 40,
                       NODE_STATUS.RELEASING: 5,
                       NODE_STATUS.ENTERING_RESCUE_MODE: 20,
----------
          ID: maas_timeout_deploying
    Function: file.replace
        Name: /usr/lib/python3/dist-packages/maasserver/node_status.py
      Result: True
     Comment: Changes were made
     Started: 13:13:40.301650
    Duration: 9.406 ms
     Changes:   
              ----------
              diff:
                  --- 
                  +++ 
                  @@ -285,7 +285,7 @@
                   
                   NODE_FAILURE_MONITORED_STATUS_TIMEOUTS = {
                       NODE_STATUS.COMMISSIONING: 10,
                  -    NODE_STATUS.DEPLOYING: 40,
                  +    NODE_STATUS.DEPLOYING: 15,
                       NODE_STATUS.RELEASING: 5,
                       NODE_STATUS.ENTERING_RESCUE_MODE: 20,
                       NODE_STATUS.EXITING_RESCUE_MODE: 5,
----------
          ID: maas_interface_default_mode_dhcp
    Function: file.replace
        Name: /usr/lib/python3/dist-packages/maasserver/models/node.py
      Result: True
     Comment: Changes were made
     Started: 13:13:40.311676
    Duration: 148.342 ms
     Changes:   
              ----------
              diff:
                  --- 
                  +++ 
                  @@ -3210,7 +3210,7 @@
                               alloc_type=IPADDRESS_TYPE.DISCOVERED, subnet__isnull=False)
                           for ip_address in discovered_addresses:
                               boot_interface.link_subnet(
                  -                INTERFACE_LINK_TYPE.AUTO, ip_address.subnet)
                  +                INTERFACE_LINK_TYPE.DHCP, ip_address.subnet)
                               auto_set = True
                           if not auto_set:
                               # Failed to set AUTO mode on the boot interface. Lets force an
----------
          ID: /etc/apache2/conf-enabled/maas-http.conf
    Function: file.managed
      Result: True
     Comment: File /etc/apache2/conf-enabled/maas-http.conf updated
     Started: 13:13:40.460674
    Duration: 204.156 ms
     Changes:   
              ----------
              diff:
                  --- 
                  +++ 
                  @@ -37,3 +37,5 @@
                       # Redirect (permanently) requests for /MAAS to /MAAS/.
                       RewriteRule ^/MAAS$ %{REQUEST_URI}/ [R=301,L]
                   </IfModule>
                  +Header set Access-Control-Allow-Origin "*"
                  +Header set X-Frame-Options "ALLOWALL"
----------
          ID: maas_apache_headers
    Function: cmd.run
        Name: a2enmod headers
      Result: True
     Comment: Command "a2enmod headers" run
     Started: 13:13:40.668972
    Duration: 348.985 ms
     Changes:   
              ----------
              pid:
                  3739
              retcode:
                  0
              stderr:
              stdout:
                  Enabling module headers.
                  To activate the new configuration, you need to run:
                    service apache2 restart
----------
          ID: /usr/share/maas/web/static/css/maas-styles.css
    Function: file.managed
      Result: True
     Comment: File /usr/share/maas/web/static/css/maas-styles.css updated
     Started: 13:13:41.018370
    Duration: 324.081 ms
     Changes:   
              ----------
              diff:
                  New file
              mode:
                  0644
----------
          ID: /etc/maas/preseeds/curtin_userdata_amd64_generic_trusty
    Function: file.managed
      Result: True
     Comment: File /etc/maas/preseeds/curtin_userdata_amd64_generic_trusty updated
     Started: 13:13:41.343186
    Duration: 387.215 ms
     Changes:   
              ----------
              diff:
                  New file
              mode:
                  0644
----------
          ID: /etc/maas/preseeds/curtin_userdata_amd64_generic_xenial
    Function: file.managed
      Result: True
     Comment: File /etc/maas/preseeds/curtin_userdata_amd64_generic_xenial updated
     Started: 13:13:41.730987
    Duration: 337.59 ms
     Changes:   
              ----------
              diff:
                  New file
              mode:
                  0644
----------
          ID: /etc/maas/preseeds/curtin_userdata_arm64_generic_xenial
    Function: file.managed
      Result: True
     Comment: File /etc/maas/preseeds/curtin_userdata_arm64_generic_xenial updated
     Started: 13:13:42.069172
    Duration: 358.745 ms
     Changes:   
              ----------
              diff:
                  New file
              mode:
                  0644
----------
          ID: Configure /root/.pgpass for MAAS
    Function: file.managed
        Name: /root/.pgpass
      Result: True
     Comment: File /root/.pgpass updated
     Started: 13:13:42.428333
    Duration: 220.832 ms
     Changes:   
              ----------
              diff:
                  New file
----------
          ID: maas_region_syncdb
    Function: cmd.run
        Name: maas-region syncdb --noinput
      Result: True
     Comment: Command "maas-region syncdb --noinput" run
     Started: 13:13:42.660241
    Duration: 10190.383 ms
     Changes:   
              ----------
              pid:
                  3781
              retcode:
                  0
              stderr:
              stdout:
                  Operations to perform:
                    Synchronize unmigrated apps: staticfiles, messages
                    Apply all migrations: sessions, sites, maasserver, contenttypes, auth, piston3, metadataserver
                  Synchronizing apps without migrations:
                    Creating tables...
                      Running deferred SQL...
                    Installing custom SQL...
                  Running migrations:
                    No migrations to apply.
----------
          ID: maas_region_services
    Function: service.running
        Name: maas-regiond
      Result: True
     Comment: Service restarted
     Started: 13:13:53.154181
    Duration: 298.192 ms
     Changes:   
              ----------
              maas-regiond:
                  True
----------
          ID: maas_region_services
    Function: service.running
        Name: bind9
      Result: True
     Comment: Service restarted
     Started: 13:13:53.765844
    Duration: 564.597 ms
     Changes:   
              ----------
              bind9:
                  True
----------
          ID: maas_region_services
    Function: service.running
        Name: apache2
      Result: True
     Comment: Service restarted
     Started: 13:13:54.594059
    Duration: 2039.226 ms
     Changes:   
              ----------
              apache2:
                  True
----------
          ID: maas_warmup
    Function: module.run
        Name: maasng.wait_for_http_code
      Result: True
     Comment: Module function maasng.wait_for_http_code executed
     Started: 13:13:56.635691
    Duration: 11410.206 ms
     Changes:   
              ----------
              ret:
                  ----------
                  comment:
                      MAAS API:http://localhost:5240/MAAS up.
                  result:
                      True
----------
          ID: maas_set_admin_password
    Function: cmd.run
        Name: maas createadmin --username opnfv --password opnfv_secret --email email@example.com && touch /var/lib/maas/.setup_admin
      Result: True
     Comment: Command "maas createadmin --username opnfv --password opnfv_secret --email email@example.com && touch /var/lib/maas/.setup_admin" run
     Started: 13:14:08.060310
    Duration: 6690.289 ms
     Changes:   
              ----------
              pid:
                  4257
              retcode:
                  0
              stderr:
              stdout:
----------
          ID: maas_login_admin
    Function: cmd.run
        Name: maas-region apikey --username opnfv > /var/lib/maas/.maas_credentials
      Result: True
     Comment: Command "maas-region apikey --username opnfv > /var/lib/maas/.maas_credentials" run
     Started: 13:14:14.756910
    Duration: 5713.162 ms
     Changes:   
              ----------
              pid:
                  4317
              retcode:
                  0
              stderr:
              stdout:
----------
          ID: maas_region_boot_source_resources_mirror
    Function: maasng.boot_source_present
      Result: True
     Comment: boot-source http://images.maas.io/ephemeral-v3/daily presented
     Started: 13:14:20.497266
    Duration: 67709.652 ms
     Changes:   
              ----------
              changes:
                  ----------
                  new:
                      Boot-resource http://images.maas.io/ephemeral-v3/daily/ deleted
              new:
                  boot resource http://images.maas.io/ephemeral-v3/daily was created
    Warnings: 'name' is an invalid keyword argument for
              'maasng.boot_source_present'. If you were trying to pass
              additional data to be used in a template context, please populate
              'context' with 'key: value' pairs. Your approach will work until
              Salt Fluorine is out. Please update your state files.
----------
          ID: maas_wait_for_region_import_done
    Function: module.run
        Name: maasng.boot_resources_import
      Result: True
     Comment: Module function maasng.boot_resources_import executed
     Started: 13:15:28.211366
    Duration: 43130.203 ms
     Changes:   
              ----------
              ret:
                  True
----------
          ID: maas_region_boot_sources_selection_xenial
    Function: maasng.boot_sources_selections_present
      Result: True
     Comment: boot-source selection for http://images.maas.io/ephemeral-v3/daily was created
     Started: 13:16:11.345040
    Duration: 12030.093 ms
     Changes:   
    Warnings: 'name' is an invalid keyword argument for
              'maasng.boot_sources_selections_present'. If you were trying to
              pass additional data to be used in a template context, please
              populate 'context' with 'key: value' pairs. Your approach will
              work until Salt Fluorine is out. Please update your state files.
----------
          ID: maas_wait_for_racks_import_done
    Function: module.run
        Name: maasng.sync_and_wait_bs_to_all_racks
      Result: True
     Comment: Module function maasng.sync_and_wait_bs_to_all_racks executed
     Started: 13:16:23.376863
    Duration: 539720.34 ms
     Changes:   
              ----------
              ret:
                  True
----------
          ID: maas_config
    Function: module.run
        Name: maas.process_maas_config
      Result: True
     Comment: Module function maas.process_maas_config executed
     Started: 13:25:23.101823
    Duration: 10457.539 ms
     Changes:   
              ----------
              ret:
                  ----------
                  errors:
                      ----------
                  success:
                      - enable_http_proxy
                      - upstream_dns
                      - commissioning_distro_series
                      - default_osystem
                      - active_discovery_interval
                      - dnssec_validation
                      - maas_name
                      - network_discovery
                      - enable_third_party_drivers
                      - default_storage_layout
                      - ntp_external_only
                      - disk_erase_with_secure_erase
                      - default_distro_series
                      - default_min_hwe_kernel
                  updated:
----------
          ID: maas_fabrics_pxe_admin
    Function: maasng.fabric_present
        Name: pxe_admin
      Result: True
     Comment: Module function maasng.fabric_present executed
     Started: 13:25:33.560250
    Duration: 690.895 ms
     Changes:   
              ----------
              new:
                  Fabric  pxe_admin created
              result:
                  True
----------
          ID: maas_vlan0_present_for_pxe_admin
    Function: maasng.vlan_present_in_fabric
        Name: vlan 0
      Result: True
     Comment: Vlan vlan 0 has been updated for pxe_admin
     Started: 13:25:34.251491
    Duration: 2770.728 ms
     Changes:   
              ----------
              new:
                  Vlan untagged was updated
----------
          ID: maas_create_subnet_10.1.0.0/24
    Function: maasng.subnet_present
        Name: 10.1.0.0/24
      Result: True
     Comment: Subnet 10.1.0.0/24 has been updated for pxe_admin
     Started: 13:25:37.023558
    Duration: 1296.833 ms
     Changes:   
----------
          ID: maas_create_iprange_1
    Function: maasng.iprange_present
      Result: True
     Comment: Module function maasng.iprange_present executed
     Started: 13:25:38.321690
    Duration: 537.687 ms
     Changes:   
              ----------
              new:
                  Iprange: Type:dynamic: 10.1.0.39-10.1.0.254 has been created
----------
          ID: maas_vlan0_present_for_pxe_admin_dhcp
    Function: maasng.vlan_present_in_fabric
        Name: vlan 0
      Result: True
     Comment: Vlan vlan 0 has been updated for pxe_admin
     Started: 13:25:38.859674
    Duration: 1519.32 ms
     Changes:   
              ----------
              new:
                  Vlan untagged was updated
----------
          ID: maas_sshkey_1
    Function: maasng.sshkey_present
        Name: opnfv
      Result: True
     Comment: SSH-key ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9iteXSOD1Du/fG2coTPKPKmlyjD5EHa0wwZm9pdya5BuWatdcIzkCgKLdLBST3LZZ6U8IB83LnCwKa7CX87OPxZVylK8W8NlzsjYt3P+ZOX8qUmhTxxkJa4i43aZXbQwv1a6QAwHY6n2/8mKdLpov5gf6Y28HpUN1d3mp3gegNK5quTNAxS2MzUXk9gOZ8ZXYLXCYjg4eWXXXCLxK+rmojXgsjdY3WruVLR+1YkXX38jJ0pirCp10knC+QGf1QTQE2quzYQNh8NQA+1weNUWi6xYd8c9Nt/bBdtoL56hqivJeNYiFw2PLeoGqfOtI1OrwR1Cp0WdoJ+cIdW9P/7ov has been added for user opnfv
     Started: 13:25:40.381300
    Duration: 825.893 ms
     Changes:   
              ----------
              new:
                  SSH Key ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9iteXSOD1Du/fG2coTPKPKmlyjD5EHa0wwZm9pdya5BuWatdcIzkCgKLdLBST3LZZ6U8IB83LnCwKa7CX87OPxZVylK8W8NlzsjYt3P+ZOX8qUmhTxxkJa4i43aZXbQwv1a6QAwHY6n2/8mKdLpov5gf6Y28HpUN1d3mp3gegNK5quTNAxS2MzUXk9gOZ8ZXYLXCYjg4eWXXXCLxK+rmojXgsjdY3WruVLR+1YkXX38jJ0pirCp10knC+QGf1QTQE2quzYQNh8NQA+1weNUWi6xYd8c9Nt/bBdtoL56hqivJeNYiFw2PLeoGqfOtI1OrwR1Cp0WdoJ+cIdW9P/7ov was added.

Summary for mas01.mcp-odl-noha.local
-------------
Succeeded: 32 (changed=28)
Failed:     0
Warnings:   2
-------------
Total states run:     32
Total run time:  720.456 s
[wait_for] OK: salt -C 'mas01*' state.apply maas.region
mas01.mcp-odl-noha.local:
----------
          ID: maas_login_admin
    Function: cmd.run
        Name: maas-region apikey --username opnfv > /var/lib/maas/.maas_credentials
      Result: True
     Comment: Command "maas-region apikey --username opnfv > /var/lib/maas/.maas_credentials" run
     Started: 13:25:56.532650
    Duration: 5291.562 ms
     Changes:   
              ----------
              pid:
                  6356
              retcode:
                  0
              stderr:
              stdout:
----------
          ID: create__machines
    Function: module.run
        Name: maas.process_machines
      Result: True
     Comment: Module function maas.process_machines executed
     Started: 13:26:01.828220
    Duration: 30563.985 ms
     Changes:   
              ----------
              ret:
                  ----------
                  errors:
                      ----------
                  success:
                      - cmp002
                      - cmp003
                      - cmp001
                      - odl01
                      - gtw01
                      - ctl01
                  updated:

Summary for mas01.mcp-odl-noha.local
------------
Succeeded: 2 (changed=2)
Failed:    0
------------
Total states run:     2
Total run time:  35.856 s
[NOTE] MaaS operations might take a long time, please be patient
tput: unknown terminal "unknown"
tput: unknown terminal "unknown"
mas01.mcp-odl-noha.local:
----------
          ID: maas_login_admin
    Function: cmd.run
        Name: maas-region apikey --username opnfv > /var/lib/maas/.maas_credentials
      Result: True
     Comment: Command "maas-region apikey --username opnfv > /var/lib/maas/.maas_credentials" run
     Started: 13:27:18.055072
    Duration: 6120.956 ms
     Changes:   
              ----------
              pid:
                  6787
              retcode:
                  0
              stderr:
              stdout:
----------
          ID: wait_for_machines_ready_or_deployed
    Function: module.run
        Name: maas.wait_for_machine_status
      Result: False
     Comment: Module function maas.wait_for_machine_status threw an exception. Exception: Machines:['cmp002', 'cmp003', 'cmp001', 'odl01', 'gtw01', 'ctl01']not in Ready|Deployed state
     Started: 13:27:24.180977
    Duration: 1506120.011 ms
     Changes:   

Summary for mas01.mcp-odl-noha.local
------------
Succeeded: 1 (changed=1)
Failed:    1
------------
Total states run:     2
Total run time: 1512.241 s
ERROR: Minions returned with non-zero exit code
[wait_for] Attempt 2/5 for: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i /var/lib/opnfv/mcp.rsa ubuntu@10.20.0.2 sudo             CI_DEBUG=false ERASE_ENV=0             /root/fuel/mcp/config/states/maas
Warning: Permanently added '10.20.0.2' (ECDSA) to the list of known hosts.

[wait_for] Waiting for cmd to return success: salt --out yaml -C 'mas01*' service.status maas-fixup | fgrep -q 'false'
[wait_for] Attempt 1/60 for: salt --out yaml -C 'mas01*' service.status maas-fixup | fgrep -q 'false'
[wait_for] OK: salt --out yaml -C 'mas01*' service.status maas-fixup | fgrep -q 'false'

[wait_for] Waiting for cmd to return success: salt -C 'mas01*' state.apply salt,iptables
[wait_for] Attempt 1/10 for: salt -C 'mas01*' state.apply salt,iptables
mas01.mcp-odl-noha.local:
----------
          ID: salt_minion_packages
    Function: pkg.installed
        Name: salt-minion
      Result: True
     Comment: All specified packages are already installed
     Started: 13:53:09.239036
    Duration: 172.267 ms
     Changes:   
----------
          ID: salt_minion_dependency_packages
    Function: pkg.installed
      Result: True
     Comment: All specified packages are already installed
     Started: 13:53:09.411915
    Duration: 14.361 ms
     Changes:   
----------
          ID: /etc/salt/minion.d/minion.conf
    Function: file.managed
      Result: True
     Comment: File /etc/salt/minion.d/minion.conf is in the correct state
     Started: 13:53:09.432401
    Duration: 612.015 ms
     Changes:   
----------
          ID: salt_minion_systemd_override
    Function: file.managed
        Name: /etc/systemd/system/salt-minion.service.d/50-restarts.conf
      Result: True
     Comment: File /etc/systemd/system/salt-minion.service.d/50-restarts.conf is in the correct state
     Started: 13:53:10.050016
    Duration: 62.85 ms
     Changes:   
----------
          ID: salt_minion_systemd_reload
    Function: module.wait
        Name: service.systemctl_reload
      Result: True
     Comment: State was not run because none of the onchanges reqs changed
     Started: 13:53:10.113606
    Duration: 0.02 ms
     Changes:   
----------
          ID: salt_minion_service
    Function: service.running
        Name: salt-minion
      Result: True
     Comment: The service salt-minion is already running
     Started: 13:53:10.114415
    Duration: 132.994 ms
     Changes:   
----------
          ID: salt_minion_sync_all
    Function: module.run
        Name: saltutil.sync_all
      Result: True
     Comment: State was not run because none of the onchanges reqs changed
     Started: 13:53:10.249409
    Duration: 0.037 ms
     Changes:   
----------
          ID: salt_minion_grains_dir
    Function: file.directory
        Name: /etc/salt/grains.d
      Result: True
     Comment: Directory /etc/salt/grains.d is in the correct state
              Directory /etc/salt/grains.d updated
     Started: 13:53:10.249817
    Duration: 2.583 ms
     Changes:   
----------
          ID: salt_minion_grains_files
    Function: file.managed
        Name: /etc/salt/grains
      Result: True
     Comment: File /etc/salt/grains exists with proper permissions. No changes made.
     Started: 13:53:10.253551
    Duration: 1.576 ms
     Changes:   
----------
          ID: salt_minion_grains_files
    Function: file.managed
        Name: /etc/salt/grains.d/placeholder
      Result: True
     Comment: File /etc/salt/grains.d/placeholder exists with proper permissions. No changes made.
     Started: 13:53:10.255839
    Duration: 1.522 ms
     Changes:   
----------
          ID: salt_minion_grain_linux_sphinx
    Function: file.managed
        Name: /etc/salt/grains.d/sphinx
      Result: True
     Comment: File /etc/salt/grains.d/sphinx is in the correct state
     Started: 13:53:10.258022
    Duration: 7.071 ms
     Changes:   
----------
          ID: salt_minion_grain_linux_sphinx_validity_check
    Function: cmd.wait
        Name: python -c "import yaml; stream = file('/etc/salt/grains.d/sphinx', 'r'); yaml.load(stream); stream.close()"
      Result: True
     Comment: 
     Started: 13:53:10.269331
    Duration: 1.527 ms
     Changes:   
----------
          ID: salt_minion_grain_linux_dns_records
    Function: file.managed
        Name: /etc/salt/grains.d/dns_records
      Result: True
     Comment: File /etc/salt/grains.d/dns_records is in the correct state
     Started: 13:53:10.271579
    Duration: 2.332 ms
     Changes:   
----------
          ID: salt_minion_grain_linux_dns_records_validity_check
    Function: cmd.wait
        Name: python -c "import yaml; stream = file('/etc/salt/grains.d/dns_records', 'r'); yaml.load(stream); stream.close()"
      Result: True
     Comment: 
     Started: 13:53:10.275322
    Duration: 1.256 ms
     Changes:   
----------
          ID: salt_minion_grain_salt_salt
    Function: file.managed
        Name: /etc/salt/grains.d/salt
      Result: True
     Comment: File /etc/salt/grains.d/salt is in the correct state
     Started: 13:53:10.277249
    Duration: 2.256 ms
     Changes:   
----------
          ID: salt_minion_grain_salt_salt_validity_check
    Function: cmd.wait
        Name: python -c "import yaml; stream = file('/etc/salt/grains.d/salt', 'r'); yaml.load(stream); stream.close()"
      Result: True
     Comment: 
     Started: 13:53:10.280886
    Duration: 1.189 ms
     Changes:   
----------
          ID: salt_minion_grains_file
    Function: cmd.wait
        Name: cat /etc/salt/grains.d/* > /etc/salt/grains
      Result: True
     Comment: 
     Started: 13:53:10.284892
    Duration: 1.453 ms
     Changes:   
----------
          ID: salt_minion_grains_publish
    Function: module.wait
        Name: mine.update
      Result: True
     Comment: 
     Started: 13:53:10.287406
    Duration: 1.041 ms
     Changes:   
----------
          ID: salt_ca_certificates_packages
    Function: pkg.installed
        Name: ca-certificates
      Result: True
     Comment: All specified packages are already installed
     Started: 13:53:10.293372
    Duration: 14.441 ms
     Changes:   
----------
          ID: salt_update_certificates
    Function: cmd.wait
        Name: update-ca-certificates
      Result: True
     Comment: 
     Started: 13:53:10.309756
    Duration: 1.402 ms
     Changes:   
----------
          ID: iptables_packages_v4
    Function: pkg.installed
        Name: iptables
      Result: True
     Comment: All specified packages are already installed
     Started: 13:53:10.311530
    Duration: 17.334 ms
     Changes:   
----------
          ID: iptables_packages_v4
    Function: pkg.installed
        Name: iptables-persistent
      Result: True
     Comment: All specified packages are already installed
     Started: 13:53:10.329220
    Duration: 13.705 ms
     Changes:   
----------
          ID: iptables_modules_v4_load
    Function: kmod.present
      Result: True
     Comment: Kernel modules iptable_filter, ip_tables are already present
     Started: 13:53:10.344838
    Duration: 42.572 ms
     Changes:   
----------
          ID: /etc/iptables/rules.v4
    Function: file.managed
      Result: True
     Comment: File /etc/iptables/rules.v4 is in the correct state
     Started: 13:53:10.388485
    Duration: 229.912 ms
     Changes:   
----------
          ID: iptables_services_v4_start
    Function: cmd.run
        Name: find /usr/share/netfilter-persistent/plugins.d/[0-9]*-ip4tables -exec {} start \;
      Result: True
     Comment: onlyif execution failed
     Started: 13:53:10.620167
    Duration: 90.264 ms
     Changes:   
----------
          ID: netfilter-persistent
    Function: service.running
      Result: True
     Comment: The service netfilter-persistent is already running
     Started: 13:53:10.712054
    Duration: 214.418 ms
     Changes:   
----------
          ID: iptables_tables_cleanup_v4
    Function: module.wait
        Name: iptables_extra.remove_stale_tables
      Result: True
     Comment: 
     Started: 13:53:10.927844
    Duration: 1.471 ms
     Changes:   
----------
          ID: iptables_services_v6_stop
    Function: cmd.run
        Name: find /usr/share/netfilter-persistent/plugins.d/[0-9]*-ip6tables -exec {} flush \;
      Result: True
     Comment: onlyif execution failed
     Started: 13:53:10.929665
    Duration: 30.362 ms
     Changes:   
----------
          ID: /etc/iptables/rules.v6
    Function: file.absent
      Result: True
     Comment: File /etc/iptables/rules.v6 is not present
     Started: 13:53:10.961628
    Duration: 1.665 ms
     Changes:   
----------
          ID: iptables_tables_flush_v6
    Function: module.wait
        Name: iptables_extra.flush_all
      Result: True
     Comment: 
     Started: 13:53:10.964419
    Duration: 1.208 ms
     Changes:   
----------
          ID: salt_minion_service_restart
    Function: cmd.run
        Name: while true; do salt-call saltutil.running|grep fun: && continue; salt-call --local service.restart salt-minion; break; done
      Result: True
     Comment: State was not run because none of the onchanges reqs changed
     Started: 13:53:10.967015
    Duration: 0.026 ms
     Changes:   

Summary for mas01.mcp-odl-noha.local
-------------
Succeeded: 31
Failed:     0
-------------
Total states run:     31
Total run time:    1.677 s
[wait_for] OK: salt -C 'mas01*' state.apply salt,iptables
mas01.mcp-odl-noha.local:
----------
          ID: maas_cluster_packages
    Function: pkg.installed
        Name: maas-rack-controller
      Result: True
     Comment: All specified packages are already installed
     Started: 13:53:25.389189
    Duration: 184.709 ms
     Changes:   
----------
          ID: maas_cluster_packages
    Function: pkg.installed
        Name: ipmitool
      Result: True
     Comment: All specified packages are already installed
     Started: 13:53:25.574336
    Duration: 13.566 ms
     Changes:   
----------
          ID: /etc/maas/rackd.conf
    Function: file.line
      Result: True
     Comment: No changes needed to be made
     Started: 13:53:25.612468
    Duration: 2.509 ms
     Changes:   
----------
          ID: /etc/maas/rackd_conf
    Function: file.managed
        Name: /etc/maas/rackd.conf
      Result: True
     Comment: File /etc/maas/rackd.conf exists with proper permissions. No changes made.
     Started: 13:53:25.615323
    Duration: 2.504 ms
     Changes:   
----------
          ID: maas_cluster_services
    Function: service.running
        Name: maas-rackd
      Result: True
     Comment: The service maas-rackd is already running
     Started: 13:53:25.619554
    Duration: 161.505 ms
     Changes:   

Summary for mas01.mcp-odl-noha.local
------------
Succeeded: 5
Failed:    0
------------
Total states run:     5
Total run time: 364.793 ms

[wait_for] Waiting for cmd to return success: salt -C 'mas01*' state.apply maas.region
[wait_for] Attempt 1/10 for: salt -C 'mas01*' state.apply maas.region
mas01.mcp-odl-noha.local:
----------
          ID: maas_region_packages
    Function: pkg.installed
        Name: maas-region-controller
      Result: True
     Comment: All specified packages are already installed
     Started: 13:53:31.357811
    Duration: 275.49 ms
     Changes:   
----------
          ID: maas_region_packages
    Function: pkg.installed
        Name: python-oauth
      Result: True
     Comment: All specified packages are already installed
     Started: 13:53:31.633800
    Duration: 35.351 ms
     Changes:   
----------
          ID: /etc/maas/regiond.conf
    Function: file.replace
      Result: True
     Comment: No changes needed to be made
     Started: 13:53:31.675282
    Duration: 18.285 ms
     Changes:   
----------
          ID: /usr/lib/python3/dist-packages/provisioningserver/templates/proxy/maas-proxy.conf.template
    Function: file.managed
      Result: True
     Comment: File /usr/lib/python3/dist-packages/provisioningserver/templates/proxy/maas-proxy.conf.template is in the correct state
     Started: 13:53:31.694281
    Duration: 166.818 ms
     Changes:   
----------
          ID: maas_timeout_commissioning
    Function: file.replace
        Name: /usr/lib/python3/dist-packages/maasserver/node_status.py
      Result: True
     Comment: No changes needed to be made
     Started: 13:53:31.861843
    Duration: 6.561 ms
     Changes:   
----------
          ID: maas_timeout_deploying
    Function: file.replace
        Name: /usr/lib/python3/dist-packages/maasserver/node_status.py
      Result: True
     Comment: No changes needed to be made
     Started: 13:53:31.869336
    Duration: 14.027 ms
     Changes:   
----------
          ID: maas_interface_default_mode_dhcp
    Function: file.replace
        Name: /usr/lib/python3/dist-packages/maasserver/models/node.py
      Result: True
     Comment: No changes needed to be made
     Started: 13:53:31.884110
    Duration: 74.833 ms
     Changes:   
----------
          ID: /etc/apache2/conf-enabled/maas-http.conf
    Function: file.managed
      Result: True
     Comment: File /etc/apache2/conf-enabled/maas-http.conf is in the correct state
     Started: 13:53:31.959711
    Duration: 80.173 ms
     Changes:   
----------
          ID: maas_apache_headers
    Function: cmd.run
        Name: a2enmod headers
      Result: True
     Comment: Command "a2enmod headers" run
     Started: 13:53:32.041856
    Duration: 137.816 ms
     Changes:   
              ----------
              pid:
                  12363
              retcode:
                  0
              stderr:
              stdout:
                  Module headers already enabled
----------
          ID: /usr/share/maas/web/static/css/maas-styles.css
    Function: file.managed
      Result: True
     Comment: File /usr/share/maas/web/static/css/maas-styles.css is in the correct state
     Started: 13:53:32.180192
    Duration: 114.624 ms
     Changes:   
----------
          ID: /etc/maas/preseeds/curtin_userdata_amd64_generic_trusty
    Function: file.managed
      Result: True
     Comment: File /etc/maas/preseeds/curtin_userdata_amd64_generic_trusty is in the correct state
     Started: 13:53:32.295707
    Duration: 199.832 ms
     Changes:   
----------
          ID: /etc/maas/preseeds/curtin_userdata_amd64_generic_xenial
    Function: file.managed
      Result: True
     Comment: File /etc/maas/preseeds/curtin_userdata_amd64_generic_xenial is in the correct state
     Started: 13:53:32.496271
    Duration: 171.04 ms
     Changes:   
----------
          ID: /etc/maas/preseeds/curtin_userdata_arm64_generic_xenial
    Function: file.managed
      Result: True
     Comment: File /etc/maas/preseeds/curtin_userdata_arm64_generic_xenial is in the correct state
     Started: 13:53:32.668043
    Duration: 194.39 ms
     Changes:   
----------
          ID: Configure /root/.pgpass for MAAS
    Function: file.managed
        Name: /root/.pgpass
      Result: True
     Comment: File /root/.pgpass is in the correct state
     Started: 13:53:32.862839
    Duration: 185.312 ms
     Changes:   
----------
          ID: maas_region_syncdb
    Function: cmd.run
        Name: maas-region syncdb --noinput
      Result: True
     Comment: Command "maas-region syncdb --noinput" run
     Started: 13:53:33.056200
    Duration: 8977.013 ms
     Changes:   
              ----------
              pid:
                  12376
              retcode:
                  0
              stderr:
              stdout:
                  Operations to perform:
                    Synchronize unmigrated apps: messages, staticfiles
                    Apply all migrations: contenttypes, maasserver, piston3, auth, metadataserver, sessions, sites
                  Synchronizing apps without migrations:
                    Creating tables...
                      Running deferred SQL...
                    Installing custom SQL...
                  Running migrations:
                    No migrations to apply.
----------
          ID: maas_region_services
    Function: service.running
        Name: maas-regiond
      Result: True
     Comment: The service maas-regiond is already running
     Started: 13:53:42.036334
    Duration: 252.899 ms
     Changes:   
----------
          ID: maas_region_services
    Function: service.running
        Name: bind9
      Result: True
     Comment: The service bind9 is already running
     Started: 13:53:42.291747
    Duration: 305.995 ms
     Changes:   
----------
          ID: maas_region_services
    Function: service.running
        Name: apache2
      Result: True
     Comment: The service apache2 is already running
     Started: 13:53:42.600175
    Duration: 194.485 ms
     Changes:   
----------
          ID: maas_warmup
    Function: module.run
        Name: maasng.wait_for_http_code
      Result: True
     Comment: Module function maasng.wait_for_http_code executed
     Started: 13:53:42.798064
    Duration: 622.158 ms
     Changes:   
              ----------
              ret:
                  ----------
                  comment:
                      MAAS API:http://localhost:5240/MAAS up.
                  result:
                      True
----------
          ID: maas_set_admin_password
    Function: cmd.run
        Name: maas createadmin --username opnfv --password opnfv_secret --email email@example.com && touch /var/lib/maas/.setup_admin
      Result: True
     Comment: /var/lib/maas/.setup_admin exists
     Started: 13:53:43.421470
    Duration: 1.433 ms
     Changes:   
----------
          ID: maas_login_admin
    Function: cmd.run
        Name: maas-region apikey --username opnfv > /var/lib/maas/.maas_credentials
      Result: True
     Comment: Command "maas-region apikey --username opnfv > /var/lib/maas/.maas_credentials" run
     Started: 13:53:43.423979
    Duration: 4363.294 ms
     Changes:   
              ----------
              pid:
                  12421
              retcode:
                  0
              stderr:
              stdout:
----------
          ID: maas_region_boot_source_resources_mirror
    Function: maasng.boot_source_present
      Result: True
     Comment: boot-source http://images.maas.io/ephemeral-v3/daily alredy exist
     Started: 13:53:47.797874
    Duration: 233.573 ms
     Changes:   
              ----------
              changes:
                  ----------
    Warnings: 'name' is an invalid keyword argument for
              'maasng.boot_source_present'. If you were trying to pass
              additional data to be used in a template context, please populate
              'context' with 'key: value' pairs. Your approach will work until
              Salt Fluorine is out. Please update your state files.
----------
          ID: maas_wait_for_region_import_done
    Function: module.run
        Name: maasng.boot_resources_import
      Result: True
     Comment: Module function maasng.boot_resources_import executed
     Started: 13:53:48.032460
    Duration: 26281.448 ms
     Changes:   
              ----------
              ret:
                  True
----------
          ID: maas_region_boot_sources_selection_xenial
    Function: maasng.boot_sources_selections_present
      Result: True
     Comment: Requested boot-source selection for http://images.maas.io/ephemeral-v3/daily already exist.
     Started: 13:54:14.315245
    Duration: 1474.322 ms
     Changes:   
    Warnings: 'name' is an invalid keyword argument for
              'maasng.boot_sources_selections_present'. If you were trying to
              pass additional data to be used in a template context, please
              populate 'context' with 'key: value' pairs. Your approach will
              work until Salt Fluorine is out. Please update your state files.
----------
          ID: maas_wait_for_racks_import_done
    Function: module.run
        Name: maasng.sync_and_wait_bs_to_all_racks
      Result: True
     Comment: Module function maasng.sync_and_wait_bs_to_all_racks executed
     Started: 13:54:15.790709
    Duration: 2891.964 ms
     Changes:   
              ----------
              ret:
                  True
----------
          ID: maas_config
    Function: module.run
        Name: maas.process_maas_config
      Result: True
     Comment: Module function maas.process_maas_config executed
     Started: 13:54:18.685372
    Duration: 7163.377 ms
     Changes:   
              ----------
              ret:
                  ----------
                  errors:
                      ----------
                  success:
                      - enable_http_proxy
                      - upstream_dns
                      - commissioning_distro_series
                      - default_osystem
                      - active_discovery_interval
                      - dnssec_validation
                      - maas_name
                      - network_discovery
                      - enable_third_party_drivers
                      - default_storage_layout
                      - ntp_external_only
                      - disk_erase_with_secure_erase
                      - default_distro_series
                      - default_min_hwe_kernel
                  updated:
----------
          ID: maas_fabrics_pxe_admin
    Function: maasng.fabric_present
        Name: pxe_admin
      Result: True
     Comment: Module function maasng.fabric_present executed
     Started: 13:54:25.849497
    Duration: 958.948 ms
     Changes:   
              ----------
              new:
                  Fabric  pxe_admin created
              result:
                  True
----------
          ID: maas_vlan0_present_for_pxe_admin
    Function: maasng.vlan_present_in_fabric
        Name: vlan 0
      Result: True
     Comment: Vlan vlan 0 has been updated for pxe_admin
     Started: 13:54:26.808866
    Duration: 1584.969 ms
     Changes:   
              ----------
              new:
                  Vlan untagged was updated
----------
          ID: maas_create_subnet_10.1.0.0/24
    Function: maasng.subnet_present
        Name: 10.1.0.0/24
      Result: True
     Comment: Subnet 10.1.0.0/24 has been updated for pxe_admin
     Started: 13:54:28.395248
    Duration: 771.281 ms
     Changes:   
----------
          ID: maas_create_iprange_1
    Function: maasng.iprange_present
      Result: True
     Comment: Iprange maas_create_iprange_1 already exist.
     Started: 13:54:29.167721
    Duration: 230.055 ms
     Changes:   
----------
          ID: maas_vlan0_present_for_pxe_admin_dhcp
    Function: maasng.vlan_present_in_fabric
        Name: vlan 0
      Result: True
     Comment: Vlan vlan 0 has been updated for pxe_admin
     Started: 13:54:29.398601
    Duration: 3091.09 ms
     Changes:   
              ----------
              new:
                  Vlan untagged was updated
----------
          ID: maas_sshkey_1
    Function: maasng.sshkey_present
        Name: opnfv
      Result: True
     Comment: SSH key ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9iteXSOD1Du/fG2coTPKPKmlyjD5EHa0wwZm9pdya5BuWatdcIzkCgKLdLBST3LZZ6U8IB83LnCwKa7CX87OPxZVylK8W8NlzsjYt3P+ZOX8qUmhTxxkJa4i43aZXbQwv1a6QAwHY6n2/8mKdLpov5gf6Y28HpUN1d3mp3gegNK5quTNAxS2MzUXk9gOZ8ZXYLXCYjg4eWXXXCLxK+rmojXgsjdY3WruVLR+1YkXX38jJ0pirCp10knC+QGf1QTQE2quzYQNh8NQA+1weNUWi6xYd8c9Nt/bBdtoL56hqivJeNYiFw2PLeoGqfOtI1OrwR1Cp0WdoJ+cIdW9P/7ov already exist for user opnfv.
     Started: 13:54:32.490585
    Duration: 386.842 ms
     Changes:   

Summary for mas01.mcp-odl-noha.local
-------------
Succeeded: 32 (changed=11)
Failed:     0
Warnings:   2
-------------
Total states run:     32
Total run time:   61.460 s
[wait_for] OK: salt -C 'mas01*' state.apply maas.region
mas01.mcp-odl-noha.local:
----------
          ID: maas_login_admin
    Function: cmd.run
        Name: maas-region apikey --username opnfv > /var/lib/maas/.maas_credentials
      Result: True
     Comment: Command "maas-region apikey --username opnfv > /var/lib/maas/.maas_credentials" run
     Started: 13:54:47.358526
    Duration: 5292.009 ms
     Changes:   
              ----------
              pid:
                  12776
              retcode:
                  0
              stderr:
              stdout:
----------
          ID: create__machines
    Function: module.run
        Name: maas.process_machines
      Result: True
     Comment: Module function maas.process_machines executed
     Started: 13:54:52.676285
    Duration: 17604.152 ms
     Changes:   
              ----------
              ret:
                  ----------
                  errors:
                      ----------
                  success:
                  updated:
                      - cmp002
                      - cmp003
                      - cmp001
                      - odl01
                      - gtw01
                      - ctl01

Summary for mas01.mcp-odl-noha.local
------------
Succeeded: 2 (changed=2)
Failed:    0
------------
Total states run:     2
Total run time:  22.896 s
tput: unknown terminal "unknown"
[NOTE] MaaS operations might take a long time, please be patient
tput: unknown terminal "unknown"
mas01.mcp-odl-noha.local:
----------
          ID: maas_login_admin
    Function: cmd.run
        Name: maas-region apikey --username opnfv > /var/lib/maas/.maas_credentials
      Result: True
     Comment: Command "maas-region apikey --username opnfv > /var/lib/maas/.maas_credentials" run
     Started: 13:55:54.048764
    Duration: 4725.028 ms
     Changes:   
              ----------
              pid:
                  13105
              retcode:
                  0
              stderr:
              stdout:
----------
          ID: wait_for_machines_ready_or_deployed
    Function: module.run
        Name: maas.wait_for_machine_status
      Result: False
     Comment: Module function maas.wait_for_machine_status threw an exception. Exception: Machines:['cmp002', 'cmp003', 'cmp001', 'odl01', 'gtw01', 'ctl01']not in Ready|Deployed state
     Started: 13:55:58.789713
    Duration: 1505988.224 ms
     Changes:   

Summary for mas01.mcp-odl-noha.local
------------
Succeeded: 1 (changed=1)
Failed:    1
------------
Total states run:     2
Total run time: 1510.713 s
ERROR: Minions returned with non-zero exit code
[wait_for] Attempt 3/5 for: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i /var/lib/opnfv/mcp.rsa ubuntu@10.20.0.2 sudo             CI_DEBUG=false ERASE_ENV=0             /root/fuel/mcp/config/states/maas
Warning: Permanently added '10.20.0.2' (ECDSA) to the list of known hosts.

[wait_for] Waiting for cmd to return success: salt --out yaml -C 'mas01*' service.status maas-fixup | fgrep -q 'false'
[wait_for] Attempt 1/60 for: salt --out yaml -C 'mas01*' service.status maas-fixup | fgrep -q 'false'
[wait_for] OK: salt --out yaml -C 'mas01*' service.status maas-fixup | fgrep -q 'false'

[wait_for] Waiting for cmd to return success: salt -C 'mas01*' state.apply salt,iptables
[wait_for] Attempt 1/10 for: salt -C 'mas01*' state.apply salt,iptables
mas01.mcp-odl-noha.local:
----------
          ID: salt_minion_packages
    Function: pkg.installed
        Name: salt-minion
      Result: True
     Comment: All specified packages are already installed
     Started: 14:21:47.897466
    Duration: 258.723 ms
     Changes:   
----------
          ID: salt_minion_dependency_packages
    Function: pkg.installed
      Result: True
     Comment: All specified packages are already installed
     Started: 14:21:48.156826
    Duration: 20.748 ms
     Changes:   
----------
          ID: /etc/salt/minion.d/minion.conf
    Function: file.managed
      Result: True
     Comment: File /etc/salt/minion.d/minion.conf is in the correct state
     Started: 14:21:48.196677
    Duration: 881.323 ms
     Changes:   
----------
          ID: salt_minion_systemd_override
    Function: file.managed
        Name: /etc/systemd/system/salt-minion.service.d/50-restarts.conf
      Result: True
     Comment: File /etc/systemd/system/salt-minion.service.d/50-restarts.conf is in the correct state
     Started: 14:21:49.109772
    Duration: 90.153 ms
     Changes:   
----------
          ID: salt_minion_systemd_reload
    Function: module.wait
        Name: service.systemctl_reload
      Result: True
     Comment: State was not run because none of the onchanges reqs changed
     Started: 14:21:49.200710
    Duration: 0.024 ms
     Changes:   
----------
          ID: salt_minion_service
    Function: service.running
        Name: salt-minion
      Result: True
     Comment: The service salt-minion is already running
     Started: 14:21:49.201495
    Duration: 344.862 ms
     Changes:   
----------
          ID: salt_minion_sync_all
    Function: module.run
        Name: saltutil.sync_all
      Result: True
     Comment: State was not run because none of the onchanges reqs changed
     Started: 14:21:49.565616
    Duration: 0.076 ms
     Changes:   
----------
          ID: salt_minion_grains_dir
    Function: file.directory
        Name: /etc/salt/grains.d
      Result: True
     Comment: Directory /etc/salt/grains.d is in the correct state
              Directory /etc/salt/grains.d updated
     Started: 14:21:49.566111
    Duration: 2.843 ms
     Changes:   
----------
          ID: salt_minion_grains_files
    Function: file.managed
        Name: /etc/salt/grains
      Result: True
     Comment: File /etc/salt/grains exists with proper permissions. No changes made.
     Started: 14:21:49.570043
    Duration: 1.547 ms
     Changes:   
----------
          ID: salt_minion_grains_files
    Function: file.managed
        Name: /etc/salt/grains.d/placeholder
      Result: True
     Comment: File /etc/salt/grains.d/placeholder exists with proper permissions. No changes made.
     Started: 14:21:49.572244
    Duration: 6.31 ms
     Changes:   
----------
          ID: salt_minion_grain_linux_sphinx
    Function: file.managed
        Name: /etc/salt/grains.d/sphinx
      Result: True
     Comment: File /etc/salt/grains.d/sphinx is in the correct state
     Started: 14:21:49.579250
    Duration: 23.463 ms
     Changes:   
----------
          ID: salt_minion_grain_linux_sphinx_validity_check
    Function: cmd.wait
        Name: python -c "import yaml; stream = file('/etc/salt/grains.d/sphinx', 'r'); yaml.load(stream); stream.close()"
      Result: True
     Comment: 
     Started: 14:21:49.606848
    Duration: 1.463 ms
     Changes:   
----------
          ID: salt_minion_grain_linux_dns_records
    Function: file.managed
        Name: /etc/salt/grains.d/dns_records
      Result: True
     Comment: File /etc/salt/grains.d/dns_records is in the correct state
     Started: 14:21:49.633468
    Duration: 2.851 ms
     Changes:   
----------
          ID: salt_minion_grain_linux_dns_records_validity_check
    Function: cmd.wait
        Name: python -c "import yaml; stream = file('/etc/salt/grains.d/dns_records', 'r'); yaml.load(stream); stream.close()"
      Result: True
     Comment: 
     Started: 14:21:49.637802
    Duration: 1.236 ms
     Changes:   
----------
          ID: salt_minion_grain_salt_salt
    Function: file.managed
        Name: /etc/salt/grains.d/salt
      Result: True
     Comment: File /etc/salt/grains.d/salt is in the correct state
     Started: 14:21:49.639696
    Duration: 2.407 ms
     Changes:   
----------
          ID: salt_minion_grain_salt_salt_validity_check
    Function: cmd.wait
        Name: python -c "import yaml; stream = file('/etc/salt/grains.d/salt', 'r'); yaml.load(stream); stream.close()"
      Result: True
     Comment: 
     Started: 14:21:49.643417
    Duration: 1.253 ms
     Changes:   
----------
          ID: salt_minion_grains_file
    Function: cmd.wait
        Name: cat /etc/salt/grains.d/* > /etc/salt/grains
      Result: True
     Comment: 
     Started: 14:21:49.647408
    Duration: 6.175 ms
     Changes:   
----------
          ID: salt_minion_grains_publish
    Function: module.wait
        Name: mine.update
      Result: True
     Comment: 
     Started: 14:21:49.654707
    Duration: 1.101 ms
     Changes:   
----------
          ID: salt_ca_certificates_packages
    Function: pkg.installed
        Name: ca-certificates
      Result: True
     Comment: All specified packages are already installed
     Started: 14:21:49.656157
    Duration: 22.802 ms
     Changes:   
----------
          ID: salt_update_certificates
    Function: cmd.wait
        Name: update-ca-certificates
      Result: True
     Comment: 
     Started: 14:21:49.680948
    Duration: 1.378 ms
     Changes:   
----------
          ID: iptables_packages_v4
    Function: pkg.installed
        Name: iptables
      Result: True
     Comment: All specified packages are already installed
     Started: 14:21:49.682655
    Duration: 12.823 ms
     Changes:   
----------
          ID: iptables_packages_v4
    Function: pkg.installed
        Name: iptables-persistent
      Result: True
     Comment: All specified packages are already installed
     Started: 14:21:49.695868
    Duration: 18.189 ms
     Changes:   
----------
          ID: iptables_modules_v4_load
    Function: kmod.present
      Result: True
     Comment: Kernel modules iptable_filter, ip_tables are already present
     Started: 14:21:49.715916
    Duration: 47.577 ms
     Changes:   
----------
          ID: /etc/iptables/rules.v4
    Function: file.managed
      Result: True
     Comment: File /etc/iptables/rules.v4 is in the correct state
     Started: 14:21:49.764702
    Duration: 564.519 ms
     Changes:   
----------
          ID: iptables_services_v4_start
    Function: cmd.run
        Name: find /usr/share/netfilter-persistent/plugins.d/[0-9]*-ip4tables -exec {} start \;
      Result: True
     Comment: onlyif execution failed
     Started: 14:21:50.330947
    Duration: 95.593 ms
     Changes:   
----------
          ID: netfilter-persistent
    Function: service.running
      Result: True
     Comment: The service netfilter-persistent is already running
     Started: 14:21:50.428118
    Duration: 211.19 ms
     Changes:   
----------
          ID: iptables_tables_cleanup_v4
    Function: module.wait
        Name: iptables_extra.remove_stale_tables
      Result: True
     Comment: 
     Started: 14:21:50.644989
    Duration: 1.893 ms
     Changes:   
----------
          ID: iptables_services_v6_stop
    Function: cmd.run
        Name: find /usr/share/netfilter-persistent/plugins.d/[0-9]*-ip6tables -exec {} flush \;
      Result: True
     Comment: onlyif execution failed
     Started: 14:21:50.647246
    Duration: 85.07 ms
     Changes:   
----------
          ID: /etc/iptables/rules.v6
    Function: file.absent
      Result: True
     Comment: File /etc/iptables/rules.v6 is not present
     Started: 14:21:50.738870
    Duration: 2.206 ms
     Changes:   
----------
          ID: iptables_tables_flush_v6
    Function: module.wait
        Name: iptables_extra.flush_all
      Result: True
     Comment: 
     Started: 14:21:50.742289
    Duration: 1.308 ms
     Changes:   
----------
          ID: salt_minion_service_restart
    Function: cmd.run
        Name: while true; do salt-call saltutil.running|grep fun: && continue; salt-call --local service.restart salt-minion; break; done
      Result: True
     Comment: State was not run because none of the onchanges reqs changed
     Started: 14:21:50.745002
    Duration: 0.024 ms
     Changes:   

Summary for mas01.mcp-odl-noha.local
-------------
Succeeded: 31
Failed:     0
-------------
Total states run:     31
Total run time:    2.711 s
[wait_for] OK: salt -C 'mas01*' state.apply salt,iptables
mas01.mcp-odl-noha.local:
----------
          ID: maas_cluster_packages
    Function: pkg.installed
        Name: maas-rack-controller
      Result: True
     Comment: All specified packages are already installed
     Started: 14:22:06.797595
    Duration: 286.074 ms
     Changes:   
----------
          ID: maas_cluster_packages
    Function: pkg.installed
        Name: ipmitool
      Result: True
     Comment: All specified packages are already installed
     Started: 14:22:07.084250
    Duration: 17.421 ms
     Changes:   
----------
          ID: /etc/maas/rackd.conf
    Function: file.line
      Result: True
     Comment: No changes needed to be made
     Started: 14:22:07.107464
    Duration: 19.375 ms
     Changes:   
----------
          ID: /etc/maas/rackd_conf
    Function: file.managed
        Name: /etc/maas/rackd.conf
      Result: True
     Comment: File /etc/maas/rackd.conf exists with proper permissions. No changes made.
     Started: 14:22:07.127220
    Duration: 2.747 ms
     Changes:   
----------
          ID: maas_cluster_services
    Function: service.running
        Name: maas-rackd
      Result: True
     Comment: The service maas-rackd is already running
     Started: 14:22:07.131716
    Duration: 183.397 ms
     Changes:   

Summary for mas01.mcp-odl-noha.local
------------
Succeeded: 5
Failed:    0
------------
Total states run:     5
Total run time: 509.014 ms

[wait_for] Waiting for cmd to return success: salt -C 'mas01*' state.apply maas.region
[wait_for] Attempt 1/10 for: salt -C 'mas01*' state.apply maas.region
mas01.mcp-odl-noha.local:
----------
          ID: maas_region_packages
    Function: pkg.installed
        Name: maas-region-controller
      Result: True
     Comment: All specified packages are already installed
     Started: 14:22:13.309230
    Duration: 218.744 ms
     Changes:   
----------
          ID: maas_region_packages
    Function: pkg.installed
        Name: python-oauth
      Result: True
     Comment: All specified packages are already installed
     Started: 14:22:13.528495
    Duration: 18.341 ms
     Changes:   
----------
          ID: /etc/maas/regiond.conf
    Function: file.replace
      Result: True
     Comment: No changes needed to be made
     Started: 14:22:13.562710
    Duration: 10.305 ms
     Changes:   
----------
          ID: /usr/lib/python3/dist-packages/provisioningserver/templates/proxy/maas-proxy.conf.template
    Function: file.managed
      Result: True
     Comment: File /usr/lib/python3/dist-packages/provisioningserver/templates/proxy/maas-proxy.conf.template is in the correct state
     Started: 14:22:13.573770
    Duration: 273.294 ms
     Changes:   
----------
          ID: maas_timeout_commissioning
    Function: file.replace
        Name: /usr/lib/python3/dist-packages/maasserver/node_status.py
      Result: True
     Comment: No changes needed to be made
     Started: 14:22:13.847821
    Duration: 14.729 ms
     Changes:   
----------
          ID: maas_timeout_deploying
    Function: file.replace
        Name: /usr/lib/python3/dist-packages/maasserver/node_status.py
      Result: True
     Comment: No changes needed to be made
     Started: 14:22:13.863273
    Duration: 6.48 ms
     Changes:   
----------
          ID: maas_interface_default_mode_dhcp
    Function: file.replace
        Name: /usr/lib/python3/dist-packages/maasserver/models/node.py
      Result: True
     Comment: No changes needed to be made
     Started: 14:22:13.870587
    Duration: 72.705 ms
     Changes:   
----------
          ID: /etc/apache2/conf-enabled/maas-http.conf
    Function: file.managed
      Result: True
     Comment: File /etc/apache2/conf-enabled/maas-http.conf is in the correct state
     Started: 14:22:13.944292
    Duration: 65.169 ms
     Changes:   
----------
          ID: maas_apache_headers
    Function: cmd.run
        Name: a2enmod headers
      Result: True
     Comment: Command "a2enmod headers" run
     Started: 14:22:14.011586
    Duration: 174.771 ms
     Changes:   
              ----------
              pid:
                  19107
              retcode:
                  0
              stderr:
              stdout:
                  Module headers already enabled
----------
          ID: /usr/share/maas/web/static/css/maas-styles.css
    Function: file.managed
      Result: True
     Comment: File /usr/share/maas/web/static/css/maas-styles.css is in the correct state
     Started: 14:22:14.186874
    Duration: 110.945 ms
     Changes:   
----------
          ID: /etc/maas/preseeds/curtin_userdata_amd64_generic_trusty
    Function: file.managed
      Result: True
     Comment: File /etc/maas/preseeds/curtin_userdata_amd64_generic_trusty is in the correct state
     Started: 14:22:14.298720
    Duration: 262.693 ms
     Changes:   
----------
          ID: /etc/maas/preseeds/curtin_userdata_amd64_generic_xenial
    Function: file.managed
      Result: True
     Comment: File /etc/maas/preseeds/curtin_userdata_amd64_generic_xenial is in the correct state
     Started: 14:22:14.562235
    Duration: 147.635 ms
     Changes:   
----------
          ID: /etc/maas/preseeds/curtin_userdata_arm64_generic_xenial
    Function: file.managed
      Result: True
     Comment: File /etc/maas/preseeds/curtin_userdata_arm64_generic_xenial is in the correct state
     Started: 14:22:14.710621
    Duration: 311.804 ms
     Changes:   
----------
          ID: Configure /root/.pgpass for MAAS
    Function: file.managed
        Name: /root/.pgpass
      Result: True
     Comment: File /root/.pgpass is in the correct state
     Started: 14:22:15.022822
    Duration: 231.034 ms
     Changes:   
----------
          ID: maas_region_syncdb
    Function: cmd.run
        Name: maas-region syncdb --noinput
      Result: True
     Comment: Command "maas-region syncdb --noinput" run
     Started: 14:22:15.281407
    Duration: 8180.645 ms
     Changes:   
              ----------
              pid:
                  19120
              retcode:
                  0
              stderr:
              stdout:
                  Operations to perform:
                    Synchronize unmigrated apps: messages, staticfiles
                    Apply all migrations: contenttypes, auth, metadataserver, sessions, piston3, maasserver, sites
                  Synchronizing apps without migrations:
                    Creating tables...
                      Running deferred SQL...
                    Installing custom SQL...
                  Running migrations:
                    No migrations to apply.
----------
          ID: maas_region_services
    Function: service.running
        Name: maas-regiond
      Result: True
     Comment: The service maas-regiond is already running
     Started: 14:22:23.465538
    Duration: 309.734 ms
     Changes:   
----------
          ID: maas_region_services
    Function: service.running
        Name: bind9
      Result: True
     Comment: The service bind9 is already running
     Started: 14:22:23.778551
    Duration: 209.604 ms
     Changes:   
----------
          ID: maas_region_services
    Function: service.running
        Name: apache2
      Result: True
     Comment: The service apache2 is already running
     Started: 14:22:23.990678
    Duration: 286.957 ms
     Changes:   
----------
          ID: maas_warmup
    Function: module.run
        Name: maasng.wait_for_http_code
      Result: True
     Comment: Module function maasng.wait_for_http_code executed
     Started: 14:22:24.280880
    Duration: 22.768 ms
     Changes:   
              ----------
              ret:
                  ----------
                  comment:
                      MAAS API:http://localhost:5240/MAAS up.
                  result:
                      True
----------
          ID: maas_set_admin_password
    Function: cmd.run
        Name: maas createadmin --username opnfv --password opnfv_secret --email email@example.com && touch /var/lib/maas/.setup_admin
      Result: True
     Comment: /var/lib/maas/.setup_admin exists
     Started: 14:22:24.304960
    Duration: 1.489 ms
     Changes:   
----------
          ID: maas_login_admin
    Function: cmd.run
        Name: maas-region apikey --username opnfv > /var/lib/maas/.maas_credentials
      Result: True
     Comment: Command "maas-region apikey --username opnfv > /var/lib/maas/.maas_credentials" run
     Started: 14:22:24.307505
    Duration: 5388.748 ms
     Changes:   
              ----------
              pid:
                  19145
              retcode:
                  0
              stderr:
              stdout:
----------
          ID: maas_region_boot_source_resources_mirror
    Function: maasng.boot_source_present
      Result: True
     Comment: boot-source http://images.maas.io/ephemeral-v3/daily alredy exist
     Started: 14:22:29.739937
    Duration: 672.413 ms
     Changes:   
              ----------
              changes:
                  ----------
    Warnings: 'name' is an invalid keyword argument for
              'maasng.boot_source_present'. If you were trying to pass
              additional data to be used in a template context, please populate
              'context' with 'key: value' pairs. Your approach will work until
              Salt Fluorine is out. Please update your state files.
----------
          ID: maas_wait_for_region_import_done
    Function: module.run
        Name: maasng.boot_resources_import
      Result: True
     Comment: Module function maasng.boot_resources_import executed
     Started: 14:22:30.413607
    Duration: 32842.949 ms
     Changes:   
              ----------
              ret:
                  True
----------
          ID: maas_region_boot_sources_selection_xenial
    Function: maasng.boot_sources_selections_present
      Result: True
     Comment: Requested boot-source selection for http://images.maas.io/ephemeral-v3/daily already exist.
     Started: 14:23:03.258500
    Duration: 1363.403 ms
     Changes:   
    Warnings: 'name' is an invalid keyword argument for
              'maasng.boot_sources_selections_present'. If you were trying to
              pass additional data to be used in a template context, please
              populate 'context' with 'key: value' pairs. Your approach will
              work until Salt Fluorine is out. Please update your state files.
----------
          ID: maas_wait_for_racks_import_done
    Function: module.run
        Name: maasng.sync_and_wait_bs_to_all_racks
      Result: True
     Comment: Module function maasng.sync_and_wait_bs_to_all_racks executed
     Started: 14:23:04.623881
    Duration: 4062.792 ms
     Changes:   
              ----------
              ret:
                  True
----------
          ID: maas_config
    Function: module.run
        Name: maas.process_maas_config
      Result: True
     Comment: Module function maas.process_maas_config executed
     Started: 14:23:08.689997
    Duration: 10523.177 ms
     Changes:   
              ----------
              ret:
                  ----------
                  errors:
                      ----------
                  success:
                      - enable_http_proxy
                      - upstream_dns
                      - commissioning_distro_series
                      - default_osystem
                      - active_discovery_interval
                      - dnssec_validation
                      - maas_name
                      - network_discovery
                      - enable_third_party_drivers
                      - default_storage_layout
                      - ntp_external_only
                      - disk_erase_with_secure_erase
                      - default_distro_series
                      - default_min_hwe_kernel
                  updated:
----------
          ID: maas_fabrics_pxe_admin
    Function: maasng.fabric_present
        Name: pxe_admin
      Result: True
     Comment: Module function maasng.fabric_present executed
     Started: 14:23:19.214124
    Duration: 705.423 ms
     Changes:   
              ----------
              new:
                  Fabric  pxe_admin created
              result:
                  True
----------
          ID: maas_vlan0_present_for_pxe_admin
    Function: maasng.vlan_present_in_fabric
        Name: vlan 0
      Result: True
     Comment: Vlan vlan 0 has been updated for pxe_admin
     Started: 14:23:19.920394
    Duration: 2297.618 ms
     Changes:   
              ----------
              new:
                  Vlan untagged was updated
----------
          ID: maas_create_subnet_10.1.0.0/24
    Function: maasng.subnet_present
        Name: 10.1.0.0/24
      Result: True
     Comment: Subnet 10.1.0.0/24 has been updated for pxe_admin
     Started: 14:23:22.219470
    Duration: 1749.387 ms
     Changes:   
----------
          ID: maas_create_iprange_1
    Function: maasng.iprange_present
      Result: True
     Comment: Iprange maas_create_iprange_1 already exist.
     Started: 14:23:23.970563
    Duration: 507.341 ms
     Changes:   
----------
          ID: maas_vlan0_present_for_pxe_admin_dhcp
    Function: maasng.vlan_present_in_fabric
        Name: vlan 0
      Result: True
     Comment: Vlan vlan 0 has been updated for pxe_admin
     Started: 14:23:24.478314
    Duration: 2471.171 ms
     Changes:   
              ----------
              new:
                  Vlan untagged was updated
----------
          ID: maas_sshkey_1
    Function: maasng.sshkey_present
        Name: opnfv
      Result: True
     Comment: SSH key ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9iteXSOD1Du/fG2coTPKPKmlyjD5EHa0wwZm9pdya5BuWatdcIzkCgKLdLBST3LZZ6U8IB83LnCwKa7CX87OPxZVylK8W8NlzsjYt3P+ZOX8qUmhTxxkJa4i43aZXbQwv1a6QAwHY6n2/8mKdLpov5gf6Y28HpUN1d3mp3gegNK5quTNAxS2MzUXk9gOZ8ZXYLXCYjg4eWXXXCLxK+rmojXgsjdY3WruVLR+1YkXX38jJ0pirCp10knC+QGf1QTQE2quzYQNh8NQA+1weNUWi6xYd8c9Nt/bBdtoL56hqivJeNYiFw2PLeoGqfOtI1OrwR1Cp0WdoJ+cIdW9P/7ov already exist for user opnfv.
     Started: 14:23:26.950839
    Duration: 73.794 ms
     Changes:   

Summary for mas01.mcp-odl-noha.local
-------------
Succeeded: 32 (changed=11)
Failed:     0
Warnings:   2
-------------
Total states run:     32
Total run time:   73.588 s
[wait_for] OK: salt -C 'mas01*' state.apply maas.region
mas01.mcp-odl-noha.local:
----------
          ID: maas_login_admin
    Function: cmd.run
        Name: maas-region apikey --username opnfv > /var/lib/maas/.maas_credentials
      Result: True
     Comment: Command "maas-region apikey --username opnfv > /var/lib/maas/.maas_credentials" run
     Started: 14:23:40.620839
    Duration: 4814.04 ms
     Changes:   
              ----------
              pid:
                  19544
              retcode:
                  0
              stderr:
              stdout:
----------
          ID: create__machines
    Function: module.run
        Name: maas.process_machines
      Result: True
     Comment: Module function maas.process_machines executed
     Started: 14:23:45.439117
    Duration: 18726.732 ms
     Changes:   
              ----------
              ret:
                  ----------
                  errors:
                      ----------
                  success:
                  updated:
                      - cmp002
                      - cmp003
                      - cmp001
                      - odl01
                      - gtw01
                      - ctl01

Summary for mas01.mcp-odl-noha.local
------------
Succeeded: 2 (changed=2)
Failed:    0
------------
Total states run:     2
Total run time:  23.541 s
[NOTE] MaaS operations might take a long time, please be patient
tput: unknown terminal "unknown"
tput: unknown terminal "unknown"
mas01.mcp-odl-noha.local:
----------
          ID: maas_login_admin
    Function: cmd.run
        Name: maas-region apikey --username opnfv > /var/lib/maas/.maas_credentials
      Result: True
     Comment: Command "maas-region apikey --username opnfv > /var/lib/maas/.maas_credentials" run
     Started: 14:24:48.946275
    Duration: 5583.197 ms
     Changes:   
              ----------
              pid:
                  19882
              retcode:
                  0
              stderr:
              stdout:
----------
          ID: wait_for_machines_ready_or_deployed
    Function: module.run
        Name: maas.wait_for_machine_status
      Result: False
     Comment: Module function maas.wait_for_machine_status threw an exception. Exception: Machines:['cmp002', 'cmp003', 'cmp001', 'odl01', 'gtw01', 'ctl01']not in Ready|Deployed state
     Started: 14:24:54.533408
    Duration: 1528562.868 ms
     Changes:   

Summary for mas01.mcp-odl-noha.local
------------
Succeeded: 1 (changed=1)
Failed:    1
------------
Total states run:     2
Total run time: 1534.146 s
ERROR: Minions returned with non-zero exit code
[wait_for] Attempt 4/5 for: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i /var/lib/opnfv/mcp.rsa ubuntu@10.20.0.2 sudo             CI_DEBUG=false ERASE_ENV=0             /root/fuel/mcp/config/states/maas
Warning: Permanently added '10.20.0.2' (ECDSA) to the list of known hosts.

[wait_for] Waiting for cmd to return success: salt --out yaml -C 'mas01*' service.status maas-fixup | fgrep -q 'false'
[wait_for] Attempt 1/60 for: salt --out yaml -C 'mas01*' service.status maas-fixup | fgrep -q 'false'
[wait_for] OK: salt --out yaml -C 'mas01*' service.status maas-fixup | fgrep -q 'false'

[wait_for] Waiting for cmd to return success: salt -C 'mas01*' state.apply salt,iptables
[wait_for] Attempt 1/10 for: salt -C 'mas01*' state.apply salt,iptables
mas01.mcp-odl-noha.local:
----------
          ID: salt_minion_packages
    Function: pkg.installed
        Name: salt-minion
      Result: True
     Comment: All specified packages are already installed
     Started: 14:51:01.977075
    Duration: 321.168 ms
     Changes:   
----------
          ID: salt_minion_dependency_packages
    Function: pkg.installed
      Result: True
     Comment: All specified packages are already installed
     Started: 14:51:02.298826
    Duration: 23.681 ms
     Changes:   
----------
          ID: /etc/salt/minion.d/minion.conf
    Function: file.managed
      Result: True
     Comment: File /etc/salt/minion.d/minion.conf is in the correct state
     Started: 14:51:02.333660
    Duration: 778.912 ms
     Changes:   
----------
          ID: salt_minion_systemd_override
    Function: file.managed
        Name: /etc/systemd/system/salt-minion.service.d/50-restarts.conf
      Result: True
     Comment: File /etc/systemd/system/salt-minion.service.d/50-restarts.conf is in the correct state
     Started: 14:51:03.118231
    Duration: 97.53 ms
     Changes:   
----------
          ID: salt_minion_systemd_reload
    Function: module.wait
        Name: service.systemctl_reload
      Result: True
     Comment: State was not run because none of the onchanges reqs changed
     Started: 14:51:03.216478
    Duration: 0.017 ms
     Changes:   
----------
          ID: salt_minion_service
    Function: service.running
        Name: salt-minion
      Result: True
     Comment: The service salt-minion is already running
     Started: 14:51:03.217332
    Duration: 193.859 ms
     Changes:   
----------
          ID: salt_minion_sync_all
    Function: module.run
        Name: saltutil.sync_all
      Result: True
     Comment: State was not run because none of the onchanges reqs changed
     Started: 14:51:03.413693
    Duration: 0.034 ms
     Changes:   
----------
          ID: salt_minion_grains_dir
    Function: file.directory
        Name: /etc/salt/grains.d
      Result: True
     Comment: Directory /etc/salt/grains.d is in the correct state
              Directory /etc/salt/grains.d updated
     Started: 14:51:03.414102
    Duration: 4.93 ms
     Changes:   
----------
          ID: salt_minion_grains_files
    Function: file.managed
        Name: /etc/salt/grains
      Result: True
     Comment: File /etc/salt/grains exists with proper permissions. No changes made.
     Started: 14:51:03.424872
    Duration: 8.424 ms
     Changes:   
----------
          ID: salt_minion_grains_files
    Function: file.managed
        Name: /etc/salt/grains.d/placeholder
      Result: True
     Comment: File /etc/salt/grains.d/placeholder exists with proper permissions. No changes made.
     Started: 14:51:03.435014
    Duration: 3.266 ms
     Changes:   
----------
          ID: salt_minion_grain_linux_sphinx
    Function: file.managed
        Name: /etc/salt/grains.d/sphinx
      Result: True
     Comment: File /etc/salt/grains.d/sphinx is in the correct state
     Started: 14:51:03.439455
    Duration: 4.65 ms
     Changes:   
----------
          ID: salt_minion_grain_linux_sphinx_validity_check
    Function: cmd.wait
        Name: python -c "import yaml; stream = file('/etc/salt/grains.d/sphinx', 'r'); yaml.load(stream); stream.close()"
      Result: True
     Comment: 
     Started: 14:51:03.452662
    Duration: 3.526 ms
     Changes:   
----------
          ID: salt_minion_grain_linux_dns_records
    Function: file.managed
        Name: /etc/salt/grains.d/dns_records
      Result: True
     Comment: File /etc/salt/grains.d/dns_records is in the correct state
     Started: 14:51:03.457524
    Duration: 4.303 ms
     Changes:   
----------
          ID: salt_minion_grain_linux_dns_records_validity_check
    Function: cmd.wait
        Name: python -c "import yaml; stream = file('/etc/salt/grains.d/dns_records', 'r'); yaml.load(stream); stream.close()"
      Result: True
     Comment: 
     Started: 14:51:03.463823
    Duration: 3.048 ms
     Changes:   
----------
          ID: salt_minion_grain_salt_salt
    Function: file.managed
        Name: /etc/salt/grains.d/salt
      Result: True
     Comment: File /etc/salt/grains.d/salt is in the correct state
     Started: 14:51:03.468063
    Duration: 10.6 ms
     Changes:   
----------
          ID: salt_minion_grain_salt_salt_validity_check
    Function: cmd.wait
        Name: python -c "import yaml; stream = file('/etc/salt/grains.d/salt', 'r'); yaml.load(stream); stream.close()"
      Result: True
     Comment: 
     Started: 14:51:03.480704
    Duration: 3.103 ms
     Changes:   
----------
          ID: salt_minion_grains_file
    Function: cmd.wait
        Name: cat /etc/salt/grains.d/* > /etc/salt/grains
      Result: True
     Comment: 
     Started: 14:51:03.487271
    Duration: 3.31 ms
     Changes:   
----------
          ID: salt_minion_grains_publish
    Function: module.wait
        Name: mine.update
      Result: True
     Comment: 
     Started: 14:51:03.492134
    Duration: 2.772 ms
     Changes:   
----------
          ID: salt_ca_certificates_packages
    Function: pkg.installed
        Name: ca-certificates
      Result: True
     Comment: All specified packages are already installed
     Started: 14:51:03.495844
    Duration: 25.74 ms
     Changes:   
----------
          ID: salt_update_certificates
    Function: cmd.wait
        Name: update-ca-certificates
      Result: True
     Comment: 
     Started: 14:51:03.524046
    Duration: 3.316 ms
     Changes:   
----------
          ID: iptables_packages_v4
    Function: pkg.installed
        Name: iptables
      Result: True
     Comment: All specified packages are already installed
     Started: 14:51:03.528139
    Duration: 35.342 ms
     Changes:   
----------
          ID: iptables_packages_v4
    Function: pkg.installed
        Name: iptables-persistent
      Result: True
     Comment: All specified packages are already installed
     Started: 14:51:03.564354
    Duration: 24.944 ms
     Changes:   
----------
          ID: iptables_modules_v4_load
    Function: kmod.present
      Result: True
     Comment: Kernel modules iptable_filter, ip_tables are already present
     Started: 14:51:03.591814
    Duration: 80.19 ms
     Changes:   
----------
          ID: /etc/iptables/rules.v4
    Function: file.managed
      Result: True
     Comment: File /etc/iptables/rules.v4 is in the correct state
     Started: 14:51:03.673735
    Duration: 401.666 ms
     Changes:   
----------
          ID: iptables_services_v4_start
    Function: cmd.run
        Name: find /usr/share/netfilter-persistent/plugins.d/[0-9]*-ip4tables -exec {} start \;
      Result: True
     Comment: onlyif execution failed
     Started: 14:51:04.077668
    Duration: 97.507 ms
     Changes:   
----------
          ID: netfilter-persistent
    Function: service.running
      Result: True
     Comment: The service netfilter-persistent is already running
     Started: 14:51:04.177342
    Duration: 201.835 ms
     Changes:   
----------
          ID: iptables_tables_cleanup_v4
    Function: module.wait
        Name: iptables_extra.remove_stale_tables
      Result: True
     Comment: 
     Started: 14:51:04.381081
    Duration: 3.064 ms
     Changes:   
----------
          ID: iptables_services_v6_stop
    Function: cmd.run
        Name: find /usr/share/netfilter-persistent/plugins.d/[0-9]*-ip6tables -exec {} flush \;
      Result: True
     Comment: onlyif execution failed
     Started: 14:51:04.384992
    Duration: 82.33 ms
     Changes:   
----------
          ID: /etc/iptables/rules.v6
    Function: file.absent
      Result: True
     Comment: File /etc/iptables/rules.v6 is not present
     Started: 14:51:04.469582
    Duration: 3.528 ms
     Changes:   
----------
          ID: iptables_tables_flush_v6
    Function: module.wait
        Name: iptables_extra.flush_all
      Result: True
     Comment: 
     Started: 14:51:04.474707
    Duration: 7.742 ms
     Changes:   
----------
          ID: salt_minion_service_restart
    Function: cmd.run
        Name: while true; do salt-call saltutil.running|grep fun: && continue; salt-call --local service.restart salt-minion; break; done
      Result: True
     Comment: State was not run because none of the onchanges reqs changed
     Started: 14:51:04.484460
    Duration: 0.03 ms
     Changes:   

Summary for mas01.mcp-odl-noha.local
-------------
Succeeded: 31
Failed:     0
-------------
Total states run:     31
Total run time:    2.434 s
[wait_for] OK: salt -C 'mas01*' state.apply salt,iptables
mas01.mcp-odl-noha.local:
----------
          ID: maas_cluster_packages
    Function: pkg.installed
        Name: maas-rack-controller
      Result: True
     Comment: All specified packages are already installed
     Started: 14:51:18.827689
    Duration: 233.122 ms
     Changes:   
----------
          ID: maas_cluster_packages
    Function: pkg.installed
        Name: ipmitool
      Result: True
     Comment: All specified packages are already installed
     Started: 14:51:19.061362
    Duration: 13.9 ms
     Changes:   
----------
          ID: /etc/maas/rackd.conf
    Function: file.line
      Result: True
     Comment: No changes needed to be made
     Started: 14:51:19.081139
    Duration: 2.935 ms
     Changes:   
----------
          ID: /etc/maas/rackd_conf
    Function: file.managed
        Name: /etc/maas/rackd.conf
      Result: True
     Comment: File /etc/maas/rackd.conf exists with proper permissions. No changes made.
     Started: 14:51:19.084421
    Duration: 2.608 ms
     Changes:   
----------
          ID: maas_cluster_services
    Function: service.running
        Name: maas-rackd
      Result: True
     Comment: The service maas-rackd is already running
     Started: 14:51:19.088776
    Duration: 191.918 ms
     Changes:   

Summary for mas01.mcp-odl-noha.local
------------
Succeeded: 5
Failed:    0
------------
Total states run:     5
Total run time: 444.483 ms

[wait_for] Waiting for cmd to return success: salt -C 'mas01*' state.apply maas.region
[wait_for] Attempt 1/10 for: salt -C 'mas01*' state.apply maas.region
mas01.mcp-odl-noha.local:
----------
          ID: maas_region_packages
    Function: pkg.installed
        Name: maas-region-controller
      Result: True
     Comment: All specified packages are already installed
     Started: 14:51:25.722065
    Duration: 274.578 ms
     Changes:   
----------
          ID: maas_region_packages
    Function: pkg.installed
        Name: python-oauth
      Result: True
     Comment: All specified packages are already installed
     Started: 14:51:25.997217
    Duration: 14.376 ms
     Changes:   
----------
          ID: /etc/maas/regiond.conf
    Function: file.replace
      Result: True
     Comment: No changes needed to be made
     Started: 14:51:26.017432
    Duration: 10.902 ms
     Changes:   
----------
          ID: /usr/lib/python3/dist-packages/provisioningserver/templates/proxy/maas-proxy.conf.template
    Function: file.managed
      Result: True
     Comment: File /usr/lib/python3/dist-packages/provisioningserver/templates/proxy/maas-proxy.conf.template is in the correct state
     Started: 14:51:26.029158
    Duration: 141.986 ms
     Changes:   
----------
          ID: maas_timeout_commissioning
    Function: file.replace
        Name: /usr/lib/python3/dist-packages/maasserver/node_status.py
      Result: True
     Comment: No changes needed to be made
     Started: 14:51:26.171892
    Duration: 6.709 ms
     Changes:   
----------
          ID: maas_timeout_deploying
    Function: file.replace
        Name: /usr/lib/python3/dist-packages/maasserver/node_status.py
      Result: True
     Comment: No changes needed to be made
     Started: 14:51:26.179355
    Duration: 12.565 ms
     Changes:   
----------
          ID: maas_interface_default_mode_dhcp
    Function: file.replace
        Name: /usr/lib/python3/dist-packages/maasserver/models/node.py
      Result: True
     Comment: No changes needed to be made
     Started: 14:51:26.192726
    Duration: 165.597 ms
     Changes:   
----------
          ID: /etc/apache2/conf-enabled/maas-http.conf
    Function: file.managed
      Result: True
     Comment: File /etc/apache2/conf-enabled/maas-http.conf is in the correct state
     Started: 14:51:26.359083
    Duration: 96.578 ms
     Changes:   
----------
          ID: maas_apache_headers
    Function: cmd.run
        Name: a2enmod headers
      Result: True
     Comment: Command "a2enmod headers" run
     Started: 14:51:26.457849
    Duration: 171.917 ms
     Changes:   
              ----------
              pid:
                  26774
              retcode:
                  0
              stderr:
              stdout:
                  Module headers already enabled
----------
          ID: /usr/share/maas/web/static/css/maas-styles.css
    Function: file.managed
      Result: True
     Comment: File /usr/share/maas/web/static/css/maas-styles.css is in the correct state
     Started: 14:51:26.630330
    Duration: 78.01 ms
     Changes:   
----------
          ID: /etc/maas/preseeds/curtin_userdata_amd64_generic_trusty
    Function: file.managed
      Result: True
     Comment: File /etc/maas/preseeds/curtin_userdata_amd64_generic_trusty is in the correct state
     Started: 14:51:26.713695
    Duration: 161.225 ms
     Changes:   
----------
          ID: /etc/maas/preseeds/curtin_userdata_amd64_generic_xenial
    Function: file.managed
      Result: True
     Comment: File /etc/maas/preseeds/curtin_userdata_amd64_generic_xenial is in the correct state
     Started: 14:51:26.875651
    Duration: 195.313 ms
     Changes:   
----------
          ID: /etc/maas/preseeds/curtin_userdata_arm64_generic_xenial
    Function: file.managed
      Result: True
     Comment: File /etc/maas/preseeds/curtin_userdata_arm64_generic_xenial is in the correct state
     Started: 14:51:27.071703
    Duration: 306.821 ms
     Changes:   
----------
          ID: Configure /root/.pgpass for MAAS
    Function: file.managed
        Name: /root/.pgpass
      Result: True
     Comment: File /root/.pgpass is in the correct state
     Started: 14:51:27.378949
    Duration: 295.355 ms
     Changes:   
----------
          ID: maas_region_syncdb
    Function: cmd.run
        Name: maas-region syncdb --noinput
      Result: True
     Comment: Command "maas-region syncdb --noinput" run
     Started: 14:51:27.681441
    Duration: 7705.607 ms
     Changes:   
              ----------
              pid:
                  26787
              retcode:
                  0
              stderr:
              stdout:
                  Operations to perform:
                    Synchronize unmigrated apps: staticfiles, messages
                    Apply all migrations: auth, sites, metadataserver, maasserver, piston3, sessions, contenttypes
                  Synchronizing apps without migrations:
                    Creating tables...
                      Running deferred SQL...
                    Installing custom SQL...
                  Running migrations:
                    No migrations to apply.
----------
          ID: maas_region_services
    Function: service.running
        Name: maas-regiond
      Result: True
     Comment: The service maas-regiond is already running
     Started: 14:51:35.390023
    Duration: 289.056 ms
     Changes:   
----------
          ID: maas_region_services
    Function: service.running
        Name: bind9
      Result: True
     Comment: The service bind9 is already running
     Started: 14:51:35.682212
    Duration: 240.219 ms
     Changes:   
----------
          ID: maas_region_services
    Function: service.running
        Name: apache2
      Result: True
     Comment: The service apache2 is already running
     Started: 14:51:35.924932
    Duration: 266.526 ms
     Changes:   
----------
          ID: maas_warmup
    Function: module.run
        Name: maasng.wait_for_http_code
      Result: True
     Comment: Module function maasng.wait_for_http_code executed
     Started: 14:51:36.193638
    Duration: 44.143 ms
     Changes:   
              ----------
              ret:
                  ----------
                  comment:
                      MAAS API:http://localhost:5240/MAAS up.
                  result:
                      True
----------
          ID: maas_set_admin_password
    Function: cmd.run
        Name: maas createadmin --username opnfv --password opnfv_secret --email email@example.com && touch /var/lib/maas/.setup_admin
      Result: True
     Comment: /var/lib/maas/.setup_admin exists
     Started: 14:51:36.239025
    Duration: 1.415 ms
     Changes:   
----------
          ID: maas_login_admin
    Function: cmd.run
        Name: maas-region apikey --username opnfv > /var/lib/maas/.maas_credentials
      Result: True
     Comment: Command "maas-region apikey --username opnfv > /var/lib/maas/.maas_credentials" run
     Started: 14:51:36.241514
    Duration: 5436.954 ms
     Changes:   
              ----------
              pid:
                  26831
              retcode:
                  0
              stderr:
              stdout:
----------
          ID: maas_region_boot_source_resources_mirror
    Function: maasng.boot_source_present
      Result: True
     Comment: boot-source http://images.maas.io/ephemeral-v3/daily alredy exist
     Started: 14:51:41.739236
    Duration: 351.393 ms
     Changes:   
              ----------
              changes:
                  ----------
    Warnings: 'name' is an invalid keyword argument for
              'maasng.boot_source_present'. If you were trying to pass
              additional data to be used in a template context, please populate
              'context' with 'key: value' pairs. Your approach will work until
              Salt Fluorine is out. Please update your state files.
----------
          ID: maas_wait_for_region_import_done
    Function: module.run
        Name: maasng.boot_resources_import
      Result: True
     Comment: Module function maasng.boot_resources_import executed
     Started: 14:51:42.091558
    Duration: 26003.235 ms
     Changes:   
              ----------
              ret:
                  True
----------
          ID: maas_region_boot_sources_selection_xenial
    Function: maasng.boot_sources_selections_present
      Result: True
     Comment: Requested boot-source selection for http://images.maas.io/ephemeral-v3/daily already exist.
     Started: 14:52:08.097522
    Duration: 490.875 ms
     Changes:   
    Warnings: 'name' is an invalid keyword argument for
              'maasng.boot_sources_selections_present'. If you were trying to
              pass additional data to be used in a template context, please
              populate 'context' with 'key: value' pairs. Your approach will
              work until Salt Fluorine is out. Please update your state files.
----------
          ID: maas_wait_for_racks_import_done
    Function: module.run
        Name: maasng.sync_and_wait_bs_to_all_racks
      Result: True
     Comment: Module function maasng.sync_and_wait_bs_to_all_racks executed
     Started: 14:52:08.589580
    Duration: 6226.201 ms
     Changes:   
              ----------
              ret:
                  True
----------
          ID: maas_config
    Function: module.run
        Name: maas.process_maas_config
      Result: True
     Comment: Module function maas.process_maas_config executed
     Started: 14:52:14.817880
    Duration: 10843.85 ms
     Changes:   
              ----------
              ret:
                  ----------
                  errors:
                      ----------
                  success:
                      - enable_http_proxy
                      - upstream_dns
                      - commissioning_distro_series
                      - default_osystem
                      - active_discovery_interval
                      - dnssec_validation
                      - maas_name
                      - network_discovery
                      - enable_third_party_drivers
                      - default_storage_layout
                      - ntp_external_only
                      - disk_erase_with_secure_erase
                      - default_distro_series
                      - default_min_hwe_kernel
                  updated:
----------
          ID: maas_fabrics_pxe_admin
    Function: maasng.fabric_present
        Name: pxe_admin
      Result: True
     Comment: Module function maasng.fabric_present executed
     Started: 14:52:25.664154
    Duration: 770.646 ms
     Changes:   
              ----------
              new:
                  Fabric  pxe_admin created
              result:
                  True
----------
          ID: maas_vlan0_present_for_pxe_admin
    Function: maasng.vlan_present_in_fabric
        Name: vlan 0
      Result: True
     Comment: Vlan vlan 0 has been updated for pxe_admin
     Started: 14:52:26.435863
    Duration: 2338.548 ms
     Changes:   
              ----------
              new:
                  Vlan untagged was updated
----------
          ID: maas_create_subnet_10.1.0.0/24
    Function: maasng.subnet_present
        Name: 10.1.0.0/24
      Result: True
     Comment: Subnet 10.1.0.0/24 has been updated for pxe_admin
     Started: 14:52:28.777281
    Duration: 1738.036 ms
     Changes:   
----------
          ID: maas_create_iprange_1
    Function: maasng.iprange_present
      Result: True
     Comment: Iprange maas_create_iprange_1 already exist.
     Started: 14:52:30.516473
    Duration: 293.857 ms
     Changes:   
----------
          ID: maas_vlan0_present_for_pxe_admin_dhcp
    Function: maasng.vlan_present_in_fabric
        Name: vlan 0
      Result: True
     Comment: Vlan vlan 0 has been updated for pxe_admin
     Started: 14:52:30.810755
    Duration: 2191.74 ms
     Changes:   
              ----------
              new:
                  Vlan untagged was updated
----------
          ID: maas_sshkey_1
    Function: maasng.sshkey_present
        Name: opnfv
      Result: True
     Comment: SSH key ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9iteXSOD1Du/fG2coTPKPKmlyjD5EHa0wwZm9pdya5BuWatdcIzkCgKLdLBST3LZZ6U8IB83LnCwKa7CX87OPxZVylK8W8NlzsjYt3P+ZOX8qUmhTxxkJa4i43aZXbQwv1a6QAwHY6n2/8mKdLpov5gf6Y28HpUN1d3mp3gegNK5quTNAxS2MzUXk9gOZ8ZXYLXCYjg4eWXXXCLxK+rmojXgsjdY3WruVLR+1YkXX38jJ0pirCp10knC+QGf1QTQE2quzYQNh8NQA+1weNUWi6xYd8c9Nt/bBdtoL56hqivJeNYiFw2PLeoGqfOtI1OrwR1Cp0WdoJ+cIdW9P/7ov already exist for user opnfv.
     Started: 14:52:33.004247
    Duration: 229.939 ms
     Changes:   

Summary for mas01.mcp-odl-noha.local
-------------
Succeeded: 32 (changed=11)
Failed:     0
Warnings:   2
-------------
Total states run:     32
Total run time:   67.394 s
[wait_for] OK: salt -C 'mas01*' state.apply maas.region
mas01.mcp-odl-noha.local:
----------
          ID: maas_login_admin
    Function: cmd.run
        Name: maas-region apikey --username opnfv > /var/lib/maas/.maas_credentials
      Result: True
     Comment: Command "maas-region apikey --username opnfv > /var/lib/maas/.maas_credentials" run
     Started: 14:52:46.524407
    Duration: 5929.517 ms
     Changes:   
              ----------
              pid:
                  27303
              retcode:
                  0
              stderr:
              stdout:
----------
          ID: create__machines
    Function: module.run
        Name: maas.process_machines
      Result: True
     Comment: Module function maas.process_machines executed
     Started: 14:52:52.465092
    Duration: 18121.368 ms
     Changes:   
              ----------
              ret:
                  ----------
                  errors:
                      ----------
                  success:
                  updated:
                      - cmp002
                      - cmp003
                      - cmp001
                      - odl01
                      - gtw01
                      - ctl01

Summary for mas01.mcp-odl-noha.local
------------
Succeeded: 2 (changed=2)
Failed:    0
------------
Total states run:     2
Total run time:  24.051 s
[NOTE] MaaS operations might take a long time, please be patient
tput: unknown terminal "unknown"
tput: unknown terminal "unknown"
mas01.mcp-odl-noha.local:
----------
          ID: maas_login_admin
    Function: cmd.run
        Name: maas-region apikey --username opnfv > /var/lib/maas/.maas_credentials
      Result: True
     Comment: Command "maas-region apikey --username opnfv > /var/lib/maas/.maas_credentials" run
     Started: 14:53:54.816908
    Duration: 5912.517 ms
     Changes:   
              ----------
              pid:
                  27649
              retcode:
                  0
              stderr:
              stdout:
----------
          ID: wait_for_machines_ready_or_deployed
    Function: module.run
        Name: maas.wait_for_machine_status
      Result: False
     Comment: Module function maas.wait_for_machine_status threw an exception. Exception: Machines:['cmp002', 'cmp003', 'cmp001', 'odl01', 'gtw01', 'ctl01']not in Ready|Deployed state
     Started: 14:54:00.757535
    Duration: 1526081.62 ms
     Changes:   

Summary for mas01.mcp-odl-noha.local
------------
Succeeded: 1 (changed=1)
Failed:    1
------------
Total states run:     2
Total run time: 1531.994 s
ERROR: Minions returned with non-zero exit code
[wait_for] Attempt 5/5 for: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i /var/lib/opnfv/mcp.rsa ubuntu@10.20.0.2 sudo             CI_DEBUG=false ERASE_ENV=0             /root/fuel/mcp/config/states/maas
Warning: Permanently added '10.20.0.2' (ECDSA) to the list of known hosts.

[wait_for] Waiting for cmd to return success: salt --out yaml -C 'mas01*' service.status maas-fixup | fgrep -q 'false'
[wait_for] Attempt 1/60 for: salt --out yaml -C 'mas01*' service.status maas-fixup | fgrep -q 'false'
[wait_for] OK: salt --out yaml -C 'mas01*' service.status maas-fixup | fgrep -q 'false'

[wait_for] Waiting for cmd to return success: salt -C 'mas01*' state.apply salt,iptables
[wait_for] Attempt 1/10 for: salt -C 'mas01*' state.apply salt,iptables
mas01.mcp-odl-noha.local:
----------
          ID: salt_minion_packages
    Function: pkg.installed
        Name: salt-minion
      Result: True
     Comment: All specified packages are already installed
     Started: 15:20:09.671422
    Duration: 383.78 ms
     Changes:   
----------
          ID: salt_minion_dependency_packages
    Function: pkg.installed
      Result: True
     Comment: All specified packages are already installed
     Started: 15:20:10.055779
    Duration: 14.188 ms
     Changes:   
----------
          ID: /etc/salt/minion.d/minion.conf
    Function: file.managed
      Result: True
     Comment: File /etc/salt/minion.d/minion.conf is in the correct state
     Started: 15:20:10.077493
    Duration: 1353.471 ms
     Changes:   
----------
          ID: salt_minion_systemd_override
    Function: file.managed
        Name: /etc/systemd/system/salt-minion.service.d/50-restarts.conf
      Result: True
     Comment: File /etc/systemd/system/salt-minion.service.d/50-restarts.conf is in the correct state
     Started: 15:20:11.451361
    Duration: 168.406 ms
     Changes:   
----------
          ID: salt_minion_systemd_reload
    Function: module.wait
        Name: service.systemctl_reload
      Result: True
     Comment: State was not run because none of the onchanges reqs changed
     Started: 15:20:11.620502
    Duration: 0.05 ms
     Changes:   
----------
          ID: salt_minion_service
    Function: service.running
        Name: salt-minion
      Result: True
     Comment: The service salt-minion is already running
     Started: 15:20:11.621290
    Duration: 285.748 ms
     Changes:   
----------
          ID: salt_minion_sync_all
    Function: module.run
        Name: saltutil.sync_all
      Result: True
     Comment: State was not run because none of the onchanges reqs changed
     Started: 15:20:11.908989
    Duration: 0.028 ms
     Changes:   
----------
          ID: salt_minion_grains_dir
    Function: file.directory
        Name: /etc/salt/grains.d
      Result: True
     Comment: Directory /etc/salt/grains.d is in the correct state
              Directory /etc/salt/grains.d updated
     Started: 15:20:11.909384
    Duration: 2.421 ms
     Changes:   
----------
          ID: salt_minion_grains_files
    Function: file.managed
        Name: /etc/salt/grains
      Result: True
     Comment: File /etc/salt/grains exists with proper permissions. No changes made.
     Started: 15:20:11.917212
    Duration: 2.087 ms
     Changes:   
----------
          ID: salt_minion_grains_files
    Function: file.managed
        Name: /etc/salt/grains.d/placeholder
      Result: True
     Comment: File /etc/salt/grains.d/placeholder exists with proper permissions. No changes made.
     Started: 15:20:11.919959
    Duration: 22.38 ms
     Changes:   
----------
          ID: salt_minion_grain_linux_sphinx
    Function: file.managed
        Name: /etc/salt/grains.d/sphinx
      Result: True
     Comment: File /etc/salt/grains.d/sphinx is in the correct state
     Started: 15:20:11.951972
    Duration: 25.057 ms
     Changes:   
----------
          ID: salt_minion_grain_linux_sphinx_validity_check
    Function: cmd.wait
        Name: python -c "import yaml; stream = file('/etc/salt/grains.d/sphinx', 'r'); yaml.load(stream); stream.close()"
      Result: True
     Comment: 
     Started: 15:20:11.981834
    Duration: 1.492 ms
     Changes:   
----------
          ID: salt_minion_grain_linux_dns_records
    Function: file.managed
        Name: /etc/salt/grains.d/dns_records
      Result: True
     Comment: File /etc/salt/grains.d/dns_records is in the correct state
     Started: 15:20:11.984015
    Duration: 23.272 ms
     Changes:   
----------
          ID: salt_minion_grain_linux_dns_records_validity_check
    Function: cmd.wait
        Name: python -c "import yaml; stream = file('/etc/salt/grains.d/dns_records', 'r'); yaml.load(stream); stream.close()"
      Result: True
     Comment: 
     Started: 15:20:12.008750
    Duration: 1.228 ms
     Changes:   
----------
          ID: salt_minion_grain_salt_salt
    Function: file.managed
        Name: /etc/salt/grains.d/salt
      Result: True
     Comment: File /etc/salt/grains.d/salt is in the correct state
     Started: 15:20:12.010630
    Duration: 26.957 ms
     Changes:   
----------
          ID: salt_minion_grain_salt_salt_validity_check
    Function: cmd.wait
        Name: python -c "import yaml; stream = file('/etc/salt/grains.d/salt', 'r'); yaml.load(stream); stream.close()"
      Result: True
     Comment: 
     Started: 15:20:12.038981
    Duration: 1.198 ms
     Changes:   
----------
          ID: salt_minion_grains_file
    Function: cmd.wait
        Name: cat /etc/salt/grains.d/* > /etc/salt/grains
      Result: True
     Comment: 
     Started: 15:20:12.042914
    Duration: 1.444 ms
     Changes:   
----------
          ID: salt_minion_grains_publish
    Function: module.wait
        Name: mine.update
      Result: True
     Comment: 
     Started: 15:20:12.066118
    Duration: 1.133 ms
     Changes:   
----------
          ID: salt_ca_certificates_packages
    Function: pkg.installed
        Name: ca-certificates
      Result: True
     Comment: All specified packages are already installed
     Started: 15:20:12.067569
    Duration: 33.942 ms
     Changes:   
----------
          ID: salt_update_certificates
    Function: cmd.wait
        Name: update-ca-certificates
      Result: True
     Comment: 
     Started: 15:20:12.103268
    Duration: 1.303 ms
     Changes:   
----------
          ID: iptables_packages_v4
    Function: pkg.installed
        Name: iptables
      Result: True
     Comment: All specified packages are already installed
     Started: 15:20:12.104939
    Duration: 38.806 ms
     Changes:   
----------
          ID: iptables_packages_v4
    Function: pkg.installed
        Name: iptables-persistent
      Result: True
     Comment: All specified packages are already installed
     Started: 15:20:12.144122
    Duration: 18.783 ms
     Changes:   
----------
          ID: iptables_modules_v4_load
    Function: kmod.present
      Result: True
     Comment: Kernel modules iptable_filter, ip_tables are already present
     Started: 15:20:12.164805
    Duration: 102.349 ms
     Changes:   
----------
          ID: /etc/iptables/rules.v4
    Function: file.managed
      Result: True
     Comment: File /etc/iptables/rules.v4 is in the correct state
     Started: 15:20:12.268184
    Duration: 382.87 ms
     Changes:   
----------
          ID: iptables_services_v4_start
    Function: cmd.run
        Name: find /usr/share/netfilter-persistent/plugins.d/[0-9]*-ip4tables -exec {} start \;
      Result: True
     Comment: onlyif execution failed
     Started: 15:20:12.652818
    Duration: 57.626 ms
     Changes:   
----------
          ID: netfilter-persistent
    Function: service.running
      Result: True
     Comment: The service netfilter-persistent is already running
     Started: 15:20:12.711992
    Duration: 302.449 ms
     Changes:   
----------
          ID: iptables_tables_cleanup_v4
    Function: module.wait
        Name: iptables_extra.remove_stale_tables
      Result: True
     Comment: 
     Started: 15:20:13.015763
    Duration: 6.321 ms
     Changes:   
----------
          ID: iptables_services_v6_stop
    Function: cmd.run
        Name: find /usr/share/netfilter-persistent/plugins.d/[0-9]*-ip6tables -exec {} flush \;
      Result: True
     Comment: onlyif execution failed
     Started: 15:20:13.022453
    Duration: 104.113 ms
     Changes:   
----------
          ID: /etc/iptables/rules.v6
    Function: file.absent
      Result: True
     Comment: File /etc/iptables/rules.v6 is not present
     Started: 15:20:13.128126
    Duration: 1.825 ms
     Changes:   
----------
          ID: iptables_tables_flush_v6
    Function: module.wait
        Name: iptables_extra.flush_all
      Result: True
     Comment: 
     Started: 15:20:13.131051
    Duration: 1.069 ms
     Changes:   
----------
          ID: salt_minion_service_restart
    Function: cmd.run
        Name: while true; do salt-call saltutil.running|grep fun: && continue; salt-call --local service.restart salt-minion; break; done
      Result: True
     Comment: State was not run because none of the onchanges reqs changed
     Started: 15:20:13.149815
    Duration: 0.066 ms
     Changes:   

Summary for mas01.mcp-odl-noha.local
-------------
Succeeded: 31
Failed:     0
-------------
Total states run:     31
Total run time:    3.366 s
[wait_for] OK: salt -C 'mas01*' state.apply salt,iptables
mas01.mcp-odl-noha.local:
----------
          ID: maas_cluster_packages
    Function: pkg.installed
        Name: maas-rack-controller
      Result: True
     Comment: All specified packages are already installed
     Started: 15:20:30.439089
    Duration: 305.828 ms
     Changes:   
----------
          ID: maas_cluster_packages
    Function: pkg.installed
        Name: ipmitool
      Result: True
     Comment: All specified packages are already installed
     Started: 15:20:30.745429
    Duration: 22.017 ms
     Changes:   
----------
          ID: /etc/maas/rackd.conf
    Function: file.line
      Result: True
     Comment: No changes needed to be made
     Started: 15:20:30.773610
    Duration: 2.715 ms
     Changes:   
----------
          ID: /etc/maas/rackd_conf
    Function: file.managed
        Name: /etc/maas/rackd.conf
      Result: True
     Comment: File /etc/maas/rackd.conf exists with proper permissions. No changes made.
     Started: 15:20:30.776729
    Duration: 2.524 ms
     Changes:   
----------
          ID: maas_cluster_services
    Function: service.running
        Name: maas-rackd
      Result: True
     Comment: The service maas-rackd is already running
     Started: 15:20:30.785381
    Duration: 280.938 ms
     Changes:   

Summary for mas01.mcp-odl-noha.local
------------
Succeeded: 5
Failed:    0
------------
Total states run:     5
Total run time: 614.022 ms

[wait_for] Waiting for cmd to return success: salt -C 'mas01*' state.apply maas.region
[wait_for] Attempt 1/10 for: salt -C 'mas01*' state.apply maas.region
mas01.mcp-odl-noha.local:
----------
          ID: maas_region_packages
    Function: pkg.installed
        Name: maas-region-controller
      Result: True
     Comment: All specified packages are already installed
     Started: 15:20:37.646207
    Duration: 299.464 ms
     Changes:   
----------
          ID: maas_region_packages
    Function: pkg.installed
        Name: python-oauth
      Result: True
     Comment: All specified packages are already installed
     Started: 15:20:37.946186
    Duration: 14.297 ms
     Changes:   
----------
          ID: /etc/maas/regiond.conf
    Function: file.replace
      Result: True
     Comment: No changes needed to be made
     Started: 15:20:37.966894
    Duration: 16.142 ms
     Changes:   
----------
          ID: /usr/lib/python3/dist-packages/provisioningserver/templates/proxy/maas-proxy.conf.template
    Function: file.managed
      Result: True
     Comment: File /usr/lib/python3/dist-packages/provisioningserver/templates/proxy/maas-proxy.conf.template is in the correct state
     Started: 15:20:37.983817
    Duration: 291.861 ms
     Changes:   
----------
          ID: maas_timeout_commissioning
    Function: file.replace
        Name: /usr/lib/python3/dist-packages/maasserver/node_status.py
      Result: True
     Comment: No changes needed to be made
     Started: 15:20:38.276440
    Duration: 6.705 ms
     Changes:   
----------
          ID: maas_timeout_deploying
    Function: file.replace
        Name: /usr/lib/python3/dist-packages/maasserver/node_status.py
      Result: True
     Comment: No changes needed to be made
     Started: 15:20:38.283874
    Duration: 6.495 ms
     Changes:   
----------
          ID: maas_interface_default_mode_dhcp
    Function: file.replace
        Name: /usr/lib/python3/dist-packages/maasserver/models/node.py
      Result: True
     Comment: No changes needed to be made
     Started: 15:20:38.291104
    Duration: 72.585 ms
     Changes:   
----------
          ID: /etc/apache2/conf-enabled/maas-http.conf
    Function: file.managed
      Result: True
     Comment: File /etc/apache2/conf-enabled/maas-http.conf is in the correct state
     Started: 15:20:38.384932
    Duration: 99.464 ms
     Changes:   
----------
          ID: maas_apache_headers
    Function: cmd.run
        Name: a2enmod headers
      Result: True
     Comment: Command "a2enmod headers" run
     Started: 15:20:38.486892
    Duration: 195.513 ms
     Changes:   
              ----------
              pid:
                  1219
              retcode:
                  0
              stderr:
              stdout:
                  Module headers already enabled
----------
          ID: /usr/share/maas/web/static/css/maas-styles.css
    Function: file.managed
      Result: True
     Comment: File /usr/share/maas/web/static/css/maas-styles.css is in the correct state
     Started: 15:20:38.682912
    Duration: 54.053 ms
     Changes:   
----------
          ID: /etc/maas/preseeds/curtin_userdata_amd64_generic_trusty
    Function: file.managed
      Result: True
     Comment: File /etc/maas/preseeds/curtin_userdata_amd64_generic_trusty is in the correct state
     Started: 15:20:38.737836
    Duration: 153.895 ms
     Changes:   
----------
          ID: /etc/maas/preseeds/curtin_userdata_amd64_generic_xenial
    Function: file.managed
      Result: True
     Comment: File /etc/maas/preseeds/curtin_userdata_amd64_generic_xenial is in the correct state
     Started: 15:20:38.892550
    Duration: 295.643 ms
     Changes:   
----------
          ID: /etc/maas/preseeds/curtin_userdata_arm64_generic_xenial
    Function: file.managed
      Result: True
     Comment: File /etc/maas/preseeds/curtin_userdata_arm64_generic_xenial is in the correct state
     Started: 15:20:39.189090
    Duration: 288.169 ms
     Changes:   
----------
          ID: Configure /root/.pgpass for MAAS
    Function: file.managed
        Name: /root/.pgpass
      Result: True
     Comment: File /root/.pgpass is in the correct state
     Started: 15:20:39.478593
    Duration: 253.74 ms
     Changes:   
----------
          ID: maas_region_syncdb
    Function: cmd.run
        Name: maas-region syncdb --noinput
      Result: True
     Comment: Command "maas-region syncdb --noinput" run
     Started: 15:20:39.740635
    Duration: 8780.444 ms
     Changes:   
              ----------
              pid:
                  1232
              retcode:
                  0
              stderr:
              stdout:
                  Operations to perform:
                    Synchronize unmigrated apps: staticfiles, messages
                    Apply all migrations: contenttypes, piston3, maasserver, metadataserver, sites, auth, sessions
                  Synchronizing apps without migrations:
                    Creating tables...
                      Running deferred SQL...
                    Installing custom SQL...
                  Running migrations:
                    No migrations to apply.
----------
          ID: maas_region_services
    Function: service.running
        Name: maas-regiond
      Result: True
     Comment: The service maas-regiond is already running
     Started: 15:20:48.532379
    Duration: 338.845 ms
     Changes:   
----------
          ID: maas_region_services
    Function: service.running
        Name: bind9
      Result: True
     Comment: The service bind9 is already running
     Started: 15:20:48.874003
    Duration: 217.3 ms
     Changes:   
----------
          ID: maas_region_services
    Function: service.running
        Name: apache2
      Result: True
     Comment: The service apache2 is already running
     Started: 15:20:49.093775
    Duration: 163.263 ms
     Changes:   
----------
          ID: maas_warmup
    Function: module.run
        Name: maasng.wait_for_http_code
      Result: True
     Comment: Module function maasng.wait_for_http_code executed
     Started: 15:20:49.259912
    Duration: 466.872 ms
     Changes:   
              ----------
              ret:
                  ----------
                  comment:
                      MAAS API:http://localhost:5240/MAAS up.
                  result:
                      True
----------
          ID: maas_set_admin_password
    Function: cmd.run
        Name: maas createadmin --username opnfv --password opnfv_secret --email email@example.com && touch /var/lib/maas/.setup_admin
      Result: True
     Comment: /var/lib/maas/.setup_admin exists
     Started: 15:20:49.728974
    Duration: 1.881 ms
     Changes:   
----------
          ID: maas_login_admin
    Function: cmd.run
        Name: maas-region apikey --username opnfv > /var/lib/maas/.maas_credentials
      Result: True
     Comment: Command "maas-region apikey --username opnfv > /var/lib/maas/.maas_credentials" run
     Started: 15:20:49.731910
    Duration: 4543.495 ms
     Changes:   
              ----------
              pid:
                  1258
              retcode:
                  0
              stderr:
              stdout:
----------
          ID: maas_region_boot_source_resources_mirror
    Function: maasng.boot_source_present
      Result: True
     Comment: boot-source http://images.maas.io/ephemeral-v3/daily alredy exist
     Started: 15:20:54.295377
    Duration: 932.425 ms
     Changes:   
              ----------
              changes:
                  ----------
    Warnings: 'name' is an invalid keyword argument for
              'maasng.boot_source_present'. If you were trying to pass
              additional data to be used in a template context, please populate
              'context' with 'key: value' pairs. Your approach will work until
              Salt Fluorine is out. Please update your state files.
----------
          ID: maas_wait_for_region_import_done
    Function: module.run
        Name: maasng.boot_resources_import
      Result: True
     Comment: Module function maasng.boot_resources_import executed
     Started: 15:20:55.230414
    Duration: 26842.891 ms
     Changes:   
              ----------
              ret:
                  True
----------
          ID: maas_region_boot_sources_selection_xenial
    Function: maasng.boot_sources_selections_present
      Result: True
     Comment: Requested boot-source selection for http://images.maas.io/ephemeral-v3/daily already exist.
     Started: 15:21:22.075745
    Duration: 1235.745 ms
     Changes:   
    Warnings: 'name' is an invalid keyword argument for
              'maasng.boot_sources_selections_present'. If you were trying to
              pass additional data to be used in a template context, please
              populate 'context' with 'key: value' pairs. Your approach will
              work until Salt Fluorine is out. Please update your state files.
----------
          ID: maas_wait_for_racks_import_done
    Function: module.run
        Name: maasng.sync_and_wait_bs_to_all_racks
      Result: True
     Comment: Module function maasng.sync_and_wait_bs_to_all_racks executed
     Started: 15:21:23.333796
    Duration: 4206.765 ms
     Changes:   
              ----------
              ret:
                  True
----------
          ID: maas_config
    Function: module.run
        Name: maas.process_maas_config
      Result: True
     Comment: Module function maas.process_maas_config executed
     Started: 15:21:27.543122
    Duration: 9266.708 ms
     Changes:   
              ----------
              ret:
                  ----------
                  errors:
                      ----------
                  success:
                      - enable_http_proxy
                      - upstream_dns
                      - commissioning_distro_series
                      - default_osystem
                      - active_discovery_interval
                      - dnssec_validation
                      - maas_name
                      - network_discovery
                      - enable_third_party_drivers
                      - default_storage_layout
                      - ntp_external_only
                      - disk_erase_with_secure_erase
                      - default_distro_series
                      - default_min_hwe_kernel
                  updated:
----------
          ID: maas_fabrics_pxe_admin
    Function: maasng.fabric_present
        Name: pxe_admin
      Result: True
     Comment: Module function maasng.fabric_present executed
     Started: 15:21:36.810687
    Duration: 990.872 ms
     Changes:   
              ----------
              new:
                  Fabric  pxe_admin created
              result:
                  True
----------
          ID: maas_vlan0_present_for_pxe_admin
    Function: maasng.vlan_present_in_fabric
        Name: vlan 0
      Result: True
     Comment: Vlan vlan 0 has been updated for pxe_admin
     Started: 15:21:37.801957
    Duration: 1447.066 ms
     Changes:   
              ----------
              new:
                  Vlan untagged was updated
----------
          ID: maas_create_subnet_10.1.0.0/24
    Function: maasng.subnet_present
        Name: 10.1.0.0/24
      Result: True
     Comment: Subnet 10.1.0.0/24 has been updated for pxe_admin
     Started: 15:21:39.250613
    Duration: 1463.569 ms
     Changes:   
----------
          ID: maas_create_iprange_1
    Function: maasng.iprange_present
      Result: True
     Comment: Iprange maas_create_iprange_1 already exist.
     Started: 15:21:40.715326
    Duration: 330.922 ms
     Changes:   
----------
          ID: maas_vlan0_present_for_pxe_admin_dhcp
    Function: maasng.vlan_present_in_fabric
        Name: vlan 0
      Result: True
     Comment: Vlan vlan 0 has been updated for pxe_admin
     Started: 15:21:41.046668
    Duration: 2452.608 ms
     Changes:   
              ----------
              new:
                  Vlan untagged was updated
----------
          ID: maas_sshkey_1
    Function: maasng.sshkey_present
        Name: opnfv
      Result: True
     Comment: SSH key ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9iteXSOD1Du/fG2coTPKPKmlyjD5EHa0wwZm9pdya5BuWatdcIzkCgKLdLBST3LZZ6U8IB83LnCwKa7CX87OPxZVylK8W8NlzsjYt3P+ZOX8qUmhTxxkJa4i43aZXbQwv1a6QAwHY6n2/8mKdLpov5gf6Y28HpUN1d3mp3gegNK5quTNAxS2MzUXk9gOZ8ZXYLXCYjg4eWXXXCLxK+rmojXgsjdY3WruVLR+1YkXX38jJ0pirCp10knC+QGf1QTQE2quzYQNh8NQA+1weNUWi6xYd8c9Nt/bBdtoL56hqivJeNYiFw2PLeoGqfOtI1OrwR1Cp0WdoJ+cIdW9P/7ov already exist for user opnfv.
     Started: 15:21:43.500644
    Duration: 98.417 ms
     Changes:   

Summary for mas01.mcp-odl-noha.local
-------------
Succeeded: 32 (changed=11)
Failed:     0
Warnings:   2
-------------
Total states run:     32
Total run time:   65.828 s
[wait_for] OK: salt -C 'mas01*' state.apply maas.region
mas01.mcp-odl-noha.local:
----------
          ID: maas_login_admin
    Function: cmd.run
        Name: maas-region apikey --username opnfv > /var/lib/maas/.maas_credentials
      Result: True
     Comment: Command "maas-region apikey --username opnfv > /var/lib/maas/.maas_credentials" run
     Started: 15:21:57.307874
    Duration: 5491.612 ms
     Changes:   
              ----------
              pid:
                  1614
              retcode:
                  0
              stderr:
              stdout:
----------
          ID: create__machines
    Function: module.run
        Name: maas.process_machines
      Result: True
     Comment: Module function maas.process_machines executed
     Started: 15:22:02.820679
    Duration: 18829.671 ms
     Changes:   
              ----------
              ret:
                  ----------
                  errors:
                      ----------
                  success:
                  updated:
                      - cmp002
                      - cmp003
                      - cmp001
                      - odl01
                      - gtw01
                      - ctl01

Summary for mas01.mcp-odl-noha.local
------------
Succeeded: 2 (changed=2)
Failed:    0
------------
Total states run:     2
Total run time:  24.321 s
tput: unknown terminal "unknown"
[NOTE] MaaS operations might take a long time, please be patient
tput: unknown terminal "unknown"
mas01.mcp-odl-noha.local:
----------
          ID: maas_login_admin
    Function: cmd.run
        Name: maas-region apikey --username opnfv > /var/lib/maas/.maas_credentials
      Result: True
     Comment: Command "maas-region apikey --username opnfv > /var/lib/maas/.maas_credentials" run
     Started: 15:23:06.400377
    Duration: 5734.137 ms
     Changes:   
              ----------
              pid:
                  1956
              retcode:
                  0
              stderr:
              stdout:
----------
          ID: wait_for_machines_ready_or_deployed
    Function: module.run
        Name: maas.wait_for_machine_status
      Result: False
     Comment: Module function maas.wait_for_machine_status threw an exception. Exception: Machines:['cmp002', 'cmp003', 'cmp001', 'odl01', 'gtw01', 'ctl01']not in Ready|Deployed state
     Started: 15:23:12.139261
    Duration: 1504206.603 ms
     Changes:   

Summary for mas01.mcp-odl-noha.local
------------
Succeeded: 1 (changed=1)
Failed:    1
------------
Total states run:     2
Total run time: 1509.941 s
ERROR: Minions returned with non-zero exit code
[wait_for] ERROR: Failed after max attempts: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i /var/lib/opnfv/mcp.rsa ubuntu@10.20.0.2 sudo             CI_DEBUG=false ERASE_ENV=0             /root/fuel/mcp/config/states/maas

[ERROR] MCP: Installation  of os-odl-sfc-noha threw a fatal error!

(B
--------------------------------------------------------
Deployment is done!
Deployment failed!
Build step 'Execute shell' marked build as failure
New run name is '212 - Scenario: os-odl-sfc-noha'
[PostBuildScript] - Executing post build scripts.
[PostBuildScript] - Build does not have any of the results [SUCCESS]. Did not execute build step #0.
[PostBuildScript] - Executing post build scripts.
[fuel-deploy-itri-pod1-daily-master] $ /bin/sh -xe /tmp/jenkins6446979469655112142.sh
+ echo export PROVISION_RESULT=FAIL
+ echo export INSTALLER=fuel
+ date +%Y-%m-%d %H:%M:%S.%3N
+ echo export TIMESTAMP_END='2019-09-26 23:48:29.149'
[fuel-deploy-itri-pod1-daily-master] $ /bin/bash /tmp/jenkins4583635193532725680.sh
Installer: fuel provision result: FAIL
{ "installer": "fuel", "version": "master", "pod_name": "itri-pod1", "job_name": "fuel-deploy-itri-pod1-daily-master", "build_id": "212", "scenario": "os-odl-sfc-noha", "upstream_job_name": "", "upstream_build_id":"", "criteria": "FAIL", "start_date": "2019-09-26 21:00:18.615", "stop_date": "2019-09-26 23:48:29.149", "details":"" }
++ gen_content
++ cat
+ curl -H 'Content-Type: application/json' -X POST -v -d '{
    "installer": "fuel",
    "version": "master",
    "pod_name": "itri-pod1",
    "job_name": "fuel-deploy-itri-pod1-daily-master",
    "build_id": "212",
    "scenario": "os-odl-sfc-noha",
    "upstream_job_name": "",
    "upstream_build_id":"",
    "criteria": "FAIL",
    "start_date": "2019-09-26 21:00:18.615",
    "stop_date": "2019-09-26 23:48:29.149",
    "details":""
}' http://testresults.opnfv.org/test/api/v1/deployresults
Note: Unnecessary use of -X or --request, POST is already inferred.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 104.154.71.112...
* Connected to testresults.opnfv.org (104.154.71.112) port 80 (#0)
> POST /test/api/v1/deployresults HTTP/1.1
> Host: testresults.opnfv.org
> User-Agent: curl/7.47.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 381
> 
} [381 bytes data]
* upload completely sent off: 381 out of 381 bytes
< HTTP/1.1 200 OK
< Server: nginx/1.10.2
< Date: Thu, 26 Sep 2019 15:48:30 GMT
< Content-Type: application/json
< Content-Length: 79
< Connection: keep-alive
< 
{ [79 bytes data]

100   460  100    79  100   381    190    919 --:--:-- --:--:-- --:--:--   920
* Connection #0 to host testresults.opnfv.org left intact
{"href": "http://localhost:8082/api/v1/deployresults/5d8cddce1b7bcc000e475f62"}+ '[' FAIL == PASS ']'
+ exit 1
Email was triggered for: Always
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
Request made to compress build log
Sending email to: Alexandru.Avadanii@enea.com
Sending email for trigger: Always
Request made to compress build log
Sending email to: Alexandru.Avadanii@enea.com
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
Request made to compress build log
Sending email to: tbramwell@linuxfoundation.org agardner@linuxfoundation.org rgrigar@linuxfoundation.org
Finished: FAILURE