SuccessConsole Output

Started by upstream project "doctor-verify-master" build number 146
originally caused by:
 Triggered by Gerrit: https://gerrit.opnfv.org/gerrit/66019
[EnvInject] - Loading node environment variables.
Building remotely on nokia-pod1 (nokia opnfv-sysinfo doctor-apex-x86_64) in workspace /home/jenkins/opnfv_slave_root/workspace/doctor-verify-maintenance-apex-sample-x86_64-master
[ssh-agent] Looking for ssh-agent implementation...
[ssh-agent]   Exec ssh-agent (binary ssh-agent on a remote machine)
$ ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-lO0v7BTUoKGB/agent.8273
SSH_AGENT_PID=8275
[ssh-agent] Started.
Running ssh-add (command line suppressed)
Identity added: /home/jenkins/opnfv_slave_root/workspace/doctor-verify-maintenance-apex-sample-x86_64-master@tmp/private_key_4314942717693627802.key (/home/jenkins/opnfv_slave_root/workspace/doctor-verify-maintenance-apex-sample-x86_64-master@tmp/private_key_4314942717693627802.key)
[ssh-agent] Using credentials jenkins-ci (Jenkins Master SSH)
using credential d42411ac011ad6f3dd2e1fa34eaa5d87f910eb2e
Wiping out workspace first.
Cloning the remote Git repository
Cloning repository https://gerrit.opnfv.org/gerrit/doctor
 > git init /home/jenkins/opnfv_slave_root/workspace/doctor-verify-maintenance-apex-sample-x86_64-master # timeout=10
Fetching upstream changes from https://gerrit.opnfv.org/gerrit/doctor
 > git --version # timeout=10
using GIT_SSH to set credentials Jenkins Master SSH
 > git fetch --tags --progress https://gerrit.opnfv.org/gerrit/doctor +refs/heads/*:refs/remotes/origin/* # timeout=15
 > git config remote.origin.url https://gerrit.opnfv.org/gerrit/doctor # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://gerrit.opnfv.org/gerrit/doctor # timeout=10
Fetching upstream changes from https://gerrit.opnfv.org/gerrit/doctor
using GIT_SSH to set credentials Jenkins Master SSH
 > git fetch --tags --progress https://gerrit.opnfv.org/gerrit/doctor refs/changes/19/66019/7 # timeout=15
Checking out Revision 73605c5c34b97ab56306bfa9af0f5888f3c7e46d (refs/changes/19/66019/7)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 73605c5c34b97ab56306bfa9af0f5888f3c7e46d # timeout=15
Commit message: "Support Fenix as admin tool"
 > git rev-parse FETCH_HEAD^{commit} # timeout=10
 > git rev-list --no-walk 33293e9c23a21ad3228f46d2063f18c915eb2b79 # timeout=10
No emails were triggered.
[doctor-verify-maintenance-apex-sample-x86_64-master] $ /usr/bin/env bash /tmp/jenkins8252613109924335893.sh
Gathering IP information for Apex installer VM
 6     undercloud                     running
Installer VM detected
Installer ip is 192.168.122.40
fetch_os_creds.info: Fetching rc file...
fetch_os_creds.info: Verifying connectivity to 192.168.122.40...
fetch_os_creds.info: 192.168.122.40 is reachable!
fetch_os_creds.info: ... from Instack VM 192.168.122.40...
Warning: Permanently added '192.168.122.40' (ECDSA) to the list of known hosts.
-------- Credentials: --------
# Clear any old environment that may conflict.
for key in $( set | awk '{FS="="}  /^OS_/ {print $1}' ); do unset $key ; done
export OS_NO_CACHE=True
export COMPUTE_API_VERSION=1.1
export OS_USERNAME=admin
export no_proxy=,192.168.37.19,192.0.2.4
export OS_USER_DOMAIN_NAME=Default
export OS_VOLUME_API_VERSION=3
export OS_CLOUDNAME=overcloud
export OS_AUTH_URL=http://192.168.37.19:5000/v3
export NOVA_VERSION=1.1
export OS_IMAGE_API_VERSION=2
#export OS_PASSWORD=N9flPG9TD7lwww7vsVewDHyBK
export OS_PASSWORD=admin
export OS_PROJECT_DOMAIN_NAME=Default
export OS_IDENTITY_API_VERSION=3
export OS_PROJECT_NAME=admin
export OS_AUTH_TYPE=password
export PYTHONWARNINGS="ignore:Certificate has no, ignore:A true SSLContext object is not available"

# Add OS_CLOUDNAME to PS1
if [ -z "${CLOUDPROMPT_ENABLED:-}" ]; then
    export PS1=${PS1:-""}
    export PS1=\${OS_CLOUDNAME:+"(\$OS_CLOUDNAME)"}\ $PS1
    export CLOUDPROMPT_ENABLED=1
fi
export OS_PROJECT_ID=6e76c202356a46cbbd626ef7c00efc0c
export OS_TENANT_NAME=admin
export OS_REGION_NAME=regionOne
[doctor-verify-maintenance-apex-sample-x86_64-master] $ /bin/sh -xe /tmp/jenkins5096031153367905369.sh
+ source /home/jenkins/opnfv-openrc.sh
+++ set
+++ awk '{FS="="}  /^OS_/ {print $1}'
++ export OS_NO_CACHE=True
++ OS_NO_CACHE=True
++ export COMPUTE_API_VERSION=1.1
++ COMPUTE_API_VERSION=1.1
++ export OS_USERNAME=admin
++ OS_USERNAME=admin
++ export no_proxy=,192.168.37.19,192.0.2.4
++ no_proxy=,192.168.37.19,192.0.2.4
++ export OS_USER_DOMAIN_NAME=Default
++ OS_USER_DOMAIN_NAME=Default
++ export OS_VOLUME_API_VERSION=3
++ OS_VOLUME_API_VERSION=3
++ export OS_CLOUDNAME=overcloud
++ OS_CLOUDNAME=overcloud
++ export OS_AUTH_URL=http://192.168.37.19:5000/v3
++ OS_AUTH_URL=http://192.168.37.19:5000/v3
++ export NOVA_VERSION=1.1
++ NOVA_VERSION=1.1
++ export OS_IMAGE_API_VERSION=2
++ OS_IMAGE_API_VERSION=2
++ export OS_PASSWORD=admin
++ OS_PASSWORD=admin
++ export OS_PROJECT_DOMAIN_NAME=Default
++ OS_PROJECT_DOMAIN_NAME=Default
++ export OS_IDENTITY_API_VERSION=3
++ OS_IDENTITY_API_VERSION=3
++ export OS_PROJECT_NAME=admin
++ OS_PROJECT_NAME=admin
++ export OS_AUTH_TYPE=password
++ OS_AUTH_TYPE=password
++ export 'PYTHONWARNINGS=ignore:Certificate has no, ignore:A true SSLContext object is not available'
++ PYTHONWARNINGS='ignore:Certificate has no, ignore:A true SSLContext object is not available'
++ '[' -z '' ']'
++ export PS1=
++ PS1=
++ export 'PS1=${OS_CLOUDNAME:+($OS_CLOUDNAME)} '
++ PS1='${OS_CLOUDNAME:+($OS_CLOUDNAME)} '
++ export CLOUDPROMPT_ENABLED=1
++ CLOUDPROMPT_ENABLED=1
++ export OS_PROJECT_ID=6e76c202356a46cbbd626ef7c00efc0c
++ OS_PROJECT_ID=6e76c202356a46cbbd626ef7c00efc0c
++ export OS_TENANT_NAME=admin
++ OS_TENANT_NAME=admin
++ export OS_REGION_NAME=regionOne
++ OS_REGION_NAME=regionOne
+ '[' -f /home/jenkins/os_cacert ']'
+ source /home/jenkins/opnfv-installer.sh
++ export INSTALLER_TYPE=apex
++ INSTALLER_TYPE=apex
++ export INSTALLER_IP=192.168.122.40
++ INSTALLER_IP=192.168.122.40
++ export SSH_KEY=/home/jenkins/installer_key_file
++ SSH_KEY=/home/jenkins/installer_key_file
+ sudo -E tox -e py34
py34 create: /home/jenkins/opnfv_slave_root/workspace/doctor-verify-maintenance-apex-sample-x86_64-master/.tox/py34
py34 installdeps: -r/home/jenkins/opnfv_slave_root/workspace/doctor-verify-maintenance-apex-sample-x86_64-master/requirements.txt
py34 develop-inst: /home/jenkins/opnfv_slave_root/workspace/doctor-verify-maintenance-apex-sample-x86_64-master
py34 installed: DEPRECATION: Python 3.4 support has been deprecated. pip 19.1 will be the last one supporting it. Please upgrade your Python as Python 3.4 won't be maintained after March 2019 (cf PEP 429).,amqp==2.2.1,aodhclient==0.9.0,appdirs==1.4.3,asn1crypto==0.22.0,Babel==2.3.4,bcrypt==3.1.3,cachetools==2.0.0,certifi==2017.4.17,cffi==1.10.0,chardet==3.0.4,click==6.7,cliff==2.8.2,cmd2==0.7.5,contextlib2==0.5.5,cryptography==2.0.2,debtcollector==1.17.1,deprecation==1.0.1,-e git+https://gerrit.opnfv.org/gerrit/doctor@73605c5c34b97ab56306bfa9af0f5888f3c7e46d#egg=doctor_tests,enum-compat==0.0.2,eventlet==0.20.0,fasteners==0.14.1,flake8==2.5.5,Flask==0.12.2,futurist==1.3.1,greenlet==0.4.15,idna==2.5,iso8601==0.1.11,itsdangerous==0.24,Jinja2==2.9.6,jsonpatch==1.16,jsonpointer==1.10,jsonschema==2.6.0,keystoneauth1==3.1.0,kombu==4.1.0,MarkupSafe==1.0,mccabe==0.4.0,monotonic==1.3,msgpack-python==0.4.8,netaddr==0.7.19,netifaces==0.10.6,openstacksdk==0.9.17,os-client-config==1.28.0,osc-lib==1.7.0,oslo.concurrency==3.21.1,oslo.config==4.11.1,oslo.context==2.17.1,oslo.i18n==3.17.1,oslo.log==3.30.2,oslo.messaging==5.30.7,oslo.middleware==3.30.1,oslo.serialization==2.20.2,oslo.service==1.25.1,oslo.utils==3.28.3,oslo.versionedobjects==1.26.2,paramiko==2.2.1,Paste==2.0.3,PasteDeploy==1.5.2,pbr==3.1.1,pep8==1.7.1,pika==0.10.0,pika-pool==0.1.3,positional==1.1.2,prettytable==0.7.2,pyasn1==0.3.1,pycparser==2.18,pyflakes==1.0.0,pyinotify==0.9.6,PyNaCl==1.1.2,pyOpenSSL==17.2.0,pyparsing==2.2.0,pyperclip==1.5.27,python-ceilometerclient==2.9.0,python-cinderclient==3.1.1,python-congressclient==1.8.0,python-dateutil==2.6.1,python-glanceclient==2.8.0,python-heatclient==1.11.1,python-keystoneclient==3.13.0,python-neutronclient==6.5.0,python-novaclient==9.1.2,python-openstackclient==3.12.1,python-swiftclient==3.4.0,python-vitrageclient==1.4.0,pytz==2017.2,PyYAML==3.12,repoze.lru==0.6,requests==2.18.2,requestsexceptions==1.3.0,rfc3986==1.1.0,Routes==2.4.1,scp==0.10.2,simplejson==3.11.1,six==1.10.0,statsd==3.2.1,stevedore==1.25.1,tenacity==4.4.0,urllib3==1.22,vine==1.1.4,virtualenv==15.1.0,warlock==1.2.0,WebOb==1.7.3,Werkzeug==0.12.2,wrapt==1.10.10
py34 runtests: PYTHONHASHSEED='2626167386'
py34 runtests: commands[0] | doctor-test
/home/jenkins/opnfv_slave_root/workspace/doctor-verify-maintenance-apex-sample-x86_64-master/.tox/py34/lib/python3.4/site-packages/paramiko/client.py:711: UserWarning: Unknown ssh-ed25519 host key for 192.168.122.40: b'3dc1628c6fe2819031608f780639f40a'
  key.get_fingerprint())))
2019-03-26 18:34:07,421 main.py 130 INFO   doctor test starting.......
2019-03-26 18:34:07,422 apex.py 43 INFO   Setup Apex installer start......
2019-03-26 18:34:07,422 base.py 113 INFO   Get SSH keys from apex installer......
2019-03-26 18:34:07,699 apex.py 67 INFO   Get overcloud config details from Apex installer......
2019-03-26 18:34:07,700 base.py 174 INFO   Run command=source stackrc; nova list | grep ' overcloud-' in apex installer......
2019-03-26 18:34:10,825 base.py 183 INFO   Output=['| 2d0aedd7-4e30-4fee-bbd2-0feb8b1078c4 | overcloud-controller-0  | ACTIVE | -          | Running     | ctlplane=192.0.2.9 |', '| 37ceec17-a130-4826-87f0-67699d3350f5 | overcloud-novacompute-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.6 |', '| 416383de-15fb-477a-9a6c-f0b21df9be70 | overcloud-novacompute-1 | ACTIVE | -          | Running     | ctlplane=192.0.2.7 |', '| 0f025a92-3365-4c3b-951e-dab9a6d9ca22 | overcloud-novacompute-2 | ACTIVE | -          | Running     | ctlplane=192.0.2.3 |'] command=source stackrc; nova list | grep ' overcloud-' in apex installer
2019-03-26 18:34:10,826 base.py 188 INFO   Check command=grep docker /home/stack/deploy_command return in apex installer......
2019-03-26 18:34:10,897 base.py 191 INFO   return 0
2019-03-26 18:34:10,898 apex.py 80 INFO   controller_ips:['192.0.2.9']
2019-03-26 18:34:10,899 apex.py 81 INFO   compute_ips:['192.0.2.6', '192.0.2.7', '192.0.2.3']
2019-03-26 18:34:10,899 apex.py 82 INFO   use_containers:True
2019-03-26 18:34:10,899 base.py 174 INFO   Run command=scp overcloudrc heat-admin@192.0.2.9: in apex installer......
2019-03-26 18:34:11,326 base.py 183 INFO   Output=[] command=scp overcloudrc heat-admin@192.0.2.9: in apex installer
2019-03-26 18:34:12,244 apex.py 102 INFO   Set apply patches start......
/home/jenkins/opnfv_slave_root/workspace/doctor-verify-maintenance-apex-sample-x86_64-master/.tox/py34/lib/python3.4/site-packages/paramiko/client.py:711: UserWarning: Unknown ssh-ed25519 host key for 192.0.2.9: b'b3b023c9e950672f64d119fecdae83fd'
  key.get_fingerprint())))
2019-03-26 18:34:12,833 base.py 218 INFO   Command sudo python set_config.py output ['Add event notifier in ceilometer', 'NOTE: add compute.instance.update to event_definitions.yaml', 'NOTE: add maintenance.scheduled to event_definitions.yaml', 'NOTE: add maintenance.host to event_definitions.yaml']
2019-03-26 18:34:14,666 base.py 218 INFO   Command sudo python restart_aodh.py output []
2019-03-26 18:34:17,647 apex.py 148 INFO   Set apply patches start......
/home/jenkins/opnfv_slave_root/workspace/doctor-verify-maintenance-apex-sample-x86_64-master/.tox/py34/lib/python3.4/site-packages/paramiko/client.py:711: UserWarning: Unknown ssh-ed25519 host key for 192.0.2.6: b'5ffdaf1f169876d8c9ac845458601952'
  key.get_fingerprint())))
/home/jenkins/opnfv_slave_root/workspace/doctor-verify-maintenance-apex-sample-x86_64-master/.tox/py34/lib/python3.4/site-packages/paramiko/client.py:711: UserWarning: Unknown ssh-ed25519 host key for 192.0.2.7: b'50858b8f4760d0c962bbdcb845047215'
  key.get_fingerprint())))
/home/jenkins/opnfv_slave_root/workspace/doctor-verify-maintenance-apex-sample-x86_64-master/.tox/py34/lib/python3.4/site-packages/paramiko/client.py:711: UserWarning: Unknown ssh-ed25519 host key for 192.0.2.3: b'890a9b151267bafe34f53791df2c09f2'
  key.get_fingerprint())))
2019-03-26 18:34:18,052 base.py 218 INFO   Command sudo python set_compute_config.py output []
2019-03-26 18:34:18,123 base.py 218 INFO   Command sudo python set_compute_config.py output []
2019-03-26 18:34:18,197 base.py 218 INFO   Command sudo python set_compute_config.py output []
2019-03-26 18:34:22,878 base.py 63 INFO   Setup ssh stunnel in apex installer......
2019-03-26 18:34:22,879 base.py 76 INFO   tunnel for port 12346
2019-03-26 18:34:22,883 base.py 76 INFO   tunnel for port 12348
2019-03-26 18:34:22,888 base.py 76 INFO   tunnel for port 12345
2019-03-26 18:34:22,892 base.py 94 INFO   tunnel for port 12347
2019-03-26 18:34:22,896 image.py 48 INFO   image create start......
2019-03-26 18:34:42,864 image.py 68 INFO   image create end......
2019-03-26 18:34:42,865 user.py 70 INFO   user create start......
2019-03-26 18:34:43,127 user.py 86 INFO   create project......
2019-03-26 18:34:43,436 user.py 95 INFO   test project <Project description=, domain_id=default, enabled=True, id=b25dae25ccc643d093a2a4971f9c8a62, is_domain=False, links={'self': 'http://192.0.2.4:35357/v3/projects/b25dae25ccc643d093a2a4971f9c8a62'}, name=doctor, parent_id=default, tags=[]>
2019-03-26 18:34:43,698 user.py 103 INFO   create user......
2019-03-26 18:34:44,300 user.py 113 INFO   test user <User domain_id=default, enabled=True, id=95d9b34a21dc4188ba386fb4a3eff1e3, links={'self': 'http://192.0.2.4:35357/v3/users/95d9b34a21dc4188ba386fb4a3eff1e3'}, name=doctor, options={}, password_expires_at=None>
2019-03-26 18:34:44,531 user.py 127 INFO   role _member_ already created......
2019-03-26 18:34:44,531 user.py 128 INFO   test role <Role domain_id=None, id=9fe2ff9ee4384b1894a90878d3e92bab, links={'self': 'http://192.0.2.4:35357/v3/roles/9fe2ff9ee4384b1894a90878d3e92bab'}, name=_member_>
2019-03-26 18:34:45,715 user.py 78 INFO   user create end......
2019-03-26 18:34:46,611 main.py 104 INFO   doctor maintenance test starting.......
2019-03-26 18:34:49,236 maintenance.py 62 INFO   checking hypervisors.......
2019-03-26 18:34:49,236 maintenance.py 95 INFO   testing 3 computes with 32 vcpus each
2019-03-26 18:34:49,236 maintenance.py 98 INFO   testing 2 actstdby and 4 noredundancy instances
2019-03-26 18:34:49,236 user.py 190 INFO   quota update start......
2019-03-26 18:34:49,237 user.py 206 INFO   default quota update start......
2019-03-26 18:34:50,300 user.py 217 INFO   user quota update start......
2019-03-26 18:34:50,677 user.py 230 INFO   quota update end......
2019-03-26 18:34:51,757 maintenance.py 117 INFO   creating maintenance stack.......
2019-03-26 18:34:51,758 maintenance.py 118 INFO   parameters: {'maint_image': 'cirros', 'ext_net': 'external', 'nonha_intances': 4, 'ha_intances': 2, 'flavor_vcpus': 16}
2019-03-26 18:36:48,918 stack.py 65 INFO   stack doctor_test_maintenance CREATE_COMPLETE
2019-03-26 18:36:48,919 sample.py 31 INFO   sample app manager start......
2019-03-26 18:36:50,484 sample.py 85 INFO   sample inspector start......
 * Running on http://0.0.0.0:12348/ (Press CTRL+C to quit)
2019-03-26 18:36:52,206 main.py 112 INFO   wait aodh for 120s.......
 * Running on http://0.0.0.0:12345/ (Press CTRL+C to quit)
2019-03-26 18:38:52,253 maintenance.py 134 INFO   start maintenance.......
2019-03-26 18:38:52,327 maintenance.py 173 INFO   admin tool not ready, retry in 10s
2019-03-26 18:39:02,345 maintenance.py 173 INFO   admin tool not ready, retry in 10s
2019-03-26 18:39:12,354 maintenance.py 173 INFO   admin tool not ready, retry in 10s
2019-03-26 18:39:22,369 maintenance.py 173 INFO   admin tool not ready, retry in 10s
2019-03-26 18:39:24,403 base.py 218 INFO   Command sudo chmod 700 set_fenix.sh;sudo ./set_fenix.sh output ['07ad2871ba58', '07ad2871ba58', 'Untagged: fenix:latest', 'Deleted: sha256:0c971558e46d847d30e4fa4d236c3a819de4ae5a6d6ad236972006ef3db380aa', 'Deleted: sha256:6fd8cb4f932f2b0d43d1c9e3818188e7de397b6fabab5104f60ab344cd5422a2', 'Deleted: sha256:be4ec8aa2c1bef3fdea8928476be22922a754c84c29e30eb967e71b4ee3ff68b', 'Deleted: sha256:fac202384d9487a482572420f0a7722f5432c6252986bb12f8f79553a843760a', 'Deleted: sha256:a3713452c1cef85e054a4ed475dd53f315b92bb925d23cd2c1cc72f4564818e0', 'Deleted: sha256:90640da525ebd0edb5197f6032b799ab66b21e748e46fffba1f566d0e23be8f0', 'Deleted: sha256:417e432d191d841512a0c04ab5408277a22b83abd9a2225cfbba88e999d4c81f', 'Deleted: sha256:165991bd8c35101c14f822ca281d8fff550de2423140a54d7ade03024b16745d', 'Deleted: sha256:c7333f6761efb5fabefd0fa5acb50e97b9f0376c8784764ce9e26ff5c605396c', 'Deleted: sha256:8e596493c88449286eb3a4263f49298fa54786f7dbf67eeacb15f187596a5f23', 'Deleted: sha256:c3861139e3e7801104561986016cb8ec0f5dd4f974b47703c694ce0cf589e80d', 'Deleted: sha256:8d6907ad26ee589253646d96ceb2964028f0c2218cabea9b86562a3c3bdef15b', 'Deleted: sha256:c9cf0922cd34ee3e1ef22b272ef893cffa645df7af8735b0fd760e5b209dedae', 'Deleted: sha256:5a0c73f182d310e6f216a0cdb18150a6d419aab16030f97463b6dbcaba3318e9', 'Deleted: sha256:700b80c07d2dc26ba4a84168d0e730cfad0768bf0acc71542ebb8ea9ed9bb164', 'Deleted: sha256:39b7fd345c904f03b12b7e282ea2604a8b65d03301009ab46d51c494c41d0fbb', 'Deleted: sha256:24460070fa79a890ee6bc21e4b32275819dc5f054fc38df97cf29fed84f69266', 'Deleted: sha256:fcb2044643377f190d9a6e6351c3969940ca0d320cb89ff259380a1dc6b842b4', 'Successfully built 8d18afd12a53', 'cb673515bb092f058f2677e92ca9f640a3f552dc3915434a74e70a6ee4dc441e', 'Fenix start: OK']
2019-03-26 18:39:24,583 base.py 218 INFO   Command sudo python set_compute_config.py output []
2019-03-26 18:39:44,789 sample.py 164 INFO   sample app manager received data = {'session_id': 'bbabcd4f-4fe5-11e9-876c-54ab3a13729a', 'service': 'fenix', 'reply_at': '2019-03-26T16:39:16', 'allowed_actions': [], 'state': 'MAINTENANCE', 'metadata': {'openstack_version': 'Rocky'}, 'project_id': 'b25dae25ccc643d093a2a4971f9c8a62', 'instance_ids': 'http://127.0.0.1:12347/v1/maintenance/bbabcd4f-4fe5-11e9-876c-54ab3a13729a/b25dae25ccc643d093a2a4971f9c8a62', 'actions_at': '2019-03-26T16:40:02', 'reply_url': 'http://127.0.0.1:12347/v1/maintenance/bbabcd4f-4fe5-11e9-876c-54ab3a13729a/b25dae25ccc643d093a2a4971f9c8a62'}
2019-03-26 18:39:44,789 sample.py 170 INFO   sample app manager state: MAINTENANCE
2019-03-26 18:39:44,836 sample.py 120 INFO   get_instance_ids {'instance_ids': ['2d153bc1-f51d-4ea6-999c-22943d6683d4', '7d35a479-0364-4a8f-9c7c-034b2556b145', '370b44c4-3a91-48f0-9e69-55b2f64ddd67', 'a6c6b1d5-68c2-42cd-8590-187609396a13', 'ef75e252-1bb0-4ab0-a458-a87069295b86', 'aad72041-7b90-4c8d-a3ea-f26d377cc820']}
2019-03-26 18:39:44,836 sample.py 240 INFO   sample app manager reply: {'session_id': 'bbabcd4f-4fe5-11e9-876c-54ab3a13729a', 'instance_ids': ['2d153bc1-f51d-4ea6-999c-22943d6683d4', '7d35a479-0364-4a8f-9c7c-034b2556b145', '370b44c4-3a91-48f0-9e69-55b2f64ddd67', 'a6c6b1d5-68c2-42cd-8590-187609396a13', 'ef75e252-1bb0-4ab0-a458-a87069295b86', 'aad72041-7b90-4c8d-a3ea-f26d377cc820'], 'state': 'ACK_MAINTENANCE'}
87.254.192.34 - - [26/Mar/2019 18:39:44] "POST /maintenance HTTP/1.1" 200 -
2019-03-26 18:40:03,208 sample.py 164 INFO   sample app manager received data = {'session_id': 'bbabcd4f-4fe5-11e9-876c-54ab3a13729a', 'service': 'fenix', 'reply_at': '2019-03-26T16:39:02', 'allowed_actions': [], 'state': 'SCALE_IN', 'metadata': {'openstack_version': 'Rocky'}, 'project_id': 'b25dae25ccc643d093a2a4971f9c8a62', 'instance_ids': 'http://127.0.0.1:12347/v1/maintenance/bbabcd4f-4fe5-11e9-876c-54ab3a13729a/b25dae25ccc643d093a2a4971f9c8a62', 'actions_at': '2019-03-26T16:39:02', 'reply_url': 'http://127.0.0.1:12347/v1/maintenance/bbabcd4f-4fe5-11e9-876c-54ab3a13729a/b25dae25ccc643d093a2a4971f9c8a62'}
2019-03-26 18:40:03,209 sample.py 170 INFO   sample app manager state: SCALE_IN
2019-03-26 18:40:16,025 stack.py 65 INFO   stack doctor_test_maintenance UPDATE_COMPLETE
2019-03-26 18:40:16,108 sample.py 144 INFO   scaled insances from 6 to 4
2019-03-26 18:40:16,190 sample.py 240 INFO   sample app manager reply: {'session_id': 'bbabcd4f-4fe5-11e9-876c-54ab3a13729a', 'instance_ids': ['7d35a479-0364-4a8f-9c7c-034b2556b145', '2d153bc1-f51d-4ea6-999c-22943d6683d4', '370b44c4-3a91-48f0-9e69-55b2f64ddd67', 'ef75e252-1bb0-4ab0-a458-a87069295b86'], 'state': 'ACK_SCALE_IN'}
87.254.192.34 - - [26/Mar/2019 18:40:16] "POST /maintenance HTTP/1.1" 200 -
2019-03-26 18:40:18,099 sample.py 164 INFO   sample app manager received data = {'session_id': 'bbabcd4f-4fe5-11e9-876c-54ab3a13729a', 'service': 'fenix', 'reply_at': '2019-03-26T16:39:57', 'allowed_actions': ['MIGRATE', 'LIVE_MIGRATE', 'OWN_ACTION'], 'state': 'PREPARE_MAINTENANCE', 'metadata': {'openstack_version': 'Rocky'}, 'project_id': 'b25dae25ccc643d093a2a4971f9c8a62', 'instance_ids': 'http://127.0.0.1:12347/v1/maintenance/bbabcd4f-4fe5-11e9-876c-54ab3a13729a/b25dae25ccc643d093a2a4971f9c8a62', 'actions_at': '2019-03-26T16:39:57', 'reply_url': 'http://127.0.0.1:12347/v1/maintenance/bbabcd4f-4fe5-11e9-876c-54ab3a13729a/b25dae25ccc643d093a2a4971f9c8a62'}
2019-03-26 18:40:18,099 sample.py 170 INFO   sample app manager state: PREPARE_MAINTENANCE
2019-03-26 18:40:18,144 sample.py 120 INFO   get_instance_ids {'instance_ids': ['370b44c4-3a91-48f0-9e69-55b2f64ddd67']}
2019-03-26 18:40:18,145 sample.py 203 INFO   sample app manager got instances: ['370b44c4-3a91-48f0-9e69-55b2f64ddd67']
2019-03-26 18:40:18,145 sample.py 240 INFO   sample app manager reply: {'session_id': 'bbabcd4f-4fe5-11e9-876c-54ab3a13729a', 'state': 'ACK_PREPARE_MAINTENANCE', 'instance_actions': {'370b44c4-3a91-48f0-9e69-55b2f64ddd67': 'MIGRATE'}}
87.254.192.34 - - [26/Mar/2019 18:40:18] "POST /maintenance HTTP/1.1" 200 -
2019-03-26 18:41:38,391 sample.py 164 INFO   sample app manager received data = {'session_id': 'bbabcd4f-4fe5-11e9-876c-54ab3a13729a', 'service': 'fenix', 'allowed_actions': [], 'state': 'INSTANCE_ACTION_DONE', 'metadata': {}, 'project_id': 'b25dae25ccc643d093a2a4971f9c8a62', 'instance_ids': ['370b44c4-3a91-48f0-9e69-55b2f64ddd67'], 'reply_url': 'http://127.0.0.1:12347/v1/maintenance/bbabcd4f-4fe5-11e9-876c-54ab3a13729a/b25dae25ccc643d093a2a4971f9c8a62'}
2019-03-26 18:41:38,391 sample.py 170 INFO   sample app manager state: INSTANCE_ACTION_DONE
2019-03-26 18:41:38,391 sample.py 230 INFO   ['370b44c4-3a91-48f0-9e69-55b2f64ddd67']
87.254.192.34 - - [26/Mar/2019 18:41:38] "POST /maintenance HTTP/1.1" 200 -
2019-03-26 18:41:44,797 sample.py 164 INFO   sample app manager received data = {'session_id': 'bbabcd4f-4fe5-11e9-876c-54ab3a13729a', 'service': 'fenix', 'reply_at': '2019-03-26T16:41:24', 'allowed_actions': ['MIGRATE', 'LIVE_MIGRATE', 'OWN_ACTION'], 'state': 'PLANNED_MAINTENANCE', 'metadata': {'openstack_version': 'Rocky'}, 'project_id': 'b25dae25ccc643d093a2a4971f9c8a62', 'instance_ids': 'http://127.0.0.1:12347/v1/maintenance/bbabcd4f-4fe5-11e9-876c-54ab3a13729a/b25dae25ccc643d093a2a4971f9c8a62', 'actions_at': '2019-03-26T16:41:24', 'reply_url': 'http://127.0.0.1:12347/v1/maintenance/bbabcd4f-4fe5-11e9-876c-54ab3a13729a/b25dae25ccc643d093a2a4971f9c8a62'}
2019-03-26 18:41:44,797 sample.py 170 INFO   sample app manager state: PLANNED_MAINTENANCE
2019-03-26 18:41:44,844 sample.py 120 INFO   get_instance_ids {'instance_ids': ['7d35a479-0364-4a8f-9c7c-034b2556b145', '370b44c4-3a91-48f0-9e69-55b2f64ddd67']}
2019-03-26 18:41:44,844 sample.py 220 INFO   sample app manager got instances: ['7d35a479-0364-4a8f-9c7c-034b2556b145', '370b44c4-3a91-48f0-9e69-55b2f64ddd67']
2019-03-26 18:41:44,845 sample.py 240 INFO   sample app manager reply: {'session_id': 'bbabcd4f-4fe5-11e9-876c-54ab3a13729a', 'state': 'ACK_PLANNED_MAINTENANCE', 'instance_actions': {'7d35a479-0364-4a8f-9c7c-034b2556b145': 'MIGRATE', '370b44c4-3a91-48f0-9e69-55b2f64ddd67': 'MIGRATE'}}
87.254.192.34 - - [26/Mar/2019 18:41:44] "POST /maintenance HTTP/1.1" 200 -
2019-03-26 18:43:04,140 sample.py 164 INFO   sample app manager received data = {'session_id': 'bbabcd4f-4fe5-11e9-876c-54ab3a13729a', 'service': 'fenix', 'allowed_actions': [], 'state': 'INSTANCE_ACTION_DONE', 'metadata': {}, 'project_id': 'b25dae25ccc643d093a2a4971f9c8a62', 'instance_ids': ['7d35a479-0364-4a8f-9c7c-034b2556b145'], 'reply_url': 'http://127.0.0.1:12347/v1/maintenance/bbabcd4f-4fe5-11e9-876c-54ab3a13729a/b25dae25ccc643d093a2a4971f9c8a62'}
2019-03-26 18:43:04,141 sample.py 170 INFO   sample app manager state: INSTANCE_ACTION_DONE
2019-03-26 18:43:04,141 sample.py 230 INFO   ['7d35a479-0364-4a8f-9c7c-034b2556b145']
87.254.192.34 - - [26/Mar/2019 18:43:04] "POST /maintenance HTTP/1.1" 200 -
2019-03-26 18:44:18,917 sample.py 164 INFO   sample app manager received data = {'session_id': 'bbabcd4f-4fe5-11e9-876c-54ab3a13729a', 'service': 'fenix', 'allowed_actions': [], 'state': 'INSTANCE_ACTION_DONE', 'metadata': {}, 'project_id': 'b25dae25ccc643d093a2a4971f9c8a62', 'instance_ids': ['370b44c4-3a91-48f0-9e69-55b2f64ddd67'], 'reply_url': 'http://127.0.0.1:12347/v1/maintenance/bbabcd4f-4fe5-11e9-876c-54ab3a13729a/b25dae25ccc643d093a2a4971f9c8a62'}
2019-03-26 18:44:18,918 sample.py 170 INFO   sample app manager state: INSTANCE_ACTION_DONE
2019-03-26 18:44:18,918 sample.py 230 INFO   ['370b44c4-3a91-48f0-9e69-55b2f64ddd67']
87.254.192.34 - - [26/Mar/2019 18:44:18] "POST /maintenance HTTP/1.1" 200 -
2019-03-26 18:44:45,134 sample.py 164 INFO   sample app manager received data = {'session_id': 'bbabcd4f-4fe5-11e9-876c-54ab3a13729a', 'service': 'fenix', 'reply_at': '2019-03-26T16:44:25', 'allowed_actions': ['MIGRATE', 'LIVE_MIGRATE', 'OWN_ACTION'], 'state': 'PLANNED_MAINTENANCE', 'metadata': {'openstack_version': 'Rocky'}, 'project_id': 'b25dae25ccc643d093a2a4971f9c8a62', 'instance_ids': 'http://127.0.0.1:12347/v1/maintenance/bbabcd4f-4fe5-11e9-876c-54ab3a13729a/b25dae25ccc643d093a2a4971f9c8a62', 'actions_at': '2019-03-26T16:44:25', 'reply_url': 'http://127.0.0.1:12347/v1/maintenance/bbabcd4f-4fe5-11e9-876c-54ab3a13729a/b25dae25ccc643d093a2a4971f9c8a62'}
2019-03-26 18:44:45,135 sample.py 170 INFO   sample app manager state: PLANNED_MAINTENANCE
2019-03-26 18:44:45,183 sample.py 120 INFO   get_instance_ids {'instance_ids': ['2d153bc1-f51d-4ea6-999c-22943d6683d4', 'ef75e252-1bb0-4ab0-a458-a87069295b86']}
2019-03-26 18:44:45,183 sample.py 220 INFO   sample app manager got instances: ['2d153bc1-f51d-4ea6-999c-22943d6683d4', 'ef75e252-1bb0-4ab0-a458-a87069295b86']
2019-03-26 18:44:45,183 sample.py 87 INFO   Switch over to: doctor_ha_app_1 7d35a479-0364-4a8f-9c7c-034b2556b145
2019-03-26 18:44:49,363 sample.py 240 INFO   sample app manager reply: {'session_id': 'bbabcd4f-4fe5-11e9-876c-54ab3a13729a', 'state': 'ACK_PLANNED_MAINTENANCE', 'instance_actions': {'2d153bc1-f51d-4ea6-999c-22943d6683d4': 'MIGRATE', 'ef75e252-1bb0-4ab0-a458-a87069295b86': 'MIGRATE'}}
87.254.192.34 - - [26/Mar/2019 18:44:49] "POST /maintenance HTTP/1.1" 200 -
2019-03-26 18:46:03,883 sample.py 164 INFO   sample app manager received data = {'session_id': 'bbabcd4f-4fe5-11e9-876c-54ab3a13729a', 'service': 'fenix', 'allowed_actions': [], 'state': 'INSTANCE_ACTION_DONE', 'metadata': {}, 'project_id': 'b25dae25ccc643d093a2a4971f9c8a62', 'instance_ids': ['2d153bc1-f51d-4ea6-999c-22943d6683d4'], 'reply_url': 'http://127.0.0.1:12347/v1/maintenance/bbabcd4f-4fe5-11e9-876c-54ab3a13729a/b25dae25ccc643d093a2a4971f9c8a62'}
2019-03-26 18:46:03,883 sample.py 170 INFO   sample app manager state: INSTANCE_ACTION_DONE
2019-03-26 18:46:03,883 sample.py 230 INFO   ['2d153bc1-f51d-4ea6-999c-22943d6683d4']
87.254.192.34 - - [26/Mar/2019 18:46:03] "POST /maintenance HTTP/1.1" 200 -
2019-03-26 18:47:20,696 sample.py 164 INFO   sample app manager received data = {'session_id': 'bbabcd4f-4fe5-11e9-876c-54ab3a13729a', 'service': 'fenix', 'allowed_actions': [], 'state': 'INSTANCE_ACTION_DONE', 'metadata': {}, 'project_id': 'b25dae25ccc643d093a2a4971f9c8a62', 'instance_ids': ['ef75e252-1bb0-4ab0-a458-a87069295b86'], 'reply_url': 'http://127.0.0.1:12347/v1/maintenance/bbabcd4f-4fe5-11e9-876c-54ab3a13729a/b25dae25ccc643d093a2a4971f9c8a62'}
2019-03-26 18:47:20,696 sample.py 170 INFO   sample app manager state: INSTANCE_ACTION_DONE
2019-03-26 18:47:20,696 sample.py 230 INFO   ['ef75e252-1bb0-4ab0-a458-a87069295b86']
87.254.192.34 - - [26/Mar/2019 18:47:20] "POST /maintenance HTTP/1.1" 200 -
2019-03-26 18:47:46,780 sample.py 164 INFO   sample app manager received data = {'session_id': 'bbabcd4f-4fe5-11e9-876c-54ab3a13729a', 'service': 'fenix', 'reply_at': '2019-03-26T16:46:46', 'allowed_actions': [], 'state': 'MAINTENANCE_COMPLETE', 'metadata': {'openstack_version': 'Rocky'}, 'project_id': 'b25dae25ccc643d093a2a4971f9c8a62', 'instance_ids': 'http://127.0.0.1:12347/v1/maintenance/bbabcd4f-4fe5-11e9-876c-54ab3a13729a/b25dae25ccc643d093a2a4971f9c8a62', 'actions_at': '2019-03-26T16:46:46', 'reply_url': 'http://127.0.0.1:12347/v1/maintenance/bbabcd4f-4fe5-11e9-876c-54ab3a13729a/b25dae25ccc643d093a2a4971f9c8a62'}
2019-03-26 18:47:46,781 sample.py 170 INFO   sample app manager state: MAINTENANCE_COMPLETE
2019-03-26 18:48:27,376 stack.py 65 INFO   stack doctor_test_maintenance UPDATE_COMPLETE
2019-03-26 18:48:27,463 sample.py 144 INFO   scaled insances from 4 to 6
2019-03-26 18:48:27,463 sample.py 240 INFO   sample app manager reply: {'session_id': 'bbabcd4f-4fe5-11e9-876c-54ab3a13729a', 'state': 'ACK_MAINTENANCE_COMPLETE'}
87.254.192.34 - - [26/Mar/2019 18:48:27] "POST /maintenance HTTP/1.1" 200 -
2019-03-26 18:48:37,001 maintenance.py 183 INFO   remove maintenance session bbabcd4f-4fe5-11e9-876c-54ab3a13729a.......
2019-03-26 18:48:37,356 maintenance.py 224 INFO   maintenance bbabcd4f-4fe5-11e9-876c-54ab3a13729a ended with state MAINTENANCE_DONE
2019-03-26 18:48:37,356 main.py 118 INFO   doctor maintenance complete.......
2019-03-26 18:48:37,356 sample.py 36 INFO   sample app manager stop......
2019-03-26 18:48:37,361 sample.py 247 INFO   shutdown app manager server at 1553618917.3611414
87.254.192.34 - - [26/Mar/2019 18:48:37] "POST /shutdown HTTP/1.1" 200 -
2019-03-26 18:48:37,363 sample.py 91 INFO   sample inspector stop......
2019-03-26 18:48:37,368 sample.py 253 INFO   shutdown inspector app server at 1553618917.3683898
87.254.192.34 - - [26/Mar/2019 18:48:37] "POST /events/shutdown HTTP/1.1" 200 -
2019-03-26 18:48:37,371 maintenance.py 236 INFO   stack delete start.......
2019-03-26 18:48:54,830 stack.py 65 INFO   stack doctor_test_maintenance DELETE_COMPLETE
2019-03-26 18:48:54,831 apex.py 166 INFO   restore apply patches start......
2019-03-26 18:48:54,914 image.py 71 INFO   image delete start.......
2019-03-26 18:48:55,286 base.py 218 INFO   Command sudo python restore_compute_config.py output ['restoring nova.bak.']
2019-03-26 18:48:55,324 base.py 218 INFO   Command sudo python restore_compute_config.py output ['nova.bak does not exist.']
2019-03-26 18:48:55,326 base.py 218 INFO   Command sudo python restore_config.py output ['restore', 'restore: /var/lib/config-data/puppet-generated/ceilometer/etc/ceilometer/event_definitions.yaml', 'Bak_file empty, so removing also: /var/lib/config-data/puppet-generated/ceilometer/etc/ceilometer/event_definitions.yaml']
2019-03-26 18:48:56,661 image.py 76 INFO   image delete end.......
2019-03-26 18:48:56,662 user.py 163 INFO   user delete start......
2019-03-26 18:48:56,662 user.py 156 INFO   restore default quota......
2019-03-26 18:48:57,117 base.py 218 INFO   Command sudo python restore_aodh.py output []
2019-03-26 18:48:57,245 base.py 218 INFO   Command sudo python restore_compute_config.py output ['nova.bak does not exist.']
2019-03-26 18:48:58,633 user.py 187 INFO   user delete end......
___________________________________ summary ____________________________________
  py34: commands succeeded
  congratulations :)
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 8275 killed;
[ssh-agent] Stopped.
Archiving artifacts
No emails were triggered.
[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
Cannot delete workspace: java.lang.NoClassDefFoundError: Could not initialize class sun.nio.fs.UnixCopyFile
Option not to fail the build is turned on, so let's continue
Finished: SUCCESS