SuccessConsole Output

Started by upstream project "doctor-verify-master" build number 152
originally caused by:
 Triggered by Gerrit: https://gerrit.opnfv.org/gerrit/50973
[EnvInject] - Loading node environment variables.
Building remotely on nokia-pod1 (nokia opnfv-sysinfo doctor-apex-x86_64) in workspace /home/jenkins/opnfv_slave_root/workspace/doctor-verify-all-apex-sample-x86_64-master
[ssh-agent] Looking for ssh-agent implementation...
[ssh-agent]   Exec ssh-agent (binary ssh-agent on a remote machine)
$ ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-uXPaBEwzjS0v/agent.28294
SSH_AGENT_PID=28296
[ssh-agent] Started.
Running ssh-add (command line suppressed)
Identity added: /home/jenkins/opnfv_slave_root/workspace/doctor-verify-all-apex-sample-x86_64-master@tmp/private_key_2021587745117371122.key (/home/jenkins/opnfv_slave_root/workspace/doctor-verify-all-apex-sample-x86_64-master@tmp/private_key_2021587745117371122.key)
[ssh-agent] Using credentials jenkins-ci (Jenkins Master SSH)
using credential d42411ac011ad6f3dd2e1fa34eaa5d87f910eb2e
Wiping out workspace first.
Cloning the remote Git repository
Cloning repository https://gerrit.opnfv.org/gerrit/doctor
 > git init /home/jenkins/opnfv_slave_root/workspace/doctor-verify-all-apex-sample-x86_64-master # timeout=10
Fetching upstream changes from https://gerrit.opnfv.org/gerrit/doctor
 > git --version # timeout=10
using GIT_SSH to set credentials Jenkins Master SSH
 > git fetch --tags --progress https://gerrit.opnfv.org/gerrit/doctor +refs/heads/*:refs/remotes/origin/* # timeout=15
 > git config remote.origin.url https://gerrit.opnfv.org/gerrit/doctor # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://gerrit.opnfv.org/gerrit/doctor # timeout=10
Fetching upstream changes from https://gerrit.opnfv.org/gerrit/doctor
using GIT_SSH to set credentials Jenkins Master SSH
 > git fetch --tags --progress https://gerrit.opnfv.org/gerrit/doctor refs/changes/73/50973/13 # timeout=15
Checking out Revision c942be45f70ec9f1a7dc78a58ba5ebd248eaf4ae (refs/changes/73/50973/13)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f c942be45f70ec9f1a7dc78a58ba5ebd248eaf4ae # timeout=15
Commit message: "[Don't merge] test for CI"
 > git rev-parse FETCH_HEAD^{commit} # timeout=10
 > git rev-list --no-walk 7bba7d67beec7b3db26ab1319bf4080565053096 # timeout=10
No emails were triggered.
[doctor-verify-all-apex-sample-x86_64-master] $ /usr/bin/env bash /tmp/jenkins329163587849018688.sh
Gathering IP information for Apex installer VM
 6     undercloud                     running
Installer VM detected
Installer ip is 192.168.122.40
fetch_os_creds.info: Fetching rc file...
fetch_os_creds.info: Verifying connectivity to 192.168.122.40...
fetch_os_creds.info: 192.168.122.40 is reachable!
fetch_os_creds.info: ... from Instack VM 192.168.122.40...
Warning: Permanently added '192.168.122.40' (ECDSA) to the list of known hosts.
-------- Credentials: --------
# Clear any old environment that may conflict.
for key in $( set | awk '{FS="="}  /^OS_/ {print $1}' ); do unset $key ; done
export OS_NO_CACHE=True
export COMPUTE_API_VERSION=1.1
export OS_USERNAME=admin
export no_proxy=,192.168.37.19,192.0.2.4
export OS_USER_DOMAIN_NAME=Default
export OS_VOLUME_API_VERSION=3
export OS_CLOUDNAME=overcloud
export OS_AUTH_URL=http://192.168.37.19:5000/v3
export NOVA_VERSION=1.1
export OS_IMAGE_API_VERSION=2
#export OS_PASSWORD=N9flPG9TD7lwww7vsVewDHyBK
export OS_PASSWORD=admin
export OS_PROJECT_DOMAIN_NAME=Default
export OS_IDENTITY_API_VERSION=3
export OS_PROJECT_NAME=admin
export OS_AUTH_TYPE=password
export PYTHONWARNINGS="ignore:Certificate has no, ignore:A true SSLContext object is not available"

# Add OS_CLOUDNAME to PS1
if [ -z "${CLOUDPROMPT_ENABLED:-}" ]; then
    export PS1=${PS1:-""}
    export PS1=\${OS_CLOUDNAME:+"(\$OS_CLOUDNAME)"}\ $PS1
    export CLOUDPROMPT_ENABLED=1
fi
export OS_PROJECT_ID=6e76c202356a46cbbd626ef7c00efc0c
export OS_TENANT_NAME=admin
export OS_REGION_NAME=regionOne
[doctor-verify-all-apex-sample-x86_64-master] $ /bin/sh -xe /tmp/jenkins497964453086355173.sh
+ source /home/jenkins/opnfv-openrc.sh
+++ set
+++ awk '{FS="="}  /^OS_/ {print $1}'
++ export OS_NO_CACHE=True
++ OS_NO_CACHE=True
++ export COMPUTE_API_VERSION=1.1
++ COMPUTE_API_VERSION=1.1
++ export OS_USERNAME=admin
++ OS_USERNAME=admin
++ export no_proxy=,192.168.37.19,192.0.2.4
++ no_proxy=,192.168.37.19,192.0.2.4
++ export OS_USER_DOMAIN_NAME=Default
++ OS_USER_DOMAIN_NAME=Default
++ export OS_VOLUME_API_VERSION=3
++ OS_VOLUME_API_VERSION=3
++ export OS_CLOUDNAME=overcloud
++ OS_CLOUDNAME=overcloud
++ export OS_AUTH_URL=http://192.168.37.19:5000/v3
++ OS_AUTH_URL=http://192.168.37.19:5000/v3
++ export NOVA_VERSION=1.1
++ NOVA_VERSION=1.1
++ export OS_IMAGE_API_VERSION=2
++ OS_IMAGE_API_VERSION=2
++ export OS_PASSWORD=admin
++ OS_PASSWORD=admin
++ export OS_PROJECT_DOMAIN_NAME=Default
++ OS_PROJECT_DOMAIN_NAME=Default
++ export OS_IDENTITY_API_VERSION=3
++ OS_IDENTITY_API_VERSION=3
++ export OS_PROJECT_NAME=admin
++ OS_PROJECT_NAME=admin
++ export OS_AUTH_TYPE=password
++ OS_AUTH_TYPE=password
++ export 'PYTHONWARNINGS=ignore:Certificate has no, ignore:A true SSLContext object is not available'
++ PYTHONWARNINGS='ignore:Certificate has no, ignore:A true SSLContext object is not available'
++ '[' -z '' ']'
++ export PS1=
++ PS1=
++ export 'PS1=${OS_CLOUDNAME:+($OS_CLOUDNAME)} '
++ PS1='${OS_CLOUDNAME:+($OS_CLOUDNAME)} '
++ export CLOUDPROMPT_ENABLED=1
++ CLOUDPROMPT_ENABLED=1
++ export OS_PROJECT_ID=6e76c202356a46cbbd626ef7c00efc0c
++ OS_PROJECT_ID=6e76c202356a46cbbd626ef7c00efc0c
++ export OS_TENANT_NAME=admin
++ OS_TENANT_NAME=admin
++ export OS_REGION_NAME=regionOne
++ OS_REGION_NAME=regionOne
+ '[' -f /home/jenkins/os_cacert ']'
+ source /home/jenkins/opnfv-installer.sh
++ export INSTALLER_TYPE=apex
++ INSTALLER_TYPE=apex
++ export INSTALLER_IP=192.168.122.40
++ INSTALLER_IP=192.168.122.40
++ export SSH_KEY=/home/jenkins/installer_key_file
++ SSH_KEY=/home/jenkins/installer_key_file
+ sudo -E tox -e py34
py34 create: /home/jenkins/opnfv_slave_root/workspace/doctor-verify-all-apex-sample-x86_64-master/.tox/py34
py34 installdeps: -r/home/jenkins/opnfv_slave_root/workspace/doctor-verify-all-apex-sample-x86_64-master/requirements.txt
py34 develop-inst: /home/jenkins/opnfv_slave_root/workspace/doctor-verify-all-apex-sample-x86_64-master
py34 installed: DEPRECATION: Python 3.4 support has been deprecated. pip 19.1 will be the last one supporting it. Please upgrade your Python as Python 3.4 won't be maintained after March 2019 (cf PEP 429).,amqp==2.2.1,aodhclient==0.9.0,appdirs==1.4.3,asn1crypto==0.22.0,Babel==2.3.4,bcrypt==3.1.3,cachetools==2.0.0,certifi==2017.4.17,cffi==1.10.0,chardet==3.0.4,click==6.7,cliff==2.8.3,cmd2==0.7.5,contextlib2==0.5.5,cryptography==2.0.2,debtcollector==1.17.2,deprecation==1.0.1,-e git+https://gerrit.opnfv.org/gerrit/doctor@c942be45f70ec9f1a7dc78a58ba5ebd248eaf4ae#egg=doctor_tests,enum-compat==0.0.2,eventlet==0.20.0,fasteners==0.14.1,flake8==2.5.5,Flask==0.12.2,futurist==1.3.2,greenlet==0.4.15,idna==2.5,iso8601==0.1.11,itsdangerous==0.24,Jinja2==2.9.6,jsonpatch==1.16,jsonpointer==1.10,jsonschema==2.6.0,keystoneauth1==3.1.0,kombu==4.1.0,MarkupSafe==1.0,mccabe==0.4.0,monotonic==1.3,msgpack-python==0.4.8,netaddr==0.7.19,netifaces==0.10.6,openstacksdk==0.9.17,os-client-config==1.28.0,osc-lib==1.7.1,oslo.concurrency==3.21.2,oslo.config==4.11.2,oslo.context==2.17.2,oslo.i18n==3.17.1,oslo.log==3.30.3,oslo.messaging==5.30.8,oslo.middleware==3.30.2,oslo.serialization==2.20.2,oslo.service==1.25.2,oslo.utils==3.28.4,oslo.versionedobjects==1.26.3,paramiko==2.2.1,Paste==2.0.3,PasteDeploy==1.5.2,pbr==3.1.1,pep8==1.7.1,pika==0.10.0,pika-pool==0.1.3,positional==1.1.2,prettytable==0.7.2,pyasn1==0.3.1,pycparser==2.18,pyflakes==1.0.0,pyinotify==0.9.6,PyNaCl==1.1.2,pyOpenSSL==17.2.0,pyparsing==2.2.0,pyperclip==1.5.27,python-ceilometerclient==2.9.0,python-cinderclient==3.1.1,python-congressclient==1.8.1,python-dateutil==2.6.1,python-glanceclient==2.8.0,python-heatclient==1.11.1,python-keystoneclient==3.13.0,python-neutronclient==6.5.0,python-novaclient==9.1.2,python-openstackclient==3.12.2,python-swiftclient==3.4.1,python-vitrageclient==1.4.1,pytz==2017.2,PyYAML==3.12,repoze.lru==0.6,requests==2.18.2,requestsexceptions==1.3.0,rfc3986==1.1.0,Routes==2.4.1,scp==0.10.2,simplejson==3.11.1,six==1.10.0,statsd==3.2.1,stevedore==1.25.2,tenacity==4.4.0,urllib3==1.22,vine==1.1.4,virtualenv==15.1.0,warlock==1.2.0,WebOb==1.7.3,Werkzeug==0.12.2,wrapt==1.10.10
py34 runtests: PYTHONHASHSEED='362373045'
py34 runtests: commands[0] | doctor-test
/home/jenkins/opnfv_slave_root/workspace/doctor-verify-all-apex-sample-x86_64-master/.tox/py34/lib/python3.4/site-packages/paramiko/client.py:711: UserWarning: Unknown ssh-ed25519 host key for 192.168.122.40: b'3dc1628c6fe2819031608f780639f40a'
  key.get_fingerprint())))
2019-04-20 10:28:13,865 main.py 130 INFO   doctor test starting.......
2019-04-20 10:28:13,866 apex.py 43 INFO   Setup Apex installer start......
2019-04-20 10:28:13,866 base.py 113 INFO   Get SSH keys from apex installer......
2019-04-20 10:28:14,161 apex.py 67 INFO   Get overcloud config details from Apex installer......
2019-04-20 10:28:14,162 base.py 174 INFO   Run command=source stackrc; nova list | grep ' overcloud-' in apex installer......
2019-04-20 10:28:17,773 base.py 183 INFO   Output=['| 2d0aedd7-4e30-4fee-bbd2-0feb8b1078c4 | overcloud-controller-0  | ACTIVE | -          | Running     | ctlplane=192.0.2.9 |', '| 37ceec17-a130-4826-87f0-67699d3350f5 | overcloud-novacompute-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.6 |', '| 416383de-15fb-477a-9a6c-f0b21df9be70 | overcloud-novacompute-1 | ACTIVE | -          | Running     | ctlplane=192.0.2.7 |', '| 0f025a92-3365-4c3b-951e-dab9a6d9ca22 | overcloud-novacompute-2 | ACTIVE | -          | Running     | ctlplane=192.0.2.3 |'] command=source stackrc; nova list | grep ' overcloud-' in apex installer
2019-04-20 10:28:17,773 base.py 188 INFO   Check command=grep docker /home/stack/deploy_command return in apex installer......
2019-04-20 10:28:17,843 base.py 191 INFO   return 0
2019-04-20 10:28:17,843 apex.py 80 INFO   controller_ips:['192.0.2.9']
2019-04-20 10:28:17,844 apex.py 81 INFO   compute_ips:['192.0.2.6', '192.0.2.7', '192.0.2.3']
2019-04-20 10:28:17,844 apex.py 82 INFO   use_containers:True
2019-04-20 10:28:19,166 apex.py 102 INFO   Set apply patches start......
/home/jenkins/opnfv_slave_root/workspace/doctor-verify-all-apex-sample-x86_64-master/.tox/py34/lib/python3.4/site-packages/paramiko/client.py:711: UserWarning: Unknown ssh-ed25519 host key for 192.0.2.9: b'b3b023c9e950672f64d119fecdae83fd'
  key.get_fingerprint())))
2019-04-20 10:28:19,623 base.py 218 INFO   Command sudo python set_config.py output ['Add event notifier in ceilometer', 'NOTE: add compute.instance.update to event_definitions.yaml', 'NOTE: add maintenance.scheduled to event_definitions.yaml', 'NOTE: add maintenance.host to event_definitions.yaml']
2019-04-20 10:28:21,290 base.py 218 INFO   Command sudo python restart_aodh.py output []
2019-04-20 10:28:21,458 base.py 218 INFO   Command sudo python set_compute_config.py output []
2019-04-20 10:28:24,253 apex.py 148 INFO   Set apply patches start......
/home/jenkins/opnfv_slave_root/workspace/doctor-verify-all-apex-sample-x86_64-master/.tox/py34/lib/python3.4/site-packages/paramiko/client.py:711: UserWarning: Unknown ssh-ed25519 host key for 192.0.2.6: b'5ffdaf1f169876d8c9ac845458601952'
  key.get_fingerprint())))
/home/jenkins/opnfv_slave_root/workspace/doctor-verify-all-apex-sample-x86_64-master/.tox/py34/lib/python3.4/site-packages/paramiko/client.py:711: UserWarning: Unknown ssh-ed25519 host key for 192.0.2.7: b'50858b8f4760d0c962bbdcb845047215'
  key.get_fingerprint())))
/home/jenkins/opnfv_slave_root/workspace/doctor-verify-all-apex-sample-x86_64-master/.tox/py34/lib/python3.4/site-packages/paramiko/client.py:711: UserWarning: Unknown ssh-ed25519 host key for 192.0.2.3: b'890a9b151267bafe34f53791df2c09f2'
  key.get_fingerprint())))
2019-04-20 10:28:24,617 base.py 218 INFO   Command sudo python set_compute_config.py output []
2019-04-20 10:28:24,695 base.py 218 INFO   Command sudo python set_compute_config.py output []
2019-04-20 10:28:24,798 base.py 218 INFO   Command sudo python set_compute_config.py output []
2019-04-20 10:28:29,481 base.py 63 INFO   Setup ssh stunnel in apex installer......
2019-04-20 10:28:29,482 base.py 76 INFO   tunnel for port 12346
2019-04-20 10:28:29,489 base.py 76 INFO   tunnel for port 12348
2019-04-20 10:28:29,494 base.py 76 INFO   tunnel for port 12345
2019-04-20 10:28:29,500 image.py 48 INFO   image create start......
2019-04-20 10:28:34,363 image.py 68 INFO   image create end......
2019-04-20 10:28:34,363 user.py 70 INFO   user create start......
2019-04-20 10:28:34,621 user.py 86 INFO   create project......
2019-04-20 10:28:35,032 user.py 95 INFO   test project <Project description=, domain_id=default, enabled=True, id=2ca07a0f752b4e4eba91e7b2137bfe10, is_domain=False, links={'self': 'http://192.0.2.4:35357/v3/projects/2ca07a0f752b4e4eba91e7b2137bfe10'}, name=doctor, parent_id=default, tags=[]>
2019-04-20 10:28:35,301 user.py 103 INFO   create user......
2019-04-20 10:28:35,974 user.py 113 INFO   test user <User domain_id=default, enabled=True, id=e359d040ac5043c58bd3c1395abfd60e, links={'self': 'http://192.0.2.4:35357/v3/users/e359d040ac5043c58bd3c1395abfd60e'}, name=doctor, options={}, password_expires_at=None>
2019-04-20 10:28:36,224 user.py 127 INFO   role _member_ already created......
2019-04-20 10:28:36,225 user.py 128 INFO   test role <Role domain_id=None, id=9fe2ff9ee4384b1894a90878d3e92bab, links={'self': 'http://192.0.2.4:35357/v3/roles/9fe2ff9ee4384b1894a90878d3e92bab'}, name=_member_>
2019-04-20 10:28:37,736 user.py 78 INFO   user create end......
2019-04-20 10:28:37,736 main.py 55 INFO   doctor fault management test starting.......
2019-04-20 10:28:38,675 fault_management.py 65 INFO   fault management setup......
2019-04-20 10:28:38,676 user.py 190 INFO   quota update start......
2019-04-20 10:28:38,676 user.py 206 INFO   default quota update start......
2019-04-20 10:28:39,652 user.py 217 INFO   user quota update start......
2019-04-20 10:28:40,010 user.py 230 INFO   quota update end......
2019-04-20 10:28:40,010 network.py 41 INFO   network create start.......
2019-04-20 10:28:41,615 network.py 47 INFO   network create end.......
2019-04-20 10:28:41,615 network.py 49 INFO   subnet create start.......
2019-04-20 10:28:42,549 network.py 58 INFO   subnet create end.......
2019-04-20 10:28:42,549 instance.py 52 INFO   instance create start......
2019-04-20 10:28:47,904 instance.py 74 INFO   instance create end......
2019-04-20 10:28:47,905 instance.py 93 INFO   wait for vm launch start......
2019-04-20 10:29:02,234 instance.py 111 INFO   wait for vm launch end......
2019-04-20 10:29:02,235 alarm.py 45 INFO   alarm create start......
2019-04-20 10:29:05,084 alarm.py 81 INFO   alarm create end......
2019-04-20 10:29:05,085 sample.py 85 INFO   sample inspector start......
2019-04-20 10:29:06,760 sample.py 26 INFO   sample consumer start......
 * Running on http://0.0.0.0:12346/ (Press CTRL+C to quit)
 * Running on http://0.0.0.0:12345/ (Press CTRL+C to quit)
2019-04-20 10:29:08,413 apex.py 86 INFO   Get host ip by hostname=overcloud-novacompute-2.opnfvlf.org from Apex installer......
2019-04-20 10:29:08,413 base.py 174 INFO   Run command=source stackrc; nova show overcloud-novacompute-2 | awk '/ ctlplane network /{print $5}' in apex installer......
2019-04-20 10:29:13,705 base.py 183 INFO   Output=['192.0.2.3'] command=source stackrc; nova show overcloud-novacompute-2 | awk '/ ctlplane network /{print $5}' in apex installer
2019-04-20 10:29:13,706 fault_management.py 118 INFO   Get host info(name:overcloud-novacompute-2.opnfvlf.org, ip:192.0.2.3) which vm(doctor_vm0) launched at
2019-04-20 10:29:13,706 sample.py 30 INFO   sample monitor start......
2019-04-20 10:29:13,707 sample.py 85 INFO   Starting Pinger host_name(overcloud-novacompute-2.opnfvlf.org), host_ip(192.0.2.3)
2019-04-20 10:31:13,807 fault_management.py 89 INFO   fault management start......
2019-04-20 10:31:13,808 base.py 113 INFO   Get SSH keys from apex installer......
2019-04-20 10:31:13,808 base.py 117 INFO   Already have SSH keys from apex installer......
2019-04-20 10:31:13,882 utils.py 91 INFO   Copy /home/jenkins/opnfv_slave_root/workspace/doctor-verify-all-apex-sample-x86_64-master/doctor_tests/disable_network.sh -> disable_network.sh
2019-04-20 10:31:14,125 utils.py 72 INFO   Executing: bash disable_network.sh > disable_network.log 2>&1 &
2019-04-20 10:31:14,176 utils.py 86 INFO   *** SUCCESSFULLY run command bash disable_network.sh > disable_network.log 2>&1 &
2019-04-20 10:31:14,177 fault_management.py 91 INFO   fault management end......
2019-04-20 10:31:15,341 sample.py 98 INFO   doctor monitor detected at 1555745475.341308
2019-04-20 10:31:15,341 sample.py 41 INFO   sample monitor report error......
2019-04-20 10:31:15,348 sample.py 238 INFO   event posted in sample inspector at 1555745475.348224
2019-04-20 10:31:15,348 sample.py 239 INFO   sample inspector = <doctor_tests.inspector.sample.SampleInspector object at 0x7f3f4e844f98>
2019-04-20 10:31:15,349 sample.py 241 INFO   sample inspector received data = b'[{"time": "2019-04-20T10:31:15.341913", "details": {"monitor": "monitor_sample", "monitor_event_id": "monitor_sample_event1", "status": "down", "hostname": "overcloud-novacompute-2.opnfvlf.org"}, "type": "compute.host.down"}]'
2019-04-20 10:31:15,398 sample.py 196 INFO   doctor compute.instance.update vm(<Server: doctor_vm0>) error 1555745475.398643
2019-04-20 10:31:15,416 sample.py 165 INFO   doctor mark host(overcloud-novacompute-2.opnfvlf.org) down at 1555745475.4161773
2019-04-20 10:31:15,536 sample.py 58 INFO   doctor consumer notified at 1555745475.5369563
2019-04-20 10:31:15,537 sample.py 61 INFO   sample consumer received data = {'reason': 'Event <id=0eac0d69-39a5-40be-a8b7-4c49aa53a659,event_type=compute.instance.update> hits the query <query=[{"field": "traits.instance_id", "op": "eq", "type": "", "value": "15073623-ef5f-4945-8958-ac37698c8ee5"}, {"field": "traits.state", "op": "eq", "type": "", "value": "error"}]>.', 'previous': 'insufficient data', 'alarm_id': '6fe697d2-9f7c-4a94-a4bb-dfd1d86f8c3e', 'severity': 'moderate', 'current': 'alarm', 'reason_data': {'event': {'event_type': 'compute.instance.update', 'message_id': '0eac0d69-39a5-40be-a8b7-4c49aa53a659', 'generated': '2019-04-20T07:31:15.350473', 'raw': {}, 'traits': [['resource_id', 1, '15073623-ef5f-4945-8958-ac37698c8ee5'], ['service', 1, 'sample'], ['state', 1, 'error'], ['project_id', 1, '2ca07a0f752b4e4eba91e7b2137bfe10'], ['instance_id', 1, '15073623-ef5f-4945-8958-ac37698c8ee5'], ['tenant_id', 1, '2ca07a0f752b4e4eba91e7b2137bfe10']], 'message_signature': '0e290bf4ae27e4aecb38d8539d4b36e5e53444d2b8ee4e98d2dc2bdaa3b3d940'}, 'type': 'event'}, 'alarm_name': 'doctor_alarm0'}
87.254.192.34 - - [20/Apr/2019 10:31:15] "POST /failure HTTP/1.1" 200 -
2019-04-20 10:31:15,564 sample.py 176 INFO   doctor mark vm(<Server: doctor_vm0>) error at 1555745475.5649312
87.254.192.34 - - [20/Apr/2019 10:31:15] "PUT /events HTTP/1.1" 200 -
2019-04-20 10:31:15,568 sample.py 101 INFO   ping timeout, quit monitoring...
2019-04-20 10:31:44,248 fault_management.py 185 INFO   doctor fault management notification_time=0.195648193359375
2019-04-20 10:31:44,248 fault_management.py 188 INFO   doctor fault management test successfully
2019-04-20 10:31:44,249 fault_management.py 198 INFO   run doctor fault management profile.......
2019-04-20 10:31:44,249 base.py 113 INFO   Get SSH keys from apex installer......
2019-04-20 10:31:44,249 base.py 117 INFO   Already have SSH keys from apex installer......
2019-04-20 10:31:51,328 utils.py 91 INFO   Copy disable_network.log -> /home/jenkins/opnfv_slave_root/workspace/doctor-verify-all-apex-sample-x86_64-master/doctor_tests/disable_network.log
2019-04-20 10:31:51,569 fault_management.py 155 INFO   Get the disable_netork.log fromdown_host(host_name:overcloud-novacompute-2.opnfvlf.org, host_ip:192.0.2.3)
2019-04-20 10:31:51,571 profiler_poc.py 97 INFO   
Total time cost: 111(ms)
==============================================================================>
       |Monitor|Inspector           |Controller|Notifier|Evaluator           |
       |-84    |75                  |?         |?       |?                   |
       |       |      |      |      |          |        |      |      |      |
link down:0    |      |      |      |          |        |      |      |      |
     raw failure:-84  |      |      |          |        |      |      |      |
         found affected:?    |      |          |        |      |      |      |
                  set VM error:139  |          |        |      |      |      |
                         marked host down:-9   |        |      |      |      |
                               notified VM error:?      |      |      |      |
                                        transformed event:?    |      |      |
                                                 evaluated event:?    |      |
                                                            fired alarm:?    |
                                                                received alarm:111  

2019-04-20 10:31:51,571 fault_management.py 94 INFO   fault management cleanup......
2019-04-20 10:31:51,571 fault_management.py 136 INFO   Already get the disable_netork.log from down_host......
2019-04-20 10:31:53,667 sample.py 91 INFO   sample inspector stop......
2019-04-20 10:31:53,707 sample.py 253 INFO   shutdown inspector app server at 1555745513.7078812
87.254.192.34 - - [20/Apr/2019 10:31:53] "POST /events/shutdown HTTP/1.1" 200 -
2019-04-20 10:31:53,710 sample.py 35 INFO   sample monitor stop......
2019-04-20 10:31:53,711 sample.py 108 INFO   Stopping Pinger host_name(overcloud-novacompute-2.opnfvlf.org), host_ip(192.0.2.3)
2019-04-20 10:31:53,711 sample.py 31 INFO   sample consumer stop......
2019-04-20 10:31:53,715 sample.py 66 INFO   shutdown consumer app server at 1555745513.7157629
87.254.192.34 - - [20/Apr/2019 10:31:53] "POST /shutdown HTTP/1.1" 200 -
2019-04-20 10:31:53,718 alarm.py 84 INFO   alarm delete start.......
2019-04-20 10:31:55,017 alarm.py 93 INFO   alarm delete end.......
2019-04-20 10:31:55,018 instance.py 77 INFO   instance delete start.......
2019-04-20 10:32:50,474 instance.py 90 INFO   instance delete end.......
2019-04-20 10:32:50,474 network.py 61 INFO   subnet delete start.......
2019-04-20 10:32:51,744 network.py 64 INFO   subnet delete end.......
2019-04-20 10:32:51,744 network.py 66 INFO   network delete start.......
2019-04-20 10:32:53,483 network.py 69 INFO   network delete end.......
2019-04-20 10:32:54,398 main.py 104 INFO   doctor maintenance test starting.......
2019-04-20 10:32:56,421 maintenance.py 62 INFO   checking hypervisors.......
2019-04-20 10:32:56,421 maintenance.py 95 INFO   testing 3 computes with 32 vcpus each
2019-04-20 10:32:56,421 maintenance.py 98 INFO   testing 2 actstdby and 4 noredundancy instances
2019-04-20 10:32:56,422 user.py 190 INFO   quota update start......
2019-04-20 10:32:56,422 user.py 206 INFO   default quota update start......
2019-04-20 10:32:56,616 user.py 217 INFO   user quota update start......
2019-04-20 10:32:56,961 user.py 230 INFO   quota update end......
2019-04-20 10:32:58,000 maintenance.py 117 INFO   creating maintenance stack.......
2019-04-20 10:32:58,001 maintenance.py 118 INFO   parameters: {'ext_net': 'external', 'nonha_intances': 4, 'ha_intances': 2, 'maint_image': 'cirros', 'flavor_vcpus': 16}
2019-04-20 10:33:36,460 stack.py 65 INFO   stack doctor_test_maintenance CREATE_COMPLETE
2019-04-20 10:33:36,461 sample.py 36 INFO   sample admin tool start......
2019-04-20 10:33:36,461 sample.py 31 INFO   sample app manager start......
 * Running on http://0.0.0.0:12347/ (Press CTRL+C to quit)
2019-04-20 10:33:37,983 sample.py 85 INFO   sample inspector start......
 * Running on http://0.0.0.0:12348/ (Press CTRL+C to quit)
2019-04-20 10:33:39,416 main.py 112 INFO   wait aodh for 120s.......
 * Running on http://0.0.0.0:12345/ (Press CTRL+C to quit)
2019-04-20 10:35:39,517 maintenance.py 134 INFO   start maintenance.......
2019-04-20 10:35:39,583 sample.py 675 INFO   maintenance message: {'hosts': ['overcloud-novacompute-2.opnfvlf.org', 'overcloud-novacompute-0.opnfvlf.org', 'overcloud-novacompute-1.opnfvlf.org'], 'state': 'MAINTENANCE', 'workflow': 'default', 'metadata': {'openstack_version': 'Rocky'}, 'maintenance_at': '2019-04-20 07:36:09'}
2019-04-20 10:35:39,584 sample.py 677 INFO   creating session: e5313ed2-633e-11e9-93d4-54ab3a1411e8
2019-04-20 10:35:39,588 sample.py 77 INFO   transport_url rabbit://guest:p1LVYe6k9JkpmYbh75q34Wii8@192.0.2.9:5672/?ssl=0
2019-04-20 10:35:39,591 sample.py 90 INFO   Admin tool session e5313ed2-633e-11e9-93d4-54ab3a1411e8 initialized
2019-04-20 10:35:39,592 sample.py 481 INFO   --==session e5313ed2-633e-11e9-93d4-54ab3a1411e8: processing state MAINTENANCE==--
87.254.192.34 - - [20/Apr/2019 10:35:39] "POST /maintenance HTTP/1.1" 200 -
2019-04-20 10:35:41,263 sample.py 436 INFO   
overcloud-novacompute-2.opnfvlf.org:
  2ca07a0f752b4e4eba91e7b2137bfe10:
    f3bf79ac-9fa9-4e74-bd84-5284729ace29: doctor_nonha_app_0
    2d9f1422-b81a-4efc-9616-4ca0e801367c: doctor_nonha_app_3
overcloud-novacompute-0.opnfvlf.org:
  2ca07a0f752b4e4eba91e7b2137bfe10:
    d5d2845c-2705-4f24-af0f-5ca4426dd7bd: doctor_nonha_app_1
    26dab7e9-97cd-4edb-879b-782e1ac11698: doctor_ha_app_1
overcloud-novacompute-1.opnfvlf.org:
  2ca07a0f752b4e4eba91e7b2137bfe10:
    a77f123a-4318-48ee-a5b1-27b13e89fc28: doctor_nonha_app_2
    ba29f300-4bb5-4e3e-97d8-c6ebdc8e1493: doctor_ha_app_0

2019-04-20 10:35:42,138 sample.py 179 INFO   
MAINTENANCE to project 2ca07a0f752b4e4eba91e7b2137bfe10

2019-04-20 10:35:42,350 sample.py 164 INFO   sample app manager received data = {'allowed_actions': [], 'session_id': 'e5313ed2-633e-11e9-93d4-54ab3a1411e8', 'instance_ids': 'http://0.0.0.0:12347/maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8/2ca07a0f752b4e4eba91e7b2137bfe10', 'metadata': {'openstack_version': 'Rocky'}, 'state': 'MAINTENANCE', 'reply_url': 'http://0.0.0.0:12347/maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8/2ca07a0f752b4e4eba91e7b2137bfe10', 'service': 'admin_tool', 'actions_at': '2019-04-20T07:36:09', 'project_id': '2ca07a0f752b4e4eba91e7b2137bfe10'}
2019-04-20 10:35:42,350 sample.py 170 INFO   sample app manager state: MAINTENANCE
87.254.192.34 - - [20/Apr/2019 10:35:42] "GET /maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8/2ca07a0f752b4e4eba91e7b2137bfe10 HTTP/1.1" 200 -
2019-04-20 10:35:42,362 sample.py 120 INFO   get_instance_ids {'instance_ids': ['d5d2845c-2705-4f24-af0f-5ca4426dd7bd', 'a77f123a-4318-48ee-a5b1-27b13e89fc28', '2d9f1422-b81a-4efc-9616-4ca0e801367c', 'f3bf79ac-9fa9-4e74-bd84-5284729ace29', 'ba29f300-4bb5-4e3e-97d8-c6ebdc8e1493', '26dab7e9-97cd-4edb-879b-782e1ac11698']}
2019-04-20 10:35:42,363 sample.py 240 INFO   sample app manager reply: {'session_id': 'e5313ed2-633e-11e9-93d4-54ab3a1411e8', 'instance_ids': ['d5d2845c-2705-4f24-af0f-5ca4426dd7bd', 'a77f123a-4318-48ee-a5b1-27b13e89fc28', '2d9f1422-b81a-4efc-9616-4ca0e801367c', 'f3bf79ac-9fa9-4e74-bd84-5284729ace29', 'ba29f300-4bb5-4e3e-97d8-c6ebdc8e1493', '26dab7e9-97cd-4edb-879b-782e1ac11698'], 'state': 'ACK_MAINTENANCE'}
87.254.192.34 - - [20/Apr/2019 10:35:42] "PUT /maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8/2ca07a0f752b4e4eba91e7b2137bfe10 HTTP/1.1" 200 -
87.254.192.34 - - [20/Apr/2019 10:35:42] "POST /maintenance HTTP/1.1" 200 -
2019-04-20 10:35:43,170 sample.py 122 INFO   all projects replied
2019-04-20 10:35:43,170 sample.py 497 INFO   Time now: 2019-04-20 07:35:43 maintenance starts: 2019-04-20 07:36:09....
2019-04-20 10:36:09,026 sample.py 503 INFO   Time to start maintenance starts: 2019-04-20 07:36:09
2019-04-20 10:36:09,026 sample.py 237 INFO   there is room for 0 instances
2019-04-20 10:36:09,026 sample.py 515 INFO   Need to down scale
2019-04-20 10:36:09,027 sample.py 524 INFO   --==State change from MAINTENANCE to SCALE_IN==--
2019-04-20 10:36:09,027 sample.py 481 INFO   --==session e5313ed2-633e-11e9-93d4-54ab3a1411e8: processing state SCALE_IN==--
2019-04-20 10:36:09,948 sample.py 155 INFO   SCALE_IN to project 2ca07a0f752b4e4eba91e7b2137bfe10
2019-04-20 10:36:10,144 sample.py 164 INFO   sample app manager received data = {'allowed_actions': [], 'session_id': 'e5313ed2-633e-11e9-93d4-54ab3a1411e8', 'instance_ids': 'http://0.0.0.0:12347/maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8/2ca07a0f752b4e4eba91e7b2137bfe10', 'metadata': {'openstack_version': 'Rocky'}, 'state': 'SCALE_IN', 'reply_url': 'http://0.0.0.0:12347/maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8/2ca07a0f752b4e4eba91e7b2137bfe10', 'service': 'admin_tool', 'actions_at': '2019-04-20T07:38:09', 'project_id': '2ca07a0f752b4e4eba91e7b2137bfe10'}
2019-04-20 10:36:10,144 sample.py 170 INFO   sample app manager state: SCALE_IN
2019-04-20 10:36:22,781 stack.py 65 INFO   stack doctor_test_maintenance UPDATE_COMPLETE
2019-04-20 10:36:22,866 sample.py 144 INFO   scaled insances from 6 to 4
2019-04-20 10:36:22,947 sample.py 240 INFO   sample app manager reply: {'session_id': 'e5313ed2-633e-11e9-93d4-54ab3a1411e8', 'instance_ids': ['d5d2845c-2705-4f24-af0f-5ca4426dd7bd', 'f3bf79ac-9fa9-4e74-bd84-5284729ace29', 'ba29f300-4bb5-4e3e-97d8-c6ebdc8e1493', '26dab7e9-97cd-4edb-879b-782e1ac11698'], 'state': 'ACK_SCALE_IN'}
87.254.192.34 - - [20/Apr/2019 10:36:22] "PUT /maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8/2ca07a0f752b4e4eba91e7b2137bfe10 HTTP/1.1" 200 -
87.254.192.34 - - [20/Apr/2019 10:36:22] "POST /maintenance HTTP/1.1" 200 -
2019-04-20 10:36:22,970 sample.py 122 INFO   all projects replied
2019-04-20 10:36:23,226 sample.py 436 INFO   
overcloud-novacompute-2.opnfvlf.org:
  2ca07a0f752b4e4eba91e7b2137bfe10:
    f3bf79ac-9fa9-4e74-bd84-5284729ace29: doctor_nonha_app_0
overcloud-novacompute-0.opnfvlf.org:
  2ca07a0f752b4e4eba91e7b2137bfe10:
    d5d2845c-2705-4f24-af0f-5ca4426dd7bd: doctor_nonha_app_1
    26dab7e9-97cd-4edb-879b-782e1ac11698: doctor_ha_app_1
overcloud-novacompute-1.opnfvlf.org:
  2ca07a0f752b4e4eba91e7b2137bfe10:
    ba29f300-4bb5-4e3e-97d8-c6ebdc8e1493: doctor_ha_app_0

2019-04-20 10:36:23,227 sample.py 536 INFO   --==State change from SCALE_IN to PREPARE_MAINTENANCE==--
2019-04-20 10:36:23,227 sample.py 481 INFO   --==session e5313ed2-633e-11e9-93d4-54ab3a1411e8: processing state PREPARE_MAINTENANCE==--
2019-04-20 10:36:23,227 sample.py 546 INFO   no empty hosts for maintenance
2019-04-20 10:36:23,228 sample.py 237 INFO   there is room for 2 instances
2019-04-20 10:36:23,228 sample.py 257 INFO   host overcloud-novacompute-2.opnfvlf.org has 0 ha and 1 non ha instances
2019-04-20 10:36:23,228 sample.py 257 INFO   host overcloud-novacompute-1.opnfvlf.org has 1 ha and 0 non ha instances
2019-04-20 10:36:23,228 sample.py 257 INFO   host overcloud-novacompute-0.opnfvlf.org has 1 ha and 1 non ha instances
2019-04-20 10:36:23,229 sample.py 266 INFO   host overcloud-novacompute-2.opnfvlf.org selected to be empty
2019-04-20 10:36:23,229 sample.py 276 INFO   PREPARE_MAINTENANCE to project 2ca07a0f752b4e4eba91e7b2137bfe10
2019-04-20 10:36:23,229 sample.py 424 INFO   Project servers:
  f3bf79ac-9fa9-4e74-bd84-5284729ace29: doctor_nonha_app_0

2019-04-20 10:36:25,200 sample.py 164 INFO   sample app manager received data = {'allowed_actions': ['MIGRATE', 'LIVE_MIGRATE', 'OWN_ACTION'], 'session_id': 'e5313ed2-633e-11e9-93d4-54ab3a1411e8', 'instance_ids': 'http://0.0.0.0:12347/maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8/2ca07a0f752b4e4eba91e7b2137bfe10', 'metadata': {'openstack_version': 'Rocky'}, 'state': 'PREPARE_MAINTENANCE', 'reply_url': 'http://0.0.0.0:12347/maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8/2ca07a0f752b4e4eba91e7b2137bfe10', 'service': 'admin_tool', 'actions_at': '2019-04-20T07:38:23', 'project_id': '2ca07a0f752b4e4eba91e7b2137bfe10'}
2019-04-20 10:36:25,201 sample.py 170 INFO   sample app manager state: PREPARE_MAINTENANCE
87.254.192.34 - - [20/Apr/2019 10:36:25] "GET /maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8/2ca07a0f752b4e4eba91e7b2137bfe10 HTTP/1.1" 200 -
2019-04-20 10:36:25,208 sample.py 120 INFO   get_instance_ids {'instance_ids': ['f3bf79ac-9fa9-4e74-bd84-5284729ace29']}
2019-04-20 10:36:25,209 sample.py 203 INFO   sample app manager got instances: ['f3bf79ac-9fa9-4e74-bd84-5284729ace29']
2019-04-20 10:36:25,209 sample.py 240 INFO   sample app manager reply: {'session_id': 'e5313ed2-633e-11e9-93d4-54ab3a1411e8', 'state': 'ACK_PREPARE_MAINTENANCE', 'instance_actions': {'f3bf79ac-9fa9-4e74-bd84-5284729ace29': 'MIGRATE'}}
87.254.192.34 - - [20/Apr/2019 10:36:25] "PUT /maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8/2ca07a0f752b4e4eba91e7b2137bfe10 HTTP/1.1" 200 -
87.254.192.34 - - [20/Apr/2019 10:36:25] "POST /maintenance HTTP/1.1" 200 -
2019-04-20 10:36:25,234 sample.py 122 INFO   all projects replied
2019-04-20 10:36:25,234 sample.py 319 INFO   Action MIGRATE server f3bf79ac-9fa9-4e74-bd84-5284729ace29: doctor_nonha_app_0
2019-04-20 10:36:25,987 sample.py 335 INFO   server f3bf79ac-9fa9-4e74-bd84-5284729ace29 state active
2019-04-20 10:37:36,975 sample.py 350 INFO   server f3bf79ac-9fa9-4e74-bd84-5284729ace29 migration confirmed
2019-04-20 10:37:37,070 sample.py 394 INFO   overcloud-novacompute-2.opnfvlf.org still has 16 vcpus reserved. wait...
2019-04-20 10:37:37,152 sample.py 164 INFO   sample app manager received data = {'allowed_actions': [], 'session_id': 'e5313ed2-633e-11e9-93d4-54ab3a1411e8', 'instance_ids': 'f3bf79ac-9fa9-4e74-bd84-5284729ace29', 'state': 'INSTANCE_ACTION_DONE', 'reply_url': 'http://0.0.0.0:12347/maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8/2ca07a0f752b4e4eba91e7b2137bfe10', 'service': 'admin_tool', 'project_id': '2ca07a0f752b4e4eba91e7b2137bfe10'}
2019-04-20 10:37:37,152 sample.py 170 INFO   sample app manager state: INSTANCE_ACTION_DONE
2019-04-20 10:37:37,152 sample.py 230 INFO   f3bf79ac-9fa9-4e74-bd84-5284729ace29
87.254.192.34 - - [20/Apr/2019 10:37:37] "POST /maintenance HTTP/1.1" 200 -
2019-04-20 10:37:52,187 sample.py 401 INFO   overcloud-novacompute-2.opnfvlf.org empty
2019-04-20 10:37:52,443 sample.py 436 INFO   
overcloud-novacompute-2.opnfvlf.org:
overcloud-novacompute-0.opnfvlf.org:
  2ca07a0f752b4e4eba91e7b2137bfe10:
    d5d2845c-2705-4f24-af0f-5ca4426dd7bd: doctor_nonha_app_1
    26dab7e9-97cd-4edb-879b-782e1ac11698: doctor_ha_app_1
overcloud-novacompute-1.opnfvlf.org:
  2ca07a0f752b4e4eba91e7b2137bfe10:
    f3bf79ac-9fa9-4e74-bd84-5284729ace29: doctor_nonha_app_0
    ba29f300-4bb5-4e3e-97d8-c6ebdc8e1493: doctor_ha_app_0

2019-04-20 10:37:52,444 sample.py 571 INFO   --==State change from PREPARE_MAINTENANCE to PLANNED_MAINTENANCE==--
2019-04-20 10:37:52,444 sample.py 481 INFO   --==session e5313ed2-633e-11e9-93d4-54ab3a1411e8: processing state PLANNED_MAINTENANCE==--
2019-04-20 10:37:52,444 sample.py 578 INFO   disable nova-compute on host overcloud-novacompute-2.opnfvlf.org
2019-04-20 10:37:52,520 sample.py 578 INFO   disable nova-compute on host overcloud-novacompute-0.opnfvlf.org
2019-04-20 10:37:52,636 sample.py 578 INFO   disable nova-compute on host overcloud-novacompute-1.opnfvlf.org
2019-04-20 10:37:52,753 sample.py 587 INFO   --==Start to maintain empty hosts==--
['overcloud-novacompute-2.opnfvlf.org']
2019-04-20 10:37:53,066 sample.py 401 INFO   overcloud-novacompute-2.opnfvlf.org empty
2019-04-20 10:37:53,066 sample.py 593 INFO   IN_MAINTENANCE host overcloud-novacompute-2.opnfvlf.org
2019-04-20 10:37:53,069 sample.py 473 INFO   maintaining host overcloud-novacompute-2.opnfvlf.org
2019-04-20 10:37:55,163 sample.py 144 INFO   sample inspector received data = {'session_id': 'e5313ed2-633e-11e9-93d4-54ab3a1411e8', 'service': 'admin_tool', 'project_id': '2ca07a0f752b4e4eba91e7b2137bfe10', 'host': 'overcloud-novacompute-2.opnfvlf.org', 'state': 'IN_MAINTENANCE'}
2019-04-20 10:37:55,163 sample.py 151 INFO   sample inspector: disable overcloud-novacompute-2.opnfvlf.org automatic fault management
87.254.192.34 - - [20/Apr/2019 10:37:55] "POST /maintenance HTTP/1.1" 200 -
2019-04-20 10:37:58,179 sample.py 602 INFO   MAINTENANCE_COMPLETE host overcloud-novacompute-2.opnfvlf.org
2019-04-20 10:37:58,180 sample.py 604 INFO   --==Start to maintain occupied hosts==--
['overcloud-novacompute-0.opnfvlf.org', 'overcloud-novacompute-1.opnfvlf.org']
2019-04-20 10:37:58,180 sample.py 606 INFO   PLANNED_MAINTENANCE host overcloud-novacompute-0.opnfvlf.org
2019-04-20 10:37:58,180 sample.py 276 INFO   PLANNED_MAINTENANCE to project 2ca07a0f752b4e4eba91e7b2137bfe10
2019-04-20 10:37:58,181 sample.py 424 INFO   Project servers:
  d5d2845c-2705-4f24-af0f-5ca4426dd7bd: doctor_nonha_app_1
  26dab7e9-97cd-4edb-879b-782e1ac11698: doctor_ha_app_1

2019-04-20 10:37:58,397 sample.py 144 INFO   sample inspector received data = {'session_id': 'e5313ed2-633e-11e9-93d4-54ab3a1411e8', 'service': 'admin_tool', 'project_id': '2ca07a0f752b4e4eba91e7b2137bfe10', 'host': 'overcloud-novacompute-2.opnfvlf.org', 'state': 'MAINTENANCE_COMPLETE'}
2019-04-20 10:37:58,397 sample.py 154 INFO   sample inspector: enable overcloud-novacompute-2.opnfvlf.org automatic fault management
87.254.192.34 - - [20/Apr/2019 10:37:58] "POST /maintenance HTTP/1.1" 200 -
2019-04-20 10:37:58,429 sample.py 164 INFO   sample app manager received data = {'allowed_actions': ['MIGRATE', 'LIVE_MIGRATE', 'OWN_ACTION'], 'session_id': 'e5313ed2-633e-11e9-93d4-54ab3a1411e8', 'instance_ids': 'http://0.0.0.0:12347/maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8/2ca07a0f752b4e4eba91e7b2137bfe10', 'metadata': {'openstack_version': 'Rocky'}, 'state': 'PLANNED_MAINTENANCE', 'reply_url': 'http://0.0.0.0:12347/maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8/2ca07a0f752b4e4eba91e7b2137bfe10', 'service': 'admin_tool', 'actions_at': '2019-04-20T07:39:58', 'project_id': '2ca07a0f752b4e4eba91e7b2137bfe10'}
2019-04-20 10:37:58,429 sample.py 170 INFO   sample app manager state: PLANNED_MAINTENANCE
87.254.192.34 - - [20/Apr/2019 10:37:58] "GET /maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8/2ca07a0f752b4e4eba91e7b2137bfe10 HTTP/1.1" 200 -
2019-04-20 10:37:58,437 sample.py 120 INFO   get_instance_ids {'instance_ids': ['d5d2845c-2705-4f24-af0f-5ca4426dd7bd', '26dab7e9-97cd-4edb-879b-782e1ac11698']}
2019-04-20 10:37:58,438 sample.py 220 INFO   sample app manager got instances: ['d5d2845c-2705-4f24-af0f-5ca4426dd7bd', '26dab7e9-97cd-4edb-879b-782e1ac11698']
2019-04-20 10:37:58,438 sample.py 240 INFO   sample app manager reply: {'session_id': 'e5313ed2-633e-11e9-93d4-54ab3a1411e8', 'state': 'ACK_PLANNED_MAINTENANCE', 'instance_actions': {'d5d2845c-2705-4f24-af0f-5ca4426dd7bd': 'MIGRATE', '26dab7e9-97cd-4edb-879b-782e1ac11698': 'MIGRATE'}}
87.254.192.34 - - [20/Apr/2019 10:37:58] "PUT /maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8/2ca07a0f752b4e4eba91e7b2137bfe10 HTTP/1.1" 200 -
87.254.192.34 - - [20/Apr/2019 10:37:58] "POST /maintenance HTTP/1.1" 200 -
2019-04-20 10:37:59,184 sample.py 122 INFO   all projects replied
2019-04-20 10:37:59,185 sample.py 319 INFO   Action MIGRATE server d5d2845c-2705-4f24-af0f-5ca4426dd7bd: doctor_nonha_app_1
2019-04-20 10:37:59,710 sample.py 335 INFO   server d5d2845c-2705-4f24-af0f-5ca4426dd7bd state active
2019-04-20 10:39:15,244 sample.py 350 INFO   server d5d2845c-2705-4f24-af0f-5ca4426dd7bd migration confirmed
2019-04-20 10:39:15,247 sample.py 319 INFO   Action MIGRATE server 26dab7e9-97cd-4edb-879b-782e1ac11698: doctor_ha_app_1
2019-04-20 10:39:15,554 sample.py 164 INFO   sample app manager received data = {'allowed_actions': [], 'session_id': 'e5313ed2-633e-11e9-93d4-54ab3a1411e8', 'instance_ids': 'd5d2845c-2705-4f24-af0f-5ca4426dd7bd', 'state': 'INSTANCE_ACTION_DONE', 'reply_url': 'http://0.0.0.0:12347/maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8/2ca07a0f752b4e4eba91e7b2137bfe10', 'service': 'admin_tool', 'project_id': '2ca07a0f752b4e4eba91e7b2137bfe10'}
2019-04-20 10:39:15,555 sample.py 170 INFO   sample app manager state: INSTANCE_ACTION_DONE
2019-04-20 10:39:15,555 sample.py 230 INFO   d5d2845c-2705-4f24-af0f-5ca4426dd7bd
87.254.192.34 - - [20/Apr/2019 10:39:15] "POST /maintenance HTTP/1.1" 200 -
2019-04-20 10:39:15,944 sample.py 335 INFO   server 26dab7e9-97cd-4edb-879b-782e1ac11698 state active
2019-04-20 10:40:30,434 sample.py 350 INFO   server 26dab7e9-97cd-4edb-879b-782e1ac11698 migration confirmed
2019-04-20 10:40:30,612 sample.py 394 INFO   overcloud-novacompute-0.opnfvlf.org still has 16 vcpus reserved. wait...
2019-04-20 10:40:30,704 sample.py 164 INFO   sample app manager received data = {'allowed_actions': [], 'session_id': 'e5313ed2-633e-11e9-93d4-54ab3a1411e8', 'instance_ids': '26dab7e9-97cd-4edb-879b-782e1ac11698', 'state': 'INSTANCE_ACTION_DONE', 'reply_url': 'http://0.0.0.0:12347/maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8/2ca07a0f752b4e4eba91e7b2137bfe10', 'service': 'admin_tool', 'project_id': '2ca07a0f752b4e4eba91e7b2137bfe10'}
2019-04-20 10:40:30,704 sample.py 170 INFO   sample app manager state: INSTANCE_ACTION_DONE
2019-04-20 10:40:30,705 sample.py 230 INFO   26dab7e9-97cd-4edb-879b-782e1ac11698
87.254.192.34 - - [20/Apr/2019 10:40:30] "POST /maintenance HTTP/1.1" 200 -
2019-04-20 10:40:46,028 sample.py 401 INFO   overcloud-novacompute-0.opnfvlf.org empty
2019-04-20 10:40:46,028 sample.py 611 INFO   IN_MAINTENANCE  host overcloud-novacompute-0.opnfvlf.org
2019-04-20 10:40:46,031 sample.py 473 INFO   maintaining host overcloud-novacompute-0.opnfvlf.org
2019-04-20 10:40:46,135 sample.py 144 INFO   sample inspector received data = {'session_id': 'e5313ed2-633e-11e9-93d4-54ab3a1411e8', 'service': 'admin_tool', 'project_id': '2ca07a0f752b4e4eba91e7b2137bfe10', 'host': 'overcloud-novacompute-0.opnfvlf.org', 'state': 'IN_MAINTENANCE'}
2019-04-20 10:40:46,135 sample.py 151 INFO   sample inspector: disable overcloud-novacompute-0.opnfvlf.org automatic fault management
87.254.192.34 - - [20/Apr/2019 10:40:46] "POST /maintenance HTTP/1.1" 200 -
2019-04-20 10:40:49,715 sample.py 696 INFO   reply: {"state": "PLANNED_MAINTENANCE"}
87.254.192.34 - - [20/Apr/2019 10:40:49] "GET /maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8 HTTP/1.1" 200 -
2019-04-20 10:40:51,094 sample.py 620 INFO   MAINTENANCE_COMPLETE host overcloud-novacompute-0.opnfvlf.org
2019-04-20 10:40:51,094 sample.py 606 INFO   PLANNED_MAINTENANCE host overcloud-novacompute-1.opnfvlf.org
2019-04-20 10:40:51,094 sample.py 276 INFO   PLANNED_MAINTENANCE to project 2ca07a0f752b4e4eba91e7b2137bfe10
2019-04-20 10:40:51,094 sample.py 424 INFO   Project servers:
  f3bf79ac-9fa9-4e74-bd84-5284729ace29: doctor_nonha_app_0
  ba29f300-4bb5-4e3e-97d8-c6ebdc8e1493: doctor_ha_app_0

2019-04-20 10:40:51,097 sample.py 122 INFO   all projects replied
2019-04-20 10:40:51,143 sample.py 144 INFO   sample inspector received data = {'session_id': 'e5313ed2-633e-11e9-93d4-54ab3a1411e8', 'service': 'admin_tool', 'project_id': '2ca07a0f752b4e4eba91e7b2137bfe10', 'host': 'overcloud-novacompute-0.opnfvlf.org', 'state': 'MAINTENANCE_COMPLETE'}
2019-04-20 10:40:51,143 sample.py 154 INFO   sample inspector: enable overcloud-novacompute-0.opnfvlf.org automatic fault management
87.254.192.34 - - [20/Apr/2019 10:40:51] "POST /maintenance HTTP/1.1" 200 -
2019-04-20 10:40:51,258 sample.py 164 INFO   sample app manager received data = {'allowed_actions': ['MIGRATE', 'LIVE_MIGRATE', 'OWN_ACTION'], 'session_id': 'e5313ed2-633e-11e9-93d4-54ab3a1411e8', 'instance_ids': 'http://0.0.0.0:12347/maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8/2ca07a0f752b4e4eba91e7b2137bfe10', 'metadata': {'openstack_version': 'Rocky'}, 'state': 'PLANNED_MAINTENANCE', 'reply_url': 'http://0.0.0.0:12347/maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8/2ca07a0f752b4e4eba91e7b2137bfe10', 'service': 'admin_tool', 'actions_at': '2019-04-20T07:42:51', 'project_id': '2ca07a0f752b4e4eba91e7b2137bfe10'}
2019-04-20 10:40:51,258 sample.py 170 INFO   sample app manager state: PLANNED_MAINTENANCE
87.254.192.34 - - [20/Apr/2019 10:40:51] "GET /maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8/2ca07a0f752b4e4eba91e7b2137bfe10 HTTP/1.1" 200 -
2019-04-20 10:40:51,266 sample.py 120 INFO   get_instance_ids {'instance_ids': ['f3bf79ac-9fa9-4e74-bd84-5284729ace29', 'ba29f300-4bb5-4e3e-97d8-c6ebdc8e1493']}
2019-04-20 10:40:51,266 sample.py 220 INFO   sample app manager got instances: ['f3bf79ac-9fa9-4e74-bd84-5284729ace29', 'ba29f300-4bb5-4e3e-97d8-c6ebdc8e1493']
2019-04-20 10:40:51,267 sample.py 87 INFO   Switch over to: doctor_ha_app_1 26dab7e9-97cd-4edb-879b-782e1ac11698
2019-04-20 10:40:56,111 sample.py 240 INFO   sample app manager reply: {'session_id': 'e5313ed2-633e-11e9-93d4-54ab3a1411e8', 'state': 'ACK_PLANNED_MAINTENANCE', 'instance_actions': {'f3bf79ac-9fa9-4e74-bd84-5284729ace29': 'MIGRATE', 'ba29f300-4bb5-4e3e-97d8-c6ebdc8e1493': 'MIGRATE'}}
87.254.192.34 - - [20/Apr/2019 10:40:56] "PUT /maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8/2ca07a0f752b4e4eba91e7b2137bfe10 HTTP/1.1" 200 -
87.254.192.34 - - [20/Apr/2019 10:40:56] "POST /maintenance HTTP/1.1" 200 -
2019-04-20 10:40:57,103 sample.py 319 INFO   Action MIGRATE server f3bf79ac-9fa9-4e74-bd84-5284729ace29: doctor_nonha_app_0
2019-04-20 10:40:57,311 sample.py 335 INFO   server f3bf79ac-9fa9-4e74-bd84-5284729ace29 state active
2019-04-20 10:40:59,733 sample.py 696 INFO   reply: {"state": "PLANNED_MAINTENANCE"}
87.254.192.34 - - [20/Apr/2019 10:40:59] "GET /maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8 HTTP/1.1" 200 -
2019-04-20 10:41:09,752 sample.py 696 INFO   reply: {"state": "PLANNED_MAINTENANCE"}
87.254.192.34 - - [20/Apr/2019 10:41:09] "GET /maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8 HTTP/1.1" 200 -
2019-04-20 10:41:19,771 sample.py 696 INFO   reply: {"state": "PLANNED_MAINTENANCE"}
87.254.192.34 - - [20/Apr/2019 10:41:19] "GET /maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8 HTTP/1.1" 200 -
2019-04-20 10:41:29,791 sample.py 696 INFO   reply: {"state": "PLANNED_MAINTENANCE"}
87.254.192.34 - - [20/Apr/2019 10:41:29] "GET /maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8 HTTP/1.1" 200 -
2019-04-20 10:41:39,810 sample.py 696 INFO   reply: {"state": "PLANNED_MAINTENANCE"}
87.254.192.34 - - [20/Apr/2019 10:41:39] "GET /maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8 HTTP/1.1" 200 -
2019-04-20 10:41:49,829 sample.py 696 INFO   reply: {"state": "PLANNED_MAINTENANCE"}
87.254.192.34 - - [20/Apr/2019 10:41:49] "GET /maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8 HTTP/1.1" 200 -
2019-04-20 10:41:59,849 sample.py 696 INFO   reply: {"state": "PLANNED_MAINTENANCE"}
87.254.192.34 - - [20/Apr/2019 10:41:59] "GET /maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8 HTTP/1.1" 200 -
2019-04-20 10:42:09,863 sample.py 696 INFO   reply: {"state": "PLANNED_MAINTENANCE"}
87.254.192.34 - - [20/Apr/2019 10:42:09] "GET /maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8 HTTP/1.1" 200 -
2019-04-20 10:42:11,967 sample.py 350 INFO   server f3bf79ac-9fa9-4e74-bd84-5284729ace29 migration confirmed
2019-04-20 10:42:11,970 sample.py 319 INFO   Action MIGRATE server ba29f300-4bb5-4e3e-97d8-c6ebdc8e1493: doctor_ha_app_0
2019-04-20 10:42:12,091 sample.py 164 INFO   sample app manager received data = {'allowed_actions': [], 'session_id': 'e5313ed2-633e-11e9-93d4-54ab3a1411e8', 'instance_ids': 'f3bf79ac-9fa9-4e74-bd84-5284729ace29', 'state': 'INSTANCE_ACTION_DONE', 'reply_url': 'http://0.0.0.0:12347/maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8/2ca07a0f752b4e4eba91e7b2137bfe10', 'service': 'admin_tool', 'project_id': '2ca07a0f752b4e4eba91e7b2137bfe10'}
2019-04-20 10:42:12,092 sample.py 170 INFO   sample app manager state: INSTANCE_ACTION_DONE
2019-04-20 10:42:12,092 sample.py 230 INFO   f3bf79ac-9fa9-4e74-bd84-5284729ace29
87.254.192.34 - - [20/Apr/2019 10:42:12] "POST /maintenance HTTP/1.1" 200 -
2019-04-20 10:42:12,519 sample.py 335 INFO   server ba29f300-4bb5-4e3e-97d8-c6ebdc8e1493 state active
2019-04-20 10:42:19,886 sample.py 696 INFO   reply: {"state": "PLANNED_MAINTENANCE"}
87.254.192.34 - - [20/Apr/2019 10:42:19] "GET /maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8 HTTP/1.1" 200 -
2019-04-20 10:42:29,905 sample.py 696 INFO   reply: {"state": "PLANNED_MAINTENANCE"}
87.254.192.34 - - [20/Apr/2019 10:42:29] "GET /maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8 HTTP/1.1" 200 -
2019-04-20 10:42:39,915 sample.py 696 INFO   reply: {"state": "PLANNED_MAINTENANCE"}
87.254.192.34 - - [20/Apr/2019 10:42:39] "GET /maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8 HTTP/1.1" 200 -
2019-04-20 10:42:49,931 sample.py 696 INFO   reply: {"state": "PLANNED_MAINTENANCE"}
87.254.192.34 - - [20/Apr/2019 10:42:49] "GET /maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8 HTTP/1.1" 200 -
2019-04-20 10:42:59,950 sample.py 696 INFO   reply: {"state": "PLANNED_MAINTENANCE"}
87.254.192.34 - - [20/Apr/2019 10:42:59] "GET /maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8 HTTP/1.1" 200 -
2019-04-20 10:43:09,968 sample.py 696 INFO   reply: {"state": "PLANNED_MAINTENANCE"}
87.254.192.34 - - [20/Apr/2019 10:43:09] "GET /maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8 HTTP/1.1" 200 -
2019-04-20 10:43:19,986 sample.py 696 INFO   reply: {"state": "PLANNED_MAINTENANCE"}
87.254.192.34 - - [20/Apr/2019 10:43:19] "GET /maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8 HTTP/1.1" 200 -
2019-04-20 10:43:28,899 sample.py 350 INFO   server ba29f300-4bb5-4e3e-97d8-c6ebdc8e1493 migration confirmed
2019-04-20 10:43:28,968 sample.py 394 INFO   overcloud-novacompute-1.opnfvlf.org still has 16 vcpus reserved. wait...
2019-04-20 10:43:29,147 sample.py 164 INFO   sample app manager received data = {'allowed_actions': [], 'session_id': 'e5313ed2-633e-11e9-93d4-54ab3a1411e8', 'instance_ids': 'ba29f300-4bb5-4e3e-97d8-c6ebdc8e1493', 'state': 'INSTANCE_ACTION_DONE', 'reply_url': 'http://0.0.0.0:12347/maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8/2ca07a0f752b4e4eba91e7b2137bfe10', 'service': 'admin_tool', 'project_id': '2ca07a0f752b4e4eba91e7b2137bfe10'}
2019-04-20 10:43:29,147 sample.py 170 INFO   sample app manager state: INSTANCE_ACTION_DONE
2019-04-20 10:43:29,147 sample.py 230 INFO   ba29f300-4bb5-4e3e-97d8-c6ebdc8e1493
87.254.192.34 - - [20/Apr/2019 10:43:29] "POST /maintenance HTTP/1.1" 200 -
2019-04-20 10:43:30,005 sample.py 696 INFO   reply: {"state": "PLANNED_MAINTENANCE"}
87.254.192.34 - - [20/Apr/2019 10:43:30] "GET /maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8 HTTP/1.1" 200 -
2019-04-20 10:43:40,017 sample.py 696 INFO   reply: {"state": "PLANNED_MAINTENANCE"}
87.254.192.34 - - [20/Apr/2019 10:43:40] "GET /maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8 HTTP/1.1" 200 -
2019-04-20 10:43:50,036 sample.py 696 INFO   reply: {"state": "PLANNED_MAINTENANCE"}
87.254.192.34 - - [20/Apr/2019 10:43:50] "GET /maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8 HTTP/1.1" 200 -
2019-04-20 10:43:54,162 sample.py 401 INFO   overcloud-novacompute-1.opnfvlf.org empty
2019-04-20 10:43:54,162 sample.py 611 INFO   IN_MAINTENANCE  host overcloud-novacompute-1.opnfvlf.org
2019-04-20 10:43:54,165 sample.py 473 INFO   maintaining host overcloud-novacompute-1.opnfvlf.org
2019-04-20 10:43:56,015 sample.py 144 INFO   sample inspector received data = {'session_id': 'e5313ed2-633e-11e9-93d4-54ab3a1411e8', 'service': 'admin_tool', 'project_id': '2ca07a0f752b4e4eba91e7b2137bfe10', 'host': 'overcloud-novacompute-1.opnfvlf.org', 'state': 'IN_MAINTENANCE'}
2019-04-20 10:43:56,015 sample.py 151 INFO   sample inspector: disable overcloud-novacompute-1.opnfvlf.org automatic fault management
87.254.192.34 - - [20/Apr/2019 10:43:56] "POST /maintenance HTTP/1.1" 200 -
2019-04-20 10:43:59,274 sample.py 620 INFO   MAINTENANCE_COMPLETE host overcloud-novacompute-1.opnfvlf.org
2019-04-20 10:43:59,364 sample.py 144 INFO   sample inspector received data = {'session_id': 'e5313ed2-633e-11e9-93d4-54ab3a1411e8', 'service': 'admin_tool', 'project_id': '2ca07a0f752b4e4eba91e7b2137bfe10', 'host': 'overcloud-novacompute-1.opnfvlf.org', 'state': 'MAINTENANCE_COMPLETE'}
2019-04-20 10:43:59,364 sample.py 154 INFO   sample inspector: enable overcloud-novacompute-1.opnfvlf.org automatic fault management
87.254.192.34 - - [20/Apr/2019 10:43:59] "POST /maintenance HTTP/1.1" 200 -
2019-04-20 10:43:59,849 sample.py 436 INFO   
overcloud-novacompute-2.opnfvlf.org:
  2ca07a0f752b4e4eba91e7b2137bfe10:
    d5d2845c-2705-4f24-af0f-5ca4426dd7bd: doctor_nonha_app_1
    26dab7e9-97cd-4edb-879b-782e1ac11698: doctor_ha_app_1
overcloud-novacompute-0.opnfvlf.org:
  2ca07a0f752b4e4eba91e7b2137bfe10:
    f3bf79ac-9fa9-4e74-bd84-5284729ace29: doctor_nonha_app_0
    ba29f300-4bb5-4e3e-97d8-c6ebdc8e1493: doctor_ha_app_0
overcloud-novacompute-1.opnfvlf.org:

2019-04-20 10:43:59,849 sample.py 481 INFO   --==session e5313ed2-633e-11e9-93d4-54ab3a1411e8: processing state PLANNED_MAINTENANCE_COMPLETE==--
2019-04-20 10:43:59,849 sample.py 625 INFO   Projects still need to up scale back to full capcity
2019-04-20 10:43:59,850 sample.py 207 INFO   MAINTENANCE_COMPLETE to project 2ca07a0f752b4e4eba91e7b2137bfe10
2019-04-20 10:44:00,056 sample.py 696 INFO   reply: {"state": "PLANNED_MAINTENANCE_COMPLETE"}
87.254.192.34 - - [20/Apr/2019 10:44:00] "GET /maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8 HTTP/1.1" 200 -
2019-04-20 10:44:00,081 sample.py 164 INFO   sample app manager received data = {'allowed_actions': [], 'session_id': 'e5313ed2-633e-11e9-93d4-54ab3a1411e8', 'instance_ids': 'http://0.0.0.0:12347/maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8/2ca07a0f752b4e4eba91e7b2137bfe10', 'metadata': {'openstack_version': 'Rocky'}, 'state': 'MAINTENANCE_COMPLETE', 'reply_url': 'http://0.0.0.0:12347/maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8/2ca07a0f752b4e4eba91e7b2137bfe10', 'service': 'admin_tool', 'actions_at': '2019-04-20T07:45:59', 'project_id': '2ca07a0f752b4e4eba91e7b2137bfe10'}
2019-04-20 10:44:00,081 sample.py 170 INFO   sample app manager state: MAINTENANCE_COMPLETE
2019-04-20 10:44:10,077 sample.py 696 INFO   reply: {"state": "PLANNED_MAINTENANCE_COMPLETE"}
87.254.192.34 - - [20/Apr/2019 10:44:10] "GET /maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8 HTTP/1.1" 200 -
2019-04-20 10:44:20,098 sample.py 696 INFO   reply: {"state": "PLANNED_MAINTENANCE_COMPLETE"}
87.254.192.34 - - [20/Apr/2019 10:44:20] "GET /maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8 HTTP/1.1" 200 -
2019-04-20 10:44:24,726 stack.py 65 INFO   stack doctor_test_maintenance UPDATE_COMPLETE
2019-04-20 10:44:24,812 sample.py 144 INFO   scaled insances from 4 to 6
2019-04-20 10:44:24,812 sample.py 240 INFO   sample app manager reply: {'session_id': 'e5313ed2-633e-11e9-93d4-54ab3a1411e8', 'state': 'ACK_MAINTENANCE_COMPLETE'}
87.254.192.34 - - [20/Apr/2019 10:44:24] "PUT /maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8/2ca07a0f752b4e4eba91e7b2137bfe10 HTTP/1.1" 200 -
87.254.192.34 - - [20/Apr/2019 10:44:24] "POST /maintenance HTTP/1.1" 200 -
2019-04-20 10:44:24,878 sample.py 122 INFO   all projects replied
2019-04-20 10:44:25,131 sample.py 436 INFO   
overcloud-novacompute-2.opnfvlf.org:
  2ca07a0f752b4e4eba91e7b2137bfe10:
    d5d2845c-2705-4f24-af0f-5ca4426dd7bd: doctor_nonha_app_1
    26dab7e9-97cd-4edb-879b-782e1ac11698: doctor_ha_app_1
overcloud-novacompute-0.opnfvlf.org:
  2ca07a0f752b4e4eba91e7b2137bfe10:
    f3bf79ac-9fa9-4e74-bd84-5284729ace29: doctor_nonha_app_0
    ba29f300-4bb5-4e3e-97d8-c6ebdc8e1493: doctor_ha_app_0
overcloud-novacompute-1.opnfvlf.org:
  2ca07a0f752b4e4eba91e7b2137bfe10:
    6cf1d5c9-b8c3-40b2-9f82-8d8cf9e57fda: doctor_nonha_app_3
    fbeba364-ec21-4a6e-8106-b5ec193ecdb3: doctor_nonha_app_2

2019-04-20 10:44:25,131 sample.py 637 INFO   --==Maintenance session e5313ed2-633e-11e9-93d4-54ab3a1411e8: MAINTENANCE_DONE==--
2019-04-20 10:44:30,118 sample.py 696 INFO   reply: {"state": "MAINTENANCE_DONE"}
87.254.192.34 - - [20/Apr/2019 10:44:30] "GET /maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8 HTTP/1.1" 200 -
2019-04-20 10:44:30,122 maintenance.py 183 INFO   remove maintenance session e5313ed2-633e-11e9-93d4-54ab3a1411e8.......
2019-04-20 10:44:30,126 sample.py 717 INFO   remove session e5313ed2-633e-11e9-93d4-54ab3a1411e8
87.254.192.34 - - [20/Apr/2019 10:44:30] "DELETE /maintenance/e5313ed2-633e-11e9-93d4-54ab3a1411e8 HTTP/1.1" 200 -
2019-04-20 10:44:30,130 maintenance.py 224 INFO   maintenance e5313ed2-633e-11e9-93d4-54ab3a1411e8 ended with state MAINTENANCE_DONE
2019-04-20 10:44:30,130 main.py 118 INFO   doctor maintenance complete.......
2019-04-20 10:44:30,131 sample.py 41 INFO   sample admin tool stop......
2019-04-20 10:44:30,135 sample.py 725 INFO   shutdown admin_tool server at 1555746270.135636
87.254.192.34 - - [20/Apr/2019 10:44:30] "POST /shutdown HTTP/1.1" 200 -
2019-04-20 10:44:30,138 sample.py 36 INFO   sample app manager stop......
2019-04-20 10:44:30,143 sample.py 247 INFO   shutdown app manager server at 1555746270.1436255
87.254.192.34 - - [20/Apr/2019 10:44:30] "POST /shutdown HTTP/1.1" 200 -
2019-04-20 10:44:30,146 sample.py 91 INFO   sample inspector stop......
2019-04-20 10:44:30,150 sample.py 253 INFO   shutdown inspector app server at 1555746270.150924
87.254.192.34 - - [20/Apr/2019 10:44:30] "POST /events/shutdown HTTP/1.1" 200 -
2019-04-20 10:44:30,153 maintenance.py 236 INFO   stack delete start.......
2019-04-20 10:44:47,771 stack.py 65 INFO   stack doctor_test_maintenance DELETE_COMPLETE
2019-04-20 10:44:47,772 apex.py 166 INFO   restore apply patches start......
2019-04-20 10:44:47,862 image.py 71 INFO   image delete start.......
2019-04-20 10:44:48,270 base.py 218 INFO   Command sudo python restore_config.py output ['restore', 'restore: /var/lib/config-data/puppet-generated/ceilometer/etc/ceilometer/event_definitions.yaml', 'Bak_file empty, so removing also: /var/lib/config-data/puppet-generated/ceilometer/etc/ceilometer/event_definitions.yaml']
2019-04-20 10:44:48,274 base.py 218 INFO   Command sudo python restore_compute_config.py output ['restoring nova.bak.']
2019-04-20 10:44:49,653 image.py 76 INFO   image delete end.......
2019-04-20 10:44:49,654 user.py 163 INFO   user delete start......
2019-04-20 10:44:49,654 user.py 156 INFO   restore default quota......
2019-04-20 10:44:49,961 base.py 218 INFO   Command sudo python restore_aodh.py output []
2019-04-20 10:44:50,121 base.py 218 INFO   Command sudo python restore_compute_config.py output ['nova.bak does not exist.']
2019-04-20 10:44:51,355 user.py 187 INFO   user delete end......

Total time cost: 111(ms)
==============================================================================>
       |Monitor|Inspector           |Controller|Notifier|Evaluator           |
       |-84    |75                  |?         |?       |?                   |
       |       |      |      |      |          |        |      |      |      |
link down:0    |      |      |      |          |        |      |      |      |
     raw failure:-84  |      |      |          |        |      |      |      |
         found affected:?    |      |          |        |      |      |      |
                  set VM error:139  |          |        |      |      |      |
                         marked host down:-9   |        |      |      |      |
                               notified VM error:?      |      |      |      |
                                        transformed event:?    |      |      |
                                                 evaluated event:?    |      |
                                                            fired alarm:?    |
                                                                received alarm:111  

py34 runtests: commands[1] | /bin/find /home/jenkins/opnfv_slave_root/workspace/doctor-verify-all-apex-sample-x86_64-master -type f -name '*.py[co]' -delete -o -type d -name __pycache__ -delete
___________________________________ summary ____________________________________
  py34: commands succeeded
  congratulations :)
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 28296 killed;
[ssh-agent] Stopped.
Archiving artifacts
No emails were triggered.
[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
[WS-CLEANUP] done
Finished: SUCCESS