SuccessConsole Output

Started by upstream project "doctor-verify-master" build number 89
originally caused by:
 Triggered by Gerrit: https://gerrit.opnfv.org/gerrit/64035
[EnvInject] - Loading node environment variables.
Building remotely on zte-virtual3 (doctor-apex-x86_64) in workspace /home/jenkins/opnfv/slave_root/workspace/doctor-verify-fault_management-apex-sample-x86_64-master
[ssh-agent] Looking for ssh-agent implementation...
[ssh-agent]   Exec ssh-agent (binary ssh-agent on a remote machine)
$ ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-u1G2ZpxColo6/agent.14923
SSH_AGENT_PID=14925
[ssh-agent] Started.
Running ssh-add (command line suppressed)
Identity added: /home/jenkins/opnfv/slave_root/workspace/doctor-verify-fault_management-apex-sample-x86_64-master@tmp/private_key_7312006778294578439.key (/home/jenkins/opnfv/slave_root/workspace/doctor-verify-fault_management-apex-sample-x86_64-master@tmp/private_key_7312006778294578439.key)
[ssh-agent] Using credentials jenkins-ci (Jenkins Master SSH)
Wiping out workspace first.
Cloning the remote Git repository
Cloning repository https://gerrit.opnfv.org/gerrit/doctor
 > git init /home/jenkins/opnfv/slave_root/workspace/doctor-verify-fault_management-apex-sample-x86_64-master # timeout=10
Fetching upstream changes from https://gerrit.opnfv.org/gerrit/doctor
 > git --version # timeout=10
 > git fetch --tags --progress https://gerrit.opnfv.org/gerrit/doctor +refs/heads/*:refs/remotes/origin/* # timeout=15
 > git config remote.origin.url https://gerrit.opnfv.org/gerrit/doctor # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://gerrit.opnfv.org/gerrit/doctor # timeout=10
Fetching upstream changes from https://gerrit.opnfv.org/gerrit/doctor
 > git fetch --tags --progress https://gerrit.opnfv.org/gerrit/doctor refs/changes/35/64035/2 # timeout=15
Checking out Revision da3ccd55ce2104a7ae338dacd74f81baf972dc3c (refs/changes/35/64035/2)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f da3ccd55ce2104a7ae338dacd74f81baf972dc3c # timeout=15
Commit message: "[Don't merge] test for CI with longer time window"
 > git rev-parse FETCH_HEAD^{commit} # timeout=10
 > git rev-list --no-walk baac6579556f8216b36db0d0f87f9c2d4f8b4ef5 # timeout=10
No emails were triggered.
[doctor-verify-fault_management-apex-sample-x86_64-master] $ /usr/bin/env bash /tmp/jenkins4387466599856599931.sh
Gathering IP information for Apex installer VM
 1     undercloud                     running
Installer VM detected
Installer ip is 192.168.122.31
fetch_os_creds.info: Fetching rc file...
fetch_os_creds.info: Verifying connectivity to 192.168.122.31...
fetch_os_creds.info: 192.168.122.31 is reachable!
fetch_os_creds.info: ... from Instack VM 192.168.122.31...
Warning: Permanently added '192.168.122.31' (ECDSA) to the list of known hosts.
-------- Credentials: --------
# Clear any old environment that may conflict.
for key in $( set | awk '{FS="="}  /^OS_/ {print $1}' ); do unset $key ; done
export OS_NO_CACHE=True
export COMPUTE_API_VERSION=1.1
export OS_USERNAME=admin
export no_proxy=,192.168.37.12,192.0.2.3
export OS_USER_DOMAIN_NAME=Default
export OS_VOLUME_API_VERSION=3
export OS_CLOUDNAME=overcloud
export OS_AUTH_URL=http://192.168.37.12:5000
export NOVA_VERSION=1.1
export OS_IMAGE_API_VERSION=2
export OS_PASSWORD=CAhmmwRN8t4JIJ7bhVxqaZRG0
export OS_PROJECT_DOMAIN_NAME=Default
export OS_IDENTITY_API_VERSION=3
export OS_PROJECT_NAME=admin
export OS_AUTH_TYPE=password
export PYTHONWARNINGS="ignore:Certificate has no, ignore:A true SSLContext object is not available"

# Add OS_CLOUDNAME to PS1
if [ -z "${CLOUDPROMPT_ENABLED:-}" ]; then
    export PS1=${PS1:-""}
    export PS1=\${OS_CLOUDNAME:+"(\$OS_CLOUDNAME)"}\ $PS1
    export CLOUDPROMPT_ENABLED=1
fi
export OS_PROJECT_ID=c9cb46561bd446b9abc5e0d222b52e0e
export OS_TENANT_NAME=admin
export OS_REGION_NAME=regionOne
[doctor-verify-fault_management-apex-sample-x86_64-master] $ /bin/sh -xe /tmp/jenkins3619304296871521400.sh
+ source /home/jenkins/opnfv-openrc.sh
+++ set
+++ awk '{FS="="}  /^OS_/ {print $1}'
++ export OS_NO_CACHE=True
++ OS_NO_CACHE=True
++ export COMPUTE_API_VERSION=1.1
++ COMPUTE_API_VERSION=1.1
++ export OS_USERNAME=admin
++ OS_USERNAME=admin
++ export no_proxy=,192.168.37.12,192.0.2.3
++ no_proxy=,192.168.37.12,192.0.2.3
++ export OS_USER_DOMAIN_NAME=Default
++ OS_USER_DOMAIN_NAME=Default
++ export OS_VOLUME_API_VERSION=3
++ OS_VOLUME_API_VERSION=3
++ export OS_CLOUDNAME=overcloud
++ OS_CLOUDNAME=overcloud
++ export OS_AUTH_URL=http://192.168.37.12:5000
++ OS_AUTH_URL=http://192.168.37.12:5000
++ export NOVA_VERSION=1.1
++ NOVA_VERSION=1.1
++ export OS_IMAGE_API_VERSION=2
++ OS_IMAGE_API_VERSION=2
++ export OS_PASSWORD=CAhmmwRN8t4JIJ7bhVxqaZRG0
++ OS_PASSWORD=CAhmmwRN8t4JIJ7bhVxqaZRG0
++ export OS_PROJECT_DOMAIN_NAME=Default
++ OS_PROJECT_DOMAIN_NAME=Default
++ export OS_IDENTITY_API_VERSION=3
++ OS_IDENTITY_API_VERSION=3
++ export OS_PROJECT_NAME=admin
++ OS_PROJECT_NAME=admin
++ export OS_AUTH_TYPE=password
++ OS_AUTH_TYPE=password
++ export 'PYTHONWARNINGS=ignore:Certificate has no, ignore:A true SSLContext object is not available'
++ PYTHONWARNINGS='ignore:Certificate has no, ignore:A true SSLContext object is not available'
++ '[' -z '' ']'
++ export PS1=
++ PS1=
++ export 'PS1=${OS_CLOUDNAME:+($OS_CLOUDNAME)} '
++ PS1='${OS_CLOUDNAME:+($OS_CLOUDNAME)} '
++ export CLOUDPROMPT_ENABLED=1
++ CLOUDPROMPT_ENABLED=1
++ export OS_PROJECT_ID=c9cb46561bd446b9abc5e0d222b52e0e
++ OS_PROJECT_ID=c9cb46561bd446b9abc5e0d222b52e0e
++ export OS_TENANT_NAME=admin
++ OS_TENANT_NAME=admin
++ export OS_REGION_NAME=regionOne
++ OS_REGION_NAME=regionOne
+ '[' -f /home/jenkins/os_cacert ']'
+ source /home/jenkins/opnfv-installer.sh
++ export INSTALLER_TYPE=apex
++ INSTALLER_TYPE=apex
++ export INSTALLER_IP=192.168.122.31
++ INSTALLER_IP=192.168.122.31
++ export SSH_KEY=/home/jenkins/installer_key_file
++ SSH_KEY=/home/jenkins/installer_key_file
+ sudo -E tox -e py34
py34 create: /home/jenkins/opnfv/slave_root/workspace/doctor-verify-fault_management-apex-sample-x86_64-master/.tox/py34
py34 installdeps: -r/home/jenkins/opnfv/slave_root/workspace/doctor-verify-fault_management-apex-sample-x86_64-master/requirements.txt
py34 develop-inst: /home/jenkins/opnfv/slave_root/workspace/doctor-verify-fault_management-apex-sample-x86_64-master
py34 installed: You are using pip version 9.0.1, however version 18.1 is available.,You should consider upgrading via the 'pip install --upgrade pip' command.,amqp==2.2.1,aodhclient==0.9.0,appdirs==1.4.3,asn1crypto==0.22.0,Babel==2.3.4,bcrypt==3.1.3,cachetools==2.0.0,certifi==2017.4.17,cffi==1.10.0,chardet==3.0.4,click==6.7,cliff==2.8.2,cmd2==0.7.5,contextlib2==0.5.5,cryptography==2.0.2,debtcollector==1.17.1,deprecation==1.0.1,-e git+https://gerrit.opnfv.org/gerrit/doctor@da3ccd55ce2104a7ae338dacd74f81baf972dc3c#egg=doctor_tests,enum-compat==0.0.2,eventlet==0.20.0,fasteners==0.14.1,flake8==2.5.5,Flask==0.12.2,futurist==1.3.1,greenlet==0.4.12,idna==2.5,iso8601==0.1.11,itsdangerous==0.24,Jinja2==2.9.6,jsonpatch==1.16,jsonpointer==1.10,jsonschema==2.6.0,keystoneauth1==3.1.0,kombu==4.1.0,MarkupSafe==1.0,mccabe==0.4.0,monotonic==1.3,msgpack-python==0.4.8,netaddr==0.7.19,netifaces==0.10.6,openstacksdk==0.9.17,os-client-config==1.28.0,osc-lib==1.7.0,oslo.concurrency==3.21.1,oslo.config==4.11.1,oslo.context==2.17.1,oslo.i18n==3.17.1,oslo.log==3.30.2,oslo.messaging==5.30.6,oslo.middleware==3.30.1,oslo.serialization==2.20.2,oslo.service==1.25.1,oslo.utils==3.28.3,oslo.versionedobjects==1.26.2,paramiko==2.2.1,Paste==2.0.3,PasteDeploy==1.5.2,pbr==3.1.1,pep8==1.7.1,pika==0.10.0,pika-pool==0.1.3,positional==1.1.2,prettytable==0.7.2,pyasn1==0.3.1,pycparser==2.18,pyflakes==1.0.0,pyinotify==0.9.6,PyNaCl==1.1.2,pyOpenSSL==17.2.0,pyparsing==2.2.0,pyperclip==1.5.27,python-ceilometerclient==2.9.0,python-cinderclient==3.1.0,python-congressclient==1.8.0,python-dateutil==2.6.1,python-glanceclient==2.8.0,python-heatclient==1.11.1,python-keystoneclient==3.13.0,python-neutronclient==6.5.0,python-novaclient==9.1.2,python-openstackclient==3.12.1,python-swiftclient==3.4.0,python-vitrageclient==1.4.0,pytz==2017.2,PyYAML==3.12,repoze.lru==0.6,requests==2.18.2,requestsexceptions==1.3.0,rfc3986==1.1.0,Routes==2.4.1,scp==0.10.2,simplejson==3.11.1,six==1.10.0,statsd==3.2.1,stevedore==1.25.1,tenacity==4.4.0,urllib3==1.22,vine==1.1.4,virtualenv==15.1.0,warlock==1.2.0,WebOb==1.7.3,Werkzeug==0.12.2,wrapt==1.10.10
py34 runtests: PYTHONHASHSEED='2765055999'
py34 runtests: commands[0] | doctor-test
/home/jenkins/opnfv/slave_root/workspace/doctor-verify-fault_management-apex-sample-x86_64-master/.tox/py34/lib/python3.4/site-packages/paramiko/client.py:711: UserWarning: Unknown ssh-ed25519 host key for 192.168.122.31: b'4661c7c49d01d0948d60eebb60cbb8c0'
  key.get_fingerprint())))
2018-10-26 17:39:32,053 main.py 128 INFO   doctor test starting.......
2018-10-26 17:39:32,053 apex.py 43 INFO   Setup Apex installer start......
2018-10-26 17:39:32,054 base.py 92 INFO   Get SSH keys from apex installer......
2018-10-26 17:39:32,356 apex.py 60 INFO   Get overcloud config details from Apex installer......
2018-10-26 17:39:32,357 base.py 110 INFO   Run command=source stackrc; nova list | grep ' overcloud-' in apex installer......
2018-10-26 17:39:37,871 base.py 119 INFO   Output=['| 7f49a6f0-a75f-43e5-a469-61ef470eb56d | overcloud-controller-0  | ACTIVE | -          | Running     | ctlplane=192.0.2.10 |', '| be97f553-86ec-4ae1-b4d8-02728d8fff09 | overcloud-novacompute-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.8  |'] command=source stackrc; nova list | grep ' overcloud-' in apex installer
2018-10-26 17:39:37,872 base.py 124 INFO   Check command=grep docker /home/stack/deploy_command return in apex installer......
2018-10-26 17:39:37,935 base.py 127 INFO   return 0
2018-10-26 17:39:37,936 apex.py 73 INFO   controller_ips:['192.0.2.10']
2018-10-26 17:39:37,937 apex.py 74 INFO   compute_ips:['192.0.2.8']
2018-10-26 17:39:37,937 apex.py 75 INFO   use_containers:True
2018-10-26 17:39:39,556 apex.py 115 INFO   Set apply patches start......
/home/jenkins/opnfv/slave_root/workspace/doctor-verify-fault_management-apex-sample-x86_64-master/.tox/py34/lib/python3.4/site-packages/paramiko/client.py:711: UserWarning: Unknown ssh-ed25519 host key for 192.0.2.10: b'7a2bc1a11eb5322873693220bd120c6b'
  key.get_fingerprint())))
2018-10-26 17:39:42,279 base.py 61 INFO   Setup ssh stunnel in apex installer......
2018-10-26 17:39:42,280 base.py 74 INFO   tunnel for port 12346
2018-10-26 17:39:42,284 image.py 48 INFO   image create start......
2018-10-26 17:39:50,218 image.py 68 INFO   image create end......
2018-10-26 17:39:50,219 user.py 70 INFO   user create start......
2018-10-26 17:39:50,656 user.py 86 INFO   create project......
2018-10-26 17:39:50,918 user.py 95 INFO   test project <Project description=, domain_id=default, enabled=True, id=29feb8c33f9d4b7a839b9b77d296e2e4, is_domain=False, links={'self': 'http://192.0.2.3:35357/v3/projects/29feb8c33f9d4b7a839b9b77d296e2e4'}, name=doctor, parent_id=default, tags=[]>
2018-10-26 17:39:51,250 user.py 103 INFO   create user......
2018-10-26 17:39:51,829 user.py 113 INFO   test user <User domain_id=default, enabled=True, id=5c48b15a43b445d691f6e2e5d2d64620, links={'self': 'http://192.0.2.3:35357/v3/users/5c48b15a43b445d691f6e2e5d2d64620'}, name=doctor, options={}, password_expires_at=None>
2018-10-26 17:39:52,281 user.py 127 INFO   role _member_ already created......
2018-10-26 17:39:52,281 user.py 128 INFO   test role <Role description=None, domain_id=None, id=fc7ca3eb155e44108b8e35009816350a, links={'self': 'http://192.0.2.3:35357/v3/roles/fc7ca3eb155e44108b8e35009816350a'}, name=_member_>
2018-10-26 17:39:54,575 user.py 78 INFO   user create end......
2018-10-26 17:39:54,575 main.py 55 INFO   doctor fault management test starting.......
2018-10-26 17:39:55,087 fault_management.py 65 INFO   fault management setup......
2018-10-26 17:39:55,087 user.py 190 INFO   quota update start......
2018-10-26 17:39:55,087 user.py 206 INFO   default quota update start......
2018-10-26 17:39:56,812 user.py 217 INFO   user quota update start......
2018-10-26 17:39:57,578 user.py 230 INFO   quota update end......
2018-10-26 17:39:57,578 network.py 41 INFO   network create start.......
2018-10-26 17:40:02,775 network.py 47 INFO   network create end.......
2018-10-26 17:40:02,776 network.py 49 INFO   subnet create start.......
2018-10-26 17:40:03,955 network.py 58 INFO   subnet create end.......
2018-10-26 17:40:03,956 instance.py 51 INFO   instance create start......
2018-10-26 17:40:12,178 instance.py 73 INFO   instance create end......
2018-10-26 17:40:12,179 instance.py 92 INFO   wait for vm launch start......
2018-10-26 17:40:30,026 instance.py 110 INFO   wait for vm launch end......
2018-10-26 17:40:30,026 alarm.py 45 INFO   alarm create start......
2018-10-26 17:40:34,400 alarm.py 81 INFO   alarm create end......
2018-10-26 17:40:34,401 sample.py 73 INFO   sample inspector start......
2018-10-26 17:40:36,845 sample.py 26 INFO   sample consumer start......
 * Running on http://0.0.0.0:12346/ (Press CTRL+C to quit)
 * Running on http://0.0.0.0:12345/ (Press CTRL+C to quit)
2018-10-26 17:40:39,525 apex.py 79 INFO   Get host ip by hostname=overcloud-novacompute-0.opnfvlf.org from Apex installer......
2018-10-26 17:40:39,526 base.py 110 INFO   Run command=source stackrc; nova show overcloud-novacompute-0 | awk '/ ctlplane network /{print $5}' in apex installer......
2018-10-26 17:40:44,334 base.py 119 INFO   Output=['192.0.2.8'] command=source stackrc; nova show overcloud-novacompute-0 | awk '/ ctlplane network /{print $5}' in apex installer
2018-10-26 17:40:44,334 fault_management.py 118 INFO   Get host info(name:overcloud-novacompute-0.opnfvlf.org, ip:192.0.2.8) which vm(doctor_vm0) launched at
2018-10-26 17:40:44,335 sample.py 30 INFO   sample monitor start......
2018-10-26 17:40:44,336 sample.py 85 INFO   Starting Pinger host_name(overcloud-novacompute-0.opnfvlf.org), host_ip(192.0.2.8)
2018-10-26 17:42:44,435 fault_management.py 89 INFO   fault management start......
2018-10-26 17:42:44,436 base.py 92 INFO   Get SSH keys from apex installer......
2018-10-26 17:42:44,436 base.py 96 INFO   Already have SSH keys from apex installer......
/home/jenkins/opnfv/slave_root/workspace/doctor-verify-fault_management-apex-sample-x86_64-master/.tox/py34/lib/python3.4/site-packages/paramiko/client.py:711: UserWarning: Unknown ssh-ed25519 host key for 192.0.2.8: b'16d49da93499660d21d8f31257fad596'
  key.get_fingerprint())))
2018-10-26 17:42:44,601 utils.py 91 INFO   Copy /home/jenkins/opnfv/slave_root/workspace/doctor-verify-fault_management-apex-sample-x86_64-master/doctor_tests/disable_network.sh -> disable_network.sh
2018-10-26 17:42:45,038 utils.py 72 INFO   Executing: bash disable_network.sh > disable_network.log 2>&1 &
2018-10-26 17:42:45,065 utils.py 86 INFO   *** SUCCESSFULLY run command bash disable_network.sh > disable_network.log 2>&1 &
2018-10-26 17:42:45,066 fault_management.py 91 INFO   fault management end......
2018-10-26 17:42:50,662 sample.py 98 INFO   doctor monitor detected at 1540546970.661899
2018-10-26 17:42:50,662 sample.py 41 INFO   sample monitor report error......
2018-10-26 17:42:50,668 sample.py 202 INFO   event posted in sample inspector at 1540546970.6686914
2018-10-26 17:42:50,669 sample.py 203 INFO   sample inspector = <doctor_tests.inspector.sample.SampleInspector object at 0x7efd81aa3048>
2018-10-26 17:42:50,670 sample.py 205 INFO   sample inspector received data = b'[{"time": "2018-10-26T17:42:50.662655", "details": {"monitor": "monitor_sample", "status": "down", "monitor_event_id": "monitor_sample_event1", "hostname": "overcloud-novacompute-0.opnfvlf.org"}, "type": "compute.host.down"}]'
2018-10-26 17:42:50,792 sample.py 149 INFO   doctor mark host(overcloud-novacompute-0.opnfvlf.org) down at 1540546970.7928843
2018-10-26 17:42:52,211 sample.py 160 INFO   doctor mark vm(<Server: doctor_vm0>) error at 1540546972.211614
127.0.0.1 - - [26/Oct/2018 17:42:52] "PUT /events HTTP/1.1" 200 -
2018-10-26 17:42:52,216 sample.py 101 INFO   ping timeout, quit monitoring...
2018-10-26 17:42:52,442 sample.py 58 INFO   doctor consumer notified at 1540546972.4422183
2018-10-26 17:42:52,443 sample.py 61 INFO   sample consumer received data = {'alarm_id': 'a6970ac7-2a6f-424c-80e6-c3497f500ac5', 'alarm_name': 'doctor_alarm0', 'severity': 'moderate', 'previous': 'insufficient data', 'reason_data': {'event': {'generated': '2018-10-26T09:42:21.838939', 'message_signature': 'a6a0562fce2456fe794425a91b8a35dba8e3515735f3c2b1d3310cc9a916c7ed', 'event_type': 'compute.instance.update', 'message_id': 'ace8030c-97ab-4440-ab76-fd867b6702ad', 'raw': {}, 'traits': [['resource_id', 1, 'e55c9c7b-d5ef-4fde-aebc-1adfc67ec3c4'], ['ephemeral_gb', 2, 0], ['instance_type_id', 2, 1], ['user_id', 1, '5c48b15a43b445d691f6e2e5d2d64620'], ['service', 1, 'compute'], ['state', 1, 'error'], ['old_state', 1, 'active'], ['project_id', 1, '29feb8c33f9d4b7a839b9b77d296e2e4'], ['launched_at', 4, '2018-10-26T09:39:58'], ['disk_gb', 2, 1], ['instance_id', 1, 'e55c9c7b-d5ef-4fde-aebc-1adfc67ec3c4'], ['host', 1, 'overcloud-controller-0.opnfvlf.org'], ['display_name', 1, 'doctor_vm0'], ['root_gb', 2, 1], ['tenant_id', 1, '29feb8c33f9d4b7a839b9b77d296e2e4'], ['memory_mb', 2, 512], ['instance_type', 1, 'm1.tiny'], ['vcpus', 2, 1], ['request_id', 1, 'req-6853fc8e-dbda-4e77-b1b7-378bd69e8dd5']]}, 'type': 'event'}, 'reason': 'Event <id=ace8030c-97ab-4440-ab76-fd867b6702ad,event_type=compute.instance.update> hits the query <query=[{"field": "traits.instance_id", "op": "eq", "type": "", "value": "e55c9c7b-d5ef-4fde-aebc-1adfc67ec3c4"}, {"field": "traits.state", "op": "eq", "type": "", "value": "error"}]>.', 'current': 'alarm'}
127.0.0.1 - - [26/Oct/2018 17:42:52] "POST /failure HTTP/1.1" 200 -
2018-10-26 17:43:15,254 fault_management.py 185 INFO   doctor fault management notification_time=1.7803192138671875
2018-10-26 17:43:15,255 fault_management.py 188 INFO   doctor fault management test successfully
2018-10-26 17:43:15,255 fault_management.py 198 INFO   run doctor fault management profile.......
2018-10-26 17:43:15,256 base.py 92 INFO   Get SSH keys from apex installer......
2018-10-26 17:43:15,256 base.py 96 INFO   Already have SSH keys from apex installer......
2018-10-26 17:43:18,384 utils.py 91 INFO   Copy disable_network.log -> /home/jenkins/opnfv/slave_root/workspace/doctor-verify-fault_management-apex-sample-x86_64-master/doctor_tests/disable_network.log
2018-10-26 17:43:18,790 fault_management.py 155 INFO   Get the disable_netork.log fromdown_host(host_name:overcloud-novacompute-0.opnfvlf.org, host_ip:192.0.2.8)
2018-10-26 17:43:18,792 profiler_poc.py 97 INFO   
Total time cost: 35600(ms)
==============================================================================>
       |Monitor|Inspector           |Controller|Notifier|Evaluator           |
       |33820  |131                 |?         |?       |?                   |
       |       |      |      |      |          |        |      |      |      |
link down:0    |      |      |      |          |        |      |      |      |
     raw failure:33820|      |      |          |        |      |      |      |
         found affected:?    |      |          |        |      |      |      |
                  set VM error:35370|          |        |      |      |      |
                         marked host down:33951|        |      |      |      |
                               notified VM error:?      |      |      |      |
                                        transformed event:?    |      |      |
                                                 evaluated event:?    |      |
                                                            fired alarm:?    |
                                                                received alarm:35600

2018-10-26 17:43:18,793 fault_management.py 94 INFO   fault management cleanup......
2018-10-26 17:43:18,793 fault_management.py 136 INFO   Already get the disable_netork.log from down_host......
2018-10-26 17:43:20,979 sample.py 79 INFO   sample inspector stop......
2018-10-26 17:43:21,058 sample.py 217 INFO   shutdown inspector app server at 1540547001.058331
127.0.0.1 - - [26/Oct/2018 17:43:21] "POST /events/shutdown HTTP/1.1" 200 -
2018-10-26 17:43:21,061 sample.py 35 INFO   sample monitor stop......
2018-10-26 17:43:21,062 sample.py 108 INFO   Stopping Pinger host_name(overcloud-novacompute-0.opnfvlf.org), host_ip(192.0.2.8)
2018-10-26 17:43:21,063 sample.py 31 INFO   sample consumer stop......
2018-10-26 17:43:21,069 sample.py 66 INFO   shutdown consumer app server at 1540547001.069482
127.0.0.1 - - [26/Oct/2018 17:43:21] "POST /shutdown HTTP/1.1" 200 -
2018-10-26 17:43:21,072 alarm.py 84 INFO   alarm delete start.......
2018-10-26 17:43:22,736 alarm.py 93 INFO   alarm delete end.......
2018-10-26 17:43:22,736 instance.py 76 INFO   instance delete start.......
2018-10-26 17:43:51,520 instance.py 89 INFO   instance delete end.......
2018-10-26 17:43:51,520 network.py 61 INFO   subnet delete start.......
2018-10-26 17:43:52,559 network.py 64 INFO   subnet delete end.......
2018-10-26 17:43:52,560 network.py 66 INFO   network delete start.......
2018-10-26 17:43:54,265 network.py 69 INFO   network delete end.......
2018-10-26 17:43:54,265 apex.py 169 INFO   restore apply patches start......
2018-10-26 17:43:58,123 image.py 71 INFO   image delete start.......
2018-10-26 17:43:59,669 image.py 76 INFO   image delete end.......
2018-10-26 17:43:59,670 user.py 163 INFO   user delete start......
2018-10-26 17:44:01,526 user.py 187 INFO   user delete end......

Total time cost: 35600(ms)
==============================================================================>
       |Monitor|Inspector           |Controller|Notifier|Evaluator           |
       |33820  |131                 |?         |?       |?                   |
       |       |      |      |      |          |        |      |      |      |
link down:0    |      |      |      |          |        |      |      |      |
     raw failure:33820|      |      |          |        |      |      |      |
         found affected:?    |      |          |        |      |      |      |
                  set VM error:35370|          |        |      |      |      |
                         marked host down:33951|        |      |      |      |
                               notified VM error:?      |      |      |      |
                                        transformed event:?    |      |      |
                                                 evaluated event:?    |      |
                                                            fired alarm:?    |
                                                                received alarm:35600

___________________________________ summary ____________________________________
  py34: commands succeeded
  congratulations :)
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 14925 killed;
[ssh-agent] Stopped.
Archiving artifacts
No emails were triggered.
[WS-CLEANUP] Deleting project workspace...[WS-CLEANUP] done
Finished: SUCCESS