FailedConsole Output

Started by upstream project "doctor-verify-master" build number 146
originally caused by:
 Triggered by Gerrit: https://gerrit.opnfv.org/gerrit/66019
[EnvInject] - Loading node environment variables.
Building remotely on nokia-pod1 (nokia opnfv-sysinfo doctor-apex-x86_64) in workspace /home/jenkins/opnfv_slave_root/workspace/doctor-verify-fault_management-apex-congress-x86_64-master
[ssh-agent] Looking for ssh-agent implementation...
[ssh-agent]   Exec ssh-agent (binary ssh-agent on a remote machine)
$ ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-XCteYTc7hO57/agent.10743
SSH_AGENT_PID=10745
[ssh-agent] Started.
Running ssh-add (command line suppressed)
Identity added: /home/jenkins/opnfv_slave_root/workspace/doctor-verify-fault_management-apex-congress-x86_64-master@tmp/private_key_7759060911746478888.key (/home/jenkins/opnfv_slave_root/workspace/doctor-verify-fault_management-apex-congress-x86_64-master@tmp/private_key_7759060911746478888.key)
[ssh-agent] Using credentials jenkins-ci (Jenkins Master SSH)
using credential d42411ac011ad6f3dd2e1fa34eaa5d87f910eb2e
Wiping out workspace first.
Cloning the remote Git repository
Cloning repository https://gerrit.opnfv.org/gerrit/doctor
 > git init /home/jenkins/opnfv_slave_root/workspace/doctor-verify-fault_management-apex-congress-x86_64-master # timeout=10
Fetching upstream changes from https://gerrit.opnfv.org/gerrit/doctor
 > git --version # timeout=10
using GIT_SSH to set credentials Jenkins Master SSH
 > git fetch --tags --progress https://gerrit.opnfv.org/gerrit/doctor +refs/heads/*:refs/remotes/origin/* # timeout=15
 > git config remote.origin.url https://gerrit.opnfv.org/gerrit/doctor # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://gerrit.opnfv.org/gerrit/doctor # timeout=10
Fetching upstream changes from https://gerrit.opnfv.org/gerrit/doctor
using GIT_SSH to set credentials Jenkins Master SSH
 > git fetch --tags --progress https://gerrit.opnfv.org/gerrit/doctor refs/changes/19/66019/7 # timeout=15
Checking out Revision 73605c5c34b97ab56306bfa9af0f5888f3c7e46d (refs/changes/19/66019/7)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 73605c5c34b97ab56306bfa9af0f5888f3c7e46d # timeout=15
Commit message: "Support Fenix as admin tool"
 > git rev-parse FETCH_HEAD^{commit} # timeout=10
 > git rev-list --no-walk 33293e9c23a21ad3228f46d2063f18c915eb2b79 # timeout=10
No emails were triggered.
[doctor-verify-fault_management-apex-congress-x86_64-master] $ /usr/bin/env bash /tmp/jenkins8763989560560431569.sh
Gathering IP information for Apex installer VM
 6     undercloud                     running
Installer VM detected
Installer ip is 192.168.122.40
fetch_os_creds.info: Fetching rc file...
fetch_os_creds.info: Verifying connectivity to 192.168.122.40...
fetch_os_creds.info: 192.168.122.40 is reachable!
fetch_os_creds.info: ... from Instack VM 192.168.122.40...
Warning: Permanently added '192.168.122.40' (ECDSA) to the list of known hosts.
-------- Credentials: --------
# Clear any old environment that may conflict.
for key in $( set | awk '{FS="="}  /^OS_/ {print $1}' ); do unset $key ; done
export OS_NO_CACHE=True
export COMPUTE_API_VERSION=1.1
export OS_USERNAME=admin
export no_proxy=,192.168.37.19,192.0.2.4
export OS_USER_DOMAIN_NAME=Default
export OS_VOLUME_API_VERSION=3
export OS_CLOUDNAME=overcloud
export OS_AUTH_URL=http://192.168.37.19:5000/v3
export NOVA_VERSION=1.1
export OS_IMAGE_API_VERSION=2
#export OS_PASSWORD=N9flPG9TD7lwww7vsVewDHyBK
export OS_PASSWORD=admin
export OS_PROJECT_DOMAIN_NAME=Default
export OS_IDENTITY_API_VERSION=3
export OS_PROJECT_NAME=admin
export OS_AUTH_TYPE=password
export PYTHONWARNINGS="ignore:Certificate has no, ignore:A true SSLContext object is not available"

# Add OS_CLOUDNAME to PS1
if [ -z "${CLOUDPROMPT_ENABLED:-}" ]; then
    export PS1=${PS1:-""}
    export PS1=\${OS_CLOUDNAME:+"(\$OS_CLOUDNAME)"}\ $PS1
    export CLOUDPROMPT_ENABLED=1
fi
export OS_PROJECT_ID=6e76c202356a46cbbd626ef7c00efc0c
export OS_TENANT_NAME=admin
export OS_REGION_NAME=regionOne
[doctor-verify-fault_management-apex-congress-x86_64-master] $ /bin/sh -xe /tmp/jenkins6260255941118353256.sh
+ source /home/jenkins/opnfv-openrc.sh
+++ set
+++ awk '{FS="="}  /^OS_/ {print $1}'
++ export OS_NO_CACHE=True
++ OS_NO_CACHE=True
++ export COMPUTE_API_VERSION=1.1
++ COMPUTE_API_VERSION=1.1
++ export OS_USERNAME=admin
++ OS_USERNAME=admin
++ export no_proxy=,192.168.37.19,192.0.2.4
++ no_proxy=,192.168.37.19,192.0.2.4
++ export OS_USER_DOMAIN_NAME=Default
++ OS_USER_DOMAIN_NAME=Default
++ export OS_VOLUME_API_VERSION=3
++ OS_VOLUME_API_VERSION=3
++ export OS_CLOUDNAME=overcloud
++ OS_CLOUDNAME=overcloud
++ export OS_AUTH_URL=http://192.168.37.19:5000/v3
++ OS_AUTH_URL=http://192.168.37.19:5000/v3
++ export NOVA_VERSION=1.1
++ NOVA_VERSION=1.1
++ export OS_IMAGE_API_VERSION=2
++ OS_IMAGE_API_VERSION=2
++ export OS_PASSWORD=admin
++ OS_PASSWORD=admin
++ export OS_PROJECT_DOMAIN_NAME=Default
++ OS_PROJECT_DOMAIN_NAME=Default
++ export OS_IDENTITY_API_VERSION=3
++ OS_IDENTITY_API_VERSION=3
++ export OS_PROJECT_NAME=admin
++ OS_PROJECT_NAME=admin
++ export OS_AUTH_TYPE=password
++ OS_AUTH_TYPE=password
++ export 'PYTHONWARNINGS=ignore:Certificate has no, ignore:A true SSLContext object is not available'
++ PYTHONWARNINGS='ignore:Certificate has no, ignore:A true SSLContext object is not available'
++ '[' -z '' ']'
++ export PS1=
++ PS1=
++ export 'PS1=${OS_CLOUDNAME:+($OS_CLOUDNAME)} '
++ PS1='${OS_CLOUDNAME:+($OS_CLOUDNAME)} '
++ export CLOUDPROMPT_ENABLED=1
++ CLOUDPROMPT_ENABLED=1
++ export OS_PROJECT_ID=6e76c202356a46cbbd626ef7c00efc0c
++ OS_PROJECT_ID=6e76c202356a46cbbd626ef7c00efc0c
++ export OS_TENANT_NAME=admin
++ OS_TENANT_NAME=admin
++ export OS_REGION_NAME=regionOne
++ OS_REGION_NAME=regionOne
+ '[' -f /home/jenkins/os_cacert ']'
+ source /home/jenkins/opnfv-installer.sh
++ export INSTALLER_TYPE=apex
++ INSTALLER_TYPE=apex
++ export INSTALLER_IP=192.168.122.40
++ INSTALLER_IP=192.168.122.40
++ export SSH_KEY=/home/jenkins/installer_key_file
++ SSH_KEY=/home/jenkins/installer_key_file
+ sudo -E tox -e py34
py34 create: /home/jenkins/opnfv_slave_root/workspace/doctor-verify-fault_management-apex-congress-x86_64-master/.tox/py34
py34 installdeps: -r/home/jenkins/opnfv_slave_root/workspace/doctor-verify-fault_management-apex-congress-x86_64-master/requirements.txt
py34 develop-inst: /home/jenkins/opnfv_slave_root/workspace/doctor-verify-fault_management-apex-congress-x86_64-master
py34 installed: DEPRECATION: Python 3.4 support has been deprecated. pip 19.1 will be the last one supporting it. Please upgrade your Python as Python 3.4 won't be maintained after March 2019 (cf PEP 429).,amqp==2.2.1,aodhclient==0.9.0,appdirs==1.4.3,asn1crypto==0.22.0,Babel==2.3.4,bcrypt==3.1.3,cachetools==2.0.0,certifi==2017.4.17,cffi==1.10.0,chardet==3.0.4,click==6.7,cliff==2.8.2,cmd2==0.7.5,contextlib2==0.5.5,cryptography==2.0.2,debtcollector==1.17.1,deprecation==1.0.1,-e git+https://gerrit.opnfv.org/gerrit/doctor@73605c5c34b97ab56306bfa9af0f5888f3c7e46d#egg=doctor_tests,enum-compat==0.0.2,eventlet==0.20.0,fasteners==0.14.1,flake8==2.5.5,Flask==0.12.2,futurist==1.3.1,greenlet==0.4.15,idna==2.5,iso8601==0.1.11,itsdangerous==0.24,Jinja2==2.9.6,jsonpatch==1.16,jsonpointer==1.10,jsonschema==2.6.0,keystoneauth1==3.1.0,kombu==4.1.0,MarkupSafe==1.0,mccabe==0.4.0,monotonic==1.3,msgpack-python==0.4.8,netaddr==0.7.19,netifaces==0.10.6,openstacksdk==0.9.17,os-client-config==1.28.0,osc-lib==1.7.0,oslo.concurrency==3.21.1,oslo.config==4.11.1,oslo.context==2.17.1,oslo.i18n==3.17.1,oslo.log==3.30.2,oslo.messaging==5.30.7,oslo.middleware==3.30.1,oslo.serialization==2.20.2,oslo.service==1.25.1,oslo.utils==3.28.3,oslo.versionedobjects==1.26.2,paramiko==2.2.1,Paste==2.0.3,PasteDeploy==1.5.2,pbr==3.1.1,pep8==1.7.1,pika==0.10.0,pika-pool==0.1.3,positional==1.1.2,prettytable==0.7.2,pyasn1==0.3.1,pycparser==2.18,pyflakes==1.0.0,pyinotify==0.9.6,PyNaCl==1.1.2,pyOpenSSL==17.2.0,pyparsing==2.2.0,pyperclip==1.5.27,python-ceilometerclient==2.9.0,python-cinderclient==3.1.1,python-congressclient==1.8.0,python-dateutil==2.6.1,python-glanceclient==2.8.0,python-heatclient==1.11.1,python-keystoneclient==3.13.0,python-neutronclient==6.5.0,python-novaclient==9.1.2,python-openstackclient==3.12.1,python-swiftclient==3.4.0,python-vitrageclient==1.4.0,pytz==2017.2,PyYAML==3.12,repoze.lru==0.6,requests==2.18.2,requestsexceptions==1.3.0,rfc3986==1.1.0,Routes==2.4.1,scp==0.10.2,simplejson==3.11.1,six==1.10.0,statsd==3.2.1,stevedore==1.25.1,tenacity==4.4.0,urllib3==1.22,vine==1.1.4,virtualenv==15.1.0,warlock==1.2.0,WebOb==1.7.3,Werkzeug==0.12.2,wrapt==1.10.10
py34 runtests: PYTHONHASHSEED='2538408964'
py34 runtests: commands[0] | doctor-test
/home/jenkins/opnfv_slave_root/workspace/doctor-verify-fault_management-apex-congress-x86_64-master/.tox/py34/lib/python3.4/site-packages/paramiko/client.py:711: UserWarning: Unknown ssh-ed25519 host key for 192.168.122.40: b'3dc1628c6fe2819031608f780639f40a'
  key.get_fingerprint())))
2019-03-26 18:50:27,087 main.py 130 INFO   doctor test starting.......
2019-03-26 18:50:27,087 apex.py 43 INFO   Setup Apex installer start......
2019-03-26 18:50:27,088 base.py 113 INFO   Get SSH keys from apex installer......
2019-03-26 18:50:27,517 apex.py 67 INFO   Get overcloud config details from Apex installer......
2019-03-26 18:50:27,517 base.py 174 INFO   Run command=source stackrc; nova list | grep ' overcloud-' in apex installer......
2019-03-26 18:50:31,462 base.py 183 INFO   Output=['| 2d0aedd7-4e30-4fee-bbd2-0feb8b1078c4 | overcloud-controller-0  | ACTIVE | -          | Running     | ctlplane=192.0.2.9 |', '| 37ceec17-a130-4826-87f0-67699d3350f5 | overcloud-novacompute-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.6 |', '| 416383de-15fb-477a-9a6c-f0b21df9be70 | overcloud-novacompute-1 | ACTIVE | -          | Running     | ctlplane=192.0.2.7 |', '| 0f025a92-3365-4c3b-951e-dab9a6d9ca22 | overcloud-novacompute-2 | ACTIVE | -          | Running     | ctlplane=192.0.2.3 |'] command=source stackrc; nova list | grep ' overcloud-' in apex installer
2019-03-26 18:50:31,463 base.py 188 INFO   Check command=grep docker /home/stack/deploy_command return in apex installer......
2019-03-26 18:50:31,535 base.py 191 INFO   return 0
2019-03-26 18:50:31,536 apex.py 80 INFO   controller_ips:['192.0.2.9']
2019-03-26 18:50:31,536 apex.py 81 INFO   compute_ips:['192.0.2.6', '192.0.2.7', '192.0.2.3']
2019-03-26 18:50:31,537 apex.py 82 INFO   use_containers:True
2019-03-26 18:50:32,436 apex.py 102 INFO   Set apply patches start......
/home/jenkins/opnfv_slave_root/workspace/doctor-verify-fault_management-apex-congress-x86_64-master/.tox/py34/lib/python3.4/site-packages/paramiko/client.py:711: UserWarning: Unknown ssh-ed25519 host key for 192.0.2.9: b'b3b023c9e950672f64d119fecdae83fd'
  key.get_fingerprint())))
2019-03-26 18:50:32,880 base.py 218 INFO   Command sudo python set_config.py output ['Add event notifier in ceilometer', 'NOTE: add compute.instance.update to event_definitions.yaml', 'NOTE: add maintenance.scheduled to event_definitions.yaml', 'NOTE: add maintenance.host to event_definitions.yaml']
2019-03-26 18:50:34,583 base.py 218 INFO   Command sudo python restart_aodh.py output []
2019-03-26 18:50:34,746 base.py 218 INFO   Command sudo python set_congress.py output []
2019-03-26 18:50:37,520 apex.py 148 INFO   Set apply patches start......
2019-03-26 18:50:37,520 base.py 63 INFO   Setup ssh stunnel in apex installer......
2019-03-26 18:50:37,521 base.py 76 INFO   tunnel for port 12346
2019-03-26 18:50:37,525 image.py 48 INFO   image create start......
2019-03-26 18:50:45,687 image.py 68 INFO   image create end......
2019-03-26 18:50:45,688 user.py 70 INFO   user create start......
2019-03-26 18:50:45,855 user.py 86 INFO   create project......
2019-03-26 18:50:46,202 user.py 95 INFO   test project <Project description=, domain_id=default, enabled=True, id=03962f8c452743feb49ac6f52451659a, is_domain=False, links={'self': 'http://192.0.2.4:35357/v3/projects/03962f8c452743feb49ac6f52451659a'}, name=doctor, parent_id=default, tags=[]>
2019-03-26 18:50:46,474 user.py 103 INFO   create user......
2019-03-26 18:50:47,094 user.py 113 INFO   test user <User domain_id=default, enabled=True, id=44cbb096822047d4962395d4f7fcd2e9, links={'self': 'http://192.0.2.4:35357/v3/users/44cbb096822047d4962395d4f7fcd2e9'}, name=doctor, options={}, password_expires_at=None>
2019-03-26 18:50:47,341 user.py 127 INFO   role _member_ already created......
2019-03-26 18:50:47,341 user.py 128 INFO   test role <Role domain_id=None, id=9fe2ff9ee4384b1894a90878d3e92bab, links={'self': 'http://192.0.2.4:35357/v3/roles/9fe2ff9ee4384b1894a90878d3e92bab'}, name=_member_>
2019-03-26 18:50:48,889 user.py 78 INFO   user create end......
2019-03-26 18:50:48,889 main.py 55 INFO   doctor fault management test starting.......
2019-03-26 18:50:50,716 main.py 82 ERROR  doctor fault management test failed, Exception='nova'
2019-03-26 18:50:50,718 main.py 83 ERROR  Traceback (most recent call last):
  File "/home/jenkins/opnfv_slave_root/workspace/doctor-verify-fault_management-apex-congress-x86_64-master/doctor_tests/main.py", line 59, in test_fault_management
    transport_url)
  File "/home/jenkins/opnfv_slave_root/workspace/doctor-verify-fault_management-apex-congress-x86_64-master/doctor_tests/scenario/fault_management.py", line 58, in __init__
    self.inspector = get_inspector(self.conf, log, transport_url)
  File "/home/jenkins/opnfv_slave_root/workspace/doctor-verify-fault_management-apex-congress-x86_64-master/doctor_tests/inspector/__init__.py", line 51, in get_inspector
    return importutils.import_object(inspector_class, conf, log)
  File "/home/jenkins/opnfv_slave_root/workspace/doctor-verify-fault_management-apex-congress-x86_64-master/.tox/py34/lib/python3.4/site-packages/oslo_utils/importutils.py", line 44, in import_object
    return import_class(import_str)(*args, **kwargs)
  File "/home/jenkins/opnfv_slave_root/workspace/doctor-verify-fault_management-apex-congress-x86_64-master/doctor_tests/inspector/congress.py", line 38, in __init__
    self._init_driver_and_ds()
  File "/home/jenkins/opnfv_slave_root/workspace/doctor-verify-fault_management-apex-congress-x86_64-master/doctor_tests/inspector/congress.py", line 47, in _init_driver_and_ds
    nova_api_version = datasources['nova']['config'].get('api_version')
KeyError: 'nova'

2019-03-26 18:50:50,718 main.py 147 ERROR  doctor test failed, Exception='DoctorTest' object has no attribute 'fault_management'
2019-03-26 18:50:50,719 main.py 148 ERROR  Traceback (most recent call last):
  File "/home/jenkins/opnfv_slave_root/workspace/doctor-verify-fault_management-apex-congress-x86_64-master/doctor_tests/main.py", line 59, in test_fault_management
    transport_url)
  File "/home/jenkins/opnfv_slave_root/workspace/doctor-verify-fault_management-apex-congress-x86_64-master/doctor_tests/scenario/fault_management.py", line 58, in __init__
    self.inspector = get_inspector(self.conf, log, transport_url)
  File "/home/jenkins/opnfv_slave_root/workspace/doctor-verify-fault_management-apex-congress-x86_64-master/doctor_tests/inspector/__init__.py", line 51, in get_inspector
    return importutils.import_object(inspector_class, conf, log)
  File "/home/jenkins/opnfv_slave_root/workspace/doctor-verify-fault_management-apex-congress-x86_64-master/.tox/py34/lib/python3.4/site-packages/oslo_utils/importutils.py", line 44, in import_object
    return import_class(import_str)(*args, **kwargs)
  File "/home/jenkins/opnfv_slave_root/workspace/doctor-verify-fault_management-apex-congress-x86_64-master/doctor_tests/inspector/congress.py", line 38, in __init__
    self._init_driver_and_ds()
  File "/home/jenkins/opnfv_slave_root/workspace/doctor-verify-fault_management-apex-congress-x86_64-master/doctor_tests/inspector/congress.py", line 47, in _init_driver_and_ds
    nova_api_version = datasources['nova']['config'].get('api_version')
KeyError: 'nova'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jenkins/opnfv_slave_root/workspace/doctor-verify-fault_management-apex-congress-x86_64-master/doctor_tests/main.py", line 84, in test_fault_management
    sys.exit(1)
SystemExit: 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jenkins/opnfv_slave_root/workspace/doctor-verify-fault_management-apex-congress-x86_64-master/doctor_tests/main.py", line 141, in run
    getattr(self, function)()
  File "/home/jenkins/opnfv_slave_root/workspace/doctor-verify-fault_management-apex-congress-x86_64-master/doctor_tests/main.py", line 86, in test_fault_management
    self.fault_management.cleanup()
AttributeError: 'DoctorTest' object has no attribute 'fault_management'

2019-03-26 18:50:50,719 apex.py 166 INFO   restore apply patches start......
2019-03-26 18:50:50,796 image.py 71 INFO   image delete start.......
2019-03-26 18:50:51,131 base.py 218 INFO   Command sudo python restore_config.py output ['restore', 'restore: /var/lib/config-data/puppet-generated/ceilometer/etc/ceilometer/event_definitions.yaml', 'Bak_file empty, so removing also: /var/lib/config-data/puppet-generated/ceilometer/etc/ceilometer/event_definitions.yaml']
2019-03-26 18:50:52,096 image.py 76 INFO   image delete end.......
2019-03-26 18:50:52,096 user.py 163 INFO   user delete start......
2019-03-26 18:50:52,542 base.py 218 INFO   Command sudo python restore_aodh.py output []
2019-03-26 18:50:52,662 base.py 218 INFO   Command sudo python restore_congress.py output ['restore: /var/lib/config-data/puppet-generated/congress/etc/congress/congress.conf']
2019-03-26 18:50:53,215 user.py 187 INFO   user delete end......
ERROR: InvocationError for command '/home/jenkins/opnfv_slave_root/workspace/doctor-verify-fault_management-apex-congress-x86_64-master/.tox/py34/bin/doctor-test' (exited with code 1)
___________________________________ summary ____________________________________
ERROR:   py34: commands failed
Build step 'Execute shell' marked build as failure
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 10745 killed;
[ssh-agent] Stopped.
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
Request made to compress build log
Sending email to: tbramwell@linuxfoundation.org agardner@linuxfoundation.org rgrigar@linuxfoundation.org
[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
Cannot delete workspace: java.lang.NoClassDefFoundError: Could not initialize class sun.nio.fs.UnixCopyFile
Option not to fail the build is turned on, so let's continue
Finished: FAILURE