ubuntu@cord-rs-s8:~/cord-tester/src/test/setup$ sudo ./cord-test.py run -m manifest-ponsim.json -t voltha:voltha_exchange.test_two_subscribers_with_voltha_for_igmp_toggling_uni_port_for_one_subscriber_and_verifying_traffic
# Host [172.17.0.2]:8101 found: line 3 type RSA
onoscli: eof exception found
onoscli:     logout

Connection to 172.17.0.2 closed.

result.ok, result.status_code are False and 409
ONOS app cord-config, version 2.0-SNAPSHOT installed
result.ok, result.status_code are False and 409
ONOS app aaa, version 2.0-SNAPSHOT installed
result.ok, result.status_code are False and 409
ONOS app igmp, version 2.0-SNAPSHOT installed
Controller IP [u'172.17.0.2'], Test type voltha:voltha_exchange.test_two_subscribers_with_voltha_for_igmp_toggling_uni_port_for_one_subscriber_and_verifying_traffic
Installing cord tester ONOS app /home/ubuntu/cord-tester/src/test/apps/ciena-cordigmp-multitable-3.0-SNAPSHOT.oar
result.ok, result.status_code are False and 409
WARNING: The DOCKER_HOST_IP variable is not set. Defaulting to a blank string.
Creating network "compose_default" with driver "bridge"
Creating network "compose_ponmgmt" with driver "bridge"
Creating compose_zookeeper_1
Creating compose_consul_1
Creating compose_fluentd_1
Creating compose_kafka_1
Creating compose_registrator_1
IP 172.18.0.2 for service consul
IP 172.18.0.5 for service kafka
IP 172.18.0.4 for service zookeeper
IP 172.18.0.6 for service registrator
IP 172.18.0.3 for service fluentd
Radius server running with IP 172.17.0.3
Running ['voltha:voltha_exchange.test_two_subscribers_with_voltha_for_igmp_toggling_uni_port_for_one_subscriber_and_verifying_traffic'] tests across 1 containers in parallel
Starting test container cord-tester1, image cordtest/nose, tag candidate
Provisioning the ports for the test container

Running PIPEWORK cmd: pipework pon1_128 -i veth0 -l l1 cord-tester1 192.168.100.1/24
Running PIPEWORK cmd: pipework ponmgmt -i veth2 -l l2 cord-tester1 192.168.100.2/24
Running PIPEWORK cmd: pipework pon1_129 -i veth4 -l l3 cord-tester1 192.168.100.3/24
Running PIPEWORK cmd: pipework pon1_130 -i veth6 -l l4 cord-tester1 192.168.100.4/24
Running PIPEWORK cmd: pipework pon1_131 -i veth8 -l l5 cord-tester1 192.168.100.5/24
Running PIPEWORK cmd: pipework pon1_132 -i veth10 -l l6 cord-tester1 192.168.100.6/24
Running PIPEWORK cmd: pipework pon1_133 -i veth12 -l l7 cord-tester1 192.168.100.7/24
Running PIPEWORK cmd: pipework pon1_134 -i veth14 -l l8 cord-tester1 192.168.100.8/24
Running PIPEWORK cmd: pipework pon1_135 -i veth16 -l l9 cord-tester1 192.168.100.9/24
Running PIPEWORK cmd: pipework pon1_136 -i veth18 -l l10 cord-tester1 192.168.100.10/24
Running PIPEWORK cmd: pipework pon1_137 -i veth20 -l l11 cord-tester1 192.168.100.11/24
Running PIPEWORK cmd: pipework pon1_128 -i veth22 -l l12 cord-tester1 192.168.100.12/24
Running PIPEWORK cmd: pipework pon1_129 -i veth24 -l l13 cord-tester1 192.168.100.13/24
Running PIPEWORK cmd: pipework pon1_130 -i veth26 -l l14 cord-tester1 192.168.100.14/24
Running PIPEWORK cmd: pipework pon1_131 -i veth28 -l l15 cord-tester1 192.168.100.15/24
Running PIPEWORK cmd: pipework pon1_132 -i veth30 -l l16 cord-tester1 192.168.100.16/24
Running PIPEWORK cmd: pipework pon1_133 -i veth32 -l l17 cord-tester1 192.168.100.17/24
Running PIPEWORK cmd: pipework pon1_134 -i veth34 -l l18 cord-tester1 192.168.100.18/24
Running PIPEWORK cmd: pipework pon1_135 -i veth36 -l l19 cord-tester1 192.168.100.19/24
Running PIPEWORK cmd: pipework pon1_136 -i veth38 -l l20 cord-tester1 192.168.100.20/24
Running PIPEWORK cmd: pipework pon1_137 -i veth40 -l l21 cord-tester1 192.168.100.21/24
Running PIPEWORK cmd: pipework pon1_138 -i veth42 -l l22 cord-tester1 192.168.100.22/24
Modifying scapy tool files before running a test: ['voltha:voltha_exchange.test_two_subscribers_with_voltha_for_igmp_toggling_uni_port_for_one_subscriber_and_verifying_traffic']
Running tests: ['voltha:voltha_exchange.test_two_subscribers_with_voltha_for_igmp_toggling_uni_port_for_one_subscriber_and_verifying_traffic']
WARNING: No route found for IPv6 destination :: (no default route?)
Connecting to controller at 172.17.0.2
onoscli: Trying to connect to 172.17.0.2
Spawning pexpect for ip 172.17.0.2
ssh connection asked for password, gave password
Command 'summary -j' sent to onoscli.
onoscli: eof exception found
onoscli:     logout

Connection to 172.17.0.2 closed.

Installing the multi table app /root/test/src/test/voltha/../apps/ciena-cordigmp-multitable-3.0-SNAPSHOT.oar for subscriber test
Test Method: ... Enabling ponsim olt
Pre-provisioning ponsim_olt with address 172.17.0.1:50060
Enabling device 340b50adba61
Checking operational status for device 340b50adba61
Installing OLT app
Adding subscribers through OLT app
172.17.0.4 - - [01/Aug/2017 09:49:10] "POST /RPC2 HTTP/1.1" 200 -
Adding group 225.0.0.1
Adding group 225.0.0.1
All subscribers have joined the channel
Subscriber on port veth4 checking data traffic receiving from group 225.0.0.1, channel 0
All subscribers have joined the channel
Subscriber on port veth0 checking data traffic receiving from group 225.0.0.1, channel 0
Subscriber on port veth4 timed out
Admin state of uni port is down and up after delay of 30 sec during tls auth flow check on voltha
Admin state of uni_port is down
Subscriber on port veth0 timed out
Joining channel 0 for subscriber port veth4
All subscribers have joined the channel
Subscriber on port veth4 checking data traffic receiving from group 225.0.0.1, channel 0
Joining channel 0 for subscriber port veth0
All subscribers have joined the channel
Subscriber on port veth0 checking data traffic receiving from group 225.0.0.1, channel 0
Subscriber on port veth4 timed out
Subscriber on port veth0 timed out
Joining channel 0 for subscriber port veth4
All subscribers have joined the channel
Subscriber on port veth4 checking data traffic receiving from group 225.0.0.1, channel 0
Joining channel 0 for subscriber port veth0
All subscribers have joined the channel
Subscriber on port veth0 checking data traffic receiving from group 225.0.0.1, channel 0
Subscriber on port veth4 timed out
Admin state of uni_port is up now
Subscriber on port veth0 timed out
Joining channel 0 for subscriber port veth4
All subscribers have joined the channel
Subscriber on port veth4 checking data traffic receiving from group 225.0.0.1, channel 0
Joining channel 0 for subscriber port veth0
All subscribers have joined the channel
Subscriber on port veth0 checking data traffic receiving from group 225.0.0.1, channel 0
Subscriber on port veth4 timed out
Subscriber on port veth0 timed out
Joining channel 0 for subscriber port veth4
All subscribers have joined the channel
Subscriber on port veth4 checking data traffic receiving from group 225.0.0.1, channel 0
Joining channel 0 for subscriber port veth0
All subscribers have joined the channel
Subscriber on port veth0 checking data traffic receiving from group 225.0.0.1, channel 0
Subscriber on port veth4 timed out
Subscriber on port veth0 timed out
Joining channel 0 for subscriber port veth4
All subscribers have joined the channel
Subscriber on port veth4 checking data traffic receiving from group 225.0.0.1, channel 0
Joining channel 0 for subscriber port veth0
All subscribers have joined the channel
Subscriber on port veth0 checking data traffic receiving from group 225.0.0.1, channel 0
Subscriber on port veth4 timed out
Subscriber on port veth0 timed out
Joining channel 0 for subscriber port veth4
All subscribers have joined the channel
Subscriber on port veth4 checking data traffic receiving from group 225.0.0.1, channel 0
Joining channel 0 for subscriber port veth0
All subscribers have joined the channel
Subscriber on port veth0 checking data traffic receiving from group 225.0.0.1, channel 0
Subscriber on port veth4 timed out
Subscriber on port veth0 timed out
Joining channel 0 for subscriber port veth4
All subscribers have joined the channel
Subscriber on port veth4 checking data traffic receiving from group 225.0.0.1, channel 0
Joining channel 0 for subscriber port veth0
All subscribers have joined the channel
Subscriber on port veth0 checking data traffic receiving from group 225.0.0.1, channel 0
Subscriber on port veth4 timed out
Subscriber on port veth0 timed out
Joining channel 0 for subscriber port veth4
All subscribers have joined the channel
Subscriber on port veth4 checking data traffic receiving from group 225.0.0.1, channel 0
Joining channel 0 for subscriber port veth0
All subscribers have joined the channel
Subscriber on port veth0 checking data traffic receiving from group 225.0.0.1, channel 0
Subscriber on port veth4 timed out
Subscriber on port veth0 timed out
Joining channel 0 for subscriber port veth4
All subscribers have joined the channel
Joining channel 0 for subscriber port veth0
Subscriber on port veth4 checking data traffic receiving from group 225.0.0.1, channel 0
All subscribers have joined the channel
Subscriber on port veth0 checking data traffic receiving from group 225.0.0.1, channel 0
Subscriber on port veth4 timed out
Subscriber on port veth0 timed out
Joining channel 0 for subscriber port veth4
All subscribers have joined the channel
Joining channel 0 for subscriber port veth0
Subscriber on port veth4 checking data traffic receiving from group 225.0.0.1, channel 0
All subscribers have joined the channel
Subscriber on port veth0 checking data traffic receiving from group 225.0.0.1, channel 0
Subscriber on port veth4 timed out
Subscriber on port veth0 timed out
Joining channel 0 for subscriber port veth4
This service is failed and other services will not run for this subscriber
This Subscriber is tested for multiple service eligibility
Joining channel 0 for subscriber port veth0
This service is failed and other services will not run for this subscriber
This Subscriber is tested for multiple service eligibility
Error deleting subscriber for device of:0000000000000001 on port 128
Error deleting subscriber for device of:0000000000000001 on port 129
Error deleting subscriber for device of:0000000000000001 on port 130
Error deleting subscriber for device of:0000000000000001 on port 131
Error deleting subscriber for device of:0000000000000001 on port 132
Error deleting subscriber for device of:0000000000000001 on port 133
Error deleting subscriber for device of:0000000000000001 on port 134
Error deleting subscriber for device of:0000000000000001 on port 135
Error deleting subscriber for device of:0000000000000001 on port 136
Error deleting subscriber for device of:0000000000000001 on port 137
Error deleting subscriber for device of:0000000000000001 on port 138
Disabling device 340b50adba61
Deleting device 340b50adba61
Uninstalling OLT app
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/root/test/src/test/voltha/volthaTest.py", line 1444, in voltha_subscribers
    assert_equal(test_status, True)
  File "/usr/lib/python2.7/unittest/case.py", line 515, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/lib/python2.7/unittest/case.py", line 508, in _baseAssertEqual
    raise self.failureException(msg)
AssertionError: False != True

Unhandled Error
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/nose-1.3.7-py2.7.egg/nose/twistedtools.py", line 57, in <lambda>
    installSignalHandlers=False))
  File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1192, in run
    self.mainLoop()
  File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1201, in mainLoop
    self.runUntilCurrent()
--- <exception caught here> ---
  File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 824, in runUntilCurrent
    call.func(*call.args, **call.kw)
  File "/root/test/src/test/voltha/volthaTest.py", line 5026, in igmp_flow_check_operating_onu_admin_state
    assert_equal(self.success, True)
  File "/usr/lib/python2.7/unittest/case.py", line 515, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/lib/python2.7/unittest/case.py", line 508, in _baseAssertEqual
    raise self.failureException(msg)
exceptions.AssertionError: False != True
FAIL
Installing back the cord igmp app /root/test/src/test/voltha/../apps/ciena-cordigmp-3.0-SNAPSHOT.oar for subscriber test on exit

======================================================================
FAIL: Test Method:
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/nose-1.3.7-py2.7.egg/nose/twistedtools.py", line 165, in wrapper
    % timeout)
TimeExpired: timeout expired before end of test (300.000000 s.)
-------------------- >> begin captured stdout << ---------------------
result.ok, result.status_code are False and 409

--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
scapy.runtime: WARNING: No route found for IPv6 destination :: (no default route?)
docker.auth.auth: DEBUG: Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
docker.auth.auth: DEBUG: No config file found
docker.auth.auth: DEBUG: Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
docker.auth.auth: DEBUG: No config file found
cordTester: INFO: Connecting to controller at 172.17.0.2
cordTester: INFO: onoscli: Trying to connect to 172.17.0.2
onoscli: INFO:
cordTester: INFO: Spawning pexpect for ip 172.17.0.2
cordTester: INFO: ssh connection asked for password, gave password
cordTester: INFO: Command 'summary -j' sent to onoscli.
cordTester: ERROR: onoscli: eof exception found
cordTester: ERROR: onoscli:     logout

Connection to 172.17.0.2 closed.

cordTester: INFO: Installing the multi table app /root/test/src/test/voltha/../apps/ciena-cordigmp-multitable-3.0-SNAPSHOT.oar for subscriber test
cordTester: INFO: Enabling ponsim olt
cordTester: INFO: Pre-provisioning ponsim_olt with address 172.17.0.1:50060
cordTester: INFO: Enabling device 340b50adba61
cordTester: INFO: Checking operational status for device 340b50adba61
cordTester: INFO: Installing OLT app
cordTester: INFO: Adding subscribers through OLT app
cordTester: DEBUG: Adding group 225.0.0.1
cordTester: DEBUG: Adding group 225.0.0.1
cordTester: INFO: All subscribers have joined the channel
cordTester: INFO: Subscriber on port veth4 checking data traffic receiving from group 225.0.0.1, channel 0
cordTester: INFO: All subscribers have joined the channel
cordTester: INFO: Subscriber on port veth0 checking data traffic receiving from group 225.0.0.1, channel 0
cordTester: INFO: Subscriber on port veth4 timed out
cordTester: INFO: Admin state of uni port is down and up after delay of 30 sec during tls auth flow check on voltha
cordTester: INFO: Admin state of uni_port is down
cordTester: INFO: Subscriber on port veth0 timed out
cordTester: INFO: Joining channel 0 for subscriber port veth4
cordTester: INFO: All subscribers have joined the channel
cordTester: INFO: Subscriber on port veth4 checking data traffic receiving from group 225.0.0.1, channel 0
cordTester: INFO: Joining channel 0 for subscriber port veth0
cordTester: INFO: All subscribers have joined the channel
cordTester: INFO: Subscriber on port veth0 checking data traffic receiving from group 225.0.0.1, channel 0
cordTester: INFO: Subscriber on port veth4 timed out
cordTester: INFO: Subscriber on port veth0 timed out
cordTester: INFO: Joining channel 0 for subscriber port veth4
cordTester: INFO: All subscribers have joined the channel
cordTester: INFO: Subscriber on port veth4 checking data traffic receiving from group 225.0.0.1, channel 0
cordTester: INFO: Joining channel 0 for subscriber port veth0
cordTester: INFO: All subscribers have joined the channel
cordTester: INFO: Subscriber on port veth0 checking data traffic receiving from group 225.0.0.1, channel 0
cordTester: INFO: Subscriber on port veth4 timed out
cordTester: INFO: Admin state of uni_port is up now
cordTester: INFO: Subscriber on port veth0 timed out
cordTester: INFO: Joining channel 0 for subscriber port veth4
cordTester: INFO: All subscribers have joined the channel
cordTester: INFO: Subscriber on port veth4 checking data traffic receiving from group 225.0.0.1, channel 0
cordTester: INFO: Joining channel 0 for subscriber port veth0
cordTester: INFO: All subscribers have joined the channel
cordTester: INFO: Subscriber on port veth0 checking data traffic receiving from group 225.0.0.1, channel 0
cordTester: INFO: Subscriber on port veth4 timed out
cordTester: INFO: Subscriber on port veth0 timed out
cordTester: INFO: Joining channel 0 for subscriber port veth4
cordTester: INFO: All subscribers have joined the channel
cordTester: INFO: Subscriber on port veth4 checking data traffic receiving from group 225.0.0.1, channel 0
cordTester: INFO: Joining channel 0 for subscriber port veth0
cordTester: INFO: All subscribers have joined the channel
cordTester: INFO: Subscriber on port veth0 checking data traffic receiving from group 225.0.0.1, channel 0
cordTester: INFO: Subscriber on port veth4 timed out
cordTester: INFO: Subscriber on port veth0 timed out
cordTester: INFO: Joining channel 0 for subscriber port veth4
cordTester: INFO: All subscribers have joined the channel
cordTester: INFO: Subscriber on port veth4 checking data traffic receiving from group 225.0.0.1, channel 0
cordTester: INFO: Joining channel 0 for subscriber port veth0
cordTester: INFO: All subscribers have joined the channel
cordTester: INFO: Subscriber on port veth0 checking data traffic receiving from group 225.0.0.1, channel 0
cordTester: INFO: Subscriber on port veth4 timed out
cordTester: INFO: Subscriber on port veth0 timed out
cordTester: INFO: Joining channel 0 for subscriber port veth4
cordTester: INFO: All subscribers have joined the channel
cordTester: INFO: Subscriber on port veth4 checking data traffic receiving from group 225.0.0.1, channel 0
cordTester: INFO: Joining channel 0 for subscriber port veth0
cordTester: INFO: All subscribers have joined the channel
cordTester: INFO: Subscriber on port veth0 checking data traffic receiving from group 225.0.0.1, channel 0
cordTester: INFO: Subscriber on port veth4 timed out
cordTester: INFO: Subscriber on port veth0 timed out
cordTester: INFO: Joining channel 0 for subscriber port veth4
cordTester: INFO: All subscribers have joined the channel
cordTester: INFO: Subscriber on port veth4 checking data traffic receiving from group 225.0.0.1, channel 0
cordTester: INFO: Joining channel 0 for subscriber port veth0
cordTester: INFO: All subscribers have joined the channel
cordTester: INFO: Subscriber on port veth0 checking data traffic receiving from group 225.0.0.1, channel 0
cordTester: INFO: Subscriber on port veth4 timed out
cordTester: INFO: Subscriber on port veth0 timed out
cordTester: INFO: Joining channel 0 for subscriber port veth4
cordTester: INFO: All subscribers have joined the channel
cordTester: INFO: Subscriber on port veth4 checking data traffic receiving from group 225.0.0.1, channel 0
cordTester: INFO: Joining channel 0 for subscriber port veth0
cordTester: INFO: All subscribers have joined the channel
cordTester: INFO: Subscriber on port veth0 checking data traffic receiving from group 225.0.0.1, channel 0
cordTester: INFO: Subscriber on port veth4 timed out
cordTester: INFO: Subscriber on port veth0 timed out
cordTester: INFO: Joining channel 0 for subscriber port veth4
cordTester: INFO: All subscribers have joined the channel
cordTester: INFO: Joining channel 0 for subscriber port veth0
cordTester: INFO: Subscriber on port veth4 checking data traffic receiving from group 225.0.0.1, channel 0
cordTester: INFO: All subscribers have joined the channel
cordTester: INFO: Subscriber on port veth0 checking data traffic receiving from group 225.0.0.1, channel 0
cordTester: INFO: Subscriber on port veth4 timed out
cordTester: INFO: Subscriber on port veth0 timed out
cordTester: INFO: Joining channel 0 for subscriber port veth4
cordTester: INFO: All subscribers have joined the channel
cordTester: INFO: Joining channel 0 for subscriber port veth0
cordTester: INFO: Subscriber on port veth4 checking data traffic receiving from group 225.0.0.1, channel 0
cordTester: INFO: All subscribers have joined the channel
cordTester: INFO: Subscriber on port veth0 checking data traffic receiving from group 225.0.0.1, channel 0
cordTester: INFO: Subscriber on port veth4 timed out
cordTester: INFO: Subscriber on port veth0 timed out
cordTester: INFO: Joining channel 0 for subscriber port veth4
cordTester: INFO: This service is failed and other services will not run for this subscriber
cordTester: INFO: This Subscriber is tested for multiple service eligibility
cordTester: INFO: Joining channel 0 for subscriber port veth0
cordTester: INFO: This service is failed and other services will not run for this subscriber
cordTester: INFO: This Subscriber is tested for multiple service eligibility
cordTester: ERROR: Error deleting subscriber for device of:0000000000000001 on port 128
cordTester: ERROR: Error deleting subscriber for device of:0000000000000001 on port 129
cordTester: ERROR: Error deleting subscriber for device of:0000000000000001 on port 130
cordTester: ERROR: Error deleting subscriber for device of:0000000000000001 on port 131
cordTester: ERROR: Error deleting subscriber for device of:0000000000000001 on port 132
cordTester: ERROR: Error deleting subscriber for device of:0000000000000001 on port 133
cordTester: ERROR: Error deleting subscriber for device of:0000000000000001 on port 134
cordTester: ERROR: Error deleting subscriber for device of:0000000000000001 on port 135
cordTester: ERROR: Error deleting subscriber for device of:0000000000000001 on port 136
cordTester: ERROR: Error deleting subscriber for device of:0000000000000001 on port 137
cordTester: ERROR: Error deleting subscriber for device of:0000000000000001 on port 138
cordTester: INFO: Disabling device 340b50adba61
cordTester: INFO: Deleting device 340b50adba61
cordTester: INFO: Uninstalling OLT app
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 1 test in 324.349s

FAILED (failures=1)
Test volthaTest.py:voltha_exchange.test_two_subscribers_with_voltha_for_igmp_toggling_uni_port_for_one_subscriber_and_verifying_traffic Failure
Done running tests
Removing test container cord-tester1
ubuntu@cord-rs-s8:~/cord-tester/src/test/setup$