I don't see any AttWorkflowServiceInstances

If you have added an OLT in the system but you see no AttWorkflowServiceInstances you can debug it following these steps.

Check the status in VOLTHA

Connect to the VOLTHA CLI and check the devices:

(voltha) devices
Devices:
+------------------+-------------------+------+------------------+------------------+-------------+-------------+----------------+----------------+------------------+------------------------+-------------------------+--------------------------+----------------------+------------------------------+
|               id |              type | root |        parent_id |    serial_number | admin_state | oper_status | connect_status | parent_port_no |    host_and_port |                 reason | proxy_address.device_id | proxy_address.channel_id | proxy_address.onu_id | proxy_address.onu_session_id |
+------------------+-------------------+------+------------------+------------------+-------------+-------------+----------------+----------------+------------------+------------------------+-------------------------+--------------------------+----------------------+------------------------------+
| 00012d28315ddb79 |           openolt | True | 000100000a5a007a | 10.90.0.122:9191 |     ENABLED |      ACTIVE |      REACHABLE |                | 10.90.0.122:9191 |                        |                         |                          |                      |                              |
| 0001d18bedd13517 | brcm_openomci_onu | True | 00012d28315ddb79 |     ALPHe3d1cfe3 |     ENABLED |      ACTIVE |      REACHABLE |      536870912 |                  | initial-mib-downloaded |        00012d28315ddb79 |                          |                    1 |                            1 |
| 00011c399faa957d | brcm_openomci_onu | True | 00012d28315ddb79 |     ALPHe3d1cf9d |     ENABLED |  DISCOVERED |      REACHABLE |      536870912 |                  |          starting-omci |        00012d28315ddb79 |                          |                    2 |                            2 |
+------------------+-------------------+------+------------------+------------------+-------------+-------------+----------------+----------------+------------------+------------------------+-------------------------+--------------------------+----------------------+------------------------------+

AttWorkflowServiceInstances are created once the ONUs reach are in oper_status=ACTIVE, so at this given time we should expect to see only one AttWorkflowServiceInstances.

Now check the logical device representing that OLT and verify that it shows active UNI ports:

(voltha) logical_devices
Logical devices:
+------------------+------------------+------------------+------------------+---------------------------+--------------------------+
|               id |      datapath_id |   root_device_id |  desc.serial_num | switch_features.n_buffers | switch_features.n_tables |
+------------------+------------------+------------------+------------------+---------------------------+--------------------------+
| 000100000a5a007a | 000000000a5a007a | 00012d28315ddb79 | 10.90.0.122:9191 |                       256 |                        2 |
+------------------+------------------+------------------+------------------+---------------------------+--------------------------+
(voltha) logical_device 000100000a5a007a
(logical device 000100000a5a007a) ports
Logical device ports:
+-----------+------------------+----------------+-----------+------------------+------------------------------+---------------+----------------+---------------+---------------------+------------------------+
|        id |        device_id | device_port_no | root_port | ofp_port.port_no |             ofp_port.hw_addr | ofp_port.name | ofp_port.state | ofp_port.curr | ofp_port.curr_speed | ofp_port_stats.port_no |
+-----------+------------------+----------------+-----------+------------------+------------------------------+---------------+----------------+---------------+---------------------+------------------------+
| nni-65536 | 00012d28315ddb79 |          65536 |      True |            65536 |     [0L, 0L, 0L, 1L, 0L, 0L] |     nni-65536 |              4 |          4128 |                  32 |                  65536 |
|    uni-32 | 00011c399faa957d |             32 |           |               32 |    [8L, 0L, 0L, 0L, 0L, 32L] |  ALPHe3d1cf9d |              4 |          4160 |                  64 |                        |
|    uni-16 | 0001d18bedd13517 |             16 |           |               16 |    [8L, 0L, 0L, 0L, 0L, 16L] |  ALPHe3d1cfe3 |              4 |          4160 |                  64 |                        |
+-----------+------------------+----------------+-----------+------------------+------------------------------+---------------+----------------+---------------+---------------------+------------------------+

If no ports are shown an error has occurred in VOLTHA, if everything looks correct you can proceed to the next step.

Check the status in ONOS

Connect to the ONOS CLI and check devices:


onos> devices id=of:0000000000000002, available=true, local-status=connected 5h33m ago, role=MASTER, type=SWITCH, mfr=Accton Corp., hw=x86-64-accton-as6712-32x-r0, sw=ofdpa 3.0.5.5+accton1.7-1, serial=671232X1538038, chassis=2, driver=ofdpa3, channelId=10.90.0.120:46211, locType=none, managementAddress=10.90.0.120, name=AGG_SWITCH, protocol=OF_13 id=of:000000000a5a007a, available=true, local-status=connected 4h19m ago, role=MASTER, type=SWITCH, mfr=VOLTHA Project, hw=, sw=, serial=10.90.0.122:9191, chassis=a5a007a, driver=voltha, channelId=10.233.102.179:59154, locType=none, managementAddress=10.233.102.179, name=SHAD OLT, protocol=OF_13

You should see the aggregation switch and the logical device representing the PON. If you can't see the logical device (mfr=VOLTHA Project) then an error occurred in the communication between the VOLTHA ofagent component and ONOS.

Check if ONOS can see the active ports on the logical device:

onos> ports -e of:000000000a5a007a
id=of:000000000a5a007a, available=true, local-status=connected 4h21m ago, role=MASTER, type=SWITCH, mfr=VOLTHA Project, hw=, sw=, serial=10.90.0.122:9191, chassis=a5a007a, driver=voltha, channelId=10.233.102.179:59154, locType=none, managementAddress=10.233.102.179, name=SHAD OLT, protocol=OF_13
  port=16, state=enabled, type=fiber, speed=0 , adminState=enabled, portMac=08:00:00:00:00:10, portName=ALPHe3d1cfe3
  port=32, state=enabled, type=fiber, speed=0 , adminState=enabled, portMac=08:00:00:00:00:20, portName=ALPHe3d1cf9d
  port=65536, state=enabled, type=fiber, speed=0 , adminState=enabled, portMac=00:00:00:01:00:00, portName=nni-65536

If you don't see the ports there's again some issue in the communication between the VOLTHA ofagent component and ONOS.

If everything looks correct you can move to the next step.

Check events in Kafka

Use Kibana

If you have installed the Logging component of the platform you can then look in Kibana for these events.

Connect to the dashboard an search for events in the onu.events topic:

Kibana ONU events

Check in Kafka

You can install kafkacat on the POD using this command:

helm install -n kfc cord/kafkacat

Then you'll need to access the container and list the messages on the onu.events topic:

kubectl exec -it kafkacat-7f45f65485-2lgp8 bash
root@kafkacat-7f45f65485-2lgp8:/# kafkacat -b cord-platform-kafka -t onu.events

Note: Replace cord-platform-kafka with cord-kafka in the command above if kafka was installed as a standalone component.

If everything is correct you should see:

{"timestamp":"2019-03-29T16:01:28.763Z","status":"activated","serialNumber":"PSMO12345678","portNumber":"128","deviceId":"of:0000aabbccddeeff"}

If you can see events in Kafka you can proceed with the next step.

Check the AttWorkflowDriver container

This is the last step of the process, if you have events in kafka, you should check the logs of the att-workflow-driver container.

You can do that using this command:

kubectl logs -f att-workflow-att-workflow-driver-7c8bc95894-xgxts