Gitiles
Code Review Sign In
gerrit.opencord.org / sdcore-docs / d15e8ac0218192aeacc8c3c3d662ff7165d9691c / . / testing / acceptance_specification.rst
blob: e1c8482729af462261730e815422d44074a26012 [file] [log] [blame]
ajayf66b38e2021-12-09 23:44:54 -0800[diff] [blame]1..
2 SPDX-FileCopyrightText: © 2020 Open Networking Foundation <support@opennetworking.org>
3 SPDX-License-Identifier: Apache-2.0
4
5Acceptance Specification
6========================
7
8Objectives
9----------
10
11The purpose of this document is to create an end-user test object list (TOL)
12for Aether Connected Edge (ACE).
13
14This document will focus on the connectivity services end-user testing. In the
15future, this document will extend to other services offered through ACE.
16
17The Automated continuous testing framework for the platform is out of the scope of this document.
18
19Integration Test (eNB-LTE Core)
20-------------------------------
21
22Before we start to test End-to-End connectivity, we have to check the
23connection (called S1-MME/S1-C interface) between eNB in an edge and MME in a
24public cloud.
25
26In order to verify this connectivity, the following test cases should be passed.
27
28Note that all the following test/verification cases have some assumptions:
29
301. eNB is connected to the Fabric switch;
312. eNB has correct network configurations;
323. eNB has correct ID configurations provided by the ONF PMFE team.
33
34IT-TOL01 Fabric Test 1: the connectivity test within the edge
35^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
36
37In order to test the fabric test, please see the following steps:
38
39
40+----------------------------------------------+------------------------------------+
41|Steps |Expected Outcome |
42+----------------------------------------------+------------------------------------+
43|1. Access to the eNB* (through SSH or Web GUI)|Able to get ICMP reply(PING reply) |
44| |from the GCP node. |
45| | |
46|2. Ping to "10.168.0.6" eNB S1-MME/S1-C |( )YES ( )NO |
47| interface IP address* | |
48| |Comments: |
49+----------------------------------------------+------------------------------------+
50
51.. note::
52 it depends on the eNB device. Some eNBs have a single network interface for the management network, S1-U, and S1-C,
53 while other eNBs have separate interfaces. The former eNB type has a single IP address,
54 and the later eNB type has multiple IP addresses for each interface.
55
56IT-TOL02 Fabric Test 2: the connectivity test between the eNB and the public cloud
57^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
58
59In order to test the fabric test, please see the following steps:
60
61
62+----------------------------------------------+------------------------------------+
63|Steps |Expected Outcome |
64+----------------------------------------------+------------------------------------+
65|1. Access to the eNB* (through SSH or Web GUI)|Able to get ICMP reply(PING reply) |
66| |from the GCP node. |
67| | |
68|2. Ping to "10.168.0.6" |( )YES ( )NO |
69| | |
70| |Comments: |
71+----------------------------------------------+------------------------------------+
72
73
74
75.. note::
76 it also depends on the eNB device. Some eNBs allow us to SSH into their device, other eNBs provide the PING tool through Web GUI.
77 Of course, some eNBs do not support both. In that case, it is okay to skip this test case.
78
79
80IT-TOL03 SCTP Connection Verification
81^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
82
83In order to verify the SCTP connection between MME and eNB, please see the following steps:
84
85
86+----------------------------------------------+------------------------------------+
87|Steps |Expected Outcome |
88+----------------------------------------------+------------------------------------+
89|1. SSH to the gateway device* for the eNB |Able to see Heart Beat |
90| S1-MME/S1-C traffic |messages |
91| | |
92|2. Capture the traffic with Wireshark |( )YES ()NO |
93| or the command: `sudo tcpdump -i any sctp` | |
94| |Comments: |
95+----------------------------------------------+------------------------------------+
96
97Capture the traffic with Wireshark or the command: `sudo tcpdump -i any sctp`
98
99.. note::
100 the eNB should have the gateway IP address for the S1-MME/S1-C traffic.
101 You can SSH there with the gateway IP address in the eNB and capture the traffic.
102 Normally, the gateway device can be one of those devices: (i) VPN router; (ii) Firewall device in between VPN router and the edge;
103 (iii) one of edge servers.
104
105IT-TOL04 PLMN Verification
106^^^^^^^^^^^^^^^^^^^^^^^^^^
107
108In order to verify the TAC number, please see the following steps:
109
110
111Able to see the correct MCC and MNC values*
112
113+-------------------------------------------------+------------------------------------+
114|Steps |Expected Outcome |
115+-------------------------------------------------+------------------------------------+
116|1. SSH to the gateway device for the eNB |Able to see the correct MCC and MNC |
117| S1-MME/S1-C traffic |values |
118| | |
119|2. Start to capture the traffic with Wireshark |( )YES ()NO |
120| or the command: `sudo tcpdump -i any sctp | |
121| -w FileName.pcap` |Comments: |
122|3. Reboot eNB | |
123| | |
124|4. Wait until FileName.pcap has `S1SetupRequest` | |
125| S1SetupResponse messages | |
126| | |
127|5. Stop the packet capturing and open | |
128| the FileName.pcap | |
129| | |
130|6. Find out the S1SetupRequest message and | |
131| open the detailed packet information | |
132| | |
133|7. Go to "Item 2: id-SupportedTAs" section | |
134| and check "MACC and "MNC" values | |
135+-------------------------------------------------+------------------------------------+
136
137Example (the MCC is 315 and MNC is 010)
138
139.. image:: images/it-tol04.png
140 :width: 840
141 :height: 840
142 :alt: Example (the MCC is 315 and MNC is 010)
143
144IT-TOL05 TAC Number Verification
145^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
146
147
148+-------------------------------------------------+------------------------------------+
149|Steps |Expected Outcome |
150+-------------------------------------------------+------------------------------------+
151|1. SSH to the gateway device for the eNB |Able to see the correct TAC number |
152| S1-MME/S1-C traffic | |
153| | |
154|2. Start to capture the traffic with Wireshark |( )YES ()NO |
155| or the command: `sudo tcpdump -i any sctp | |
156| -w FileName.pcap` |Comments: |
157|3. Reboot eNB | |
158| | |
159|4. Wait until FileName.pcap has `S1SetupRequest` | |
160| S1SetupResponse messages | |
161| | |
162|5. Stop the packet capturing and open | |
163| the FileName.pcap | |
164| | |
165|6. Find out the S1SetupRequest message and | |
166| open the detailed packet information | |
167| | |
168|7. Go to "Item 0: id-SupportedTAs" section | |
169| and check TAC " | |
170+-------------------------------------------------+------------------------------------+
171
172.. note::
173 if you already captured packets in IT-TOL03, you can skip steps from 1 to 5.
174 Just you can check the expected outcome with the file you captured at IT-TOL03.
175
176Example (the TAC number is 19)
177
178.. image:: images/it-tol05.png
179 :width: 840
180 :height: 840
181 :alt: Example (the TAC number is 19)
182
183IT-TOL06 eNB Verification
184^^^^^^^^^^^^^^^^^^^^^^^^^
185
186In order to test the eNB, please see the following steps:
187
188+-------------------------------------------------+------------------------------------+
189|Steps |Expected Outcome |
190+-------------------------------------------------+------------------------------------+
191|1. SSH to the gateway device for the eNB |Able to see the correct eNBID |
192| S1-MME/S1-C traffic | |
193| | |
194|2. Start to capture the traffic with Wireshark |( )YES ()NO |
195| or the command: `sudo tcpdump -i any sctp | |
196| -w FileName.pcap` |Comments: |
197|3. Reboot eNB | |
198| | |
199|4. Wait until FileName.pcap has `S1SetupRequest` | |
200| S1SetupResponse messages | |
201| | |
202|5. Stop the packet capturing and open | |
203| the FileName.pcap | |
204| | |
205|6. Find out the S1SetupRequest message and | |
206| open the detailed packet information | |
207| | |
208|7. Go to "Item 0: id-Global-eNB-ID" section | |
209| and check "eNB-ID: macroENB-ID" | |
210+-------------------------------------------------+------------------------------------+
211
212.. note::
213 if you already captured packets in IT-TOL03, you can skip steps number 1 to 5.
214 Just you can check the expected outcome with the file you captured at IT-TOL03.
215
216Example (the eNB ID is 19)
217
218.. image:: images/it-tol06.png
219 :width: 840
220 :height: 840
221 :alt: Example (the eNB ID is 19)
222
223Connectivity Services
224---------------------
225
226Aether provides only data connectivity for end-user devices and systems.
227So the voice service over LTE is not available. However, users can use
228any OTT services over the Aether network for voice connectivity.
229
230The test specifications are only covering the data connectivity focused tests.
231
232
233CS-TOL01 Device Attach/Connect
234^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
235
236To test device can attach to Aether network
237
238
239+----------------------------------------------+------------------------------------+
240|Steps |Expected Outcome |
241+----------------------------------------------+------------------------------------+
242|1. Turn off the mobile device |Able to attach the device and |
243| |connect to the internet/Aether |
244|2. Turn on the mobile device |Network |
245| | |
246|3. Check whether the device is showing |( )YES ( )NO |
247| connected on the status, depending on | |
248| the device it will show "Aether" or | |
249| "MCCMNC" format. | |
250|4. Browse http://www.google.com/? |( )YES ( )NO |
251| From the device web browser | |
252| |Comments: |
253+----------------------------------------------+------------------------------------+
254
255CS-TOL02 Device Detach/Disconnect
256^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
257
258To test device can detach/disconnected by user initiation
259
260
261+----------------------------------------------+------------------------------------+
262|Steps |Expected Outcome |
263+----------------------------------------------+------------------------------------+
264|1. Make sure the device is connected to Aether|Able to detach the device and |
265| |disconnect from the internet/Aether |
266|2. Deselect the network (or forget the network|Network |
267| , depending on device configuration) | |
268|3. Try to browse http://www.google.com/? |( )YES ( )NO |
269| From your web browser | |
270| |Comments: |
271+----------------------------------------------+------------------------------------+
272
273
274CS-TOL03 Bandwidth Test - Internet
275^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
276
277To test bandwidth available to a mobile device over Aether network.
278
279Please note the following, the bandwidth test depends on the eNB hardware,
280your local breakout bandwidth, and the overall radio environment.
281If you face an unexpected result, please explain it in the comment section in the outcome column.
282
283
284+----------------------------------------------+------------------------------------+
285|Steps |Expected Outcome |
286+----------------------------------------------+------------------------------------+
287|1. Open Speedtest app from your mobile device |Expected Bandwidth/Throughput |
288| |observed |
289| | |
290|2. Run Speedtest 3 times, take the average as |( )YES ( )NO |
291| the final result | |
292| |Comments: |
293+----------------------------------------------+------------------------------------+
294
295
296CS-TOL04 Bandwidth Test - Edge Application
297^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
298
299To test bandwidth available to a mobile device over Aether network.
300
301Please note the following, the bandwidth test depends on the eNB hardware,
302your local breakout bandwidth, and the overall radio environment. If you face an unexpected result,
303please explain it in the comment section in the outcome column.
304
305
306+----------------------------------------------+------------------------------------+
307|Steps |Expected Outcome |
308+----------------------------------------------+------------------------------------+
309|1. Initiate FTP Download from a local server |Expected Bandwidth/Throughput |
310| (same location) connected to the enterprise|observed |
311| network (through local breakout) | |
312| | |
313|2. Download 3 times, take the average as the |( )YES ( )NO |
314| final result | |
315| |Comments: |
316+----------------------------------------------+------------------------------------+
317
318
319
320Monitoring Services
321-------------------
322
323ACE uses the Grafana dashboard for monitoring services.
324Each ACE will be provided with Read-Only Access to our centralized monitoring platform.
325
326
327Application Services
328--------------------
329
330Aether uses Rancher to onboard applications to ACE.
331Each ACE host will be provided with access to rancher to onboard applications on their ACE cluster.
332