Created by Suchitra Vemuri, last modified on Jun 20, 2018

TEST PLAN for CORD 6.0 Release

Test Plan consists of various sections/feature areas for CORD 6.0 Release and tests included here are at high level.

CORD Platform Related

  • Kubernetes Sanity POD Tests (expected resources to be looked up dependings on the profile/helm-charts deployed). This test is meant to give us a good level of certainty that the cluster has been brought up as expected, and we can then get into deeper level of testing each individual component/service on CORD. [High]

    • Validate all deployments are successfully rolled out and matches replicas to available replicas

    • Validate all pods are running

    • Validate all services are running

    • Validate pods have external connectivity*

    • Validate all pods can ping the kube-system namespace*

    • Validate all nodes are healthy (checking “ready”, “outofdisk”, “memorypressure”, “diskpressure”)

    • Validate all containers are in running state

    • Health checks

  • Helm upgrade test [Not ready for 6.0]

    • Possibly after testing a successful build -> upgrade helm charts and re-test

  • Validate Tests on  Minikube and KubeSpary environments

  • KubeSpray and Minikube (Hardware )

    • Helm install RCORD Lite profile

    • Post Installation Tests (needs to be realigned)

      • Profile based container checks

      • Validating container logs for errors

    • Control Plane Tests

    • Data Plane Tests (Manual Scenarios will be tested in the release using OLT/ONU)

  • New tests for validating the scripts to install environment

    • Install Kubespray + helm (Verify base containers)

    • Install OpenStack

    • Install base packages on ubuntu (required for OpenStack)

R-CORD Specific

  • Validate the change in the existing CORDSubscriberRoot Service Chain

    • Validate using `rcord` profile on PODs and `rcord-virtual` on Virtual Environment

    • New control plane tests to validate new service chain for CordSubscriber creations

    • Verify that by creating a subscriber, vOLTServiceInstance is created and subsequently vSGServiceInstances, Core Instances are created properly.

    • New models defined for CORDSubscriberRoot, OLT Device and PON-Port, ONU Device modelsare used to POST to the APIs

    • Do we need to still validate the old way of R-CORD profile for 6.0?

  • Validate “rcord-lite” (or no vSG) profile [High]

    • Verify the profile on Kubernetes Environment

    • New models defined for CORDSubscriberRoot, OLT Device, PON-Port and ONU Device models are used to POST to the APIs

    • Write new control plane tests to verify the new service chain

      • creating a subscriber, a vOLTServiceInstance is created and a vSG-HWServiceInstance is created

      • No vSG instances/containers are created

    • Data Plane Verification

      • Testing will be performed once the implementation has been completely defined

      • Testing is also dependant on the availability of OLT/ONU devices

      • Manual data plane verifications will be performed in this release

      • Investigate how to validate data plane connectivity with `rcord-lite` profile in automated tests

  • R-CORD Lite Data Plane with ONU/OLT

    • Currently QA does not have any hardware available to test/validate using ONU/OLT devices

    • What are the most important tests required for 6.0 ? [Ping tests, multiple subscriber traffic tests]

  • M-CORD

    • Openstack changes and instance validations

    • Validate installation of M-CORD using helm-charts

    • Perform a simple end-end test.

    • Platform that will be used is a single node platform only (one node- CiaB)


Jenkins Environment

  • Migrate old jenkins jobs to new JJB. Migration will be done based on the following prioritized list

    • `xos-api-sanity-pipeline` will be migrated first along with changes that includes usage of new light weight profiles with minikube environment.

      • The new ‘xos-api-sanity-pipeline’ job have been ported over to ‘verify_<project>_api_test’ jobs in the new jenkins. Still using the old ‘make <profile>-local’ configs (docker-compose).

      • WIP on the environment to be deployed and run the tests. (minikube)

      • [JOB Migrated by Zack and tests will be provided by QA]

    • POD related jobs

      • Build procedure updates (test on Calix POD for master branch with Kubernetes deployment)

      • Port changes on other PODs

      • Migrate Soak PODs to 6.0 towards the end

R-CORD Performance/Scale/Stability Testing

  • Performance/Scale/Stability Testing will be performed using CORD 5.0 stable version

  • Scale vCPEs in a single vSG

    • Required test scripts will be achieved for CORD 5.0 release

    • Tests will be performed Flex POD

  • Scale vSGs

    • Verify that the vSGs created are ACTIVE

    • Verify that the respective containers are created within

    • Verify the stability of CORD (memory utilization, restart checks and any error checks)

  • Stability Tests

    • POD runs for almost 3 month time

    • Following checks are performed to check the health of the POD

      • Errors in Containers, ONOS Logs

      • Restart checks on any containers

      • Memory utilization on the containers

      • Out of Memory issues


    • Traffic Emulation tests using Spirent Test Tools

      • Generation of continuous traffic scenarios using multiple vSGs

      • Scaling of vSGs and vCPEs followed by generation of more taffic

  • TestBed Setup is shown below

    • Flex POD-1

      • Compute (Flex 1U Victoria)

        • 2x Intel E5-2630v3 8c 2.4Ghz, 192GB RAM

        • 2x1G, 2x40G dual port Mellanox NIC

        • 1x Intel SSD 480GB

      • Switch (Accton AS6712-54X 48x40G, 6x40G)


  


M-CORD

  • Validate EPC in M-CORD 4.1 release if it is compliant to 3GPP standards or not

  • Functional Tester Tool from Polaris is used to verify

  • MME FT - 786 tests

    • S1-MME Management

    • UE Attach

    • UE Context Management

    • UE Detach

    • Service Request

    • Tracking Area Update

    • Dedicated Bearer Handling

    • S1 based Handover

    • X2 based Handover

    • Trace

    • Location Services

    • And other functionality

  • PGW FT - 387 tests

  • SGW FT - 344 tests

  • SPGW FT - 173 tests


Diagram below how the Functional Tester tool connects to M-CORD and perform tests.

The System Under Test can be one of the EPC Nodes like MME, SGW or PGW. SPGW is a combined SGW + PGW.


In M-CORD, few nodes from the EPC are available such as (NG4T provides MME, Intel provides SPGW and Sprint(?) provides MME & HSS in 5.0).

R-CORD Performance/Scale/Stability Testing

Additional Information about test setup and actual tests


 

Scenario 1 -

As RCORD Solution is deployed on Open-stack – Spirent test center virtual(STCv)VM’s are created and deployed on the same environment.

Detailed Packet flow

  • The test client (STCv) emits double-tagged VLAN packets (S-Tag and C-Tag) onto the “vSG” interface on the compute node.

  • Depending on the functionality configured on the vSG the test client (STCv) on the receiver end receives the traffic.


 

Wireshark Capture

Scenario 2

Detailed Packet flow

  • The test client (STCv) emits double-tagged VLAN packets (S-Tag and C-Tag) onto the “fabric”.

    • Fabric here in the above picture is a switch running ONOS.

    • Flows are configured on this switch in ONOS such that when it receives double-tagged traffic, the traffic is routed to corresponding vSG’s.

  • All the packets are always forwarded on the “fabric” interface of the compute nodes, where they are forwarded to the corresponding vSG.

  • Depending upon the functionality of the vSG the test client (STCv) on the receiver end receives the traffic.

  • Analytical report is generated by the Spirent tool. This report has a detailed insight that provides us various details like frame size, latency, throughput, dropped frames, etc.

  • These reports help us analyze the life of the packet in the CORD environment.



Spirent Frame Details -