Framework

Sources: TVR v2.3 §2.6, §2 · SAI v2.2 §2.2, §4.1.1.7

Phase 2 Testing Scope

5G-EMERGE aims to evaluate the comparative effectiveness of different video distribution methods, with a focus on their ability to support reliable, high-quality, and sustainable content delivery.

Use Case Categories

The 5G-EMERGE ecosystem targets a range of market segments served by a common infrastructure. Four use case categories are in scope, each with its own service requirements and deployment scenarios:

Service Scenarios

Four end-to-end service scenarios are defined as representative of the media delivery challenges in scope. They can be applied across all four use case categories:

Further detail on use case categories and service scenarios is provided in the SAI document, which also defines the system architecture.

Phase 1 vs Phase 2

Strand 2 Cross-Strand Testing

Strand 2 has defined a comprehensive set of cross-strand end-to-end tests designed to verify that the 5G-EMERGE system achieves the KVIs. The aims are:

• A common approach to end-to-end testing across the consortium
• The expression of Key Value Indicators (KVIs) via measurable Key Performance Indicators (KPIs)
• Reporting standards for consistent evaluation
• Scenarios pertinent to assessing the media delivery platform as a whole

Testing specific to Strands 3–6 (component-level and strand-specific functional validation) is documented separately and is a pre-condition for end-to-end service-level testing.

Testing is a cornerstone of systems engineering. It ensures quality, reduces risks, and verifies that solutions perform reliably in real-world conditions. Within 5G-EMERGE, testing serves to:

• Identify Defects — uncover bugs, glitches, and performance issues before they reach end-users.
• Ensure Quality — confirm the system meets requirements, performs consistently, and provides a positive user experience.
• Reduce Risks — minimise risks relating to quality, security, and performance.
• Lower Costs — detect and resolve issues early, avoiding expensive fixes after deployment.

All components must also comply with relevant standards for the technologies and regions in which they operate. These may include physical, accessibility, and intellectual property standards, among others.

Test Structure

Testing is organised into a clear hierarchy for comprehension and manageability.

This hierarchical structure ensures traceability from high-level requirements through to individual test outcomes, supporting both systematic coverage and reporting.

Test Execution

Execution of tests can be either conducted manually (Manual Testing) or automated through tools (Automated Testing).

Test Harnesses support component testing. For end-to-end validation in 5G-EMERGE, these are referred to as Test Beds. Where real components are unavailable, simulated components (mocks, stubs) may be substituted.

Execution scope is tailored according to risk and impact:

Automated testing conducted after each change is known as Continuous Testing.

It is recommended that, where possible, testing be automated and continuous, with full testing conducted periodically and change-based testing on each change.

Additional deliverables related to the test beds (e.g. design of demos) are covered by the deliverables in the associated work packages and are out of scope for this document.

Test Scale

In Phase 2, the test beds are designed to support the necessary Test Levels but are not scaled to support load tests and soak tests at production-expected volumes. A full set of large-scale test scenarios has been included in Appendix 1 of the TVR for completeness.

Where full-scale testing is not feasible, test scenarios may be conducted at scaled-down volumes to assess behaviour and determine limiting factors. Such tests should be designed so that they do not simply re-identify known architectural constraints (e.g. limitations arising from available spectrum, cluster sizes, or throughput).

The four use case categories defined in the SAI give the framing for what scale means in each deployment context:

Test Types

Different types of tests focus on specific aspects of the system to isolate and evaluate particular characteristics. Test types fall into two main categories.

Functional Testing

Checks that the system does exactly what it is specified to do. This was the primary focus in Phase 1. In Phase 2, functional testing of strand-specific components is a pre-condition for end-to-end service-level testing.

Non-Functional Testing

Verifies the readiness of a system according to non-functional parameters — performance, security, accessibility, scale — which functional testing alone cannot address.

5G-EMERGE Phase 2 is predominantly focused on non-functional testing, and in particular, performance testing.

Test Reporting

To ensure results are comparable, reproducible, and interpretable across the consortium, each test is conducted under a standardised reporting framework defining inputs, outputs, and expectations.

Inputs

All tests must begin with a clear definition of required inputs:

• System-Level Metrics — network logs, playback sessions, server utilisation, concurrency levels, energy monitoring, crash/error reports.
• Environmental Parameters — geographic distribution of test nodes, network variability (latency, packet loss, bandwidth), simulated user loads.
• Method Identifiers — explicit classification of the distribution method under test (CDN, Edge, etc.).

Inputs must be captured in structured, timestamped logs with a uniform schema so that results can be aggregated across test campaigns.

Outputs

Each test must produce outputs that are quantitative, comparable, and traceable:

• Primary Metrics aligned to the KPI (e.g. watch time, session duration, completion rate, video resolution/fps, latency, kWh consumed).
• Derived Indicators — correlations (e.g. buffering vs. abandonment) or efficiency ratios (e.g. energy per GB delivered).
• Comparative Data — presented per distribution method, region, and test condition.

All outputs should be stored in a centralised results repository in standardised units (ms, %, kWh, MB, GB).

Expectations

Each use case is guided by explicit expectations:

• Benchmark Performance — define what resilience, scalability, or quality should look like under given conditions (e.g. near-linear energy scaling, <2 s startup delay, >95% uptime under stress).
• Identify Inefficiencies — highlight scenarios where energy, capacity, or network performance deviate from expected efficiency.
• Support Comparability — ensure differences in performance can be attributed to the distribution method rather than test inconsistency.

Expectations must be written as testable hypotheses (e.g. "Edge delivery will reduce latency and energy per viewing minute compared to centralised CDN") to maintain a scientific research framework.

Report Requirements

Test reports must:

• Document Context — record test date, location, duration, planned/actual network conditions, and distribution method.
• Present Structured Results — use consistent table and chart formats to display inputs, outputs, and derived metrics.
• Enable Cross-Test Analysis — support aggregation across methods and KPIs for holistic evaluation.
• Flag Deviations — clearly identify where results diverge from expectations, positively or negatively.

All reports should be archived and versioned, allowing stakeholders to reproduce or validate findings.

Test Metrics — KVIs and KPIs

Phase 2 employs a structured set of Key Value Indicators (KVIs) aligned to six overarching evaluation dimensions. Each KVI is linked to specific measurable Key Performance Indicators (KPIs), enabling comparison of distribution approaches across the full range of use case categories.

KVI Dimensions:

Most KPIs are recorded as datasets. The following statistical measures are applied where appropriate: mean, standard deviation, lower/upper quartile, 95th percentile (p95), 99.9th percentile (p99.9).

Engagement and Usage KPIs

Engagement KPIs function as proxies for usability and adoption potential under different delivery frameworks. They help identify not only which distribution methods sustain attention, but why users may disengage.

Technical Performance KPIs

Technical performance KPIs are central to comparing the efficiency, reliability, and scalability of different delivery methods. Variations across methods directly reflect differences in architectural efficiency.

Energy and Resource Efficiency KPIs

Different distribution approaches carry distinct energy and resource footprints. These KPIs ensure evaluation moves beyond technical feasibility to include environmental and systemic impact.

Note: Some proposed energy tests assume large-scale deployment beyond the scope of Phase 2. Tests feasible within Phase 2 test beds are included in the overall test plan; others are recorded in TVR Appendix 1 for future reference.

Compute, AI and Security KPIs

Computing KPIs

These KPIs assess Nuvla's capability to deploy and operate ETSI-MEC / 3GPP edge applications on distributed edge infrastructure.

AI-Related KPIs

These KPIs evaluate the effectiveness, accuracy, adaptability, and performance efficiency of the AI-based WAAP (Web Application and API Protection) solution.

Security KPIs

These KPIs address confidentiality, integrity, and availability of edge infrastructure and communication channels, measured through controlled scenarios including attack simulations and misconfiguration attempts.

Far-Edge Security:

Far-Edge Firewall:

Satellite Link Security:

For the full list of test cases mapped to these KPIs, see Global Test Register. For the E2E scenario catalogue, see E2E Scenarios Reference.