Metrics
Overview
Magma gateways and orc8r generate a lot of metrics which provides a great deal of visibility into gateways, base stations, subscribers, reliability, throughput etc. These metrics are regularly pushed into prometheus, which along with grafana enables us to store and query for these metrics. All the metrics are stored in prometheus for a default of 30 days. For unlimited retention and a more scaled metrics pipeline, we also support deploying magma with Thanos.
Metrics Explorer
Metrics explorer provides an easy way to learn and explore the metrics available in our system. Metrics explorer can be viewed through NMS. We can search and filter the metrics by the name or description. Additionally if we click on the detailed view on the metric, we enable exploring the current trends on the metric via grafana explorer.
Grafana
Grafana provides a powerful, configurable, and user-friendly dashboarding solution. Any users within an organization can create and edit custom timeseries dashboards that will be visible to all other users in their organization. An important detail is that Grafana access is limited only to users in an organization with the "Super-User" title (you will select this when provisioning users in an organization). This is a technical workaround to ensure that users with additional network visibility restrictions within an organization can't see information from networks that they are restricted from as Grafana will allow all users to query across any network that the organization owns.
When you click on this link we have to do some book-keeping on the backend, so the initial load may take a few seconds.
You’ll see built in dashboards available to you.
These dashboards contain dropdown selectors to choose which network(s) and gateway(s) you want to look at.
In Grafana you can look at any collection of networks or gateways your organization has access to at once. Simply select or
deselect the networks/gateways that you want to see and the graphs will be updated. In the top right corner, there is an option to choose the time range that the graphs display. The default is 6 hours.
Custom Dashboards
With Grafana, you can create your own custom dashboards and populate them with any graphs and queries you want. These custom dashboards will be visible to all other users in the organization that you belong to in the NMS. The simple way is to just click on the “+” icon on the left sidebar, then create a new dashboard. There is ample documentation about grafana dashboards online if you need help creating your dashboard.
- Grafana documentation on creating dashboards: Grafana Dashboards
- Prometheus documentation on writing queries: Prometheus Querying
If you want to replicate the networkID or gatewayID variables that you find in the preconfigured dashboards, we provide a “template” dashboard to make that easy. Simply open the Template dashboard, and click on the gear icon near the
top right. From there, click “Save As” and enter the name you want. Your new dashboard will now have the gatewayID and networkID variables. An example of how to use these variables in your queries:
Some technical details: You need to use =~ when matching label names with these variables in order to see more than one network or gateway at a time. This is because the =~ operator tells Prometheus to match the value as a regex.
Enabling Access
The feature flag is enabled by default for all new organizations created in the NMS. If you want to turn this feature off or on, you can do so from the master organization. Login to the master organization, navigate to the feature flag page using
the left sidebar, then edit the feature flag named "Include tab for Grafana in the Metrics page". Support can be turned on and off for individual organizations.
List of metrics which are currently available
Up-to-date view would be available through metrics explorer
| Metric Identifier | Description | Category | Service |
|---|---|---|---|
| s1_connection | eNodeB S1 connection status | eNodeB | MME |
| user_plane_bytes_ul | User plane uplink bytes | eNodeB | |
| user_plane_bytes_dl | User plane downlink bytes | eNodeB | |
| enodeb_mgmt_connected | eNodeB management plane connected | eNodeB | enodebd |
| enodeb_mgmt_configured | eNodeB is in configured state | eNodeB | enodebd |
| enodeb_rf_tx_enabled | eNodeB RF transmitter enabled | eNodeB | enodebd |
| enodeb_rf_tx_desired | eNodeB RF transmitter desired state | eNodeB | enodebd |
| enodeb_gps_connected | eNodeB GPS synchronized | eNodeB | enodebd |
| enodeb_ptp_connected | eNodeB PTP/1588 synchronized | eNodeB | enodebd |
| enodeb_opstate_enabled | eNodeB operationally enabled | eNodeB | enodebd |
| enodeb_reboot_timer_active | Is timer for eNodeB reboot active | eNodeB | enodebd |
| enodeb_reboots | eNodeb reboots counter | eNodeB | enodebd |
| rcc_estab_attempts | RRC establishment attempts | eNodeB | enodebd |
| rrc_estab_successes | RRC establishment successes | eNodeB | enodebd |
| rrc_reestab_attempts | RRC re-establishment attempts | eNodeB | enodebd |
| rrc_reestab_attempts_reconf_fail | RRC re-establishment attempts due to reconfiguration failure | eNodeB | enodebd |
| rrc_reestab_attempts_ho_fail | RRC re-establishment attempts due to handover failure | eNodeB | enodebd |
| rrc_reestab_attempts_other | RRC re-establishment attempts due to other cause | eNodeB | enodebd |
| rrc_reestab_successes | RRC re-establishment successes | eNodeB | enodebd |
| erab_estab_attempts | ERAB establishment attempts | eNodeB | enodebd |
| erab_estab_failures | ERAB establishment failures | eNodeB | enodebd |
| erab_estab_successes | ERAB establishment successes | eNodeB | enodebd |
| erab_release_requests | ERAB release requests | eNodeB | enodebd |
| erab_release_requests_user_inactivity | ERAB release requests due to user inactivity | eNodeB | enodebd |
| erab_release_requests_normal | ERAB release requests due to normal cause | eNodeB | enodebd |
| erab_release_requests_radio_resources_not_available | ERAB release requests due to radio resources not available | eNodeB | enodebd |
| erab_release_requests_reduce_load | ERAB release requests due to reducing load in serving cell | eNodeB | enodebd |
| erab_release_requests_fail_in_radio_proc | ERAB release requests due to failure in the radio interface procedure | eNodeB | enodebd |
| erab_release_requests_eutran_reas | ERAB release requests due to EUTRAN generated reasons | eNodeB | enodebd |
| erab_release_requests_radio_conn_lost | ERAB release requests due to radio connection with UE lost | eNodeB | enodebd |
| erab_release_requests_oam_intervention | ERAB release requests due to OAM intervetion | eNodeB | enodebd |
| pdcp_user_plane_bytes_ul | User plane uplink bytes at PDCP | eNodeB | enodebd |
| pdcp_user_plane_bytes_dl | User plane downlink bytes at PDCP | eNodeB | enodebd |
| ip_address_allocated | Total IP addresses allocated | AGW | mobilityd |
| ip_address_released | Total IP addresses released | AGW | mobilityd |
| s6a_auth_success | Total successful S6a auth requests | AGW | subscriberdb |
| s6a_auth_failure | Total failed S6a auth requests | AGW | subscriberdb |
| s6a_location_update | Total S6a lcoation update requests | AGW | subscriberdb |
| diameter_capabilities_exchange | Total Diameter capabilities exchange requests | AGW | subscriberdb |
| diameter_watchdog | Total Diameter watchdog requests | AGW | subscriberdb |
| diameter_disconnect | Total Diameter disconnect requests | AGW | subscriberdb |
| dp_send_msg_error | Total datapath message send errors | AGW | pipelined |
| arp_default_gw_mac_error | Total errors with default gateway MAC resolution | AGW | pipelined |
| openflow_error_msg | Total openflow error messages received by code and type | AGW | pipelined |
| unknown_pkt_direction | Counts number of times a packet is missing its flow direction | AGW | pipelined |
| enforcement_rule_install_fail | Counts number of times rule install failed in enforcement app | AGW | pipelined |
| enforcement_stats_rule_install_fail | Counts number of times rule install failed in enforcement stats app | AGW | pipelined |
| network_iface_status | Status of a network interface required for data pipeline | AGW | pipelined |
| subscriber_icmp_latency_ms | Reported latency for subscriber in milliseconds | AGW | monitord |
| magmad_ping_rtt_ms | Gateway ping metrics in milliseconds | AGW | magmad |
| cpu_percent | System-wide CPU utilization as percentage over 1 second | AGW | magmad |
| swap_memory_percent | Percent of memory that can be assigned to processes | AGW | magmad |
| virtual_memory_percent | Percent of memoery that can be assigned to processes without the system going to swap | AGW | magmad |
| mem_total | Total memory | AGW | magmad |
| mem_available | Available memory | AGW | magmad |
| mem_used | Used memory | AGW | magmad |
| mem_free | Free memory | AGW | magmad |
| disk_percent | Percent of disk space used for the volume mounted at root | AGW | magmad |
| bytes_sent | System-wide network I/O bytes sent | AGW | magmad |
| bytes_received | System-wide network I/O bytes received | AGW | magmad |
| temperature | Temperature readings from system sensors | AGW | magmad |
| checkin_status | 1 for checkin success, and 0 for failure | AGW | magmad |
| bootstrap_exception | Count for exceptions raised by bootstrapper | AGW | magmad |
| unexpected_service_restarts | Count of unexpected service restarts | AGW | magmad |
| unattended_upgrade_status | Unattended Upgrade status | AGW | magmad |
| service_restart_status | Count of service restarts | AGW | magmad |
| enb_connected | Number of eNodeb connected to MME | AGW | MME |
| ue_registered | Number of UE registered succesfully | AGW | MME |
| ue_connected | Number of UE connected | AGW | MME |
| ue_attach | Number of UE attach success | AGW | MME |
| ue_detach | Number of UE detach | AGW | MME |
| s1_setup | Counter for S1 setup success | AGW | MME |
| mme_sgs_eps_detach_indication_sent | SGS EPS detach indication sent | AGW | MME |
| sgs_eps_detach_timer_expired | SGS EPS Detach Timer expired | AGW | MME |
| sgs_eps_implicit_detach_timer_expired | SGS EPS Implicit detach Timer expired | AGW | MME |
| mme_sgs_imsi_detach_indication_sent | SGS IMSI detach indication sent | AGW | MME |
| sgs_imsi_detach_timer_expired | SGS IMSI detach timer expired | AGW | MME |
| sgs_imsi_implicit_detach_timer_expired | SGS IMSI implicit detach timer expired | AGW | MME |
| mme_spgw_delete_session_rsp | SPGW delete session response | AGW | MME |
| initial_context_setup_failure_received | Initial context setup failure received | AGW | MME |
| nas_service_reject | NAS Service Reject | AGW | MME |
| mme_s6a_update_location_ans | S6a Update location | AGW | MME |
| sgsap_paging_reject | SGS Paging Reject | AGW | MME |
| duplicate_attach_request | Duplicate attach request | AGW | MME |
| authentication_failure | Auth failure | AGW | MME |
| nas_auth_rsp_timer_expired | NAS Auth response timer expired | AGW | MME |
| emm_status_rcvd | EMM status received | AGW | MME |
| emm_status_sent | EMM status sent | AGW | MME |
| nas_security_mode_command_timer_expired | NAS security mode command timer expired | AGW | MME |
| extended_service_request | Extended service request | AGW | MME |
| tracking_area_update_req | Tracking area update request | AGW | MME |
| tracking_area_update | Tracking area update success | AGW | MME |
| service_request | Service request | AGW | MME |
| security_mode_reject_received | Security mode reject received | AGW | MME |
| mme_new_association | New SCTP association | AGW | MME |
| ue_context_release_command_timer_expired | UE context release command timer expired | AGW | MME |
| enb_sctp_shutdown_ue_clean_up_timer_expired | SCTP shutdown UE clean up timer expired | AGW | MME |
| ue_context_release_req | UE context release request | AGW | MME |
| s1ap_error_ind_rcvd | S1AP error indication received | AGW | MME |
| s1_reset_from_enb | S1 reset from eNB | AGW | MME |
| nas_non_delivery_indication_received | NAS non delivery indication received | AGW | MME |
| spgw_create_session | SPGW create session success | AGW | MME |
| ue_pdn_connection | UE PDN connection | AGW | MME |
| ue_pdn_connectivity_req | UE PDN connectivity request | AGW | MME |
| ue_reported_usage / up | Reported TX traffic for subscriber / session in bytes | AGW | sessiond |
| ue_reported_usage / down | Reported RX traffic for subscriber / session in bytes | AGW | sessiond |
| ue_dropped_usage / up | Reported dropped TX traffic for subscriber / session in bytes | AGW | sessiond |
| ue_dropped_usage / down | Reported dropped RX traffic for subscriber / session in bytes | AGW | sessiond |
REST API for querying metrics
Troubleshooting Metrics
On the gateways, magmad service collects metrics from all the services and pushes them to Orc8r. In Orc8r, metricsd receives the metrics and pushes them to registered metric exporters. Prometheus is one the main metric exporters. Specifically, Orc8r pushes the metrics to the edge-hub, which later scraped by prometheus instance.
On the query side, When we make queries through NMS or swagger, the Orc8r queries the prometheus instance directly.
We can effectively troubleshoot the metrics by looking at the logs in all these components involved. On the gateways, syslog might have error logs in case there is a failure during metric upload
ERROR:root:Metrics upload error! [StatusCode.UNKNOWN] rpc error: code = Unavailable desc = client_loop: send disconnect: Broken pipe
On the orc8r, we can debug the issues by dumping the logs on prometheus, prometheus-configmanager, prometheus-cache and metricsd
kubectl --namespace orc8r logs -l app.kubernetes.io/component=prometheus-configurer -c prometheus-configurer
kubectl --namespace orc8r logs -l app.kubernetes.io/component=prometheus -c prometheus
kubectl --namespace orc8r logs -l app.kubernetes.io/component=metricsd
helm --debug -n orc8r get values orc8r