SPARK Performance Testing

The testing was completed during Phase 4 - Attestation Campaign where the highest demand for SPARK occurs.

Tenant Details

Total Analyzed Sites
Approximately 125K
Total Admins/Owners (A/O)
Approximately 75K
Email Notification Time
1200 Eastern
Email Notification Method
Phased; ~15K A/O Notifications on Day 1; +15K A/O
Notifications on each subsequent day over the first five days of attestation campaign.
App Service Plan Tier
P3V3
Instances
15
SQL Tier
Premium 1 (125 DTU, High Performance SSD)

Metrics

CPU Percentage (Average) By Instance (15 Instances)

App Service Plan

Function Execution Counts (Sum) By Instance (15 Instances)

Function Executions

SQL Server DTU Limit (Average), DTU Used (Average)

SQL Server

Conclusion

Each day, the notification email was sent to admin/owners at 1200 eastern time. This is during peak business hours. Immediately proceeding the email notification, all App Service Plan instances would spike in CPU utilization, and would remain elevated over a period of approximately 15 to 20 minutes. This coincided with the rapid increase in Function Executions within the Function App running on each virtual machine instance. A significant increase in DTU consumption against the SQL server can also be observed. After the initial 30 minute window proceeding the email notification, then CPU utilization significantly drops. Function Executions stabilize, and maintain a regular average. DTU utilization stabilizes.

Within the first hour, organizations can begin their scale down procedures, such as:

  • Lower Instance Count from 15 to 10; monitor demand
  • Continue to Lower Instance Count and Tier (P3V3 -> P2V3 -> P1V3) relative to CPU utilization and total Function Executions

From this, we may draw the following conclusions:

  • Upon receiving the first notification that action is required, organizations should expect a significant user demand increase within the first hour. Organizations should be prepared for this significant increase by properly turning and scaling.
  • Organizations should expect user demand to stabilize after the first hour. In turn, they may begin their scale down procedures for that day.
  • Organizations may need to continue to monitor and manage the App Service Plan as the attestation campaign proceeds. A standard rhythm within the organization should emerge which can be leveraged to plan daily procedures.

Why does SPARK not utilize Elastic Premium App Service plans?

Organizations are free to leverage Elastic Premium App Service plans. However, by doing so, this results in a mandatory premium tier VM (always on, always available, always consuming). As such, during off hours organizations are unable to scale down into the consumption only tiers which may reduce overall consumption and expense of SPARK.

In turn, the SPARK development team has opted to provide guidance on manually managing the App Server Plan tier and instance configuration.