[Grant Application] Autoscaling Endpoints V1

1) Title

Autoscaling Endpoints V1

2) Project Category

Infrastructure: for supporting the underlying code base of the blockchain - Infrastructure supporting tools, bug patches, node maintenance tools, etc.

3) Project Description

This project aims to build a globally distributed set of geo-routed autoscaling endpoints for the ICON blockchain. It will be developed completely in code such that new users can easily adopt the architecture or modify it to suit their needs. It is the first part in a series of grants aimed at upgrading the API infrastructure supporting ICON and handing off support of the network’s endpoints to the P-Rep community. As such, there will be two versions developed, both open source but with different objectives. The first version is a VM based solution that is easily deployed and operated by any community member. The second will be a more robust kubernetes based deployment for the main endpoints and experienced operators.

The community based deployment will be an extension of the existing automated P-Rep deployment with two flavors, a single node and multi-node deployment. The single node deployment is based on an ansible role to configure the node and is essentially done with minor modifications needed. The multi-node deployment needs to have Packer integrated with a load balancer along with a custom health check to inform routing. Both deployments will be provisioned with terraform on AWS with options to extend to GCP or other clouds where the features are matching on community request.

The main endpoints will be primarily deployed on Kubernetes and leverage the autoscaling groups developed in the community modules for application nodes. There will be three clusters deployed geographically distributed and georouted through cloudflare with a goal to achieve a response time under 100ms. A fast sync agent will be deployed such that scaling of the cluster can happen as fast as possible.

All deployments will include sidecar prometheus monitoring exporters and logging agents. Integration tests will be built with terratest and run in CI. Endpoints will be later extended to include websockets with ICON 2.0 and any additional APIs being built by the community as laid out in the adjacent stream data processing grant.

4) Project Duration

  • Development over 3 months
  • Operation for 3 months as soon as development is finished

5) Project Milestones

All milestones built from a bottoms up estimate of tasks. All items open to feedback on priority with associated trade-offs in risks running the endpoints.

Phase 1 - Community Modules - 3 Weeks

Initial development will focus on a set of robust community modules to deploy both single nodes and clustered environments. They will be used later in the main deployment due to their ability to scale with locally attached NVMe storage.

When operating autoscaling groups, it is important to be able to minimize scaling time. Currently fast sync options take up to 2 hours from node bootstrap largely due to how the whole blockchain is zipped up. As the chain grows, this time will grow. This forces node operators to use volumes 75% larger than the actual chain data due to downloads hitting the buffer and additional time to unzip the content. To minimize this, we will build our own sync agent to take incremental uncompressed backups of the blockchain and store them in an S3 bucket within the region of the cluster. A sync client sidecar container will be initially used with the code eventually integrated into the main container distribution with foundation approval for fast sync options. We are aiming to bring scaling time down to under 10 minutes.

The deployment will generally consist of a single terraform module which calls an Ansible role that is uploaded to Galaxy. This role then pulls a repo with a docker-compose and renames appropriate override files to run the application in the desired configuration. Users will be able to use any component of the stack independently (ie run the ansible against any desired host) though the terraform will generally be suitable for most users.

  • Docker Compose Stacks
    • Repo with base application and override files
  • Sync client and agent (source of truth node)
  • Ansible Roles
    • Main application (updates)
    • Custom health check
    • Source of truth node
  • AWS Modules
    • Single node module
      • SSL with certbot
    • Autoscaling group module
      • Network load balancer with SSL tied to domain in Route53
  • GCP Modules (Optional - Add 2,000$ to project)
    • Same components AWS deployment

Phase 2 - Main Deployment Modules - 1 Month

The second phase will focus on elements needed to bring the endpoints into production including logging, monitoring, and alerts. For logging, an ELK stack will be used to show status code and request type (method) for all requests. For monitoring, a prometheus stack will be used with a custom exporter to get metrics from the node and application. Alerts will then be built from Elasticsearch queries using Elastalert and metrics with Alertmanager then forwarded to Slack and Telegram channels. Grafana and Kibana dashboards for each region will be available.

The deployment will be built on kubernetes and will either route traffic down to the citizen nodes from the community module or to pods running inside the cluster. The decision will be based on tests run on using local volumes in kubernetes which is in active development though ultimately the goal will be to run entirely within kubernetes using the P-Rep application helm chart we developed. Using kubernetes will allow us to more easily integrate production features as they become available such as websockets and an L4 firewall for a private sync point.

On completion, we will run a single region endpoint and offer it to the community.

  • Kubernetes cluster (AWS will be primary development - GCP on request)
  • Kubernetes baseline setup
    • Service accounts
    • Prometheus operator
    • Logging / Elasticsearch operators
    • Consul (if using ASG)
  • Fluentd and Elasticsearch configurations
  • Grafana and kibana dashboards
  • Alertmanager and Elastalert alert configurations

Phase 3 - Master Cluster Deployment - 2 weeks

In order to achieve global redundancy and minimize latency between requests, three clusters (US, Europe, Asia) will be deployed. Cloudflare will be used for georouting with aggregated health checks bubbling up from each region to inform routing. A single master cluster will be developed to take high level metrics from each region and perform various cross-cluster elasticsearch queries to build a complete map of the use and health of the endpoints. A status page will be built to display this information publicly. Lastly a load test will be performed to understand thresholds for use and scaling in the event of a DoS attack.

  • Status page
  • Alerta feeding status page and distribution of alerts
  • Load test with Locust
  • Multi-region production deployment

Phase 4 - Operations - 3 months

Once developed, the endpoints will be operated for 3 months with continued operational expenses priced at cost.

6) Funding Amount Requested

Requested amount falls into two budgetary categories, development and operations. For development we are requesting 25,000$ USD with options to extend based on GCP inclusion (2k$) and feature requests. For operations, we are requesting a baseline operations fee of 1,500$ USD / month plus at cost expenses (~2k$/mo estimate) to be provided as a report. All improvements made to the cluster will be integrated into the main codebase so that best practices can be shared and adopted by the community.

7) Official Team Name


8) Team Contact Information (e.g. e-email, telegram id, social media)

Email: insighticon.prep@gmail.com

Telegram ID: @robcio

Social Media: twitter.com/icon_insight

GitHub: github.com/insight-icon

9) Public Address (To receive the grant)


Hash: SHA256