[Grant Application] Event Based Architecture

1) Title

Event Based Architecture

2) Project Category

Infrastructure: for supporting the underlying code base of the blockchain - Infrastructure supporting tools, bug patches, node maintenance tools, etc.

3) Project Description

This project aims to bring event streaming support to the ICON ecosystem and enable a broad set of APIs to access the blockchain. It will take newly produced blocks and broadcast them to a configurable set of middleware layers and backends. Additional APIs such as REST and GraphQl will then be exposed on top of the newly created backend.

This service will enable events on ICON to integrate with numerous off-chain application architectures and infrastructure components. This is needed in any application where complex business logic needs to be processed off-chain, or representations of the chain can’t be easily queried with the standard JSON-RPC interface.

For event processing, in Ethereum the equivalent service is called eventeum which this project will replicate the functionality of with improvements in implementation. Instead of a monolithic application, the service will be decoupled into a streaming event producer and event registration and broadcasting service. Users will then have the option of running it as a standalone service or on top of kafka from which exactly-once messaging can be configured so that even in the event a service disruption that events will be processed.

Several projects under active development right now will benefit from this project including Balanced DEX API and Omm though we are confident many more exist (please leave comments to let us know others). This project will extend the existing icon-etl tool with streaming support to populate a kafka backend from which the event registration service can be built on top of.

4) Project Duration

3 months

5) Project Milestones

The project is broken down to 2 key phases for development: an event polling websocket agent, and an event registration plus broadcasting layer. Each development phase has been laid out in the following bottoms up task analysis for the first two milestones.

**Note the cache and JSON-RPC Proxy is waiting on the availability of websockets in 2.0 and is out of scope of this grant and in diagram only for demonstrative purposes.

Streaming Event Producer - 3 Weeks

Eventeum has two modes of operation, polling and websockets. Until websockets are released in ICON 2.0, the polling mode will be developed and later integrated with websockets when released. To do this, we will start by extending icon-etl to short poll a local citizen node on the order of every 100 milliseconds or less. When a new block is produced, transactions and receipts are extracted through a series of API calls and then forwarded on to a configurable set of broadcasters. For ethereum-etl, these include postgres and Google’s Pub/Sub. For icon-etl, we will extend into kafka which will be the primary message queue for consecutive broadcasts.

Event Registration and Broadcasting Service - 1 Month

Events need to be registered with a corresponding middleware to be broadcasted on. For eventeum, broadcast routes are set through a file and events are registered with both a config file and a REST API. ICON service will replicate this functionality. Each registered broadcast will create a kafka topic and a subscriber from which events will be broadcasted on. Each registered event will then subscribe to the main data feed and produce to the given broadcasting topics. Event subscriptions can be along blocks, transactions, or receipts and filtered based on a variety of useful parameters as exposed by the REST API including event logs from contracts and transactions from addresses. Eventlog registrations will be parsed based on a registered schema in either JSON Schema or OpenAPI.

REST and GraphQL APIs - 1 Month

Built on top of a postgres database populated with a live stream of block data, both REST and GraphQL APIs should then be exposed. Both endpoints will leverage FastAPI framework with GraphQL needing additional support from graphene. These endpoints will by no means be optimized and merely mark the start of building a full suite of developer tools. Beta testing will start with members of the community with the expectation that additional effort will need to be spent to bring to production. Infrastructure developed during the adjacent project, Autoscaling Endpoints, will be used to bring the endpoints into production with it’s associated logging and monitoring infrastructure.

6) Funding Amount Requested

We are requesting 30,000$ USD to complete this project.

7) Official Team Name

Insight

8) Team Contact Information (e.g. e-email, telegram id, social media)

Email: insighticon.prep@gmail.com

Telegram ID: @robcio

Social Media: twitter.com/icon_insight

GitHub: github.com/insight-icon

9) Public Address (To receive the grant)

----BEGIN PGP SIGNED MESSAGE-----

Hash: SHA256

hxd278152d68a34baff1492d9abaf652fa02cbd088

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEfeqitHBQR7zWx7uGhofO4QrZt98FAl9JQHsACgkQhofO4QrZ

t98XXhAAlLPoKISf9LJMtCUS6flV1kpNqJb3iPvfGObNZqYpwcwYSeRObszgUz5b

aqOwz1JzDnrW0RrS/S9KGPawkT7byH16iu/c0EpU7vmgs/hSNmXLxbjdKHt/uEqs

pLjQ1YqmnglO7kNHYTD3bSe+uX1kngw9KaVLUzn1jRmVMB3avqf5PrFuYRry+elj

DdBPpg9KyLbmadFU3axtYLC25uA6n0LsbBbBMD8NZHzYzcOzz82t89iF+ws/iKy7

9oLdEKNAXv24PSlrX06dv7+zwyEyKnG+4h5KNz+MnsZlkcoweOCRsU+E71ZSwa5h

6ZjZ5VxWINuB5lvCHT4H1MkDFlt4OP2uTqFNVsxS7MNQQg8OFGvIJHooFomyoNL8

MIV/pAx1aPfYL3i6YN+usdxBCFdoClw69aGFPslusSzutXNvhrDW/j9cUhKZ3Xsh

qph8Ywj+tvKEPZOeLAh3B4+2KEUBcoyS+AfBfiPOo+5vLpo4C0xHmmsnIhVhtuiL

3NfqCfmXw5XXgXDvGF56VJuP455kbbcGfrGIOF6eI54FqDs7CQlEyFy4OqGUcgZD

vS01hDN5xxWWE1o/W2Mnme2cHAQNP6lYWelONuDwxhVFJh6j3Sysi5igMbTOtl05

kXoRdelS6pKOgc23nrH9tjlL9ktAF2kMFHUy/kvHS2bxeh11Oo8=

=5PUl

-----END PGP SIGNATURE-----