Post Mortem on Network Pause
At block height 32,031,263, 14:52:31 UTC, March 20th, 2021, the ICON Network paused due to a large amount of spam transactions. These spam transactions caused the ICON Network to halt temporarily until the ICON Foundation fully blocked the ip address from the foundation’s citizen group and released a hotfix. We suspect it may have been an attack but are not sure.
What happened?
At block height 32,031,263, the address (hx542550181ea328195d3b02850692b2988fcb5447) generated a huge amount of spam transactions on the foundation’s citizen group from Virginia region. The attacker repeatedly generated ICX transferring transactions, which resulted in too many transactions in the queue on the node. These transactions continued to be broadcasted to other P-Reps, causing the entire network to hang.
Despite blocking access to the Foundation’s citizen group, the address continuously generated spam transactions on the network using more than 100 ip addresses.
Since then, the Foundation has released a hotfix update on March 20th at 17:55 UTC, and many Main P-Reps have completed the update. At the same time, the Foundation implemented a more strong defense policy on the citizen group on nginx level. After this time, most spam transactions were blocked, but large transactions were still being processed in the queue.
On March 20, 18:58:25 UTC, the network was completely restored.
How did we solve this issue?
We have applied infrastructure-level and core-level solutions to address this attack.
Basically, at the infrastructure level, when the citizen group receives a request from external, each request is divided into ‘read’ and ‘write (sendTransaction)’ and classified on the citizen group’s nginx. Therefore, the citizen group can have a strong rate limit. Now the citizen group’s nginx will block the sender if the same sender continues to request a ‘sendTransaction’ command.
Also, at the core level, the ‘dosGuardEnable’ option was added through the release of version 1.8.10 of the ICON Service. This option has a function to block the creation of a transaction for a certain period when the same address repeatedly creates transactions that exceed the threshold.