Earlier this evening I discovered the root cause of behind the Super Node outages.
A single incorrect character when specifying the automation’s schedule changed “Run once every 22 hours” to “Run each minute for an hour after 22 hours”. This flooded the Super Nodes with 60 challenges and left their the ZenCash provided Super Node Tracker app in a confused state. The schedule has been corrected, and the previously adjusted automation to tackle other exceptions that crop up has been working as expected
Secure Node Cluster Update
The first pilot of 9 Secure Nodes have been migrated in to the new cluster, with room for 21 more. As nodes migrate away from the first-generation of our infrastructure their former hosts are re-purposed to join the third-generation cluster.
Two nodes which were not scheduled to join the cluster were moved to it after it became clear that migrating them would be faster than fixing them by hand. I expect this to continue whenever extreme blockchain corruption is detected going forward.
So far, challenge times for migrated nodes have dropped from ~150-200 seconds to ~40-60 seconds. I’m looking forward to rapidly moving all Secure Nodes in to the cluster over the course of the next few weeks.
Super Node Orders – Temporarily halted
Until the Super Node offering is stabilized, I’ve made the call to halt spinning up new Super Nodes. I expect to leave them as “out of stock” for one week until stability is proven.
Emails and a Survey
We are still struggling to keep up with the load of emails. New Secure Node orders are being completed within 6-48 hours, but emailed inquiries are not being responded to in a timely manner.
I would like to get your opinion on how we can best improve the situation. If you are able, please fill out the survey I’ve linked below* (redacted from the web post). The first question deals specifically with how we should increase our headcount, and the remaining 9 questions are for you to rank us in various areas of customer satisfaction. By answering honestly, you help us know where to focus our efforts.