That was bad.

As a quarter our customers know, last week was rough. Three primary servers struggled as the ZenCash block size increased last week, with one struggling for the entire week before being brought completely back online. To compound matters, the new server to which some load was shifted began struggling as unassigned nodes entered a race condition. They complained without pause about their lack of a Node ID in their configuration file and proceeded to consume all available CPU.

I’ll say it again. That was bad. We failed to live up to your expectations in numerous areas. Emails went unanswered. Nodes were down. Orders went unfulfilled. We lost some of you as customers and I hold myself personally responsible for this failure.

The stats

As I write this there are 23 nodes with active exceptions, which will be repaired by hand. At the peak of the outage, we had 323 nodes with open exceptions, with only 75% of node days seeing ZEN rewards earned. We’re climbing out of that hole now in our reports. For reference, I’ve included a chart of rewards earned per day versus excluded for the past three months, grouped by week. The exclude data is slightly skewed as almost all new nodes take an exclusion the first day of their life, but you can see that we have had three outages of varying sizes. After outage two we began creating something to serve our customers better.

What’s next

We have constructed a cluster to attack the problem from two directions. First: All nodes in the cluster are receiving increased access to resources from where they are today. Disk IO killed us as the block size increased, so we need to reduce the number of nodes fighting for disk access. RAM will be increased for each node. This is going to increase our costs, but challenge times will go down. With these two changes, reliability will go way up. Here’s the momentous change: true fail-over for nodes.

In the new cluster, if a node isn’t able to be immediately and automatically fixed, it is rebuilt from a known good copy. If a server dies, knocking its nodes offline, the nodes are automatically brought back up on random servers across three countries and two continents. During testing, nodes that were knocked offline by a simulated host failure were back online elsewhere in under 10 minutes.

The migration will begin slowly. It will take place with care and in batches to ensure that no nodes experience any more downtime than is absolutely necessary. For the time being new orders will still be provisioned in the soon to be legacy environment until everything is cut over and the last of the old environment is turned off.

Redundancy and automatic fail-over aren’t the only things we need more of. It’s become clear that email can’t go unaddressed, and Chainsaw needs more heads and hands. I’ve begun the process of looking for additional help to increase our manpower. Everything that contributed to this outage needs an upgrade.

Amends

Tales of a future state made of shining towers of ZEN nodes which never go offline are not enough. You did not receive the service you expected, and that is not the way it should be. To that end, we offer existing customers the following:

nodefire

For those of you who pre-ordered Super Nodes: We have dropped our price to complete directly with the discount providers. You have not locked yourselves in at the pre-order price, and we look forward to helping you avoid some sticker shock in August.

We are going to continue to work tirelessly, learn from our mistakes, and iterate quickly to better serve our customers. Substantial change is coming to Chainsaw.Ninja, and we hope to see your faith in us rewarded.