September 14th Outage Postmortem

On September 14, 2020, the VoltBuilder server went down for several hours. The purpose of this document is to clarify what happened and what we are doing to make sure it doesn’t happen again.

It’s usual in these cases that there isn’t a single failure - one problem uncovers the next and so on. We used this failure to see where other weaknesses might exist in our processes.

The root cause was the main build server running out of space. While we have jobs in place to clear app data soon after builds are completed we found that one of our build tools was caching intermediate build products. We’ve improved our existing clean-up process to include the cached files. We’re also more closely monitoring the free space on the build server in case another anomaly crops up.

Generally resolving the space issue would have solved the problem, but in this case we continued to see a number of problems with the main build server. Further investigation showed that we were experiencing a localized DNS outage. In this particular case it appears that when the host ran out of space, the network adapter settings were somehow corrupted, leaving the main build server able to access the network, but unable to resolve any DNS names. Resetting the network interface resolved this issue.

Once the DNS issue was resolved we were able to get our build infrastructure back to building apps. No submissions were lost during this time, though we did have a bit of a backlog to work through. It took about an hour: no jobs were lost.

Over the course of dealing with this outage we discovered a few other changes we’re implementing to keep VoltBuilder running smoothly and consistently. These include:

• Making our network retry code more resilient so we can deal with a larger variety of outages, should they occur.
• Fixing some minor bugs in our build status checking logic so that longer than average build times don’t cause unnecessary network traffic.
• Continuing to enhance and improve out monitoring and notification so that our engineers are notified of any system anomalies so they can be resolved quickly and efficiently.

In order to make our systems resilient against such outages we will continue to expand our server infrastructure.

We are very sorry for the inconvenience this issue caused.

What VoltBuilder Does

VoltBuilder converts your web projects into native Android and iOS apps. VoltBuilder is a wrapper for Apache Cordova and Ionic Capacitor, so you can add native features to your apps via Cordova and Capacitor plugins.

VoltBuilder's Key features:

• Produces native executables — for easy testing & app store publishing.
• Publish in minutes — VoltBuilder uploads directly to Google Play and the Apple App Store.
• Easily generate certificates — VoltSigner makes Android and iOS certificates. No Mac needed!
• Nothing to install — just head to the Uploads page to get building.
• Nothing to store — we don't keep your passwords, certs, or code in our cloud.
• Always up-to-date — we install and maintain the latest SDKs and tooling so you don't have to.

How to Use VoltBuilder

It's as simple as Upload, Download, Run. Done.

1. Zip your web project.
2. Upload it to our platform.
3. Download the native app or let us deploy it to stores.
4. Run it instantly via QR code or direct link for testing.

Ready to Build?

Skip the native-dev hassle. Sign up and start your first build today!