February 2023 outage information and updates

Well, I guess I will be using this blog after all. This isn’t exactly how I’d hoped to start my sojourn into blogging, but such is life, I guess.

It was intended that this new website would go live at or around the time the next Survive the Wild update got released. Unfortunately, on Friday, February 3rd 2023 at about 05:10 PM central time, the company that hosts the server I’d been using since late 2017 detected a fault in the main solid state drive that ran the server’s OS, my website, Survive the Wild, and pretty much all of my services accept Any Audio. As of February 4th, I do not think I can recover the data on that drive, particularly considering that I am unable to remotely access it even in the server’s rescue operating system.

As it happens, however, I’ve been setting up this WordPress website very slowly as a side project since early January, if you’re seeing this post shortly after it’s publication date you can see that the site still needs work. In order to communicate to the largest number of people possible though, and also because much of my old one is now destroyed or at least scattered, we’ve decided to just bring this website out of staging early instead.

The loss of data is only most of the problem, but the loss of my old running environment is unfortunately another. Though luckily I’ve been preparing a new, much better server ahead of time for Survive the Wild, this failure will still result in a lot of reconfiguration and redeploying of services that may unfortunately result in their continuing downtime or instability until further notice, or else as otherwise described later in this post for each service.

Updates since this post

• February 2023 outage update 1
• Survive the Wild early March update

I do recommend checking the main blog page for updates, as you’ll get them instantly rather than having to wait until I remember to update this post to include them in this section.

Survive the Wild

At this time of course, most of you will be interested in the status of Survive the Wild. Unfortunately the game was running on the drive that failed, and we lost a lot of data. We are almost certainly going to have to restore to an off-site backup, the most recent of which at this time was made early on January 11, 2023. Anyone who has purchased credits from the store since the backup date will of course have their credits restored, as transactions are recorded on PayPal. If we manage to somehow gain access to our logs since the backup date, we can then help restore many more items that players may have collected since that time. Me and the staff genuinely do apologize to everybody for the 3 weeks of lost gameplay, and promise to improve our off-site backup strategy considerably in the future so that we never get more than a day or 2 behind, even if multiple disk drives crash at the same time. Aside from restoring credits, we may not be able to do too much about lost gameplay accept hope and try for the data back, but we will certainly learn from this incident and insure that this is far, far less likely to happen in the future.

I’m guessing that what most people want to know is when the game will be back. Here, I have some bad news and some good news depending on how you view it. Me and the staff team have discussed it, and have reached a completely unanimous decision, we are of course very sorry to have to do this and hope we never have reason to do it again. Quite simply, the bad news is that Survive the Wild is going to remain offline for the next couple/few weeks at the very least. The good news is that the next time Survive the Wild comes back online, you all will be playing the public beta!

Every long time player knows it at least to some extent, so I’m just going to admit it here. The public version of Survive the Wild has been a bit of a mess now for months for several reasons. First, development has been almost exclusively focused on the next game version for a while. This has been for several reasons. First, the client updater for the public version is at least somewhat unstable unfortunately. Every time we try to release an incremental update, we end up having to distribute patches to a bunch of people who are unable to download it. Creating patches that then fail to download for some people and can sometimes break themselves eventually started to seem redundant when we could just fix all of the bugs properly for the next version during a well planned public beta phase if I haven’t fixed them already while coding the update. Furthermore given the current updating workflow, backported code from the new version to the public one ends up usually being rather untested, as it gets piped directly to the public server after the porting. Again, we keep running into the fundamental issue that, for example, spinning up a tertiary beta server and calling up the entire beta team to test bugfixes for the public version which we already intend to replace with the update ASAP seems like spending time that could otherwise be used to get this update out faster. In short, the public game version is a nightmare to update incrementally at the present time.

At least partially because of this initial failure to implement a proper incremental updating system (which I profusely apologize for and promise to improve), the public version of the game has also unfortunately become increasingly difficult to run particularly while trying to develop an update on the scale that we’re trying for. As has been mentioned in previous announcements on the game, the staff ranking system on the public server is very underdeveloped and does not allow us to promote anything other than completely trusted help. Aside from that though, we don’t want to train new staff on two versions of the game at a time. We think it would be a very stressful endeavor for new staff members to learn a bunch of legacy administration commands while also learning the newer systems on the upcoming game version at the same time as they’re trying to help players on the public game. It’s just not viable, and as such, the same couple of long time admins have been putting every bit of spare energy they have into helping as many players on the public game as they can. However, me and those same long time admins are the only people who are able to train new staff and who are able to make some of the larger executive decisions needed to move this update forward. They are not able to spend as much time as needed doing that while trying to help players on the public game as well, particularly as new bugs continue to be found and old ones continue to cause an increasing pile of bugged deaths, dfc losses, and other negative experiences that need tending to.

All of this and more has lead to a less than optimal experience to say the least both for players and for staff. As the number of minor incidents continued to increase on the game, me and the staff have been questioning whether to make the decision to temporarily shut down the public server for a couple of months now so that we could all reset and exclusively focus on pounding out this update as fast as we can. Despite the increasing stress of running the game in this state though, despite me saying that I understand if they need a break I might add, my amazing admins have continued to tough it out over these last couple of months to the absolute best of their ability, and I give them huge props for that. Now, though, less than a week after taking a break from renting an apartment for a while thus freeing me of much financial stress, the drive that the game was running on has crashed, taking the entire server operating system down with it. After discussing it with the team, we decided that it was time to use this rather unfortunate crash combined with the lack in financial stress as an opportunity to get this update done, and to get it done right. With the public server down, we’ll have as much time as we need to convert the character and account data from the public server to the new format used by the upcoming version, and test for any errors in the conversions. The long time admins will be able to spend more time directing the worldmakers and answering their questions, gathering and training more moderators on the new systems, and generally just being around to help with the update where ever they might do so, without worrying that a small crisis might pop up on the public game at any time what so ever to distract them from their work. This also gives me more time to try to recover our lost data, which would mean that we wouldn’t have to revert to a backup. I don’t expect this to be successful, but taking Survive the Wild offline for a bit at least gives me more time to try. There are more reasons for this decision as well. A surgery I had performed last year has sadly been slowly undoing itself, and this drive crash happened to take place days away from the re-op. Our new server machine runs linux and not windows, so the old bgt server application would have to be virtualized through wine or a VM, thus likely slowing it down. While repurposing the old server would probably be an option, it would mean replacing the broken drive, and probably the loss of any remaining chance, slim though it may be, that we’ll ever see our missing data again. Dealing with hectic server migration, reconfiguration and virtualization plus maybe a client update to point to the new server (not sure if that’s needed or not because of an anticheat measure) plus maybe people’s relays no longer pointing to the new location plus several distressed players who want their 3 weeks of game play back plus who knows what else directly following surgery is probably not the healthiest idea in the world. While the last surgical recovery wasn’t very painful or draining, the operation failed in the long-term and so a new method is being tried this time, where I have been warned that the recovery process may be more intense than the last one. For all these reasons and more, we’ve decided that though it’s disappointing in the short-term, the most wise decision in this case is to shut down the public game, finish pounding out this update, and come back online in public beta mode when the update is ready for testers. Again me and the staff genuinely apologize for any disappointment this causes, and hope we can make some of that up to you with the contents of the upcoming update.

There are several ways to get updates on this situation, particularly since we can give no concrete dates or times in this post. If you want to directly communicate with us, the best way to reach any administrator is currently through the game’s discord server, where significant announcements also always get posted when something happens. I will also post on this blog, so feel free to check here or use the rss feed to get more instant updates. Though I can’t promise given the current state of twitter, an announcement may be posted there as well, if nothing else when public beta goes live. Finally, you can try to keep launching your Survive the Wild clients every couple of days if all else fails. At the time of writing this post there is simply some sort of connection error, but I will try to get that replaced with an easily updated status message within the next couple of days so that people can also get a few notifications that way. If you happen to be a member of this, though the skype group is considered completely inactive, we will most likely post there when the public beta goes live as well for good measure. More slowly, you will see updates propagate to the audiogames.net forum. Just keep a close eye out, and you’ll likely find new information on the situation quickly, if nothing else you can simply feel free to bookmark this blog post and check back every few days for updates (denoted by headings) if needed. Thank you all for your continuing patients and support during this unfortunate time, we look forward to seeing you at the launch of the public beta!

Missing games and other website content

As to my smaller games, I will make sure they get uploaded to the games page on this new website over the next couple weeks. If you’re seeing this post soon after it was published, no doubt you’re seeing quite an empty website. This page should have been made public several weeks from now, instead it’s being rushed. Feel free to check the games page every few days for more updates.

I’m not completely sure when the Ah Damn It scoreboard will become active again, it depends if I can easily find the related php scripts. I don’t know how many people still play that game and generally I must spend most of my time focusing on Survive the Wild.

All of my old BGT code that used to be public on my development page will be uploaded to my Github also over the next couple of weeks.

Any Audio Network and Awesome Tech archives

I was able to image the disk that contains Any Audio’s data. Well, most of it. What I think we may have lost is the most recent version of Any Audio’s sql database. I do have a version of it, but it appears to be a few months older than when the website became frozen. I’m not sure how much metadata I’ll be able to restore from clips newer than mid November 2018, of course I will do my best though and of course apologize for anybody who may need to reupload clips some day. As with other sections in this post, I’ll update this paragraph if I end up finding a more recent version of the site’s database.

I would unfortunately expect Any Audio’s archive to remain unavailable for the next few weeks at least, again because my main focus must remain much more on Survive the Wild’s upcoming update than anything else. Basically, I’ve intended to rewrite Any Audio in python for a couple of years now, but have never gotten around to it. It’s honestly a side project that I’m very excited to work on though when I have time. In fact, I may do it as soon as the next STW update is released and is stable enough for me to decrease my focus on it for a few weeks. The new version of Any Audio will include post tagging, better moderation features, multiple file formats, a working password reset form, view filters, an API and much more. At the time the new version of Any Audio comes online, as much of the old archive as possible will be available under an archived category, the Awesome Tech archives will be available there as well.

If there is huge demand for the Awesome Tech archives before Any Audio comes back, I may provide those as a zip file that can be downloaded, basically a bunch of mp3 files and a feed.xml file with their metadata. I’m guessing though that this archive will simply become available when Any Audio returns.

Final notes and thoughts

Well, in general, talk about an unpleasant and entirely unexpected mess! I want to generally apologize to anyone who was at all effected by this outage in any way, be it by some lost stw game play or a deleted any audio clip, or even a prematurely cleared game score. I want to assure everyone that I am going to learn from this situation and be more prudent about backups in the future, particularly in regards to data that I did not create and which I have been entrusted with keeping safe. By losing user data I have failed to handle it properly, and for that I am genuinely sorry and seriously intend to do better in the future. Whatever happens, next time my server’s disk drive fails, I’ll be much more ready to handle it and it won’t effect users of my services nearly as much if I have any say in the matter. Thank you again everybody for your patients, kind support and well wishes during this time.

And with that, lets see if we can’t manage to make the third post on this here blog a bit more of a positive one, shall we? Because I think this is a really bad start to a blogging hobby! 😀