February 2023 outage update 1

Hello everyone, I just wanted to give a formal update as to what’s been going on over the last couple of weeks since I’ve posted last. There have been some developments, but my communication has been a bit fragmented, taking place mostly on the Survive the Wild discord server as well as one blog comment reply. I’m here to fix that.

My physical health

Thank you everybody for your concern and well wishes regarding my recent surgery. It took place on February 10th, and I’m happy to say that the recovery has all but concluded aside from a few painful twinges here and there as well as a weight lifting restriction I must follow for the next couple of weeks. I actually had quite a bit of energy for the first couple days immediately following surgery, but then after the worst of the pain was past and it became easier to sleep, I crashed pretty hard for a couple of days. Now I’m just slowly increasing physical activity day by day depending on how much my body will let me do so without complaining. We’re already to the point now where the post-surgery is only occasionally mildly annoying and I completely expect that to dissipate in the coming days!

Data recovery

The following is the support ticket that took place between me and OVH, the company that hosts my old server. The only things I’ve done were to redact personal information such as names and unique IDs and delete a couple brackets in the subject lines that were confusing the WordPress block editor, other than that, these emails are provided unmodified. You can probably make most necessary conclusions on your own after reading these.

tsXXXXX-ovh CS7XXXXX5 New Case – looking into data recovery options after disk fault – ns3XXXXX2.ip-19X-XX-XX.eu OVHcloud Support 2/13/2023, 7:31 PM

Hi,
We’ve created a support request CS7XXXXX5: looking into data recovery options after disk fault.
Server concerned:
dedicated server 19X.XX.XX.X44
Additional information:
10 days ago, my server went offline and I was informed via email that the disk with the serial number S2TXXXXXXXXX9 was defective and prevented my server from booting. I was told to backup my data in rescue mode. I have backed up the data on the one remaining disk that still appears in rescue mode, though for some reason it’s ntfs header information was missing and I had to create a raw image of the disk. I am now wondering if I have any options for recovering data on the defective disk? it is completely nonexistent to the rescue operating system. I’m guessing it failed completely and nothing can be done, but I wanted to check. If no other options are available to me, is it possible to perhaps post the drive to me physically encase I wish to consider some sort of professional data recovery option? Thank you for any information regarding this matter.
One of our support team members will contact you as soon as possible.
Sincerely,
The OVHcloud team

tsXXXXXX-ovh CS7XXXXX5 Case assigned – looking into data recovery options after disk fault – ns3XXXXX2.ip-19X-XX-XX.eu OVHcloud Support 2/13/2023, 7:54 PM

Hi,
One of our support specialists has started treating your ticket CS7XXXXX5: looking into data recovery options after disk fault and will respond to you shortly.
Sincerely,
XXXXXX XXX.
The OVHcloud team

tsXXXXXX-ovh CS7XXXXX5 Solution proposed – looking into data recovery options after disk fault – ns3XXXXX2.ip-19X-XX-XX.eu OVHcloud Support 2/13/2023, 7:59 PM

Hi,
A solution has been proposed for your support ticket CS7XXXXX5: looking into data recovery options after disk fault.
Resolution comment:
Hello, Thank you for contacting the technical support of OVHcloud.
It is a pleasure to meet you!
Normally, if your server was installed in a Raid 1 configuration, all data in one SSD will be duplicated in the other.
However, if you configured your server in a JBOD configuration the data in the defective SSD will not be retrievable as per the technologies that will prevent to read any information as the chips that is defective on the SSD corrupted the information on the device itself.
Unfortunately, as this is a SSD the data cannot be retrieved.
Thank you for your understanding.
To confirm the resolution of your ticket, you can either reply to this email indicating “Accept” in the first line of your response, or by closing the ticket in the control panel.
To reject the proposed solution, please either respond to this email indicating “Reject” in the first line of your response, and include additional information, or or continue the exchange in the control panel.
Please note that in the absence of any feedback in the next seven days, this ticket will be closed automatically.
Sincerely,
XXXXXX XXX.
The OVHcloud team

I have not responded yet

Me and the Survive the Wild administrators have been discussing what to do about it for a few days now. Of course, working on the update is a more important fish to fry than getting back that data in the short-term, so conversation about it has been a bit slow (a few messages a day kind of thing). To sum it up, the only option for continuing to pursue data recovery would be to continue to press OVH to ship the drive somewhere which would allow us to then consider a professional data recovery option. Most likely, such a recovery option would be rather expensive, and I don’t know at this time about shipping costs. Also, OVH at least maybe thinks that the drive is pretty fried. If you look at the message timestamps above though you’ll understand why I say maybe… I’ll let you draw your own conclusions there. Add to all that, we’re also a bit pressed for time here. I have no doubt that OVH is going to charge me for every day that this server stays online in rescue mode, which is of course a complete waste. We need to destroy this server as soon as possible to avoid being charged for a server we’re not using. Our backup is 3 weeks old and all credits will be restored. Is it really worth continuing this potentially fruitless fight to recover 3 weeks of data? We’re considering trying to see if we can simply just obtain physical possession of the drive just so that we have options down the road. As you can see though I literally asked for that and the support agent just skipped over that part of the question, apparently confident in the fact that they would be sending me a worthless slab of metal.

So those are more or less the current facts on the data recovery situation, we’re not sure it’s worth pushing forward with. The time could probably be spent on much more long lasting endeavors, such as insuring that we never run into this situation again. It’s sad what happened to OVH. When I purchased my server, I’m pretty sure we had 1gbps bandwidth. I could call their phone support line and get helpful resolutions to my issues. Then, very silently indeed (I didn’t notice for months), our bandwidth cap dropped significantly, and when I tried to get phone support about this drive failure, I was told that I am not paying for a high enough level of support to talk to somebody. If I try to use their live chat system, I get this hilarious error message, pasted verbatim. “We are sorry we could not serve you this time. Please email us at <put email address here> and we would respond to you as soon as possible.” I tried later when I was sure their offices were open, and got the same result. Being completely honest, I lost all faith in their support system when I realized that they couldn’t even be bothered to replace the support unavailable message placeholder in their live chat system. This is why I ended up procrastinating so long before managing to bother going through their only somewhat accessible ticket creation system, but in the end I felt it would be irresponsible to not at least knowingly ask somebody at the company. After all is said and done, I again apologize for not being more prudent about transferring backups off-site more often, and promise to improve immediately. We will not release the Survive the Wild update for example until we are sure that we have a much more stable backup plan in place. It also probably goes without saying that we are no longer doing business with our previous server company and hope that our new one will treat us better. Thank you everybody for your understanding in this unfortunate matter.

Survive the Wild

This server crash has been an absolute blessing in disguise in regards to actually moving forward with this update. Though we knew the public server was indeed distracting us, it’s still another thing entirely to see what a general lack of stress in regards to the situation can achieve. First and foremost all of the staff can finally reside where they are needed most, which is of course the development server. More importantly though, it is priceless to be able to know that we can sit down and focus on something involving the update for hours with the secure knowledge that we will not be distracted by the public game in any way. Because of the aforementioned surgery/recovery not to mention getting this website and a couple of games online, data recovery attempts etcetera the last 2 weeks regarding Survive the Wild have actually been slower on my end than I would like. However, before the surgery I managed to finish the little SQLite Object Relational Mapping class that we needed, and afterword’s during the recovery I fixed a few remaining long standing bugs and shared some needed script functions for worldmakers. A couple days before writing this post, I began converting the data storage for support tickets and user accounts from homegrown file formats to SQLite, using the ORM class I’d just finished before surgery. I also need to do this for characters and for a few other sections of data. There are several reasons we’re switching to SQLite, from robustness of the format to easier and quicker retrieval of data to reduced stress on the server disk drives etc. Of course it makes sense to do this now while we have a few weeks to convert the data in a state where it will remain unchanging due to the game being down. Some long-time players may remember the account database crash in January 2022, switching to SQLite should insure that such a thing never happens again. As I said I’ve already started on the conversions, and hope to be done with them in the next week or 2.

Meanwhile, objects are being built, converted and finished faster than they ever have been before. While I work on this data conversion, admins and worldmakers both have been and will continue to be working on the conversion, creation, and finalization of various game objects, npcs and items. Some may remember that we mentioned replacing the old RTig scripting engine we previously used for game content with a shared angelscript module. Originally we really wanted to release one final update using RTig, then focus on an Angelscript conversion afterword. Unfortunately, it became clear in October last year that this wasn’t going to work, RTig would be too slow and bulky for worldmakers to continue using. The Angelscript conversion thus has been underway for a couple of months now, and the worldmakers are now fortunately on the last leg of that process. Hopefully they’ll finish around the same time I’m done converting all of our data to SQLite.

We’re excited to say that currently, we believe the SQLite and the RTig conversions to be nearly our last giant hurtles! After everything is converted to a usable degree, we will spend a couple weeks dealing with more minor aspects to the update. For example, as I’m converting support tickets to use SQLite, I am adding categories to them so that we can use the same system for bug reports and suggestions, or really any other place a little conversation thread could be useful. This has been on our radar for months, and the only reason we haven’t gotten to it yet is because we’ve always had bigger fish to fry up until this point. It worked out, since I have to modify the ticket code anyway for the SQLite conversion it will be easier to just add ticket categories at the same time. We also need to give the new tracking system about one full day of undivided attention, which probably means 2 or 3 days of real world time. It needs polishing in a few areas, particularly regarding 3d distance calculations and the hiding of objects that are no longer in site, a few other things too. A bit of UI for admins needs doing as well, which was again saved for last due to the fact that we’ve had bigger things to focus on lately. At the time of writing, the only other system that may eat up a couple/few days of time that we must finish is the ability for worldmakers to transfer scripts from the beta server to the main server. In short, the plan is that worldmakers will do all of their actual object development on the game’s beta/development server, and then with just a couple of buttons, quickly transfer all of their work to the public game once it is complete and ready for integration. This is so that a worldmaker can’t accidentally crash or inhibit the public game while working on a script, and also to not give them undue power on the public server. Our plan is to use a programmatic git repository to solve this problem, using something like libgit2. A worldmaker on beta can click an option in their interface to commit their changes to the scripts repository, then on the public/main server, they can click another option to pull their changes from that repository into the game. This way we can also see with much more detail than before who changed what maps, data or scripts and how they were changed. The library I’m intending to use for this looks very cool, and will hopefully allow me to finish this system in days or less… hopefully. After all that? we simply intend to spend a few days looking over all of our work. We’ll likely all get together and add several new objects and quests during that time without the distractions of bugs in the core systems to inhibit us, I’ll look through a lot of code specifically to see if there are any other ways I can find to reduce lag, we’ll call all the beta testers at once to test as much as possible one last time etc. My plan is to spend one work week on this last bit, so 5 consecutive days. And then, finally, oh finally, it will be time for what everyone is waiting for, a new chapter in Survive the Wild’s history to begin!

The only downside is that the thing we were planning on getting at least slightly working was shelters. That is the other huge reason why the update was going to take longer had the old server not crashed. As it is, all shelter items will be returned to players inventories for the time being. If we chose to finish shelters to a point where players could use them, this could take several more weeks on top of the things we already have to finish. At this point, we’re pretty sure that Survive the Wild with no shelters for a few weeks is better than no Survive the Wild at all. We think that shelters are a much les crucial element to the game than Animals, and so we do not foresee issues such as what happened when all animals were disabled in late October 2021. So as you can see we still have a few things to finish up over these next weeks, but especially since we’re saving finishing shelters for after the release, we’re genuinely nearing the end of this long and boring wait and are so excited to share what we’ve been working on with all of you!

Two other small games online

In the mean time, I did decide to take a day and get a couple of smaller games I’ve created in the past back online, one of which has never been public before now!

While considering bringing the Ah Damn It scoreboard online again, I realized that I had another older project I’d worked on just sitting on my computer collecting virtual dust. Some members of the community may have heard of this from 2019 when I publicly talked about it a little, but due to me developing a somewhat negative relationship with python in the audiogame development realm, I stopped developing the project for a while. Rest assured I do fully intend to rewrite it though in my new NVGT engine, the same one that Survive the Wild now uses instead of BGT. Regardless of the game’s development state, enough was done with Constant Motion (my collection of endless runners) that I knew people could still enjoy it if I released it as a rough concept demo, until such time comes when I rewrite it.

I have also brought my Ah Damn It project back online. Both Constant Motion and Ah Damn It have working scoreboards, so you can feel free to compete with your friends to see who’s the best! You can learn more and download these restored games on my games page. Hopefully they at least provide a bit of fun while me and the Survive the Wild staff continue to work at bringing the game online as fast as we are reasonably able to do so.

And that’s an update

Well, I think that’s about it for now. As I said before, I will continue to slowly bring more games and services back online in the upcoming weeks, and apologize that some of them, such as Any Audio, will take a while to come back. Thank you everyone for your support, well wishes, general comments and endless patients during this unfortunate period. I look forward to everything returning back to normal as time goes on, and for goodness sakes, to be able to blog about something other than a data disaster! 😀 Not that I know what I’ll write about though…