February 2023 outage information and updates

Well, I guess I will be using this blog after all. This isn’t exactly how I’d hoped to start my sojourn into blogging, but such is life, I guess.

It was intended that this new website would go live at or around the time the next Survive the Wild update got released. Unfortunately, on Friday, February 3rd 2023 at about 05:10 PM central time, the company that hosts the server I’d been using since late 2017 detected a fault in the main solid state drive that ran the server’s OS, my website, Survive the Wild, and pretty much all of my services accept Any Audio. As of February 4th, I do not think I can recover the data on that drive, particularly considering that I am unable to remotely access it even in the server’s rescue operating system.

As it happens, however, I’ve been setting up this WordPress website very slowly as a side project since early January, if you’re seeing this post shortly after it’s publication date you can see that the site still needs work. In order to communicate to the largest number of people possible though, and also because much of my old one is now destroyed or at least scattered, we’ve decided to just bring this website out of staging early instead.

The loss of data is only most of the problem, but the loss of my old running environment is unfortunately another. Though luckily I’ve been preparing a new, much better server ahead of time for Survive the Wild, this failure will still result in a lot of reconfiguration and redeploying of services that may unfortunately result in their continuing downtime or instability until further notice, or else as otherwise described later in this post for each service.

Updates since this post

I do recommend checking the main blog page for updates, as you’ll get them instantly rather than having to wait until I remember to update this post to include them in this section.

Survive the Wild

At this time of course, most of you will be interested in the status of Survive the Wild. Unfortunately the game was running on the drive that failed, and we lost a lot of data. We are almost certainly going to have to restore to an off-site backup, the most recent of which at this time was made early on January 11, 2023. Anyone who has purchased credits from the store since the backup date will of course have their credits restored, as transactions are recorded on PayPal. If we manage to somehow gain access to our logs since the backup date, we can then help restore many more items that players may have collected since that time. Me and the staff genuinely do apologize to everybody for the 3 weeks of lost gameplay, and promise to improve our off-site backup strategy considerably in the future so that we never get more than a day or 2 behind, even if multiple disk drives crash at the same time. Aside from restoring credits, we may not be able to do too much about lost gameplay accept hope and try for the data back, but we will certainly learn from this incident and insure that this is far, far less likely to happen in the future.

I’m guessing that what most people want to know is when the game will be back. Here, I have some bad news and some good news depending on how you view it. Me and the staff team have discussed it, and have reached a completely unanimous decision, we are of course very sorry to have to do this and hope we never have reason to do it again. Quite simply, the bad news is that Survive the Wild is going to remain offline for the next couple/few weeks at the very least. The good news is that the next time Survive the Wild comes back online, you all will be playing the public beta!

Every long time player knows it at least to some extent, so I’m just going to admit it here. The public version of Survive the Wild has been a bit of a mess now for months for several reasons. First, development has been almost exclusively focused on the next game version for a while. This has been for several reasons. First, the client updater for the public version is at least somewhat unstable unfortunately. Every time we try to release an incremental update, we end up having to distribute patches to a bunch of people who are unable to download it. Creating patches that then fail to download for some people and can sometimes break themselves eventually started to seem redundant when we could just fix all of the bugs properly for the next version during a well planned public beta phase if I haven’t fixed them already while coding the update. Furthermore given the current updating workflow, backported code from the new version to the public one ends up usually being rather untested, as it gets piped directly to the public server after the porting. Again, we keep running into the fundamental issue that, for example, spinning up a tertiary beta server and calling up the entire beta team to test bugfixes for the public version which we already intend to replace with the update ASAP seems like spending time that could otherwise be used to get this update out faster. In short, the public game version is a nightmare to update incrementally at the present time.

At least partially because of this initial failure to implement a proper incremental updating system (which I profusely apologize for and promise to improve), the public version of the game has also unfortunately become increasingly difficult to run particularly while trying to develop an update on the scale that we’re trying for. As has been mentioned in previous announcements on the game, the staff ranking system on the public server is very underdeveloped and does not allow us to promote anything other than completely trusted help. Aside from that though, we don’t want to train new staff on two versions of the game at a time. We think it would be a very stressful endeavor for new staff members to learn a bunch of legacy administration commands while also learning the newer systems on the upcoming game version at the same time as they’re trying to help players on the public game. It’s just not viable, and as such, the same couple of long time admins have been putting every bit of spare energy they have into helping as many players on the public game as they can. However, me and those same long time admins are the only people who are able to train new staff and who are able to make some of the larger executive decisions needed to move this update forward. They are not able to spend as much time as needed doing that while trying to help players on the public game as well, particularly as new bugs continue to be found and old ones continue to cause an increasing pile of bugged deaths, dfc losses, and other negative experiences that need tending to.

All of this and more has lead to a less than optimal experience to say the least both for players and for staff. As the number of minor incidents continued to increase on the game, me and the staff have been questioning whether to make the decision to temporarily shut down the public server for a couple of months now so that we could all reset and exclusively focus on pounding out this update as fast as we can. Despite the increasing stress of running the game in this state though, despite me saying that I understand if they need a break I might add, my amazing admins have continued to tough it out over these last couple of months to the absolute best of their ability, and I give them huge props for that. Now, though, less than a week after taking a break from renting an apartment for a while thus freeing me of much financial stress, the drive that the game was running on has crashed, taking the entire server operating system down with it. After discussing it with the team, we decided that it was time to use this rather unfortunate crash combined with the lack in financial stress as an opportunity to get this update done, and to get it done right. With the public server down, we’ll have as much time as we need to convert the character and account data from the public server to the new format used by the upcoming version, and test for any errors in the conversions. The long time admins will be able to spend more time directing the worldmakers and answering their questions, gathering and training more moderators on the new systems, and generally just being around to help with the update where ever they might do so, without worrying that a small crisis might pop up on the public game at any time what so ever to distract them from their work. This also gives me more time to try to recover our lost data, which would mean that we wouldn’t have to revert to a backup. I don’t expect this to be successful, but taking Survive the Wild offline for a bit at least gives me more time to try. There are more reasons for this decision as well. A surgery I had performed last year has sadly been slowly undoing itself, and this drive crash happened to take place days away from the re-op. Our new server machine runs linux and not windows, so the old bgt server application would have to be virtualized through wine or a VM, thus likely slowing it down. While repurposing the old server would probably be an option, it would mean replacing the broken drive, and probably the loss of any remaining chance, slim though it may be, that we’ll ever see our missing data again. Dealing with hectic server migration, reconfiguration and virtualization plus maybe a client update to point to the new server (not sure if that’s needed or not because of an anticheat measure) plus maybe people’s relays no longer pointing to the new location plus several distressed players who want their 3 weeks of game play back plus who knows what else directly following surgery is probably not the healthiest idea in the world. While the last surgical recovery wasn’t very painful or draining, the operation failed in the long-term and so a new method is being tried this time, where I have been warned that the recovery process may be more intense than the last one. For all these reasons and more, we’ve decided that though it’s disappointing in the short-term, the most wise decision in this case is to shut down the public game, finish pounding out this update, and come back online in public beta mode when the update is ready for testers. Again me and the staff genuinely apologize for any disappointment this causes, and hope we can make some of that up to you with the contents of the upcoming update.

There are several ways to get updates on this situation, particularly since we can give no concrete dates or times in this post. If you want to directly communicate with us, the best way to reach any administrator is currently through the game’s discord server, where significant announcements also always get posted when something happens. I will also post on this blog, so feel free to check here or use the rss feed to get more instant updates. Though I can’t promise given the current state of twitter, an announcement may be posted there as well, if nothing else when public beta goes live. Finally, you can try to keep launching your Survive the Wild clients every couple of days if all else fails. At the time of writing this post there is simply some sort of connection error, but I will try to get that replaced with an easily updated status message within the next couple of days so that people can also get a few notifications that way. If you happen to be a member of this, though the skype group is considered completely inactive, we will most likely post there when the public beta goes live as well for good measure. More slowly, you will see updates propagate to the audiogames.net forum. Just keep a close eye out, and you’ll likely find new information on the situation quickly, if nothing else you can simply feel free to bookmark this blog post and check back every few days for updates (denoted by headings) if needed. Thank you all for your continuing patients and support during this unfortunate time, we look forward to seeing you at the launch of the public beta!

Missing games and other website content

As to my smaller games, I will make sure they get uploaded to the games page on this new website over the next couple weeks. If you’re seeing this post soon after it was published, no doubt you’re seeing quite an empty website. This page should have been made public several weeks from now, instead it’s being rushed. Feel free to check the games page every few days for more updates.

I’m not completely sure when the Ah Damn It scoreboard will become active again, it depends if I can easily find the related php scripts. I don’t know how many people still play that game and generally I must spend most of my time focusing on Survive the Wild.

All of my old BGT code that used to be public on my development page will be uploaded to my Github also over the next couple of weeks.

Any Audio Network and Awesome Tech archives

I was able to image the disk that contains Any Audio’s data. Well, most of it. What I think we may have lost is the most recent version of Any Audio’s sql database. I do have a version of it, but it appears to be a few months older than when the website became frozen. I’m not sure how much metadata I’ll be able to restore from clips newer than mid November 2018, of course I will do my best though and of course apologize for anybody who may need to reupload clips some day. As with other sections in this post, I’ll update this paragraph if I end up finding a more recent version of the site’s database.

I would unfortunately expect Any Audio’s archive to remain unavailable for the next few weeks at least, again because my main focus must remain much more on Survive the Wild’s upcoming update than anything else. Basically, I’ve intended to rewrite Any Audio in python for a couple of years now, but have never gotten around to it. It’s honestly a side project that I’m very excited to work on though when I have time. In fact, I may do it as soon as the next STW update is released and is stable enough for me to decrease my focus on it for a few weeks. The new version of Any Audio will include post tagging, better moderation features, multiple file formats, a working password reset form, view filters, an API and much more. At the time the new version of Any Audio comes online, as much of the old archive as possible will be available under an archived category, the Awesome Tech archives will be available there as well.

If there is huge demand for the Awesome Tech archives before Any Audio comes back, I may provide those as a zip file that can be downloaded, basically a bunch of mp3 files and a feed.xml file with their metadata. I’m guessing though that this archive will simply become available when Any Audio returns.

Final notes and thoughts

Well, in general, talk about an unpleasant and entirely unexpected mess! I want to generally apologize to anyone who was at all effected by this outage in any way, be it by some lost stw game play or a deleted any audio clip, or even a prematurely cleared game score. I want to assure everyone that I am going to learn from this situation and be more prudent about backups in the future, particularly in regards to data that I did not create and which I have been entrusted with keeping safe. By losing user data I have failed to handle it properly, and for that I am genuinely sorry and seriously intend to do better in the future. Whatever happens, next time my server’s disk drive fails, I’ll be much more ready to handle it and it won’t effect users of my services nearly as much if I have any say in the matter. Thank you again everybody for your patients, kind support and well wishes during this time.

And with that, lets see if we can’t manage to make the third post on this here blog a bit more of a positive one, shall we? Because I think this is a really bad start to a blogging hobby! 😀

49 Comments

  • DJWolfy says:

    I believe that some of your old games are on the audiogames archive, for anyone who wants to play them

  • Draq says:

    Hey there Sam. I hope all goes well with the surgery. Take as much time as you need to recover and try not to push things too hard.

  • Literary says:

    Best of luck, man!

  • Pedro Dias says:

    Hello, Sam. I hope you recover very quickly. Try not to work too hard even if the world is falling apart around you. The stw is the better game that i play in years and i have sure that you will recover it. If I can make someting to help, same that be oferting my hopes, there are it so. Take care and good luck. I hope see you getting success shortly.

  • aakash says:

    take care. and your health is importent not the game.

  • Ordrin says:

    Hi!
    I recommend looking at the “duplicity” project, designed for creating backups.
    I’ve been using it on my server for a few years and I’m happy with the functionality and ease of use.
    This thing can create incremental backups, automatically delete old copies, reuse blocks for duplicate data, which saves space.

    P.S.
    This is not an advertisement. The utility is really cool! 😉

    • Sam says:

      Hey thanks, I actually decided to look at this thing because of your comment and it actually seems really cool. I may indeed use this for general backups.

  • Underworld says:

    Hey Sam, I wish you speedy recovery. As much as the game is important to us, your health is more important so do not try to push too hard.
    I have something to say though about your Anticheat system in the old Survive the Wild and I think you should take some time to look at fixing it in the upcoming beta. Lately a few players were banned by your admins as the system detected that their keyboard press is not human. While there are some cases of using scripts, there are also genuine cases of people who just happen to use mechanical keyboard and perform fast physical key press which were flagged by your anticheat system. No doubt, some people enjoy spamming their keyboards for long jumps, running, or just spamming the t key while on the boat.
    When this happen, there is no way a player can dispute that they do not use any scripts as alleged, as the admins only rely on what’s shown in the cheat system. I would always advise those who were caught in this situation to just admit and accept whatever punishment given out, as verbal admission against your cheat detection cannot hold any solid evidence for the accused.
    I do not intend to speak out for you to unbanned those who were wrongly accused, but for you to maybe improve the anticheat pertaining to key presses so that similar incidents could well be avoided. Another suggestion that you can consider is by limiting the speed of activities such that pressing the arrow keys very fast will not give any extra advantage like the ability to run at faster than usual speed. I think you have done this very well in the melee weapon, whereby spamming the space key will not cause weapons to swing faster.
    Please give this a good thought to prevent those who are genuine players from becoming victim of the game anticheat system. This beta is a good trial for you to unbanned players which have done minor mistakes before. Players involved in manipulating paid items, crashing the games etc should continue to be banned.

    • Seico. says:

      the system does not detect quick presses, but presses with the exact same time between pressing the keys, if the time is exactly the same it means that it is a machine doing it, a human cannot in any way press always in the same interval

      • Acer says:

        Most people think that, but that’s only 99% true. There was actually a musician from many years ago (can’t remember how long) that was so good at timing that he could literally hear 5 ms discrepinseas in his playing and resolv them. That just goes to show how rare, but possible this kind of thing is.

      • Seico says:

        I understand, but it’s one thing to notice the difference in time and quite another to adjust it with manual grips so that it’s always at the same time as you press the keys.

  • animal killer says:

    Hi Mr Sam. I hope to every thing will be ok.

  • aliiq says:

    if you are using linux it will be better, linux can take daly backup and send it to cloud services and save them, if anything happened you have backup for your data. that will be better more

  • Hamad says:

    Hi Sam, I’m sorry to hear about your recent server issues. It’s never easy to lose data and experience downtime, but it sounds like you’re taking the necessary steps to restore your website and services.

    As for your message to the community, it’s good that you’re being transparent about the current situation and giving them an update on the status of Survive the Wild. While the delay in the game’s availability may be disappointing to some, the good news about the game entering public beta is a positive spin.

    Here are some tips for improving your off-site backup strategy in the future:

    Schedule frequent backups to minimize data loss.
    Store backups in multiple locations, such as cloud storage or off-site physical storage.
    Test backups regularly to ensure they can be restored if necessary.

    Good luck!

  • hi sam,i hope everything is going well an,hoping Survive the wild will become soon

  • Dallas says:

    hey Sam I hope you do well with the update I will try to find a new game to play from you’re website. I love this game so much and wish you the best. From Dallas.

  • golden warrior says:

    hey man. wishing you all the best. and take the time you need to recover.

  • luara daneluz ribeiro says:

    oi Sam,muito obrigada por avisar e estarei orando para que corra tudo certo na cirurgia.
    Estou achando muita falta do survive, mais a sua saúde sempre em primeiro lugar.

  • Legendary says:

    hello sam, this is me tenzin, my real name and i use as legendary in game from bhutan. stw was the first online i ever played and belief me, it was my favourite game i ever played. i was completely shocked and disappointed wehn i heard about the server issue you encountered. anyway, as saying goes, “health is wealth”, please do recover your health first and then work slowly on the game part. i know, we are really missing stw, yet, as someone said earlier, health is more important than a game. wish you all the best sam. may you bless with utmost happiness and success in any career you move into.

  • Kevin Roberts says:

    Hey sam,
    I am sorry to here about your server issues I myself as a linux user know how emportant it is to keep backups as much as I can. I hope you can get your stuff back up and running as soon as you can and as far as stw looking forward to seeing the new game when it comes in to public beta.
    Thanks
    Kevin Roberts

  • Lauren says:

    Thank you for the update and hope to see it out in the next month!

  • mehdi-t! says:

    hello, thank you all for this great website, and your great game survive the wild! i am waiting for the stw update very much, everyday i listen to the audio demo about the stw which was posted on july. take care, your health is much more important than the game! again, thanks!

  • Mister Santu says:

    hello sam I am very sorry to hear your heard drive is bad and server down. This game has become very dear to me, we are in no hurry. You keep trying to bring new beta update gradually. And take care of your health. I am Mister Santu.. thank you Sam

  • ahmed says:

    hi there
    good day sam
    i was know about this issu whin hard disk is failed
    please try to take care of yourself ferst,and after that try to fix this problem
    i wish you quik healing

  • Pedro Dias says:

    Hey guys! I can’t keep up with the Discord channel so does anyone know if there’s an update on the health of samtupy and your developments?

  • Bravo says:

    Hello Sam!
    Feeling sad to hear that, It will take a bit of long time to come back.
    But hoping that, when It will come up.
    There will no technical issue that I have faced in stw several times.
    And if possible, try to add system to see our death.
    So no one will claim without checking that, his death was unfair.
    And must focus on the previous data.
    As, no one lose their stuffs and character type.
    Waiting for getting it back.
    Be success.

  • Sam says:

    Thank you everybody for your kind comments! I am now out of surgery and recovering. As predicted above this recovery isn’t nearly as easy as the first surgical attempt a year ago, but that being said the recovery is going exactly as expected and no issues are currently foreseen. Hopefully in a few days I’ll be my old self again more or less completely! As for backups, thank you all for the suggestions! Indeed we had been making on-site backups of STW every day or 2, but our issue was that we only transferred the latest one off-site every couple of weeks. Of course this entire incident has insured that by the time public beta comes back, we’ll have a system in place to push daily backups to multiple remote locations at the same time, so that nothing short of a cascading multi-continental computer failure or something else just as unlikely or unexpected can result in the loss of any more of our data. I’ll provide more official updates in the next days of course. Thanks all!

    • Pedro Dias says:

      Thank you @Samtupy for the little update by commentary. Did me and a lot of other people more calm about your healph. I continui hopping that you be better soon and go back work in this project. The STW isn’t game only more, it’s part of our life and every day that stay off, miss a part of our day.
      Maybe you will like know that yesterday afternoon, in stw’s whatsapp group Many people were talking how the STW helpe us make friends all arrownd of world and how it was great in our life. Everyone is hopping that you be better soon. Send noticies, please and: continui healling nice how you’re doing. Take care bro!

  • rishi says:

    hi sam.really unfortunate to see stw down. as much as i know it is the longest break in the stw journey. but hope it to be running soon soon. nothing is above health. so be fine first, then you will have enough time to do stuffs. stay blessed and hope this boring time will pass soon. considering myself regular stw player i can wait with patience. stay blessed!

  • Ramon says:

    it’s so nice when life gives us so many coincidences at the same time in our life to help.

  • amir says:

    hi sam, even tho we have lost so meny things if you get back the thing to the 11th, but still, take care and i wish you the best.

  • Samuel04 says:

    Hey Sam, glad you had a good recovery and looking forward to a great new public beta when it comes out.

  • golden_secret says:

    Hello Sam, I really hope that your recovery all went well and I hope that you are doing okay now. I know how hard surgery can be both physically and mentally, so please take care of yourself and I look forward to seeing the game when it comes back online.

  • abolfazlghasemi says:

    Hello, I’m very happy that you came out of surgery healthy, I hope you’re happy and can always fight with problems. Meanwhile, we’re all impatiently waiting for the game to be updated! Your health is important first! Good luck!

  • VanilaIce says:

    hi sam.
    I wish you to return to normal and continue working on the stw part, and may stw continue to be a great game all over the world.

  • rashid says:

    I hope stw will back soon,

  • night-wolf says:

    Hello Sam, I wish this game will come back cause I am honestly bored and have nothing to do 😀 and i am very excited for exploring the beta as well as chatting with people

  • alano says:

    i hope stw come back soon I haven’t played it for a long time, I miss it

  • Nico Alfaro says:

    hola estimado sam! pronta recuperación espero que todo salga de lo mejor amigo! de argentina te saludo

  • ddt.one says:

    yes. I whaiting your games.

  • Dawabsheh says:

    Hello, I hope you are well,
    and I hope the game come back as soon as possible,
    actually I miss the game so much,
    You have to take care about you’re self
    then you should to take care also for the game, there is a lot of players waiting you’re game to go up again, I hope that actually, I hope to not be like what’s happened for Redspot
    thank you a lot and good luck.

    • jim pickens says:

      yeah, that would be really sad

    • Sam says:

      I promise that STW will not go in the direction of Redspot. The only thing that is going to stop stw from coming back online is some sort of terrible tragedy, which lord willing will not befall me or us. STW development continues, and today I will write another update post describing the latest.

      • Dawabsheh says:

        This is very great, I hope you are well, and that the game is back as it was before, it works without problems, thank you very much.

  • Tuxo says:

    Until this moment I’m waiting for any new explanation that reveals news about the game, but we only have promises and the hope that by 2045 or so something will change in this obscure scenario where all the players who paid for the catastrophe are.

  • Tuxo says:

    I recognize that bad things happen, but being in limbo without any update, or plausible explanation, is even a lack of respect for those who have always been there…

    • Sam says:

      Hello, I just want to make sure that you and anybody else reading this knows that this blog contains more posts after this one which describe the development of the game in as much detail as possible. If you did not know about them, can you please describe how you got here so that I can provide more up-to-date information in the linking source? If you have read them, I apologize that they do not provide enough information for you and can only promise that I am working every day on the next version of the game and that it will come online. No credits are lost, this I can assure you. Anyway I just wanted to make sure you know about the other posts on this blog, everything else I can say about when the game will be back has been repeated by me time and again in the blog comments, posts, and on the discord. I genuinely apologize for the wait, it’s just as frustrating for me and the staff as it is for you.

  • dawood tiyes says:

    hi sam how are you?i can knot wate for the game.i am bord but, how is your helth?i olwase checking when it wil cum it is that which make me to have so many friends. and i giv you many thanks for making that game,and take your time

  • jim pickens says:

    for anyone who didn’t see this timer is now live https://samtupy.com/blog/2023/05/stw-early-may-update/

Leave a Reply to Ordrin Cancel reply

Your email address will not be published. Required fields are marked *