Don't panic if your service breaks over night

budding

planted Jun 11, 2021

If something is broken: Keep calm and analyze, before trying any quick fixes that break even more.

Your internet is gone, your server is broken, your PhotoPrism, Nginx, or Nextcloud crashes. It may happen from time to time that one or all of your service do not work as expected.

Keep calm and analyze, before trying any quick fixes from Google!

The worst thing you can do, is copy and paste any random command you don't even know the function of. You easily destroy more, than what was broken in the first place. Maybe the issue wasn't even on your end, but now it is.

This has happened to me more than once, and it is super annoying.

Try to remain calm and find out if the problem is your software, or something external, that you depend on.

Follow these steps as a rule of thumb:

1. Is your internet working? Reboot your router and whatever device is not working correctly
2. If it is a self-hosted service, that is not accessible from the outside anymore: how about from the inside? Can you still access it via its local IP? If so, does anything seem unusual from there?
   • If you are using and external service to map your IPv6 or something similar, consider waiting a few days. It already happened to me that it worked again later. Maybe set up the HTTPS-Proxy again (and if that fails, also wait. It worked again later).
   • If it doesn't go to the next step
3. Try to carefully find the problem. Only read logs and find out how to access the logs. Do not try any quick fixes before you have an idea what the issue is.
4. Maybe you have a recent backup to revert to? Consider using it before fiddling around. It is much less of a pain in the ass and getting the data from a few days back might be much easier.
5. Prefer steps that just install a missing dependency over fiddling with some configs. This is much less fatal.
6. Always note, what you changed in the configs and make a backup file! Oh, how often I forget this and do not know what I have done if it worked (and I would like to know what fixed it) or it didn't work, and I don't know all the steps to revert…

With these steps in mind you will hopefully find the issue without breaking even more. If you found the problem, it is generally a good idea to note the problem somewhere. This way you can build a checklist of your common issues and hopefully can fix your service much quicker next time.