I have a process which copies emojos from the hajkey blahaj.zone site to our lemmy.
Last night (for me) it copied an invalid emojo into the database and it broke lemmy for hours while Ada and I were sleeping.
I've found the issue and fixed it now, so everything should be purring along once again, but for those who were left without the instance to use, I sincerely apologise.
No worries, things happen — and you’re community-building volunteers, not an on-call rotation. 💜 I’m just glad we’re back now! Thanks for your efforts during these past 24 hours. 🏳️⚧️😊
30
Paige (she/her) - 2.5yr
I'm going to imagine that :capybara_animated: was chewing on the server racks
It wasn't an actual emojo. The script processed the SQL header column names as an emojo and tried to add them. Unfortunately publicUrl is not a valid URL, so lemmy's /api/v3/site metadata endpoint started returning an error relative URL without a base instead of the JSON that the website was expecting and so the site just stopped working for everyone the next time it tried to load that url.
28
bamboo - 2.5yr
This coincided with me resetting my password, and I thought I brought down the site somehow
Excellent news! The site loads perfectly for me. I am still unable to log in for some reason, but that could just be a me thing.
8
Noodles - 2.5yr
clear your cookies/cache for this site, it should work after that
8
Krrygon - 2.5yr
It works! I am back in the blahaj zone. Feels good.
9
LuckingFurker (Any/All) - 2.5yr
Thank you for all you do, dear admins ❤️
5
Wirlocke - 2.5yr
I was worried that it got hacked again, since that just happened. Glad that Blahaj lives again!
5
stercar - 2.5yr
So glad it‘s up again. I was thinking it was something like that since it kept giving a parsing error when I was trying to log in. Good that it‘s fixed now.
5
djstini - 2.5yr
{"data":{"msg":"Couln't upload file, Error in DB, IO error: Os { code: 28, kind: StorageFull, message: "No space left on device" }","files":null},"state":"success"}
Hmm, still getting errors, for example when trying to upload profile images
4
Kaity A - 2.5yr
Should be fixed now.
2
essi - 2.5yr
It happens! Thanks for the writeup, and good work bringing things back online again :)
4
Rozaŭtuno - 2.5yr
Mistakes happen, I'm just happy the instance is back up.
4
alx - 2.5yr
Welcome back to the Friends From Sweden and thank you to the admins
4
Lotus Eater - 2.5yr
Appreciate the fix, but I am having trouble uploading images, wonder if anyone else is having them?
4
Kaity A - 2.5yr
Should be fixed now
2
Moss - 2.5yr
I can't log into the website now. It tells me I logged in successfully, then simply doesn't log me in.
Jerboa still works fine tho
2
Xtallll - 2.5yr
After the Hack they changed the secret key, you have to clear the cookies for Lemmy.Blahja.zone and re-login.
2
Moss - 2.5yr
I already did that it's not working
1
Jon - 2.5yr
What browser are you using? I have that issue in Firefox, but not in Chrome.
supakaity in main
Broken emojo caused site outage,
I have a process which copies emojos from the hajkey blahaj.zone site to our lemmy.
Last night (for me) it copied an invalid emojo into the database and it broke lemmy for hours while Ada and I were sleeping.
I've found the issue and fixed it now, so everything should be purring along once again, but for those who were left without the instance to use, I sincerely apologise.
https://blahaj.zone/files/e106824f-b5e7-4f5f-b506-909632fd79e0
No worries, things happen — and you’re community-building volunteers, not an on-call rotation. 💜 I’m just glad we’re back now! Thanks for your efforts during these past 24 hours. 🏳️⚧️😊
I'm going to imagine that :capybara_animated: was chewing on the server racks
https://blahaj.zone/files/9723acde-4835-47fc-a8e5-64f7305f0227
That checks out
https://blahaj.zone/files/9723acde-4835-47fc-a8e5-64f7305f0227 thanks for the update! what emojo was it
It wasn't an actual emojo. The script processed the SQL header column names as an emojo and tried to add them. Unfortunately
publicUrlis not a valid URL, so lemmy's/api/v3/sitemetadata endpoint started returning an errorrelative URL without a baseinstead of the JSON that the website was expecting and so the site just stopped working for everyone the next time it tried to load that url.This coincided with me resetting my password, and I thought I brought down the site somehow
happy the best instance is back up!! https://blahaj.zone/files/431442e0-bcac-4058-907c-983866239c9f
Glad to be back and thanks for your hard work!
Excellent news! The site loads perfectly for me. I am still unable to log in for some reason, but that could just be a me thing.
clear your cookies/cache for this site, it should work after that
It works! I am back in the blahaj zone. Feels good.
Thank you for all you do, dear admins ❤️
I was worried that it got hacked again, since that just happened. Glad that Blahaj lives again!
So glad it‘s up again. I was thinking it was something like that since it kept giving a parsing error when I was trying to log in. Good that it‘s fixed now.
{"data":{"msg":"Couln't upload file, Error in DB, IO error: Os { code: 28, kind: StorageFull, message: "No space left on device" }","files":null},"state":"success"}
Hmm, still getting errors, for example when trying to upload profile images
Should be fixed now.
It happens! Thanks for the writeup, and good work bringing things back online again :)
Mistakes happen, I'm just happy the instance is back up.
Welcome back to the Friends From Sweden and thank you to the admins
Appreciate the fix, but I am having trouble uploading images, wonder if anyone else is having them?
Should be fixed now
I can't log into the website now. It tells me I logged in successfully, then simply doesn't log me in. Jerboa still works fine tho
After the Hack they changed the secret key, you have to clear the cookies for Lemmy.Blahja.zone and re-login.
I already did that it's not working
What browser are you using? I have that issue in Firefox, but not in Chrome.
Firefox yeah