Major Upgrades & Maintenance - Week of July 14th

[Matt K]

Hi everyone,

Apologies for being out of the loop for a while. Good news is that I have another little one in the house! I should be available again in the coming weeks.

At any rate, Neobri and I have been working on planning out how to make sure this site is stable, and we're executing on it as I type.

Beginning tomorrow at ~9:15AM EST, I'll be taking the site down briefly for several upgrades; we are a few minor releases behind, and bugfixes / new features have been added that are intended to ameliorate some of the instability the site has been experiencing.

Additionally, I have a major OS upgrade that needs to take place, so I'll be migrating to a fresh OS as well. If I can complete this as well tomorrow, I will do so. Otherwise, I will similarly take the site down around 9:15 a.m. on Wednesday and perform the upgrade separately.

I understand people, particularly coming from behind CGNAT (and particularly on cell phones), have been sporadically banned - this is a byproduct of some excessive scraping that has been occurring from LLMs. I'm working on a workaround for this as well, though I don't anticipate needing any downtime for these fixes. I will announce them if I decide they are necessary.

-Matt

[Geordie]

Thanks for all you do. Congratulations on your new arrival.

[Posaunus]

Thanks Matt and Brian - can't tell you how much we appreciate your keeping up with the challenges and maintaining / improving the TromboneChat site.

[claf]

Congratulations Matt for your kid!

[jpwell]

Thank you very much for all the updates

[Oslide]

Big thanks for all your efforts, and all my best wishes to you and your family!

[mgladdish]

Thanks Matt and Congratulations!

Do say if I can be of any help, even if it's just another techie to take a look if something goes bang during UK hours. And I'd like to help spread the risk a little - life always gets in the way and if we could have a few more people who can help run the place when you're indisposed that would be even better.

[WGWTR180]

Congratulations at home and Thank You!

[Matt K]

All righty, upgrade was successful. We're now on the most up-to-date version of phpBB, latte and prosilver themes have been updated, and I've made a few tweaks to the authentication that may help with the banning issue.

I won't be able to finish the server upgrade today, but will begin making arrangements - these will no affect the usage of the site today, but plan on the site being down tomorrow around 9:15AM for the transition. It may actually go fairly quick as I'll work on transferring the site today and just do diffs tomorrow AM.

[AtomicClock]

Is that why the TromboneChat logo has been replaced with an (admittedly attractive) phpBB logo?

[Kbiggs]

Thank you Matt and Neobri. I appreciate the work you guys do, and I recognize that you’re volunteers, so it’s a labor of love.

[Matt K]

Is that why the TromboneChat logo has been replaced with an (admittedly attractive) phpBB logo?

Ah I didn't even notice that! Yes, when you upgrade styles you have to replace all the assets. I have the old one I'll put it back momentarily!

[atopper333]

Thanks again Matt, your work is really appreciated! Congratulations and hope all is well.

[Matt K]

I'm going to keep the site up for a little while longer as I do some other things to prep in the background. I will put the site in maintenance mode soon though to prep for migration to another server. Stay tuned!

[Matt K]

Test

[BGuttman]

Thanks for all you are doing, Matt,

[Matt K]

I made some significant progress, but I was unable to finish today. However, things are now in what appears to be a stable state. I believe I can now continue most of the work without outages, but I will announce them in advance if that changes! I would expect this to be done around EOW.

[tim]

Congrats on your new little one! Hope all are doing well.

[harrisonreed]

Congratulations Matt!

I noticed that I don't get an email and when someone private messages me now. I missed one from someone important on the forum and feel bad about responding late to them. Maybe it was just a one time fluke.

Thanks for what you do, Matt!

[Matt K]

Email may be tricky to triage, but I can look into it. It's becoming a massively unreliable mechanism for notifications, regrettably. There's so much spam and all it takes is a few people to hit "mark as spam" from our domain for the big players to start to filter out messages. Adding trombonechat.com in your allow list (or as a contact) is likely to help.

[harrisonreed]

I understand. It's not in my spam though. I would get email notifications for everything from the site. This is the first time I didn't get one. I'm not, like, upset or anything. Just putting out an experience.

The new site looks great!

[Stustang05]

Thank you and congratulations!

[Kevbach33]

Thank you for your hard work, Matt. A minor inconvenience to get in is nothing compared to the big downtimes before this.

And congratulations on your little one!

[Matt K]

I understand. It's not in my spam though. I would get email notifications for everything from the site. This is the first time I didn't get one. I'm not, like, upset or anything. Just putting out an experience.

The new site looks great!

Sometimes what happens is the provider you have decides it isn’t even worth receiving the mail and categorically rejects from a domain. There’s a ton of consternation about this in the “self hosted” community about how it’s become virtually impossible to do email because of the big players rejecting stuff that isn’t from them selves. We don’t self host it, but SMTP still is frequently rejected. We really don’t WANT to go with a big player like Google and Outlook if we can avoid it. Once I finish migrating I’ll look and see if I can get something more reliable than what we currently have though. When we first installed the site my recollection is phpBB didn’t support mailgun, SES, or mailchimp which are pretty common players in the “automated” mail space.

[claf]

And even those players would easily be rejected because "whatever".
I hate dealing with automated mails at my job for that reason.

[sf105]

Thanks for all your work.

For email, I've started using maibox.org (based in Germany).

[WGWTR180]

I understand. It's not in my spam though. I would get email notifications for everything from the site. This is the first time I didn't get one. I'm not, like, upset or anything. Just putting out an experience.

The new site looks great!

Sometimes what happens is the provider you have decides it isn’t even worth receiving the mail and categorically rejects from a domain. There’s a ton of consternation about this in the “self hosted” community about how it’s become virtually impossible to do email because of the big players rejecting stuff that isn’t from them selves. We don’t self host it, but SMTP still is frequently rejected. We really don’t WANT to go with a big player like Google and Outlook if we can avoid it. Once I finish migrating I’ll look and see if I can get something more reliable than what we currently have though. When we first installed the site my recollection is phpBB didn’t support mailgun, SES, or mailchimp which are pretty common players in the “automated” mail space.

Morning Matt. Thanks for all of the work on The Chat! Question: I just tried to respond to a message from a member and I receive a message that states I am a Spammer and kicks me off the site. Any thoughts? Thanks.

[Matt K]

Can you send me a screenshot? You aren’t in the special spammer group, so you’re probably being caught by cloud flare. Phone systems and popular ISPs seem to be particularly susceptible, so try seem to put additional scrutiny on people on those systems

[WGWTR180]

Here ya go:

[Matt K]

That issue should be fixed. It appears that one of the moderators (there's no log of which one of us) manually banned someone via IP instead of emai/username, which is typically what we do. I can lookup people based on their associated IPs and there were approximately 10 of you affected by that. You all appear to have a rotating IP, which explains the inconsistency.

LMK if anyone is still receiving the ban messages; between the fixes I did in cloudflare and this newest fix that issue should be fully resolved.

[Matt K]

Okay apologies for short notice, but I'll have some downtime today, which should be short. I want to make sure I have a good, cold backup of the site so I don't miss anything before doing the cutoff.

Could be either 6:30 EST (yes 15 minutes from now ) or 9:30EST.

[Matt K]

Bringing the site down momentarly. With luck I'll have it fully up and running and migrated shortly. Keep your arms and legs inside the ride at all times!

[Matt K]

Getting closer! I wasn't fully able to do the transition. I almost have it running on K8s (Kubernetes), which should be substantially better than what we had before, but I'm running into a few snags on getting the database to sync up from within a container. I won't be able to finish today, so I'm bringing the site back up now. Tomorrow should *actually* be a brief outage once I get this figured out. Will keep updated.

[Posaunus]

[claf]

Wow, TC on K8S?
Almost feel at my job

[PhilG]

Matt, thanks for all your hard work on this. And congratulations on your new little one!

[Matt K]

Fingers crossed, one more outage today at 9:30 AM EST. I was unfortunately not able to get it running in K8s (:sad trombone:), but I have a test site fully working more closely to what we had but with some tweaks in between the site itself and the big bad internet, so I'll do the transition today. New machine has around twice the resources of the old one and is on an LTS OS that is supported until 2029 and will still allow some of the ideas I had to have things like RSS feeds available again and controlled access for LLM scraping, so chatgpt can have all the hottest news of all things trombone fresh off the press.

[Matt K]

Testing 123

[Matt K]

Another test

[Posaunus]

Success!

Thanks, Matt, for all your diligent effort. Keep up the good work.

[Matt K]

Okay everything looks good.

For our EU users, you should notice a reduction in latency. For US based folks... you might notice a little bit more than you were used to, although I've been using it now for the past hour without much problem. You should also notice that the site will be slightly slow after a restart and should get progressively faster. I have a redis cache turned on now, so anything that has NOT been viewed by someone will be slightly slower to retrieve, but subsequent views should be nearly instant.

This server has around twice the resources of the previous one, so we are unlikely to be running into performance issues of the box itself.

Report any issues you have here and I'll see what I can do to address them.

I WILL need about 15 minutes of downtime sometime this week for a cold backup. I will annonce that later.

[EriKon]

Performance in Germany is a lot worse since a few hours with some short outages. Thought it's my device but pretty sure it is not. Site takes quite a bit longer to load.

[Matt K]

Performance in Germany is a lot worse since a few hours with some short outages. Thought it's my device but pretty sure it is not. Site takes quite a bit longer to load.

That makes sense, the site was migrating until about an hour ago. (It finished right about the time I posted the message). If it KEEPS being slow that could be an issue I can investigate.

The way ids setup now, it will be slightly slower the first one a page is visited, but subsequent access should be faster

[RobL]

I'm very grateful to those who provide this gift to the worldwide trombone community!

[Posaunus]

Still slow to respond here, but tolerable. Just not as quick and convenient as it once was.

[JohnL]

It seems that opening individual topics has some delay, while opening other pages is pretty quick.

[Matt K]

Thanks for the reports. Can you all let me know if you're on mobile or pc/laptop? I'm working on profiling it... I THINK it may be related to Cloudflare. I have the screws turned down pretty hard on it and I'd like to pull some off but want to make sure I do so in a way that doesn't give us the problems we were having earlier this summer.

[JohnL]

Ancient Linux-converted Chromebook laptop normally, but I just got the same behavior on a Windows machine running a freshly updated Chrome browser.

Example: "Active Topics" pops up right away, but if I click on the "Transcription Apps", it takes a bit to load. It's not a really long delay, but it's noticeable. It's like the database query for posts with that topic number is bogging down.

[Posaunus]

Thanks for the reports. Can you all let me know if you're on mobile or pc/laptop?

Desktop, laptop, mobile phone - all experiencing slow loading of pages.

[mrdeacon]

I might be going crazy but I swear that the forum is back to normal fast speed this morning. I’m on mobile.