Forum very unreliable lately

[CalgaryTbone]

Really bad today - tried to log on multiple times, and the first time that worked was after 10PM local time.

JS

[jabice]

I'm a software engineer and would be happy to help. phpBB is written in PHP and likely has a mySQL or SQLite database; these are somewhat adjacent to my specialties.

[paysonmcc]

I’ve noticed that when it is really bad the number of users will say that there is 12 registered users online, and up to 1,400+ unregistered. Check out the bottom of the home page- it looks like we’re getting a lot of artificial traffic recently. Is there a way to stop that? The only way I can think is to require an account to access the forum.

[dwcarder]

That's what I noticed too. In another group, this was attributed to bots scraping content and bringing the database to a crawl. I'd be happy to contribute to help get things stabilized.

[Posaunus]

MUCH better today.
Thanks Matt?

[Doug Elliott]

Matt is working on it and preparing to move it to a better service. He will probably check in today. I think he has a full time job plus a new kid, so maybe he just hasn't had time to deal with it.

[mgladdish]

It would be nice to be able to spread the load and have more than just Matt who can look after the board. Everyone has stuff that comes up from time to time and having such a useful resource as this dependent on the availability of just one person will inevitably lead to issues.

I'm more than willing to help, and I'm also in the UK which will give cover over more of the day.

[Trav1s]

I’ve noticed that when it is really bad the number of users will say that there is 12 registered users online, and up to 1,400+ unregistered. Check out the bottom of the home page- it looks like we’re getting a lot of artificial traffic recently. Is there a way to stop that? The only way I can think is to require an account to access the forum.

Bots scraping date to feed AI?

[Doug Elliott]

Who is online
In total there are 1094 users online :: 13 registered, 2 hidden and 1079 guests

Here are the listed Bots:

Ahrefs [Bot],
Amazon [Bot],
Bing [Bot],
Google [Bot],
Google Adsense [Bot],
Semrush [Bot]

[ghmerrill]

By comparison,

TubaForum.net: 249 users online, 6 registered, 243 guests
Euphonium-Tuba Forum: 341 users online, 2 members, 339 guests
Trumpet Herald: 496 users online, 4 registered, 492 guests.

I won't try to interpret those figures in any meaningful way. I suppose it's quite possible that there are just a LOT more trombonists in the world then euphers and tubists (I wouldn't be surprised), although Werden's forum lists the most ever online as 26,956 on April 17, 2024. Of course, many of those might be bots. Maybe there are just a lot more forum-watching trombonists. But the trumpet forum figures seems to suggest that something is a bit funny with this forums usage figures -- don't you think? It's "most users ever online" is 1797.

[BPBasso]

Who is online
In total there are 1094 users online :: 13 registered, 2 hidden and 1079 guests

Here are the listed Bots:

Ahrefs [Bot],
Amazon [Bot],
Bing [Bot],
Google [Bot],
Google Adsense [Bot],
Semrush [Bot]

Geez.

At the time of my post: "In total there are 1392 users online :: 14 registered, 3 hidden and 1375 guests (based on users active over the past 5 minutes)
Most users ever online was 2384 on Mon Apr 14, 2025 6:57 am"

Reminds me of the influencers and streamers who use bots to boost likes/views. I don't know how any of that works.. This forum doesn't have ads that I've seen. Is the forum being maliciously targeted? DDoS type of thing?

[ParLawGod]

By comparison,

TubaForum.net: 249 users online, 6 registered, 243 guests
Euphonium-Tuba Forum: 341 users online, 2 members, 339 guests
Trumpet Herald: 496 users online, 4 registered, 492 guests.

I won't try to interpret those figures in any meaningful way. I suppose it's quite possible that there are just a LOT more trombonists in the world then euphers and tubists (I wouldn't be surprised), although Werden's forum lists the most ever online as 26,956 on April 17, 2024. Of course, many of those might be bots. Maybe there are just a lot more forum-watching trombonists. But the trumpet forum figures seems to suggest that something is a bit funny with this forums usage figures -- don't you think? It's "most users ever online" is 1797.

We were having problems at TubaForum.net with DDoS attacks a while back (I do the tech. side of that board). I'm not privy to things here, but maybe something similar is happening when it lists over 1000 "users" online...the server is being overloaded. We upgraded our firewall to help with that and haven't had any issues since.

[ghmerrill]

We upgraded our firewall to help with that and haven't had any issues since.

I know that Dave Werden was having problems of similar scope and managed to add more effective bot protection. That assault nowadays is simply relentless. Some years ago I set up and managed a phpBB board for the community band I was in (along with an email server), and simply didn't need to worry about this problem. Now, it's a major factor.

As an aside ... that board I set up failed -- in the sense of not achieving "user acceptance". At that time the band was heavily weighted in the direction of people in their 60s and 70s who simply couldn't/wouldn't adapt to online technology (basically, wouldn't even try to because it was "too complicated"). Fast forward almost 10 years, and I'm sure that people in that age group are now almost 100% comfortable with the use of online environments -- largely because they now have to be in order to manage their medical care and appointments. Of course, now invasive bots are a problem they didn't used to be.

[Matt K]

A few people have identified the problem: there is an excessive amount of traffic that is nefarious in nature, almost certainly originating from LLM scraping. There is a reasonably good Hackernews thread on this:
https://news.ycombinator.com/item?id=43397361

It isn't unusual for us to see sustained 8MB/s outbound for an extensive period. A firewall would likely be helpful (if you wouldn't mind emailing me the details of what you've firewalled off, I would appreciate it). The concern I have with firewalls is that there is already a significant issue with reports of bans, and I don't want to exacerbate that problem for genuine users.

It appears that most of the bans are originating from individuals behind CGNAT, particularly when using AT&T, Verizon, and Comcast networks. (Humorously, I actually briefly banned myself when I logged in yesterday to post the announcement, as I am posting from behind Comcast CGNAT at the moment.)

I actually conducted a test yesterday with an LLM and found that asking a question that I had validated as not being found on other sites (Reddit, in particular) resulted in an immediate and significant spike in traffic that lasted for hours. We've added some Cloudflare mitigations, and I could really put the screws down on it too... You'll notice that with other phpBB sites, it's fairly common for EVERY SCREEN to have a recaptcha now (check out the odroid site, for example: https://forum.odroid.com/viewforum.php?f=175). That is a miserable experience that I would prefer to avoid.

And it's a real shame because I would personally prefer LLMs to be able to scrape our site, as long as it's done reasonably and respectfully. One of the main aims was to provide a corpus of knowledge about trombone, and LLMs are going to be the way people access that type of content... but non-stop scraping to merely validate nothing has changed is just excessive.

The concern about me disappearing is well-founded, although I should be back now. I had some personal stuff happen (as noted!), and I think everything is mostly winding down. I also had several people threaten me personally with lawsuits over moderation, and I really didn't want to deal with that with everything else I had going on. I probably would have had a more aggressive reaction had I not been dealing with all the personal issues. Perhaps it might be best for me to step back from moderating or take a back seat on that front.

At any rate, the reason I'm mentioning that is that there was some concern about the backup being lost and what would happen if I were to disappear. It's a valid concern, and setting up something where the backups would be publicly available (though with obvious redactions, such as excluding PMs and user data) is technically feasible; however, there might be some complications to it. And I am not a lawyer, so it's quite possible the concerns I have are not valid. In the event someone DOES want their posts deleted, what does that mean practically? If backups are posted, would we need to modify the backups retroactively? Does it depend on jurisdiction? Those are all questions that would need to be answered and actively monitored.

Though there may be workarounds or alternative solutions. I'm going to focus on hardening the site this week and will follow up with leadership later in the week/next week regarding those other items. Do note that we have a pretty rigorous backup process that both Neobri and I have access to, so if the server farm gets sucked into the earth, we can still recover and be at worst a few days out of sync. However, someone would need the technical know-how to restore the site, although it's pretty straightforward, so that if I were to go away, a suitable substitute should not be hard to find.

[Matt K]

Speaking of, in case you've been wondering what has happened since the last post, we've been undergoing a major DDoS attack that was peaking at about 38k requests per second. I've had to resort to that hammer I was referring to, which will put an annoying cloudflare popup much more frequently than I would like (possibly as much as every time you load a page...) but it seems to have stopped the issue.

I have some errands to run but am monitoring the situation. The steps I'm taking tomorrow should help, and I'll be investigating how I can filter out these as obviously we do not naturally have 38k+ visitors per second, as cool as we are.

[AndrewMeronek]

Great news!

I wish I had more relevant experience with online security systems (and could help, at least with advice), but alas although I have IT experience that's not in my background.

[eln]

I will admit to having asked trombone questions to LLMs recently, and watched them as they scoured the forum for answers.

[ghmerrill]

I will admit to having asked trombone questions to LLMs recently, and watched them as they scoured the forum for answers.

Ultimately, there is great benefit in AIs making the knowledge and instruction available on forums widely and quickly to all of us. We WANT the "right" AIs to do this. At the moment we just lack the model and mechanism to achieve these goals.

[Matt K]

FWIW I do have a plan to provide LLMs with accessible access. Actually, several... going to focus on shoring up making the site usable for HUMANS first!!! Then circle back. The sledgehammer approach would be a read-replica that only bots can use which would be a mirror of this site. It isn't trivial to set something like that up but it also isn't an enormous burden. I may be able to finesse the cloudflare controls I setup yesterday but given that they seem to be working, I'm reluctant to "fix" something that isn't broken

[ghmerrill]

FWIW I do have a plan to provide LLMs with accessible access.

If you do that, it might be worth writing it up and publishing it ... say in Software Practice and Experience. I notice that in June they published a paper on "Safeguarding Networks From Malicious Intrusions ...". SP&E seems to have drifted in a somewhat more academic direction than when I dealt with them several eons ago, but I still think they might go for good a application and case study -- and people would benefit from seeing it. I mean, the title still includes practice and experience.

[jabice]

Thanks for the hard work, Matt. I'll take the CF pop-ups over the hours-long outages!

[tbdana]

Thanks for the hard work, Matt. I'll take the CF pop-ups over the hours-long outages!

Ditto

[Matt K]

At good to hear! I suspect this will stop the “bans” that were happening. This is a bit of a double edged sword here— a few people have been unable to get past cloudflare which I’m working on (emailing me is the best if somehow you see this and cannot get in). I was seeing the logs that indeed phpBB was treating people on - for example - Verizon as a single user which is fine for even a few thousand organic users but starts to fall over when its tens and hundreds of thousands all requesting the same thing non stop

[robcat2075]

... we've been undergoing a major DDoS attack that was peaking at about 38k requests per second.

What does the attack get them?

i suppose it's an honor to be noticed but what does crowding out a trombone forum get them? It' s not a business that competes with anyone.

Is it really a general attack on every site that the host hosts?

[Matt K]

Definitely not seeing this on other sites in this provider. I have several of my own sites on the same host (actually, there's another site hosted on the same server), and it gets virtually no traffic.

There are really two somewhat plausible explanations based on the patterns, and both would have a similar pattern. They aren't merely going to "trombonechat.com" - the front page - the traffic is going to specific, and different pages. Otherwise, the redis queue would serve up the page to an anonymous user from the cache, and there wouldn't be an issue. Each one goes to a unique thread and specific page. Because of the way phpBB works, each of these requires a brief connection to retrieve the page from the database.

To me that means either: 1) someone knows this and is/was intentionally, maliciously trying to bring the site down (there have been some heated exchanges, though nothing really IMO that would ever cause me to want to do something like that but ) or 2) it's LLMs that essentially constantly ping every page they've ever scraped to not miss out on whatever hot takes we have here. As silly as the latter may be, I have seen numerous reports from other similar sites with even less content than we have here, about identical traffic patterns.

[Trav1s]

I work at major US bank as PM in IT and would agree that 2 is the likely option. As a younger engineer likes to say, the LLM is looking for the hot goss from the forum and is always on the quest to find it...

Thanks Matt for your work behind the scenes!

Like the Wizard of Oz, we know the important one is the man behind the curtain.

[ScottZigler]

Just want to put this here for Matt K:

I use an RSS reader (feedly.com) to read posts, and since 7/15 11:30A Eastern Time, there are no new posts coming over. I'm assuming it is seeing Feedly scraping and it looks just like an LLM scraping. There may be no fix, or maybe it's a long term fix (read-only copy for the bots) but just wanted to make you aware. Thanks for all you do to keep this running!

[Matt K]

Just want to put this here for Matt K:

I use an RSS reader (feedly.com) to read posts, and since 7/15 11:30A Eastern Time, there are no new posts coming over. I'm assuming it is seeing Feedly scraping and it looks just like an LLM scraping. There may be no fix, or maybe it's a long term fix (read-only copy for the bots) but just wanted to make you aware. Thanks for all you do to keep this running!

Thanks for the report, that is unfortunately a necessary side-effect of the filtering I had to put in place. I do have a few ideas for how I might re-enable it, or enable it in a different form once I get this up and running. I'm almost there... I have all of the site migrated, database cloning in place, etc.

I'm hoping that I'm dealing with the last migration bug as I type... CSS and stylings are not showing up for some reason now, but I have the database synchronizing properly now. If I can get that running today, I'll probably have a solution for RSS feeds shortly thereafter.

The way that I'm architecting the site now will allow substantially more control over access and rerouting. Even at the peak amount of requests we were getting, that actually isn't THAT much traffic in the grand scheme of things, so enabling it is really just a matter of elbow grease in fitting a square peg in a round hole and getting phpbb to do something it wasn't wholly designed for.

[ScottZigler]

Any updates on RSS feeds? Is this just a lost cause? Thanks for all you do!

Just want to put this here for Matt K:

I use an RSS reader (feedly.com) to read posts, and since 7/15 11:30A Eastern Time, there are no new posts coming over. I'm assuming it is seeing Feedly scraping and it looks just like an LLM scraping. There may be no fix, or maybe it's a long term fix (read-only copy for the bots) but just wanted to make you aware. Thanks for all you do to keep this running!

Thanks for the report, that is unfortunately a necessary side-effect of the filtering I had to put in place. I do have a few ideas for how I might re-enable it, or enable it in a different form once I get this up and running. I'm almost there... I have all of the site migrated, database cloning in place, etc.

I'm hoping that I'm dealing with the last migration bug as I type... CSS and stylings are not showing up for some reason now, but I have the database synchronizing properly now. If I can get that running today, I'll probably have a solution for RSS feeds shortly thereafter.

The way that I'm architecting the site now will allow substantially more control over access and rerouting. Even at the peak amount of requests we were getting, that actually isn't THAT much traffic in the grand scheme of things, so enabling it is really just a matter of elbow grease in fitting a square peg in a round hole and getting phpbb to do something it wasn't wholly designed for.

[Matt K]

Definitely not a lost cause. I've been monitoring the site for a while, and it seems that the changes/optimizations/etc. that I did are sustainable for reliable operations. I've been a little sidetracked starting a band ( ) and am back at work now that my paternity leave ended ( ). Have our first gig next week and then I'll circle back on trying to re-enable some of the lost features, like RSS feeds. The biggest problem is going to be keeping some of the protections we get from Cloudflare (virtually 99%+ of the traffic was bots) but allow, well, what you could technically call a bot through.

What that is probably going to look like is going to be publishing an rss.trombonechat.com and enabling people to paste the stuff after trombonechat.com/etc.etc.etc. into it to get such feeds. The problem is that if I use phpbb, it treats every access as a single, "anonymous" user for this type of traffic, which requires serious resources from the DB, which impedes reliability for like, you know, real users. So I'll likely spin up some kind of read replica and have the RSS feeds be slightly behind the real site.

Unfortunately, I do not intend to put the classifieds sections - as was the case before too - on RSS, because we got tons of scammers when that was enabled for non-logged-in users. Technically, there's nothing really stopping people from doing that, but it seems to put enough friction in place that many of the previous, very pervasive problems do not exist, but still people need to do due diligence, etc., as this is still a minor roadblock. As usual, the <.1% of bad people make life worse for the honest people.

[BGuttman]

I've noticed that CloudFlare is making me check a box every time where it didn't used to.

Wonder if my Internet connection is to blame.

[Matt K]

I've noticed that CloudFlare is making me check a box every time where it didn't used to.

Wonder if my Internet connection is to blame.

Looking at your ISP, or at least what the forum thinks your ISP is, that is very possible. You appear to be behind a CGNAT, which basically means that several "people" share your IP address. Cloudflare looks at patterns not just from our site but from ALL sites they are involved with and ratchets up the filtering when it detects that IP is causing larger than expected traffic.

That said, it's unlikely that you have an internet neighbor (or perhaps literal neighbor, as it will still be likely people physically close to you) is causing large amounts of traffic aimed at us. I'll dial down some of the protections this weekend and see if that helps, and still allows the site to effectively function.

[BGuttman]

My ISP is in a nursing home and I probably share my connection with a large number of other users, although I'd bet none of them are looking for trombone stuff. I've been here for a long time. When you first initiated the CloudFlare check I'd sometimes get the box to click but way more often it would "pass" me directly on. Having to click the box every time isn't a deal breaker by any means, though.