yha, what do people think the FBI is for… this isn’t crazy. They can get access to ISP logs, VPN provider logs, etc.

CSAM (Child Sexual Assault Material) posts

The federal governments of several nations should be in pursuit of this, and IP addresses and specific time logs shared.

Nothing like a little bit of corporate sabotage!

The software developers who created Lemmy openly criticize systems of government and economics. These are nation-state battlegrounds too. The barrier to entrance is very low, as Lemmy doesn’t even do routine tracking of account creation, rate-limiting alone isn’t really defensive. 15 years ago sites like Reddit had major vote manipulation detection logic behind the scenes. This is pretty much unleashed playground for a lot of known tactics.

@ruud@lemmy.world and @sunaurus@lemm.ee

I added some self-awareness of concurrency to the Lemmy API Rust logic: https://github.com/LemmyNet/lemmy/pull/3805/files - this should allow a more standard in-band response to concurrency slow-response server overloads. Keywords (‘woodstocksnoopy’ as watchdog, and even more alarming keyworld to find in your logs: ‘SITE_OVERLOAD’) were added to the logging to make it explicit when this is happening.

@sunaurus@lemm.ee

The bugs in Lemmy are such that you don’t even need to touch a server for it to be vulnerable. Cloudflare does not defend against such mistakes. Other servers can trigger deep PostgreSQL logic problems within Lemmy. Growing pains, a lot of the federation code was never tested, and today’s crash is due to a logic issue with lemmy_server mistakenly updating 1700 servers it knows of through federation for a delete instead of the 1 local server.

Good News

Most of these ‘attacks’ are targeted at the database

A major PostgreSQL performance issue, logic mistake, was discovered today in lemmy_server and is an easy fix. Details: https://lemmy.world/post/2008987

Good to see a heavy production server taking on the scaling issues. Thank you! To discuss Lemmy performance issues, there is a dedicated community: !lemmyperformance@lemmy.ml

Edit still see some performance issues. Needs more troubleshooting

Federation overheard is putting a lot of load on servers. Creating one task for every single post, comment, and vote in RAM-only queue… pending changes: https://github.com/LemmyNet/lemmy/pull/3466

we created extra lemmy containers to spread the load. (And extra lemmy-ui containers).

Is Rust HTTP server running into thread limits? database connection pooling? All kinds of internal questions bout that solution.

My lemmy server is also running the same versions are you are now. I was getting timeouts with lemmy.world federation so I recompiled lemmy_server with timeout changes:

diff --git a/crates/utils/src/lib.rs b/crates/utils/src/lib.rs
index e5d07db2c..e2c592d82 100644
--- a/crates/utils/src/lib.rs
+++ b/crates/utils/src/lib.rs
@@ -18,7 +18,7 @@ use std::time::Duration;

 pub type ConnectionId = usize;

-pub const REQWEST_TIMEOUT: Duration = Duration::from_secs(10);
+pub const REQWEST_TIMEOUT: Duration = Duration::from_secs(13);

 #[macro_export]
 macro_rules! location_info {
diff --git a/src/lib.rs b/src/lib.rs
index cc77ca48f..45c621a7c 100644
--- a/src/lib.rs
+++ b/src/lib.rs
@@ -37,7 +37,7 @@ use tracing_subscriber::{filter::Targets, layer::SubscriberExt, Layer, Registry}
 use url::Url;

 /// Max timeout for http requests
-pub(crate) const REQWEST_TIMEOUT: Duration = Duration::from_secs(10);
+pub(crate) const REQWEST_TIMEOUT: Duration = Duration::from_secs(16);

 /// Placing the main function in lib.rs allows other crates to import it and embed Lemmy
 pub async fn start_lemmy_server() -> Result<(), LemmyError> {

I think a static page page explaining to users that the server is being upgraded would be best.

Beehaw.org did this, their nginx seems configured to return a nice error message.

That’s for encryption signatures.

I’m talking about HTTP timeouts for connection:

https://github.com/LemmyNet/lemmy/blob/0f91759e4d1f7092ae23302ccb6426250a07dab2/src/lib.rs#L39

/// Max timeout for http requests
pub(crate) const REQWEST_TIMEOUT: Duration = Duration::from_secs(10);

From what I’ve seen, there is a 10-second hard-coded timeout for HTTP, seems too low for the kind of load going on. Especially if the server is opening tons of connects to the same peer server.