Today, like the past few days, we have had some downtime. Apparently some script kids are enjoying themselves by targeting our server (and others). Sorry for the inconvenience.
Most of these ‘attacks’ are targeted at the database, but some are more ddos-like and can be mitigated by using a CDN. Some other Lemmy servers are using Cloudflare, so we know that works. Therefore we have chosen Cloudflare as CDN / DDOS protection platform for now. We will look into other options, but we needed something to be implemented asap.
For the other attacks, we are using them to investigate and implement measures like rate limiting etc.
Thank you for the amazing job, as always! Cloudflare is a solid solution :)
Sure but maybe something less centralized/proprietary would be preferable
Such as?
Nothing. DDoS mitigation is inherently an ISP or someone like cloudflare. You will not have success against anybody who knows what they are doing without their help.
This is bullshit. Just take this as an example. I found it with one quick search and there are plenty more. Perhaps we should broaden our horizons a little rather than entrusting everything to some corpos.
My dude, I think you’re not super familiar with these technologies.
The most basic form of a content delivery network is a set of globally distributed servers that replicate content from a source of truth and a network to direct traffic to the closest server with a valid replica. So the cost here is servers.
With Lemmy, this problem is solved by eliminating the need for individuals to own many servers and a lack of need for trust between servers. The effort and cost is distributed among individual humans, making it manageable.
Now, if you’re familiar with blockchain, you probably perked up when you heard “lack of need for trust.” That’s what the blockchain was built for! Perfect fit, right? Ehh, not so much.
There’s two problems: acting as a proxy for content requires trust, and some single service needs to direct clients to the right local server. If I can arbitrarily join some network of serving content, I can always tell other servers in the network that I’m serving what they ask… and then serve ads. There’s no (reasonable and fast) way for the network to verify that I’m serving the correct content to every client. There’s no way to avoid the need for trust. Additionally, DNS, which directs you from mysite.com to 120.1.2.1, isn’t intelligent. It can’t direct clients to a geographically (or route-efficient, fucking ISPs) local IP. The best it can do is pick a random one from the pool. So when you go to lemmy.world, DNS can’t pick the correct server for you. So some set of servers needs to do the logic to select which local server to actually get content from. Those servers need to be central for the whole content delivery network.
This company you linked is just another company using “blockchain” to get investment money. If you read through their page to get a cursory understanding of how things work, an easy question comes up: what is the purpose of
mediatokens? Sure, maybe you can buy CDN time with it, but when you pay that token to someone providing compute… what do they do with that token? It’s worthless, just like crypto currency. Fucking scams. All that said, blockchain is a super, super interesting technology. There’s just very, very few suitable applications of it.I’ve worked in IT for about 12 years now. Everything from infrastructure monitoring to data analysis to data engineering to DevOps to backend engineering to product management. I’ve worked with systems serving tens of users and tens of millions of users. Happy to answer any questions. I love this shit.
If someone could figure out a trustless, decentralized way to implement a CDN, I’d eat that up in a second, but with my current understanding of the internet and available technologies, I don’t see a way it can work. At least, not with making every web page take >3s to load, which would absolutely kill websites.
Great breakdown
I could a agree with the first part and it does not contradict with the idea of a distributed network for content saving. Think about it this way. Instead of one big local server farm you have multiple small local servers which together form a global network. Now we come to the blockchain. As you pointed out you get these tokens for the CDN time the storage or more generally the server operation costs. Of course the blockchain these tokens are hosted on (Solana) do have to be trustworthy (which in this case they may not be. I don’t like solana that much either). But does that mean that this could not be achieved? It seems logical to me that with a distributed storage and computing network something like this could be achieved very efficiently and cheaply. Heck I’m using a decentralized VPN right now that works with the same principles I mentioned. Or take the Helium network for example? Don’t you see the potential there? Like with all technology these things have to mature but with my understanding they are pretty much doable.
Sure, its doable, but if we return to OP issue, is it available and usable now? If there’s a service provider I’d trust to do this, it’s CF, they have a good, solid product and they have not given a reason to doubt their business ethics yet.
They exist sure, but as others have said, there’s still a lot of links in the chain to smooth out. And for a mission critical application like this you’ll always want to chose the most stable offering.
They lost me at building a CDN on top of a blockchain. Why?
It’s rather the other way round. Complement a distributed CDN with a blockchain.
Stupid
Well for now we’ll have to stick around with cloudflare. I’d just would like to see something managed by a decentralized network. I don’t know if it exists, it’s more of a sentiment or a general idea.
If you don’t know what a content delivery network is, here : https://www.cloudflare.com/learning/cdn/what-is-a-cdn/
A CND is very costly to run in an effective way. And because it is an intermediary server between the user and content server, the market is already pretty full. So competing with the CDN giants is practically impossible in a decentralised manner.
Because of what a CDN does (cache website elements closer to the user, protect the website against ddos…), it cannot be a cheap weak server, or it’s the one which will get overwhelmed by the ddos, or even the users.
Another limiting factor is that in decentralisation, that means different companies, and so many separate plans to pay, which is just impossible for a company.
If it was decentralized, a company would have to go and pay 100 different companies (which is more expensive, du to the server costs and each companies having their own staff to may (even if it’s just 1 person per company)) just to offer a quick access to the users around the world, which is just impossible.
A CDN isn’t a great comparison to DDOS mitigations. CDN spreads the load amongst multiple locations that are distinct entities. Any one can be down and the rest functions fine. They generally exist on separate domains and are not inherently codependent.
DDOS requires an inline solution. A layer acting as a man in the middle to deflect or absorb the traffic destined to Lemmy.world, for example. That’s not something that can be readily be decentralized while there’s only one ingress to Lemmy.world.
I know well what a CDN is and that’s why I don’t understand why you build a DISTRIBUTED content delivery network on a single corporation. I mean, the whole architecture is based on decentralised servers that precache the content and share the service load. Why not create an independent network that provides this bandwidth and where each node is rewarded according to its contribution? I know blockchain is a term that pisses a lot of people off, but it’s basically the best way to incorporate trust and monetisation into a decentralised system.
Idiot
I think the biggest problem with such services is that they require lots of money to run which means that any well-meaning effort will eventually end up becoming a commercial service.
…and that’s where the blockchain comes in. This means that the individual contributions of the node operators can be directly recorded and compensated adequately.
…and that’s where the blockchain comes in.
Sure.
Tell me a good argument why not? How would you reward those people that contribute to said netowork?
It’s an interesting question but the knee jerk reaction towards decentralization isn’t always a silver bullet. Bitcoin always screamed that concept while ignoring the role of clearinghouses. Decentralization can actually compound the issue. Not to dispel the solution but good to keep these things in mind.
It isn’t a silver bullet but in this case it is particularly suitable. I mean, the architecture of CDN is decentralised, but all these servers are controlled by ONE company. So why not leave the whole task to an independent network?
You’re being down voted, but a p2p cdn is something that sort of already exists. IPFS is probably the most mature. As far as I know, it’d only work for static content though. It’s also an entirely different protocol so you’d have to use some sort of local gateway or plugin to make use of it.
I have several vms and dedicated servers that I sort of use as a DIY cdn. No where near as spread out or capable as something like cloudflare, but its also not incredibly expensive to do on a small low performance scale. DDOS mitigation is another story though, generally that is best handled by large networks that can soak up the throughput.
Yeah it’s also more of a potential that I wanted to point out. Over the years that I have been involved with blockchain projects, I have developed a feeling for where blockchains and decentralised networks are suitable and where they are not. In this case, however, it seems very feasible to me. In the end, CDNs are nothing more than a server network that caches the data locally and distributes the bandwidth. This is exactly what an independent network could do with the advantage of the blockchain to remunerate the contributions of the individual node operators. But I see that the notion of blockchain triggers a great aversion in most people.
I don’t have half the knowledge in IT you have, but i totally agree we should find a solution to seperate from mastadons who owns the whole network.
It’s very similar to how we shouldn’t give big corpos like GAFAM willingly our data/privacy or our foodchain shouldn’t be controled by a few corpos who serve poison… (the list goes on).
Most people just don’t care, they have nothing to hide or they won’t die if they eat one cheesburger from McDonald’s a week…
But in the case of lemmy I think (personal opinion) It’s because it’s easier, simpler, faster to setup right now. I’m sure if they had a better solution to not depend on cloudflare they would chose the other solution.
I mean your idea seems great, but how long would it take to put it inplace? How many highly qulified people are needed to make it work? How much will it cost…
I hope that in the long run, lemmy instances are going to find a better solution 😀
I’m only talking about the long run. For now cloudflare is a solid service. I’d love to see some experental approaches tho maybe from other smaller instances.
Build your own then.
Wanna know the beauty of Lemmy? If you don’t like how instances are ran you can create your own🙂
sure, this is just a wish of mine and I’m totally happy with our mods here.
This is a polite way of saying get lost.
That’s easier said than done, DDoS mitigation requires a large amount of servers that are only really useful to persist an active DDoS attack. It’s why everyone uses Cloudflare, because of the amount of customers they serve there’s pretty much always an active attack to fend off. Decentralization wouldn’t work great for it because you would have to trust every decentralized node not to perform man in the middle attacks. But if you know of any such solution I’d love to hear it.
Yeah I see the issue but on the other side you would get a more robust network which could also be incentivised by some sort of underlying blockchain technology. The man in the middle attack could also be mitigated on a technical level.
Oh man, you lost me at blockchain.
I block anyone who mentions a blockchain.
Blockchains are bullshit.
Wait whaat?
Chances are that you’re being sarcastic, but in the event you’re not or if others want to learn…
Interesting tech. Almost zero practically useful applications.
Blockchains are effectively reproducible, verifiable ledger systems. But if the ledger grows infinitely, your storage and compute costs also grow infinitely. I’ve heard this has been solved, but I haven’t seen an implementation yet. (If anyone knows of one, please share!)
Another issue is the proofing system. Bitcoin uses proof of work, which means you need to do more computational work to produce new blocks on the chain. If the computational work grows, that means you need more and more powerful computers. This means increased cost which means centralization as participants with less money to pay for compute get pushed out. Alternatively, there’s proof of stake, where having some amount of a token or some similar value/stake allows you to write new blocks. This does reduce the computation cost but still causes those with lots of tokens/stake to get even more tokens/stake, which in turn allows them to spend more for new blocks… which creates a loop towards centralization.
So basically, the technology that preaches decentralization naturally centralizes in practical use over time.
Blockchains are bullshit.
You can’t mitigate a man in the middle attack on a technical level… Because they are a man in the middle… That’s the point of using DDoS mitigation. Nothing’s stopping them from just sending incoming traffic to a phishing site if a bad actor was in control of it.
Dunno if this guy is just so stupid or is trolling at this point. Using random tech buzzwords that have no relevance to the issue.
You’ve never blockchained your decentralized DDoS backend with a bi-duplex CDN enumerator?
Well I did mitigate an attack before using quantum entanglement calibrated against the cosmological constant to mitigated carbon decay. Does that count? Oh and, blockchain and decentralized. Haha
I myself am not sure who here understands anything about blockchain technology. For you it’s just NFT images and shitcoins that you associate with blockchain, isn’t it? That knowledge is enough for you to understand the whole technology. Read my other comments and ask yourself first if you have a balanced information base.
For sure it’s not you. You sound like an amateur developer no-one would hire.
You are smoking crack. You clearly do not know what you are talking about.
You had me until you mentioned blockchain technology. How would a blockchain system help in that regard, anyway?
A blockchain can complement a decentralised network by introducing trust into such a network, where the individual members cannot be trusted. This makes it possible to accurately document actions and reward or punish them accordingly. If you take such a distributed CDN network as an example, a blockchain could help to directly reward the individual members according to their contributions instead of building everything on voluntariness and goodwill as in the Tor network.
Which viable alternative could work to mitigate ddos?
Out of my head, I think OVH offers such a service (but without free tier).
OVH is cheap but their anti-spam/abuse departments are ineffective. Too often they do not action blatant spam reports so in effect OVH is part of the problem with network abuse on the Internet. I’ve had to blackhole many of their netblocks while the people who run mxroute (solid email providers) have written about doing the same.
OVH needs to clean up their act.
HAProxy has some really good features a server admin can use locally without sending all of our data to Cloudflare or OVH.
https://www.haproxy.com/blog/application-layer-ddos-attack-protection-with-haproxyThere are many protection modules for most reverse proxies that provide basic (limiting) or sophisticated (captcha, calculation challenge, etc) DDoS protection. HAProxy is just a very powerful and easily extensible proxy.
Sure, but you still have to pay for servers to run the proxy instances on. Any DDoS of appreciable size will knock over the number of instances that lemmy.world could stand up. Interesting thought, though. Maybe CloudFlare or others use HAProxy internally? I’m actually not sure what tech they use
It’s usually very effective unless the amount of connections is too much for one machine anyways. Along with bandwidth shaping and connection throttling it can fend off smaller attacks.
A huge botnet would bring down a single proxy instance in any case, true.
Is “decentralised” the new “blockchain”?
Well, no. Unlike the blockchain, decentralized platforms aren’t snake oil.
This explains nothing. It’s literally saying “one thing is bad, the other thing isn’t”. Try to explain why instead, if you do happen to have an explanation.
One of the things that makes Lemmy unique is the underlying decentralized infrastructure. I think it’s just a request to keep that mantra.
Why are the Lemmy devs asking for snake oil on their Donate page then?
Sitting comfy in a country where the financial system works for you elites is the real snake oil.
…what are you even talking about? Your sentence makes zero sense.
100% of the crypto hate I see is from citizens of neocolonial states. You lord your control of the financial system over us and when something threatens it, it’s always delegitimised for any number of reasons.
Take your pick: scam, destroying the environment, eroding state power etc.
A decentralised system/society will need a value layer to transact. You think Visa should be in control of that?
Just because you don’t like it, doesn’t make it snake oil. I hope you never find yourself at the mercy of a government that persecutes you and imposes capital control so you can’t even run away with your money. If crypto existed when my people were literally being genocided, my parents would not have to end up in a new country with nothing to their name.
Just because you’re smart at writing code doesn’t mean you’re smart at other things :) Or more likely, maybe they’re ideology-driven rather than by practicality.
Lemmy is an unusual but fortunate example of where ideology and practicality line up.
If you can find an entire nation state that runs on crypto currency with a functional, stable economy, I’ll eat my words.
Why would I want a nation state to run on anything? The end of the nation state is the communist utopia!
I am for whatever erodes the illegitimate violence exerted by nation states to safeguard their parasitic domain. If it’s crypto, it’s crypto. If people not eating apples brought about their end, I would be out there burning orchards.
Blockchain can bring trust and thus monetisation to a decentralised network. A good example is the Tor network, which is based on voluntariness, and dVPNs, which can have the same network architecture, but where the nodes are paid for their services.
this is a meme right
It is obviously not. Why would it be?
Stupid
There are a couple elements that a DDOS mitigation system needs to have.
It needs to be able to absorb the raw network traffic of the attack. A purely volumetric attack seeks to just overload the network pipes that lead to the servers. This can be with junk packets that don’t even make sense to an OS kernel, but have a valid destination IP address so they get through the routers. If the DDOS mitigation system acts as a filter in front of the servers, it has to not get overloaded in the same way the routers do.
It needs to allow good traffic through to the servers. If the attack causes the pipes to just shut down and reject all traffic, then the attack has succeeded. So the mitigation system has to distinguish attack traffic from good traffic, and keep the pipes open enough to let the good traffic through.
For attacks trying to do expensive stuff on the database, or create spam posts, one useful reflex the system can have is to notice when an endpoint is doing those attacks, and then block it at the network layer.
That is not necessarily easy, and it requires control of the network ingress, which arbitrary hosting providers may not be able to provide.
Thank you for the clear explanation. It seems a lot of folks here don’t understand the tech, but this explains things clearly and accurately
The goal is to mitigate attacks, it costs a lot of money to purpose build world spanning networks than can absorb large amounts of traffic. P2P type options are not a good fit.
Thanks to the fediverse we were all able to read and search old posts on other instances and interact freely with communities on other instances. Pretty damn great i think.
@jimmy90 @PropaGandalf And I was able to finally open up and stop being a lurker
I meant the CDN. The Fediverse per see is great!
You sound fanatical with this statement
What a
This isn’t a helpful reply. There’s no reason to just call someone a name without even explaining why you think what he said is moronic.
Agreed
Don’t forget. Donate to them. There are no ads here. So we have to maintain the staff and servers.
Lemmy World
https://www.patreon.com/mastodonworld?utm_campaign=creatorshare_fan
Lemmy Devs
https://www.patreon.com/dessalines?utm_campaign=creatorshare_fan
Didn’t the admins for Lemmy[.]world post their expenses recently-ish? I can’t remember how much it would be for a single user to donate. I’d want to donate, but I’d like to know how much of my contribution would affect operation of the server.
Wow if I’m reading their expenses correctly, the maintenance bill doubled from May to June…
Yes. Here you go. https://opencollective.com/mastodonworld
Is this the combined bill for mastodonworld and lemmyworld?
Not sure. I would reach out to them.
Imagine hosting a service for anyone else to use it, free of charge, no ads, free & open API, yet some idiots think it’s fair to (D)DOS it.
There are more “interesting” targets, worst case - Reddit, who thinks everyone is just a number/noise.
Just leave Lemmy alone. :(
we will all still be here when their hyperactivity wears off.
with the old Reddit simulator, personally I’m not going anywhere anytime soon. This place has a great user base and it feels so old-school.
The new layout with old.lemmy I came back, and new apps coming out for it. It’s been a good replacement. Was on tildes, but got banned for just discussing difficult topics…the admin there is just ban happy and yea he owns the site but will just ban people for no reason. Not to mention that the users over there, assuming new people are using the malicious tag as a down vote button which probably goes right to the admin. So you step out of line and you get banned. I really liked the place too, but it’s not wanting to be a serious place to discuss topics with an admin like that.
I wonder if the owners of deddit, fb, tweetster, et al, might think it financially worthwhile to cause disruption in the fediverse, and even its ultimate failure.
I wouldn’t be surprised, we didn’t take their whole user base of anything but it’s in their interest to keep viable competitors out of the way.
Every account they lose hits them in the pocketbook. The bigger the fediverse gets, the more adherents, the greater the momentum it will have and the harder it will be to stop.
Nipping it in the bud is the best, easiest, and least expensive place to nip it.
The downvotes suggest their operatives are reading the comments.
Counterpoint- people are down voting because they think its unlikely and many people are inherently gaurded against conspiratorial thinking- especially if they think it’s unrealistic.
Whether you think its happening or not, the idea that the only reason anyone would downvote is because they’re “opperatives” of the big social platforms is kind of out of touch with the fact that there are lots of people who don’t think like you do. I’m a real person, love open source, and love the fediverse (have 3 lemmy accounts, plus an account for mastodon and pixelfed each) and I was tempted to down vote certain comments just because they seemed silly and a bit like fearmongering that there’s a big bad boogey man out to get us.
I hope I’m being clear, communicating on the internet devoid of tone or facial expressions is hard- my point isn’t that your perspective is silly, my point is that there are lots of people who would sincerely see it that way and disagree with you. Assuming that being disagreed with is a sign of the sort of conspiratorial situation you’re describing is a self fulfilling prophecy. I hope I’m not coming across as hostile, that isn’t my intent
Personally I think the other platforms are unlikely to see the fediverse as a problem until it proves it can be, because CEOs are stupid, and after eons of not having meaningful competition in this space I think they’re likely to be overly proud and look down on our nice little platform. I think its far more likely its just the internet being shitty because lots of people on the internet like breaking or ruining anything they can, regardless of whether its a good thing to have exist. I could very easily be wrong, and perhaps other platform’s owners do want to kill what we have before it can manifest into something bigger, but either way there are lots of sincerely held perspectives that might drive someone to down vote some of the comments here just because they think the situation being described is unrealistic.
Points well made and taken, thanks. No hostility perceived at all.
Reasonable minds can differ and frequently do. And it could be that people may think my suggestion is unrealistic or even silly.
There’s no shortage of miscreants out there who just like to mess with things, thrown wrenches into spokes, etc. And these types could well be behind the daily local issues.
But here’s an important point, and no offense intended. Corporations are like The Terminator. But instead of getting Sarah Connor, they purse profits. And regardless of CEO intelligence or accumen, every Fortune 500 company has a department that deals in these areas. They all have their skunk works and use them. It’s been this way for centuries. A primer: https://en.m.wikipedia.org/wiki/Industrial_espionage
So whether they’re operating here atm or not, there is nothing paranoid about assuming they are. If they’re not, they will be. It’s what they do.
Thanks for the input. :)
But here’s an important point, and no offense intended. Corporations are like The Terminator. But instead of getting Sarah Connor, they purse profits. And regardless of CEO intelligence or accumen, every Fortune 500 company has a department that deals in these areas. They all have their skunk works and use them. It’s been this way for centuries. A primer: https://en.m.wikipedia.org/wiki/Industrial_espionage
Lol, all very fair, corporations suck and are prone to doing anything shitty they can think of to even marginally improve their bottom line. Its an understandable sentiment.
I’m glad I was able to convey what I meant without it coming across as my being a dick :)
Take care! ❤️
It’s an important life skill, being able to plant a thought in the mind of another and in a way that is likely to be accepted.
It crossed my mind since my last writing that, in the 80s, I got a money back guarantee for any counter-surveillance equipment purchased that didn’t reveal surveillance equipment in a Fortune 100 facility. It was that pervasive back then. And my perception is that morals and business ethics have not improved in the interim. Far from it.
Good luck and thanks for the valuable, respectful input.
I agree, many of them appear to be edgy script kiddies upset that people don’t wanna use their precious reddit anymore
The downvotes suggest their operatives are reading the comments.
Let’s not do this. People are allowed to downvote without being a paid operative. This was a very common mentality on Reddit I would like to avoid here.
What makes Lemmy interesting is that you can see the combined upvotes and downvotes. It’s not a “net” votes system like some shithole site whose name I will not mention. So I think people can read into the voting system much more than they might have been able to do on some other awful and alienating place.
But, I too disagree with the conspiratorial comment that there are operatives downvoting people on Lemmy, as if that could do anything meaningful. I think the notion that Lemmy is being hacked because the major social media companies are afraid of it, is also very extreme and conspiratorial.
I agree we should support this community and people’s ability to react positively or poorly to a post or comment.
Wondering if reddit or Musk are behind the attacks?
Most likely their parasocial fans. The Reddit stans who want to be edgy and follow their meme leader. Who will never acknowledge them no matter how much they do.
It’s sad that they could target the real people making the world worse, yet only prop up the people who are oppressors.
I don’t understand why people want to take down websites. Especially sites like Lemmy, which isn’t exactly sticking it to anyone because no one owns it!
Are they just Reddit groupies?
For most hackers or wanna-bes (often called Script Kiddies, that is, people (generally young, even children thus the “Kiddies”) who are not technologically inclined enough to be real hackers and see a tutorial online on how to run pre-written scripts that repeatedly perform various functions), the answer to “Why do you do it?” is often:
-
“Because I was bored.”
-
“Because I can.”
Very rarely are other reasons given.
deleted by creator
More like “I get zero action, so I take my anger out on other people”
-
Some people enjoy causing suffering to others. On the internet they are termed trolls. Irl people usually just call them assholes. Most people have encountered them before.
I think they are far more common and likely than anyone giving two shits about reddit.
I was using voip.ms last year when they were DDoS’d for over a week, by a group demanding payment via anonymous crypto. The DDoS ended when they switched to CloudFlare (which was probably pretty difficult because they’re a SIP provider.)
Almost any website with a small number of servers is vulnerable to this attack, which happens to be great business for CloudFlare. I wonder which companies are most effectively competing with CloudFlare?
There are others, but I think the craziest thing about Cloudflare is its basic level of protection is free. Free, unmetered, DDOS protection. It’s so popular because so many hobbyists use it for free, and are familiar with it. Then they convince their workplaces to adopt it when the need arises because they are already familiar with it.
They make money by selling support to companies, and selling access to some more advanced features (that often have a free tier as well). It’s honestly so impressive, it made me wonder how much they actually make because it seems unnecessary for most to pay at all. Turns out they cleared almost a billion dollars in revenue in 2022.
They’re just trolls. Lemmy is popular enough that it’s fun target for them, but still small and infantile enough that you don’t have to be hackerman to ddos it. Reddit, twitter, etc… would be constantly getting ddos’d just for the lulz by people if they didn’t have the infrastructure to make it a challenge.
Nah, it’s not the 00s anymore. Hacker gangs are a real thing today.
I’m not actually in the security field so take this with a grain of salt. But I believe that these attacks play a similar role to random attacks in low level gangs. It proves that your criminal group has power and the ability to deface a website.
So if you publish that Lemmy.world will go down next week because your hackers are on it… It’s advertising. Its just business. It proves that your hackers have an ability and that you are up for sale.
Cyberspace. A consensual hallucination experienced daily by billions of legitimate operators, in every nation, by children being taught mathematical concepts… A graphic representation of data abstracted from banks of every computer in the human system. Unthinkable complexity. Lines of light ranged in the nonspace of the mind, clusters and constellations of data. Like city lights, receding…
Some people just want to watch the world burn.
there are some people salty at a given instances, like exploding-heads for defederation or this @lmao dude for no clear reason, there was some spammer activity, and then you have regular drama seekers with usual ensemble of suspects
You don’t think just being bored is enough reason for some?
If I’m bored I find something productive and/or fun to do.
Launching a DDoS attack is neither.
You, sure. It’s not difficult to imagine a teen who’s not you
With my tinfoil hat on, I’d say one concern is that Cloudfare is basically a monopoly and nothing is stopping them from DDoSing sites to force them to use their product.
While it’s good to be suspicious, I don’t think we can call CloudFlare a monopoly quite yet.
Akamai is a big, giant competitor. You also have the big cloud providers like AWS that have their own CDN systems, like CloudFront. (I don’t recall GCP’s or Azure’s product names.) Then you have specialized CDNs like Google’s AMP system.
Now, is it possible that there could be a horizontal trust between these companies? Certainly. There’s few enough players for that to happen, but so far, I haven’t seen signs of it happening.
Or paid for by Reddit…
My guess: lemmy.world is attacking itself to make people believe that it is more popular enough than Reddit to be attacked so hard.
Upvoting because this has to be satire
It’s coming from someone over on Kbin. Wonder if that’s the motivation.
You have more faith in people than I do…
Delete your account and go back to reddit
Genius
In case you haven’t considered this, some helpful advice. To keep them from the lemmy.world door after the CDN installation
- Change the public IP addresses
- rotate your certificates
- block all traffic appart from the CDN and only allow a limited known good IP addresses (like yours and your support team). These steps will make your server harder to find, hopefully they move on.
You might have Cloudflare add a request header to the origin request, like
x-cloudflare-key: <somesecret>, and then configure nginx on the server to block everything not containing that header.just block ingress to your server from non cloudflare IPs or use argo tunnel.
Good News
Most of these ‘attacks’ are targeted at the database
A major PostgreSQL performance issue, logic mistake, was discovered today in lemmy_server and is an easy fix. Details: https://lemmy.world/post/2008987
That’s great!
Growing pains. This server and the platform will be better for it. If not for these script kids, some other attacker would eventually be motivated to try it.
old.lemmy.world still exposes your hetzner server to the internet, just a quick heads up.
Exposing your hetzner sounds fucked up
Wait till you see Oppenheimer.
Thank you as always for the transparency. This instance is going to be the most targeted because of its size. Y’all dealing with this is hard but you’re going to figure things out that will help the other instances.
deleted by creator
Seems like we had downtime again just a little while earlier?
Yep
Anything we can do as “users” to help, other than donating?
Hmm, best would be if those kids find a real hobby so they stop bothering us. On the other hand, it helps us understand Lemmy better and secure it.
That’s true. Free stress testing the system I guess? Still they need to touch grass lol
If it’s the same people, they’ll probably get tired of it and move on. But the more we talk about it, the more likely it is that new people want to get in on the “fun”. I’d say to not make memes about the downtime and pretty much act like it doesn’t exist (as users, obviously the admins should take action as necessary to mitigate it and post to be transparent).
Excellent! CDN and DDoS protection are essential. Also would recommend looking into load balancing if you haven’t.
Thank you for your hard work, and for keeping us updated on the situation.
Wonder why this wasn’t done earlier. Hopefully we’ll see less of the 404-type pages that has plagued this instance.
It costs more money right?
Not necessarily. I have several servers behind Cloudflare for free. I’m just limited on analytics, some advanced firewall settings, advanced cache management and maybe a few other features that I don’t use. But the basic service is free.
Nothings free, someone’s paying cloudflare something to operate
I imagine this is more of a “If we give people the basic stuff for free when they are small, they are more likely to buy our better stuff when they grow and need to update”
Dont understand this either, who the fck would expose real server IP in first place?
anyone cuz it doesn’t matter
This post would not existed if they properly hide their real IP in the first place lol, so yeah it does matter a lot ;)
Actually it does. There’s a reason why Cloudflare is a multi-billion dollar company…
Because CDNs lighten load and work as a global cache for load times? Game servers and plenty of other types of servers have exposed their IP since the dawn of time.
Thats not how it works
Thank you! I will donate tomorrow
Be aware that you use another server so you might consider donating to them instead.
I have an account on yours too, but I might split it between both indeed :)
I’m curious, why bother with multiple accounts? It seems counterintuitive when taking federation into account
Edit: All of these reasons are why I host my own instance
I do it for the following reasons:
- The big main lemmy servers can and do go down regularly. having accounts on other instances still gives me the ability to log in and participate on the communities I care about. I sure do wish the underlying federation logic allowed for associating selected logins on various instances together somehow.
- Testing: often, I will create a post or comment into a community hosted on a different lemmy server and not see it update. When this happens, I’ll log in on the other instance to see what that post or comment looks like from there.
- Insurance against defederation: I participated/followed some beehaw.org communities before they defederated, and then I was forced to open an account there to continue participating in those communities.
Taking some load from the biggest servers such as LW. I still have a community on LW however, and mod with my local account
Well, I can’t answer for them but this situation in particular makes it nice to have accounts on different instances. If I can’t log on to/load my lemmy.world account then I can switch to my lemm.ee account and load content there.
I have accounts on both lemmy.world and outpost.zeuslink.net. I have had a lot of problems with seeing posts from the smaller instance, while lemmy.world has been overloaded due to the massive Reddit exodus and script kiddies.
I created the account on the smaller instance first, but I’m still having a ton of issues with federation on that one (many communities still come up blank there, while lemmy.world has enough users that I can see everything when it isn’t down).
So far federation seems to be a lot better since the fix from last week - I can’t say that I’ve seen anything out of place or out of sync myself, but I am trying to keep a vigilant eye out for anything that’s been overlooked!
since 0.18.2 I havn’t noticed any federation issues with my own personal instance, nothing obvious anyway.
