It’s excruciatingly obnoxious to have to rely on third party sources for what should be a first-party feature.
Like, I select all and then search a query. “Oh no, nobody on your server used a third party service to find it, so you won’t see it here.”
Like, how short-sighted is that, really? If I search for a string in the ‘all’ servers, I should have a list of ‘all’ the servers containing that string.
It’s a really simple concept. Not sure why this post even has to be made, but I’m wondering if there’s something I can do to make these ‘features’ more intuitive.
totally understand the frustration, and i’m not going to try and invalidate it!
… however, it’s definitely not a problem with a simple solution
since anyone can start an instance, when you search “all”, where should it search? i don’t mean generally like “all the instances”, i mean where specifically? things like lemmy.world, lemmy.ml, kbin.social, etc are obvious… but what about lemmy.mydomainforfriends.social (not real but let’s pretend someone created their own little instance for friends there!)?
let’s say you say yes that should be searched, okay… how does your instance know it’s there? does it tell all other instances that it exists at some point? where does IT get that list from? (the current solution to this is that your instance starts to “know about” an instance after someone interacts with it, but this has the problem you’ve described)
let’s say that instance shouldn’t be searched… now, what are the rules (automatic id assume; not with human intervention) that would allow an instance to be added to some big list somewhere? also where is that list? now we’re back at problem 1: how do you store a federated list of servers?
the problem gets even harder when you consider mastodon, pixelfed, peertube, etc… all these services interact: should all include them? only certain things in them?
While it has problems of its own, instances could pool and share that knowledge. The first time an instance talks to a different insta ce it could just ask “hey, what other instances are you aware of?”. The main issue there is just instances obsessively sending exponentially growing lists of instances back and forth.
But no, that is the main bane of federated social media, discoverability without a center of truth
yup! 100% agree! federation is kind of a new thing and we have some issues to work out that’s for sure!
heck, i could even see some kind of federated search service: activitypub instances could submit their content for indexing and individual instance could choose an existing, or run their own federated fediverse search… importantly, there would need to be choice for each individual instance with no centralised repository
So many options, doing none seems lazy. I can source all kinds of lists for my pihole to block traffic. I can put a lot of repos in my yum.conf. It’s not like this should be reliant on any one single source of truth. There could certainly be an open source list maintained. I’m surprised this is considered such a difficult problem with so many smart folks involved, I’m obviously really ignorant to how this stuff works. I just don’t get how a problem that seems to have been solved across a litany of technical products using shared sources in defederated environments is such an exotic hurdle here.
okay so now you have a decentralised list with 1000 servers on it. does your instance… make 1000 requests when you search?
Lists can be cached and updated. Even if posts from all doesn’t include all active content it would be very manageable to have a queries include communities across instances based on names and other fields. All this shit is already solved problems.
since anyone can start an instance, when you search “all”, where should it search?
Easy! It should search all the servers your server is federated with! Servers should contain a list of their community names that can be easily and quickly queried by other servers.
Federation isn’t opt-in though. It would be VERY easy to spin up a bunch of instances with millions or billions of fake communities and use them to DDOS a server’s search function.
Searching current active subscriptions helps mitigate that vector a little.
I would suggest that instances should have settings that allow them to decide whether to “advertise” a community list. With configurable settings like "all, “most active”, “top X”, or even a manually maintained list depending on the admins and instances preferences.
Then your home instance, when searching, should have it’s own settings to decide what results it’s going to ping other servers for. Big/popular/high confidence instances can have an open all/all relationship, while you might query only the top 10 communities from unknown or new instances to handle the scenario you describe.
Federation can be binary yes/no but there should be room to add more logic around enabling search on communities from your instance and controlling the search results from other instances. I don’t think the two are mutually exclusive, unless I fundamentally misunderstand how federation works!
I… don’t think you know what ddossing means but okay.
Would it really be very easy? Especially considering once instances find your doing that, they just block you? Would it be worth people’s time?
Is there any way around this, perhaps querying a global repository of federated instances and sorting them by popularity?
In all honesty, you don’t have a point. If you did, third-party services already wouldn’t offer this. Seeing as they can, it’s clearly possible.
Sorry you’re right that I wasn’t being precise with my terminology. It’s not a DDOS but it could be used to slow down targeted features, take up some HTTP connections, inflate the target’s DB, and waste CPU cycles, so it shares some characteristics of one.
In general, you want to be very very careful of implementing features that allow untrusted parties to supply potentially unbounded resources to your server.
And yeah, it would be trivial to write a set of scripts that pretend to be a lemmy instance and supply an endless number of fake communities to the target server. The nice thing about this attack vector is that it’s also not bound by the normal rate limiting since it’s the target server making the requests. There are definitely a bunch of ways lemmy could mitigate such an attack, but the current approach of “list communities current users are subscribed to” seems like a decent first approach.
Pleroma calls their equivalent of “All” the “Known Network” instead, which does a better job explaining what will show up there in my opinion.
What does ‘All’ mean to you?
In this context it means all posts which are stored on the server. And only things are stored which people subscribed to.
There are good reasons why the protocol has been desihned like that, if you’re interested then you can find out about it. If not, reddit still exists for people who like it more.
All means all. If it isn’t actually All (it isn’t) then it should be called something else.
But it is all, just not the all you think, it’s all things the server is aware of, not all things in the universe.
This is not obvious to anyone who doesn’t have some understanding of how networking and federation work, which is most people. Especially if we’re talking about users who have only ever experienced centralized platforms.
It should be called “Known Network” or something more transparent that doesn’t require an explanation of indexing
I agree. But the attitude of those users just ribbs me the wrong way.
It’s an understandable response. They were previously in a position where this was such an obvious concept that it didn’t merit any thought, and now they are required to have an understanding of networking and federation in order to understand how well actually this a fundamental part of how distributed systems work and isn’t technically a bug.
From their perspective this seems like a fairly straightforward problem. Obviously (to us) it’s not, but the threshold for the fediverse shouldn’t be that you deeply understand federation if there’s ever going to be meaningful adoption.
As an aside, your personal domain is timing out.
Damn, thanks, I have a bad implementation of getting Twitter avatars and now that Twitter redirects everything which is not logged in my implementation goes into redirect hell every time someone opens a page with a Twitter comment. Perhaps I’ll find the time tonight to look for a fix.
It seems I was able to fix it by adding
curl.max_redirects = 3to my caching code. No idea why it would hang without it because it gets the image from Twitter just fine now too.
Uh… no it’s not.
I’m sorry, but what you’re doing is actively making this service harder to use by suggesting that ‘all’ should only mean ‘the communities other community members have subscribed to that contain that string.’
Gee, fediverse design strikes again. Sorry, it has to be said. It really does.
The Internet also has all information made available to it, not all information that is in the universe. Same as the fediverse users, internet users rely on search engines to find what they are looking for.
When all would mean all available in fediverse you would get issues:
- even the smallest instance would explode like lemmy.world
- to much choice makes people freeze up
- you’d get reddit/facebook all over again. (with the required corporate mess as you need to play loads of employers to support the environment to supply the intel to the bubble that wants it)
It’s to the user to search what he wants to see. I for one have 3 ‘main’ accounts (world, ml and studio) and I subscribe to specific communities for each account (ml for development, studio for music related, world for ‘rest’) and each one I open the feed with subscribed. I’m totally not interested in all, to much junk (even on Lemmy).
To freely quote: All, you want all? You can’t handle all. ;)
I cannot believe you’re making this argument.
I’m sorry, I can, and it’s a big reason why fediverse design has a lot of progress to make.
deleted by creator
🥱
Calm down buddy. Your analogy has no water and you’re clearly fueled by emotion.
Take a step back, breath some fresh air, then come back when you’re ready to discuss civilly. Right now you’ve said nothing of substance.
The word “all” fundamentally means everything? By calling it “all” they are really doing a disservice to everyone who, gasp, assumes “all” means “all” when it really means “local communities and local user foreign subscriptions”. I don’t know what they should call it, but redefining the words “all” to be “not all” is super confusing, especially for users new to lemmy.
‘All’ to me means “”“all”“” the servers my instance can connect to that contain that string.
It’s a very simple concept.
Resource wise, it makes sense to only retrieve content the users of the instance are interested in. Think about all the nsfw communities popping on lemmynsfw and pornlemmy, as well as all the content in languages your users don’t speak.
Some instances are multilingual, so you don’t want to defederate from them (and defederation shouldn’t really be used in this context anyway), but retrieving 50% of content none of your users is ever going to read seems like a waste.
Yeah. And I’m interested in retrieving all of the servers my instance is connected to that contain a string in that community name.
Why is this so hard to get across to you?
Why don’t you have a solution for how server members find communities on different servers in the first place? Are you really defending relying on third-party services and ‘other means’ to find communities on different servers?
I think that’s really bad design and a testament to why the fediverse is inaccessible to the general public.
It’s just not very simple, quite the contrary, you would need to have a server park like reddit has it to store everuthing on every instance, the databases would be so big that you would need specialosts running just the database servers.
Oh yes, it really is.
The implementation may not be easy, but the concept is very simple.
Just bringing this to everyone’s awareness, the issues is already tracked here: https://github.com/LemmyNet/lemmy/issues/2951
From the Lemmy devs
I think the lemmy-ui’s could very much benefit from a “global community discovery service” like https://browse.feddit.de , but integrated into the front ends. I’d of course prefer that each lemmy back-end do their own crawling of communities and instances, to make it as distributed as possible.
Thank you for sharing this!
You are welcome!
It’s a massive usability issue and a massive content discovery issue, imo.
For lemmy users who got lucky and had their first lemmy experience on a top 5 instance where a lot of popular off-instance communities are already subscribed to, then users would see a huge list of both local and foreign communities. For users who got unlucky and had their first lemmy experience on a small instance, their view of “all” looks like a ghost town.
Part of the problem is semantical. If they are going to call it “all” then it should really be all (all lemmy communities available on all federated instances). If it isn’t going to actually show everything, then they should call it something else that indicates it’s only local communities plus whatever local users are subscribed to.
I like the idea of calling it “Known Network” and “Local”
That could be better, but based on this thread I guess it could create other confusions down the line
What prevents users from switching to a top 10 instance using https://github.com/CMahaff/lasim or other migration tools?
Resource wise, it makes sense to only retrieve content the users of the instance are interested in. Think about all the nsfw communities popping on lemmynsfw and pornlemmy, as well as all the content in languages your users don’t speak.
You are obviously speaking from the privilege of someone not only familiar with how lemmy works, and who understands the difference and pros/cons of joining a large vs small instance and can probably even name a bunch, but also someone who knows of obscure tools and github repos that host those tools. What prevents users from switching using that obscure tool you referenced is that most users never heard of it and didn’t know it even existed. You are using the argument that new and casual users should have god-level knowledge and understanding…which is exactly the point I’m trying to argue against. Casual and new users don’t know what they don’t know. They don’t know what other communities are out there and they don’t know that when they view “all” that they aren’t seeing them. Think about this from the perspective of someone who doesn’t know what you know.
Regarding your argument about NSFW and foreign language search results…that already happens now when users of your instance have subscribed to those things. You can’t argue that it would become a problem when it’s already happening right now. If it really was the problem that you think it is, then the solution would be to mimic what every other search tool figured out three decades ago and put an option to exclude nsfw results when viewing/searching “all” communities. It’s already possible for communities to flag themselves as NSFW, and it’s already possible for communities to designate their community language setting, it would make sense that those options be presented to users for filtering. These filtering options are things we need now regardless of whether search results come from actual-all or subset-all. I’m just suggesting that “all” mean “actual all”.
But, just for fun, let’s steelman your claim that it would be technologically infeasible for “all” to be “all fediverse” as opposed to a subset of just what this server’s users subscribe to (it’s absolutely not technologically infeasible, but lets pretend it is) - even it that scenario, they should at least change up the UI for the communities page to make it clear that when the user selects “all” that they aren’t really getting all - it should be made clear what the user is actually getting, which is “local, plus foreign content that is subscribed to locally”. It simply is not truly “all”, so presenting it as “all” is only leading to more confusion about what the users are seeing.
Casual and new users don’t know what they don’t know. They don’t know what other communities are out there and they don’t know that when they view “all” that they aren’t seeing them. Think about this from the perspective of someone who doesn’t know what you know.
Then they can ask. Plenty of new joiners ask plenty of other questions, and the community has answered them. I agree there should probably be a consolidated wiki at a later point to address this kind of questions, but with the constant evolution of the platform, and the animation of the different communities, I guess everyone is just busy with something else.
I’m not going to address the rest of your comments as this is a decision made on the Lemmy platform level, actually I just had a look at the GitHub issues, and this issue is already tracked here: https://github.com/LemmyNet/lemmy/issues/2951
Omg I didn’t know this! Though I did wonder why my “all” feed seemed to empty.
The simple explanation is your instance doesn’t “know” what’s out there. lemmy.world doesn’t know when lemmy.ml adds a community, and it doesn’t know when hypothetical.server pops up as a new instance. There’s not really a good way of knowing that without having a central repository, which defeats the purpose of a centralized platform.
One thing you can do is use Lemmy Explorer to search for communities on other instances and subscribe to them. This will fill up All for everyone on your instance.
Looks to me like lemmy explorer could just be sourced for results fairly easy. Even if it was just added as an additional source to the default listings. Similar to setting up yum repos etcetera. Is there a good reason this isn’t a thing? I know my use and exposure to communities is severely limited by the current cluster fuck of finding communities. I just don’t care enough to go further than searching in the app and closing out if nothing shows up. I realize my laziness contributes to my user experience but saying an instance doesn’t know what’s out there and then providing a site that will let me search for what’s out there doesn’t seem logical.
You could always write something and submit a merge request:
Based on my reading of the other comments here their is a huge push against this type of functionality from folks that understand the service better than I do. I doubt any contributions from an outside perspective would be at all welcomed.
It should be able to search a list of communities available on other servers its federated with.
This would be a very simple feature to implement and should not cause significant overhead.
TIL. That explains why I couldn’t find some things.
Someone will implement it.
The protocol itself is decentralized. Which is good.
If a app wants to use a central service to search thats a option available to them.
Userbase don’t care about how the tech works under the hood - user base sees no content and goes back to Reddit.
The original poster asked why.
I was answering why
If they can’t bother with investigating the platform for 10 minutes, I think they should stay on Reddit and keep complaining about the awful app and website over there.
Winning tactic. I’d like to stay a place with people with knowledge and interesting viewpoint, regardless of their ability to find search services on other websites to locate content.
At this very moment, there is a choice between two options
- an easy to use place managed by a company that see user data as resources to be sold to advertisers
- an emerging platform where some features are still being implemented, but without any tracking of its users, and managed by volunteers
Hopefully in the near future some features such as the one highlighted by OP will be integrated in the platform, but right now, it’s not, which is why I said that if people cannot search a bit about the current state of Lemmy, they should probably head back to Reddit. And I say that hoping that once the platform is polished enough, they’ll come back.
How does posting and reading posts work - and how do you know that nobody tracks their users? I was under the assumption that admins of a node have totals access to data going in/out/through their instance.
Just have a look at the data accessed by the apps, both stores display them. It’s something else than the Reddit app.
There might be some data agregation on the server side indeed, but compared to the ads promotion machine than Reddit has become (and even announced openly, with subreddits now being platform to promote products), it’s a completely different story.
What app? As far as I’m concerned there’s no reason to believe that fediverse users aren’t tracked. Probably not all, but where there’s users interacting with each other discussing different subjects there’s money to be made, and data to sell to AI companies for training.
Thanks for the heads up.
I’m not exactly sure what features are up to the admins and which are standard. If there’s a server that implements this and mine doesn’t, I can definitely see myself switching.
Based on https://lemmy.fediverse.observer/list, your instance has 29 active monthly users. You might indeed want to switch to an instance such as lemm.ee and its 3602 amu. You can use https://github.com/CMahaff/lasim to do so.
Sh.itjust.works, sopuli.xyz and reddthat.com are also solid options. I’m on sopuli and my All feed is the same as on Lemmyworld
So, small instances are bad if you don’t want to rely on third party services to find other communities?
Gee, what great design. Thank you for defending it for me and telling me to just switch to a different server. That really is the solution.
You seem to be quite a negative person.
That’s okay, but I don’t think that adds much to the discussion.
I offered you a solution to the very issue you are facing. If you are unhappy with it, you can probably just leave this place.
Have a good day.
This is a solution for 1 single person. It doesn’t solve the greater problem that the experience is VERY inconsistent for new users depending on whether they got lucky and first joined a large instance or if they got unlucky and joined a small instance. It also doesn’t retroactively repair the horrible first experience that new users have with lemmy.
What you offered really isn’t an solution as much as it is mansplaining…and that’s not really helpful at all when what we’re talking about here are the perceptions of new users.
You didn’t really offer me a solution nor did you answer my question. You also made the sny remark, “If they can’t bother with investigating the platform for 10 minutes, I think they should stay on Reddit and keep complaining about the awful app and website over there.”
You’re unwilling to take any criticism of lemmy. You are the kind of person that prevents it from getting better and more accessible to a general audience.
I’m sorry for not giving you praise for not contributing anything of value.
Forgive what is probably a silly naive question…
Can someone point me to an explanation of the federated architecture of lemmy? I haven’t found one yet that has helped me build a good mental model. I either get a step-by-step startup guide, or discussions on the merrits/demerits of a distributed system.
I think I’ve pieced together that it’s basically independent “instances” of the machine each with their own communities within. Sort of like if there were multiple instances of reddit, each with its own r/aww or whatever. I don’t yet understand, however how these interact/relate/ovelap/collaborate…which I think is the basis for this thread.
when a user (let’s call them Kim) on one instance (let’s call it “Works”), subscribes to a community on another (let’s call that one “World”), Works creates a copy of the community on its own database. It also asks World to notify it when there is an update to the community – when there is a new post, new comment, up/downvote, something gets deleted, etc. Kim can now browse and interact with the community on Works. Works will also notify World when Kim does something in the community so everything syncs and everyone sees the same thing.
So really, the problem OP is describing is simply a natural consequence of communities not existing on Works until someone subscribes to it.
Thanks, this makes sense. So, the last thing I’m wondering about is the redundancy/exclusivity of communities. For example, could there be a community called ‘gardening’ on the “Works” instance and also an independent community by the same name on “World” (before anyone is mutuallt subscribed)? Seems like it could… And if so, what happens when someone cross subscribes to ‘gardening’.
Specifically, (from a user experience standpoint) do these redundant communities coelesce into one? Because some of the benefit of these communities (particularly the more niche) is pulling together the experts into one community.
The gardening community on World will be called gardening@World on Works. they will continue to be distinct communities, and you can subscribe to either or both independently
All you need to do is subscribe to it yourself. You don’t need to rely on someone else. You can find the place with the search feature on your own, then subscribe so it starts getting pulled in all.
