23O/ Scuttlebutt - the gossiping protocol
Scuttlebutt - The Gossiping Protocol
Thursday 23O
Convener: zelf (Zenna)
Notes-taker(s): Charles E. Lehner
Tags for the session - technology discussed/ideas considered:
Data Sovereignty, Distributed Systems, Community, Trust, Applications, Fun, Standards
Discussion notes, key understandings, outstanding questions, observations, and, if appropriate to this discussion: action items, next steps:
Zelf (Zenna), Project manager for Scuttlebutt NGI - developing on the backend of Scuttlebutt
Scuttlebutt is a gossipping protocol, in both the technical and social sense. It is a protocol born of the IRC chat “mad scientists” that D. Tarr was part of. D. Tarr is a wild mad genius who lives in New Zealand on the boat. That is the origin of its creation - to be able to offline compatibly communicate with friends. It took off and grew rapidly into a larger social network. Now as far as we know about 25000(?) nodes/people…
[Presentation slides unable to be presented because of technical difficulties]
Offline-first social protocol based on social trust between people and nodes. Some of the reasons why the re-design of an internet architecture is needed is Data Sovereignty - people need to own their own data - and that goes for Communities as well. Maori communities using Scuttlebutt as part of Ahau to keep sovereignty of their cultural heritage and have the data move in a way that aligns with their culture.
Internet “centralized”... We don’t own our data but need a continuous upstream to access it. Cumbersome and unnecessary for access to our own data - and our friends’.
Scuttlebutt is as p2p as we can imagine without NAT punching(?). Development in the past year: “Rooms” (?).
Rise of cyber warfare… Google services down for one hour… November(?) 2020… People couldn’t even turn on lights in their own home because they relied on the connectivity.
Last week, Facebook and Whatsapp and Instagram were down for 6 hours, which is also due to a large extent to the Internet architecture design… continuous upstream, energy use…
Current Internet Infrastructure is quite exclusive, requires people to have high-tech devices to be able to access, they way we are used to in the Western World… 48% of the world does not have Internet access… easy to forget [when we have] open space technology online…
Climate crisis… 20 years collapse of society left?
Aaron: We’re good at patching things…
Z: We’re trying…
Our solution: open source trust-based(?) protocol…
Humans trusting humans with their communications… offline access to data (Data Sovereingy), open source (developed for and by the community)...
Illustrations....
Aaron: What’s your relationship to the project?
Z: Project manager for recent core development of the protocol. I’ve been doing that for the past year through NGI pointer. A team of 6-20 people working on it in various ways (core group of 6 people).
What is Scuttlebutt? How does it radically change the pattern of communication and Internet as we know it?
… It’s a combination of 3 different elements:
Version control (like GitHub)
Torrent-based sharing (kindof - sharing small packages of data)
Ledger-based storage.
These three, together with the format of how the data spreads - which starts with you, having a subjective perspective… We’re each in our world… We have friends… You choose your friends, and they choose you… effectively establishing a data transfer agreement.
You store data on your computer… Each node acts as a server.
Your friends’ friends - you also share their data and
Your friends’ friends’ friends’ you see their data but do not store it.
Subjective view of the network: what you receive you have access to even offline - or over Mesh networks.
Aaron: Sounds like a lot of copies of any particular message…
Z: Yes…
A: Is the data explosion a problem?
Z: It could be if very large files are uploaded… That was a problem in the early versions of the protocol…
Media files (“blobs”) take up a lot of space. “Messages” (text-based) do not take up a lot of space.
Another solution developed in past year (“partial replication”)...
If someone out of your social proximity, you don’t know they are on the network, and they don’t know you are… no way to know unless there is some social path of trust…
Aaron: Is there a concept that smells like a retweet? Reshares… may be 5 steps away… deliberate action, chain…
Z: Could be. So far none of the applications have that functionality, but hypothetically it would be possible.
You can also have separate network keys - that is completely private unless you have access to that key. Highly private if you want it to be.
That’s the basic outline of how Scuttlebutt functions.. Above mentioned 3 points, and the pattern of social trust of spreading data
Radical new way of having a social network… Also meant that there are many “hacky”, “messy” aspects - very different from usual user experience… no passwords, for example. There’s a hack called “pubs” - an automated node/peer in the network to act as a constantly online peer… to enable spreading data…
Rooms developed to address that in part… a relay server that does not share any data… just connects peers.
Aaron: Like a Torrent tracker?
Z: Yes
Also implemented “partial replication”. Problem: have log you can’t change or delete data, might continue together, people continue talking for 30 years… that’s going to be a lot of data, even if it’s text messages.
So we implemented partial replication. That means you can have multiple logs of identity - and can choose which logs should be replicated… which kind of media it is you replicate… you can put a max cap on the data replication, can make ephemeral logs… that was a big redesign of Scuttlebutt from the protocol perspective. Also a new indexing format… 10x faster. If you tried joining Scuttlebutt you might have seen it took an hour to do the initial sync… Now we have it down to about 8 minutes.
Aaron: is there a process for discovery? How to find peers?
Z: Yes, that’s what Rooms and Pubs are for.
Rooms are quite new, just finished last week(?)
Some rooms are running, but you may need an invite.
Aaron: like with Git… if I want to do a “shallow clone” I can limit by directories or by number of commits… is it all different channels and I pick some, or go back 100 MBs…?
Z: Everything possible, depends on UX perspective. From protocol perspective you can do both - but it depends on how it’s implemented.
Aaron: what does the protocol look like? Do I have to follow messages back to the beginning of time?
Charles: traditional replication…
Synchronization
Sync from log id and byte...
Security audit of new protocol
15 people designing the protocol together, trying to see what would be possible to build within the time frame, and what is important to build right now.
Went for partial replication but also made “fusion identity” - still a bit out there but has something to do with “tangles”... to link two identities basically. The identities can then share a log. Design is finalized but not developed yet.
We’re not an organization or company… just doing it as an interested [community], we’re decentralized… next phase: “P2Panda”
Aaron: … Fusion identity…?
Z: A difficulty for Scuttlebutt because it’s traditionally based on [device keys].
Solved [many problems]. At this point pretty much starting to wrap up the package for our goals for offline-first trust-based communication protocol.
A: Ephemerality solved? Deleting data in distributed systems quite a thorny problem. If many nodes replicate, how do you know they deleted it?
Z: My understanding is that when you create a new branch you have to decide if it’s ephemeral. If you decide that it has a timer on it…
C: Could be based on trust?
A: Could be… need to ask [...], doing security audit… quite critical. But yes, social trust is important.
A: Need to be careful who your peers are…
Z: Also it is now possible to be incognito on the network so that only some peers see you and replicate, to avoid unwanted interactions.
A: Solution for large blobs?
Z: Long-time conversation... one is to offline to another distributed protocol (such as hypercore) - another is to limit your local storage.
A: IPFS? Content centric.. Scuttlebutt message-centric…
Z: Scuttlebutt is more community-oriented. IPFS more similar to hypercore in my understanding… a very different way of distributing the data… Different individuals choose to distribute certain data. IPFS used more for business backend solutions. Different communities. Scuttlebutt doesn’t do VC funding(?)
A: Thanks!
Brent: Thanks. Could you add slides to the notes when they become available?
Z: Yes.
A: Links to protocol specifications?
Z: Many links… trying to clean that up.
A “treasure map”
Scuttlebutt developed in a distributed fashion, different values, different ways of working…
---
[Discussion about Twitter BlueSky]
Aaron: Value of network control
… Dimensionality of trust… e.g. speech mostly commercial - do you want to see it or not?
… Closed systems causing a lot of problems that open systems can solved… could be the future of platforms. Hard to get things off the ground, could gain learnings…
… like Windows NT was an experimental kernel but now it’s “the kernel” - but they also had a … drawbridge kernel… then they killed that project and moved the learnings into the main product.
---
Scuttlebutt compared to git?
Z: Subjective sharing of knowledge…
A: Necessary for distributed system. One global truth means need consensus algorithm, means need to connect to Internet…
Enddy: Can identity cease to exist? If people decide not to store it…
Z: No, it always exists at least for yourself.
… One person once failed to get online, started using it as just a diary… Then they connected with someone and their “diary” went public!
… If you have friends, it exists with them too.
Dmitri: Huge fan of Scuttlebutt… tried it last year… but lost key…
… Is there movement towards more like DIDs…?
Z: Yes…. good to write down in notebook… (I lost my first one too…)
A: Key generated randomly?
Z: Yes… can use [mnemonic] words
… Dark Crystal project, developed a beautiful way of sharing your key and getting back through social trust…. But unfortunately couldn’t figure out to send it back without something like email…
… DID part: scoped out of current R&D in past year… now handing it over to the next team of developers (“P2Panda”)... Scuttlebutt is as distributed in organizing as it is in protocol…
Domain name owners?
… Individuals, no foundation
Fusion identity? Can’t say how specifically it’s designed, but can say by linking identities together you can have the same messages go to different devices.
A: What do you use for NAT traversal? STUN and TURN servers?
Z: Not doing firewall bypass… we’re doing services that don’t store…
C: Room servers are like TURN servers…
End-to-end encryption
A: HTTPS?
Z: Need to send you the documentation.
A: Most projects using distributed hash tables…
Dmitri: do you have an ask for us? As developers or standards community?
Z: So many ways… but hard to say…
A: Standards people tend to be good at writing a spec.
Z: We’ve covered as much as we need at this point… Next would like to have mesh networks (not proprietarily). From a Scuttlebutt perspective, we are getting to a point where… documentation needs to be organized. Needs to be a larger security audit of the protocol…
Some parts have been audited - but not as a whole.
From a standards perspective - I’m new to that, would love guidance on that, personally.
Next steps: fusion identity - ready to go
[Dmitri, Aaron and Balazs talking about standards]
Aaron: … groups not having best practices on standards… how do you start?
Balazs: Hard… some standards written and not used… some broadly used but never received formal specification…
… “We made a new organization for this standard”
… Multiple approaches for standards… Every organization that works on them ends up having their own format, usually. Big boring background, political…
Audit Report: Secure Scuttlebutt Partial Replication and Fusion Identity
Fusion identity ^ (Not yet built) - Does not cover Rooms or the new indexing format.
Z: Standards…?
Dmitri: depends on how much time you have. Easiest way: join conferences like this. Then, join working groups or interest groups… like the DIF Interop WG (https://identity.foundation/interop/)… Many have bi-weekly calls. Getting in touch with community leaders [...] is good too..
Z: Great!
Balazs: https://identity.foundation/faq/ - educational page for going from new to decentralized identity to understanding why it gets complicated… with different layers and considerations, building up a stack, to be helpful for certain stages. From this you can get an understanding of the DID framework - without too specific/technical (hard to write…)
[Talking about Berlin]
[Talking about Boston]
E: Open Source?
Z: Yes.
A: Then you get bought by Oracle...
Z: But what is there to buy…?
Best way to engage with Scuttlebutt is to go on it.
People flow in and out depending on energy levels. It’s a very fluid community.
Best way to engage is with your friends.
Because of the decentralization… one must feel their way around the parts and pieces to see what to engage with… One thing always desired is documentation. (D: That sounds familiar!) Specifically, organizing the docs… it’s everywhere… including the repos themselves.
The primary thing is to fun.
One way to have fun: we recently built a demo app, where you can build whatever apps you want - and it runs over Scuttlebutt. A distributed application sharer.
E: I’d like to join...
Z: Just join and ask “stupid questions”...
A: I’ve got plenty of those…
Enddy: Big apps in production?
Z: Most maintained one currently is Manyverse - also building a desktop app. Patchwork has been “tombstoned” - not being continuously developed… beyond that there are many… Oasis (still maintained?), Patchfox, browser-based applications (popular when you can’t download whole applications), patchbay still used but not maintained.
