A Gentle Introduction to Decentralized Storage
A blog post explaining Decentralized storage, how it works, and popular dStorage protocols & platforms like IPFS, Swarm, Filecoin, etc.
Hey, fren! gm. ☀️
In the previous article, we deployed some cool NFTs with the help of VRF and IPFS. We didn't go over IPFS or the concept of decentralized storage in depth then, but Decentralized storage is something that has fascinated me for a while now.
So, today we're gonna learn about Decentralized Storage (let's call it dStorage for short) and some of the most popular dStorage protocols in this blog post.
Let's dive in!
So, what is Decentralized Storage?
As its name implies, Decentralized storage is a type of storage system that does not rely on a central server or authority.
Unlike a centralized storage system which is managed & operated by a single centralized entity, decentralized storage is operated by a peer-to-peer network of user-operated nodes, each of which stores a copy of the data creating a resilient file storage-sharing system.
This makes it much more resistant to censorship and attack than traditional storage systems.
The Need for Decentralized Storage?
In modern days, cloud services, such as Google Drive, AWS, and Dropbox, have been the go-to option for hosting files and websites.
While these services have helped individuals, startups, and large companies from the hassle of managing their own storage infrastructure, the centralized nature of such services present some deep flaws.
One of the many flaws of Centralized Storage services is that they are easy to censor. If a government or other authority does not want certain information to be out there, they can simply order the centralized storage providers to remove it and not store it in the future.
Furthermore, they can even make changes to the contents of the data stored, resulting in poor information integrity.
Another flaw is that centralized storage services are vulnerable to attack. If hackers can gain access to the servers of a centralized storage provider, they can potentially access and even delete all of the data that is stored on those servers.
Since these services have data hosted on centralized servers, having a single point of failure means any outage can result in temporary, or even permanent loss of data.
Decentralized Storage provides a solution to all these problems by providing a censorship-resistant, secure, distributed, efficient, robust, cost-effective, and resilient form of data storage.
How does Decentralized Storage even work?
The Decentralized Storage model works by distributing a copy or a portion of the entire data across a peer-to-peer network of nodes, each of which is incentivized to store the data.
By storing data redundantly across multiple nodes, the decentralized storage system ensures data security and accessibility. When the same piece of data is stored across multiple nodes, so even if a couple of nodes go down, the data can still be retrieved from the rest of the storage nodes.
What's in it for the node operators?
We all have unused space available in our computer and mobile device storage. The decentralized storage system makes use of available storage space on the node operator's disk drives, and node operators are incentivized to rent out unused storage space on their devices to the decentralized storage networks.
That was a general overview of how dStorage works.
Decentralized Storage Protocols and Platforms
Well, for starters, we can even consider the Ethereum blockchain itself to be a decentralized storage system, and to some extent, it is, for code (smart contracts) storage. But when it comes to storing large amounts of data (those JPEGs, gifs, and massive MP4s), it becomes a bit unfeasible, not to mention gas-inefficient, since this isn't what it was originally designed for.
So, we need to look at other solutions that are tailor-made for storage purposes.
IPFS is a peer-to-peer protocol for storing, accessing, and sharing data in a distributed file system.
What does that mumbo jumbo even mean? Let's take an example.
We all love Twitter (the little bird app is so cool). To access Twitter, you put in the
https://twitter.com URL in your browser, and since it points to Twitter's IP address, we get what webpage the server on that address stores.
But if we put Twitter on IPFS, we don't get an IP address. Instead, we get a content identifier, or CID, to access Twitter that looks something like this:
Instead of finding data by its location, IPFS finds it by its contents.
How does IPFS work?
Right now, there's a problem that exists on the Web2 Internet that content is found by its location. You wanna Netflix and chill? Cool, go to
netflix.com. You wanna read CryptoShuriken's blog? Visit
cryptoshuriken.com. But what if the location of the content changes for some reason? 🤔
Content addressing fixes this problem. By content addressing, every piece of content has a CID, which is based on the data's cryptographic hash, that points to data in IPFS.
That means two exactly similar files will have the same CID. Any minute difference in the content will generate a completely different CID. IPFS uses the SHA256 hashing algorithm by default to generate CIDs.
IPFS uses content addressing to identify and find content, rather than looking at where it's located.
Filecoin is a peer-to-peer network built on top of IPFS, created by the same team behind IPFS, that incentivizes users to rent out unused storage space by rewarding Filecoin's native 'FIL' token.
In Filecoin, users pay for storage space, and anyone who wants to store other users' files can join the Filecoin network, and get paid.
Swarm is another decentralized storage protocol that is a part of Ethereum's holy trinity: Ethereum for computing power, Whisper for messaging, and Swarm for storage.
Swarm provides a completely decentralized storage infrastructure that allows people from all over the world to become storage providers and get paid while doing so.
Swarm is designed to be highly scalable and resilient, and to provide a platform for applications that require a high degree of security and censorship resistance.
The idea for Swarm was presented by Gavin Wood and its development is mainly funded by the Ethereum foundation.
Difference between IPFS & Swarm
While both protocols may look similar from a high level, there are subtle differences that exist when we dive deeper into the inner workings and philosophies behind these protocols.
Some of them are:
🔹 Swarm's core storage component uses an immutable content-addressed chunk store while IPFS uses distributed hash tables to find which peers are hosting the content.
🔹 Since Swarm is a part of the Ethereum holy trinity, it has deep integration with the Ethereum blockchain for the incentive system, whereas IPFS has no incentive system of its own and utilizes Filecoin to add an incentivization layer.
🔹 From a development standpoint, IPFS is much further along in code maturity, adoption, and community engagement as compared to Swarm. As a result, IPFS has a lot of content available in terms of documentation, whereas Swarm doesn't have a large documentation base (heck, even the info I found while researching about Swarm was from third-party blogs 😞).
Arweave is a decentralized storage platform that uses a new type of data structure called a blockweave. Blockweaves allow Arweave to offer scalable, resilient, and efficient storage.
So, what are Blockweaves? 🤔
Blockweave is the core technology behind Arweave.
We all know that a blockchain is just a chain of linked blocks that contains transaction data. Blockweaves are similar to blockchain in that they are a chain of blocks, but it contains storage data, linking to multiple previous blocks from the network
Blockweaves work by enforcing that the miners provide a 'Proof-of-Access' to old data in order to add new blocks.
Unlike a more traditional blockchain, where miners spend compute power and electricity and compete to mine a block in order to earn tokens, the Arweave network encourages miners to store & replicate valuable data in order to earn tokens.
The Permaweb, just like the traditional web, is a collection of interlinked documents and applications that are stored permanently. The Permaweb sits on top of the Arweave data storage layer.
Since the Arweave network is built on HTTP, just like the traditional web, web browsers have access to the data stored in the network.
Storj is a decentralized storage platform that provides secure, scalable, private, efficient, and S3-compatible decentralized cloud object storage.
Similar to other decentralized storage platforms, Storj connects people with unused bandwidth and storage space to those who need cheap, accessible, and private file storage.
Storj uses 'STORJ', their native ERC-20 token, to incentivize people who are renting out bandwidth & storage space.
Sia is another popular decentralized cloud storage platform that connects renters (who rent out storage space to host their files & applications) and hosts (who lend their storage space to renters) together in a peer-to-peer network. Sia has its own blockchain and the hosts get rewarded with Sia's own native utility token 'SiaCoin'.
Once the renter uploads their file to the Sia network, it gets split up, encrypted, and sent all over the world. The network ensures the files are always accessible by making multiple copies of them.
And since the files are split up into multiple pieces & encrypted, they are inaccessible to the hosts.
With the mainstream adoption of centralized storage networks in the past decade and with centralized cloud services, like Google & AWS, providing a fast & cheap storage infrastructure to individuals and organizations, threats of poor information integrity, privacy, and censorship have been omnipresent.
The decentralized storage model addresses the issue of centralized storage by distributing the data across a network of distributed nodes, each of which stores a copy of the data.
This makes it much more resistant to censorship & attacks with better information integrity and data availability than traditional storage systems.
- A Detailed Overview of Decentralized Cloud Storage - Leeway Hertz
- IPFS Documentation
- Filecoin Docs
- Swarm Docs
- Arweave Whitepaper
That's it for now, fren!
If you enjoyed reading this article, please consider subscribing here so that you get a notification in your mail whenever I post new stuff. Subscribing is free, and it gives me a ton of motivation to keep going.