This past September, my team at Citrusbyte embarked on a research project. Our mission was to evaluate the maturity of IPFS’s network and tools with the current frontend development tech stack and to weigh strategies and approaches for building decentralized apps.

To achieve this we planned on building a clone of Vine using Offline First and Distributed Web technologies. Vine was a short-form video hosting service where users could share six-second-long looping video clips. You can think of it as the “Instagram for GIFs.” If the necessary networks and protocols were mature enough, we would deliver a decentralized app where users could create GIFs and share them with friends. Decentralized Apps can also be called Distributed Apps, which people refer to as “Dapps.”

What is a Dapp?

Centralized, decentralized and distributed

Most people are familiar with the term “application” as it pertains to software. There are millions of software applications currently in use, and the vast majority of web software applications follow a centralized server-client model. A Dapp has its backend code running on a decentralized peer-to-peer network. Contrast this with an app where the backend code is running on centralized servers. Dapps will power the next wave of computing.

As explained in my coworker Chad’s talk, a Dapp is an app that works in the most extreme network conditions. Can you imagine horrible network conditions such as incredibly high latency, extremely slow connectivity from being very far from the server? Picture yourself on Mars if you want to easily imagine such conditions, apps built in this way would even work on a future Mars colony!

What exactly is decentralized about them? Two things: Data and Compute.

  • Data: Rather than storing all of your data on one company’s servers, and always fetching all of that data from said company, Dapps allow you to store your own data, or to fetch it from whoever already has it nearby.
  • Compute: Rather than using one company’s “cloud” for doing data processing, transactional logic, and other computation, Dapps make use of emerging new networks that don’t rely on the goodwill and stability of any single actor, but instead leverage the computing resources of the entire network.

For our GIF-based research, we planned mostly to explore the Data side of building a Dapp. We especially wanted to explore using IPFS — the InterPlanetary File System.

Our design goals:

  1. Build an app that runs in the browser, and requires no additional downloads (no need for extensions, native apps and the like)
  2. Store all data locally, on users’ individual machines
  3. Broadcast data directly from one user to another, with no servers intermediating

The second design goal clearly marks this project as research — we expect the data used by our website to grow too quickly for this storage strategy to be practical for a consumer app. Long-term, there are ways to work around this problem — we could offer users the opportunity to back up their own data on a device plugged into their home router, or we could allow them to pay other people to store their data on the IPFS network using Filecoin. But those sorts of long-term solutions aren’t ready yet, and are outside the scope of our research.

“Beware of the dragons”

What is IPFS, anyway?

There’s a lot to say, and they’ll tell you about it better than we can. But to start out, it’s important to understand that IPFS is a new protocol. A protocol is not a library; it is not a single tool. Instead, the protocol can be implemented in any language you care to write it in.

The IPFS team’s primary implementation is written in Go, a language that would work well on servers or on a desktop via a downloaded app. But per our design goals, we need the whole thing to work in a browser, no downloads required. To this end, we can use js-ipfs, a JavaScript implementation of the IPFS protocol which works in Node or in the browser.

IPFS as a whole is still in alpha, with known bugs and missing features. And though it is also built and maintained by the core IPFS team, the JS implementation is even less mature than the Go implementation. On the Project Status section of their README, they say explicitly that there’s “lots of development happening” and to “beware of the dragons.”

Well! I was named after Saint George, slayer of dragons. Bring ‘em on!

Our first dragon: the build process

  • Created a new React app with create-react-app
  • GIF creation with gifshot
  • Storing GIF data locally with js-ipfs
  • Broadcasting GIF data to others using the app with ipfs-pubsub-room
  • Merge real-time interactions and solve potential conflicts using the library YJS

It was working on our local machines and looking great.

And then: we tried to deploy it.

$ yarn build
yarn run v1.2.0
$ react-scripts build
Creating an optimized production build...
Failed to compile.

Failed to minify the code from this file:

./node_modules/cids/src/index.js:23

Read more here: http://bit.ly/2tRViJ9

error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

The shortened bit.ly URL redirected us to a section of the create-react-app README with the following:

npm run build fails to minify

You may occasionally find a package you depend on needs compiled or ships code for a non-browser environment. This is considered poor practice in the ecosystem and does not have an escape hatch in Create React App.

To resolve this:

Open an issue on the dependency’s issue tracker and ask that the package be published pre-compiled (retaining ES6 Modules). Fork the package and publish a corrected version yourself. If the dependency is small enough, copy it to your src/ folder and treat it as application code.

Apparently we were using a dependency that didn’t play nicely with create-react-app’s minify logic.

How to slay a dragon

As create-react-app runs its build script to prepare code for a production environment, one of the things it does is use UglifyJS2 to make the code as small as possible. The version of UglifyJS2 that it uses does not understand JavaScript code written in newer es6 syntax. Only es5 syntax and older.

Indeed, when we investigated where exactly our build script was failing, it failed on keywords like class and let — keywords that only exist in newer versions of JavaScript.

We tried to solve this in two different ways:

  1. Transpile all dependencies to ES5 syntax ourselves
  2. Remove minification altogether

Strategy 1: Transpile all dependencies to ES5 syntax ourselves

I thought “Hey, it’s probably just one or two libraries that are behaving poorly like this. How long could it take to manually add each one to a script that transpiles each to an older version of JavaScript?”

And I started adding them.

And each time, I’d get a little further. The build process would fail at a different spot. But with the same error.

And each time, one by one, I continued. I had to find the last dependency that needed transpiling. With each try I was sure I had reached the last one but the error kept showing up. Eventually the error stopped. I had built up a list of 255 files that needed to be transpiled! From 52 different dependencies!

We didn’t exhaustively investigate all of them, but most of these problem libraries are part of the IPFS ecosystem. It appears that most of these use a tool called AEgir — also built by the IPFS team — for their build process. This would explain why so many of our dependencies fail.

I persevered. 225 problematic files later, I got our build script to get past that problem…

…And onto a new one.

Webpack bundle failing at CID: Unexpected token: name (CID)

We had actually come across a Github issue about this one before, when researching our initial problem. Now we could finally make sense of it.

It turns out that transpiling our dependencies to ES5 wasn’t enough. The minification step also removes constructor names, which causes the problems explained here.

So then. Strategy 2.

Strategy 2: Remove minification altogether

In order to solve this we had to eject the create-react-app tool.

create-react-app wraps up various complex logic related to webpack, babel, and other tools. It’s great. When you run create-react-app eject, it stops hiding all that logic. It dumps Webpack and Babel configuration files into your codebase.

This gave us the flexibility we needed to remove minification altogether.

As you might expect, this increased the size of our built bundle by quite a bit. In fact, from 1.5MB to 3.8MB.

But it worked. We can at least deploy our app now.

We didn’t like this solution. We tried to avoid it. We tried to use the latest version of the minify tool for webpack (UglifyJS2 version 3) with a set parameter to maintain classnames (keep_classnames). It would get us around the constructor name problem but another one would show up: Uncaught SyntaxError: Unexpected token (. We did not reach far trying to debug this new error.

We decided to cut our losses, and move on with an unminified build.

The two headed dragon

Perhaps we’re being generous with ourselves. Maybe what we just described wasn’t so much slaying a dragon, but merely running away from it. Well, we felt like we had slain a dragon!

But just when we thought we were past it, another error appeared in our consoles:

Mixed Content: The page at 'PUBLIC_URL' was loaded over HTTPS, but attempted to connect to the insecure WebSocket endpoint 'ws://star-signal.cloud.ipfs.team/socket.io/?EIO=3&transport=websocket'. This request has been blocked; this endpoint must be available over WSS.

So, something to do with our app connecting to an insecure WebSocket (WS) instead of a secure one (WSS).

We found out we were dealing with a two headed monster! We tried some quick hacking, but nothing. We had wounded the creature, but it had resurrected.

In addition to this error, our app felt slower than it used to! After creating a new GIF and clicking “save”, it would take half a minute before it showed up in the list. It would take half a minute to show up on our friend’s devices. Why was it so slow? Did it have something to do with this WSS error?

Then we found this comment. The team at IPFS are constantly updating their code, we probably had cached our IPFS repo settings from an old codebase and that caused the error to appear.

We applied the suggested solution and the beast finally surrendered. Our beleaguered app was working again. We could share GIFs with each other, with no servers in between us.

But it was still slower than it used to be…

IPFS, their network, and tools are a work in progress.

Developer WIP

As mentioned in the above-linked forum post:

Performance improvements are being actively developed, but these things take time.

These are the problems we found during our Dapp development:

Development using create-react-app and js-ipfs will produce a build error during the minify step. Such error happens because the tool used to minify (UglifyJS version 2) is not able to understand code written in es6 syntax - js-ipfs and some of its dependencies have code written in es6 syntax.

  • Transpiling the code written in es6 syntax into es5 syntax will make the minify step succeed, thus the build script will succeed too.
  • Using another minify tool (one that is able to understand code written in es6 syntax such as UglifyJS version 3 or babili) will make the minify step and the build script succeed too.
    • Using either the js-ipfs transpiled or minified code will generate an error. Such error happens because the minify step removes classnames from the source code and some IPFS dependencies have checks in place for specific classnames.
      • By setting the corresponding configuration parameters (keepClassNames and/or keepFnNames) the resulting code will pass the classnames check.
    • A generic, very hard to debug error will appear if you’ve reached this far: Uncaught SyntaxError: Unexpected token (
  • Removing the minify step from the build script will make the process succeed.
    • The app will most likely more than double in size.

To sum up, follow these steps to build a Dapp if you want to use the IPFS toolset and modern frontend tools:

  • Execute the eject script from create-react-app.
  • Remove the minify step from the production webpack configuration file (this file can be found under the config/ folder).

Epilogue

Even when we used an older version of our codebase, from before removing the minification step, it was still sluggish. But… we hadn’t changed anything. Why did the app performance change, even with the same codebase?

While we were starting this project in late September, a political situation was becoming tense outside my window here in Barcelona, Catalonia, Spain. A planned independence referendum was about to take place: Catalonians were going to choose whether or not they would continue to be part of Spain or undertake their own autonomous governance. Spain’s central government had other plans. Disruption would take over.

On the days prior to the event some measures where taken: on Wednesday morning, police entered the offices of the .cat internet registry’s headquarters in Barcelona and seized all of its computers. They arrested six members of the staff and held four of them for two days. Finally the CTO was accused of sedition.

Then, the Catalan government used IPFS to sidestep Spain’s legal block.

It turns out, the way the Catalan government used IPFS to skirt Spanish censorship had attracted some attention. The IPFS network grew quickly from having about 1000 nodes to having about 6000.

But neither the Go implementation nor the JS implementation are built to deal with this increased scale. At least not yet. That’s coming!

We were very happy to see how these tools can help in real situations happening, like the Catalonia referendum. But there is more work to be done.

We hope our tale of the journey through the difficult build process — the js-ipfs library, create-react-app, webpack, uglify, and the rest — can help other people experimenting with these technologies. We also hope it might spur library owners to make their tools play more nicely with others in the wider ecosystem.

We encourage others to try out these new technologies. Let us know what dragons you find!