“Hope this email finds you well” is how 2020-2021 has been in a nutshell. Since we’ve all been working remotely since last year, actively collaborating with teammates became one notch harder, from activities like brainstorming a topic on a whiteboard to building documentation.
Having tools powered by collaborative systems had become a necessity, and to explore the same following the principle of build fast fail fast, I started building up a collaborative editor using existing available, open-source tools, which can eventually be extended for needs across different projects.
Conflicts, as they say, are inevitable, when multiple users are working on the same document constantly modifying it, especially if it’s the same block of content. Ultimately, the end-user experience is defined by how such conflicts are resolved.
There are various conflict resolution mechanisms, but two of the most commonly discussed ones are Operational Transformation (OT) and Conflict-Free Replicated Data Type (CRDT). So, let’s briefly talk about those first.
The order of operations matter in OT, as each user will have their own local copy of the document, and since mutations are atomic, such as insert V at index 4 and delete X at index 2. If the order of these operations is changed, the end result will be different. And that's why all the operations are synchronized through a central server. The central server can then alter the indices and operations and then forward to the clients. For example, in the below image, User2 makes a delete(0) operation, but as the OT server realizes that User1 has made an insert operation, the User2’s operation needs to be changed as delete(1) before applying to User1.
OT with a central server is typically easier to implement. Plain text operations with OT in its basic form only has three defined operations: insert, delete, and apply.
Source: Conclave
“Fully distributed OT and adding rich text operations are very hard, and that’s why there’s a million papers.”
Instead of performing operations directly on characters like in OT, CRDT uses a complex data structure to which it can then add/update/remove properties to signify transformation, enabling scope for commutativity and idempotency. CRDTs guarantee eventual consistency.
There are different algorithms, but in general, CRDT has two requirements: globally unique characters and globally ordered characters. Basically, this involves a global reference for each object, instead of positional indices, in which the ordering is based on the neighboring objects. Fractional indices can be used to assign index to an object.
Source: Conclave
As all the objects have their own unique reference, delete operation becomes idempotent. And giving fractional indices is one way to give unique references while insertion and updation.
There are two types of CRDT, one is state-based, where the whole state (or delta) is shared between the instances and merged continuously. The other is operational based, where only individual operations are sent between replicas. If you want to dive deep into CRDT, here's a nice resource.
For our purposes, we choose CRDT since it can also support peer-to-peer networks. If you directly want to jump to the code, you can visit the repo here.
As our goal was for a quick implementation, we targeted off-the-shelf tools for editor and backend to manage collaborative operations.
To keep it simple, we'll set up a client and server both in the same code base. Initialize a project with npm init and install the below dependencies:
CODE: https://gist.github.com/velotiotech/983eb97b92b560d4290165a96eb798eb.js
CODE: https://gist.github.com/velotiotech/b4e82fdca68d6f518ad541481a899240.js
This is a basic webpack config where we have provided which file is the starting point of our frontend project, i.e., the index.js file. Webpack then uses that file to build the internal dependency graph of your project. The output property is to define where and how the generated bundles should be saved. And the devServer config defines necessary parameters for the local dev server, which runs when you execute “npm start”.
We'll first create an index.html file to define the basic skeleton:
CODE: https://gist.github.com/velotiotech/f96264879f6fed49035a8dfee99cceed.js
The index.html has a pretty basic structure. In <head>, we’ve provided the path of the bundled js file that will be created by webpack, and the css theme for the quill editor. And for the <body> part, we've just created a button to connect/disconnect from the backend and a placeholder div where the quill editor will be plugged.
CODE: https://gist.github.com/velotiotech/2e6fcbf543a357c990b096fe0d5774c6.js
CODE: https://gist.github.com/velotiotech/ec1e994203ee2311b697556e8b6a60b6.js
CODE: https://gist.github.com/velotiotech/4a0d183bdc626a27424bb7c386c71eb5.js
CODE: https://gist.github.com/velotiotech/aa5730fba5f35fa78327ad61929c7321.js
For simplicity, we'll directly use the y-websocket-server out of the box.
NOTE: You can either let it run and open a new terminal for the next commands, or let it run in the background using `&` at the end of the command.
Start the client by npm start. On successful compilation, it should open on your default browser, or you can just go to http://localhost:8080.
You can find the repository here.
Conflict resolution approaches are not relatively new, but with the trend of remote culture, it is important to have good collaborative systems in place to enhance productivity.
Although this example was just on rich text editing capabilities, we can extend existing resources to build more features and structures like tabular data, graphs, charts, etc. Yjs shared types can be used to define your own data format based on how your custom editor represents data internally.
Did you like the blog? If yes, we're sure you'll also like to work with the people who write them - our best-in-class engineering team.
We're looking for talented developers who are passionate about new emerging technologies. If that's you, get in touch with us.
“Hope this email finds you well” is how 2020-2021 has been in a nutshell. Since we’ve all been working remotely since last year, actively collaborating with teammates became one notch harder, from activities like brainstorming a topic on a whiteboard to building documentation.
Having tools powered by collaborative systems had become a necessity, and to explore the same following the principle of build fast fail fast, I started building up a collaborative editor using existing available, open-source tools, which can eventually be extended for needs across different projects.
Conflicts, as they say, are inevitable, when multiple users are working on the same document constantly modifying it, especially if it’s the same block of content. Ultimately, the end-user experience is defined by how such conflicts are resolved.
There are various conflict resolution mechanisms, but two of the most commonly discussed ones are Operational Transformation (OT) and Conflict-Free Replicated Data Type (CRDT). So, let’s briefly talk about those first.
The order of operations matter in OT, as each user will have their own local copy of the document, and since mutations are atomic, such as insert V at index 4 and delete X at index 2. If the order of these operations is changed, the end result will be different. And that's why all the operations are synchronized through a central server. The central server can then alter the indices and operations and then forward to the clients. For example, in the below image, User2 makes a delete(0) operation, but as the OT server realizes that User1 has made an insert operation, the User2’s operation needs to be changed as delete(1) before applying to User1.
OT with a central server is typically easier to implement. Plain text operations with OT in its basic form only has three defined operations: insert, delete, and apply.
Source: Conclave
“Fully distributed OT and adding rich text operations are very hard, and that’s why there’s a million papers.”
Instead of performing operations directly on characters like in OT, CRDT uses a complex data structure to which it can then add/update/remove properties to signify transformation, enabling scope for commutativity and idempotency. CRDTs guarantee eventual consistency.
There are different algorithms, but in general, CRDT has two requirements: globally unique characters and globally ordered characters. Basically, this involves a global reference for each object, instead of positional indices, in which the ordering is based on the neighboring objects. Fractional indices can be used to assign index to an object.
Source: Conclave
As all the objects have their own unique reference, delete operation becomes idempotent. And giving fractional indices is one way to give unique references while insertion and updation.
There are two types of CRDT, one is state-based, where the whole state (or delta) is shared between the instances and merged continuously. The other is operational based, where only individual operations are sent between replicas. If you want to dive deep into CRDT, here's a nice resource.
For our purposes, we choose CRDT since it can also support peer-to-peer networks. If you directly want to jump to the code, you can visit the repo here.
As our goal was for a quick implementation, we targeted off-the-shelf tools for editor and backend to manage collaborative operations.
To keep it simple, we'll set up a client and server both in the same code base. Initialize a project with npm init and install the below dependencies:
CODE: https://gist.github.com/velotiotech/983eb97b92b560d4290165a96eb798eb.js
CODE: https://gist.github.com/velotiotech/b4e82fdca68d6f518ad541481a899240.js
This is a basic webpack config where we have provided which file is the starting point of our frontend project, i.e., the index.js file. Webpack then uses that file to build the internal dependency graph of your project. The output property is to define where and how the generated bundles should be saved. And the devServer config defines necessary parameters for the local dev server, which runs when you execute “npm start”.
We'll first create an index.html file to define the basic skeleton:
CODE: https://gist.github.com/velotiotech/f96264879f6fed49035a8dfee99cceed.js
The index.html has a pretty basic structure. In <head>, we’ve provided the path of the bundled js file that will be created by webpack, and the css theme for the quill editor. And for the <body> part, we've just created a button to connect/disconnect from the backend and a placeholder div where the quill editor will be plugged.
CODE: https://gist.github.com/velotiotech/2e6fcbf543a357c990b096fe0d5774c6.js
CODE: https://gist.github.com/velotiotech/ec1e994203ee2311b697556e8b6a60b6.js
CODE: https://gist.github.com/velotiotech/4a0d183bdc626a27424bb7c386c71eb5.js
CODE: https://gist.github.com/velotiotech/aa5730fba5f35fa78327ad61929c7321.js
For simplicity, we'll directly use the y-websocket-server out of the box.
NOTE: You can either let it run and open a new terminal for the next commands, or let it run in the background using `&` at the end of the command.
Start the client by npm start. On successful compilation, it should open on your default browser, or you can just go to http://localhost:8080.
You can find the repository here.
Conflict resolution approaches are not relatively new, but with the trend of remote culture, it is important to have good collaborative systems in place to enhance productivity.
Although this example was just on rich text editing capabilities, we can extend existing resources to build more features and structures like tabular data, graphs, charts, etc. Yjs shared types can be used to define your own data format based on how your custom editor represents data internally.
.svg)
.svg)
