Software development best practices

ArangoDB: Perfect Database for Projects with High Uncertainty

Time for reading: 9 min

We often deal with projects that face a high level of uncertainty. With networking applications, you can’t predict how many connections you’ll have in the future and how complex they’ll get. So it can be hard to pick an optimal tech stack. However, there might be a perfect database for such cases. And this database is called ArangoDB.

As we learned during one of our recent projects, it is fast, flexible, and has a fantastic community. But first, let’s take a look at the requirements we were facing when choosing our tech stack.

The challenge

Some time ago, an early-stage startup contacted our custom software development company to build a new type of social network.

The idea was simple. There are many startups who need funding and there are lots of investors who are interested in the tech sector. But bringing them together isn’t as simple as it sounds.

When searching for a startup that’s worth their money, venture funds have to consider dozens of parameters. And one of the most important factors for any investor is the team composition. Is it well connected? What skills do they have? What networks do they have outside of the company?

These are all essential questions but investors often have no quantitative ways to assess the team besides gut feeling.

And this is where our client’s idea comes into play.

They wanted to develop an app for the visualization of connections within and outside of startup teams. With social network analysis, investors would be able to assess startup teams based on quantitative methods.

visualization in d3 library

Different types of visualization available in the D3 library

The app had a number of requirements that complicated its architecture and made it hard to choose an optimal tech stack:

  • All connections for each user had to be visualized as a graph using the D3.js library.
  • 2 user types: companies and individuals (with the ability to add universities and other user types in the future).
  • A quick and efficient search that would match users with companies and individuals based on their skills and social connections.
  • All users should be able to share a part of their profile (i.e. fields in a form) into various networks (i.e. individual entities and field values needed to have different access rights).
  • Attributes of all entities in the database needed to be dynamic so that field values could be shared between different social networks.
  • The application should be able to recommend different social networks based on connection strength, the data you share in your forms and other attributes.
  • The final solution had to include a chat for all user types and a document library with the ability to share files.

social network visualization in d3

A social network visualized as a graph in the D3 library

Now, choosing a database that would satisfy all of these requirements was quite a challenge.

It’s possible to use dynamic field attributes in a regular SQL database but sharing field values between different social networks is a big problem. Creating additional connecting tables would’ve complicated the application’s architecture and slowed down the queries.

The result would be poor performance.

Now, the need to have different access rights to individual entities like social networks meant creating even more relationships between the database entities. The increased complexity was another reason to look into non-relational databases (also useful for the live chat functionality).

What’s more, our client wanted the app to suggest existing social networks to users based on information in their profiles.

And this is a very hard task for a relational database.

NoSQL vs SQL

Relational databases Non-relational databases
SQL Pros:SQL Cons:NoSQL Pros:NoSQL Cons:
StandardizationDifficult to architectureSchemalessNo query language standard
SimplicityPoor organization of diverse dataObject sizesNo Joins
Severity of structureLow performance on hyper-complex queriesArchitecture flexibilityWork with bidirectional relationships
Clarity of connections Great performanceAggregation
Complex queries with many relationships  Multiple relations queries
Joins   

As our app was based around social networks and graph visualizations, our choice fell on graph databases where relationships between data are just as important as the data itself.

Database entities are represented as nodes that can have as many attributes as needed. Each node has a direct pointer towards all other related nodes.

These connections between nodes, called relationships, are stored alongside the data.

They must have a starting point, an end node, a direction, and a type. Just like nodes, relationships can have attributes (e.g. weights, strengths, distance, or time intervals).

This network of nodes connected via relationships is known as a graph.

Graph example

A graph with nodes (circles) and relationships (arrows)

Graph databases shine at complex relationships-intense queries and datasets with many connections. They can search through the neighboring nodes having only a set of starting points and a pattern.

Our initial research pointed us towards Neo4j, one of the leading graph databases.

Its main downside was the high price for use in production. We had to search for another option.

And this option came in the form of ArangoDB, a relatively uncommon database made by German developers.

What is ArangoDB?

ArangoDB stats and reviews

Source: G2 ArangoDB reviews

ArangoDB is an open source native multi-model database. It supports graph, document, and key-value data models allowing users to freely combine all data models in a single query.

As applications become increasingly complex, you often need more than one NoSQL database. Using a multi-model database can simplify your architecture by combining several NoSQL types in a single infrastructure.

So, what are the main features that define ArangoDB?

  • Easy performance scaling

The database allows you to quickly adapt to increasing requirements for performance and storage with both vertical and horizontal scaling. It also supports the independent scaling of different data models and allows you to quickly scale down your application to save on hardware and operational expenses.

  • Consolidation

ArangoDB drastically reduces the number of components to be maintained making your tech stack much less complex.

  • Decreased operational complexity

Multi-model databases allow you to use different storage technologies that better fit the way your data is used by different components within your application (i.e. Polyglot Persistence).

ArangoDB uses binary JSON to store data. It is lean, self-contained, doesn’t allocate too much memory, and covers all of JSON plus dates, binary data, integers, as well as arbitrary precision numbers.

  • Powerful fault tolerance system enabled by default
  • ArangoDB Query Language (AQL)

ArangoDB allows you to access any data (regardless of its model) using a single declarative query language. It is quite similar to a standard query SQL language with some minor differences:

SELECT * FROM users;//SQL
FOR user IN users RETURN user;//AQL<br />
SELECT * FROM users WHERE active = 1; //SQL
FOR user IN users FILTER user.active == 1 RETURN user;//AQL

As a NoSQL database, ArangoDB features high performance in data storage and retrieval.

But to realize social networks in our application, we needed relations that are absent from typical NoSQL databases.

So we dug into ArangoDB documentation and discovered that it actually supports relations. AQL allows Joining related documents on the fly inside a query without affecting the app’s performance.

Our research showed that ArangoDB could satisfy all our requirements. A single database to cover the whole business logic.

Yet, we had no experience with ArangoDB. The database is comparatively young and has a small community. Moreover, there were very few integrations with frameworks meaning we’d have to write these components by ourselves.

As we analyzed the documentation, we came to the conclusion that all of the ArangoDB cons were canceled out by its pros.
So we took the chance and started the development.

companies using ArangoDB

Companies using ArangoDB in their projects

The result

By choosing ArangoDB, we could use a single database for all features: chats, relationships, document sharing, various types of entities (companies, persons, universities, employees, venture funds, etc.).

It can store all connections between nodes and construct graphs that will be constantly updated when you add a new connection.

With ArangoDB, we could easily architect dynamic field attributes. The database allows implementing relations in two ways: by making links to primary keys or via connections and graphs. As we discovered, it’s better to use connections when you have dynamic attributes. And primary keys are better suited for situations when you have a clear understanding of relations between different entities.

What’s more, ArangoDB has an excellent web-interface that allows writing queries and instant viewing of the resulting graphs.

arangodb web interface

ArangoDB web interface as shown in the documentation

The database supports all the modern deployment options (e.g. Kubernetes, Docker, and clusters for high-load).

It features a flexible and powerful query language (AQL) that covers all the needs of a modern application. The whole graph visualization process is performed with a single query.

Overall, it’s one of the most elegant query languages I’ve ever seen.

ArangoDB fully supports ACID transactions and has excellent performance.

The average search query takes around 0.1 seconds. Even the most complex queries that affect hundreds of social networks take less than half a second. Joining so much data in a relational database would’ve required denormalization or using Elasticsearch with a flat connecting structure to help with the visualization.

We achieved the same thing in ArangoDB without any additional effort.

arangodb performance graphs

arangodb performance

Source: arangodb.com

All our worries about the small community turned out to be unfounded. The database has awesome documentation. It includes over a dozen case studies that highlight success stories of various companies and a cookbook with detailed solutions for various problems.

Moreover, it has a great community in Google groups and an awesome Slack channel.

Once I had an issue with ArangoDB that I couldn’t solve on my own, so I asked a question in the Slack channel. Within 30 minutes I got a detailed answer from one of the database developers.

This is really awesome!

ArangoDB developers constantly organize free webinars that delve into the latest versions of the database which are presented in a very interesting and unusual format.

Conclusions

It was a great pleasure working with ArangoDB.

Of course, there were some difficulties like integrating the database with a framework but, overall, the experiment was a great success and the project was finished on time.

Now, when you choose a database for your project, it’s always worth considering ArangoDB.

If you face a high degree of uncertainty or have complex relations between the entities, ArangoDB might be the database you are looking for.

It has an excellent web interface for managing queries, a powerful query language for managing data, and VelocyPack (VPack) for storage and serialization.

Everything you need in one powerful database.

And if you need help with your next project, you can always contact MindK and schedule a free consultation.

Subscribe to MindK Blog

Get our greatest hits delivered to your inbox once a month.
MindK uses the information you provide to us to contact you about our relevant content andservices. You may unsubscribe at any time. For more information, check out our privacy policy.

Read next

Custom LMS Development: Complete 2024 Guide

Custom LMS Development: Complete 2024 Guide

Read more
How to Create a Job Search Engine: 5 Things to Consider

How to Create a Job Search Engine: 5 Things to Consider

Read more
macbook

SPA vs MPA: What Are Single Page Applications and When to Use Them

Read more