Building a highly scalable microservice architecture

Hey everyone, today will be about a project I've been thinking of for a long time. But before that, just some news.

I'm still in South Korea, and today, some friends will join me for two weeks, so I will chill a bit, with less work as the past weeks were about developing stuff.

The genesis

This project is called Onyx, and I've been working on it since 2013. My goal was to make a personal assistant like Jarvis in Iron Man.

At this time, it was maybe too much as the technology wasn't mature enough regarding AI and stuff like that, and because I was 13.

But I built a system like that that helped me manage my home like changing my light colors, managing my schedule, weather, and stuff like that. It was a simple PHP app doing that.

After that, I added a kind of AI like a chatbot and managed it using my voice. It wasn't working that well as the open-source world about that wasn't good.

Then, Google just released the Google Home ( yes I was trying to build it years before ) and I decided to forget about this project, I was a solo developer, without any marketing skills and I was young.

Back in 2020, I tried to make a new version of the project, in Python and Javascript, with a plugin system, and it was quite cool. But for the client side, the plugin system wasn't working well as it was impossible to share the UI client side from different plugins, and then Webpack Module Federation appeared, but the technology was very new and I wasn't that good at helping them build it.

So I gave up again.

Now

We are now in 2024 and I still have this idea behind my head. The aim of this project is to build for me a framework to build anything and create a plugin for each stuff I want to incorporate, like an assistant.

For instance, I can create a plugin to help me manage my bug bounty programs. Another for my schedule and so on. And I wanted to give it another try but in another way.

At first, the project was more about making a home assistant. Now, it will be more about building a whole framework, scalable, of different microservices to help anyone who wants to build stuff and having a nice dashboard to manage everything.

So here we are.

Architecture

The past two weeks were about to figure it out how to build such a system. It's been a while since I developed huge stuff and I wasn't updated about all the new ways to build an architecture.

I discovered the monolith architecture vs microservice one. And I dig deep into everything, reading books to understand them the best way and how can I use that to build mine.

I will describe what will be the architecture. It was after many failed attempts and tries with a lot of headaches, but it was worth it.

For this architecture, we will have the main app which consists of a gateway and a web part.

This gateway will be made in NestJS and the Web part ( which is the dashboard ) will be using NextJS.

They will communicate using a REST API.

Now the important part is how to make a plugin system, and that was the hardest part. Because I wanted each plugin to be fully independent, and working on any server.

This is called the microservice architecture, and it works great for the backend part, but not for the frontend part.

So for the backend part, it will consist of using NestJS ( or anything else ) and each plugin will communicate with each other using an event-based system, using RabbitMQ.

The dashboard will be able to communicate with them using the Gateway that will redirect each request to the good service.

The problem was that I also wanted the plugins to be able to share UI-like widgets or pages. And it wasn't that easy.

You can use the micro frontend framework using the webpack module federation, and it's now better. But this is still experimental, and quite huge. For instance, for a simple weather plugin, I don't want to set up a full nextjs app just for a simple page.

So I decided to use the method the Caido.io team used for their plugins. Building an SDK and simply importing a minified script.js for each plugin that will contain the pages and widgets. This way, it's possible to create a front in any language, and still use the Onyx UI and API.

It was quite hard building such a system, but now it's working pretty well.

Oh and as I see this as a second brain, I called the plugins neurons just like you build your brain by adding new neurons to it.

Ok but what's the aim of all of it?

It's nice to build such a system, but why and how it can help me and you too?

Let's take an example of a neuron I want to build next. And we will use the bug bounty world as I want to build automation around it.

With this system, I will be able to build a fully scalable recon system by simply building simple neurons. For instance, a neuron to analyze javascript, another to monitor them, another to fetch new subdomains, and so on.

Each of them will be separate projects, and they can be built in any language. They just need to follow the guidelines to be connected to the gateway.

This way, I will have a single dashboard, with all of my information, for each stuff that will be in any server working on something specific.

The main framework is working, but it's still very experimental, so if you want to try that, read the code carefully and don't use it in production.

This new version is very early, but it's just to show you what I'm currently working on. And it will help me a lot in the next months to build my automation system.

If you have any questions about that, feel free to ask, and if you want to follow the project to get news, here is the website :