Automation: n8n strikes back

Loic Tosser -

Almost 2 years ago, we introduced a new automation tool in this article: n8n, and we know that some people were using our article to deploy it in production. Here is the small issue: our deployment was made back in the days, and nowadays, a lot of improvements has been made, and I wanted to show you how to properly deploy n8n in 2022!

Architecture

Little advice for all people out there: throwing a software into a platform will always create problems! You want to know how it works, in order to ensure the continuity of your services!

Hope for the best, prepare for the worst!
graph TD A[API/Web] --> D[PostgreSQL] R[Redis] --> W[Worker] --> D A --> R

As you can see, you can split the architecture into 3 parts:

  • The Web/API, also called "main": it is in charge of deploying the web interface, all incoming endpoints/triggers, the API, and forward to the queue system
  • The queue system: n8n relies heavily on a tool that I have some problems with (especially after testing celery): BullJS. Bull has a lot of missing features, uses Redis (and only Redis) to transmit messages, and is going in a direction that I don't appreciate: an open-core solution! Nonetheless, it explains the new requirements for Redis and a worker system!
  • the database, which is going to store every configuration of workflows, authorization, credentials and executions!

How to

Step 0: Clone the repo

We are maintaining a repo on Github, that you can fork in order to deploy the n8n: n8n.kalvad.tech.

Step 1: Understand the repo

Inside the repo, you are going to find a first file which is interesting: package.json

{
    "name": "n8n-clever-cloud",
    "description": "n8n on clevercloud",
    "version": "1.0.0",
    "author": "Loic Tosser (wowi42) <loic@kalvad.com>",
    "scripts": {
        "start": "./run.sh",
        "main": "./main.sh",
        "worker": "./worker.sh",
        "http": "mkdir -p /tmp/http_n8n && http-server /tmp/http_n8n"
    },
    "dependencies": {
        "http-server": "14.1.0",
        "n8n": "0.177.0"
    },
    "engines": {
        "node": "16"
    }
}

As you can see, we have 2 dependencies: http-server and n8n. But why do we need http-server? Because we are going to split the deployment of the main and the workers, and Clever Cloud needs something to listen on port 8080

We also have 3 scripts:

  • run.sh: to debug locally
  • main.sh: how to run the main service, the central piece!
  • worker.sh: how to run the workers!

The main difference between run.sh and main.sh is a small parameter called EXECUTIONS_MODE and the associated Redis configuration!

the only difference between main.sh and worker.sh is the command!

# main.sh
./node_modules/.bin/n8n

# worker.sh
./node_modules/.bin/n8n worker --concurrency=$CONCURRENCY

Step 2: Prepare Clever Cloud

N8N can work with multiple "databases" (hum hum MongoDB), but as a sysadmin, I'm going to recommend you PostgreSQL.

On Clever Cloud, go to Create -> an add-on -> PostgreSQL -> DEV. Enter a name and you are ready for the database.

Then go to Create -> an add-on -> Redis -> S. Enter a name and you are ready for the Redis.

Then go to Create -> an application -> select your GitHub repo -> NodeJS -> Select the instance size (XS is ok) -> enter the name, and add -main behind and the description and click on create. Click on I don't need any addons. Then you can define your environment variables.

For your environment, you will need at least 4:

  • N8N_BASIC_AUTH_PASSWORD: the password to access n8n
  • N8N_BASIC_AUTH_USER: the username to access n8n
  • N8N_ENCRYPTION_KEY: key used to encrypt the credentials before they get saved to the database
  • CC_RUN_COMMAND: npm run main

A fifth field is optional: N8N_HOST which will allow you to define the host used for n8n, as the default value will be the domain name in cleverapps.io, which is not recommended.

Then go to service dependencies and add the PostgreSQL database and the Redis Database.

Finally, click on rebuild.

Then we need to deploy the workers!

Go to Create -> an application -> select your GitHub repo -> NodeJS -> Select the instance size (XS is ok) -> enter the name, and add -workers behind and the description and click on create. Click on I don't need any addons. Then you can define your environment variables.

For your environment, you will need at least 6:

  • N8N_BASIC_AUTH_PASSWORD: the password to access n8n
  • N8N_BASIC_AUTH_USER: the username to access n8n
  • N8N_ENCRYPTION_KEY: key used to encrypt the credentials before they get saved to the database
  • CC_RUN_COMMAND: npm run http
  • CC_WORKER_COMMAND: npm run worker
  • CC_WORKER_RESTART: always

Another field is optional: N8N_HOST which will allow you to define the host used for n8n, as the default value will be the domain name in cleverapps.io, which is not recommended.

Finally, we recommend the following parameters:

  • EXECUTIONS_TIMEOUT: 36000
  • EXECUTIONS_TIMEOUT_MAX: 72000
  • QUEUE_BULL_REDIS_TIMEOUT_THRESHOLD: 1000000
  • N8N_PAYLOAD_SIZE_MAX: 64

Then go to service dependencies and add the PostgreSQL database and the Redis Database.

Finally, click on rebuild.

Step 3: Enjoy

Let's resume the situation:

  • you have a main instance, with its own scalability
  • you have a Redis instance and a PG instance
  • you have your own set of workers which can scale up and down

Congratulations, you are ready for launching n8n in production!

You still have room for improvments, like having a dedicated endpoint for Webhooks, but with our current setup, you are going to be able to handle around 1kreq/s without problems!

If you have a problem and no one else can help, maybe you can hire the Kalvad-Team.