Open engineering / Infrastructure
Queue-based workflow execution
A self-hosted n8n study that separates the editor, job queue, execution workers and persistent data.
- n8n
- Docker Compose
- PostgreSQL
- Redis
- Python
Illustrative connections, not live system traffic.
Context
Workflow automation becomes an infrastructure problem when editing workflows, receiving requests and executing jobs all compete for the same process. In this public repository, I explore a queue-based arrangement for self-hosted n8n.
My role
This is a repository I publish under my GitHub account. It contains the deployment configuration, a small example workflow and a Python trigger utility. I present it here as an architecture study, rather than evidence of a production service or a measured capacity result.
Architecture
The Compose file defines an n8n main service, three worker services, PostgreSQL, Redis and a database seeder. PostgreSQL holds application data, while Redis provides the queue connection. Named volumes retain service data between container restarts.
The example workflow accepts a webhook and includes the worker label or hostname in its output. This makes the execution location visible when investigating how the components behave.
Decisions and trade-offs
Separating execution from the editor makes worker capacity an independent concern. It also adds a queue, shared configuration and more failure modes to operate.
The startup sequence deserves particular care: the included seeder drops and recreates the application database. I would replace that lab-style bootstrap with controlled migrations and recovery procedures before using this pattern with important data.
The worker definitions also need configuration review. Their environment mappings replace the mapping inherited through the YAML merge, so queue and database settings must be carried through explicitly. A service definition alone does not establish that jobs execute correctly.
What is verified
The public source contains the Compose arrangement, health checks, example workflow and trigger utility. It does not establish throughput, high availability, recovery performance or a working multi-region deployment. Kubernetes manifests and fuller disaster-recovery playbooks are listed as future work in the repository documentation.
What I would examine next
I would check job distribution, retry behavior, worker termination and database recovery with representative workflows. Those checks provide more useful evidence than a diagram or a successful container startup on its own.