KodMQ ^βeta

Because development must be simple

Overview

KodMQ is a simple and easy to use Node.js background jobs. Everything is included, scheduling, CLI, UI and much more 🎉

Features

👍 Simple: KodMQ is designed to be simple to use and easy to understand
🔋 Batteries Included: KodMQ comes with a built-in UI to manage your jobs
🧘‍♀️ Flexible: KodMQ can be used with Redis or Prisma (In Development)
TypeScript: KodMQ is written in TypeScript and provides type definitions

Installation (Recommended)
Installation (Manual)
Usage
- Perform Job
- Schedule Job
Core Concepts
- Job
- Worker
- Concurrency
Environment Variables
Jobs API
Workers API
Sponsor
That’s it!

Installation (Recommended)

Just run this command and follow the instructions:

bun create kodmq@latest

Installation (Manual)

This installation guide assumes that you are using Bun and TypeScript.

If you aren’t using Bun, you need to replace bun run with node or ts-node in the package.json.

If you are using JavaScript, you need to change the file extension from .ts to .js and remove type annotations.

Let’s get started!

Install packages

bun install @kodmq/core @kodmq/adapter-redis @kodmq/ui

Create configuration file ./jobs/index.ts. Ideally, to keep your code organized, you should create a separate file for each handler and put them under ./jobs/handlers directory.

import KodMQ from "@kodmq/core"
import { Handlers } from "@kodmq/core/types"

const handlers: Handlers = {
  sendEmail: async ({ to, text }) => {
    // Logic to send email
  },
}

const kodmq = new KodMQ({ handlers })

export default kodmq

Create file ./jobs/launch.ts. This file will be used to launch your workers.

import kodmq from "./index"
import kodmqLauncher from "@kodmq/core/launcher"

await kodmqLauncher(kodmq)

Add these scripts to your package.json

{
  // ...
  "scripts": {
    "workers": "bun run ./jobs/launch.ts",
    "workers:ui": "kodmq-ui"
  },
  // ...
}

That’s it! Now you can launch your workers and KodMQ UI 🎉
```
bun run workers
bun run workers:ui
```

Usage

Okay, now you have your workers running. Let’s see how you can perform some jobs.

Perform Job

When you’re performing a job, it will be pushed to the queue and performed by the first available worker.

import kodmq from "./jobs"

const job = await kodmq.jobs.perform("sendEmail", { to: "Andrew Kodkod", text: "Have a good day!" })
console.log(job)

Schedule Job

When you’re scheduling a job, it will be saved to the database and performed by the first available worker at the specified time.

import kodmq from "./jobs"

const runAt = new Date(Date.now() + 1000 * 60 * 60 * 24) // 1 day later
const job = await kodmq.jobs.performAt(runAt, "sendEmail", { to: "Andrew Kodkod", text: "I hope you had a good day!" })
console.log(job)

// or

const runIn = 1000 * 60 * 60 * 24 // 1 day later
const job = await kodmq.jobs.performIn(runIn, "sendEmail", { to: "Andrew Kodkod", text: "I hope you had a good day!" })
console.log(job)

Core Concepts

Job

A job is a task that needs to be performed. It can be performed immediately or scheduled to be performed later.

Job may have one of the following statuses:

Pending – Job is waiting to be performed
Scheduled – Job is scheduled to be performed later
Active – Job is being performed
Completed – Job has been performed successfully
Failed – Job has failed to be performed

Worker

A worker is a single thread that performs jobs.

Worker may have one of the following statuses:

Idle – Worker is waiting for jobs
Busy – Worker is performing a job
Stopping – Worker is in the process of stopping (waiting for the current job to be completed)
Stopped – Worker is stopped
Killed – Worker was killed while performing a job (job was not completed and will be retried)

Concurrency

Concept of concurrency in KodMQ is a bit different from other libraries. As we’re trying to keep things simple, we don’t have concept of worker threads. What does it mean? It means 1 thread = 1 worker.

Let me explain it with an example.

When you’re launching your application with concurrency of 5:

kodmqLauncher(kodmq, { concurrency: 5 })
// or
bun run workers --concurrency=5

It runs a single process that can handle 5 jobs simultaneously, but you’ll see 5 workers in the UI.

If you want to run multiple processes, you can just run the launcher multiple times:

bun run workers --concurrency=2
bun run workers --concurrency=3

To make sure things don’t get messy in KodMQ UI, or you want to benchmark performance of different servers, you can specify a name for each “cluster”:

// Command to run locally
bun run workers --concurrency=5 --cluster-name="Local"

// Command to run on server 1
bun run workers --concurrency=5 --cluster-name="AWS 4vCPU"

// Command to run on server 2
bun run workers --concurrency=5 --cluster-name="DigitalOcean 8vCPU"

By default, KodMQ uses number of CPU cores as concurrency, so you don’t need to specify it manually.

Environment Variables

You can use environment variables to configure KodMQ

KODMQ_REDIS_URL=redis://localhost:6379
KODMQ_DATABASE_URL=postgresql://localhost:5432/kodmq
KODMQ_CONCURRENCY=5

Jobs API

import { Pending, Completed } from "@kodmq/core/constants"
import kodmq from "./jobs"

// Get all jobs
const jobs = await kodmq.jobs.all()

// Get jobs by status
const pendingJobs = await kodmq.jobs.all({ status: Pending })
const completedJobs = await kodmq.jobs.all({ status: Completed })

// Get specific job
const job = await kodmq.jobs.find(1)

// Boost job
await kodmq.jobs.boost(job.id)

// Retry job (if it failed)
await kodmq.jobs.retry(job.id)

// Cancel job (keeps it in the database)
await kodmq.jobs.cancel(job.id)

// Remove job from database
await kodmq.jobs.remove(job.id)

// Remove all jobs from database (including pending and scheduled, but not active)
await kodmq.jobs.removeAll()

// Remove all finished jobs from database (completed, failed, canceled)
await kodmq.jobs.removeFinished()

// Wait until all jobs are finished (completed, failed, canceled)
await kodmq.jobs.waitUntilAllFinished()

Workers API

import KodMQ from "kodmq"
import { Idle, Busy, Stopped } from "@kodmq/core/constants"
import kodmq from "./jobs"

// Get all workers
const workers = await kodmq.workers.all()

// Get workers by status
const idleWorkers = await kodmq.workers.all({ status: Idle })
const busyWorkers = await kodmq.workers.all({ status: Busy })

// Get specific worker
const worker = await kodmq.workers.find(1)

// Stop specific worker
await kodmq.workers.stop(worker.id)

// Stop all workers (launched in current process)
await kodmq.workers.stopAll()

// Wait until all workers are stopped (launched in current process)
await kodmq.workers.waitUntilAllInStatus(Stopped)

KvikPhotos helps you share your RAW photos with friends, family and clients

– Easily upload photos
– Automatically convert them into JPEGs
– Share your best photos and albums with a link

→ Register today and receive 50GB of free storage

That’s it!

I hope you’ll find KodMQ useful and enjoy using it! If you have any questions or suggestions, feel free to open an issue.

Have a good day! 🐈 🐈‍⬛

KodMQ βeta