Optimizing Web Applications with Redis: A Practical Journey

This post details our journey of solving two common performance bottlenecks in web applications—email delivery and database load—using Redis and BullMQ.

Problem 1: The Email Deluge

The Challenge: Email Rate Limiting

Nearly every email provider imposes a rate limit on how many emails you can send per second. These limits are crucial for them to prevent spam and maintain their sending reputation.

Amazon SES: ~14 emails/sec (default)
SendGrid: 10-20 emails/sec (depending on the plan)
Postmark: Optimized for transactional emails, but still has limits.

Our application needed to send hundreds of emails for various reasons: order confirmations, password resets, and marketing campaigns. When a burst of activity caused us to exceed the provider's limit, emails would start to fail. Some were delayed, and worse, others were lost entirely.

First Attempt: A Simple Redis Rate Limiter

Our initial thought was to use Redis to simply track the number of emails sent per second.

The result was chaotic. While it prevented us from exceeding the rate limit, any email that would have exceeded it was simply dropped. There were no retries and no queuing. This was not a viable solution.

The Solution: Robust Queueing with BullMQ

We realized we needed a proper queueing system. While we could have built one manually with Redis Lists, BullMQ provided a robust, battle-tested solution out of the box.

Step 1: Queue Emails Instead of Sending Them Directly

First, we changed our sendEmail function. Instead of trying to send the email immediately, it now adds a "job" to a dedicated email queue.

import { Queue } from "bullmq";

// Connect to your Redis instance
const emailQueue = new Queue("email-queue");

const sendEmail = async (userId, emailData) => {
  // Add a new job to the queue with the email data
  const job = await emailQueue.add("send-email", emailData, {
    attempts: 3, // Retry the job up to 3 times if it fails
    backoff: {
      // Use exponential backoff for retries
      type: "exponential",
      delay: 1000,
    },
  });
  console.log(`Queued job ${job.id} for user ${userId}`);
  return job.id;
};

Step 2: Process the Queue with a Worker

Next, we created a worker. This is a separate process that listens for jobs on the email-queue. We configured it to process jobs at a rate that respects our email provider's limits.

import { Worker } from 'bullmq';

const processEmail = async (job) => {
  try {
    const { to, from, subject, body } = job.data;

    // Call your actual email sending service here
    // await emailService.send({ to, from, subject, body });

    console.log(`Email job ${job.id} completed successfully.`);
    return { status: "Success" };
  } catch (error) {
    console.error(`Email job ${job.id} failed:`, error);
    // The error is thrown so BullMQ knows to retry the job
    throw error;
  }
};

// Initialize a worker to process jobs from the 'email-queue'
const worker = new Worker("email-queue", processEmail, {
  limiter: {
    max: 10, // Process a maximum of 10 jobs
    duration: 1000, // per 1000 ms (1 second)
  },
});

console.log("Email worker started...");

The Result:

Emails are queued instantly instead of failing during traffic spikes.
The worker processes the queue at a controlled, sustainable rate.
Built-in retries and backoff strategies ensure transient errors don't cause lost emails.
The email overload issue was completely resolved.

Problem 2: The Database Bottleneck

The Challenge: Sluggish Performance from Database Overload

On our product pages, fetching all the necessary data for a single product required multiple database queries.

// Example of data fetching for a single product page
const getProductData = async (id) => {
  const product = await Product.findById(id);
  const variants = await Variant.find({ productId: id });
  const pricing = await Pricing.findOne({ productId: id });
  // ... and so on
};

This approach led to significant performance problems:

Slow Page Load Times: Complex pages could take several seconds to load.
High Database Load: Our MongoDB instance was constantly under heavy read load.
Scalability Issues: Performance degraded quickly as the number of concurrent users grew.

The Solution: Smart Caching with Redis and BullMQ

Instead of hitting the database for every single request, we implemented a caching layer with Redis. We then used a BullMQ queue as a clever way to manage cache updates and versions.

Step 1: Create a Queue to Manage Cache Versions

We set up a queue that would hold our cached data. Each "job" in this queue represents a version of the cache at a specific point in time.

import { useQueue } from 'bullmq';

const productCacheQueue = useQueue("product-cache");

Step 2: Periodically Refresh the Cache via a Scheduled Job

We used a setInterval to periodically fetch fresh data from the database and add it as a new job to our cache queue. This process also prunes old cache versions to prevent the queue from growing indefinitely.

const MAX_CACHE_ENTRIES = 2; // Keep the current and previous cache versions

setInterval(async () => {
  console.log("Refreshing product cache...");

  // 1. Fetch the complete, fresh data from the database
  const freshProducts = await fetchAllProductsFromDB();

  // 2. Get all existing cache jobs
  const existingJobs = await productCacheQueue.getJobs(["waiting", "completed"]);

  // 3. Prune older cache entries to keep only the last N versions
  if (existingJobs.length >= MAX_CACHE_ENTRIES) {
    existingJobs.sort((a, b) => a.timestamp - b.timestamp); // Sort oldest to newest
    const jobsToRemove = existingJobs.slice(0, existingJobs.length - (MAX_CACHE_ENTRIES - 1));
    for (const job of jobsToRemove) {
      await job.remove();
    }
  }

  // 4. Add the new data as a new job in the queue
  const timestamp = Date.now();
  await productCacheQueue.add(
    "update-cache",
    {
      timestamp,
      products: JSON.parse(JSON.stringify(freshProducts)), // Ensure proper serialization
    },
    {
      jobId: `cache-${timestamp}`, // Use a timestamp-based ID for easy identification
      removeOnComplete: false, // Keep the job in the queue as our cache entry
      attempts: 1,
    }
  );
  console.log(`Cache updated with job ID: cache-${timestamp}`);
}, 60000); // Refresh every 60 seconds

Step 3: Serve API Requests from the Cache

Finally, we updated our API routes to read from the latest entry in our cache queue instead of querying the database directly.

// Helper function to get the latest data from our cache queue
const getLatestFromCache = async () => {
    const jobs = await productCacheQueue.getJobs(['completed'], 0, 0, false); // Get latest completed job
    if (jobs && jobs.length > 0) {
        return jobs[0].data.products;
    }
    return null; // Cache is empty
}

// API endpoint to get products
const getProducts = async (req, res) => {
  let products = await getLatestFromCache();

  // Fallback: If cache is empty, fetch directly from the DB
  if (!products) {
    console.log("Cache miss! Fetching from DB...");
    products = await fetchAllProductsFromDB();
  }

  return res.json(products);
};