Introduction

Imagine this: You're building a social media app. A user loads their feed, and each post shows the author's name.

Sounds simple? No... if you have 10 posts from 10 different authors, you'd normally need 10 separate database calls to fetch each author's data.

This is the N+1 query problem.

The Problem: Three Core Issues

1. Redundant Requests: If two posts are from the same author, we're querying that author's data twice.

2. Network Communication: Each request adds latency, making 10 small requests is much slower than one bigger request.

3. Poor Batching: Even if we could batch requests, our code would get messy trying to coordinate them

Introducing DataLoader

DataLoader solves these issues by following two key principles:

1. Batch What's Batchable: If you're going to request multiple things of the same type, do it in one go.

2. Cache What's Cacheable: If you've already fetched something, don't fetch it again.

Here's the magic: DataLoader does this automatically while letting you write simple, clean code.

How It Works

// Without DataLoader
async function getAuthorNames(posts) {
  // 😱 N+1 queries!
  // We literally make a database call for each post.
  return Promise.all(posts.map((post) => db.getAuthor(post.authorId)));
}

// With DataLoader
const authorLoader = new DataLoader(async (authorIds) => {
  // 🎉 One query!
  const authors = await db.getAuthors(authorIds);
  return authorIds.map((id) => authors.find((a) => a.id === id));
});

async function getAuthorNames(posts) {
  return Promise.all(posts.map((post) => authorLoader.load(post.authorId)));
}

What's actually happening

When you call authorLoader.load(), DataLoader:

1. Collects: Instead of immediately fetching, it collects all load requests made in the same tick of the event loop.

2. Batches: Once the tick is done, it calls your batch function with all collected IDs at once.

3. Caches: Stores the results in memory so repeated requests for the same ID return the cached value.

Real-World Impact

Let's say you have a GraphQL query like:

{
  posts {
    title
    author {
      name
      avatar
    }
    comments {
      text
      author {
        name
        avatar
      }
    }
  }
}

Without DataLoader, you might make:

• 1 query for posts

• N queries for post authors

• N queries for comments

• M queries for comment authors

With DataLoader, you get:

• 1 query for posts

• 1 query for all unique post authors

• 1 query for all comments

• 1 query for all unique comment authors

Ordering matters

One of DataLoader's most important requirements is maintaining the exact order between your input keys and output values. This isn't just a nice-to-have, it's a fundamental contract that makes DataLoader work.

When you batch things, you have to be really careful about keeping everything in the right order. Like if you ask for users [4,2,9], and your database returns them as [2,9,4], you MUST put them back in the original order [4,2,9].

For example:

const postsLoader = new DataLoader(async (postIds) => {
  // Original order: [4, 2, 9]
  const posts = await db.getPosts(postIds);

  // Database returned them in a different order: [2, 9, 4]

  // For each postId from the original array, find the post in the array of the db results and return it
  // This will return [{result of 4}, {result of 2}, {result of 9}]
  return postIds.map((id) => posts.find((p) => p.id === id));
});

The Mental Model

Think of DataLoader as your smart personal assistant:

• You: "Get me users 1, 4, and 7"

• Assistant (DataLoader): "I see you need multiple users. I'll get them all at once, and I'll remember them in case you ask again."