Pretext

Before jumping in, this is up until NextJS 12^. NextJS 13 has however solved this issue with dedupe requests, but this is in beta.

The Scenario

Let’s take up a scenario here with SSG ( Static Site Generation );

1. Let’s say you wanted to build a customer-facing web application that needed good SEO and incredible page speed loads. The built page routes don’t need to have dynamic but rather static content.
2. How would one usually go about this? You can read up about getStaticPaths and getStaticProps and move further.
3. We want to build dynamic routes here. This would mean we’d be using both of the above in a file, let’s say [id].tsx for example.
4. Let’s assume you have a CMS, with a quotas limit for the requests per minute you can make to the CMS as 300. This is pretty standard with the free tier of GCP.
5. TLDR on data fetching;
   1. getStaticPaths runs once during build time to fetch a list of objects with params, which in laymen's terms is just all the routes of the pages to be built. Each params object is passed onto a getStaticProps function.
   2. getStaticProps runs on build time for each page/route which was fetched from getStaticPaths. This is usually where one would make another API request to fetch the page details and return them as props.

Ideally, this format for each object would be →

{params: {your-dynamic-filename: your-route-id}}
Example -> {params: {id: 4}}, given your filename is [id].tsx

So far so good right?

The Problem

Let’s understand what was the problem we faced with the above scenario.

1. What if I wanted to build 301 pages?
   1. The first request would be made by getStaticPaths
   2. After that, there will be 301 requests made to the API to fetch each page's data.
   3. Isn’t this quite exorbitant? You will be passing the per-minute quota limits in this case as it is asynchronously executed.
2. The question one would want to ask is, why cannot I pass additional data relating to a page from getStaticPaths itself? Why am I just passing in the params, which is an id or a slug per se?
   1. TLDR; You cannot. We can view this as a limitation by all means.
3. Well, the build time is going to be huge here, so why not just go with SSR?

Let’s get to solving

The ideal solution is pretty simple here.

1. Create a cache / db dump file to store the list of page data on getStaticPaths.
2. On getStaticProps return the page data from the cache file based on a slug or id.
3. Since this is SSG, on each build, a new cache file will be built with the updated data.

The process

We know that all of this takes place on build time and thereby this runs on the Server Side technically at the end of the day. This means we can make use of NodeJS File System modules ( fs-modules ).

Logics

1. Example - Let’s say you have an API that returns a list of tasks with slug as your to-be route.
2. Let’s clear out some decisions here.
   1. We’ll name our dynamic file as [taskSlug].tsx.
   2. Let’s have a helper file to have the cache logic.
3. Helpers

• First, you need fallbacks on the first requests when a cache file isn’t present. Let’s encapsulate all functions inside an object.

{
    fetchTaskList: async () => {
    return getAllTasks(); // Your fetch / API request handler
  },

  fetchTask: async (slug) => {
    const tasks = await api.fetchTaskList();
    return tasks.find((task: Task) => task?.slug === slug);
  },
}

Let’s write our Cache logic. There should be three functions for our basic needs here.

• Let’s set our file name in a constant for better readability.

const TASKS_CACHE_FILE = "tasks.db";

• Getting the cached Task List :

getCacheTasks: async (): Promise<Task[] | null | undefined> => {
  let tasks: Task[] | null = null;
  try {
        /* First check if we are able to access the cache file 
             / if it is present */
    await fs.access(TASKS_CACHE_FILE);
    const data = await fs.readFile(
      path.join(process.cwd(), TASKS_CACHE_FILE)
    );
    tasks = JSON.parse(data as unknown as string);
  } finally {
    return tasks;
  }
}

• Getting the cached Task :

getCachedTask: async (
  slug: string
): Promise<Task | null | undefined> => {
  let task: Task | null = null;
  try {
    await fs.access(TASKS_CACHE_FILE);
    const data = await fs.readFile(
      path.join(process.cwd(), TASKS_CACHE_FILE)
    );
    task =
      JSON.parse(data as unknown as string)?.find(
        (task: Task) => task.slug === slug
      ) ?? null;
  } finally {
    return task;
  }
},

• Setting the cache file :

setTasksCache: async (tasks: Task[]) => {
  return await fs.writeFile(
    path.join(process.cwd(), TASKS_CACHE_FILE),
    JSON.stringify(tasks)
  );
},

• Combine this together -

const TASKS_CACHE_FILE = "tasks.db";

export const api = {
  fetchTaskList: async () => {
    return getAllTasks(); // Your fetch / API request handler
  },
    // Fallbacks, in case we don't find a cache
  fetchTask: async (slug) => {
    const tasks = await api.fetchTaskList();
    return tasks.find((task: Task) => task?.slug === slug);
  },
  cache: {
    getCacheTasks: async (): Promise<Task[] | null | undefined> => {
      let tasks: Task[] | null = null;
      try {
                /* First check if we are able to access the cache file 
                     / if it is present */
        await fs.access(TASKS_CACHE_FILE);
        const data = await fs.readFile(
          path.join(process.cwd(), TASKS_CACHE_FILE)
        );
        tasks = JSON.parse(data as unknown as string);
      } finally {
        return tasks;
      }
    },
    getCachedTask: async (
      slug: string
    ): Promise<Task | null | undefined> => {
      let task: Task | null = null;
      try {
        await fs.access(TASKS_CACHE_FILE);
        const data = await fs.readFile(
          path.join(process.cwd(), TASKS_CACHE_FILE)
        );
        task =
          JSON.parse(data as unknown as string)?.find(
            (task: Task) => task.slug === slug
          ) ?? null;
      } finally {
        return task;
      }
    },
    setTasksCache: async (tasks: Task[]) => {
      return await fs.writeFile(
        path.join(process.cwd(), TASKS_CACHE_FILE),
        JSON.stringify(tasks)
      );
    },
  },
};

Views

• Our Page component would ideally be -

const TaskPage: NextPage = (props: Props) => {
  const { task } = props;

  return (
      <Task {...props} />
  );
};

• Our getStaticPaths would contain setting our caches and building the route params.

export const getStaticPaths: GetStaticPaths = async () => {
  let tasks = await api.cache.getCachedTasks();

  if (!tasks) {
    tasks = await api.fetchTaskList();
    await api.cache.setTasksCache(tasks);
  }

  const paths = tasks.map((task) => ({
    params: {
      taskSlug: task.slug,
    },
  }));

  return { paths, fallback: false };
};

• Our getStaticProps would execute on each page with the respective param object as it’s a parameter.

export const getStaticProps: GetStaticProps = async ({ params }) => {
  let task = await api.cache.getCachedTask(params?.taskSlug as string);

  if (!task) {
    task = await api.fetchTask(params?.taskSlug as string);
  }

// If the file does not exist, we return null either way

  if (!task) { 
    return {
      notFound: true, // This would handle 404s
    };
  }

  return {
    props: {
      task,
    },
  };
};

We can put all the above together -

const TaskPage: NextPage = (props: Props) => {
  const { task } = props;

  return (
      <Task {...props} />
  );
};

export const getStaticProps: GetStaticProps = async ({ params }) => {
  let task = await api.cache.getCachedTask(params?.taskSlug as string);

  if (!task) {
    task = await api.fetchTask(params?.taskSlug as string);
  }

// If the file does not exist, we return null either way

  if (!task) { 
    return {
      notFound: true, // This would handle 404s
    };
  }

  return {
    props: {
      task,
    },
  };
};

export const getStaticPaths: GetStaticPaths = async () => {
  let tasks = await api.cache.getCachedTasks();

  if (!tasks) {
    tasks = await api.fetchTaskList();
    await api.cache.setTasksCache(tasks);
  }

  const paths = tasks.map((task) => ({
    params: {
      taskSlug: task.slug,
    },
  }));

  return { paths, fallback: false };
};

export default TaskPage;

And you’re good to go. Hopefully.

Results

Before looking at differences, one small change depending on the number of pages you might want to build might take a lot of time than originally NextJS intended.

You can mention a property inside your next config →

module.exports = {staticPageGenerationTimeout: 5000, ..}

What have we solved? What are our trade-offs?

The number of requests:

• We have successfully reduced the number of requests to just 1. Throughout the build time of the application.

We did not go with SSR. With SSR -

• We could go Serverless:
  • We risk the possibility of cold lambdas occurring and thereby slowing the initial page load arbitrarily for a certain group of requests.
  • For a static customer-facing web application, this is not preferred.
• We could go with a full-blown server such as AWS EB / EC2:
  • More maintenance and lesser scalability.
  • CI/CD processes will have to be clearly defined.
  • For a customer-facing application that is meant to be static, this is pretty overkill.
  • This is going to be more expensive. With SSG, we are just going to serve a static set of pages.

Advantages during development: With NextJS, during development, all data-fetching methods are run on each request.

• That’s right. You can imagine trying to fetch tasks/1 and waiting for a while here.
• NextJS provides the ability to skip during development, by all means, this means this would be fetched on demand with SSR, which would take a few seconds as well.
• With caching, we solve this issue pretty much after the first load on development. It makes a huge difference when debugging or styling pages.

GCP Request quota caps:

• We were using the free-tier of GCP as our API with a request quota cap of 300 requests per second.
• If we wanted to stay with SSG, we would have needed to pay up. 🙂.

Build times on Cloud platforms:

Build times on Development:

• Something to keep in mind is that post-cache builds do not happen in production. This is because when we deploy, a fresh cache is always built to reflect the latest data.

• On Development, this does not make sense. Rather we would want to have post-cache builds here.

I think we can definitely understand the trade-offs here. This is pretty relative to what you are trying to solve here.

How has NextJS 13 solved this?

Although this is in beta, they have →

1. Removed the standard functions such as getStaticProps and getStaticPaths.
2. Extended the native fetch method to provide the above features with just this single function.
3. React has introduced Automatic fetch request deduping.

That’s right! The above introduces performance improvements through caching and minimizing duplicate requests being made.

Does this mean the above effort is quite useless?

1. Nope, being in beta and the fact that it is quite an effort to migrate from NextJS 12 and get accustomed to the new app directory flow, this is going to take some time.
2. At the end of the day, if your request endpoint has different params, each request is at the end of the day, not deduped. Think of it like a key-value pair where the request URL is the key and the response is the value.

Key Takeaways :

1. Always try to think from a DRY mindset. Don’t repeat yourself. Ask yourself questions.
2. Measure performance, space, and business requirements and always think about trade-offs to make your decision to solve a problem.
3. Try to be good at requirement gathering and breaking down a problem you want to solve. Do not be chained by your framework or language, concepts are permanent here.
4. Do not be discouraged if there is no documentation, similar issues, or threads out there bringing this up.