50% Improvement to NextJS Build Times -- Developer Diary

At BlackBean, we’ve undergone numerous transformations in our web development toolkit to keep pace with the rapidly evolving web development landscape. Staying in front of the latest innovations is crucial; failing to do so might mean missing out on significant efficiencies and advancements!

Keeping up with the latest and greatest in technology can also introduce risk. Innovation isn’t easy! We recently uncovered an interesting optimization issue affecting our current Next.JS powered development process. More importantly, we found a fix. Read on to learn how we cut website build times by 50% while reducing server requests by 80%.

There and back again: WordPress to static sites, back to hybrid WP

Let’s look at our six-year journey in web development: where we started, and how we got here.

When we started in 2018, we were fully bought into the WordPress ecosystem. We used WordPress to launch numerous successful websites. Despite its strengths, WordPress is not without flaws. Minimizing website downtime while ensuring seamless editing, solid security, and fast load speeds on WordPress requires a lot of time and effort. Over the years, we encountered the limitations of WordPress: a sometimes-cumbersome development workflow, a constant need for updates to ensure security, and a plugin ecosystem that frequently resulted in bugs and instability.

Driven by a need to streamline website development and reduce management overhead, we began exploring alternative solutions. We had talked internally about sites that were developed in the early 2000s built with basic HTML, CSS, and JS. They looked and performed exactly the same way they did 20 years ago when they first launched—no plugin updates or security patches. How could we replicate that stability with modern technologies?

This exploration led us to the world of static website generators. Starting with smaller projects, we experimented with Eleventy, a frontend framework, coupled with Decap CMS for content management. This combination was a revelation for us. We saw first-hand the efficiency and potential that static sites offer.

However, as our projects grew in complexity, we encountered limitations with this setup. Static websites were great for informational websites or simple lead generating pages, but they lacked the ability to easily manage things like memberships and gated content. This prompted us to explore more robust solutions for enterprise-level websites. It led us to adopt Next.js, a leading frontend framework. Initially, we chose to develop our own custom CMS to offer clients a secure, fool-proof way to manage content without jeopardizing website integrity—an issue we’d faced in the past.

Despite positive feedback for our custom CMS, its development and maintenance proved challenging for our small team. After much deliberation, we came full circle, opting for a headless WordPress configuration. This setup leverages WordPress solely for content management, bypassing its frontend capabilities. By minimizing plugin use, we’ve reduced (almost eliminated) update and security concerns. We have carefully crafted a custom editor experience that retains WordPress’s strengths while eliminating its drawbacks. Using this technology, we can better provide the benefits of a conventional server-based website, while benefitting from the security and stability of serverless tech. This strategic pivot has enabled us to harness the best of WordPress, tailoring it to our evolving needs and those of our clients.

Discovering the Issue

After implementing our latest technology stack, we rigorously tested it on our own website before transitioning some of our client websites from our proprietary CMS back to WordPress. Many of these websites featured complex data structures with numerous pages and functionalities. Although the migration process was smooth, it was time-consuming to do. A warning to would-be developers: technical debt is expensive to pay off!

After completing the migration, we gradually became aware of an issue. One of our clients made especially frequent updates to their site content. For those unfamiliar, static sites require a complete rebuild to incorporate new data points. This allows the site to serve the refreshed content as a single web application. The rebuild duration varies based on the site’s complexity and the nature of the data involved, with more extensive data resulting in longer build times.

Our client informed us that every time an update was made to the site, the whole WordPress editor slowed down to a crawl. This was especially concerning given our use of a WordPress multisite configuration for this headless CMS approach. The slowdown suggested that updates on one site could affect the back-end performance of all other sites on the server. This is when we sprang into action to figure out what was going on!

Fixing the Issue

We began our investigation by examining the WordPress server for any misconfigurations, but everything looked good on that end. Then we activated an Application Performance Monitoring (APM) tool to assess the server’s load. The APM revealed a significantly high volume of requests originating from the Next.js site during its build process—approximately 12,000 requests per minute with three sites concurrently building. Although our server configuration is robust enough to handle a considerable amount of traffic, the influx of simultaneous uncached requests nearly overwhelmed it.

The excessive number of requests during the site builds pointed to an issue within our Next.js configuration, suggesting unnecessary data requests. Essentially, when big or complex websites pushed an update, our WordPress server got blasted with a huge amount of traffic, as though thousands of users were trying to load the website at once.

Our attention turned to the dynamic pages, as they were the primary source of these requests. Dynamic routes in Next.js are typically utilized for content like blog or news posts, where the base URL isn’t predetermined and must be retrieved from the CMS. In our case, the news pages were the culprits.

To implement dynamic routes in Next.js, a few functions are necessary. Firstly, getStaticPaths is used to fetch the URL paths for the dynamic pages. Following this, we use getStaticProps to retrieve the necessary data and pre-render the pages, ensuring that all content is ready before being served to the user.

Here is an example of the setup we were using in typescript:

export const getStaticPaths = async () => {
  return {
    paths: await getSlugs('posts'),
    fallback: false,
  };
};

export const getStaticProps: GetStaticProps = async ({ params }: GetStaticPropsContext) => {
  const slug = String(params?.id || '');
  const newsData = await getIds("posts");
  const index = newsData.findIndex((item: any) => item.params.slug === slug);
  const postID = index !== -1 ? newsData[index]?.params.id : null;

  if (!postID) {
    // Handle case where postID is null or undefined
    return {
			notFound: true,
    };
  }

  const pageData = await getPostData('posts', postID);
  
  return {
    props: {
      pageData,
      key: slug,
      index,
      recentNewsData: await getRecentPosts('posts', 4),
    },
  };
};

As you can see above, we first use getStaticPaths to determine which paths for the dynamic pages should be pre-rendered at build time. We call a function called getSlugs that queries the WordPress rest API to grab all these slugs.

Once that is complete, we move onto getting the page data and pre-rendering the page. getStaticProps contains a couple of lines that first extract the slug. Next, we fetch a list of IDs from the CMS using getIds("posts"). Then we search through newsData to find the index of the post that matches the slug and check for the ****Post ID validity. Finally, we fetch the post data using getPostData('posts', postID); and return all the data to use on the page. We also call for a couple more articles to display in a “recent news” section.

This is where we can start to see where the main issue is happening. Each dynamic page needs to call four unique requests to get the info needed to build out the page. If you had 50+ pages, you would have to call each one of those requests per page. That starts to turn into a lot of requests for one dynamic route—and this particular site has many such routes.

The Solution

Now that we have determined the main cause of the slowdown on our headless WordPress setup, we can start to work towards a fix! There are a couple different methods that we explored to help reduce the number of requests coming into the WordPress site.

1. NextJS – Incremental Static Regeneration (ISR)
2. JSON File Generation

Incremental Static Regeneration (ISR)

We first looked into Incremental Static Regeneration (ISR). ISR empowers you to create or update static pages without having to rebuild your entire site. This feature supports static generation on a per-page basis, blending the benefits of static sites with the capacity to scale to thousands of pages.

Upon a request to a page that was pre-rendered during the build phase, the cached version of the page is initially displayed. Subsequent requests within a specific time limit (e.g., 10 seconds) will instantly retrieve the cached version. Beyond this period, the next request will still present the cached version, while Next.js discreetly initiates a page regeneration in the background. Once regenerated successfully, Next.js updates the cache to reflect the latest version of the page. Using this feature will help reduce the heavy load on the server after content is updated. We will still see a number of requests on build that don’t really need to happen, but this is definitely a step in the right direction. This is the preferred method, but we would have to make some heavy adjustments to our setup for this specific build.

Here is an example of what that would look like:

export const getStaticPaths = async () => {
	// We'll pre-render only these paths at build time.
  // { fallback: 'blocking' } will server-render pages
  // on-demand if the path doesn't exist.
  return {
    paths: await getSlugs('posts'),
    fallback: blocking,
  };
};

export const getStaticProps: GetStaticProps = async ({ params }: GetStaticPropsContext) => {
  const slug = String(params?.id || '');
  const newsData = await getIds("posts");
  const index = newsData.findIndex((item: any) => item.params.slug === slug);
  const postID = index !== -1 ? newsData[index]?.params.id : null;

  if (!postID) {
    // Handle case where postID is null or undefined
    return {
			notFound: true,
    };
  }

  const pageData = await getPostData('posts', postID);
  
  return {
    props: {
      pageData,
      key: slug,
      index,
      recentNewsData: await getRecentPosts('posts', 4),
    },
		// Next.js will attempt to re-generate the page:
    // - When a request comes in
    // - At most once every 30 seconds
		revalidate: 30,
  };
};

JSON File Generation

Next, we looked into creating some simple on-site JSON files from the different needed data paths that were reused throughout the build. The idea is this script would run before the site build and pull in all the needed slugs, id, and post data from the CMS before it is called for the pre render. We would then be able to query these files for the data rather than using requests directly to WordPress. The only requests to WordPress would come from the initial call to build out these JSON files. For our usage, this method worked perfectly! Querying these internal files are fast and most importantly do not add any strain to the WordPress install.

Here is an example of what the getStaticPaths and the getStaticProps would look like now:

export const getStaticPaths = async () => {
  const schemaData = require('/public/assets/post_slugs.json'); // Import the schema data from the file
  return {
    paths: schemaData,
    fallback: false,
  };
};

export const getStaticProps: GetStaticProps = async ({ params }: GetStaticPropsContext) => {
  const slug = String(params?.id || '');
  const newsData = require('/public/assets/post_ids.json'); // Import the schema data from the file
  const index = newsData.findIndex((item: any) => item.params.slug === slug);
  const postID = index !== -1 ? newsData[index]?.params.id : null;

  if (!postID) {
    // Handle case where postID is null or undefined
    return {
      notFound: true,
    };
  }

  const pageData = await getPostData('posts', postID);
  const recentNewsData = require('/public/assets/post_recent.json');

  return {
    props: {
      pageData,
      key: slug,
      index,
      recentNewsData,
    },
  };
};

After we implemented this JSON strategy, we can see our overall NextJS build time reduced by about 50% across the board.

Before

After

 

We also ran the exact same tests on the WordPress server using our AMP tool as we did earlier: three sites building at the same time. This is a night-and-day difference between the two tests. The overall PHP transaction time went from 162 seconds down to 16 seconds. We also reduced our overall requests per minute from 26 to 2. There is a similar trend for all other transaction types on the tests that we had run.

Before

After

Conclusion

BlackBean’s journey through the evolving landscape of web development tools shows the critical need for continuous innovation and adaptation. From the initial commitment to the WordPress ecosystem to the strategic shift towards static site generation and the integration of NextJS with a headless WordPress setup, we have navigated through many challenges to enhance operational efficiency and site performance.

A pivotal moment came with the identification and resolution of build-time and server load issues, through the use of Incremental Static Regeneration (ISR) and a JSON file generation approach. This not only markedly improved build times by about 50% but also exemplified a proactive approach to problem-solving. The key lesson learned is the importance of not just reacting to technological challenges but anticipating them and creatively leveraging technology to turn potential setbacks into opportunities for advancement. This journey shows the path for others in the industry, highlighting that the right combination of technology and innovation can lead to major improvements in both product quality and client satisfaction.