How to Use Node-Schedule to Build and Deploy a Hacker News Aggregator16 min
Modern software applications perform a multitude of recurring tasks, such as processing and exchanging data, generating reports, and more. While some of these tasks require specific timing, others take longer to complete and cannot be executed on the application's primary thread. Consequently, a scheduling mechanism is necessary to execute these tasks efficiently.
Cron is a widely used scheduling tool that allows users to automate and schedule recurring tasks at specific intervals, dates or times. It helps eliminate manual execution and ensures the timely execution of commands and scripts. The
node-schedule library is a flexible cron-like job scheduler for Node.js, offering cron-style, date-based, and recurrent rule scheduling.
In this tutorial, we will delve into the process of setting up and executing cron jobs in Node.js using the
node-schedule library by building a Hacker News story aggregator. Upon completing this tutorial, you will have developed a Node.js script that gathers the ten most commented-on stories on Hacker News and subsequently forwards them via scheduled daily emails.
You can deploy and preview the
node-schedule application from this guide using the Deploy to Koyeb button below:
You can consult the repository on GitHub to find out more about the application that this guide builds.
To successfully follow this tutorial, you need the following:
- Node.js and
npminstalled on your development machine. The demo app in this tutorial uses Node.js version 18.16.1.
- Git installed on your development machine.
- A Mailgun account to send the emails.
- A Koyeb account to deploy the application.
Set up the Mailgun account
Mailgun provides APIs for sending, receiving and tracking emails. In this section, you will log into your Mailgun account and get your API key and sandbox domain information.
While logged into your verified Mailgun account, retrieve your private API key by clicking the API keys link located on the right sidebar of the dashboard page. Copy the Private API key displayed on the page and store it securely for future use.
Next, select the Sending link in the left sidebar. This will direct you to the Domains page, where you can find your sandbox domain address presented in the following format:
Be sure to copy and save this domain address in a safe place for your future reference.
For testing purposes, Mailgun restricts sending emails to authorised email addresses only. To authorize an email address, click your domain URL on the Domains page to access its Overview page. On the right sidebar of the Overview page, enter the email address you want to send test emails to in the Authorized Recipients input field and click the Save Recipient button.
Mailgun will send a verification email to the provided address. In the verification email, click the I Agree button to complete the authorisation process. If you refresh the page in Mailgun, you see that the target email address is now marked as verified.
You now have all of the information you need to send emails using your Mailgun account. In the next section, you will set up the project and install the necessary libraries and dependencies.
Set up the project
In this section, we will set up an
npm project with TypeScript and install all necessary dependencies to build the Hacker News story aggregator.
To begin, create a project root directory on your development machine by typing:
hn_post_aggregator directory serves as the main directory for the demo application. The
src directory nested within will hold the actual project code.
Next, execute the commands below to initialise a Git repository in the newly created
The first command above changes your terminal's current working directory to the
hn_post_aggregator directory, while the second command initializes a Git repository there.
Next, generate an
npm project within the
hn_post_aggregator directory by typing:
This will initialize an
npm project using default values and creates a
package.json file in the project's root directory.
Having initialized the
npm project, proceed to install the necessary libraries and packages for building the Hacker News story aggregator:
npm install command installs the specified libraries and type definitions. The first command installs dependencies required to run the demo application, while the second command installs development-related dependencies.
The dependencies installed include:
mailgun.js: An SDK for Mailgun.
node-schedule: A job scheduling library.
dotenv: A library for handling environment variables.
axios: A promise-based HTTP client.
Additionally, the libraries installed for development purposes include:
typescript: Enables the execution of TypeScript code.
nodemon: Detects code changes to restart the application during development.
ts-node: To execute and rebuild TypeScript efficiently.
@types/node-schedule: Type definitions for
Afterwards, create a
tsconfig.json file in the root directory of the project, and insert the following code into the file:
tsconfig.json file indicates that this is a TypeScript project and defines the necessary options to compile the project.
With that last step, the project setup is complete. In the next section, you will start building the Hacker News story aggregator by adding the ability to retrieve the top Hacker News stories.
Retrieve top Hacker News stories
Hacker News offers a publicly available API that allows access to its stories, comments, polls, and other content on its platform in near real-time. The API provides access to these resources, known as "items", through their respective ids.
Furthermore, the API provides an endpoint for accessing a list of item IDs for 500 top, best, and new stories. This endpoint makes it possible to fetch the Hacker News stories with the highest number of comments.
To fetch the top ten most commented-on Hacker News stories, start by creating an
index.ts file in the
src directory and add the following code to the file:
The above code imports the
axios library and defines an
Item object type representing a Hacker News "item".
Next it defines a
fetchTopHNPosts function which makes two HTTP requests: one to retrieve a list of IDs for the top stories on Hacker News and another to fetch the corresponding items associated with those IDs.
The second HTTP request's response is sorted in descending order according to the number of comments they've received, and then the top ten items from the sorted list are returned. Finally, to test the
fetchTopHNPosts function, we invoke the function and display the return value in the console.
To test out the code, update the
scripts section of your project's
package.json with the following line:
dev command added above runs the code in your
index.ts file using
ts-node. To execute it, run the command below in your terminal window:
If everything works correctly, you should see an output like the one below in your terminal window:
Once you've confirmed that it is working correctly, open the
src/index.ts file again and comment out the line executing the
fetchTopHNPosts function. Moving forward, we will want this information formatted into an email instead of printing the results directly to the console.
In the next section, you will create an email containing the retrieved top ten stories with the highest number of comments. Furthermore, you will implement the necessary functionality to send this email using Mailgun.
Create email content
Now that we can generate the list of the top ten Hacker News stories with the most comments, we can compose an HTML email showcasing these stories and send it to specified recipients using Mailgun.
To get started, create an
.env file in your project's root directory and add the following lines of code to the file, replacing the placeholder values with your own:
.env file contains secret values and should be kept private. To ensure the file and its content aren't committed to Git, create a
.gitignore file by typing:
The command above creates a
.gitignore file containing lines that will exclude the
.env file and the
Next, update the top of your
index.ts file with the following code:
The code above imports the
mailgun.js libraries and instantiates a new Mailgun client. The Mailgun client will handle sending composed emails.
Next, add the following lines to the bottom of the
The code above adds a
sendEmail function which iterates through an array of
Item objects and generates an HTML email content with a list of stories, their comment counts, and a link to view each story. In addition, the email message data (the sender and recipient's email, the email subject, and the content) is prepared and passed alongside your Mailgun domain as arguments to the Mailgun client. The client will attempt to send the email and logs the outcome to the console.
In the next section, we will schedule a cron job to fetch the top Hacker News stories and send them via email at a specific, designated time.
The necessary logic for retrieving the top ten most commented-on Hacker News stories and sending them through an email has been successfully implemented. In this section, you'll schedule a cron job to automate the execution of this code.
Cron jobs are managed through a table called
crontab, which holds entries for each scheduled task. Each task on the
crontab is entered using a cron syntax consisting of five fields delimited by spaces representing the schedule of when the task should run followed by the command to be executed:
node-schedule library handles the scheduling and execution of cron jobs and accepts both the cron syntax and other formats for time-based scheduling.
To schedule fetching and sending of the top ten most commented-on Hacker News stories, add the following
import to the top of your
Next, add the code below to the end of your
The code above schedules a cron job using the
scheduleJob method. The method takes a cron syntax schedule from a
SCHEDULE environment variable, falling back on a default value of
"*/2 * * * *". The cron syntax
"*/2 * * * *" defines the following schedule:
"*/2"means "every 2 minutes."
"*"means "every hour."
- Day of the month:
"*"means "every day of the month."
"*"means "every month."
- Day of the week:
"*"means "every day of the week."
node-schedule "recurrence rule" with the
scheduleJob method for greater flexibility in scheduling tasks. Visit the node-schedule documentation to learn about other usage options.
When the job is triggered, the
fetchTopHNPosts function is executed to gather the ten most commented-on Hacker News stories. These stories are then sent via email using the
To test out the scheduling functionality, run the
npm run dev command in your terminal window. After about two minutes, you should see output like the one below in your terminal window:
After a few more minutes, you should have an email like the one below in your email:
Note: The email might show up in your spam folder if your Mailgun domain is unverified.
You have now successfully scheduled a cron job to fetch and send the top ten most commented-on stories on Hacker News. In the next section, you will deploy your code to Koyeb.
Deploy to Koyeb
Since the Hacker News story aggregator operates solely in the background without a frontend UI, deploying it as a worker is the optimal choice. Fortunately, Koyeb allows the deployment of a wide range of services, including background workers.
To prepare the demo app for deployment, adjust the
scripts section of your
package.json file to include the following lines:
build command compiles the TypeScript code in your
index.js file. The
start command executes the compiled code.
Next, create a GitHub repository for your code and run the following commands in your terminal window to commit and push your changes to the repository:
In the Koyeb control panel, navigate to the Secrets tab and enter a new Secret for each environment variable defined in your local
.env file. Next, go to the Overview tab and click the Create App button to start the App creation and deployment process.
On the App deployment page:
- Select GitHub as your deployment method.
- From the repository dropdown menu, select the GitHub repository that contains your code. Alternatively, you can deploy from the example repository associated with this tutorial by entering
https://github.com/koyeb/example-node-schedulein the Public GitHub repository field.
- Select the specific branch you wish to deploy (e.g.,
- Choose a Worker Service type.
- Click on the Advanced button to view additional options and then select the Add Variable button.
- For each environment variable defined in your
.envfile, create a corresponding variable. For the variable type, select Secret and then choose the matching Secret you defined earlier.
- To change the schedule of the cron job from running every two minutes after deployment, add a
SCHEDULEenvironment variable and set its value to your preferred cron schedule. To help you generate the cron schedule easily, you can use an online cron syntax generator such as Cronitor's cron guru.
- You can optionally give your app a custom name; otherwise, the default value will be used.
- Finally, click the Deploy button to initiate the deployment process.
During the app deployment process, Koyeb detects and utilizes the
start scripts specified in your
package.json file to build and start the application. Throughout the deployment, you can monitor the progress with the displayed logs. After the deployment has finished and all necessary health checks have passed, your application will be operational.
At the scheduled time, the app will retrieve the top ten most commented-on stories on Hacker News and send them to the specified email address. Note that the scheduled task will execute according to the local time of the app's deployment region.
Congratulations! You have successfully created and deployed a Node.js cron job that collects the top ten most commented-on Hacker News stories and sends them via scheduled emails. Feel free to explore the other scheduling options offered by the
node-schedule library to create more flexible schedules for your cron job.
As the worker was deployed using the Git deployment method, any new push to the deployed branch will automatically trigger a fresh build for your worker. Updates to your worker will go live as soon as the deployment successfully passes all required health checks. In the event of a deployment failure, Koyeb preserves the last functioning deployment in production, guaranteeing that your application remains continuously operational.