Setting Up an Automated URL Status Checker with Notifications on macOS

by David Yates

I finished reading Node.js Design Patterns a while ago and thoroughly recommend it. This Saturday morning I decided to have a bit of downtime from project work and instead write my self a little program that checks the status of a list of URLs and sends me a notification if any of them are down. I’ve been meaning to do this for a while, and it was a fun little project to work on.

A quick note on the pattern used

In the book, they use the Parallel Stream Processing design pattern, which makes asynchronous calls (such as checking a bunch of urls) incredibly fast. The pattern uses a ParallelStream class that extends Nodes Transform class, and a checkUrls.js file that uses the ParallelStream class to check the status of the URLs. This patterns allows for chunked processing of the URLs, which is much faster than processing them one at a time, in other words as one url is being checked, the next one is already being checked, and so on. I’ve just given you the finished version here, but read the book to get the full picture.

This tutorial will guide you through setting up an automated service that checks the status of URLs from a file and sends notifications or emails if any services are down. The service will be scheduled to run at regular intervals on a macOS system.

Prerequisites

Node.js installed on your system. npm or yarn to install Node.js packages. A Gmail account for sending email notifications. Access to the terminal and basic familiarity with shell scripting.

Step 1: Project Setup

I’m terrible at documentation. I also really dislike having to remember a bunch of steps to get a project up and running, so lets use a Makefile to automate the process.

# .Makefile
NODE = $(shell which node)
NPM = $(shell which npm)
USERNAME = $(shell whoami)
PLIST = ~/Library/LaunchAgents/com.$(USERNAME).checkurls.plist

all: check install create_urls_file reload run

check:
	@echo "Checking for Node.js..."
	@test -n "$(NODE)" || (echo "Node.js is not installed. Please install it before proceeding." && exit 1)
	@echo "Checking for npm..."
	@test -n "$(NPM)" || (echo "npm is not installed. Please install Node.js which includes npm." && exit 1)

install:
	@echo "Installing required npm packages..."
	@$(NPM) install superagent split nodemailer dotenv

create_urls_file:
	@test -f urls.txt || (echo "Creating a new urls.txt file..." && touch urls.txt)

reload:
	@echo "Setting up the scheduled task..."
	@./setup-scheduled-task.sh
	@echo "Reloading the plist file..."
	-@launchctl unload $(PLIST)
	@launchctl load -w $(PLIST)
	@echo "Scheduled task has been reloaded."

run:
	@echo "Running the URL status checker..."
	@$(NODE) check-urls.js urls.txt

.PHONY: all check install create_urls_file reload run

Now, when I want to run the script, I can just run make all and it will check for Node.js, install the required packages, create a urls.txt file, set up the scheduled task, and run the script.

Remember to set the permissions on the shell script to allow execution. You can do this with chmod +x setup-scheduled-task.sh. Also, remember this is the finished version, read to the end to get all the pieces.

Step 2: Node.js Script Setup

Borrowing from the Node.js Design Patterns book, I created a Node.js script that reads a list of URLs from a file, checks their status, and writes the results to another file. I used the superagent package to check the status of the URLs, and the split package to split the file into an array of URLs.

I’ve just given you the finished version here, so keep reading to get the full picture. This pattern uses two files, aParallelStream clas sthat extends Nodes Transform class, and a checkUrls.js file that uses the ParallelStream class to check the status of the URLs.

// parallel-stream.js
import { Transform } from "stream";

export class ParallelStream extends Transform {
  constructor(userTransform) {
    super({ objectMode: true });
    this.userTransform = userTransform;
    this.running = 0;
    this.terminateCallback = null;
  }

  _transform(chunk, enc, done) {
    this.running++;
    this.userTransform(
      chunk,
      enc,
      this.push.bind(this),
      this._onComplete.bind(this)
    );
    done();
  }

  _flush(done) {
    if (this.running > 0) {
      this.terminateCallback = done;
    } else {
      done();
    }
  }

  _onComplete(err) {
    this.running--;
    if (err) {
      return this.emit("error", err);
    }
    if (this.running === 0) {
      this.terminateCallback && this.terminateCallback();
    }
  }
}
// check-urls.js
import { pipeline } from "stream";
import { createReadStream, createWriteStream } from "fs";
import split from "split";
import superagent from "superagent";
import { ParallelStream } from "./parallel-stream.js";
import { exec } from "child_process";
import nodemailer from "nodemailer";
import "dotenv/config";

const resultsFilePath = process.env.RESULTS_FILE_PATH || "results.txt";

function showNotification(message) {
  const escapedMessage = message.replace(/"/g, '\\"');

  const script = `display notification "${escapedMessage}" with title "Service Status"`;

  exec(`osascript -e '${script}'`, error => {
    if (error) {
      console.error(`exec error: ${error}`);
    }
  });
}
const transporter = nodemailer.createTransport({
  service: "gmail",
  auth: {
    user: process.env.GMAIL_USERNAME,
    pass: process.env.GMAIL_PASSWORD,
  },
});

function sendEmailNotification(url) {
  const mailOptions = {
    from: process.env.GMAIL_USERNAME,
    to: process.env.EMAIL_RECIPIENT,
    subject: "Service Down Alert",
    text: `The service at ${url} is down!`,
  };

  transporter.sendMail(mailOptions, (error, info) => {
    if (error) {
      return console.error(`Error sending email: ${error}`);
    }
    console.log(`Email sent: ${info.response}`);
  });
}

pipeline(
  createReadStream(process.argv[2]),
  split(),
  new ParallelStream(async (url, enc, push, done) => {
    if (!url) return done();
    try {
      await superagent.head(url, { timeout: 5 * 1000 });
      push(`${url} is up\n`);
    } catch {
      push(`${url} is down\n`);
      if (url.trim()) {
        showNotification(`${url} is down`);
        sendEmailNotification(url);
      }
      done();
    }
    done();
  }),
  createWriteStream(resultsFilePath),
  err => {
    if (err) {
      console.error(err);
      process.exit(1);
    }
    console.log("All urls have been checked");
  }
);

Take note of the environment variables in the script. These will be used to store sensitive information such as your Gmail username and password. We’ll set these up later.

Step 3: Environment Configuration

Create a .env file to store sensitive information such as your Gmail username and password.

# .env
GMAIL_USERNAME="your-gmail-username"
GMAIL_PASSWORD="your-gmail-password"
EMAIL_RECIPIENT="your-email-address"
PLIST_LABEL='com.username.checkurls'
SCHEDULE_MINUTE="*/60" # every hour
RESULTS_FILE_PATH="/path/to/results.txt"

Ensure your script reads environment variables from the .env file.

Step 4: Shell Script for Scheduling

Write a shell script touch setup-scheduled-task.sh && chmod +x setup-scheduled-task.sh that will set up a launchd job to run your Node.js script at specified intervals.

# setup-scheduled-task.sh
#!/bin/bash

if [ -f .env ]; then
  set -a
  source .env
  set +a
fi

USERNAME=$(whoami)
CURRENT_DIR=$(pwd)

PLIST_LABEL=${PLIST_LABEL:-"com.$USERNAME.checkurls"}
PLIST_FILE="$HOME/Library/LaunchAgents/$PLIST_LABEL.plist"
NODE_PATH=$(which node)
SCRIPT_PATH="$(pwd)/check-urls.js"
URLS_FILE="$(pwd)/urls.txt"

if [ -z "$NODE_PATH" ]; then
  echo "Node.js is not installed. Please install it first."
  exit 1
fi

SCHEDULE_MINUTE=${SCHEDULE_MINUTE:-0}

if [ "$SCHEDULE_MINUTE" = "*/1" ]; then
  INTERVAL_KEY="<key>StartInterval</key>"
  INTERVAL_VALUE="<integer>60</integer>"
  SCHEDULE_MESSAGE="every minute"
else
  INTERVAL_KEY="<key>StartCalendarInterval</key>"
  INTERVAL_VALUE="<dict><key>Minute</key><integer>$SCHEDULE_MINUTE</integer></dict>"
  SCHEDULE_MESSAGE="at minute $SCHEDULE_MINUTE of every hour"
fi

PLIST_CONTENT="<?xml version=\"1.0\" encoding=\"UTF-8\"?>
<!DOCTYPE plist PUBLIC \"-//Apple//DTD PLIST 1.0//EN\" \"http://www.apple.com/DTDs/PropertyList-1.0.dtd\">
<plist version=\"1.0\">
<dict>
  <key>Label</key>
  <string>${PLIST_LABEL}</string>
  <key>ProgramArguments</key>
  <array>
      <string>${NODE_PATH}</string>
      <string>${SCRIPT_PATH}</string>
      <string>${URLS_FILE}</string>
  </array>
  <key>RunAtLoad</key>
  <true/>
  ${INTERVAL_KEY}
  ${INTERVAL_VALUE}
  <key>EnvironmentVariables</key>
  <dict>
    <key>GMAIL_USERNAME</key>
    <string>${GMAIL_USERNAME}</string>
    <key>GMAIL_PASSWORD</key>
    <string>${GMAIL_PASSWORD}</string>
    <key>EMAIL_RECIPIENT</key>
    <string>${EMAIL_RECIPIENT}</string>
    <key>RESULTS_FILE_PATH</key>
    <string>${RESULTS_FILE_PATH}</string>
  </dict>
 <key>StandardOutPath</key>
  <string>${CURRENT_DIR}/output.log</string>
  <key>StandardErrorPath</key>
  <string>${CURRENT_DIR}/error.log</string>
</dict>
</plist>"

if launchctl list | grep -q "$PLIST_LABEL"; then
  launchctl unload "$PLIST_FILE"
fi

echo "$PLIST_CONTENT" > "$PLIST_FILE"

launchctl load -w "$PLIST_FILE"

if [ "$SCHEDULE_MINUTE" = "*/1" ]; then
  SCHEDULE_MESSAGE="every minute"
else
  SCHEDULE_MESSAGE="at minute $SCHEDULE_MINUTE of every hour"
fi

echo "Scheduled task set up to run $SCHEDULE_MESSAGE."

Step 5: Testing the Script

At this point you should be able to run the script with make run and see the results in the results.txt file. If you have any issues, check the log files for errors.

If you read through the lines in the ./setup-schedueld-task.sh file, you’ll see that it creates a .plist file for launchd. This is what allows us to schedule the task to run at regular intervals. The script also sets up environment variables for the Node.js script, which allows us to store sensitive information such as our Gmail username and password in a .env file. It also sets up log files for both the errors and outputs inthe script, which are setup to use your current projects directory, these will be useful for troubleshooting. Once you’re happy with the script, you can run make reload to reload the .plist file and make run to run the script.

Troubleshooting

Node.js and npm checks

npm package installation checks

Script execution and stream pipeline

Launchd and plist management

File existence check

File and directory permissions

Environment variable and .env file checks

Error and output logs checks

Editing and validating plist files

Script running interval adjustment

Some General troubleshooting tips

I ran into a few errors getting al lthis to run together, so here are some general tips to help you troubleshoot.

Wrapping Up

You now have an automated URL status checker running on your macOS system, with notifications to keep you informed about the status of your services. You can adjust the scheduling, notification methods, and even the Node.js script to better suit your needs.

Remember to secure your .env file, and consider using more secure methods for handling Gmail credentials, such as OAuth 2.0, if you plan to use this service extensively.

Happy Coding!

All Posts