Monitoring DigitalOcean Apps with GitHub Actions

I just added a feature to all of my app platform hosted apps on DigitalOcean (DO) that I wanted to share here. It has to do with automated error monitoring.

What’s the issue?

When we deploy code to the wild, sooner or later we will have unexpected/unhandled exceptions bubble up. Third-party services failing in ways we didn’t expect, and users may run into edge cases in our code we hadn’t considered, etc.

There are two primary ways for developers to find out about these issues: 1) by getting user reports about “things not working”, or 2) by looking at their app run-time logs and finding unhandled exception stack traces in the logs.

It’s a good idea to know about these issues ahead of user reports, and that’s where some sort of automated error reporting comes in. Usually, some jobs are run that look at the logs, and if they detect an error, an alert is generated.

Also, it’s not uncommon to have developers check out runtime logs from time to time manually to make sure all is well.

DigitalOcean runtime logs

DO apps have the ability to expose runtime logs via their UI or API. Here is what it looks like on their UI:

For automation tasks, they provide a CLI utility, doctl, which can be used to run a runtime log command:

doctl apps logs {app-id}

Automated monitoring with GitHub action

You can avoid having to login into the console to check out the runtime logs and instead use GitHub actions to run a periodic check for you. Here is the template of the action I have written that I use:

name: App Log Monitoring

on:
    schedule:
        - cron: '*/5 * * * *'  # Run every 5 minutes
          
    # also run on push to main branch, but only if applog_monitoring.yml file was in the changes
    push:
        branches:
            - main
        paths:
            - .github/workflows/applog_monitoring.yml

jobs:
    monitor_logs:
        runs-on: ubuntu-latest
        
        steps:
            - name: Checkout code
              uses: actions/checkout@v2

            - name: Install doctl
              uses: digitalocean/action-doctl@v2
              with:
                  token: ${{ secrets.DIGITALOCEAN_ACCESS_TOKEN }}

            - name: Retrieve app logs
              id: get_logs
              run: |
                  echo "APP_LOGS<<EOF" >> $GITHUB_OUTPUT
                  doctl apps logs ${{ secrets.DIGITALOCEAN_APP_ID }} >> $GITHUB_OUTPUT
                  echo "EOF" >> $GITHUB_OUTPUT
            - name: Output logs
              run: echo "${{ steps.get_logs.outputs.APP_LOGS }}"
                  
            - name: Check for errors
              id: check_errors
              if: contains(steps.get_logs.outputs.APP_LOGS, 'error:') || contains(steps.get_logs.outputs.APP_LOGS, 'fail:')
              env:
                  GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
              run: |
                  echo "Errors found in the app logs:"
                  echo "${{ steps.get_logs.outputs.APP_LOGS }}"
                  echo "Disabling the workflow to prevent further notifications."
                  gh workflow disable "App Log Monitoring"
                  exit 1

What it does is use `doctl` to query the runtime logs, and check for the presence of error level logs. If it finds a match, it fails the action. The failing action will result in GitHub sending me an emailm which I then can inspect and see what’s happening.

Note that when a failure occurs, I disable the action with GitHub’s CLI utility because we don’t want the job to periodically be pinging us about this, in case we don’t have a chance to address the issue right away.

You will need to generate a DO API access token and store it in repo’s secrets, as well as obtain the application ID and store it in the secrets so that it can be used by DO’s utility.

This has been a nice time saver and a safety net where I get notified if something not so nice is happening on the server, and I can address it right away.