tutorial

.NET — Hangfire

Using Hangfire to recurrently trigger HTTP endpoints

One common scenario when developing applications is the need to perform background processing than can either be run only once (like sending an email) or scheduled to be run multiple times within a given interval (like doing database housekeeping), usually defined by a cron expression.

When I need to implement such requirements my first choice for the last few years as always been Hangfire.

It integrates seamlessly with ASP.NET Core applications, it has a simple but very powerful dashboard to monitor and manually trigger recurring jobs and is open source and completely free for commercial use.

In this article I’m going to explain how to configure both Hangfire server and dashboard into an ASP.NET Core application and recurrently send requests to HTTP endpoints, which will be configured based on application settings without requiring a service restart to detect changes.

Why use Hangfire to trigger HTTP endpoints?

An extremely common approach I use when implementing schedulers that may run millions of jobs per day is to use Hangfire to trigger HTTP endpoints, offloading work to external APIs instead of in process.

By design, Hangfire is implemented to be easily distributed on different machines by orchestrating the job execution using a persistent data storage shared across all servers.

The problem comes when some jobs need more processing power than others or millions of executions per day. You’ll have to maintain multiple nodes looking at different queues, code changes must be properly distributed to the relevant nodes, the database will probably become a bottleneck - even if using some message bus or in-memory cache.

To attenuate this problem I usually prefer to keep Hangfire instances as simple as possible, by recurrently doing simple HTTP requests that can be configured either by settings files or database providers (like exemplified in this article I recently made). This allows to offload the heavy work to external APIs that can be more easily scaled, tested, distributed and monitored and ensuring you’ll need much fewer Hangfire instances to run millions of jobs per day while using the dashboard to keep track of job executions.

Solution setup

In this example, which I have available on GitHub, I’m going to setup an ASP.NET Core application with .NET 8 to host both Hangfire server and dashboard. For simplicity, the job storage will be in-memory but feel free to use any other of the supported ones (like SQL Server).

Start by creating an empty ASP.NET Core project and register the following NuGet packages:

  • Hangfire.AspNetCore - depends on Hangfire.NetCore, which is used to run the jobs server on Microsoft.Extensions.Hosting as a hosted service, but also brings a Web dashboard that can be used to manually trigger recurring jobs or check execution status;
  • Hangfire.InMemory - in-memory job storage but you can swap for any other of your preference, like Hangfire.SqlServer;

Open the Program.cs file, register both the server and dashboard services into the dependency injection container and configure the dashboard to listen on base address:

1
2
3
4
5
6
7
8
9
10
11
12
var builder = WebApplication.CreateBuilder(args);

builder.Services.AddHangfire(config => config
    .UseInMemoryStorage()
);
builder.Services.AddHangfireServer();

var app = builder.Build();

app.MapHangfireDashboard("");

app.Run();

If you run the application, the Hangfire dashboard will be shown with a single server running but without recurring jobs.

As you can see, in just a few minutes you have Hangfire up and ready do run. As stated previously, this is one of the reasons I always use Hangfire for scheduling jobs, the simplicity of it!

HTTP job options

Now that we have both the server and dashboard running, let’s start by defining the recurring HTTP options that we’ll read from the application settings and dynamically configure endpoints to be recurrently triggered.

Let’s start by defining the options for requesting a single endpoint. To keep things simple, I’ll assume the need for an HTTP address, method, headers and a request timeout. It will also need a cron expression, so we can define the recurrency interval, and a flag to enable it so you can manage per environment which jobs are enabled.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
public class HttpJobEndpointOptions
{
    public string Cron { get; init; }

    public bool Enabled { get; init; }

    public string Address { get; init; }

    public string Method { get; init; }

    public IReadOnlyDictionary<string, string> Headers { get; init; }

    public TimeSpan? Timeout { get; init; }

    public bool IgnoreInvalidStatusCode { get; init; }
}

We could now load a collection of HttpJobEndpointOptions from the application settings but let’s try to organize the endpoints into categories and also give a name for each. This will be handy later to have a more readable job name when looking at the Hangfire dashboard.

1
2
3
4
5
6
7
8
9
public class HttpJobOptions
{
    public TimeSpan DefaultTimeout { get; init; } = TimeSpan.FromSeconds(5);

    public IReadOnlyDictionary<
        string, 
        IReadOnlyDictionary<string, HttpJobEndpointOptions>
    > Endpoints { get; init; }
}

Open the appsettings.json file and configure an IsAlive job inside a Core category that’ll run every 5 minutes. This job will be used later for testing, so assume an HTTP request GET /api/is-alive to itself:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
{
  "HttpJobs": {
    "DefaultTimeout": "00:00:05",
    "Endpoints": {
      "Core": {
        "IsAlive": {
          "Cron": "*/5 * * * *",
          "Enabled": true,
          "Address": "https://localhost:7076/api/is-alive",
          "Method": "GET",
          "Headers": {
            "accept": "application/json"
          },
          "Timeout": null,
          "IgnoreInvalidStatusCode": false
        }
      }
    }
  },
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "AllowedHosts": "*"
}

Open the Program.cs file and configure the HttpJobOptions from the HttpJobs section. Let’s also add the testing endpoint to be triggered later:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
var builder = WebApplication.CreateBuilder(args);

builder.Services.AddHangfire(config => config
    .UseInMemoryStorage()
);
builder.Services.AddHangfireServer();

builder.Services.Configure<HttpJobOptions>(
    builder.Configuration.GetSection("HttpJobs")
);

var app = builder.Build();

app.MapGet("/api/is-alive", () => "I'm alive!");

app.MapHangfireDashboard("");

app.Run();

HTTP job runner

Now that we have defined the job options, let’s create the class that will be recurrently executed by the Hangfire server.

The core idea of this class is to be responsible to do a single HTTP request by receiving both the category and endpoint names as a parameter, looking up for the configuration in the HttpJobOptions instance and doing the expected HTTP request. It will also receive a PerformContext instance which is a special (but optional) class provided by Hangfire to give context to your methods.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
public class HttpJobRunner(
    ILogger<HttpJobRunner> logger,
    IOptionsMonitor<HttpJobOptions> options,
    IHttpClientFactory clientFactory
)
{
    private HttpJobOptions Options => options.CurrentValue;

    public async Task RunAsync(string category, string name, PerformContext ctx)
    {
        using var _ = logger.BeginScope(
            "Category:{Category} Name:{Name} JobId:{JobId}",
            category,
            name,
            ctx.BackgroundJob.Id
        );

        if (!Options.Endpoints.TryGetValue(category, out var endpoints))
        {
            logger.LogWarning("Configuration for category not found");
            return;
        }

        if (!endpoints.TryGetValue(name, out var endpoint))
        {
            logger.LogWarning("Configuration for endpoint not found");
            return;
        }

        if (!endpoint.Enabled)
        {
            logger.LogWarning("Endpoint was disabled while still scheduled, nothing will be done");
            return;
        }

        using var client = clientFactory.CreateClient(
            $"{nameof(HttpJobRunner)}.{category}.{name}"
        );

        client.Timeout = endpoint.Timeout ?? Options.DefaultTimeout;

        logger.LogDebug(
            "Starting HTTP request '{Method} {Address}' [Timeout:{Timeout}]",
            endpoint.Method,
            endpoint.Address,
            client.Timeout
        );

        using var request = new HttpRequestMessage(
            new HttpMethod(endpoint.Method),
            endpoint.Address
        );

        if (endpoint.Headers?.Count > 0)
        {
            foreach (var (key, value) in endpoint.Headers)
                request.Headers.Add(key, value);
        }

        var ct = ctx.CancellationToken.ShutdownToken;

        HttpResponseMessage response;
        try
        {
            response = await client.SendAsync(request, ct);
        }
        catch (TaskCanceledException e) when (!ct.IsCancellationRequested)
        {
            throw new TimeoutException("Request timed out", e);
        }

        using (response)
        {
            logger.LogInformation(
                "Completed HTTP request '{Method} {Address}' with status code {StatusCode}",
                endpoint.Method,
                endpoint.Address,
                response.StatusCode
            );

            if (!endpoint.IgnoreInvalidStatusCode)
                response.EnsureSuccessStatusCode();
        }
    }
}

Register into the DI container the HTTP client services (because we are using the IHttpClientFactory) and also this class as transient.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
var builder = WebApplication.CreateBuilder(args);

builder.Services.AddHangfire(config => config
    .UseInMemoryStorage()
);
builder.Services.AddHangfireServer();

builder.Services.Configure<HttpJobOptions>(
    builder.Configuration.GetSection("HttpJobs")
);

builder.Services.AddHttpClient();
builder.Services.AddTransient<HttpJobRunner>();

var app = builder.Build();

app.MapGet("/api/is-alive", () => "I'm alive!");

app.MapHangfireDashboard("");

app.Run();

HTTP hosted service

It is time to let Hangfire know that we have endpoints that must be requested on a recurrent basis. 

To configure the class HttpJobRunner as a recurring job, we can use either the static method RecurringJob.AddOrUpdate or the equivalent methods provided by IRecurringJobManager.

A manual registration of the HTTP:Core:IsAlive job we defined in the application settings would be as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
// using static class RecurringJob
RecurringJob.AddOrUpdate<HttpJobRunner>(
    "HTTP:Core:IsAlive",
    runner => runner.RunAsync("Core", "IsAlive", null),
    "*/5 * * * *",
    new RecurringJobOptions
    {
        TimeZone = TimeZoneInfo.Utc
    }
);

// using IRecurringJobManager instance
jobManager.AddOrUpdate<HttpJobRunner>(
    "HTTP:Core:IsAlive",
    runner => runner.RunAsync("Core", "IsAlive", null),
    "*/5 * * * *",
    new RecurringJobOptions
    {
        TimeZone = TimeZoneInfo.Utc
    }
);

As you can see, we pass an expression to the AddOrUpdate method that will be used by Hangfire to build dynamic code to call the RunAsync method of our HttpJobRunner class, including the parameters we specify. As stated before, the PerformContext parameter is special and despite we are passing null when defining the expression, Hangfire will actually create a context each time the method is run.

With this simple yet effective approach, Hangfire doesn’t impose any interface or abstract class that must be implemented, making our life much easier.

This registration could be done at application startup but since we want do detect changes to the application settings without restarting the server, let’s create a hosted service that will be running in the background and either add or remove scheduler jobs. As a note, since Hangfire doesn’t provide a way to get currently registered jobs so we can remove inexistent, we’ll register the hosted service as a singleton and have a collection in-memory to track registered job ids.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
public class HttpJobHostedService(
    ILogger<HttpJobHostedService> logger,
    IOptionsMonitor<HttpJobOptions> options,
    IRecurringJobManager jobManager
) : IHostedService
{
    private readonly HashSet<string> _jobIds = new();
    private IDisposable _onOptionsChange;

    public Task StartAsync(CancellationToken ct)
    {
        _onOptionsChange = options.OnChange(jobOptions =>
        {
            logger.LogDebug("Configuration changed, scheduling jobs");
            ScheduleJobs(jobOptions);
        });

        logger.LogDebug("Initial configuration, scheduling jobs");
        ScheduleJobs(options.CurrentValue);

        return Task.CompletedTask;
    }

    public Task StopAsync(CancellationToken ct)
    {
        _onOptionsChange?.Dispose();

        return Task.CompletedTask;
    }

    private void ScheduleJobs(HttpJobOptions options)
    {
        var currentJobIds = new HashSet<string>();

        if (options.Endpoints is { Count: > 0 })
        {
            foreach (var (category, endpoints) in options.Endpoints)
            {
                if (endpoints is not { Count: > 0 })
                    continue;

                foreach (var (name, endpoint) in endpoints)
                {
                    if (!endpoint.Enabled)
                        continue;

                    var jobId = $"HTTP:{category}:{name}";

                    jobManager.AddOrUpdate<HttpJobRunner>(
                        jobId,
                        runner => runner.RunAsync(category, name, null),
                        endpoint.Cron,
                        new RecurringJobOptions
                        {
                            TimeZone = TimeZoneInfo.Utc
                        }
                    );

                    currentJobIds.Add(jobId);
                }
            }
        }

        _jobIds.RemoveWhere(jobId => currentJobIds.Contains(jobId));

        foreach (var jobId in _jobIds) 
            jobManager.RemoveIfExists(jobId);

        _jobIds.Clear();

        foreach (var jobId in currentJobIds)
            _jobIds.Add(jobId);
    }
}

Register this class into the Dependency Injection container as a hosted service.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
var builder = WebApplication.CreateBuilder(args);

builder.Services.AddHangfire(config => config
    .UseInMemoryStorage()
);
builder.Services.AddHangfireServer();

builder.Services.Configure<HttpJobOptions>(
    builder.Configuration.GetSection("HttpJobs")
);

builder.Services.AddHttpClient();
builder.Services.AddTransient<HttpJobRunner>();

builder.Services.AddHostedService<HttpJobHostedService>();

var app = builder.Build();

app.MapGet("/api/is-alive", () => "I'm alive!");

app.MapHangfireDashboard("");

app.Run();

If you now run the application, the Hangfire dashboard will show the HTTP:Core:IsAlive recurring job to be run every 5 minutes.

Try to change the application settings while the application is running, like changing the cron expression, disabling, removing or creating jobs, so you can see the recurring job list to change accordingly.

If you wait for the jobs to run (or manually trigger) and go to the Jobs tab, open the Succeeded status and you’ll see the list of all jobs that have run successfully.

If you open one of the jobs, you’ll see details about it’s execution, like the parameters that were passed, the execution time and, if it was failed, the exception details.

Job filters

The code is looking nice and simple and seems to work very well but there’s still three problems that I think should be solved: 

  1. If you have multiple HTTP endpoints you can’t tell just looking at the list which one was called without checking execution details. In this case every job is called HttpJobRunner.RunAsync, independently the endpoint;
  2. Since this is a recurring and stateless job so, when it fails, there’s no point in keeping it in the failed state to be retried, we can move it to the deleted stated that would be automatically housekept by Hangfire;
  3. If for some reason a job for a given endpoint takes more time to run than its cron interval, multiple executions will overlap and cause requests to happen concurrently, which may not be the expected behavior;

To solve these problems we are going to use Hangfire filters, which are attributes that can be used to annotate methods (or registered globally) to add custom behavior when running a job, working similarly to ASP.NET Core MVC filters.

We are going to use the JobDisplayName filter to define the job name in the dashboard, the AutomaticRetry filter to mark the job as deleted when failed, and create a custom filter than will lock job executions per endpoint until the previous one has completed.

Starting by the custom filter, we must extend the JobFilterAttribute and because it’s changing the behavior of a job execution, it must also implement the IServerFilter interface. We are going to use both the category and endpoint name parameters to create a unique identifier per endpoint execution and use an Hangfire distributed lock feature to ensure a job for the same endpoint only runs after the lock is disposed by a previous execution.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
public class EnableDistributedMutexAttribute : JobFilterAttribute, IServerFilter
{
    private const string Key = nameof(EnableDistributedMutexAttribute);

    private readonly string _nameFormat;
    private readonly TimeSpan _timeout;

    public EnableDistributedMutexAttribute(string nameFormat, int timeoutInSeconds)
    {
        ArgumentNullException.ThrowIfNull(nameFormat);
        ArgumentOutOfRangeException.ThrowIfLessThanOrEqual(timeoutInSeconds, 0);

        _nameFormat = nameFormat;
        _timeout = TimeSpan.FromSeconds(timeoutInSeconds);
    }

    public void OnPerforming(PerformingContext ctx)
    {
        var distributedLockName = string.Format(
            _nameFormat,
            ctx.BackgroundJob.Job.Args.ToArray()
        );

        var distributedLock = ctx.Connection.AcquireDistributedLock(
            distributedLockName,
            _timeout
        );

        ctx.Items[Key] = distributedLock;
    }

    public void OnPerformed(PerformedContext ctx)
    {
        if (!ctx.Items.TryGetValue(Key, out var distributedLock))
            throw new InvalidOperationException("Can not release a distributed lock: it was not acquired.");

        ((IDisposable)distributedLock).Dispose();
    }
}

Open the HttpJobRunner class and annotate the RunAsync method with a JobDisplayName, EnableDistributedMutex and AutomaticRetry attributes.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
public class HttpJobRunner(
    ILogger<HttpJobRunner> logger,
    IOptionsMonitor<HttpJobOptions> options,
    IHttpClientFactory clientFactory
)
{
    private HttpJobOptions Options => options.CurrentValue;

    [JobDisplayName("HTTP:{0}:{1}")]
    [EnableDistributedMutex("HTTP:{0}:{1}", 15)]
    [AutomaticRetry(OnAttemptsExceeded = AttemptsExceededAction.Delete, Attempts = 0)]
    public async Task RunAsync(string category, string name, PerformContext ctx)
    {
        // ...
    }
}

If you now run the application and wait or manually trigger the HTTP:Core:IsAlive job you’ll see the new display name in the dashboard.

Feel free to also force some exceptions to see the job in the deleted state instead of failed, or delaying the job execution for more time than the defined cron interval to see jobs waiting for the previous ones to complete.

Conclusion

In this article I explained how easy it is to configure the Hangfire server to run recurring jobs in the background of an ASP.NET Core application, and use its dashboard to manage and monitor job executions.

For simplicity we used the in-memory job storage and created the base architecture for jobs that recurrently trigger HTTP endpoints, everything configured using application settings.

As a reminder, this example is available on GitHub so feel free to give it a look.