Open, source-available — the new KeygenStar us on GitHub arrow_right_alt

How to Safely Change the Argument Signature of a Sidekiq Job

Friday, Jan 28th 2022

When I upgraded our Sidekiq version to 6.4 yesterday, I started receiving a lot of warnings while running our test suite:

Job arguments to RequestLogWorker do not serialize to JSON safely. This
will raise an error in Sidekiq 7.0. See https://github.com/mperham/sidekiq/wiki/Best-Practices
or raise an error today by calling `Sidekiq.strict_args!` during Sidekiq
initialization.

So the first thing I did was follow the link, which told me I was doing some things wrong concerning best practices. The second thing I did was throw Sidekiq.strict_args! into my Sidekiq initializer, to start raising errors instead of warnings.

These warnings were pointing out that one of my jobs, RequestLogWorker, was not following best practices. It accepted 2 hashes, a request hash and a response hash, which we used to log API requests and responses to the database, respectively.

As an example, my request hash looked like this when queuing a job:

RequestLogWorker.perform_async(
{
id: 'f14b9029-78ab-4470-9391-d665b9279b29',
time: <Fri, 28 Jan 2022 20:11:41.576563000 UTC +00:00>,
method: 'POST',
path: '/v1/accounts/demo/licenses',
},
{
# ...
}
)

According to Sidekiq best practices, there are a few problems here.

  1. The hash is using symbol keys, which is not JSON-safe. If I were expecting my hash to have symbol keys in RequestLogWorker#perform, I'd be mistaken.
  2. The time key is a Time object, not a string. If I were expecting time to be a time object in #perform, I'd once again be mistaken.

To fix these problems, we'd need to pass in the following hash:

RequestLogWorker.perform_async(
{
'id' => 'f14b9029-78ab-4470-9391-d665b9279b29',
'time' => '2022-01-28T20:23:53.943088Z',
'method' => 'POST',
'path' => '/v1/accounts/demo/licenses',
},
{
# ...
}
)

I also could've used to_json on the original hash, but that's kind of a code smell and would require modification of the job to parse the JSON. Rather than pass in JSON-safe arguments, I decided to follow best practices and not pass in any complex objects, including hashes. I decided to only pass in scalar values as positional arguments.

But then I realized there was a problem…

How do you change the argument signature of a Sidekiq job?

The job class

To give more context, let's define our current job class and its signature.

class RequestLogWorker
include Sidekiq::Worker
 
def perform(req, res)
# ...
end
end

The problem

With that, let's take the naive route! Let's change the arguments for #perform.

class RequestLogWorker
include Sidekiq::Worker
 
- def perform(req, res)
+ def perform(request_id, request_time, request_method, request_path, ...)
# ...
end
end

Seems straight forward, right? But there's a problem! If we went ahead and deployed this, any RequestLogWorker already queued would begin raising an ArgumentError. This is because Sidekiq would be trying to pass in worker.perform(req, res), while our class now takes a different set of arguments. The job's signature was changed.

So how do we work around this?

Duplicating our job

Rather than modify our existing RequestLogWorker job, we're going to create a new job class — a duplicate. The only difference in these job classes will be their arguments. We'll eventually get things back to normal, but it will take a few carefully executed steps.

class RequestLogWorker2
include Sidekiq::Worker
 
def perform(request_id, request_time, request_method, request_path, ...)
# ...
end
end

Now we have a new job class, RequestLogWorker2. We need to begin calling our new class anywhere we previously called our old job class, RequestLogWorker.

At this point, we'll want to run our tests, ensure that they pass and then deploy the new job code. During this time, the backlog of RequestLogWorker will begin to clear, and all new jobs should be using RequestLogWorker2.

At some point, the backlog for the old job class will clear. Whether this happens within minutes or hours will depend on how your jobs are processed.

Renaming our job

At this point, we should no longer have any jobs queued for RequestLogWorker. All jobs should now be for RequestLogWorker2. Our next step is to rename our job, removing the 2 suffix. Ultimately, we want to get things back to normal.

Now again, let's take the naive route! Let's simply delete RequestLogWorker and rename our RequestLogWorker2 job to RequestLogWorker.

-class RequestLogWorker
- include Sidekiq::Worker
- 
- def perform(req, res)
- # ...
- end
-end
-class RequestLogWorker2
+class RequestLogWorker
include Sidekiq::Worker
 
def perform(request_id, request_time, request_method, request_path, ...)
# ...
end
end

If we went ahead and deployed this, we'd start seeing lots of uninitialized constant errors being raised by Sidekiq. This is because Sidekiq will be attempting to process jobs for the old RequestLogWorker2 class, but that job class no longer exists.

So how do we safely rename a job?

We need to create an alias.

class RequestLogWorker
include Sidekiq::Worker
 
def perform(request_id, request_time, request_method, request_path, ...)
# ...
end
end
 
+# FIXME(ezekg) Remove after backlog for RequestLogWorker2 is clear
+RequestLogWorker2 = RequestLogWorker

After this change, we'd be safe to deploy the second step of our migration. Once again, after the backlog for RequestLogWorker2 has cleared, it will be safe to perform the last step — remove our RequestLogWorker2 alias.

class RequestLogWorker
include Sidekiq::Worker
 
def perform(request_id, request_time, request_method, request_path, ...)
# ...
end
end
 
-# FIXME(ezekg) Remove after backlog for RequestLogWorker2 is clear
-RequestLogWorker2 = RequestLogWorker

Things should now be back to normal.

In conclusion

Today we learned how to resolve warnings introduced in Sidekiq 6.4. We learned how to change the argument signature of a job class, as well as how to safely rename a class to get things back to normal. Each step here is not too complicated, but they do need to be performed carefully to avoid errors in production.

Using this technique, we were able to migrate a few job classes from accepting complex objects to accepting simple scalar values. (One of the jobs even used keyword arguments, but judging by this new warning, I'm assuming that is not what the Sidekiq maintainer wants us to be doing.)

Until next time.


If you find any errors in my code, or if you can think of ways to improve things, ping me via Twitter.