How to Safely Change the Argument Signature of a Sidekiq Job
Friday, Jan 28th 2022
When I upgraded our Sidekiq version to 6.4 yesterday, I started receiving a lot of warnings while running our test suite:
Job arguments to RequestLogWorker do not serialize to JSON safely. Thiswill raise an error in Sidekiq 7.0. See https://github.com/mperham/sidekiq/wiki/Best-Practices or raise an error today by calling `Sidekiq.strict_args!` during Sidekiqinitialization.So the first thing I did was follow the link, which told me I was doing some things wrong
concerning best practices. The second thing I did was throw Sidekiq.strict_args! into
my Sidekiq initializer, to start raising errors instead of warnings.
These warnings were pointing out that one of my jobs, RequestLogWorker, was not following
best practices. It accepted 2 hashes, a request hash and a response hash, which we used
to log API requests and responses to the database, respectively.
As an example, my request hash looked like this when queuing a job:
RequestLogWorker.perform_async( { id: 'f14b9029-78ab-4470-9391-d665b9279b29', time: <Fri, 28 Jan 2022 20:11:41.576563000 UTC +00:00>, method: 'POST', path: '/v1/accounts/demo/licenses', }, { # ... })According to Sidekiq best practices, there are a few problems here.
- The hash is using symbol keys, which is not JSON-safe. If I were expecting my
hash to have symbol keys in
RequestLogWorker#perform, I'd be mistaken. - The
timekey is aTimeobject, not a string. If I were expecting time to be a time object in#perform, I'd once again be mistaken.
To fix these problems, we'd need to pass in the following hash:
RequestLogWorker.perform_async( { 'id' => 'f14b9029-78ab-4470-9391-d665b9279b29', 'time' => '2022-01-28T20:23:53.943088Z', 'method' => 'POST', 'path' => '/v1/accounts/demo/licenses', }, { # ... })I also could've used to_json on the original hash, but that's kind of a code smell and
would require modification of the job to parse the JSON. Rather than pass in JSON-safe
arguments, I decided to follow best practices and not pass in any complex objects, including
hashes. I decided to only pass in scalar values as positional arguments.
But then I realized there was a problem…
How do you change the argument signature of a Sidekiq job?
The job class
To give more context, let's define our current job class and its signature.
class RequestLogWorker include Sidekiq::Worker def perform(req, res) # ... endendThe problem
With that, let's take the naive route! Let's change the arguments for #perform.
class RequestLogWorker include Sidekiq::Worker - def perform(req, res) + def perform(request_id, request_time, request_method, request_path, ...) # ... end endSeems straight forward, right? But there's a problem! If we went ahead and deployed this,
any RequestLogWorker already queued would begin raising an ArgumentError. This is
because Sidekiq would be trying to pass in worker.perform(req, res), while our class
now takes a different set of arguments. The job's signature was changed.
So how do we work around this?
Duplicating our job
Rather than modify our existing RequestLogWorker job, we're going to create a new
job class — a duplicate. The only difference in these job classes will be their
arguments. We'll eventually get things back to normal, but it will take a few
carefully executed steps.
class RequestLogWorker2 include Sidekiq::Worker def perform(request_id, request_time, request_method, request_path, ...) # ... endendNow we have a new job class, RequestLogWorker2. We need to begin calling our new
class anywhere we previously called our old job class, RequestLogWorker.
At this point, we'll want to run our tests, ensure that they pass and then deploy
the new job code. During this time, the backlog of RequestLogWorker will begin
to clear, and all new jobs should be using RequestLogWorker2.
At some point, the backlog for the old job class will clear. Whether this happens within minutes or hours will depend on how your jobs are processed.
Renaming our job
At this point, we should no longer have any jobs queued for RequestLogWorker. All
jobs should now be for RequestLogWorker2. Our next step is to rename our job,
removing the 2 suffix. Ultimately, we want to get things back to normal.
Now again, let's take the naive route! Let's simply delete RequestLogWorker and
rename our RequestLogWorker2 job to RequestLogWorker.
-class RequestLogWorker - include Sidekiq::Worker- - def perform(req, res)- # ...- end-end -class RequestLogWorker2 +class RequestLogWorker include Sidekiq::Worker def perform(request_id, request_time, request_method, request_path, ...) # ... end endIf we went ahead and deployed this, we'd start seeing lots of uninitialized constant
errors being raised by Sidekiq. This is because Sidekiq will be attempting to process
jobs for the old RequestLogWorker2 class, but that job class no longer exists.
So how do we safely rename a job?
We need to create an alias.
class RequestLogWorker include Sidekiq::Worker def perform(request_id, request_time, request_method, request_path, ...) # ... end end +# FIXME(ezekg) Remove after backlog for RequestLogWorker2 is clear +RequestLogWorker2 = RequestLogWorker After this change, we'd be safe to deploy the second step of our migration. Once again,
after the backlog for RequestLogWorker2 has cleared, it will be safe to perform the
last step — remove our RequestLogWorker2 alias.
class RequestLogWorker include Sidekiq::Worker def perform(request_id, request_time, request_method, request_path, ...) # ... end end -# FIXME(ezekg) Remove after backlog for RequestLogWorker2 is clear -RequestLogWorker2 = RequestLogWorker Things should now be back to normal.
In conclusion
Today we learned how to resolve warnings introduced in Sidekiq 6.4. We learned how to change the argument signature of a job class, as well as how to safely rename a class to get things back to normal. Each step here is not too complicated, but they do need to be performed carefully to avoid errors in production.
Using this technique, we were able to migrate a few job classes from accepting complex objects to accepting simple scalar values. (One of the jobs even used keyword arguments, but judging by this new warning, I'm assuming that is not what the Sidekiq maintainer wants us to be doing.)
Until next time.
If you find any errors in my code, or if you can think of ways to improve things, ping me via Twitter.