We have now added a user group privilege which can restrict users from removing the off-limits restriction on a candidate or contact record in Tracker.
With this setting enabled, users can clear this setting and bypass the off-limits. With this disabled, users would be unable to bypass the off-limits set and it would be blank as shown below: