You may want to have a look at permissions at the Lifecycle level, which (as far as I understand it) work with the Team applied. Context Team Roles generally populate Object Team Roles when an Assigned Activity uses Context Team as a Resource Pool. What I would try is adding the Reviewer Role to your Change Activity Lifecycle (Roles Tab) and then set permissions on the Access Tab for the Reviewer Role. The out of the box Change Activity Team has (I suspect) the Reviewer Role in it, but it looks like the Lifecycle does not.
I suspect that when the Design Engineer permission was removed there was no remaining permission. My guess/hope is that if you apply the permission at the Lifecycle level then you'll get the Object by Object permissions that you are looking for.