After making the controller models vanish when holding an interactable. I had to move the Hand tag ridged body from the controller prefab to the XR toolkit prefab as to not disturb the hand count on the toolkit animation. otherwise, the hands would not register to leave the toolkit collider.
Drop script