I think it could work. Just depends how they use it.
If you take away the option of the referee going to the video ref, and put the onus on the players to challenge decisions, then it takes a lot of pressure off the match officials and may even speed the game up.
For example, Team A score a try, the referee wants to give the try but a player from Team B challenges the decision. Video Referee called upon. No try. Team B still has their 3 challenges.
Or...
Team A score a try, the referee wants to disallow the try. Player from Team A challenges. Video Referee called upon. No try. Team A loses a challenge.
The same situation but with the referee having contrasting views. It will stop players trying to claim for everything and anything under the Sun if they're insignificant, because it won't be worth losing a "challenge" for if they've got it wrong. All it takes is the ref to say "do you want to challenge that?", them to say "no" and the ref to say "go away then!"