Thank you for the criticism, I totally agree. I think the shaky cam in the beginning feels wrong because it moves a bit too much for how minor the actions are on screen, in the second half there is a lot more body movement so the larger camera moves feel more appropriate and are better motivated.
I am surprised I didn't notice the timing issues before, I think it's still a good idea to leave points in the fight to let the viewer process for a moment and catch their breath but the pauses in this often overstay their welcome. I will keep these things in mind next time I start an action scene and do my best to not make the same mistakes, thank you for the feedback.