Ok, I owed you this. Here's one part of the Pinch-from-the-middle-of-your-fingers demo I've been working on. (The original project is way bigger and can't share it for the time being).
For this demo, I decided to go with Joshua Tucker's awesome method to connect the module to a Layer. I removed all your logic though! I included mine specifically to demostrate this solution.
The key function that solved the issue I was trying to solve is _fixScaledObject, please check it out. Also, there are better solutions in this group to deal with scale, so that's one of my next steps for sure.
If you want to go into further observation, I left commented some lines to handle the draggable behavior of the layer. There's certainly a (small) difference, so that's something worth exploring.
And please, before pointing to a random Pinch demo, refer to this conversation ( https://www.facebook.com/groups/framerjs/permalink/752433014883772/ ) to fully understand what this is trying to solve, specially the issue of changing the originX & originY values of a scaled layer.
This is super abstract, but I can put together a post with some sketches and visual cues to tell my journey through problem-frustation-solution. It was a good challenge.