Some of the issue is the lack of people close to your power/level not joining as well. I am seeing a much decline in who those type of joining matsuri so I am left with matching with whales regardless of a win or loss

It all comes down to get the maximum amount of in game resources you can get within a week and your choice of not getting them. If you are able max Sun and moon coins without doing matsuri and SWB, then fine you can skip it. For me, however, I would rather get as much as I can as it would just hurt my progress if I just don't participate at all. I understand some aren't willing to allow people beat them but they should accept their effort contributes to their progress in any game.