Usually what the practice would evolve to is accepting observation of what is, which in turn creates that calming of the body and mind because there is no more resistance to whatever is present to the moment which would create these tensions in the first line. SO whenever you are trying to "calm the body" by in fact resisting your body not being calm right now hence wanting it to be calm and trying (eventually hard) to calm your body and/or mind, this will obviously create more tension because you are not calming by accepting reality as it is.
To get from a tensious state into a calming state thereof the first step seems to be to accept any tensions being there fully without resisting them which in turn may eventually produce (in experience and as practice continues it certainly will) a calming, because you lessen your resistance more and more - and your tensions are in fact just that, resistances. You won`t get rid of them by resisting them, calm happens through acceptance, which is equanimity.
Thereof, "calming" is a more or less indirect path in my experience, which eventually becomes kind of direct once you get "the hang of it", which is the right amount of effort TO let go of resistances, a balance act of "letting go and putting effort into that" - which is kinda contradictory, because effortlessness is kind of implied in "letting go"; and it takes effort to get there oddly enough.
That`s the best way I could describe it, and in relation to that:
"I can also calm the breath by observing and letting the breath become more fluid. After I have done all this I let the rest be. If I am still tense or breathing rapid then I just accept it and observe, trying to steady the mind and being equal to whatever happens.
By accepting the tenseness or the rapidity of breath there is space created for calm/release to happen. That comes with broader sense of equanimity, so I think you`re on the right track by just accepting things as they are and let any consequences follow naturally without your influence, expectation or wishes. Whatever happens is bound to happen, and in that practice, it`s calm more often than not :-P If you have an initial sense of calming your mind/body (as a result of consistent practice), that's wonderful to do at the start before, as you say, you "let the rest be". That`s usually also how I handle it, not always - it differs from sitting to sitting. When I don`t feel it is necessary, which is when I feel my level of equanimity is stable and broad enough for calm to happen by itself over time, I just start scanning or being aware of all my body and/or breath and let itself release as an eventual consequence.
I feel like I have been too detailed here, but hey maybe it helps :-P Always good to not get hung up in the words but where they point you at, which you will find in your own personal experience of course!
Have fun practicing! :-)
Ps: Instead of equanimity, I have also invented the synonym of "non-resistance" to my practice, as in, whatever I encounter, I will not resist it in the first line. Which is basically the same as equanimity or acceptance, but gives another perspective or dimension to the way I practice, because I won`t forget to not-resist even when things get kinda.. messy or uncomfortable, and first impulse would be to resist that, because we are usually craving the goods (e.g. calm) and avoiding the bads (e.g. tensions). Especially when I think that I`m "done" now practicing, and feel like the urge to stand up now or stop earlier than the time set, I be like "I`m done not-resisting? really? how does that work?" :-D and I keep practicing, most of the time :-P Which will eventually bring calm again. But it`s easy to get hung up in stuff like that is it.