That is not the rule. The rule is that you will have to pay the import VAT, if VAT to be paid is CHF 5 or more.
Depending on if the VAT Rate is 7.7% (8.1% in the future) or 2.5% (2.6%) this equates into an import value (including shipping cost) of CHF 64 (CHF 61.70) and CHF 200 (CHF 192) respectively.
2.5% (2.6%) has, for example books and "Lebensmittel nach dem Lebensmittelgesetz vom 20. Juni 2014, mit Ausnahme alkoholischer Getränke" / "foodstuffs under the Foodstuffs Act of 20 June 2014, with the exception of alcoholic beverages,"
Under this falls food itself, but also herbs, nutritional supplements or in general as defined in Art. 4 Foodstuffs Act
As you can see, nutritional value is not a criteria. Important is human consumption. Tea and coffee falls under the "drinks" clause. Also dry tea leaves and coffee beans. https://www.fedlex.admin.ch/eli/cc/2017/220/de
Next to VAT there is also duty. Like VAT duty has to be paid when it is CHF 5 or more. Duty can be looked up in the Tares database: https://xtares.admin.ch/tares/
Duty for green and black teas is 0 CHF/kg.
So what went wrong?
The CHF 1.70 are the VAT on the disbursement fee.
Means the CHF 6.25 is the assessed the import VAT.
One can only assume that the customs declaration by the sendelr/seller was wrong or incomplete and thus it was not clear it was tea whihc has a i,mport VAT of 2.5%
On your FedEx bill you should see a big long number, with which you can download the official e-dec customs declaration, which FedEx made on your behalf, based on the information provided by the sender.
In the end you can try and contest the bill with fedex by showing them that they did a mistake (if the senders documentation was correct and complete). Might be possible that they reimburse you.