I found the Inspur X540-T2 card for around $15 on Aliexxpress with free delivery!
It’s remarkable how these have fallen in price now. They’re still sold under Intel brand on Digitec for >$200 and were originally around $800 or so.
1 Like
I wonder if the average Joe would even notice these kind of speeds.
I get about 800mbps using ExpressVPN and over 900 without using cable connection. Home wifi is about 500mbps. This on UPC “Giga”. Frankly I experience no difference to the previous deal of 500.
If you’re running a server, then yes. But for 99% of users it’s simply a way to get you to pay more…
1 Like
In reality, you will probably not see much difference beyond 1Gb because:
- Most things you connect to will anyway be < 1 Gb (exception might be if you are torrenting so aggregating from multiple difference sources)
- Your link might anyway be shared/limited
- Your home hardware is likely to be at most 2.5Gb and more likely 1Gb
I think many providers are charging the same for 1Gb upwards so I think it is more a marketing thing.
Faster network cards are mainly required for local network transfers e.g. between computers in the same home network.
Latency goes down. But you need a device that can route at wirespeed.
That’s where you realize that a „cheap“ router still has a price.
I don’t know why the average home user would need such speeds, for example, one can download a video very quickly but what is the benefit of getting it faster than you can watch it. Anyway sooner or later you have to go back and watch the saved version at normal speed.
Overall an interesting read although most of the technicalities go over my head. I was a telecoms expert in the late sixties where my claim to fame was persuading the UK Post Office to give permission to run our 200bps modems at 300 over telephone lines.
They don’t.
1 Like
Not for Downloads and movie streaming anyway.
Regarding the NIC you bought, please note that init7 terminates fiber (UPC, green plug) in your house. So you need something with SFP+ (10G) or SFP28 (25G) ports, and you need to use a compatible BiDi LR transceiver. That is just for connecting to their network. I bought my optics off of fs.com to negotiate a 25G link with init7.
I will not share 25G to all clients. My media server will get a 25G link to the internet, and the rest of the house will likely sit at 10G or 2.5G, as the price is just too high for 25G everywhere (you need transceivers and adapters).
It may even be 10G for all devices, and 25G just for the router (so that two devices can reach their full 10G speeds). I went for a ConnectX-4 for the router (i7 7700K) and will test speeds for various appliance OSes (vyos/opnsense/openwrt), I am curious how close you can get to 25G routing on a fast single core.
For switches, I’m looking at https://www.digitec.ch/en/s1/product/qnap-qsw-m5216-1t-ddr3-1-ports-rj45-16-ports-sfp28-network-switches-17959203 as the main backbone for the house (as this provides 25G everywhere), then something like https://www.digitec.ch/en/s1/product/zyxel-xmg-105hp-5-ports-network-switches-40383111 to terminate 10G connections in a room with devices for 2.5G links, or https://www.digitec.ch/en/s1/product/mikrotik-crs305-1g-4sin-5-ports-network-switches-9876046?ip=CRS305-1G-4S%2BIN to provide 10G.
https://www.digitec.ch/en/s1/product/qnap-qsw-m3216r-8s8t-16-ports-network-switches-39444754 is also nice, but expensive and has fans (of which I am not a fan of (no pun intended) for the home office rooms).
And as for anyone asking “Why”: “Because we can”. I would argue that a 10G connection is useful indeed for backing up data on self-hosted hardware (for example all my photos for the last 15 years are at home, backing them up to a new provider takes forever), but realistically its just for the fact that “we can”.
3 Likes
vyOS is probably the fastest.
You could ask for a TNSR eval from netgate. Don’t know if they’re inclined to do that.
My guess is that TNSR could be even faster, depending on the workload.
25G hardware is crazy expensive - and mostly oversized unless you fully wire up a house with WiFi7 APs in almost every room.
I would go for a regular 1G switch with one or two 10G ports - and then replace that as hardware becomes available.
However, who has cabling for 10G to every outlet?
from the mellanox vs intel google search:
https://www.reddit.com/r/homelab/comments/112etty/what_do_i_need_to_know_about_chosing_a_nic/
/r/homlab is probably better suited for this discussion anyway - but I don’t really have a homelab 
My apartment came pre-wired with 2 runs of Cat 5e between the pantry and each of 3 network outlet boxes. While you officially should have Cat 6 for 10 Gbps, 5e is said to work for short runs, and has worked fine for me so far. Therefore, I could use a bonding setup to get 20 Gbps through the apartment and really cannot justify pulling fiber to get 25Gbps.
So far though, outside of said pantry, I’m just running 10Gbps and slower, all through a Netgear MS510TXUP to keep some APs powered. It just barely has the ports for the other stuff in the pantry - the next upgrade would be to insert a 25Gbps switch of some sort between it and the rest of the pantry. Within the pantry, SFP28 and SFP+ passive interconnects seem to be the way to go. Compare for example the price difference between a 10Gbps RJ-45 module and a 25Gbps interconnect.
Init7 has the same ongoing cost for 1Gbps through 25Gbps, just a higher setup fee for 25Gbps. I’ve always asked them to sell me an optic (1Gbps years ago, 25Gbps more recently) as one less thing to worry about.
As per my other post, I seem to have enough for 10G to every plug in practice and 20G to every network box if I bothered to set up bonding.
As for routing, I currently have an all software solution, and that software is nothing more exotic than a modern linux kernel with the netfilter kernel modules compiled in. It is a stateful firewall performing source NAT and has a few other firewall rules. Running iperf3 from an interior box with a 10Gbps card, I see reports in the 8-10Gbps range. No sign of routing bottleneck so far.
During this test, I saw one kernel thread on the router burn something like 30% of a core. So while I haven’t set up a full wire-speed test, I strongly suspect it’d pass or very nearly so.
Yes, TNSR or more broadly anything DPDK based would burn even less CPU, and if I really wanted a networking project, developing an Open Source DPDK based firewall/router would be a cool one. But I don’t think it is particularly important unless you are doing something more complex than my FW/router setup, or running beyond 25Gbps, or are really trying to keep power consumption down, etc, etc.
What CPU do you have in your router box? I was digging around and found an old Core 2 Duo class CPU (I’m guessing must be E6xxx class) and wondered how far that would stretch.
It is has a D-1736NT. I now believe that it’s little brother D-1718T would be more than sufficient what I’m actually using it for. These SOCs are not high TDP or clock speed chip by modern standards, but not budget devices either (partially from the 2x 25Gbps controller) and Core 2 is a few generations older.
So Stapelberg’s write up is probably a better reference point. But, my reaction is that if what you have is a more or less complete system and the PCI lanes are sufficient on paper, I’d try it. But if you just have a bare CPU, I’d probably look for something newer before putting money into that generation.
1 Like
Yes. I have an old Core 2 system. So old it has PCI-X slots in it, but luckily also an 8x PCIe and 4x PCIe which I can use for the network card.
I oddly discovered that my standard Swisscom IB4 has a 10GB RJ45 port. Connected this to WAN1 of my UDM Pro (SFP+), and I seem to get around 4-5GB DL / 5-7GB UL speeds.
Fast enough for me.
I did a test upgrade to 10G for a few of my computers (after getting impatient transferring large files at 2.5G speeds). They annoying thing was that although line speeds of 9G+ were easily achievable, in practice, since I was transferring mostly via SCP, these were getting bottlenecked by CPU encryption performance limiting it to between 350MB/s - 600MB/s depending on the machine.
I didn’t want to upgrade the machines, so reverted back to the 2.5G NICs as the minor speed improvement wasn’t worth the higher energy consumption and lack of multi-gig support.
1 Like
If I had a spare mini PC with PCIe slot I’d try this router-as-a-card https://www.digitec.ch/en/s1/product/mikrotik-ccr2004-1g-2xs-pcie-pci-express-30-x8-pci-express-30-pci-network-cards-25322851?ip=mikrotik+pcie
I guess that even if you have “weak” CPU, and less than x8 PCI, it’ll still work flawlessly on the two SFP28 ports, so it could serve as a good edge router, plus additional server capability of your mini PC, with whatever network bandwidth your PCI allows.
If I understand it correctly, the card will work by default as standard 2 port NIC for the host PC, unless you activate the router.
Unfortunately I don’t have any, so I’ll buy a standalone router.
The problem is that my computer can only encrypt/decrypt at around, say, 500MB/s so cannot keep the NICs fed with data.
I guess I could recompile ssh/scp on all my machines to accept the ‘None’ cipher.
I’m blaming the cipher, but it could just be other things in scp. I guess I could install hpn-ssh instead.
I believe the above router NIC will just draw power from the computer, but work independently routing traffic between those two SFP ports with no use of host CPU.