What a time to be alive! I learn now that there is up to 25Gbps internet connection available here in Switzerland!
I wondered are any of you on 10Gbps or 25Gbps? How is the actual performance?
I think some providers have 10Gbps on P2MP instead of P2P so wondering whether 1Gbps on P2P might be even better than 10Gbps on P2MP if there’s a lot of contention.
I just saw on Init7’s page that they took action against Swisscomm so if I understand correctly, future installations should be P2P anyway:
For Internet providers with a market share of less than 30%, installing and maintaining a splitter is not worthwhile. The reason for this is the fact that the splitters are placed close to the end customers and therefore only a small number of customers can be served per splitter. Smaller Internet providers can therefore hardly operate the last section of the route themselves for economic reasons. Instead, they must rent the services of the company that did the network expansion. Init7 is committed to a monopoly-free and liberal Internet and is therefore actively fighting against expansion using P2MP network topology.
On December 9, 2020, Init7 filed a complaint against Swisscom with the Competition Commission (COMCO). On December 17, 2020, the ComCo decided that Swisscom is not allowed to expand the fiber optic network according to the P2MP network topology until a final ruling is available. Swisscom has lodged an appeal against these so-called precautionary measures.
Addendum:
I’m on 10Gig Swisscom fibre, but I think it is shared…ie. if my neighbours are heavy users, my speed could suffer.
I just did a speed test using my UDM Pro SE: 902 Mbps DL, 927 Mbps upload.
I’m not sure if I’ve seen any reports of > 1Gb, but that might be because few people have >1Gb clients after the router.
This guy has a blog:
What a time to be alive, indeed.
Thanks for the links. Just a few months ago I experimented with upgrading a few of my machines to 2.5Gbps using cheap Chinese NICs/Switches off Aliexpress. I’m pretty happy with the results.
If I can go to 10Gbps without breaking the bank, I’d be interested in doing that if anyone has experience or tips on that.
I would also suggest the ConnectX cards. You can get them via eBay.de
As I’m on a work-sponsored 50/25 connection and it doesn’t look that my apartment-building at the very end of the village is getting Fiber any time soon, I can’t really relate any personal experience.
But the people who build our Openstack servers at work also use them (though probably ConnectX5 and later by now).
I went with the 25Gbps option from Init7. Running speedtest-cli on my border machine tends to only show 3-4Gbps, while iperf seems to have less of an internal bottleneck and often shows around 15Gbps.
Unless you are doing something very specific, my current thinking is that 10Gbps is the actual sweet spot. It isn’t so easy to saturate 10Gbps, and there are a lot more prebuilt, e.g. SOHO-ish, router options.
$ speedtest-cli
Retrieving speedtest.net configuration...
Testing from Init7 (Switzerland) Ltd. (85.X.X.X)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Init7 AG (Winterthur) [24.X km]: 14.676 ms
Testing download speed................................................................................
Download: 3667.49 Mbit/s
Testing upload speed......................................................................................................
Upload: 3011.86 Mbit/s
$ iperf3 -c speedtest.init7.net
Connecting to host speedtest.init7.net, port 5201
[ 5] local 85.X.X.X port 47600 connected to 77.109.175.63 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 2.52 GBytes 21.6 Gbits/sec 0 3.06 MBytes
[ 5] 1.00-2.00 sec 2.57 GBytes 22.0 Gbits/sec 0 3.06 MBytes
[ 5] 2.00-3.00 sec 2.00 GBytes 17.2 Gbits/sec 403 1.72 MBytes
[ 5] 3.00-4.00 sec 1.65 GBytes 14.2 Gbits/sec 98 1.83 MBytes
[ 5] 4.00-5.00 sec 1.33 GBytes 11.4 Gbits/sec 133 1.29 MBytes
[ 5] 5.00-6.00 sec 1.49 GBytes 12.8 Gbits/sec 0 1.99 MBytes
[ 5] 6.00-7.00 sec 1.80 GBytes 15.5 Gbits/sec 103 1.48 MBytes
[ 5] 7.00-8.00 sec 1.40 GBytes 12.1 Gbits/sec 146 1.49 MBytes
[ 5] 8.00-9.00 sec 1.81 GBytes 15.5 Gbits/sec 0 2.20 MBytes
[ 5] 9.00-10.00 sec 2.11 GBytes 18.2 Gbits/sec 126 2.21 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 18.7 GBytes 16.1 Gbits/sec 1009 sender
[ 5] 0.00-10.00 sec 18.7 GBytes 16.1 Gbits/sec receiver
The recommended MikroTik router for 25Gbit has an internal limit of 15Gbit.
Thanks for the hint. I’ll check it out at home. On my phone I couldn’t tell whether €40 was just for the card or included the SFP+ module. I also want to make sure it plays well with FreeNAS.
They should work very well: FreeBSD has a vendor-supported driver.
Netflix uses these cards in their OpenConnect Appliances.
That isn’t the limit that I’m hitting - at times I do see more than 15, including the start of the run I included. Also my border router is more of a fanless server than a dedicated router, exactly because router options in that speed range are hard to find and start to be priced like servers.
Well, also because I wanted a project.
I wouldn’t go above 10Gbps for sure. As to sweetspot, I guess you could argue there are 2:
It is interesting to see that. While I built DIY routers in the 10Mbps and 100 Mbps era, at the time I seem to remember that DIY was not typically the way to go due to power and throughput.
I guess that was back when you had throughput limitations on PCI and maybe OS wasn’t as tuned. Perhaps now with the abundance of PCIe lanes and cheaper multi-port HBAs, DIY is a more attractive option. I’ll have to look into it!
I’d be tempted to run FreeNAS virtualized on the router hardware to save a box.
Do you have experience of the X540 cards? They were expensive and maybe buggy initially, but now seem cheap and widely cloned.
Check out Stapelberg’s write up. Things have surely gotten easier since then. I’d expect the result to be larger, noisier and (maybe) less energy efficient than the dedicated routers, but also cheaper. At least, if your goal is 25Gbps wire speed. For 10Gbps there are lot more options, some of which are very well priced. I’d hesitate to guess at what price point they can actually firewall at wire speed though.
For myself, I managed to hunt down a SYS-E302-12D-8C, which is serious overkill but gives lots of room to play. A SYS-E302-12D-4C would also be plenty. As I see it, the advantage over the DIY route is the actual lack of fans (no dust collection) a smallish form factor, and that the integration testing is largely done. Disadvantage is cost, sourcing in CH, etc.
Software wise, I looked around a lot and then decided to use linux and nft directly to learn what the cool kids are doing. Ended up running Arch linux with vanilla kernel and a .config that I’ve been slowly tweaking over time. But all this is absolutely an example of me choosing what I want a project to be rather than a recommendation.
Thanks. That’s an interesting blog. He went for the no-expenses-spared option. He also merged the router and fileserver, which makes a lot of sense due to the synergies and spare hardware capacity on the router.
Before I moved to Linux, I used to use BSD and I never understood why Linux just didn’t take the BSD firewall and use that instead of coming up with the horror show that was ipchains, then iptables. I’m not familiar with nftables, but anything has to be better than ipchains/iptables.
I find the ruleset format to be fairly readable, and like that it is dumpable and supports atomic replacement. I still spent some time looking at the diagram to understand what I to block where to get the desired behavior, but that may be the nature of firewalling.
At one point I had some crazy thoughts of building up and maybe open sourcing a DPDK based firewall. Commercial versions of the approach claim crazy low CPU for high volume processing. I decided that I didn’t have time for that much of a project though.
I am not too familiar with Linux firewalling - but apart from hardware offloading, maybe you also need something like https://www.dpdk.org/ for routing at 25G - with linespeed.
While FreeBSD can deliver and sustain 100G or more delivering movies, it cannot really do 10G firewalling at linespeed.
Commercial firewalls these days do all the lowlevel TCP/IP and packet filtering in hardware, so CPU isn’t an issue.
Netgate created tnsr for this. But it’s also commercial.
I get that too. I feel this project might get infectious…