So.... instead of waiting for all that and never fixing a known issue:
They could just take a little Linux box with nginx (which is F5 now ... funnily), assign the IPv6 address to that and proxy with that. Voila. Solved.
No need to have a load balancer for that as I doubt that sbb.ch gets more than a few 100mbit in IPv6 traffic. And as it is broken today, it is not like they are losing redundancy.
The F5 box has a bug, something with the checksum goes wrong and the F5 discards the ICMP packet.
As noted in previous comments that is standard ICMPv6 PtB handling.
See https://blog.cloudflare.com/path-mtu-discovery-in-practice/ and many other similar explanations.
Greets, Jeroen
--
On 2019-03-12 12:03, Silvia Hagen wrote:
Hi guys
Here's some info from SBB (I was working with them and just spoke with them today).
. They are aware of the problem. . The problem only happens when someone uses smaller packet sizes (often when using some tunnelling techniques). . Currently the webserver is in an IPv4 zone, the Internet router is a Cisco box which does 64 Translation. The packets go through an F5 LB to reach the webserver. . When the packets go out and the Cisco box asks for fragmention, it sends the ICMP packet to the webserver. The F5 box has a bug, something with the checksum goes wrong and the F5 discards the ICMP packet. . They have had a neverending incident with F5 and F5 does not seem to be able to fix that. SBB has given up on this incident.
The plan: . SBB is currently enabling IPv6 on the routing layer, plan to be accomplished by summer 2019. . Next step on the plan is to enable v6 out to the datacenter, with priority on the webserver zone. So with that the problems should go away.
SBB was attending the last swinog event in Switzerland. They will also come again and they offered to have a talk if desired. I can connect to the right person if you are interested.
Thanks, Silvia
-----Ursprüngliche Nachricht----- Von: swinog-bounces@lists.swinog.ch [mailto:swinog-bounces@lists.swinog.ch] Im Auftrag von Nico Schottelius Gesendet: Dienstag, 12. März 2019 10:33 An: swinog@lists.swinog.ch Betreff: [swinog] SBB.ch / IPv6 MTU / fragmentation problem
Good morning,
is anyone from sbb.ch reading here?
https://sbb.ch does not load on IPv6 for us. It seems that packets > 1420 bytes are dropped inside the SBB network,
Local PMTU / fragmentation seems to work, my local outgoing MTU is 1420. MTR below.
Best,
Nico
[10:23] line:~% mtr -w -c1 -s 1500 sbb.ch Start: 2019-03-12T10:24:17+0100 HOST: line Loss% Snt Last Avg Best Wrst StDev 1.|-- 2a0a:e5c1:111:111::42 0.0% 1 11.2 11.2 11.2 11.2 0.0 2.|-- ??? 100.0 1 0.0 0.0 0.0 0.0 0.0 3.|-- 2a0a:e5c0:2:12::7 0.0% 1 69.8 69.8 69.8 69.8 0.0 4.|-- 2a0a:e5c0:1:1::9 0.0% 1 74.3 74.3 74.3 74.3 0.0 5.|-- 2001:1620:20e6::1 0.0% 1 69.4 69.4 69.4 69.4 0.0 6.|-- r1zrh2.core.init7.net 0.0% 1 69.1 69.1 69.1 69.1 0.0 7.|-- r1olt2.core.init7.net 0.0% 1 58.0 58.0 58.0 58.0 0.0 8.|-- r1brn1.core.init7.net 0.0% 1 62.8 62.8 62.8 62.8 0.0 9.|-- r2brn1.core.init7.net 0.0% 1 65.4 65.4 65.4 65.4 0.0 10.|-- r1epe1.core.init7.net 0.0% 1 75.2 75.2 75.2 75.2 0.0 11.|-- r1qls1.core.init7.net 0.0% 1 78.4 78.4 78.4 78.4 0.0 12.|-- r1gva3.core.init7.net 0.0% 1 81.0 81.0 81.0 81.0 0.0 13.|-- gw-sunrise.init7.net 0.0% 1 64.4 64.4 64.4 64.4 0.0 14.|-- 2001:1700:1:7:120::2 0.0% 1 84.4 84.4 84.4 84.4 0.0 15.|-- 2001:1700:4d00:2::2 0.0% 1 81.3 81.3 81.3 81.3 0.0 16.|-- 2a00:4bc0:ffff:ff00::1d 0.0% 1 67.0 67.0 67.0 67.0 0.0 17.|-- ??? 100.0 1 0.0 0.0 0.0 0.0 0.0 [10:24] line:~% mtr -w -c1 -s 1400 sbb.ch Start: 2019-03-12T10:24:35+0100 HOST: line Loss% Snt Last Avg Best Wrst StDev 1.|-- 2a0a:e5c1:111:111::42 0.0% 1 3.2 3.2 3.2 3.2 0.0 2.|-- 2a0a:e5c1:100::1 0.0% 1 69.0 69.0 69.0 69.0 0.0 3.|-- 2a0a:e5c0:2:12::7 0.0% 1 74.7 74.7 74.7 74.7 0.0 4.|-- 2a0a:e5c0:1:1::9 0.0% 1 69.9 69.9 69.9 69.9 0.0 5.|-- 2001:1620:20e6::1 0.0% 1 60.5 60.5 60.5 60.5 0.0 6.|-- r1zrh2.core.init7.net 0.0% 1 75.3 75.3 75.3 75.3 0.0 7.|-- r1olt2.core.init7.net 0.0% 1 70.7 70.7 70.7 70.7 0.0 8.|-- r1brn1.core.init7.net 0.0% 1 69.1 69.1 69.1 69.1 0.0 9.|-- r2brn1.core.init7.net 0.0% 1 54.6 54.6 54.6 54.6 0.0 10.|-- r1epe1.core.init7.net 0.0% 1 75.9 75.9 75.9 75.9 0.0 11.|-- r1qls1.core.init7.net 0.0% 1 78.8 78.8 78.8 78.8 0.0 12.|-- r1gva3.core.init7.net 0.0% 1 79.8 79.8 79.8 79.8 0.0 13.|-- gw-sunrise.init7.net 0.0% 1 69.9 69.9 69.9 69.9 0.0 14.|-- 2001:1700:1:7:120::2 0.0% 1 77.5 77.5 77.5 77.5 0.0 15.|-- 2001:1700:4d00:2::2 0.0% 1 59.3 59.3 59.3 59.3 0.0 16.|-- 2a00:4bc0:ffff:ff00::1d 0.0% 1 70.1 70.1 70.1 70.1 0.0 17.|-- ??? 100.0 1 0.0 0.0 0.0 0.0 0.0 18.|-- ??? 100.0 1 0.0 0.0 0.0 0.0 0.0 19.|-- ??? 100.0 1 0.0 0.0 0.0 0.0 0.0 20.|-- ??? 100.0 1 0.0 0.0 0.0 0.0 0.0 21.|-- ??? 100.0 1 0.0 0.0 0.0 0.0 0.0 22.|-- ??? 100.0 1 0.0 0.0 0.0 0.0 0.0 23.|-- ??? 100.0 1 0.0 0.0 0.0 0.0 0.0 24.|-- ??? 100.0 1 0.0 0.0 0.0 0.0 0.0 25.|-- 2a00:4bc0:ffff:ffff::c296:f58e 0.0% 1 58.3 58.3 58.3 58.3 0.0 [10:24] line:~%
[10:25] line:~% mtr -w -c1 -s 1420 sbb.ch Start: 2019-03-12T10:25:44+0100 HOST: line Loss% Snt Last Avg Best Wrst StDev 1.|-- 2a0a:e5c1:111:111::42 0.0% 1 16.3 16.3 16.3 16.3 0.0 2.|-- 2a0a:e5c1:100::1 0.0% 1 77.0 77.0 77.0 77.0 0.0 3.|-- 2a0a:e5c0:2:12::7 0.0% 1 67.0 67.0 67.0 67.0 0.0 4.|-- 2a0a:e5c0:1:1::9 0.0% 1 66.7 66.7 66.7 66.7 0.0 5.|-- 2001:1620:20e6::1 0.0% 1 78.8 78.8 78.8 78.8 0.0 6.|-- r1zrh2.core.init7.net 0.0% 1 64.5 64.5 64.5 64.5 0.0 7.|-- r1olt2.core.init7.net 0.0% 1 68.3 68.3 68.3 68.3 0.0 8.|-- r1brn1.core.init7.net 0.0% 1 74.9 74.9 74.9 74.9 0.0 9.|-- r2brn1.core.init7.net 0.0% 1 73.6 73.6 73.6 73.6 0.0 10.|-- r1epe1.core.init7.net 0.0% 1 62.2 62.2 62.2 62.2 0.0 11.|-- r1qls1.core.init7.net 0.0% 1 74.3 74.3 74.3 74.3 0.0 12.|-- r1gva3.core.init7.net 0.0% 1 63.6 63.6 63.6 63.6 0.0 13.|-- gw-sunrise.init7.net 0.0% 1 69.1 69.1 69.1 69.1 0.0 14.|-- 2001:1700:1:7:120::2 0.0% 1 77.4 77.4 77.4 77.4 0.0 15.|-- 2001:1700:4d00:2::2 0.0% 1 78.8 78.8 78.8 78.8 0.0 16.|-- 2a00:4bc0:ffff:ff00::1d 0.0% 1 75.7 75.7 75.7 75.7 0.0 17.|-- ??? 100.0 1 0.0 0.0 0.0 0.0 0.0 18.|-- ??? 100.0 1 0.0 0.0 0.0 0.0 0.0 19.|-- ??? 100.0 1 0.0 0.0 0.0 0.0 0.0 20.|-- ??? 100.0 1 0.0 0.0 0.0 0.0 0.0 21.|-- ??? 100.0 1 0.0 0.0 0.0 0.0 0.0 22.|-- ??? 100.0 1 0.0 0.0 0.0 0.0 0.0 23.|-- ??? 100.0 1 0.0 0.0 0.0 0.0 0.0 24.|-- ??? 100.0 1 0.0 0.0 0.0 0.0 0.0 25.|-- 2a00:4bc0:ffff:ffff::c296:f58e 0.0% 1 83.8 83.8 83.8 83.8 0.0 [10:25] line:~% mtr -w -c1 -s 1430 sbb.ch Start: 2019-03-12T10:25:55+0100 HOST: line Loss% Snt Last Avg Best Wrst StDev 1.|-- 2a0a:e5c1:111:111::42 0.0% 1 7.3 7.3 7.3 7.3 0.0 2.|-- ??? 100.0 1 0.0 0.0 0.0 0.0 0.0 3.|-- 2a0a:e5c0:2:12::7 0.0% 1 60.4 60.4 60.4 60.4 0.0 4.|-- 2a0a:e5c0:1:1::9 0.0% 1 61.9 61.9 61.9 61.9 0.0 5.|-- 2001:1620:20e6::1 0.0% 1 72.2 72.2 72.2 72.2 0.0 6.|-- r1zrh2.core.init7.net 0.0% 1 65.2 65.2 65.2 65.2 0.0 7.|-- r1olt2.core.init7.net 0.0% 1 64.9 64.9 64.9 64.9 0.0 8.|-- r1brn1.core.init7.net 0.0% 1 64.9 64.9 64.9 64.9 0.0 9.|-- r2brn1.core.init7.net 0.0% 1 71.7 71.7 71.7 71.7 0.0 10.|-- r1epe1.core.init7.net 0.0% 1 64.4 64.4 64.4 64.4 0.0 11.|-- r1qls1.core.init7.net 0.0% 1 63.2 63.2 63.2 63.2 0.0 12.|-- r1gva3.core.init7.net 0.0% 1 77.9 77.9 77.9 77.9 0.0 13.|-- gw-sunrise.init7.net 0.0% 1 64.5 64.5 64.5 64.5 0.0 14.|-- 2001:1700:1:7:120::2 0.0% 1 63.5 63.5 63.5 63.5 0.0 15.|-- 2001:1700:4d00:2::2 0.0% 1 81.7 81.7 81.7 81.7 0.0 16.|-- 2a00:4bc0:ffff:ff00::1d 0.0% 1 74.4 74.4 74.4 74.4 0.0 17.|-- ??? 100.0 1 0.0 0.0 0.0 0.0 0.0 [10:26] line:~%
icmp6, frag works locally:
10:29:44.919328 IP6 2a0a:e5c1:111:111:3185:e802:6548:658c > 2a00:4bc0:ffff:ffff::c296:f58e: frag (0|1368) ICMP6, echo request, seq 33000, length 1368 10:29:44.919368 IP6 2a0a:e5c1:111:111:3185:e802:6548:658c > 2a00:4bc0:ffff:ffff::c296:f58e: frag (1368|92)
-- Your Swiss, Open Source and IPv6 Virtual Machine. Now on www.datacenterlight.ch.
swinog mailing list swinog@lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog
swinog mailing list swinog@lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog