PPPoE (5.9 still): https gets stuck

[Harald Dunkel]

Hi folks,

I am using an openbsd (5.9) box as gateway/firewall to the
internet. ISP is Deutsche Telekom. In between is a Vigor 130
VDSL2 modem, configured to PPPoE passthrough. The PPPoE
connection is initiated on the openbsd box.

Problem: https via the tunnel gets stuck for some sites, e.g.
https://telekom.de/ (please note the irony). Other sites work
fine, e.g. https://kundencenter.telekom.de/. I tried a lot of
clients: chrome, firefox, Safari, wget, etc. and all platforms
I have at home.

Other services (http, smtp, dns, ntp, vnc, ...) seem to work
flawless.

The problem came up with the migration from ADSL to VDSL this
weekend. The gateway wasn't changed, but I wonder if there are
some issues or pitfalls with PPPoE and fragmented packages or
whatever, possibly breaking https negotiation?


Every helpful comment is highly appreciated
Harri

[Daniel Gillen]

Post by Harald Dunkel
Hi folks,
I am using an openbsd (5.9) box as gateway/firewall to the
internet. ISP is Deutsche Telekom. In between is a Vigor 130
VDSL2 modem, configured to PPPoE passthrough. The PPPoE
connection is initiated on the openbsd box.
Problem: https via the tunnel gets stuck for some sites, e.g.
https://telekom.de/ (please note the irony). Other sites work
fine, e.g. https://kundencenter.telekom.de/. I tried a lot of
clients: chrome, firefox, Safari, wget, etc. and all platforms
I have at home.
Other services (http, smtp, dns, ntp, vnc, ...) seem to work
flawless.
The problem came up with the migration from ADSL to VDSL this
weekend. The gateway wasn't changed, but I wonder if there are
some issues or pitfalls with PPPoE and fragmented packages or
whatever, possibly breaking https negotiation?
Every helpful comment is highly appreciated
Harri

Hi

I had a similar problem. In my case it had to do with Path MTU issues.

This site f.ex.: http://test-ipv6.com/ will check for that.

The solution for me was to switch to "jumbo" frames below the pppoe
device (1508 bytes if I remember correctly) and set the pppoe's MTU to 1500.

Daniel

[Harald Dunkel]

Hi Daniel,

Post by Daniel Gillen
I had a similar problem. In my case it had to do with Path MTU issues.
This site f.ex.: http://test-ipv6.com/ will check for that.
The solution for me was to switch to "jumbo" frames below the pppoe
device (1508 bytes if I remember correctly) and set the pppoe's MTU to 1500.
Daniel

I will surely try asap, but I wonder if this is allowed?
RFC 2516 says "... the PPP MTU MUST NOT be greater than
1492". https://tools.ietf.org/html/rfc2516

???


Regards
Harri

[Stuart Henderson]

Post by Harald Dunkel
Hi Daniel,

Post by Daniel Gillen
I had a similar problem. In my case it had to do with Path MTU issues.
This site f.ex.: http://test-ipv6.com/ will check for that.
The solution for me was to switch to "jumbo" frames below the pppoe
device (1508 bytes if I remember correctly) and set the pppoe's MTU to 1500.
Daniel

I will surely try asap, but I wonder if this is allowed?
RFC 2516 says "... the PPP MTU MUST NOT be greater than
1492". https://tools.ietf.org/html/rfc2516
???

RFC 4638 supercedes this. Some providers + some xDSL bridges support it,
some don't, you have to try it to find out. And you have to actively test
rather than just see if it connects; not all ISPs have paths between LNS
and LAC that can cope with baby jumbos - also sometimes it can fail if
there are changes to the ISP's internal network or their L2/L3 providers.

[Peter J. Philipp]

Hello Harri,

This interests me because I'm switching to Deutsche Telekom in february
2017. I did research back in

march or april of 2016 on how to connect to Telekom with an allnet vdsl
modem and I came across hints that Telekom uses vlan tagging. I made
notes but I don't know how updated they are, but you

can try this:

T-Online uses vlan tag 7, IP-TV uses vlan tag 8. So it depends on your
plan I guess? I'd appreciate if someone told me if this information is
outdated but I'm probably going to have to ask in february again
anyhow. Apparently my allnet vdsl modem can vlan tag and give a
passthrough to the router, I don't know if the Draytek Vigor can do this.

Regards,

-peter

Post by Harald Dunkel
Hi folks,
I am using an openbsd (5.9) box as gateway/firewall to the
internet. ISP is Deutsche Telekom. In between is a Vigor 130
VDSL2 modem, configured to PPPoE passthrough. The PPPoE
connection is initiated on the openbsd box.
Problem: https via the tunnel gets stuck for some sites, e.g.
https://telekom.de/ (please note the irony). Other sites work
fine, e.g. https://kundencenter.telekom.de/. I tried a lot of
clients: chrome, firefox, Safari, wget, etc. and all platforms
I have at home.
Other services (http, smtp, dns, ntp, vnc, ...) seem to work
flawless.
The problem came up with the migration from ADSL to VDSL this
weekend. The gateway wasn't changed, but I wonder if there are
some issues or pitfalls with PPPoE and fragmented packages or
whatever, possibly breaking https negotiation?
Every helpful comment is highly appreciated
Harri

[Harald Dunkel]

Hi Peter,

Post by Peter J. Philipp
T-Online uses vlan tag 7, IP-TV uses vlan tag 8. So it depends on your
plan I guess? I'd appreciate if someone told me if this information is
outdated but I'm probably going to have to ask in february again
anyhow. Apparently my allnet vdsl modem can vlan tag and give a
passthrough to the router, I don't know if the Draytek Vigor can do this.

Nice hint (esp. about vlan8), but please note that the problem
is on or above the PPPoE layer. I have a working tunnel (except
for https).

Vlan tagging (7) appears to be handled by the modem in its default
configuration.


Thanx very much for your hint

Harri

[Christian Weisgerber]

Post by Peter J. Philipp
This interests me because I'm switching to Deutsche Telekom in february
2017. I did research back in
march or april of 2016 on how to connect to Telekom with an allnet vdsl
modem and I came across hints that Telekom uses vlan tagging. I made
notes but I don't know how updated they are, but you

That setup is straightforward.

The Allnet VDSL modem doesn't require any configuration; you just
plug it in. (The device is actually a fully-fledged CPE with the
usual home gateway functionality, but that's all disabled by default.)

On the OpenBSD gateway, I have this:

==> hostname.em2 <==
description WAN
up

==> hostname.vlan0 <==
description "T-Online Internet"
vlan 7 vlandev em2

==> hostname.pppoe0 <==
description T-Online
inet 0.0.0.0 255.255.255.255 NONE \
pppoedev vlan0 authproto pap \
authname ***@t-online.de authkey kkkkkkkk up
dest 0.0.0.1
!route add default -ifp \$if 0.0.0.1


The authentication information is derived like above from the four
magic numbers provided by Telekom:

"Zugangsnummer" zzzzzzzzzzzz
"Kennwort" kkkkkkkk
"Anschlusskennung" aaaaaaaaaaaa
"Mitbenutzernummer" mmmm

[Markus Hennecke]

Post by Harald Dunkel
Hi folks,
I am using an openbsd (5.9) box as gateway/firewall to the
internet. ISP is Deutsche Telekom. In between is a Vigor 130
VDSL2 modem, configured to PPPoE passthrough. The PPPoE
connection is initiated on the openbsd box.
Problem: https via the tunnel gets stuck for some sites, e.g.
https://telekom.de/ (please note the irony). Other sites work
fine, e.g. https://kundencenter.telekom.de/. I tried a lot of
clients: chrome, firefox, Safari, wget, etc. and all platforms
I have at home.
Other services (http, smtp, dns, ntp, vnc, ...) seem to work
flawless.
The problem came up with the migration from ADSL to VDSL this
weekend. The gateway wasn't changed, but I wonder if there are
some issues or pitfalls with PPPoE and fragmented packages or
whatever, possibly breaking https negotiation?
Every helpful comment is highly appreciated
Harri

I use the same VDSL modem with Deutsche Telekom and can reach
https://telekom.de/
The only MTU related setting in pf.conf seems to be this:

ext_if = pppoe0
match in on $ext_if all scrub (no-df max-mss 1440)

It is an old soekris, which does not support gbit ethernet.
I do the VLAN tagging on the OpenBSD router, I think I disabled the
automatic tagging the modem supports.

Regards,
Markus

[Markus Hennecke]

Post by Markus Hennecke

Post by Harald Dunkel
Hi folks,
I am using an openbsd (5.9) box as gateway/firewall to the
internet. ISP is Deutsche Telekom. In between is a Vigor 130
VDSL2 modem, configured to PPPoE passthrough. The PPPoE
connection is initiated on the openbsd box.
Problem: https via the tunnel gets stuck for some sites, e.g.
https://telekom.de/ (please note the irony). Other sites work
fine, e.g. https://kundencenter.telekom.de/. I tried a lot of
clients: chrome, firefox, Safari, wget, etc. and all platforms
I have at home.
Other services (http, smtp, dns, ntp, vnc, ...) seem to work
flawless.
The problem came up with the migration from ADSL to VDSL this
weekend. The gateway wasn't changed, but I wonder if there are
some issues or pitfalls with PPPoE and fragmented packages or
whatever, possibly breaking https negotiation?
Every helpful comment is highly appreciated
Harri

I use the same VDSL modem with Deutsche Telekom and can reach
https://telekom.de/
ext_if = pppoe0
match in on $ext_if all scrub (no-df max-mss 1440)
It is an old soekris, which does not support gbit ethernet.
I do the VLAN tagging on the OpenBSD router, I think I disabled the
automatic tagging the modem supports.
Regards,
Markus

Damn. Of course without this line it won't work:

match out on $ext_if all scrub (max-mss 1440)

Regards,
Markus

[Harald Dunkel]

Hi Markus,

Post by Markus Hennecke

Post by Markus Hennecke
I use the same VDSL modem with Deutsche Telekom and can reach
https://telekom.de/
ext_if = pppoe0
match in on $ext_if all scrub (no-df max-mss 1440)
It is an old soekris, which does not support gbit ethernet.
I do the VLAN tagging on the OpenBSD router, I think I disabled the
automatic tagging the modem supports.
Regards,
Markus

match out on $ext_if all scrub (max-mss 1440)

I have

match in on pppoe0 scrub (no-df random-id max-mss 1440)

but no "match out". I will check.

Unfortunately I cannot check right now, since I use this line
to access my home network.


Thanx very much
Harri

[Kapfhammer, Stefan]

Hello Harald,

use the setup from there:

http://www.un.geeig.net/openbsd-vdsl.html

‎if you don't have IPTV, leave everything after
'ping' probe and reboot out. I didn't use
the 5 sysctl settings.

Update ALLNET Bridge to at least fw c46a,
reboot the device and reset to factory defaults.
Don't change anything here after defaults apply!

Wait for VDSL synchronization ‎(LINK/WAN)
permanent green LED on.‎ Then start soekris,
alix, apu/2, whatever is after the bridge.

My setup:
ALLNET ALL-BM100VDSL2V‎ with fw c46a
APU2B4 previously with OpenBSD 5.9 now 6.0
with fw 20160311
Grandstream HT-702 ATA ‎with fw 1.82
Telekom VDSL 50 Business Premium

Works well, even with https connections.
Good luck.


Freundliche Grüße / Regards
-stefan kapfhammer
Originalnachricht
Von: Harald Dunkel
Gesendet: Dienstag, 13. September 2016 13:08
An: ***@openbsd.org
Betreff: Re: PPPoE (5.9 still): https gets stuck


Hi Markus,

Post by Markus Hennecke

Post by Markus Hennecke
I use the same VDSL modem with Deutsche Telekom and can reach
https://telekom.de/
ext_if = pppoe0
match in on $ext_if all scrub (no-df max-mss 1440)
It is an old soekris, which does not support gbit ethernet.
I do the VLAN tagging on the OpenBSD router, I think I disabled the
automatic tagging the modem supports.
Regards,
Markus

match out on $ext_if all scrub (max-mss 1440)

I have

match in on pppoe0 scrub (no-df random-id max-mss 1440)

but no "match out". I will check.

Unfortunately I cannot check right now, since I use this line
to access my home network.


Thanx very much
Harri

[Harald Dunkel]

Hi Markus,

Post by Harald Dunkel
Hi Markus,

Post by Markus Hennecke
match out on $ext_if all scrub (max-mss 1440)

I have
match in on pppoe0 scrub (no-df random-id max-mss 1440)
but no "match out". I will check.

Congratulations, you are the winner. Replacing the

match in on pppoe0 scrub (max-mss 1440)
by
match on pppoe0 scrub (max-mss 1440)

did the trick.

I tried the suggested jumbo mtu approach, too, but this
didn't work for me, even though tracepath showed
pmtu = 1500 between my desktop PC and www.telekom.de.

I highly appreciate the helpful suggestions of all.


Regards
Harri

[Stuart Henderson]

Post by Harald Dunkel
Hi folks,
I am using an openbsd (5.9) box as gateway/firewall to the
internet. ISP is Deutsche Telekom. In between is a Vigor 130
VDSL2 modem, configured to PPPoE passthrough. The PPPoE
connection is initiated on the openbsd box.
Problem: https via the tunnel gets stuck for some sites, e.g.
https://telekom.de/ (please note the irony). Other sites work
fine, e.g. https://kundencenter.telekom.de/. I tried a lot of
clients: chrome, firefox, Safari, wget, etc. and all platforms
I have at home.
Other services (http, smtp, dns, ntp, vnc, ...) seem to work
flawless.
The problem came up with the migration from ADSL to VDSL this
weekend. The gateway wasn't changed, but I wonder if there are
some issues or pitfalls with PPPoE and fragmented packages or
whatever, possibly breaking https negotiation?
Every helpful comment is highly appreciated
Harri

See "MTU/MSS ISSUES" in pppoe(4).

[Harald Dunkel]

Hi folks,

Post by Stuart Henderson
See "MTU/MSS ISSUES" in pppoe(4).

indeed, its documented, but its also a little bit misleading.
Reading the man page I had the first impression that modifying
the mtu and max-mss are equally effective. Apparently they
are not.

AFAIU setting the max-mss affects TCP traffic only (e.g.
HTTPS). It defines the maximum payload block size on sending
and receiving(!) data via TCP. UDP and other protocols are not
affected. Very important to know if you run IPsec or openvpn
or other non-TCP protocols over the PPPoE tunnel.

The mtu works on ethernet level, giving the maximum packet
size to send(!) to the next hop without fragmentation,
regardless if the higher level protocol is TCP, UDP, ESP or
whatever.

Point is, setting the MTU does not affect the data flow from
the peer back to your site. AFAIU this made the difference
in this case. Reducing the max-mss seems to be a workaround
for some networking issue I cannot fix.

My current configuration uses both baby jumbo frames (mtu = 1508
on re0 and mtu = 1500 on pppoe0) and "scrub (max-mss 1452)" for
TCP traffic on pppoe0 (1500 bytes - 40 bytes TCP/IP header -
8 bytes PPPoE frame).


Regards
Harri

[Stuart Henderson]

Post by Harald Dunkel
Hi folks,

Post by Stuart Henderson
See "MTU/MSS ISSUES" in pppoe(4).

indeed, its documented, but its also a little bit misleading.
Reading the man page I had the first impression that modifying
the mtu and max-mss are equally effective. Apparently they
are not.
AFAIU setting the max-mss affects TCP traffic only (e.g.
HTTPS). It defines the maximum payload block size on sending
and receiving(!) data via TCP. UDP and other protocols are not
affected. Very important to know if you run IPsec or openvpn
or other non-TCP protocols over the PPPoE tunnel.
The mtu works on ethernet level, giving the maximum packet
size to send(!) to the next hop without fragmentation,
regardless if the higher level protocol is TCP, UDP, ESP or
whatever.

You are supposed to be able to get large packets through; routers
are meant to either fragment or send ICMP "frag-needed" messages
to allow the end host to do it. But some idiots misconfigure their
firewalls to drop the necessary ICMP messages causing breakage to
some sites.

In practice, people configuring non-TCP services often try to keep
packets below the common breakage points (especially 1492)..

Post by Harald Dunkel
Point is, setting the MTU does not affect the data flow from
the peer back to your site. AFAIU this made the difference
in this case. Reducing the max-mss seems to be a workaround
for some networking issue I cannot fix.

The MSS in the TCP handshake is based on the MTU on the outgoing
route from the end host. So if you have

router - pppoe0 - 1492
router - internal - 1500
host - internal - 1500

the host will base its calculation on 1500 not 1492.

"scrub (max-mss ..)" will cap this in the TCP handshake (and adjust
checksums to match).

However: if you have working 1500 MTU pppoe via baby jumbos on
the ethernet interface, "scrub (max-mss..)" shouldn't fix anything.
The restricted MTU is usually towards the "client" side of things
rather than towards the server, plus if it's towards the server
then many people will be affected so it's easier for the server
admin to spot problems.

Post by Harald Dunkel
My current configuration uses both baby jumbo frames (mtu = 1508
on re0 and mtu = 1500 on pppoe0) and "scrub (max-mss 1452)" for
TCP traffic on pppoe0 (1500 bytes - 40 bytes TCP/IP header -
8 bytes PPPoE frame).

You should not need both. If baby jumbos are working correctly
(you can "ping -D -s 1472 $some_internet_host" from an end host)
then you should be able to get rid of the scrub. If they are not
working correctly, you should get rid of the baby jumbo config
and just use 1492, this will at least fix things for larger
packets in the normal case where there is no bad firewall config
in the way.

[Harald Dunkel]

Hi Stuart,

AFAIU setting the max-mss affects TCP traffic only (e.g. HTTPS). It defines the maximum payload block size on sending and receiving(!) data via TCP. UDP and other protocols are not affected. Very important to know if you run IPsec or openvpn or other non-TCP protocols over the PPPoE tunnel.
The mtu works on ethernet level, giving the maximum packet size to send(!) to the next hop without fragmentation, regardless if the higher level protocol is TCP, UDP, ESP or whatever.

You are supposed to be able to get large packets through; routers are meant to either fragment or send ICMP "frag-needed" messages to allow the end host to do it. But some idiots misconfigure their firewalls to drop the necessary ICMP messages causing breakage to some sites.

Would you say that setting "scrub (no-df random-id)" is best practice
for IPv4 traffic forwarded into the internet?

The MSS in the TCP handshake is based on the MTU on the outgoing route from the end host. So if you have
router - pppoe0 - 1492 router - internal - 1500 host - internal - 1500
the host will base its calculation on 1500 not 1492.
"scrub (max-mss ..)" will cap this in the TCP handshake (and adjust checksums to match).

I figured this out, but maybe pf.conf(5) could be more explicit on
this detail, too?

However: if you have working 1500 MTU pppoe via baby jumbos on the ethernet interface, "scrub (max-mss..)" shouldn't fix anything. The restricted MTU is usually towards the "client" side of things rather than towards the server, plus if it's towards the server then many people will be affected
so it's easier for the server admin to spot problems.

My current configuration uses both baby jumbo frames (mtu = 1508 on re0 and mtu = 1500 on pppoe0) and "scrub (max-mss 1452)" for TCP traffic on pppoe0 (1500 bytes - 40 bytes TCP/IP header - 8 bytes PPPoE frame).

You should not need both. If baby jumbos are working correctly (you can "ping -D -s 1472 $some_internet_host" from an end host) then you should be able to get rid of the scrub. If they are not working correctly, you should get rid of the baby jumbo config and just use 1492, this will at least
fix things for larger packets in the normal case where there is no bad firewall config in the way.

Deutsche Telekom seems to support baby jumbo frames ("Dumbos"? :-) )
on my side, but they told me that they don't actively support per-
customer MTUs on their BNGs.


Thanx for your detailed response. That was really helpful.


Regards
Harri