Discussion:
rdomain/rtable
(too old to reply)
Paul B. Henson
2017-12-20 01:54:48 UTC
Permalink
Raw Message
I've got a box with an LTE cellular modem in it whose purpose is to provide
a backup connection to the Internet if the hardwire service goes down. It's
running OSPF to connect to the rest of the network, and the only time any
traffic should go over the cellular link (which is slower and bandwidth
capped) is if the hardwire interconnection is down, including ideally
traffic generated from the system itself.

I have that part working, by adding in a local static default route to the
cellular gateway with less priority than the OSPF default route. However,
for testing purposes, I'd like to be able to poke out the cellular link on
an as-needed basis without having to switch the entire box over to using it.
Virtual routing tables looked perfect for this purpose, as I could just
spawn a single process with a different default route, we do something
similar with network name spaces under Linux.

However, I can't quite get it to work. What I'd really like is to be able to
make a copy of the current system routing table, then change one thing about
it. However, a new rdomain shows up with no routes or interfaces in the
routing table. I can add the new default route pointing out the cellular
link, and get traffic to go out there. But I haven't sorted out how to make
all the traffic for my internal network still go through the internal link
rather than get sent out the default route. While ideally all the OSPF
routes would propagate to the other routing domain I tried just adding a
static to the /16 for our internal address space:

Internet:
Destination Gateway Flags Refs Use Mtu Prio
Iface
default 24.x.x.x UGS 0 6 - 8 umb0
10.0/16 10.128.0.21 UGS 0 0 - 8 em0

That doesn't work; the documentation says you need to get pf to pass packets
across routing domains. However, it says:

rtable number
Used to select an alternate routing table for the routing lookup.
Only effective before the route lookup happened, i.e. when
filtering inbound.

Unfortunately, for traffic originating from the system itself, there isn't
really an "inbound" interface? So I'm not sure what pf rule would make this
work. Is it just not possible, or am I missing something?

Thanks much.
Sebastian Benoit
2017-12-23 16:07:37 UTC
Permalink
Raw Message
Post by Paul B. Henson
I've got a box with an LTE cellular modem in it whose purpose is to provide
a backup connection to the Internet if the hardwire service goes down. It's
running OSPF to connect to the rest of the network, and the only time any
traffic should go over the cellular link (which is slower and bandwidth
capped) is if the hardwire interconnection is down, including ideally
traffic generated from the system itself.
I have that part working, by adding in a local static default route to the
cellular gateway with less priority than the OSPF default route. However,
for testing purposes, I'd like to be able to poke out the cellular link on
an as-needed basis without having to switch the entire box over to using it.
Virtual routing tables looked perfect for this purpose, as I could just
spawn a single process with a different default route, we do something
similar with network name spaces under Linux.
However, I can't quite get it to work. What I'd really like is to be able to
make a copy of the current system routing table, then change one thing about
it. However, a new rdomain shows up with no routes or interfaces in the
routing table. I can add the new default route pointing out the cellular
link, and get traffic to go out there.
When you create a new routing domain, for example by adding an interface to
a routing domain (e.g. ifconfig umb0 rdomain 10), you create a new routing
table 10. It will be empty until you add an address on umb0 or, for example
add your default route.

This routing table will be used to forward packets that are "in that routing
domain" (the packet is marked with the rdomain or rather the rtable it will
use). How does the packet get marked?

Three ways:

* with pf, as you have discovered. As the manpage documents, the
mark needs to be set before route lookup is done.

* when a paket comes in on an interface in rdomain 10, it will stay in
rdomain 10 (unless pf changes it).

* a packet is generated on the local machine by a process that "is in that
routing domain". I.e. processes are also marked with a rdomain.

To start a process in a specific rdomain (10), use "route -T 10 exec
command", for example

route -T 10 exec ping -n ip

or even

route -T 10 exec ksh

Processes spawned by that shell will inherit the rdomain.

Note that i used -n in the ping example. DNS resolving using the resolvers
in resolv.conf might not work, as long as those resolvers are not reachable
in rdomain 10.

Hope this helps ...
Post by Paul B. Henson
But I haven't sorted out how to make
all the traffic for my internal network still go through the internal link
rather than get sent out the default route. While ideally all the OSPF
routes would propagate to the other routing domain I tried just adding a
Destination Gateway Flags Refs Use Mtu Prio
Iface
default 24.x.x.x UGS 0 6 - 8 umb0
10.0/16 10.128.0.21 UGS 0 0 - 8 em0
That doesn't work; the documentation says you need to get pf to pass packets
rtable number
Used to select an alternate routing table for the routing lookup.
Only effective before the route lookup happened, i.e. when
filtering inbound.
Unfortunately, for traffic originating from the system itself, there isn't
really an "inbound" interface? So I'm not sure what pf rule would make this
work. Is it just not possible, or am I missing something?
Thanks much.
--
Paul B. Henson
2017-12-24 20:55:45 UTC
Permalink
Raw Message
Thanks for the info. I don't want to move any interfaces to a
non-default routing domain, I just want to be able to run a process with
a different default route. I can make that work, via the route -T 10
exec you mention after setting a default route in that domain.

But I can't seem to get traffic for my local subnet sent out my
internal interface, even after I add a route to it in the non-default
routing domain. Dunno, maybe I'm missing something.

I set it up like:

Internet:
Destination Gateway Flags Refs Use Mtu Prio Iface
default 24.x.x.x UGS 0 2 - 8 umb0
10.0/16 10.128.0.20 UGS 0 0 - 8 em0

But 'ping 10.128.0.20' shows the packets going out umb0, not em0?

Thanks again.
Post by Sebastian Benoit
When you create a new routing domain, for example by adding an interface to
a routing domain (e.g. ifconfig umb0 rdomain 10), you create a new routing
table 10. It will be empty until you add an address on umb0 or, for example
add your default route.
This routing table will be used to forward packets that are "in that routing
domain" (the packet is marked with the rdomain or rather the rtable it will
use). How does the packet get marked?
* with pf, as you have discovered. As the manpage documents, the
mark needs to be set before route lookup is done.
* when a paket comes in on an interface in rdomain 10, it will stay in
rdomain 10 (unless pf changes it).
* a packet is generated on the local machine by a process that "is in that
routing domain". I.e. processes are also marked with a rdomain.
To start a process in a specific rdomain (10), use "route -T 10 exec
command", for example
route -T 10 exec ping -n ip
or even
route -T 10 exec ksh
Processes spawned by that shell will inherit the rdomain.
Note that i used -n in the ping example. DNS resolving using the resolvers
in resolv.conf might not work, as long as those resolvers are not reachable
in rdomain 10.
Hope this helps ...
Scott Nicholas
2017-12-24 21:39:07 UTC
Permalink
Raw Message
Hello

You may need a direct route to the gateway as well

Happy holidays,
Scott


On Dec 24, 2017 4:08 PM, "Paul B. Henson" <***@acm.org> wrote:

Thanks for the info. I don't want to move any interfaces to a
non-default routing domain, I just want to be able to run a process with
a different default route. I can make that work, via the route -T 10
exec you mention after setting a default route in that domain.

But I can't seem to get traffic for my local subnet sent out my
internal interface, even after I add a route to it in the non-default
routing domain. Dunno, maybe I'm missing something.

I set it up like:

Internet:
Destination Gateway Flags Refs Use Mtu Prio
Iface
default 24.x.x.x UGS 0 2 - 8 umb0
10.0/16 10.128.0.20 UGS 0 0 - 8 em0

But 'ping 10.128.0.20' shows the packets going out umb0, not em0?

Thanks again.
Post by Sebastian Benoit
When you create a new routing domain, for example by adding an interface to
a routing domain (e.g. ifconfig umb0 rdomain 10), you create a new routing
table 10. It will be empty until you add an address on umb0 or, for example
add your default route.
This routing table will be used to forward packets that are "in that routing
domain" (the packet is marked with the rdomain or rather the rtable it will
use). How does the packet get marked?
* with pf, as you have discovered. As the manpage documents, the
mark needs to be set before route lookup is done.
* when a paket comes in on an interface in rdomain 10, it will stay in
rdomain 10 (unless pf changes it).
* a packet is generated on the local machine by a process that "is in that
routing domain". I.e. processes are also marked with a rdomain.
To start a process in a specific rdomain (10), use "route -T 10 exec
command", for example
route -T 10 exec ping -n ip
or even
route -T 10 exec ksh
Processes spawned by that shell will inherit the rdomain.
Note that i used -n in the ping example. DNS resolving using the resolvers
in resolv.conf might not work, as long as those resolvers are not reachable
in rdomain 10.
Hope this helps ...
Loading...