no route to host (when there is a route )

[Tom Smyth]

Hello,

I have encountered this issue for a while, it happens irregularly
on my systems on this lan
basically when the issue occurs
I cant route out the interface with the default route on it,
I cant ping the gateway
I cant see the arp of the gateway
but i can see the routes installed in the routing table


are there other commands I should be looking at to debug it more
Im using

ifconfig
arp
route

when i run run sh /etc/netstart em0
then normal operation returns

The only (unusual network config) im using is that im deploying
more specific static routes (than the connected route) to allow
clients on a non broadcast network to route to each other
ie if a client wants to talk to another client send packet to default gateway
(icmp redirects are off on the gateway)

the output of ping and arp when it happens are as follows
# ping 5.134.92.1
PING 5.134.92.1 (5.134.92.1): 56 data bytes
ping: sendto: No route to host
ping: wrote 5.134.92.1 64 chars, ret=-1
ping: sendto: No route to host
ping: wrote 5.134.92.1 64 chars, ret=-1
ping: sendto: No route to host
ping: wrote 5.134.92.1 64 chars, ret=-1
--- 5.134.92.1 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss

#ifconfig em0
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:0d:b9:46:33:50
index 1 priority 0 llprio 3
groups: egress
media: Ethernet autoselect (100baseTX full-duplex)
status: active
inet 5.134.92.142 netmask 0xfffffc00 broadcast 5.134.95.255





below is an output of the routing table when it is not working

# route -n -T0 show
Routing tables

Internet:
Destination Gateway Flags Refs Use Mtu Prio Iface
default 5.134.92.1 UGS 2 78893 - 8 em0
224/4 127.0.0.1 URS 0 609 32768 8 lo0
5.134.92/22 5.134.92.142 UC 0 0 - 4 em0
5.134.92/23 5.134.92.1 UGS 0 80897 - 8 em0
5.134.94/23 5.134.92.1 UGS 0 396 - 8 em0
5.134.92.142 00:0d:b9:46:33:50 UHLl 0 52 - 1 em0
5.134.95.255 5.134.92.142 UHb 0 0 - 1 em0
127/8 127.0.0.1 UGRS 0 0 32768 8 lo0
127.0.0.1 127.0.0.1 UHl 1 6 32768 1 lo0
185.55.204/23 5.134.92.1 UGS 0 24 - 8 em0
185.55.206/23 5.134.92.1 UGS 0 21 - 8 em0

Internet6:
Destination Gateway
Flags Refs Use Mtu Prio Iface
::/96 ::1 UGRS
0 0 32768 8 lo0
::/104 ::1 UGRS
0 0 32768 8 lo0
::1 ::1 UHl
14 14 32768 1 lo0
::127.0.0.0/104 ::1 UGRS
0 0 32768 8 lo0
::224.0.0.0/100 ::1 UGRS
0 0 32768 8 lo0
::255.0.0.0/104 ::1 UGRS
0 0 32768 8 lo0
::ffff:0.0.0.0/96 ::1 UGRS
0 0 32768 8 lo0
2002::/24 ::1 UGRS
0 0 32768 8 lo0
2002:7f00::/24 ::1 UGRS
0 0 32768 8 lo0
2002:e000::/20 ::1 UGRS
0 0 32768 8 lo0
2002:ff00::/24 ::1 UGRS
0 0 32768 8 lo0
fe80::/10 ::1 UGRS
0 0 32768 8 lo0
fec0::/10 ::1 UGRS
0 0 32768 8 lo0
fe80::1%lo0 fe80::1%lo0 UHl
0 0 32768 1 lo0
ff01::/16 ::1 UGRS
21 21 32768 8 lo0
ff01::%lo0/32 ::1 Um
0 1 32768 4 lo0
ff02::/16 ::1 UGRS
21 21 32768 8 lo0
ff02::%lo0/32 ::1 Um
0 1 32768 4 lo0



---------------------------------------------
---------------------------------------------

below is the output of the routing table when it is working
# route -n -T0 show
Routing tables

Internet:
Destination Gateway Flags Refs Use Mtu Prio Iface
default 5.134.92.1 UGS 5 51573 - 8 em0
224/4 127.0.0.1 URS 0 1060 32768 8 lo0
5.134.92/22 5.134.92.142 UC 1 0 - 4 em0
5.134.92/23 5.134.92.1 UGS 0 116 - 8 em0
5.134.94/23 5.134.92.1 UGS 0 51 - 8 em0
5.134.92.1 de:10:0d:96:90:34 UHLc 5 180 - 4 em0
5.134.92.142 00:0d:b9:46:33:50 UHLl 0 80378 - 1 em0
5.134.95.255 5.134.92.142 UHb 0 0 - 1 em0
127/8 127.0.0.1 UGRS 0 0 32768 8 lo0
127.0.0.1 127.0.0.1 UHl 1 6 32768 1 lo0
185.55.204/23 5.134.92.1 UGS 0 12 - 8 em0
185.55.206/23 5.134.92.1 UGS 0 6 - 8 em0

Internet6:
Destination Gateway
Flags Refs Use Mtu Prio Iface
::/96 ::1 UGRS
0 0 32768 8 lo0
::/104 ::1 UGRS
0 0 32768 8 lo0
::1 ::1 UHl
14 14 32768 1 lo0
::127.0.0.0/104 ::1 UGRS
0 0 32768 8 lo0
::224.0.0.0/100 ::1 UGRS
0 0 32768 8 lo0
::255.0.0.0/104 ::1 UGRS
0 0 32768 8 lo0
::ffff:0.0.0.0/96 ::1 UGRS
0 0 32768 8 lo0
2002::/24 ::1 UGRS
0 0 32768 8 lo0
2002:7f00::/24 ::1 UGRS
0 0 32768 8 lo0
2002:e000::/20 ::1 UGRS
0 0 32768 8 lo0
2002:ff00::/24 ::1 UGRS
0 0 32768 8 lo0
fe80::/10 ::1 UGRS
0 0 32768 8 lo0
fec0::/10 ::1 UGRS
0 0 32768 8 lo0
fe80::1%lo0 fe80::1%lo0 UHl
0 0 32768 1 lo0
ff01::/16 ::1 UGRS
21 21 32768 8 lo0
ff01::%lo0/32 ::1 Um
0 1 32768 4 lo0
ff02::/16 ::1 UGRS
21 21 32768 8 lo0
ff02::%lo0/32 ::1 Um
0 1 32768 4 lo0


em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:0d:b9:46:33:50
index 1 priority 0 llprio 3
groups: egress
media: Ethernet autoselect (100baseTX full-duplex)
status: active
inet 5.134.92.142 netmask 0xfffffc00 broadcast 5.134.95.255


----------------------------------------------------------------------------------
----------------------------------------------------------------------------------

[jin&hitman&Barracuda]

I'm not an expert but i had same issue while using 6.1 as a guest os on a
kvm.
I updated to 6.2 and problem vanished.

Post by Tom Smyth
Hello,
I have encountered this issue for a while, it happens irregularly
on my systems on this lan
basically when the issue occurs
I cant route out the interface with the default route on it,
I cant ping the gateway
I cant see the arp of the gateway
but i can see the routes installed in the routing table
are there other commands I should be looking at to debug it more
Im using
ifconfig
arp
route
when i run run sh /etc/netstart em0
then normal operation returns
The only (unusual network config) im using is that im deploying
more specific static routes (than the connected route) to allow
clients on a non broadcast network to route to each other
ie if a client wants to talk to another client send packet to default gateway
(icmp redirects are off on the gateway)
the output of ping and arp when it happens are as follows
# ping 5.134.92.1
PING 5.134.92.1 (5.134.92.1): 56 data bytes
ping: sendto: No route to host
ping: wrote 5.134.92.1 64 chars, ret=-1
ping: sendto: No route to host
ping: wrote 5.134.92.1 64 chars, ret=-1
ping: sendto: No route to host
ping: wrote 5.134.92.1 64 chars, ret=-1
--- 5.134.92.1 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss
#ifconfig em0
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:0d:b9:46:33:50
index 1 priority 0 llprio 3
groups: egress
media: Ethernet autoselect (100baseTX full-duplex)
status: active
inet 5.134.92.142 netmask 0xfffffc00 broadcast 5.134.95.255
below is an output of the routing table when it is not working
# route -n -T0 show
Routing tables
Destination Gateway Flags Refs Use Mtu Prio Iface
default 5.134.92.1 UGS 2 78893 - 8 em0
224/4 127.0.0.1 URS 0 609 32768 8 lo0
5.134.92/22 5.134.92.142 UC 0 0 - 4 em0
5.134.92/23 5.134.92.1 UGS 0 80897 - 8 em0
5.134.94/23 5.134.92.1 UGS 0 396 - 8 em0
5.134.92.142 00:0d:b9:46:33:50 UHLl 0 52 - 1 em0
5.134.95.255 5.134.92.142 UHb 0 0 - 1 em0
127/8 127.0.0.1 UGRS 0 0 32768 8 lo0
127.0.0.1 127.0.0.1 UHl 1 6 32768 1 lo0
185.55.204/23 5.134.92.1 UGS 0 24 - 8 em0
185.55.206/23 5.134.92.1 UGS 0 21 - 8 em0
Destination Gateway
Flags Refs Use Mtu Prio Iface
::/96 ::1 UGRS
0 0 32768 8 lo0
::/104 ::1 UGRS
0 0 32768 8 lo0
::1 ::1 UHl
14 14 32768 1 lo0
::127.0.0.0/104 ::1 UGRS
0 0 32768 8 lo0
::224.0.0.0/100 ::1 UGRS
0 0 32768 8 lo0
::255.0.0.0/104 ::1 UGRS
0 0 32768 8 lo0
::ffff:0.0.0.0/96 ::1 UGRS
0 0 32768 8 lo0
2002::/24 ::1 UGRS
0 0 32768 8 lo0
2002:7f00::/24 ::1 UGRS
0 0 32768 8 lo0
2002:e000::/20 ::1 UGRS
0 0 32768 8 lo0
2002:ff00::/24 ::1 UGRS
0 0 32768 8 lo0
fe80::/10 ::1 UGRS
0 0 32768 8 lo0
fec0::/10 ::1 UGRS
0 0 32768 8 lo0
fe80::1%lo0 fe80::1%lo0 UHl
0 0 32768 1 lo0
ff01::/16 ::1 UGRS
21 21 32768 8 lo0
ff01::%lo0/32 ::1 Um
0 1 32768 4 lo0
ff02::/16 ::1 UGRS
21 21 32768 8 lo0
ff02::%lo0/32 ::1 Um
0 1 32768 4 lo0
---------------------------------------------
---------------------------------------------
below is the output of the routing table when it is working
# route -n -T0 show
Routing tables
Destination Gateway Flags Refs Use Mtu Prio Iface
default 5.134.92.1 UGS 5 51573 - 8 em0
224/4 127.0.0.1 URS 0 1060 32768 8 lo0
5.134.92/22 5.134.92.142 UC 1 0 - 4 em0
5.134.92/23 5.134.92.1 UGS 0 116 - 8 em0
5.134.94/23 5.134.92.1 UGS 0 51 - 8 em0
5.134.92.1 de:10:0d:96:90:34 UHLc 5 180 - 4 em0
5.134.92.142 00:0d:b9:46:33:50 UHLl 0 80378 - 1 em0
5.134.95.255 5.134.92.142 UHb 0 0 - 1 em0
127/8 127.0.0.1 UGRS 0 0 32768 8 lo0
127.0.0.1 127.0.0.1 UHl 1 6 32768 1 lo0
185.55.204/23 5.134.92.1 UGS 0 12 - 8 em0
185.55.206/23 5.134.92.1 UGS 0 6 - 8 em0
Destination Gateway
Flags Refs Use Mtu Prio Iface
::/96 ::1 UGRS
0 0 32768 8 lo0
::/104 ::1 UGRS
0 0 32768 8 lo0
::1 ::1 UHl
14 14 32768 1 lo0
::127.0.0.0/104 ::1 UGRS
0 0 32768 8 lo0
::224.0.0.0/100 ::1 UGRS
0 0 32768 8 lo0
::255.0.0.0/104 ::1 UGRS
0 0 32768 8 lo0
::ffff:0.0.0.0/96 ::1 UGRS
0 0 32768 8 lo0
2002::/24 ::1 UGRS
0 0 32768 8 lo0
2002:7f00::/24 ::1 UGRS
0 0 32768 8 lo0
2002:e000::/20 ::1 UGRS
0 0 32768 8 lo0
2002:ff00::/24 ::1 UGRS
0 0 32768 8 lo0
fe80::/10 ::1 UGRS
0 0 32768 8 lo0
fec0::/10 ::1 UGRS
0 0 32768 8 lo0
fe80::1%lo0 fe80::1%lo0 UHl
0 0 32768 1 lo0
ff01::/16 ::1 UGRS
21 21 32768 8 lo0
ff01::%lo0/32 ::1 Um
0 1 32768 4 lo0
ff02::/16 ::1 UGRS
21 21 32768 8 lo0
ff02::%lo0/32 ::1 Um
0 1 32768 4 lo0
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:0d:b9:46:33:50
index 1 priority 0 llprio 3
groups: egress
media: Ethernet autoselect (100baseTX full-duplex)
status: active
inet 5.134.92.142 netmask 0xfffffc00 broadcast 5.134.95.255
----------------------------------------------------------------------------------
----------------------------------------------------------------------------------

[Ingo Schwarze]

Hi Tom,

i think i am able to reproduce your problem on -current:

# sh /etc/netstart em0
# route -n show -inet
default 192.168.2.1 UGS 0 0 - 8 em0
192.168.2/24 192.168.2.100 UCn 1 0 - 4 em0
192.168.2.1 link#1 UHLch 1 2 - 3 em0
[...]
# route add 192.168.2.0/25 192.168.2.1
add net 192.168.2.0/25: gateway 192.168.2.1
# route -n show -inet | grep /25
192.168.2.0/25 192.168.2.1 UGS 0 0 - 8 em0
# route delete 192.168.2/24
delete net 192.168.2/24
# route add 192.168.2/24 -iface -cloning -nostatic 192.168.2.100
add net 192.168.2/24: gateway 192.168.2.100
# route -n show -inet|grep /2
192.168.2/24 192.168.2.100 UC 0 0 - 56 em0
192.168.2.0/25 192.168.2.1 UGS 0 0 - 8 em0

That is very similar to what you have, see the quote from your
posting at the very end.

Now ping 192.168.2.1 doesn't work because
(1) there is no specific route to 192.168.2.1/32
(2) the closest matching route is to 192.168.2.0/25
(3) but that is a *gateway* route via 192.168.2.1, which doesn't
help because how to get to 192.168.2.1 is exactly what we are
trying to figure out...

I can make it work as follows, continuing the previous example:

# route add 192.168.2.1 -iface -cloning 192.168.2.100
add host 192.168.2.1: gateway 192.168.2.100
# route -n show -inet | grep UHCS
192.168.2.1 192.168.2.100 UHCS 0 0 - 8 em0
# ping 192.168.2.1
PING 192.168.2.1 (192.168.2.1): 56 data bytes
64 bytes from 192.168.2.1: icmp_seq=0 ttl=64 time=0.651 ms
# route delete default
delete net default
# route add default 192.168.2.1
add net default: gateway 192.168.2.1
# route -n show -inet
default 192.168.2.1 UGS 1 1 - 8 em0
192.168.2/24 192.168.2.100 UC 0 0 - 56 em0
192.168.2.0/25 192.168.2.1 UGS 0 165 - 8 em0
192.168.2.1 74:31:70:fd:f0:12 UHLch 1 4 - 7 em0
192.168.2.1 192.168.2.100 UHCS 1 0 - 8 em0
# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=61 time=34.990 ms

Like that, we *also* have a specific route to the gateway,
given that the closest matching route is unusable for the purpose.
Note the "h" flag on the 192.168.2.1 UHLch route. That's because
i added the default route after the route to the gateway.
The other way round, the "h" would be missing and it wouldn't work.



So what you are doing seems fragile to me. It may sometimes work
due to order of configuration, timeouts, whatever, i'm not sure.
But once part of it gets broken for whatever reason, i don't see
how it could possibly automatically recover via the normal RTF_CLONING
mechanism.

If you must have a static route to a subnet that includes the
gateway address (5.134.92.1), thus defeating the purpose of the
RTF_CLONING flag on the 5.134.92/22 route, it would seem safest
to me to manually add the following route *before* adding the
default route:

# route add 5.134.92.1 -iface -cloning 5.134.92.142
# route add default 5.134.92.1

Take all this with a grain of salt, even though i committed to
the route(8) program lately, i'm not at all an expert on it.

Yours,
Ingo


P.S.
Your setup seems weird and is likely still fragile even with the
above improvement. Having gateway routes for hosts in directly

The only (unusual network config) im using is that im deploying
more specific static routes (than the connected route) to allow
clients on a non broadcast network to route to each other
ie if a client wants to talk to another client send packet to
default gateway (icmp redirects are off on the gateway)
#ifconfig em0
inet 5.134.92.142 netmask 0xfffffc00 broadcast 5.134.95.255
# route -n -T0 show
5.134.92/22 5.134.92.142 UC 0 0 - 4 em0
5.134.92/23 5.134.92.1 UGS 0 80897 - 8 em0

[Martin Pieuchot]

Post by Ingo Schwarze
[...]
So what you are doing seems fragile to me. It may sometimes work
due to order of configuration, timeouts, whatever, i'm not sure.

It can work if the ARP entry, what Ingo called the /32 is created
before you add your /23.

Post by Ingo Schwarze
But once part of it gets broken for whatever reason, i don't see
how it could possibly automatically recover via the normal RTF_CLONING
mechanism.

It can't because as you described the /23 will be a better match. And the
reason will be the expiration of the ARP cache.

[Tom Smyth]

Hello Ingo, Martin,All,

I think you hit the nail on the head, (I was too busy looking at the
routing table (and forgot the fundamental principle of longest prefix
match)

so if I have a static arp entry before adding in the
(more specific than the connected route) i should be OK

just to explain (the method in my madness )
(what i agree in hindsight is a fragile setup)
basically I have a /22 network the clients are isolated from each
other on Layer 2 (so the clients cant see each others
arp requests / replys) (Bridge horizon / protected ports / privatevlan)
to limit bandwith wasting stuff such as broadcasts and other
security issues such as rogue DHCP servers etc.

The clients can only see the gateway
and the gateway can see all the clients .. I have heard of people
using some proxy arp solutions on the gateway and perhaps
that is what I should be doing rather than the ( more specific
than connected static routes)

Does anyone who operate Access layer in ISPS have suggestions
I appreciate the help and the reminder of the longest prefix match
rule :)

Thanks again

Post by Martin Pieuchot

Post by Ingo Schwarze
[...]
So what you are doing seems fragile to me. It may sometimes work
due to order of configuration, timeouts, whatever, i'm not sure.

It can work if the ARP entry, what Ingo called the /32 is created
before you add your /23.

Post by Ingo Schwarze
But once part of it gets broken for whatever reason, i don't see
how it could possibly automatically recover via the normal RTF_CLONING
mechanism.

It can't because as you described the /23 will be a better match. And the
reason will be the expiration of the ARP cache.

[Tom Smyth]

Ingo , Martin, All,

i can confirm
when the issue occured the command

arp -s gateway-ip-address gateway-mac-address
worked to restore connectivity

Cheers,

Tom Smyth

Post by Tom Smyth
Hello Ingo, Martin,All,
I think you hit the nail on the head, (I was too busy looking at the
routing table (and forgot the fundamental principle of longest prefix
match)
so if I have a static arp entry before adding in the
(more specific than the connected route) i should be OK
just to explain (the method in my madness )
(what i agree in hindsight is a fragile setup)
basically I have a /22 network the clients are isolated from each
other on Layer 2 (so the clients cant see each others
arp requests / replys) (Bridge horizon / protected ports / privatevlan)
to limit bandwith wasting stuff such as broadcasts and other
security issues such as rogue DHCP servers etc.
The clients can only see the gateway
and the gateway can see all the clients .. I have heard of people
using some proxy arp solutions on the gateway and perhaps
that is what I should be doing rather than the ( more specific
than connected static routes)
Does anyone who operate Access layer in ISPS have suggestions
I appreciate the help and the reminder of the longest prefix match
rule :)
Thanks again

Post by Martin Pieuchot

Post by Ingo Schwarze
[...]
So what you are doing seems fragile to me. It may sometimes work
due to order of configuration, timeouts, whatever, i'm not sure.

It can work if the ARP entry, what Ingo called the /32 is created
before you add your /23.

Post by Ingo Schwarze
But once part of it gets broken for whatever reason, i don't see
how it could possibly automatically recover via the normal RTF_CLONING
mechanism.

It can't because as you described the /23 will be a better match. And the
reason will be the expiration of the ARP cache.