CARP without IP on the physical interfaces of carp group?

[Peus, Christoph]

Hi all,

in the official CARP/pfsync faq here: http://www.openbsd.org/faq/pf/carp.html
I found an information, that suggests that it's possible to use CARP without
IPs attached to the physical interfaces used in a CARP group:

ipaddress
This is the shared IP address assigned to the redundancy group. This address
does not have to be in the same subnet as the IP address on the physical
interface (if present). This address needs to be the same on all hosts in the
group, however.

And in this netbsd related guide:
https://www.netbsd.org/docs/guide/en/chap-carp.html I found this :
"...although with the introduction of the carpdev directive, there is no more
need for IP addresses on the physical interfaces."

Is it really possible to use CARP without IPs assigned to the physical
interfaces? How does the communication between the interfaces of a group work
if there are no IPs assigned to them? Which disadvantages could this mode of
operation have compared to the classic mode with IPs assigned? (Each and every
configuration example I found on the net including those on the OpenBSD site,
use IPs assigned to the physical interfaces...)

Thanks!

Regards
Christoph



Private Universit?t Witten/Herdecke gGmbH
Alfred-Herrhausen-Stra?e 50
D - 58448 Witten

Homepage: http://www.uni-wh.de
Twitter: http://twitter.com/UniWH
Facebook: http://www.facebook.com/UniWH

Gesch?ftsf?hrung: Prof. Dr. Martin Butzlaff (Pr?sident), Dipl. oec. Jan Peter
Nonnenkamp (Kanzler)

Sitz der Gesellschaft: Witten
Handelsregister des Amtsgerichts Bochum Nr. HRB 8671

[Henning Brauer]

Post by Peus, Christoph
Is it really possible to use CARP without IPs assigned to the physical
interfaces?

Sure.

Post by Peus, Christoph
How does the communication between the interfaces of a group work
if there are no IPs assigned to them?

multicast

Post by Peus, Christoph
Which disadvantages could this mode of operation have compared to the
classic mode with IPs assigned?

the backup node might not be able to reach the network on the carp if

[Peus, Christoph]

Henning, thanks for your quick reply.

Post by Henning Brauer

Post by Peus, Christoph
Which disadvantages could this mode of operation have compared to the
classic mode with IPs assigned?

the backup node might not be able to reach the network on the carp if

Hmm... what does this mean to me..? To make it more precise - my setup looks like this:

Physical NIC -> trunk interface -> vlan interface = physical interface of the carpdev -> carpdev -> virtual IP (no IP on the vlan interface)

I tested this setup and experienced unpredictable transitions from and to master/backup state on different carp groups configured that way and also unstable connections with >50% packet loss. I strictly followed the carp/pfsync configuration guidance in respect of advskew, sysctl.conf options and so on. So I wondered if this may have been caused by the "no IP" configuration.

Is multicast only used when there are no IPs configured for the physical interfaces of the carpdevs?

Regards
Christoph


Private Universität Witten/Herdecke gGmbH
Alfred-Herrhausen-Straße 50
D - 58448 Witten

Homepage: http://www.uni-wh.de
Twitter: http://twitter.com/UniWH
Facebook: http://www.facebook.com/UniWH

Geschäftsführung: Prof. Dr. Martin Butzlaff (Präsident), Dipl. oec. Jan Peter Nonnenkamp (Kanzler)

Sitz der Gesellschaft: Witten
Handelsregister des Amtsgerichts Bochum Nr. HRB 8671

[Adam Thompson]

Post by Peus, Christoph
Henning, thanks for

your quick reply.

Post by Peus, Christoph

Post by Peus, Christoph
Which disadvantages could this mode of

operation have compared to the classic mode with IPs assigned?

Post by Peus, Christoph

the

backup node might not be able to reach the network on the carp if
Hmm... what does this mean to me..? To make it more precise - my setup

Post by Peus, Christoph
Physical NIC -> trunk interface -> vlan interface

= physical interface of the carpdev -> carpdev -> virtual IP (no IP on
the vlan interface)

Post by Peus, Christoph
I tested this setup and experienced

unpredictable transitions from and to master/backup state on different
carp groups configured that way and also unstable connections with >50%
packet loss. I strictly followed the carp/pfsync configuration guidance
in respect of advskew, sysctl.conf options and so on. So I wondered if
this may have been caused by the "no IP" configuration.

Post by Peus, Christoph
Is

multicast only used when there are no IPs configured for the physical
interfaces of the carpdevs?

Not exactly.
CARPPEER _peer_address_

Post by Peus, Christoph
Send the carp advertisements to a specified

point-to-point peer

Post by Peus, Christoph
or multicast group instead of sending the messages

to the default

Post by Peus, Christoph
carp multicast group. The _peer_address_ is the IP

address of the

Post by Peus, Christoph
other host taking part in the carp cluster. With this

option,

Post by Peus, Christoph
carp(4) traffic can be protected using ipsec(4) and it may

be

Post by Peus, Christoph
desired in networks that do not allow or have problems with IPv4

multicast traffic.

So it always uses multicast unless you specify
"carppeer". Reading ip_carp.c, I don't see any reason carppeer can't be
a completely arbitrary address, including an address bound to a loopback
adapter. Using carppeer unwisely can defeat the purpose of the multicast
transmission, i.e. detecting layer 2 failures, if your layer 3 ensures
the carppeer address remains reachable (e.g. both hosts are running OSPF
and can re-route over another interface).

Recap:
- Interface state
(carrier/no-carrier) detects local layer 1 failures only
- Multicast
heartbeat detects layer 2 failures, and remote layer 1 failures; this is
desirable in VLAN scenarios, I believe, where carrier/no-carrier does
not signify anything useful.
- Using carpdev specifies the interface on
which to bind the CARP address and on which to send multicast packets.
If it is not specified, it is inferred from the subnet.
- Using
carppeer disables multicast - the nominal use case is to protect CARP
traffic with IPSec. Other uses are possible, but questionable because
they may break lower-level assumptions. (or so I believe, anyway. I'm
sure Henning will correct me if not.)

FWIW, I don't use carppeer even
though it could save me substantial IP address space, for a couple of
reasons:
1) I want the canary-in-the-coal-mine to inform me of any
layer 2 weirdness
2) I prefer predictability and "normal" use cases
3)
if I ever stop using CARP and switch to HSRP or VRRP, I'll need those
addresses again

-Adam

[Henning Brauer]

Post by Adam Thompson
traffic with IPSec. Other uses are possible, but questionable because
they may break lower-level assumptions. (or so I believe, anyway. I'm
sure Henning will correct me if not.)

I don't think carppeer uses than manually specifying the IP on the
carpdev of the other node are very well tested, so there might be
surprises, but I really don't why other uses shouldn't work as long as
the nodes see each other.

Post by Adam Thompson
FWIW, I don't use carppeer even
though it could save me substantial IP address space, for a couple of
1) I want the canary-in-the-coal-mine to inform me of any
layer 2 weirdness
2) I prefer predictability and "normal" use cases
3)
if I ever stop using CARP and switch to HSRP or VRRP, I'll need those
addresses again

you are creating massive confusion here regarding carppeer and
unnumbered carpdevs - those really have nothing to do with each other.

That said, I do use unnumbered carpdevs in some cases and places.

If carp0 has 10.0.0/24, and carp0 is backup on nodeX, nodeX might not
be able to reach 10.0.0/24. No more, no less. Can hurt, esp when the
default gateway is in that net, but is perfectly fine in many cases.

[Adam Thompson]

Post by Henning Brauer

Post by Adam Thompson
FWIW, I don't use

carppeer even though it could save me substantial IP address space, for
a couple of reasons: 1) I want the canary-in-the-coal-mine to inform me
of any layer 2 weirdness 2) I prefer predictability and "normal" use
cases 3) if I ever stop using CARP and switch to HSRP or VRRP, I'll need
those addresses again

Post by Henning Brauer
you are creating massive confusion here

regarding carppeer and

Post by Henning Brauer
unnumbered carpdevs - those really have nothing

to do with each other.

Post by Henning Brauer
That said, I do use unnumbered carpdevs in

some cases and places.

Post by Henning Brauer
If carp0 has 10.0.0/24, and carp0 is backup

on nodeX, nodeX might not

Post by Henning Brauer
be able to reach 10.0.0/24. No more, no

less. Can hurt, esp when the

Post by Henning Brauer
default gateway is in that net, but is

perfectly fine in many cases.

Whoops, you're right - I fixed a few
errors in my email before sending, but missed those ones.

#1 is
somewhat valid - using carppeer would prevent me from learning that
multicast was broken. I'm not sure how it could ever break on a L2 VLAN,
but still...

#2 is somewhat valid - using carppeer isn't the typical
scenario. Nor is using carpdev, although I have the impression that use
of carpdev (and therefore only needing 1 IP address) is increasing.

#3
really has nothing to do with carppeer, unless I deliberately used
addresses "foreign" to that interface/subnet, which would probably be
silly, but could be another way around needing 3 IPs in the same subnet
to implement CARP. Using carpdev is a much more sensible option to avoid
IP address proliferation with CARP.

-Adam

[Henning Brauer]

Nor is using carpdev [the typical case], although I have the
impression that use of carpdev (and therefore only needing 1 IP
address) is increasing.

I consider carpdev that "natural" use, we're stacking interfaces after
all.

I even wouldn't be surprised if the !carpdev case bites the bullet at
some point, should we change/redesign basics. There's nothing up in
that direction tho, call it a vague feeling.

[Janne Johansson]

Post by Adam Thompson
#1 is
somewhat valid - using carppeer would prevent me from learning that
multicast was broken. I'm not sure how it could ever break on a L2 VLAN,
but still...

I've had bad broadcom (bnx (4)) cards do that to me.
Worked better with carppeer but best with intels instead.