Using relayd as a reverse proxy for multiple local servers

[Philip Kaludercic]

Hi,

I have been trying to configure relayd for a few days now to multiplex
multiple servers running on the same local machine, while at the same
time taking care of TLS.

A simplified state of my configuration looks something like this:

log connection
log state changes

table <httpd> { 127.0.0.1 }
table <serv1> { 127.0.0.1 }
table <serv2> { 127.0.0.1 }
table <acme> { 127.0.0.1 }

http protocol "http" {
match request header "Host" value "example.com" forward to <httpd>
match request header "Host" value "sub.example.com" forward to <serv1>
match request header "Host" value "beispiel.de" forward to <serv2>
match request path "/.well-known/acme*" forward to <acme>
}

http protocol "https" {
tls keypair "example.com" # responsible for example.com and sub.example.com
tls keypair "beispiel.de"

match request header "Host" value "example.com" forward to <httpd>
match request header "Host" value "sub.example.com" forward to <serv1>
match request header "Host" value "beispiel.de" forward to <serv2>
match request path "/.well-known/acme*" forward to <acme>
}

relay plain {
listen on * port http

protocol "http"

forward to <httpd> port 8080
forward to <serv1> port 8081
forward to <serv2> port 8082
forward to <acme> port 8080
}

relay secure {
listen on * port https tls

protocol "https"

forward to <httpd> port 8080
forward to <serv1> port 8081
forward to <serv2> port 8082
forward to <acme> port 8080
}

The "plain" relayd works just the way it should, it redirects every
request to the right destination. "secure" on the other hand triggers an
error I cannot make sense of:

# relayd -nvvv
relay_load_certfiles: using certificate /etc/ssl/example.com:443.crt
relay_load_certfiles: using private key /etc/ssl/private/example.com:443.key
relay_load_certfiles: using certificate /etc/ssl/beispiel.de:443.crt
relay_load_certfiles: using private key /etc/ssl/private/beispiel.de:443.key
/etc/relayd.conf:46: cannot load certificates for relay secure4:443

I have looked into the source code, but couldn't find where "secure4"
comes from. The certificates and keys were generated using acme-client,
and they have the default permissions (crt is 444, key is 400).

Am I doing the right thing here, considering what I want to achieve? I
would be very grateful for any comments or hints on what I could be
doing wrong.

[Kent Watsen]

I did this too, because I have:

1) a single external IP
2) multiple internal HTTP-based services
3) a port-based firewall policy

This whole issue would disappear, and remove a single point of failure (relayd), if my firewall directed inbound traffic based on URLs (for port 80) and SNI (for port 443). Alas, I’m not there yet, and this setup has been working okay for me for a couple years now.

My biggest complaint is that, if one internal site is down (e.g., 'www'), `relayd` will direct traffic to that site to another (‘blog’), which may seem innocent, but could be a real problem if, e.g., the external IP is shared by multiple domains and traffic to 'www.domain1.com' gets mapped to ‘www.domain2.com’. I’ve never really looked into solving the issue, as it’s easier to restart the downed service, but would be thrilled if someone could explain how to fix it.

A redacted version of my /etc/relayd.conf follows. But note that I also have `httpd` running on this machine, listening for inbound port 80 requests, in order to 1) handle ACME requests and 2) redirect all port 80 requests to port 443. Both configs follow.

PS: there are many ways to skin the cat. For example, you’re running different httpd instances on ports versus my running them on different VMs. Also, how we approach handling port 80 and ACME requests. Still, hopefully seeing my config helps.

K.


==== /etc/httpd.conf ====

# This rule is used to redirect all (except ACME) external
# HTTP/80 requests to the HTTPS/443 equivalent.
#
# Note that `relayd` (/etc/relayd.conf) terminates *all*
# external HTTPS/443 requests and forward them to
# the appropriate HTTP/80 server

server "default" {
listen on egress port 80
location "/.well-known/acme-challenge/*" {
root "/acme"
request strip 2
}
location "*" {
block return 301 "https://$HTTP_HOST$REQUEST_URI"
}
}


==== /etc/relayd.conf ====

# this is the ONLY machine that accepts inbound connections to
# <external-ip>:443
#
# it uses the certificate maintained by Let's Encrypt (acme-client)
#
# it fowards the request to the correct <destination-server>:80
# via inspecting the "Host" HTTP header field's value

# define some variables
www_example_net="10.0.1.X"
blog_example_net=“10.0.1.Y"
git_example_net=“10.0.1.Z”

# make a table out of each
table <www_example_net_table> { $www_example_net }
table <blog_example_net_table> { $blog_example_net }
table <git_example_net_table> { $git_example_net }

# http protocol-specific rules
http protocol "my_http_protocol_config" {
match request header "Host" value "www.example.net" forward to <www_example_net_table>
match request header "Host" value "blog.example.net" forward to <blog_example_net_table>
match request header "Host" value "git.example.net" forward to <git_example_net_table>

match response header remove "Server"

# is this supposed to be "request" or "response"? (I see both in the forums!)
match request header set "Connection" value "close"
match response header set "Connection" value "close"

tcp { nodelay, sack }

tls keypair example.net
}

# handle inbound port 443 traffic
relay "my_relay" {
listen on egress port 443 tls
protocol my_http_protocol_config
forward to <www_example_net_table> port 80 check tcp
forward to <blog_example_net_table> port 80 check tcp
forward to <git_example_net_table> port 80 check tcp
}

[Philip Kaludercic]

Post by Kent Watsen
A redacted version of my /etc/relayd.conf follows. But note that I
also have `httpd` running on this machine, listening for inbound port
80 requests, in order to 1) handle ACME requests and 2) redirect all
port 80 requests to port 443. Both configs follow.

Could it be that you have only one certificate, for every service? My
understanding was that a protocol could specify more than one "tls
keypair" directive, and the "right one" would be chosen, depending on
the actual request.

Post by Kent Watsen
PS: there are many ways to skin the cat. For example, you’re running
different httpd instances on ports versus my running them on different
VMs.

I am not sure if this makes a difference, after all non-encrypted
traffic operates the way it should.

[Joel Carnat]

Hi,

In my testings, using « listen on * port https tls » doesn’t work either.
What I did is replace the « * » with the IP address where I want relayd to listen to. And as my gateway has several interfaces, I created a relay section for each single interface I wanted relayd to bind to.

Regards,
Joel

Envoyé de mon iPad

Post by Philip Kaludercic
listen on * port https tls

[Philip Kaludercic]

Post by Joel Carnat

Post by Philip Kaludercic
listen on * port https tls

In my testings, using « listen on * port https tls » doesn’t work either.
What I did is replace the « * » with the IP address where I want
relayd to listen to. And as my gateway has several interfaces, I
created a relay section for each single interface I wanted relayd to
bind to.

It seems I oversimplified my example, my configuration has the exact
IPv4 and IPv6 ports, but that does not seem to make any difference :/

[Jean-Pierre de Villiers]

Personally, I would drop the keypairs you define and rename the
certificates as 'localhost.crt' for example.com and its subdomain and a
certificate 'localhost:8082' for handling beispiel.de. Similarly,
repeat this for the private keys as well.

No further configuration is needed after that. See the description of
'keypair' under the PROTOCOLS section in relayd.conf(8).

Regards,
JP

Post by Philip Kaludercic
Hi,
I have been trying to configure relayd for a few days now to multiplex
multiple servers running on the same local machine, while at the same
time taking care of TLS.
log connection
log state changes
table <httpd> { 127.0.0.1 }
table <serv1> { 127.0.0.1 }
table <serv2> { 127.0.0.1 }
table <acme> { 127.0.0.1 }
http protocol "http" {
match request header "Host" value "example.com" forward to <httpd>
match request header "Host" value "sub.example.com" forward to <serv1>
match request header "Host" value "beispiel.de" forward to <serv2>
match request path "/.well-known/acme*" forward to <acme>
}
http protocol "https" {
tls keypair "example.com" # responsible for example.com and sub.example.com
tls keypair "beispiel.de"
match request header "Host" value "example.com" forward to <httpd>
match request header "Host" value "sub.example.com" forward to <serv1>
match request header "Host" value "beispiel.de" forward to <serv2>
match request path "/.well-known/acme*" forward to <acme>
}
relay plain {
listen on * port http
protocol "http"
forward to <httpd> port 8080
forward to <serv1> port 8081
forward to <serv2> port 8082
forward to <acme> port 8080
}
relay secure {
listen on * port https tls
protocol "https"
forward to <httpd> port 8080
forward to <serv1> port 8081
forward to <serv2> port 8082
forward to <acme> port 8080
}
The "plain" relayd works just the way it should, it redirects every
request to the right destination. "secure" on the other hand triggers an
# relayd -nvvv
relay_load_certfiles: using certificate /etc/ssl/example.com:443.crt
relay_load_certfiles: using private key /etc/ssl/private/example.com:443.key
relay_load_certfiles: using certificate /etc/ssl/beispiel.de:443.crt
relay_load_certfiles: using private key /etc/ssl/private/beispiel.de:443.key
/etc/relayd.conf:46: cannot load certificates for relay secure4:443
I have looked into the source code, but couldn't find where "secure4"
comes from. The certificates and keys were generated using acme-client,
and they have the default permissions (crt is 444, key is 400).
Am I doing the right thing here, considering what I want to achieve? I
would be very grateful for any comments or hints on what I could be
doing wrong.
--
Philip K.

[Philip Kaludercic]

Post by Jean-Pierre de Villiers
Personally, I would drop the keypairs you define and rename the
certificates as 'localhost.crt' for example.com and its subdomain and a
certificate 'localhost:8082' for handling beispiel.de. Similarly,
repeat this for the private keys as well.

I tried this out, but it didn't help ._.

Now it doesn't even appear to notice the certificates, as the output now
is just

relayd -nvvv
/etc/relayd.conf:43: cannot load certificates for relay secure

But "at least", it says "secure" instead of "secure4:443"?

I am wondering if this could be a bug? It appears to make no sense to
me...

Post by Jean-Pierre de Villiers
No further configuration is needed after that. See the description of
'keypair' under the PROTOCOLS section in relayd.conf(8).

That confuses me, as one the one hand the manual says

The relay will attempt to look up a private key in
/etc/ssl/private/name:port.key and a public certificate in
/etc/ssl/name:port.crt, WHERE PORT IS THE SPECIFIED PORT THAT THE
RELAY LISTENS ON.

which would mean that the certificate should be called localhost:443 (or
127.0.0.1:443), but then again the same paragraph says

If not specified, a keypair will be loaded using the specified IP
address of the relay as name.

Which I read as saying that it will try to use /etc/ssl/secure.key, in
my case. That obviously won't work, as I need different certificates for
different domains.

[Jean-Pierre de Villiers]

Apologies yes, my error. I forgot I divert traffic using pf to my
relayd listener.

I've never seen/used a wildcard listen address in relayd before but I'm
guessing that, in your case, a listener is created for each ip on every
interface. This ofcourse raises the question: does every ip on every
one of your interfaces map to one of your hosts?

I ask this as I'm relatively certain relayd will expect a TLS
certificate for each address it listens on, regardless of whether it
forwards traffic from that IP to some host or not.

This would explain why it complains that the IPv4 listener 'secure4' was
unable to load a certificate.

Regards,
JP

Post by Philip Kaludercic
That confuses me, as one the one hand the manual says
The relay will attempt to look up a private key in
/etc/ssl/private/name:port.key and a public certificate in
/etc/ssl/name:port.crt, WHERE PORT IS THE SPECIFIED PORT THAT THE
RELAY LISTENS ON.
which would mean that the certificate should be called localhost:443 (or
127.0.0.1:443), but then again the same paragraph says
If not specified, a keypair will be loaded using the specified IP
address of the relay as name.
Which I read as saying that it will try to use /etc/ssl/secure.key, in
my case. That obviously won't work, as I need different certificates for
different domains.
--
Philip K.

[Philip Kaludercic]

Post by Jean-Pierre de Villiers
Apologies yes, my error. I forgot I divert traffic using pf to my
relayd listener.

Ok, I will look into this too.

Post by Jean-Pierre de Villiers
I've never seen/used a wildcard listen address in relayd before but I'm
guessing that, in your case, a listener is created for each ip on every
interface. This ofcourse raises the question: does every ip on every
one of your interfaces map to one of your hosts?

I only have one I and one interface, but multiple hosts. My
understanding is that if I had multiple IPs/interfaces, non of this
would be an issue.

[Jean-Pierre de Villiers]

I should mention, as I did in a thread months ago, there are extensive
example configurations available in '/etc/examples'. Yours would be
'/etc/examples/relayd.conf', which illustrates several typical use
cases.

Also, in future you might want to post the entirety of your
configuration files. Otherwise, the responses will be just as vague and
it wastes everyone's time (including yours). You may of course
"sanitize" any identifying information like hostnames and public ip
adressess.

[Ronan Viel]

Post by Philip Kaludercic
which would mean that the certificate should be called localhost:443 (or
127.0.0.1:443)

No, it means the certificate must be named /etc/ssl/localhost:443.crt or /etc/ssl/127.0.0.1:443.crt, and the matching key must be in file /etc/ssl/private/localhost:443.key or /etc/ssl/private/127.0.0.1:443.key

The manual is clear enough but requires attention to filenames and paths.

Ronan

[Philip Kaludercic]

Post by Ronan Viel

Post by Philip Kaludercic
which would mean that the certificate should be called localhost:443 (or
127.0.0.1:443)

No, it means the certificate must be named /etc/ssl/localhost:443.crt
or /etc/ssl/127.0.0.1:443.crt, and the matching key must be in file
/etc/ssl/private/localhost:443.key or
/etc/ssl/private/127.0.0.1:443.key

Yes, of course, that was I typo. My point was that the port in the file
names doesn't appear to be the port the traffic is being directed *towards*.

Post by Ronan Viel
The manual is clear enough but requires attention to filenames and paths.
Ronan