openup and OpenBSD ftp and pkg_add via proxy

[Steve Shockley]

I'm trying to get openup to work through a proxy. I'm able to get it to
work through an antique Bluecoat proxy, but it fails using a Websense
Content Gateway (more or less Inktomi/Apache Traffic Server) or a
Fortigate firewall device. I'm using 5.8-stable, and this happened on
earlier releases as well. If I try pkg_add with curl or wget it always
works, but openup makes it difficult to use anything but ftp. (You can
set FETCH in openup.conf, but it gets reset in line 249. It also
doesn't appear to pass the FETCH_CMD environment variable to pkg_add.)

Anyway, if I run:
# export PKG_PATH=https://stable.mtier.org/updates/$(uname -r)/$(arch
-s):ftp://mirrors.nycbug.org/pub/OpenBSD/$(uname -r)/packages/$(arch -s)
# pkg_add -vvvvv binpatch58-amd64-kernel binpatch58-amd64-libcrypto
binpatch58-amd64-smtpd binpatch58-amd64-sshd binpatch58-amd64-sslhello

it sits for a minute and I get:
Update candidates: quirks-2.114 -> quirks-2.114
quirks-2.114 signed on 2015-08-09T11:57:52Z
No change in quirks-2.114
parsing binpatch58-amd64-kernel-2.0
Error from
https://stable.mtier.org/updates/5.8/amd64/binpatch58-amd64-kernel-2.0.tgz
ftp: connect: Connection timed out
Can't find CONTENTS from
https://stable.mtier.org/updates/5.8/amd64/binpatch58-amd64-kernel-2.0.tgz
parsing binpatch58-amd64-libcrypto-1.0p0
Error from
https://stable.mtier.org/updates/5.8/amd64/binpatch58-amd64-libcrypto-1.0p0.tgz
ftp: SSL read error: 4294967295
Can't find CONTENTS from
https://stable.mtier.org/updates/5.8/amd64/binpatch58-amd64-libcrypto-1.0p0.tgz
parsing binpatch58-amd64-smtpd-1.0
No change in binpatch58-amd64-smtpd-1.0
parsing binpatch58-amd64-sshd-1.0
^C

With more verbose ftp and less verbose pkg_add:
# export FETCH_CMD='/usr/bin/ftp -v'
# pkg_add -ui
quirks-2.114 signed on 2015-08-09T11:57:52Z
Error from
ftp://mirrors.nycbug.org/pub/OpenBSD/5.8/packages/amd64/apache-httpd-openbsd-1.3.20140502p4.tgz
ftp: connect: Connection timed out
^C

Note that the proxy thought the transaction was successful:
a.b.c.d - - [30/Oct/2015:15:20:15 -0500] "GET
ftp://mirrors.nycbug.org/pub/OpenBSD/5.8/packages/amd64/apache-httpd-common-2.4.16.tgz
HTTP/1.0" 200 160810 000 0 0 0 128 195 0 0 0

I can always download the files using ftp, I only have problems when
using pkg_add.

What else can I do to diagnose? Thanks.

[Raf Czlonka]

Post by Steve Shockley
I'm trying to get openup to work through a proxy. I'm able to get it to
work through an antique Bluecoat proxy, but it fails using a Websense
Content Gateway (more or less Inktomi/Apache Traffic Server) or a Fortigate
firewall device. I'm using 5.8-stable, and this happened on earlier
releases as well. If I try pkg_add with curl or wget it always works, but
openup makes it difficult to use anything but ftp. (You can set FETCH in
openup.conf, but it gets reset in line 249. It also doesn't appear to pass
the FETCH_CMD environment variable to pkg_add.)
# export PKG_PATH=https://stable.mtier.org/updates/$(uname -r)/$(arch
-s):ftp://mirrors.nycbug.org/pub/OpenBSD/$(uname -r)/packages/$(arch -s)
# pkg_add -vvvvv binpatch58-amd64-kernel binpatch58-amd64-libcrypto
binpatch58-amd64-smtpd binpatch58-amd64-sshd binpatch58-amd64-sslhello
Update candidates: quirks-2.114 -> quirks-2.114
quirks-2.114 signed on 2015-08-09T11:57:52Z
No change in quirks-2.114
parsing binpatch58-amd64-kernel-2.0
Error from
https://stable.mtier.org/updates/5.8/amd64/binpatch58-amd64-kernel-2.0.tgz
ftp: connect: Connection timed out
Can't find CONTENTS from
https://stable.mtier.org/updates/5.8/amd64/binpatch58-amd64-kernel-2.0.tgz
parsing binpatch58-amd64-libcrypto-1.0p0
Error from https://stable.mtier.org/updates/5.8/amd64/binpatch58-amd64-libcrypto-1.0p0.tgz
ftp: SSL read error: 4294967295
Can't find CONTENTS from https://stable.mtier.org/updates/5.8/amd64/binpatch58-amd64-libcrypto-1.0p0.tgz
parsing binpatch58-amd64-smtpd-1.0
No change in binpatch58-amd64-smtpd-1.0
parsing binpatch58-amd64-sshd-1.0
^C
# export FETCH_CMD='/usr/bin/ftp -v'
# pkg_add -ui
quirks-2.114 signed on 2015-08-09T11:57:52Z
Error from ftp://mirrors.nycbug.org/pub/OpenBSD/5.8/packages/amd64/apache-httpd-openbsd-1.3.20140502p4.tgz
ftp: connect: Connection timed out
^C
a.b.c.d - - [30/Oct/2015:15:20:15 -0500] "GET ftp://mirrors.nycbug.org/pub/OpenBSD/5.8/packages/amd64/apache-httpd-common-2.4.16.tgz
HTTP/1.0" 200 160810 000 0 0 0 128 195 0 0 0
I can always download the files using ftp, I only have problems when using
pkg_add.
What else can I do to diagnose? Thanks.

Hi Steve,

You hadn't mentioned it and I don't have a proxy at hand to test it, but
won't either simply honour the "http_proxy" environment variable?

Regards,

Raf

[Steve Shockley]

Post by Raf Czlonka
Hi Steve,
You hadn't mentioned it and I don't have a proxy at hand to test it, but
won't either simply honour the "http_proxy" environment variable?

Thanks for the reply. Everything does seem to honor the http_proxy
environment variable, but I can only get /usr/bin/ftp to reliably use
one of the three brands of proxy servers I have available (and the one
that works is slated for decom).

(I also noticed that I copied the wrong log line for the Apache
download. I'll re-check that tomorrow when I'm near the machine. I've
been staring that this problem too long.)