Discussion:
Addblock + Badhost blocking via unbound(8) and pf anchors
Add Reply
Jordan Geoghegan
2017-12-29 22:50:31 UTC
Reply
Permalink
Raw Message
Hi everyone,

Due to the number of people who have requested my add-blocking scripts,
I figured I would also post them to @misc so anyone can easily enjoy
network-wide bad-host/add-blocking.

I won't go into detail on how to set up routing/dhcp/unbound/anchors
etc, for that see: https://www.openbsd.org/faq/pf/example1.html

I've included some example files from my an Edgerouter I have set up .
They are trimmed down for brevities sake; the conf files are not
production ready, these are merely examples.

This setup is easily customizable, if you come across any other block
lists you prefer, then they can be dropped in no problem. I chose to use
solely the StevenBlack hosts file because it is a master list compiled
from all the major banlists found in popular blocking products such as
uBlock Origin, Addblock Plus et al. I also chose this file because it is
filtered for duplicates as unbound(8) is said to struggle when there are
redundancies in the blocklists, I'm told -- though I've never had any issue.

You're going to have to read the scripts and create the directories the
scripts are calling and edit the anchor macros to fit your interface
layout (I doubt everyone here is running cnmac0 as egress) and also will
have to make the scripts executable and set them to run at regular
intervals with crontab, ideally nightly.

I didn't make these scripts intelligent because I figured it was simpler
to just run mkdir once rather than add extra lines to the script.

I know the pf.conf is fairly long, I thought I would show an example of
my prio and queing setup as an example, or conversely to see if anyone
can poke any holes in it.

All the relevant bits regarding the anchors and blocklists are found at
the end of the pf.conf file. See below that for the anchor conf files
we're calling as well.

Hope this helps,

Jordan Geoghegan


First, the scripts:

*DNS addblock script:*

StevenBlack.sh:

cd /var/unbound/etc/banlist && \
ftp https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts && \
cat hosts | grep '^0\.0\.0\.0' | awk '{print "local-zone: \""$2"\" redirect\nlocal-data: \""$2" A 0.0.0.0\""}' > ads.conf
rcctl reload unbound

###

*IP based malicious IP blocking:*

banlist.sh:

cd /etc/blocklist && ftp https://www.binarydefense.com/banlist.txt\
&& ftp https://rules.emergingthreats.net/blockrules/compromised-ips.txt\
&& ftp https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt\
&& ftp https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level3.netset\
&& pfctl -a banlist -f /etc/banlist.conf

###

As you can see, we are going to have to make an anchor in pf called
'banlist' and modify the unbound.conf to load our banlist 'ads.conf'

If that's all you need, then you're pretty much good to go. If you would
like to see my example conf files, see below.

*


Example unbound.conf:*

# $OpenBSD: unbound.conf,v 1.7 2016/03/30 01:41:25 sthen Exp $

server:
interface: 172.17.17.1
interface: 127.0.0.1
access-control: 172.17.17.0/24 allow
access-control: 172.17.0.0/24 allow
do-not-query-localhost: no
hide-identity: yes
hide-version: yes
include: /var/unbound/etc/banlist/ads.conf

forward-zone:
name: "."
forward-addr: UR.DNS.GO.HERE
forward-addr: UR.DNS.GO.HERE

###


*Example pf.conf:*

# $OpenBSD: pf.conf,v 1.54 2014/08/23 05:49:42 deraadt Exp $
#
# See pf.conf(5) and /etc/examples/pf.conf

# By default, do not permit remote connections to X11
block return in on ! lo0 proto tcp to port 6000:6010
#
ext_if="{ cnmac0 }"
int_if="{ cnmac1 cnmac2 }"
lan_if="{ cnmac1 }"
wifi_if="{ cnmac2 }"
goodguys="{ 172.17.17.0/24 }"
wifiguys="{ 172.17.0.0/24 }"
chromecast="{ 172.17.0.12 172.17.0.13 172.17.0.23 }"
xbox360="{ 172.17.0.19 }"
printer="{ 172.17.0.17 }"
Jordan="{ XXX.XX.XXX.XX }"

table <martians> { 0.0.0.0/8 10.0.0.0/8 127.0.0.0/8 169.254.0.0/16 \
172.16.0.0/12 192.0.0.0/24 192.0.2.0/24 224.0.0.0/3 \
192.168.0.0/16 198.18.0.0/15 198.51.100.0/24 \
203.0.113.0/24 }


# Queue List [ Download ]
queue download on cnmac2 bandwidth 70M max 70M
queue media-down parent download bandwidth 20M min 5M max 20M burst 24M for 200ms
queue xbox-down parent media-down bandwidth 4M max 4M burst 8M for 200ms
queue chrome-down parent media-down bandwidth 16M max 16M burst 20M for 225ms
queue std-down parent download bandwidth 50M min 5M max 50M burst 70M for 500ms default


set block-policy drop
set loginterface egress
set skip on lo0
match in all scrub (no-df random-id max-mss 1440)
match out on egress inet from !(egress:network) to any nat-to (egress:0)
block in quick on egress from <martians> to any
block return out quick on egress from any to <martians>
block quick inet6
block all

# A bit of edgy prio and bandwidth queuing, I felt like taking pf out for a test drive here

pass in on $lan_if from $goodguys tag LAN set prio 6
pass in on $wifi_if from $wifiguys tag WIFI modulate state set queue std-down
pass in on $wifi_if from $chromecast tag CHROME modulate state set prio 2 \
set queue chrome-down
block out on $lan_if tagged WIFI
block out on $lan_if tagged CHROME
antispoof for { egress cnmac0 cnmac1 cnmac2 lo0 }
pass in quick on $ext_if from $Jordan to any tag Jordan
block in on $ext_if proto { tcp udp } from any to any port ssh ! tagged Jordan
pass out on $ext_if inet


# Printers Ruleset | Block Printer on Egress && allow $goodguys subnet
block out on $ext_if from $printer to any
pass out quick on $wifi_if from $goodguys to $printer

# Spammers
anchor banlist
load anchor banlist from "/etc/banlist.conf"

# DNS Redirect
anchor dns
load anchor dns from "/etc/dns-redirect.conf"


###

*Anchor banlist.conf:*


# $OpenBSD: pf.conf,v 1.54 2014/08/23 05:49:42 deraadt Exp $
#
## Spammers ##

table <banlist> persist file "/etc/blocklist/banlist.txt"\
file "/etc/blocklist/compromised-ips.txt"\
file "/etc/blocklist/emerging-Block-IPs.txt"\
file "/etc/blocklist/firehol_level3.netset"
block in on egress from <banlist> to any
block out log on egress from any to <banlists>


####

*Anchor dns-redirect.conf:***


# $OpenBSD: pf.conf,v 1.54 2014/08/23 05:49:42 deraadt Exp $
#

wifi_lan="{ cnmac2 }"

# DNS Redirect
pass in on $wifi_lan proto { tcp udp } from any to \
{ 8.8.8.8 8.8.4.4 208.67.222.222 208.67.220.220 209.244.0.3 } port 53 \
tag google rdr-to 172.17.17.1

# I added this because several devices were aggressively pinging 8.8.8.8 on my network and it was annoying me
pass in on $wifi_lan from any to \
{ 8.8.8.8 8.8.4.4 } \
tag google rdr-to 172.17.17.1
Freddy DISSAUX
2017-12-30 08:21:48 UTC
Reply
Permalink
Raw Message
> Hi everyone,
Hello,

[ snip ]

> cat hosts | grep '^0\.0\.0\.0' | awk '{print "local-zone: \""$2"\" redirect\nlocal-data: \""$2" A 0.0.0.0\""}' > ads.conf

awk 'NF == 2 && $1 == "0.0.0.0" && $2 ~ /^[a-z0-9]/ { print "local-zone: \"" $2 "\" redirect\nlocal-data: \"" $2 " A " $1 "\"" }' host > ads.conf


Regards,
Jordan Geoghegan
2017-12-30 19:36:05 UTC
Reply
Permalink
Raw Message
I have tried using all awk for the script before, but I find piping the
grep output into awk to be 2-3x faster on the Edgerouter Lite. I just
ran some timed tests for your script against mine on the ErLite, and I
got similar results, with my script completing in ~6 seconds against the
StevenBlack hosts file, and yours at ~14 seconds. This may not be the
case on more conventional architectures. I am considering rewriting the
script in Perl to see if that runs any faster.


On 12/30/17 00:21, Freddy DISSAUX wrote:
>> Hi everyone,
> Hello,
>
> [ snip ]
>
>> cat hosts | grep '^0\.0\.0\.0' | awk '{print "local-zone: \""$2"\" redirect\nlocal-data: \""$2" A 0.0.0.0\""}' > ads.conf
> awk 'NF == 2 && $1 == "0.0.0.0" && $2 ~ /^[a-z0-9]/ { print "local-zone: \"" $2 "\" redirect\nlocal-data: \"" $2 " A " $1 "\"" }' host > ads.conf
>
>
> Regards,
>
Jordan Geoghegan
2017-12-31 19:29:40 UTC
Reply
Permalink
Raw Message
Hi Freddy,

I just ran some further benchmarks between your first and second script,
compared to mine, and again similar results were found. Your second
script was significantly faster than the first, but still didn't match
the grep-piped-into-awk config.

This shouldn't be the case though. I did further testing on my PowerMac
G4 500Mhz workstation running 6.2, which I chose because I thought a
single core ppc G4 500Mhz vs a mips64 dual core 500Mhz would be a pretty
epic showdown. I ran each script twice and wrote the output to /dev/null
to ensure disk I/O wasn't a factor. The StevenBlack hosts file has on
average ~47,000 lines including comments.

The results were somewhat surprising:

The G4 cranked out the scripts with these times:

*Your 1st script: an average of 1.415 seconds**
*

*Your 2nd script: an average of 0.54 seconds*

*My script: an average of 1.71 seconds*


This clearly shows things the way they are supposed to be, with my
script being grossly inefficient and yours being clearly superior. See
below for the times on the Edgerouter Lite:

(Note: All tested times are slower than previous results from last email
due to the machine being under a modest network load during testing.
Load remained consistent due to it being a long running slow 5 megabit
bulk network transfer it was routing. This was unavoidable due to it
being a production machine.)

*Your first script came in at an average of 20.8 seconds*

*Your second script came in at an average of 13.75 seconds *

*And my script came in at an average of 10.25 seconds. *

These results are shockingly poor compared to a G4 of the same clock
speed. The leads me to believe there may be some Octeon specific
inefficiencies at play here, namely floating point. None of the
Edgerouter units have an FPU I believe ( I know for sure the Lite
doesn't) and I am wondering if awk makes heavy use of floating point,
and thus it having to abuse the emulated fpu? During the all awk
scripts, the ERLite becomes cpu bound on 1 core.

It would be awesome if an awk guru here could confirm whether awk makes
heavy use of the fpu.

If this is indeed the case, then the PowerPC would have an extreme
advantage with its beefy AltiVec unit.

So I suppose for those folks running my addblocking scripts, it would be
wise to use Freddy Dissaux's all awk hostfile conversion method if
you're running a more conventional architecture. It would be great if
someone here could post some test results on an arm64 board!

I am now very curious to see how Perl compares against these results. I
hope I can find the time to play around with making a nice optimized
script.


On 12/31/17 03:41, Freddy DISSAUX wrote:
> Hello Jordan,
>
>> I have tried using all awk for the script before, but I find piping the
>> grep output into awk to be 2-3x faster on the Edgerouter Lite. I just
>> ran some timed tests for your script against mine on the ErLite, and I
>> got similar results, with my script completing in ~6 seconds against the
>> StevenBlack hosts file, and yours at ~14 seconds. This may not be the
>> case on more conventional architectures. I am considering rewriting the
>> script in Perl to see if that runs any faster.
>
> Could you try
>
> awk 'BEGIN { OFS = "" } NF == 2 && $1 == "0.0.0.0" { print "local-zone: \"", $2, "\" redirect"; print "local-data: \"", $2, " A 0.0.0.0\"" }' hosts > ads.conf
>
> If i understand my tests, 2 print without concat are faster than
> 1 print with concat (and faster than 1 printf)
>
>>>> cat hosts | grep '^0\.0\.0\.0' | awk '{print "local-zone: \""$2"\" redirect\nlocal-data: \""$2" A 0.0.0.0\""}' > ads.conf
> UUOC:
>
> grep '^0\.0\.0\.0' host | awk '{print "local-zone: \""$2"\" redirect\nlocal-data: \""$2" A 0.0.0.0\""}' > ads.conf
>
>
> Regards,
jin&hitman&Barracuda
2018-08-24 13:32:39 UTC
Reply
Permalink
Raw Message
Hello

Thanks for sharing all those informations. I've been looking a way to
create a blacklist and you sent this mail just on time. Your web page help
me a lot.
On the OpenBSD your script do all jobs but on linux based systems I wrote a
shell script for update iptables rules.

http://analog-radyo.blogspot.com/2018/08/dynamic-block-list-on-linux-iptables.html


Jordan Geoghegan <***@gmail.com>, 30 Ara 2017 Cmt, 01:52 tarihinde
şunu yazdı:

> Hi everyone,
>
> Due to the number of people who have requested my add-blocking scripts,
> I figured I would also post them to @misc so anyone can easily enjoy
> network-wide bad-host/add-blocking.
>
> I won't go into detail on how to set up routing/dhcp/unbound/anchors
> etc, for that see: https://www.openbsd.org/faq/pf/example1.html
>
> I've included some example files from my an Edgerouter I have set up .
> They are trimmed down for brevities sake; the conf files are not
> production ready, these are merely examples.
>
> This setup is easily customizable, if you come across any other block
> lists you prefer, then they can be dropped in no problem. I chose to use
> solely the StevenBlack hosts file because it is a master list compiled
> from all the major banlists found in popular blocking products such as
> uBlock Origin, Addblock Plus et al. I also chose this file because it is
> filtered for duplicates as unbound(8) is said to struggle when there are
> redundancies in the blocklists, I'm told -- though I've never had any
> issue.
>
> You're going to have to read the scripts and create the directories the
> scripts are calling and edit the anchor macros to fit your interface
> layout (I doubt everyone here is running cnmac0 as egress) and also will
> have to make the scripts executable and set them to run at regular
> intervals with crontab, ideally nightly.
>
> I didn't make these scripts intelligent because I figured it was simpler
> to just run mkdir once rather than add extra lines to the script.
>
> I know the pf.conf is fairly long, I thought I would show an example of
> my prio and queing setup as an example, or conversely to see if anyone
> can poke any holes in it.
>
> All the relevant bits regarding the anchors and blocklists are found at
> the end of the pf.conf file. See below that for the anchor conf files
> we're calling as well.
>
> Hope this helps,
>
> Jordan Geoghegan
>
>
> First, the scripts:
>
> *DNS addblock script:*
>
> StevenBlack.sh:
>
> cd /var/unbound/etc/banlist && \
> ftp https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts && \
> cat hosts | grep '^0\.0\.0\.0' | awk '{print "local-zone: \""$2"\"
> redirect\nlocal-data: \""$2" A 0.0.0.0\""}' > ads.conf
> rcctl reload unbound
>
> ###
>
> *IP based malicious IP blocking:*
>
> banlist.sh:
>
> cd /etc/blocklist && ftp https://www.binarydefense.com/banlist.txt\
> && <https://www.binarydefense.com/banlist.txt%5C&&> ftp
> https://rules.emergingthreats.net/blockrules/compromised-ips.txt\
> && <https://rules.emergingthreats.net/blockrules/compromised-ips.txt%5C&&>
> ftp https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt\
> && <https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt%5C&&>
> ftp
> https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level3.netset\
> &&
> <https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level3.netset%5C&&>
> pfctl -a banlist -f /etc/banlist.conf
>
> ###
>
> As you can see, we are going to have to make an anchor in pf called
> 'banlist' and modify the unbound.conf to load our banlist 'ads.conf'
>
> If that's all you need, then you're pretty much good to go. If you would
> like to see my example conf files, see below.
>
> *
>
>
> Example unbound.conf:*
>
> # $OpenBSD: unbound.conf,v 1.7 2016/03/30 01:41:25 sthen Exp $
>
> server:
> interface: 172.17.17.1
> interface: 127.0.0.1
> access-control: 172.17.17.0/24 allow
> access-control: 172.17.0.0/24 allow
> do-not-query-localhost: no
> hide-identity: yes
> hide-version: yes
> include: /var/unbound/etc/banlist/ads.conf
>
> forward-zone:
> name: "."
> forward-addr: UR.DNS.GO.HERE
> forward-addr: UR.DNS.GO.HERE
>
> ###
>
>
> *Example pf.conf:*
>
> # $OpenBSD: pf.conf,v 1.54 2014/08/23 05:49:42 deraadt Exp $
> #
> # See pf.conf(5) and /etc/examples/pf.conf
>
> # By default, do not permit remote connections to X11
> block return in on ! lo0 proto tcp to port 6000:6010
> #
> ext_if="{ cnmac0 }"
> int_if="{ cnmac1 cnmac2 }"
> lan_if="{ cnmac1 }"
> wifi_if="{ cnmac2 }"
> goodguys="{ 172.17.17.0/24 }"
> wifiguys="{ 172.17.0.0/24 }"
> chromecast="{ 172.17.0.12 172.17.0.13 172.17.0.23 }"
> xbox360="{ 172.17.0.19 }"
> printer="{ 172.17.0.17 }"
> Jordan="{ XXX.XX.XXX.XX }"
>
> table <martians> { 0.0.0.0/8 10.0.0.0/8 127.0.0.0/8 169.254.0.0/16 \
> 172.16.0.0/12 192.0.0.0/24 192.0.2.0/24 224.0.0.0/3 \
> 192.168.0.0/16 198.18.0.0/15 198.51.100.0/24 \
> 203.0.113.0/24 }
>
>
> # Queue List [ Download ]
> queue download on cnmac2 bandwidth 70M max 70M
> queue media-down parent download bandwidth 20M min 5M max 20M burst 24M
> for 200ms
> queue xbox-down parent media-down bandwidth 4M max 4M burst 8M for 200ms
> queue chrome-down parent media-down bandwidth 16M max 16M burst 20M for
> 225ms
> queue std-down parent download bandwidth 50M min 5M max 50M burst 70M for
> 500ms default
>
>
> set block-policy drop
> set loginterface egress
> set skip on lo0
> match in all scrub (no-df random-id max-mss 1440)
> match out on egress inet from !(egress:network) to any nat-to (egress:0)
> block in quick on egress from <martians> to any
> block return out quick on egress from any to <martians>
> block quick inet6
> block all
>
> # A bit of edgy prio and bandwidth queuing, I felt like taking pf out for
> a test drive here
>
> pass in on $lan_if from $goodguys tag LAN set prio 6
> pass in on $wifi_if from $wifiguys tag WIFI modulate state set queue
> std-down
> pass in on $wifi_if from $chromecast tag CHROME modulate state set prio 2 \
> set queue chrome-down
> block out on $lan_if tagged WIFI
> block out on $lan_if tagged CHROME
> antispoof for { egress cnmac0 cnmac1 cnmac2 lo0 }
> pass in quick on $ext_if from $Jordan to any tag Jordan
> block in on $ext_if proto { tcp udp } from any to any port ssh ! tagged
> Jordan
> pass out on $ext_if inet
>
>
> # Printers Ruleset | Block Printer on Egress && allow $goodguys subnet
> block out on $ext_if from $printer to any
> pass out quick on $wifi_if from $goodguys to $printer
>
> # Spammers
> anchor banlist
> load anchor banlist from "/etc/banlist.conf"
>
> # DNS Redirect
> anchor dns
> load anchor dns from "/etc/dns-redirect.conf"
>
>
> ###
>
> *Anchor banlist.conf:*
>
>
> # $OpenBSD: pf.conf,v 1.54 2014/08/23 05:49:42 deraadt Exp $
> #
> ## Spammers ##
>
> table <banlist> persist file "/etc/blocklist/banlist.txt"\
> file "/etc/blocklist/compromised-ips.txt"\
> file "/etc/blocklist/emerging-Block-IPs.txt"\
> file "/etc/blocklist/firehol_level3.netset"
> block in on egress from <banlist> to any
> block out log on egress from any to <banlists>
>
>
> ####
>
> *Anchor dns-redirect.conf:***
>
>
> # $OpenBSD: pf.conf,v 1.54 2014/08/23 05:49:42 deraadt Exp $
> #
>
> wifi_lan="{ cnmac2 }"
>
> # DNS Redirect
> pass in on $wifi_lan proto { tcp udp } from any to \
> { 8.8.8.8 8.8.4.4 208.67.222.222 208.67.220.220 209.244.0.3 } port 53 \
> tag google rdr-to 172.17.17.1
>
> # I added this because several devices were aggressively pinging 8.8.8.8
> on my network and it was annoying me
> pass in on $wifi_lan from any to \
> { 8.8.8.8 8.8.4.4 } \
> tag google rdr-to 172.17.17.1
>
>
>
>

--
*There is no place like "/home"*
*Tuco (Benedicto Pasifico Juan Maria) Ramirez*
Jordan Geoghegan
2018-08-25 07:31:22 UTC
Reply
Permalink
Raw Message
You may want to check out the more recent guides I wrote for the updated
version of these scripts:

www.geoghegan.ca/unbound-adblock.html

www.geoghegan.ca/pfbadhost.html


On 08/24/18 06:32, jin&hitman&Barracuda wrote:
> Hello
>
> Thanks for sharing all those informations. I've been looking a way to
> create a blacklist and you sent this mail just on time. Your web page
> help me a lot.
> On the OpenBSD your script do all jobs but on linux based systems I
> wrote a shell script for update iptables rules.
>
> http://analog-radyo.blogspot.com/2018/08/dynamic-block-list-on-linux-iptables.html
>
>
> Jordan Geoghegan <***@gmail.com
> <mailto:***@gmail.com>>, 30 Ara 2017 Cmt, 01:52 tarihinde
> şunu yazdı:
>
> Hi everyone,
>
> Due to the number of people who have requested my add-blocking
> scripts,
> I figured I would also post them to @misc so anyone can easily enjoy
> network-wide bad-host/add-blocking.
>
> I won't go into detail on how to set up routing/dhcp/unbound/anchors
> etc, for that see: https://www.openbsd.org/faq/pf/example1.html
>
> I've included some example files from my an Edgerouter I have set
> up .
> They are trimmed down for brevities sake; the conf files are not
> production ready, these are merely examples.
>
> This setup is easily customizable, if you come across any other block
> lists you prefer, then they can be dropped in no problem. I chose
> to use
> solely the StevenBlack hosts file because it is a master list
> compiled
> from all the major banlists found in popular blocking products
> such as
> uBlock Origin, Addblock Plus et al. I also chose this file because
> it is
> filtered for duplicates as unbound(8) is said to struggle when
> there are
> redundancies in the blocklists, I'm told -- though I've never had
> any issue.
>
> You're going to have to read the scripts and create the
> directories the
> scripts are calling and edit the anchor macros to fit your interface
> layout (I doubt everyone here is running cnmac0 as egress) and
> also will
> have to make the scripts executable and set them to run at regular
> intervals with crontab, ideally nightly.
>
> I didn't make these scripts intelligent because I figured it was
> simpler
> to just run mkdir once rather than add extra lines to the script.
>
> I know the pf.conf is fairly long, I thought I would show an
> example of
> my prio and queing setup as an example, or conversely to see if
> anyone
> can poke any holes in it.
>
> All the relevant bits regarding the anchors and blocklists are
> found at
> the end of the pf.conf file. See below that for the anchor conf files
> we're calling as well.
>
> Hope this helps,
>
> Jordan Geoghegan
>
>
> First, the scripts:
>
> *DNS addblock script:*
>
> StevenBlack.sh:
>
> cd /var/unbound/etc/banlist && \
> ftp
> https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts && \
> cat hosts | grep '^0\.0\.0\.0' | awk '{print "local-zone: \""$2"\"
> redirect\nlocal-data: \""$2" A 0.0.0.0\""}' > ads.conf
> rcctl reload unbound
>
> ###
>
> *IP based malicious IP blocking:*
>
> banlist.sh:
>
> cd /etc/blocklist && ftp https://www.binarydefense.com/banlist.txt\
> && <https://www.binarydefense.com/banlist.txt%5C&&> ftp
> https://rules.emergingthreats.net/blockrules/compromised-ips.txt\
> &&
> <https://rules.emergingthreats.net/blockrules/compromised-ips.txt%5C&&>
> ftp https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt\
> &&
> <https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt%5C&&>
> ftp
> https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level3.netset\
> &&
> <https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level3.netset%5C&&>
> pfctl -a banlist -f /etc/banlist.conf
>
> ###
>
> As you can see, we are going to have to make an anchor in pf called
> 'banlist' and modify the unbound.conf to load our banlist 'ads.conf'
>
> If that's all you need, then you're pretty much good to go. If you
> would
> like to see my example conf files, see below.
>
> *
>
>
> Example unbound.conf:*
>
> # $OpenBSD: unbound.conf,v 1.7 2016/03/30 01:41:25 sthen Exp $
>
> server:
>         interface: 172.17.17.1
>         interface: 127.0.0.1
>         access-control: 172.17.17.0/24 <http://172.17.17.0/24> allow
>         access-control: 172.17.0.0/24 <http://172.17.0.0/24> allow
>         do-not-query-localhost: no
>         hide-identity: yes
>         hide-version: yes
>         include: /var/unbound/etc/banlist/ads.conf
>
> forward-zone:
>         name: "."
>         forward-addr: UR.DNS.GO.HERE
>         forward-addr: UR.DNS.GO.HERE
>
> ###
>
>
> *Example pf.conf:*
>
> #       $OpenBSD: pf.conf,v 1.54 2014/08/23 05:49:42 deraadt Exp $
> #
> # See pf.conf(5) and /etc/examples/pf.conf
>
> # By default, do not permit remote connections to X11
> block return in on ! lo0 proto tcp to port 6000:6010
> #
> ext_if="{ cnmac0 }"
> int_if="{ cnmac1 cnmac2 }"
> lan_if="{ cnmac1 }"
> wifi_if="{ cnmac2 }"
> goodguys="{ 172.17.17.0/24 <http://172.17.17.0/24> }"
> wifiguys="{ 172.17.0.0/24 <http://172.17.0.0/24> }"
> chromecast="{ 172.17.0.12 172.17.0.13 172.17.0.23 }"
> xbox360="{ 172.17.0.19 }"
> printer="{ 172.17.0.17 }"
> Jordan="{ XXX.XX.XXX.XX }"
>
> table <martians> { 0.0.0.0/8 <http://0.0.0.0/8> 10.0.0.0/8
> <http://10.0.0.0/8> 127.0.0.0/8 <http://127.0.0.0/8>
> 169.254.0.0/16 <http://169.254.0.0/16>     \
> 172.16.0.0/12 <http://172.16.0.0/12> 192.0.0.0/24
> <http://192.0.0.0/24> 192.0.2.0/24 <http://192.0.2.0/24>
> 224.0.0.0/3 <http://224.0.0.0/3>  \
> 192.168.0.0/16 <http://192.168.0.0/16> 198.18.0.0/15
> <http://198.18.0.0/15> 198.51.100.0/24 <http://198.51.100.0/24> \
> 203.0.113.0/24 <http://203.0.113.0/24> }
>
>
> # Queue List [ Download ]
> queue download on cnmac2 bandwidth 70M max 70M
> queue media-down parent download bandwidth 20M min 5M max 20M
> burst 24M for 200ms
> queue xbox-down parent media-down bandwidth 4M max 4M burst 8M for
> 200ms
> queue chrome-down parent media-down bandwidth 16M max 16M burst
> 20M for 225ms
> queue std-down parent download bandwidth 50M min 5M max 50M burst
> 70M for 500ms default
>
>
> set block-policy drop
> set loginterface egress
> set skip on lo0
> match in all scrub (no-df random-id max-mss 1440)
> match out on egress inet from !(egress:network) to any nat-to
> (egress:0)
> block in quick on egress from <martians> to any
> block return out quick on egress from any to <martians>
> block quick inet6
> block all
>
> # A bit of edgy prio and bandwidth queuing, I felt like taking pf
> out for a test drive here
>
> pass in on $lan_if from $goodguys tag LAN set prio 6
> pass in on $wifi_if from $wifiguys tag WIFI modulate state set
> queue std-down
> pass in on $wifi_if from $chromecast tag CHROME modulate state set
> prio 2 \
> set queue chrome-down
> block out on $lan_if tagged WIFI
> block out on $lan_if tagged CHROME
> antispoof for { egress cnmac0 cnmac1 cnmac2 lo0 }
> pass in quick on $ext_if from $Jordan to any tag Jordan
> block in on $ext_if proto { tcp udp } from any to any port ssh !
> tagged Jordan
> pass out on $ext_if inet
>
>
> # Printers Ruleset      | Block Printer on Egress && allow
> $goodguys subnet
> block out on $ext_if from $printer to any
> pass out quick on $wifi_if from $goodguys to $printer
>
> # Spammers
> anchor banlist
> load anchor banlist from "/etc/banlist.conf"
>
> # DNS Redirect
> anchor dns
> load anchor dns from "/etc/dns-redirect.conf"
>
>
> ###
>
> *Anchor banlist.conf:*
>
>
> #   $OpenBSD: pf.conf,v 1.54 2014/08/23 05:49:42 deraadt Exp $
> #
> ## Spammers ##
>
> table <banlist> persist file "/etc/blocklist/banlist.txt"\
> file "/etc/blocklist/compromised-ips.txt"\
> file "/etc/blocklist/emerging-Block-IPs.txt"\
> file "/etc/blocklist/firehol_level3.netset"
> block in on egress from <banlist> to any
> block out log on egress from any to <banlists>
>
>
> ####
>
> *Anchor  dns-redirect.conf:***
>
>
> #   $OpenBSD: pf.conf,v 1.54 2014/08/23 05:49:42 deraadt Exp $
> #
>
> wifi_lan="{ cnmac2 }"
>
> # DNS Redirect
> pass in on $wifi_lan proto { tcp udp } from any to \
> { 8.8.8.8 8.8.4.4 208.67.222.222 208.67.220.220 209.244.0.3 } port
> 53 \
> tag google rdr-to 172.17.17.1
>
> # I added this because several devices were aggressively pinging
> 8.8.8.8 on my network and it was annoying me
> pass in on $wifi_lan from any to \
> { 8.8.8.8 8.8.4.4  } \
> tag google rdr-to 172.17.17.1
>
>
>
>
>
> --
> /There is no place like "/home"/
> /Tuco (Benedicto Pasifico Juan Maria) Ramirez/
jin&hitman&Barracuda
2018-08-25 09:33:13 UTC
Reply
Permalink
Raw Message
Thanks Jordan, i will look at those links.

On Sat, 25 Aug 2018, 10:31 Jordan Geoghegan, <***@gmail.com> wrote:

> You may want to check out the more recent guides I wrote for the updated
> version of these scripts:
>
> www.geoghegan.ca/unbound-adblock.html
>
> www.geoghegan.ca/pfbadhost.html
>
>
> On 08/24/18 06:32, jin&hitman&Barracuda wrote:
> > Hello
> >
> > Thanks for sharing all those informations. I've been looking a way to
> > create a blacklist and you sent this mail just on time. Your web page
> > help me a lot.
> > On the OpenBSD your script do all jobs but on linux based systems I
> > wrote a shell script for update iptables rules.
> >
> >
> http://analog-radyo.blogspot.com/2018/08/dynamic-block-list-on-linux-iptables.html
> >
> >
> > Jordan Geoghegan <***@gmail.com
> > <mailto:***@gmail.com>>, 30 Ara 2017 Cmt, 01:52 tarihinde
> > şunu yazdı:
> >
> > Hi everyone,
> >
> > Due to the number of people who have requested my add-blocking
> > scripts,
> > I figured I would also post them to @misc so anyone can easily enjoy
> > network-wide bad-host/add-blocking.
> >
> > I won't go into detail on how to set up routing/dhcp/unbound/anchors
> > etc, for that see: https://www.openbsd.org/faq/pf/example1.html
> >
> > I've included some example files from my an Edgerouter I have set
> > up .
> > They are trimmed down for brevities sake; the conf files are not
> > production ready, these are merely examples.
> >
> > This setup is easily customizable, if you come across any other block
> > lists you prefer, then they can be dropped in no problem. I chose
> > to use
> > solely the StevenBlack hosts file because it is a master list
> > compiled
> > from all the major banlists found in popular blocking products
> > such as
> > uBlock Origin, Addblock Plus et al. I also chose this file because
> > it is
> > filtered for duplicates as unbound(8) is said to struggle when
> > there are
> > redundancies in the blocklists, I'm told -- though I've never had
> > any issue.
> >
> > You're going to have to read the scripts and create the
> > directories the
> > scripts are calling and edit the anchor macros to fit your interface
> > layout (I doubt everyone here is running cnmac0 as egress) and
> > also will
> > have to make the scripts executable and set them to run at regular
> > intervals with crontab, ideally nightly.
> >
> > I didn't make these scripts intelligent because I figured it was
> > simpler
> > to just run mkdir once rather than add extra lines to the script.
> >
> > I know the pf.conf is fairly long, I thought I would show an
> > example of
> > my prio and queing setup as an example, or conversely to see if
> > anyone
> > can poke any holes in it.
> >
> > All the relevant bits regarding the anchors and blocklists are
> > found at
> > the end of the pf.conf file. See below that for the anchor conf files
> > we're calling as well.
> >
> > Hope this helps,
> >
> > Jordan Geoghegan
> >
> >
> > First, the scripts:
> >
> > *DNS addblock script:*
> >
> > StevenBlack.sh:
> >
> > cd /var/unbound/etc/banlist && \
> > ftp
> > https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts &&
> \
> > cat hosts | grep '^0\.0\.0\.0' | awk '{print "local-zone: \""$2"\"
> > redirect\nlocal-data: \""$2" A 0.0.0.0\""}' > ads.conf
> > rcctl reload unbound
> >
> > ###
> >
> > *IP based malicious IP blocking:*
> >
> > banlist.sh:
> >
> > cd /etc/blocklist && ftp https://www.binarydefense.com/banlist.txt\
> > && <https://www.binarydefense.com/banlist.txt%5C&&> ftp
> > https://rules.emergingthreats.net/blockrules/compromised-ips.txt\
> > &&
> > <
> https://rules.emergingthreats.net/blockrules/compromised-ips.txt%5C&&>
> > ftp
> https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt\
> > &&
> > <
> https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt%5C&&>
> > ftp
> >
> https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level3.netset\
> > &&
> > <
> https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level3.netset%5C&&
> >
> > pfctl -a banlist -f /etc/banlist.conf
> >
> > ###
> >
> > As you can see, we are going to have to make an anchor in pf called
> > 'banlist' and modify the unbound.conf to load our banlist 'ads.conf'
> >
> > If that's all you need, then you're pretty much good to go. If you
> > would
> > like to see my example conf files, see below.
> >
> > *
> >
> >
> > Example unbound.conf:*
> >
> > # $OpenBSD: unbound.conf,v 1.7 2016/03/30 01:41:25 sthen Exp $
> >
> > server:
> > interface: 172.17.17.1
> > interface: 127.0.0.1
> > access-control: 172.17.17.0/24 <http://172.17.17.0/24> allow
> > access-control: 172.17.0.0/24 <http://172.17.0.0/24> allow
> > do-not-query-localhost: no
> > hide-identity: yes
> > hide-version: yes
> > include: /var/unbound/etc/banlist/ads.conf
> >
> > forward-zone:
> > name: "."
> > forward-addr: UR.DNS.GO.HERE
> > forward-addr: UR.DNS.GO.HERE
> >
> > ###
> >
> >
> > *Example pf.conf:*
> >
> > # $OpenBSD: pf.conf,v 1.54 2014/08/23 05:49:42 deraadt Exp $
> > #
> > # See pf.conf(5) and /etc/examples/pf.conf
> >
> > # By default, do not permit remote connections to X11
> > block return in on ! lo0 proto tcp to port 6000:6010
> > #
> > ext_if="{ cnmac0 }"
> > int_if="{ cnmac1 cnmac2 }"
> > lan_if="{ cnmac1 }"
> > wifi_if="{ cnmac2 }"
> > goodguys="{ 172.17.17.0/24 <http://172.17.17.0/24> }"
> > wifiguys="{ 172.17.0.0/24 <http://172.17.0.0/24> }"
> > chromecast="{ 172.17.0.12 172.17.0.13 172.17.0.23 }"
> > xbox360="{ 172.17.0.19 }"
> > printer="{ 172.17.0.17 }"
> > Jordan="{ XXX.XX.XXX.XX }"
> >
> > table <martians> { 0.0.0.0/8 <http://0.0.0.0/8> 10.0.0.0/8
> > <http://10.0.0.0/8> 127.0.0.0/8 <http://127.0.0.0/8>
> > 169.254.0.0/16 <http://169.254.0.0/16> \
> > 172.16.0.0/12 <http://172.16.0.0/12> 192.0.0.0/24
> > <http://192.0.0.0/24> 192.0.2.0/24 <http://192.0.2.0/24>
> > 224.0.0.0/3 <http://224.0.0.0/3> \
> > 192.168.0.0/16 <http://192.168.0.0/16> 198.18.0.0/15
> > <http://198.18.0.0/15> 198.51.100.0/24 <http://198.51.100.0/24> \
> > 203.0.113.0/24 <http://203.0.113.0/24> }
> >
> >
> > # Queue List [ Download ]
> > queue download on cnmac2 bandwidth 70M max 70M
> > queue media-down parent download bandwidth 20M min 5M max 20M
> > burst 24M for 200ms
> > queue xbox-down parent media-down bandwidth 4M max 4M burst 8M for
> > 200ms
> > queue chrome-down parent media-down bandwidth 16M max 16M burst
> > 20M for 225ms
> > queue std-down parent download bandwidth 50M min 5M max 50M burst
> > 70M for 500ms default
> >
> >
> > set block-policy drop
> > set loginterface egress
> > set skip on lo0
> > match in all scrub (no-df random-id max-mss 1440)
> > match out on egress inet from !(egress:network) to any nat-to
> > (egress:0)
> > block in quick on egress from <martians> to any
> > block return out quick on egress from any to <martians>
> > block quick inet6
> > block all
> >
> > # A bit of edgy prio and bandwidth queuing, I felt like taking pf
> > out for a test drive here
> >
> > pass in on $lan_if from $goodguys tag LAN set prio 6
> > pass in on $wifi_if from $wifiguys tag WIFI modulate state set
> > queue std-down
> > pass in on $wifi_if from $chromecast tag CHROME modulate state set
> > prio 2 \
> > set queue chrome-down
> > block out on $lan_if tagged WIFI
> > block out on $lan_if tagged CHROME
> > antispoof for { egress cnmac0 cnmac1 cnmac2 lo0 }
> > pass in quick on $ext_if from $Jordan to any tag Jordan
> > block in on $ext_if proto { tcp udp } from any to any port ssh !
> > tagged Jordan
> > pass out on $ext_if inet
> >
> >
> > # Printers Ruleset | Block Printer on Egress && allow
> > $goodguys subnet
> > block out on $ext_if from $printer to any
> > pass out quick on $wifi_if from $goodguys to $printer
> >
> > # Spammers
> > anchor banlist
> > load anchor banlist from "/etc/banlist.conf"
> >
> > # DNS Redirect
> > anchor dns
> > load anchor dns from "/etc/dns-redirect.conf"
> >
> >
> > ###
> >
> > *Anchor banlist.conf:*
> >
> >
> > # $OpenBSD: pf.conf,v 1.54 2014/08/23 05:49:42 deraadt Exp $
> > #
> > ## Spammers ##
> >
> > table <banlist> persist file "/etc/blocklist/banlist.txt"\
> > file "/etc/blocklist/compromised-ips.txt"\
> > file "/etc/blocklist/emerging-Block-IPs.txt"\
> > file "/etc/blocklist/firehol_level3.netset"
> > block in on egress from <banlist> to any
> > block out log on egress from any to <banlists>
> >
> >
> > ####
> >
> > *Anchor dns-redirect.conf:***
> >
> >
> > # $OpenBSD: pf.conf,v 1.54 2014/08/23 05:49:42 deraadt Exp $
> > #
> >
> > wifi_lan="{ cnmac2 }"
> >
> > # DNS Redirect
> > pass in on $wifi_lan proto { tcp udp } from any to \
> > { 8.8.8.8 8.8.4.4 208.67.222.222 208.67.220.220 209.244.0.3 } port
> > 53 \
> > tag google rdr-to 172.17.17.1
> >
> > # I added this because several devices were aggressively pinging
> > 8.8.8.8 on my network and it was annoying me
> > pass in on $wifi_lan from any to \
> > { 8.8.8.8 8.8.4.4 } \
> > tag google rdr-to 172.17.17.1
> >
> >
> >
> >
> >
> > --
> > /There is no place like "/home"/
> > /Tuco (Benedicto Pasifico Juan Maria) Ramirez/
>
>
Loading...