Discussion:
pflow on PE router
Denis Fondras
2021-05-14 14:52:41 UTC
Permalink
Hello,

I used OpenBSD as a PE router on my network. The router is connected to an IX, a
transit and multiple peers with OpenBGPd.

Earlier this week, I enabled pflow(4) to track traffic usage.
Unfortunately enabling pf(4) on a edge router does not seems like a good idea.
Some peers called in to tell they notice multiple problems (ranging from what
seems MTU problem to cut in lengthy TCP sessions), deactivating pf(4)
instantaneously fixed the problem on their side, reactivating pf($) and the
problems are back.

I tried to push up the state table (I reached 300k states), to no avail.

Do you know what are the "right settings" to have pflow(4) enabled on PE router
?

Thank you in advance,
Denis
Denis Fondras
2021-05-16 17:30:47 UTC
Permalink
- does running pf(4) without pflow(4) cause issue?
Yes, the issue is linked to pf(4) being enabled.
- can you confirm you were running with pf(4) disabled prior to enabling pflow(4)?
I do confirm. I never enable pf(4) on edge routers, it bit in the past with
assymetric routing :)
- are you able to provide or indicate your pf.conf?
--- /etc/pf.conf ---
set state-defaults pflow
set limit states 1000000

pass
--- /etc/pf.conf ---
- how many pf(4) states are you seeing in # pfctl -s info ? what is the removal rate?
depending on the period of the day, it ranges from 300 to 300000.
The removal rate was 112761228.5/s when I disabled pf(4) again.
- was traffic to the pflow sink machine transiting MPLS?
No, there is no MPLS involved at all. (I guess PE was not the right word, but
edge router might have triggered Ubiquiti fans...)
- can you provide a dmesg
I upgraded this morning, problem is still the same :

OpenBSD 6.9-current (GENERIC.MP) #20: Sun May 16 00:32:45 MDT 2021
***@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 34228760576 (32643MB)
avail mem = 33175949312 (31639MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xdab19000 (51 entries)
bios0: vendor American Megatrends Inc. version "1.0c" date 06/30/2020
bios0: Supermicro AS -5019D-FTN4
acpi0 at bios0: ACPI 6.1
acpi0: sleep states S0 S5
acpi0: tables DSDT FACP APIC FPDT FIDT SSDT SPMI SSDT MCFG SSDT CRAT CDIT BERT EINJ HEST HPET SSDT UEFI SSDT WSMT
acpi0: wakeup devices S0D0(S3) S0D1(S3) S0D2(S3) S0D3(S3) S1D0(S3) S1D1(S3) S1D2(S3) S1D3(S3)
acpitimer0 at acpi0: 3579545 Hz, 32 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: AMD EPYC 3251 8-Core Processor, 2500.55 MHz, 17-01-02
cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,SHA,IBPB,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
cpu0: 64KB 64b/line 4-way I-cache, 32KB 64b/line 8-way D-cache, 512KB 64b/line 8-way L2 cache
cpu0: ITLB 64 4KB entries fully associative, 64 4MB entries fully associative
cpu0: DTLB 64 4KB entries fully associative, 64 4MB entries fully associative
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 99MHz
cpu0: mwait min=64, max=64, C-substates=1.1, IBE
cpu1 at mainbus0: apid 2 (application processor)
cpu1: AMD EPYC 3251 8-Core Processor, 2500.01 MHz, 17-01-02
cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,SHA,IBPB,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
cpu1: 64KB 64b/line 4-way I-cache, 32KB 64b/line 8-way D-cache, 512KB 64b/line 8-way L2 cache
cpu1: ITLB 64 4KB entries fully associative, 64 4MB entries fully associative
cpu1: DTLB 64 4KB entries fully associative, 64 4MB entries fully associative
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 4 (application processor)
cpu2: AMD EPYC 3251 8-Core Processor, 2500.01 MHz, 17-01-02
cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,SHA,IBPB,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
cpu2: 64KB 64b/line 4-way I-cache, 32KB 64b/line 8-way D-cache, 512KB 64b/line 8-way L2 cache
cpu2: ITLB 64 4KB entries fully associative, 64 4MB entries fully associative
cpu2: DTLB 64 4KB entries fully associative, 64 4MB entries fully associative
cpu2: smt 0, core 2, package 0
cpu3 at mainbus0: apid 6 (application processor)
cpu3: AMD EPYC 3251 8-Core Processor, 2500.01 MHz, 17-01-02
cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,SHA,IBPB,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
cpu3: 64KB 64b/line 4-way I-cache, 32KB 64b/line 8-way D-cache, 512KB 64b/line 8-way L2 cache
cpu3: ITLB 64 4KB entries fully associative, 64 4MB entries fully associative
cpu3: DTLB 64 4KB entries fully associative, 64 4MB entries fully associative
cpu3: smt 0, core 3, package 0
cpu4 at mainbus0: apid 8 (application processor)
cpu4: AMD EPYC 3251 8-Core Processor, 2500.01 MHz, 17-01-02
cpu4: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,SHA,IBPB,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
cpu4: 64KB 64b/line 4-way I-cache, 32KB 64b/line 8-way D-cache, 512KB 64b/line 8-way L2 cache
cpu4: ITLB 64 4KB entries fully associative, 64 4MB entries fully associative
cpu4: DTLB 64 4KB entries fully associative, 64 4MB entries fully associative
cpu4: smt 0, core 4, package 0
cpu5 at mainbus0: apid 10 (application processor)
cpu5: AMD EPYC 3251 8-Core Processor, 2500.01 MHz, 17-01-02
cpu5: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,SHA,IBPB,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
cpu5: 64KB 64b/line 4-way I-cache, 32KB 64b/line 8-way D-cache, 512KB 64b/line 8-way L2 cache
cpu5: ITLB 64 4KB entries fully associative, 64 4MB entries fully associative
cpu5: DTLB 64 4KB entries fully associative, 64 4MB entries fully associative
cpu5: smt 0, core 5, package 0
cpu6 at mainbus0: apid 12 (application processor)
cpu6: AMD EPYC 3251 8-Core Processor, 2500.01 MHz, 17-01-02
cpu6: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,SHA,IBPB,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
cpu6: 64KB 64b/line 4-way I-cache, 32KB 64b/line 8-way D-cache, 512KB 64b/line 8-way L2 cache
cpu6: ITLB 64 4KB entries fully associative, 64 4MB entries fully associative
cpu6: DTLB 64 4KB entries fully associative, 64 4MB entries fully associative
cpu6: smt 0, core 6, package 0
cpu7 at mainbus0: apid 14 (application processor)
cpu7: AMD EPYC 3251 8-Core Processor, 2500.01 MHz, 17-01-02
cpu7: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,SHA,IBPB,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
cpu7: 64KB 64b/line 4-way I-cache, 32KB 64b/line 8-way D-cache, 512KB 64b/line 8-way L2 cache
cpu7: ITLB 64 4KB entries fully associative, 64 4MB entries fully associative
cpu7: DTLB 64 4KB entries fully associative, 64 4MB entries fully associative
cpu7: smt 0, core 7, package 0
cpu8 at mainbus0: apid 1 (application processor)
cpu8: AMD EPYC 3251 8-Core Processor, 2500.01 MHz, 17-01-02
cpu8: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,SHA,IBPB,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
cpu8: 64KB 64b/line 4-way I-cache, 32KB 64b/line 8-way D-cache, 512KB 64b/line 8-way L2 cache
cpu8: ITLB 64 4KB entries fully associative, 64 4MB entries fully associative
cpu8: DTLB 64 4KB entries fully associative, 64 4MB entries fully associative
cpu8: smt 1, core 0, package 0
cpu9 at mainbus0: apid 3 (application processor)
cpu9: AMD EPYC 3251 8-Core Processor, 2500.01 MHz, 17-01-02
cpu9: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,SHA,IBPB,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
cpu9: 64KB 64b/line 4-way I-cache, 32KB 64b/line 8-way D-cache, 512KB 64b/line 8-way L2 cache
cpu9: ITLB 64 4KB entries fully associative, 64 4MB entries fully associative
cpu9: DTLB 64 4KB entries fully associative, 64 4MB entries fully associative
cpu9: smt 1, core 1, package 0
cpu10 at mainbus0: apid 5 (application processor)
cpu10: AMD EPYC 3251 8-Core Processor, 2500.01 MHz, 17-01-02
cpu10: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,SHA,IBPB,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
cpu10: 64KB 64b/line 4-way I-cache, 32KB 64b/line 8-way D-cache, 512KB 64b/line 8-way L2 cache
cpu10: ITLB 64 4KB entries fully associative, 64 4MB entries fully associative
cpu10: DTLB 64 4KB entries fully associative, 64 4MB entries fully associative
cpu10: smt 1, core 2, package 0
cpu11 at mainbus0: apid 7 (application processor)
cpu11: AMD EPYC 3251 8-Core Processor, 2500.01 MHz, 17-01-02
cpu11: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,SHA,IBPB,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
cpu11: 64KB 64b/line 4-way I-cache, 32KB 64b/line 8-way D-cache, 512KB 64b/line 8-way L2 cache
cpu11: ITLB 64 4KB entries fully associative, 64 4MB entries fully associative
cpu11: DTLB 64 4KB entries fully associative, 64 4MB entries fully associative
cpu11: smt 1, core 3, package 0
cpu12 at mainbus0: apid 9 (application processor)
cpu12: AMD EPYC 3251 8-Core Processor, 2500.01 MHz, 17-01-02
cpu12: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,SHA,IBPB,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
cpu12: 64KB 64b/line 4-way I-cache, 32KB 64b/line 8-way D-cache, 512KB 64b/line 8-way L2 cache
cpu12: ITLB 64 4KB entries fully associative, 64 4MB entries fully associative
cpu12: DTLB 64 4KB entries fully associative, 64 4MB entries fully associative
cpu12: smt 1, core 4, package 0
cpu13 at mainbus0: apid 11 (application processor)
cpu13: AMD EPYC 3251 8-Core Processor, 2500.01 MHz, 17-01-02
cpu13: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,SHA,IBPB,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
cpu13: 64KB 64b/line 4-way I-cache, 32KB 64b/line 8-way D-cache, 512KB 64b/line 8-way L2 cache
cpu13: ITLB 64 4KB entries fully associative, 64 4MB entries fully associative
cpu13: DTLB 64 4KB entries fully associative, 64 4MB entries fully associative
cpu13: smt 1, core 5, package 0
cpu14 at mainbus0: apid 13 (application processor)
cpu14: AMD EPYC 3251 8-Core Processor, 2500.01 MHz, 17-01-02
cpu14: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,SHA,IBPB,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
cpu14: 64KB 64b/line 4-way I-cache, 32KB 64b/line 8-way D-cache, 512KB 64b/line 8-way L2 cache
cpu14: ITLB 64 4KB entries fully associative, 64 4MB entries fully associative
cpu14: DTLB 64 4KB entries fully associative, 64 4MB entries fully associative
cpu14: smt 1, core 6, package 0
cpu15 at mainbus0: apid 15 (application processor)
cpu15: AMD EPYC 3251 8-Core Processor, 2500.01 MHz, 17-01-02
cpu15: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,SHA,IBPB,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
cpu15: 64KB 64b/line 4-way I-cache, 32KB 64b/line 8-way D-cache, 512KB 64b/line 8-way L2 cache
cpu15: ITLB 64 4KB entries fully associative, 64 4MB entries fully associative
cpu15: DTLB 64 4KB entries fully associative, 64 4MB entries fully associative
cpu15: smt 1, core 7, package 0
ioapic0 at mainbus0: apid 128 pa 0xfec00000, version 21, 24 pins, can't remap
ioapic1 at mainbus0: apid 129 pa 0xefff0000, version 21, 32 pins, can't remap
acpimcfg0 at acpi0
acpimcfg0: addr 0xf0000000, bus 0-127
acpihpet0 at acpi0: 14318180 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus -1 (D0A0)
acpiprt2 at acpi0: bus 1 (D0A2)
acpiprt3 at acpi0: bus 2 (ASTB)
acpiprt4 at acpi0: bus 3 (D0A4)
acpiprt5 at acpi0: bus 5 (D0B0)
acpiprt6 at acpi0: bus -1 (D0B1)
acpiprt7 at acpi0: bus 7 (S0D0)
acpiprt8 at acpi0: bus 8 (BR17)
acpipci0 at acpi0 PCI0: 0x00000010 0x00000011 0x00000000
acpicmos0 at acpi0
"IPI0001" at acpi0 not configured
dwiic0 at acpi0 I2CA addr 0xfedc2000/0x1000 irq 10
iic0 at dwiic0
dwiic1 at acpi0 I2CB addr 0xfedc3000/0x1000 irq 11
iic1 at dwiic1
dwiic2 at acpi0 I2CC addr 0xfedc4000/0x1000 irq 12
iic2 at dwiic2
dwiic3 at acpi0 I2CD addr 0xfedc5000/0x1000 irq 13
iic3 at dwiic3
dwiic4 at acpi0 I2CE addr 0xfedc6000/0x1000 irq 14
iic4 at dwiic4
dwiic5 at acpi0 I2CF addr 0xfedcb000/0x1000 irq 15
iic5 at dwiic5
amdgpio0 at acpi0 GPIO uid 0 addr 0xfed81500/0x300 irq 7, 184 pins
acpibtn0 at acpi0: PWRB
acpicpu0 at acpi0: C2(***@400 ***@0x814), C1(***@1 mwait), PSS
acpicpu1 at acpi0: C2(***@400 ***@0x814), C1(***@1 mwait), PSS
acpicpu2 at acpi0: C2(***@400 ***@0x814), C1(***@1 mwait), PSS
acpicpu3 at acpi0: C2(***@400 ***@0x814), C1(***@1 mwait), PSS
acpicpu4 at acpi0: C2(***@400 ***@0x814), C1(***@1 mwait), PSS
acpicpu5 at acpi0: C2(***@400 ***@0x814), C1(***@1 mwait), PSS
acpicpu6 at acpi0: C2(***@400 ***@0x814), C1(***@1 mwait), PSS
acpicpu7 at acpi0: C2(***@400 ***@0x814), C1(***@1 mwait), PSS
acpicpu8 at acpi0: C2(***@400 ***@0x814), C1(***@1 mwait), PSS
acpicpu9 at acpi0: C2(***@400 ***@0x814), C1(***@1 mwait), PSS
acpicpu10 at acpi0: C2(***@400 ***@0x814), C1(***@1 mwait), PSS
acpicpu11 at acpi0: C2(***@400 ***@0x814), C1(***@1 mwait), PSS
acpicpu12 at acpi0: C2(***@400 ***@0x814), C1(***@1 mwait), PSS
acpicpu13 at acpi0: C2(***@400 ***@0x814), C1(***@1 mwait), PSS
acpicpu14 at acpi0: C2(***@400 ***@0x814), C1(***@1 mwait), PSS
acpicpu15 at acpi0: C2(***@400 ***@0x814), C1(***@1 mwait), PSS
acpipwrres0 at acpi0: P0SA
acpipwrres1 at acpi0: P3SA
acpipwrres2 at acpi0: P0SA
acpipwrres3 at acpi0: P3SA
acpipwrres4 at acpi0: P0SA
acpipwrres5 at acpi0: P3SA
acpipwrres6 at acpi0: P0SA
acpipwrres7 at acpi0: P3SA
acpipwrres8 at acpi0: P0SA
acpipwrres9 at acpi0: P3SA
acpipwrres10 at acpi0: P0SA
acpipwrres11 at acpi0: P3SA
acpipwrres12 at acpi0: P0SA
acpipwrres13 at acpi0: P3SA
acpipwrres14 at acpi0: P0SA
acpipwrres15 at acpi0: P3SA
ipmi at mainbus0 not configured
cpu0: 2500 MHz: speeds: 2500 1800 1200 MHz
pci0 at mainbus0 bus 0
ksmn0 at pci0 dev 0 function 0 "AMD 17h Root Complex" rev 0x00
pchb0 at pci0 dev 1 function 0 "AMD 17h PCIE" rev 0x00
ppb0 at pci0 dev 1 function 3 "AMD 17h PCIE" rev 0x00: msi
pci1 at ppb0 bus 1
ppb1 at pci1 dev 0 function 0 "ASPEED Technology AST1150 PCI" rev 0x04
pci2 at ppb1 bus 2
vga1 at pci2 dev 0 function 0 "ASPEED Technology AST2000" rev 0x41
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
ppb2 at pci0 dev 1 function 5 "AMD 17h PCIE" rev 0x00: msi
pci3 at ppb2 bus 3
em0 at pci3 dev 0 function 0 "Intel I350" rev 0x01: msi, address 3c:ec:ef:47:b3:7c
em1 at pci3 dev 0 function 1 "Intel I350" rev 0x01: msi, address 3c:ec:ef:47:b3:7d
em2 at pci3 dev 0 function 2 "Intel I350" rev 0x01: msi, address 3c:ec:ef:47:b3:7e
em3 at pci3 dev 0 function 3 "Intel I350" rev 0x01: msi, address 3c:ec:ef:47:b3:7f
pchb1 at pci0 dev 2 function 0 "AMD 17h PCIE" rev 0x00
pchb2 at pci0 dev 3 function 0 "AMD 17h PCIE" rev 0x00
ppb3 at pci0 dev 3 function 1 "AMD 17h PCIE" rev 0x00: msi
pci4 at ppb3 bus 5
ixl0 at pci4 dev 0 function 0 "Intel X710 SFP+" rev 0x02: port 1, FW 6.0.48442 API 1.7, msix, 8 queues, address 00:00:00:00:01:00
ixl1 at pci4 dev 0 function 1 "Intel X710 SFP+" rev 0x02: port 0, FW 6.0.48442 API 1.7, msix, 8 queues, address 00:00:00:00:01:01
ixl2 at pci4 dev 0 function 2 "Intel X710 SFP+" rev 0x02: port 2, FW 6.0.48442 API 1.7, msix, 8 queues, address 00:00:00:00:01:02
ixl3 at pci4 dev 0 function 3 "Intel X710 SFP+" rev 0x02: port 3, FW 6.0.48442 API 1.7, msix, 8 queues, address 00:00:00:00:01:03
pchb3 at pci0 dev 4 function 0 "AMD 17h PCIE" rev 0x00
pchb4 at pci0 dev 7 function 0 "AMD 17h PCIE" rev 0x00
ppb4 at pci0 dev 7 function 1 "AMD 17h PCIE" rev 0x00
pci5 at ppb4 bus 7
vendor "AMD", unknown product 0x145a (class instrumentation unknown subclass 0x00, rev 0x00) at pci5 dev 0 function 0 not configured
ccp0 at pci5 dev 0 function 2 "AMD 17h Crypto" rev 0x00
xhci0 at pci5 dev 0 function 3 "AMD 17h xHCI" rev 0x00: msi, xHCI 1.0
usb0 at xhci0: USB revision 3.0
uhub0 at usb0 configuration 1 interface 0 "AMD xHCI root hub" rev 3.00/1.00 addr 1
pchb5 at pci0 dev 8 function 0 "AMD 17h PCIE" rev 0x00
ppb5 at pci0 dev 8 function 1 "AMD 17h PCIE" rev 0x00
pci6 at ppb5 bus 8
vendor "AMD", unknown product 0x1455 (class instrumentation unknown subclass 0x00, rev 0x00) at pci6 dev 0 function 0 not configured
ccp1 at pci6 dev 0 function 1 "AMD 17h Crypto" rev 0x00
ahci0 at pci6 dev 0 function 2 "AMD FCH AHCI" rev 0x51: msi, AHCI 1.3.1
ahci0: port 0: 6.0Gb/s
ahci0: port 2: 1.5Gb/s
scsibus1 at ahci0: 32 targets
sd0 at scsibus1 targ 0 lun 0: <ATA, C300-CTFDDAC064M, 0007> naa.500a075103008fe4
sd0: 61057MB, 512 bytes/sector, 125045424 sectors, thin
sd1 at scsibus1 targ 2 lun 0: <ATA, ST3320620AS, 3.AA> t10.ATA_ST3320620AS_3QF00ECJ
sd1: 305241MB, 512 bytes/sector, 625134827 sectors
piixpm0 at pci0 dev 20 function 0 "AMD FCH SMBus" rev 0x59: SMI
iic6 at piixpm0
iic7 at piixpm0
pcib0 at pci0 dev 20 function 3 "AMD FCH LPC" rev 0x51
pchb6 at pci0 dev 24 function 0 "AMD 17h Data Fabric" rev 0x00
pchb7 at pci0 dev 24 function 1 "AMD 17h Data Fabric" rev 0x00
pchb8 at pci0 dev 24 function 2 "AMD 17h Data Fabric" rev 0x00
pchb9 at pci0 dev 24 function 3 "AMD 17h Data Fabric" rev 0x00
pchb10 at pci0 dev 24 function 4 "AMD 17h Data Fabric" rev 0x00
pchb11 at pci0 dev 24 function 5 "AMD 17h Data Fabric" rev 0x00
pchb12 at pci0 dev 24 function 6 "AMD 17h Data Fabric" rev 0x00
pchb13 at pci0 dev 24 function 7 "AMD 17h Data Fabric" rev 0x00
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
vmm0 at mainbus0: SVM/RVI
dt: 443 probes
uhub1 at uhub0 port 1 configuration 1 interface 0 "Genesys Logic USB2.0 Hub" rev 2.00/32.98 addr 2
uhub2 at uhub0 port 2 configuration 1 interface 0 "Genesys Logic USB2.0 Hub" rev 2.00/32.98 addr 3
uhub3 at uhub2 port 3 configuration 1 interface 0 "ATEN International product 0x7000" rev 2.00/0.00 addr 4
uhidev0 at uhub3 port 1 configuration 1 interface 0 "ATEN International product 0x2419" rev 1.10/1.00 addr 5
uhidev0: iclass 3/1
ukbd0 at uhidev0: 8 variable keys, 6 key codes
wskbd0 at ukbd0: console keyboard, using wsdisplay0
uhidev1 at uhub3 port 1 configuration 1 interface 1 "ATEN International product 0x2419" rev 1.10/1.00 addr 5
uhidev1: iclass 3/1
ums0 at uhidev1: 3 buttons, Z dir
wsmouse0 at ums0 mux 0
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
root on sd0a (57f289ff8ce4b14e.a) swap on sd0b dump on sd0b
Chris Cappuccio
2021-05-28 22:30:58 UTC
Permalink
Post by Denis Fondras
Hello,
I used OpenBSD as a PE router on my network. The router is connected to an IX, a
transit and multiple peers with OpenBGPd.
Earlier this week, I enabled pflow(4) to track traffic usage.
Unfortunately enabling pf(4) on a edge router does not seems like a good idea.
Some peers called in to tell they notice multiple problems (ranging from what
seems MTU problem to cut in lengthy TCP sessions), deactivating pf(4)
instantaneously fixed the problem on their side, reactivating pf($) and the
problems are back.
I tried to push up the state table (I reached 300k states), to no avail.
Do you know what are the "right settings" to have pflow(4) enabled on PE router
?
Pflow requires pf to be enabled to create states otherwise there is nothing to
export. You could use a different flow generator tool (there is at least one
in ports) that will watch the traffic over bpf and generate flow data.

You might try "set state-defaults pflow, sloppy", also in some scenarios you
might need "set state-policy floating"

If "sloppy" fixes it, there may be some bugs to hunt.
Denis Fondras
2021-05-30 11:30:45 UTC
Permalink
Post by Chris Cappuccio
You might try "set state-defaults pflow, sloppy", also in some scenarios you
might need "set state-policy floating"
If "sloppy" fixes it, there may be some bugs to hunt.
"sloppy" seems to fix the issue. I will do more tests this week before declaring
victory :)

Thank you Chris.
Patrick Dohman
2021-05-30 13:09:48 UTC
Permalink
Post by Denis Fondras
"sloppy" seems to fix the issue. I will do more tests this week before declaring
victory :)
Thank you Chris.
Get somme ;)
Regards
Patrick
Chris Cappuccio
2021-06-01 16:48:39 UTC
Permalink
Post by Denis Fondras
"sloppy" seems to fix the issue. I will do more tests this week before declaring
victory :)
If that really works, then there could be a problem with PF sequence number tracking. Can you develop a specific sequence of events to reproduce the failures?
Stuart Henderson
2021-06-01 20:44:12 UTC
Permalink
Post by Denis Fondras
Post by Chris Cappuccio
You might try "set state-defaults pflow, sloppy", also in some scenarios you
might need "set state-policy floating"
If "sloppy" fixes it, there may be some bugs to hunt.
"sloppy" seems to fix the issue. I will do more tests this week before declaring
victory :)
Thank you Chris.
Oh watch out with sloppy. Keep an eye on your state table size.
Patrick Dohman
2021-06-03 12:19:42 UTC
Permalink
I suspect that you’ll be out of luck until TLSv1.3 is implemented.
I’ve found the same to be true with the new 10 gb sfp switches in our infrastructure which surprisingly still implement TLSv1.0 & broken CGI web server.
Regards
Patrick
Post by Stuart Henderson
Post by Denis Fondras
Post by Chris Cappuccio
You might try "set state-defaults pflow, sloppy", also in some scenarios you
might need "set state-policy floating"
If "sloppy" fixes it, there may be some bugs to hunt.
"sloppy" seems to fix the issue. I will do more tests this week before declaring
victory :)
Thank you Chris.
Oh watch out with sloppy. Keep an eye on your state table size.
Chris Cappuccio
2021-06-03 22:04:31 UTC
Permalink
Post by Stuart Henderson
Oh watch out with sloppy. Keep an eye on your state table size.
Really? Wouldn't sloppy keep the state table smaller if anything since it's tracking less specifically?

Anyways I use sloppy across four boxes that run in parallel with pfsync. There could easily be 10,000 devices behind it at any given time. I keep my state table limit at 1,000,000. It's around 300,000 during this lighter traffic period today. I had to do sloppy after moving to several boxes in parallel, I didn't notice sloppy making any significant difference?

Chris
Stuart Henderson
2021-06-04 11:43:56 UTC
Permalink
Post by Chris Cappuccio
Post by Stuart Henderson
Oh watch out with sloppy. Keep an eye on your state table size.
Really? Wouldn't sloppy keep the state table smaller if anything since it's tracking less specifically?
Anyways I use sloppy across four boxes that run in parallel with pfsync. There could easily be 10,000 devices behind it at any given time. I keep my state table limit at 1,000,000. It's around 300,000 during this lighter traffic period today. I had to do sloppy after moving to several boxes in parallel, I didn't notice sloppy making any significant difference?
Chris
The problem I had was in conjunction with synfloods. I didn't get
captures for everything to figure it out (it was in 2018 and my
network was in flames, with the full state table bgp sessions were
getting dropped / not reestablishing) but I think what happened was
this,

spoofed SYN to real server behind PF
SYN+ACK from server

and the state entry ended up as ESTABLISHED:ESTABLISHED where it
remained until the tcp.established timer expired (24h default
or 5h with "set optimization aggressive").

My "fix" was to move as much as possible to "pass XX flags any no state"
but that's clearly not going to help with what Denis would like to do.
(fwiw - I'm not doing flow monitoring regularly, but when I do it's
usually via sflow on switches instead, which solves some problems,
though it's only possible in some situations).
Patrick Dohman
2021-06-06 12:18:41 UTC
Permalink
Perhaps it has something to do with Citrix being a dinosaur.
God forbid the powers that be choose on premise unix.
Regards
Patrick
Post by Stuart Henderson
Post by Chris Cappuccio
Post by Stuart Henderson
Oh watch out with sloppy. Keep an eye on your state table size.
Really? Wouldn't sloppy keep the state table smaller if anything since it's tracking less specifically?
Anyways I use sloppy across four boxes that run in parallel with pfsync. There could easily be 10,000 devices behind it at any given time. I keep my state table limit at 1,000,000. It's around 300,000 during this lighter traffic period today. I had to do sloppy after moving to several boxes in parallel, I didn't notice sloppy making any significant difference?
Chris
The problem I had was in conjunction with synfloods. I didn't get
captures for everything to figure it out (it was in 2018 and my
network was in flames, with the full state table bgp sessions were
getting dropped / not reestablishing) but I think what happened was
this,
spoofed SYN to real server behind PF
SYN+ACK from server
and the state entry ended up as ESTABLISHED:ESTABLISHED where it
remained until the tcp.established timer expired (24h default
or 5h with "set optimization aggressive").
My "fix" was to move as much as possible to "pass XX flags any no state"
but that's clearly not going to help with what Denis would like to do.
(fwiw - I'm not doing flow monitoring regularly, but when I do it's
usually via sflow on switches instead, which solves some problems,
though it's only possible in some situations).
Stuart Henderson
2021-06-06 22:27:52 UTC
Permalink
Post by Patrick Dohman
Perhaps it has something to do with Citrix being a dinosaur.
God forbid the powers that be choose on premise unix.
Regards
Patrick
Your message doesn't appear to relate in any way to the message to which you're replying.
Post by Patrick Dohman
Post by Stuart Henderson
Post by Chris Cappuccio
Post by Stuart Henderson
Oh watch out with sloppy. Keep an eye on your state table size.
Really? Wouldn't sloppy keep the state table smaller if anything since it's tracking less specifically?
Anyways I use sloppy across four boxes that run in parallel with pfsync. There could easily be 10,000 devices behind it at any given time. I keep my state table limit at 1,000,000. It's around 300,000 during this lighter traffic period today. I had to do sloppy after moving to several boxes in parallel, I didn't notice sloppy making any significant difference?
Chris
The problem I had was in conjunction with synfloods. I didn't get
captures for everything to figure it out (it was in 2018 and my
network was in flames, with the full state table bgp sessions were
getting dropped / not reestablishing) but I think what happened was
this,
spoofed SYN to real server behind PF
SYN+ACK from server
and the state entry ended up as ESTABLISHED:ESTABLISHED where it
remained until the tcp.established timer expired (24h default
or 5h with "set optimization aggressive").
My "fix" was to move as much as possible to "pass XX flags any no state"
but that's clearly not going to help with what Denis would like to do.
(fwiw - I'm not doing flow monitoring regularly, but when I do it's
usually via sflow on switches instead, which solves some problems,
though it's only possible in some situations).
Loading...