Sparc64 LDOM not working past OpenBSD 6.5

[Ax0n]

I have a SunFire T2000 that I originally installed 6.1 on. I set up LDOMs
way back in May 2017. I kept all of the domains up to date until OpenBSD
6.6. After that, LDOMs would no longer work. The system would not boot
unless I reverted back to the single domain default using
bootmode config="factory-default"

I kind of just forgot about the machine until 6.7 came out. I upgraded, and
got the same errors upon trying to boot. I re-generated the LDOM config as
outlined in this blog post I wrote:

http://www.h-i-r.net/2017/05/logical-domains-on-sunfire-t2000-with.html

That is, I dumped the factory-default config, used it as a template for the
new LDOM configuration, edited a config file, applied the config to the
directory and used ldomctl download to apply the LDOM config before
resetting the system.

Specifically, the errors I get now (and yes, some are repeats, but it's ALL
I get from the console while booting) are:

ERROR: /***@780: Invalid hypervisor argument(s). function: b4
ERROR: /***@780: Invalid hypervisor argument(s). function: b4
ERROR: /***@780: Invalid hypervisor argument(s). function: b5
WARNING: /***@7c0/***@0/***@1/network: Missing network-vpd MD node
WARNING: /***@7c0/***@0/***@1/network: Missing network-vpd MD node

And after that, the system hangs and I must exit to the ALOM system
controller prompt to do anything further, such as revert the configuration
and reset to make the system able to boot again.

I searched and found one other person with this problem a while back ago,
but no resolution. I have hardware right here in front of me and I'm not
afraid to run -CURRENT and/or test patches to help. I am also willing to
provide remote SSH access to the system controller if someone wants to hack
on the hardware directly if it would help, though I think there are a few
LDOM-capable sparc64 machines in developers' hands already.

dmesg:
console is /virtual-***@100/***@1
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
Copyright (c) 1995-2021 OpenBSD. All rights reserved.
https://www.OpenBSD.org

OpenBSD 6.9 (GENERIC.MP) #794: Sun Apr 18 12:34:31 MDT 2021
***@sparc64.openbsd.org:/usr/src/sys/arch/sparc64/compile/GENERIC.MP
real mem = 34225520640 (32640MB)
avail mem = 33608228864 (32051MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root: Sun Fire T200
cpu0 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu1 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu2 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu3 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu4 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu5 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu6 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu7 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu8 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu9 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu10 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu11 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu12 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu13 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu14 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu15 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu16 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu17 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu18 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu19 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu20 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu21 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu22 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu23 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu24 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu25 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu26 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu27 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu28 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu29 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu30 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu31 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
vbus0 at mainbus0
"flashprom" at vbus0 not configured
cbus0 at vbus0
vldc0 at cbus0
vldcp0 at vldc0 chan 0x0: ivec 0x0, 0x1 channel "hvctl"
"ldom-primary" at vldc0 chan 0x1 not configured
"fmactl" at vldc0 chan 0x3 not configured
vldc1 at cbus0
"ldmfma" at vldc1 chan 0x4 not configured
vldc2 at cbus0
vldcp1 at vldc2 chan 0x14: ivec 0x28, 0x29 channel "spds"
"system-management" at vldc2 chan 0xd not configured
vcons0 at vbus0: ivec 0x111: console
vrtc0 at vbus0
"fma" at vbus0 not configured
"sunvts" at vbus0 not configured
"sunmc" at vbus0 not configured
"explorer" at vbus0 not configured
"led" at vbus0 not configured
"flashupdate" at vbus0 not configured
"ncp" at vbus0 not configured
vpci0 at mainbus0: bus 2 to 7, dvma map 80000000-ffffffff
pci0 at vpci0
ppb0 at pci0 dev 0 function 0 "PLX PEX 8532" rev 0xbc
pci1 at ppb0 bus 3
ppb1 at pci1 dev 1 function 0 "PLX PEX 8532" rev 0xbc
pci2 at ppb1 bus 4
em0 at pci2 dev 0 function 0 "Intel 82571EB" rev 0x06: ivec 0x795, address
00:14:4f:9d:60:ac
em1 at pci2 dev 0 function 1 "Intel 82571EB" rev 0x06: ivec 0x796, address
00:14:4f:9d:60:ad
ppb2 at pci1 dev 2 function 0 "PLX PEX 8532" rev 0xbc
pci3 at ppb2 bus 5
ppb3 at pci1 dev 8 function 0 "PLX PEX 8532" rev 0xbc: msi
pci4 at ppb3 bus 6
ppb4 at pci1 dev 9 function 0 "PLX PEX 8532" rev 0xbc
pci5 at ppb4 bus 7
mpi0 at pci5 dev 0 function 0 "Symbios Logic SAS1064E" rev 0x02: msi
mpi0: UNUSED, firmware 1.9.0.0
scsibus1 at mpi0: 63 targets
sym0 at scsibus1 targ 0 lun 0: <SEAGATE, ST914602SSUN146G, 0400>
naa.5000c500061f070b
sd0 at scsibus0 targ 0 lun 0: <SEAGATE, ST914602SSUN146G, 0400>
naa.5000c500061f070b
sd0: 140009MB, 512 bytes/sector, 286739329 sectors
sym1 at scsibus1 targ 1 lun 0: <SEAGATE, ST914602SSUN146G, 0400>
naa.5000c500061f0d3b
sd1 at scsibus0 targ 1 lun 0: <SEAGATE, ST914602SSUN146G, 0400>
naa.5000c500061f0d3b
sd1: 140009MB, 512 bytes/sector, 286739329 sectors
sd2 at scsibus1 targ 2 lun 0: <HITACHI, H101414SCSUN146G, SA23>
naa.5000cca0000c2b34
sd2: 140009MB, 512 bytes/sector, 286739329 sectors
sym2 at scsibus1 targ 3 lun 0: <SEAGATE, ST914602SSUN146G, 0400>
naa.5000c5000546b3ef
sd3 at scsibus0 targ 2 lun 0: <SEAGATE, ST914602SSUN146G, 0400>
naa.5000c5000546b3ef
sd3: 140009MB, 512 bytes/sector, 286739329 sectors
vpci1 at mainbus0: bus 2 to 9, dvma map 80000000-ffffffff
pci6 at vpci1
ppb5 at pci6 dev 0 function 0 "PLX PEX 8532" rev 0xbc
pci7 at ppb5 bus 3
ppb6 at pci7 dev 1 function 0 "PLX PEX 8532" rev 0xbc
pci8 at ppb6 bus 4
ppb7 at pci8 dev 0 function 0 "Intel 41210 PCIE-PCIX" rev 0x09
pci9 at ppb7 bus 5
ebus0 at pci9 dev 2 function 0 "Acer Labs M1533 ISA" rev 0x00
com0 at ebus0 addr 3f8-3ff ivec 0x2: ns16550a, 16 byte fifo
ohci0 at pci9 dev 5 function 0 "Acer Labs M5237 USB" rev 0x03: ivec 0x7c1,
version 1.0, legacy support
ohci1 at pci9 dev 6 function 0 "Acer Labs M5237 USB" rev 0x03: ivec 0x7c3,
version 1.0, legacy support
pciide0 at pci9 dev 8 function 0 "Acer Labs M5229 IDE" rev 0xc4: DMA,
channel 0 configured to native-PCI, channel 1 configured to native-PCI
pciide0: using ivec 0x7c4 for native-PCI interrupt
atapiscsi0 at pciide0 channel 0 drive 0
scsibus2 at atapiscsi0: 2 targets
cd0 at scsibus2 targ 0 lun 0: <TEAC, DV-28SL, 1.0A> removable
cd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
pciide0: channel 1 disabled (no drives)
usb0 at ohci0: USB revision 1.0
uhub0 at usb0 configuration 1 interface 0 "Acer Labs OHCI root hub" rev
1.00/1.00 addr 1
usb1 at ohci1: USB revision 1.0
uhub1 at usb1 configuration 1 interface 0 "Acer Labs OHCI root hub" rev
1.00/1.00 addr 1
ppb8 at pci8 dev 0 function 2 "Intel 41210 PCIE-PCIX" rev 0x09
pci10 at ppb8 bus 6
ppb9 at pci7 dev 2 function 0 "PLX PEX 8532" rev 0xbc
pci11 at ppb9 bus 7
em2 at pci11 dev 0 function 0 "Intel 82571EB" rev 0x06: ivec 0x7d6, address
00:14:4f:9d:60:ae
em3 at pci11 dev 0 function 1 "Intel 82571EB" rev 0x06: ivec 0x7d7, address
00:14:4f:9d:60:af
ppb10 at pci7 dev 8 function 0 "PLX PEX 8532" rev 0xbc: msi
pci12 at ppb10 bus 8
ppb11 at pci7 dev 9 function 0 "PLX PEX 8532" rev 0xbc: msi
pci13 at ppb11 bus 9
uhub2 at uhub1 port 1 configuration 1 interface 0 "Atmel Standard USB Hub"
rev 1.10/3.00 addr 2
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
sd4 at scsibus4 targ 1 lun 0: <OPENBSD, SR RAID 5, 006>
sd4: 280018MB, 512 bytes/sector, 573477376 sectors
bootpath: /***@780,0/***@0,0/***@9,0/***@0,0/***@0,0
root on sd0a (f8f19189c21ce1c4.a) swap on sd0b dump on sd0b

[Andrew Grillet]

I have a T1000, and it runs 6.9 in primary and 7 guests.
However, attempts to create and install a new ldom config result
in complete loss of the device tree, and consequent inability to boot.

restore to factory, and then restore the ldom config created with OBSD 6.3
will produce a working system.

This system is available and currently could be used for testing, although
not on the public internet, and only during office hours in Europe/London
timezone - machine must be shut down out of office hours.

Andrew
.

Post by Ax0n
I have a SunFire T2000 that I originally installed 6.1 on. I set up LDOMs
way back in May 2017. I kept all of the domains up to date until OpenBSD
6.6. After that, LDOMs would no longer work. The system would not boot
unless I reverted back to the single domain default using
bootmode config="factory-default"
I kind of just forgot about the machine until 6.7 came out. I upgraded, and
got the same errors upon trying to boot. I re-generated the LDOM config as
http://www.h-i-r.net/2017/05/logical-domains-on-sunfire-t2000-with.html
That is, I dumped the factory-default config, used it as a template for the
new LDOM configuration, edited a config file, applied the config to the
directory and used ldomctl download to apply the LDOM config before
resetting the system.
Specifically, the errors I get now (and yes, some are repeats, but it's ALL
And after that, the system hangs and I must exit to the ALOM system
controller prompt to do anything further, such as revert the configuration
and reset to make the system able to boot again.
I searched and found one other person with this problem a while back ago,
but no resolution. I have hardware right here in front of me and I'm not
afraid to run -CURRENT and/or test patches to help. I am also willing to
provide remote SSH access to the system controller if someone wants to hack
on the hardware directly if it would help, though I think there are a few
LDOM-capable sparc64 machines in developers' hands already.
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
Copyright (c) 1995-2021 OpenBSD. All rights reserved.
https://www.OpenBSD.org
OpenBSD 6.9 (GENERIC.MP) #794: Sun Apr 18 12:34:31 MDT 2021
real mem = 34225520640 (32640MB)
avail mem = 33608228864 (32051MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root: Sun Fire T200
vbus0 at mainbus0
"flashprom" at vbus0 not configured
cbus0 at vbus0
vldc0 at cbus0
vldcp0 at vldc0 chan 0x0: ivec 0x0, 0x1 channel "hvctl"
"ldom-primary" at vldc0 chan 0x1 not configured
"fmactl" at vldc0 chan 0x3 not configured
vldc1 at cbus0
"ldmfma" at vldc1 chan 0x4 not configured
vldc2 at cbus0
vldcp1 at vldc2 chan 0x14: ivec 0x28, 0x29 channel "spds"
"system-management" at vldc2 chan 0xd not configured
vcons0 at vbus0: ivec 0x111: console
vrtc0 at vbus0
"fma" at vbus0 not configured
"sunvts" at vbus0 not configured
"sunmc" at vbus0 not configured
"explorer" at vbus0 not configured
"led" at vbus0 not configured
"flashupdate" at vbus0 not configured
"ncp" at vbus0 not configured
vpci0 at mainbus0: bus 2 to 7, dvma map 80000000-ffffffff
pci0 at vpci0
ppb0 at pci0 dev 0 function 0 "PLX PEX 8532" rev 0xbc
pci1 at ppb0 bus 3
ppb1 at pci1 dev 1 function 0 "PLX PEX 8532" rev 0xbc
pci2 at ppb1 bus 4
em0 at pci2 dev 0 function 0 "Intel 82571EB" rev 0x06: ivec 0x795, address
00:14:4f:9d:60:ac
em1 at pci2 dev 0 function 1 "Intel 82571EB" rev 0x06: ivec 0x796, address
00:14:4f:9d:60:ad
ppb2 at pci1 dev 2 function 0 "PLX PEX 8532" rev 0xbc
pci3 at ppb2 bus 5
ppb3 at pci1 dev 8 function 0 "PLX PEX 8532" rev 0xbc: msi
pci4 at ppb3 bus 6
ppb4 at pci1 dev 9 function 0 "PLX PEX 8532" rev 0xbc
pci5 at ppb4 bus 7
mpi0 at pci5 dev 0 function 0 "Symbios Logic SAS1064E" rev 0x02: msi
mpi0: UNUSED, firmware 1.9.0.0
scsibus1 at mpi0: 63 targets
sym0 at scsibus1 targ 0 lun 0: <SEAGATE, ST914602SSUN146G, 0400>
naa.5000c500061f070b
sd0 at scsibus0 targ 0 lun 0: <SEAGATE, ST914602SSUN146G, 0400>
naa.5000c500061f070b
sd0: 140009MB, 512 bytes/sector, 286739329 sectors
sym1 at scsibus1 targ 1 lun 0: <SEAGATE, ST914602SSUN146G, 0400>
naa.5000c500061f0d3b
sd1 at scsibus0 targ 1 lun 0: <SEAGATE, ST914602SSUN146G, 0400>
naa.5000c500061f0d3b
sd1: 140009MB, 512 bytes/sector, 286739329 sectors
sd2 at scsibus1 targ 2 lun 0: <HITACHI, H101414SCSUN146G, SA23>
naa.5000cca0000c2b34
sd2: 140009MB, 512 bytes/sector, 286739329 sectors
sym2 at scsibus1 targ 3 lun 0: <SEAGATE, ST914602SSUN146G, 0400>
naa.5000c5000546b3ef
sd3 at scsibus0 targ 2 lun 0: <SEAGATE, ST914602SSUN146G, 0400>
naa.5000c5000546b3ef
sd3: 140009MB, 512 bytes/sector, 286739329 sectors
vpci1 at mainbus0: bus 2 to 9, dvma map 80000000-ffffffff
pci6 at vpci1
ppb5 at pci6 dev 0 function 0 "PLX PEX 8532" rev 0xbc
pci7 at ppb5 bus 3
ppb6 at pci7 dev 1 function 0 "PLX PEX 8532" rev 0xbc
pci8 at ppb6 bus 4
ppb7 at pci8 dev 0 function 0 "Intel 41210 PCIE-PCIX" rev 0x09
pci9 at ppb7 bus 5
ebus0 at pci9 dev 2 function 0 "Acer Labs M1533 ISA" rev 0x00
com0 at ebus0 addr 3f8-3ff ivec 0x2: ns16550a, 16 byte fifo
ohci0 at pci9 dev 5 function 0 "Acer Labs M5237 USB" rev 0x03: ivec 0x7c1,
version 1.0, legacy support
ohci1 at pci9 dev 6 function 0 "Acer Labs M5237 USB" rev 0x03: ivec 0x7c3,
version 1.0, legacy support
pciide0 at pci9 dev 8 function 0 "Acer Labs M5229 IDE" rev 0xc4: DMA,
channel 0 configured to native-PCI, channel 1 configured to native-PCI
pciide0: using ivec 0x7c4 for native-PCI interrupt
atapiscsi0 at pciide0 channel 0 drive 0
scsibus2 at atapiscsi0: 2 targets
cd0 at scsibus2 targ 0 lun 0: <TEAC, DV-28SL, 1.0A> removable
cd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
pciide0: channel 1 disabled (no drives)
usb0 at ohci0: USB revision 1.0
uhub0 at usb0 configuration 1 interface 0 "Acer Labs OHCI root hub" rev
1.00/1.00 addr 1
usb1 at ohci1: USB revision 1.0
uhub1 at usb1 configuration 1 interface 0 "Acer Labs OHCI root hub" rev
1.00/1.00 addr 1
ppb8 at pci8 dev 0 function 2 "Intel 41210 PCIE-PCIX" rev 0x09
pci10 at ppb8 bus 6
ppb9 at pci7 dev 2 function 0 "PLX PEX 8532" rev 0xbc
pci11 at ppb9 bus 7
em2 at pci11 dev 0 function 0 "Intel 82571EB" rev 0x06: ivec 0x7d6, address
00:14:4f:9d:60:ae
em3 at pci11 dev 0 function 1 "Intel 82571EB" rev 0x06: ivec 0x7d7, address
00:14:4f:9d:60:af
ppb10 at pci7 dev 8 function 0 "PLX PEX 8532" rev 0xbc: msi
pci12 at ppb10 bus 8
ppb11 at pci7 dev 9 function 0 "PLX PEX 8532" rev 0xbc: msi
pci13 at ppb11 bus 9
uhub2 at uhub1 port 1 configuration 1 interface 0 "Atmel Standard USB Hub"
rev 1.10/3.00 addr 2
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
sd4 at scsibus4 targ 1 lun 0: <OPENBSD, SR RAID 5, 006>
sd4: 280018MB, 512 bytes/sector, 573477376 sectors
root on sd0a (f8f19189c21ce1c4.a) swap on sd0b dump on sd0b