DUETi - Installer for DUET UEFI on BIOS Bootloader

[SugoiDogo]

I’ve put together a small python script to make it easier to install DUET. If you don’t already know what that is, DUET is a BIOS bootloader that implements a UEFI interface. DUETi doesn’t actually include DUET, you'll find it packaged with Clover and OpenCore for legacy systems. DUETi works on all modern desktop operating systems, and operates on whatever you point it at, whether that be a system drive, raw disk image, or loop device. There are no safety checks in place, so if you don’t know anything about partition tables, wait for the GUI release. The script and technical details are at https://github.com/sugoidogo/DUETi

[Slice]

We already know all about DUET since 2011. But may be you know something new about it. Tell us!

[SugoiDogo]

56 minutes ago, Slice said:

We already know all about DUET since 2011. But may be you know something new about it. Tell us!

Not sure what you're asking here. My post is about some new software I've written, not anything new about DUET itself. Just makes it easier to get DUET installed, especially for edge cases.

[MacNB]

2 hours ago, SugoiDogo said:

I’ve put together a small python script to make it easier to install DUET. If you don’t already know what that is, DUET is a BIOS bootloader that implements a UEFI interface. DUETi doesn’t actually include DUET, you'll find it packaged with Clover and OpenCore for legacy systems. DUETi works on all modern desktop operating systems, and operates on whatever you point it at, whether that be a system drive, raw disk image, or loop device. There are no safety checks in place, so if you don’t know anything about partition tables, wait for the GUI release. The script and technical details are at https://github.com/sugoidogo/DUETi

 

Good effort but it's not really a DUET installer.

It just installs the first stage loader and the user has to manually find & copy Duet file onto the EFI and your script relies on the user supplying ALL the required files.

 

Take a look at the FULL Duet installer in OpenCore's LegayBoot utility folder which is called BootInstallBase.sh.

It does not need arguments and it prompts you to choose which drive to install it on.

If you wish create a GUI version, use BootInstallBase.sh as your base.

Other than that, you may be re-inventing the wheel.

[SugoiDogo]

17 minutes ago, MacNB said:

 

Good effort but it's not really a DUET installer.

It just installs the first stage loader and the user has to manually find & copy Duet file onto the EFI and your script relies on the user supplying ALL the required files.

 

Take a look at the FULL Duet installer in OpenCore's LegayBoot utility folder which is called BootInstallBase.sh.

It does not need arguments and it prompts you to choose which drive to install it on.

If you wish create a GUI version, use BootInstallBase.sh as your base.

Other than that, you may be re-inventing the wheel.

There's several reasons my script is an improvement over bootinstallbase.sh, but the big one is cross-platform. That script uses functions specific to osx fdisk, so like any other DUET installer I could find, only works on the one platform. For the users of this forum maybe a non-issue, but for my own purposes I needed to be able to install duet from multiple different operating systems and onto virtual, physical, and removable disks.

I'm in the middle of a move at the moment, so I decided to release it as-is and make it feature-complete later. There's several places around the web you'll find duet with os-specific install instructions or tools, including the opencore install guide, which claims it can only be installed from windows with bootice.

[Slice]

https://github.com/CloverHackyColor/CloverBootloader/wiki/Installing

[SugoiDogo]

12 minutes ago, Slice said:

https://github.com/CloverHackyColor/CloverBootloader/wiki/Installing

sorry, I'm still not sure what you're getting at. I have seen that page, and I'm aware the clover installer can also install DUET, if you're already inside OSX. I made DUETi to be as generic a tool as possible, you can use it for any version of DUET, on any operating system, and install to any device. The existing tools didn't fit my needs so I made a new one, and I'm sharing it here for those who might have need of it.

[MacNB]

24 minutes ago, SugoiDogo said:

There's several reasons my script is an improvement over bootinstallbase.sh, but the big one is cross-platform.

 

Good goal to have cross-platform installer.

I would just make sure it does NOT rely on users having to hunt for or pulldown files from REFIND, or OpenCore, etc to truly install Duet.

I.e. the GUI should just allow the User select the drive onto which to install Duet. Then your tool needs to source the required files from appropriate repositories or bundle them within the tool.

Once tested and working, suggest it to OpenCore developers to add it to the  LegacyBoot Utility folder as an additional tool.

[SugoiDogo]

5 minutes ago, MacNB said:

 

Good goal to have cross-platform installer.

I would just make sure it does NOT rely on users having to hunt for or pulldown files from REFIND, or OpenCore, etc to truly install Duet.

I.e. the GUI should just allow the User select the drive onto which to install Duet. Then your tool needs to source the required files from appropriate repositories or bundle them within the tool.

Once tested and working, suggest it to OpenCore developers to add it to the  LegacyBoot Utility folder as an additional tool.

That's the plan for the future, as well as providing a choice of opencore, clover, or refind. I'd have it stream the relevant release archives and extract the specific files needed, rather than bundling them, so that it continues to install the most up-to-date files. Past that I'd like to set up automated builds of DUET, since the refind version I can currently only find via a google drive link.

Edited January 26, 2023 by SugoiDogo

[deeveedee]

Very interesting.  I've never been able to get my Dell Latitude E6410 to UEFI boot macOS even though BIOS allows UEFI boot configuration.  Maybe the solution is to 'CLOVERX64.EFI -> /EFI/Microsoft/Boot/bootmgfw.efi'

[MacNB]

14 minutes ago, deeveedee said:

Very interesting.  I've never been able to get my Dell Latitude E6410 to UEFI boot macOS even though BIOS allows UEFI boot configuration.  Maybe the solution is to 'CLOVERX64.EFI -> /EFI/Microsoft/Boot/bootmgfw.efi'

Huh ?!? You posted in a wrong thread. Your issue is off topic in this thread.

 

[deeveedee]

@MacNB Good thing we still have someone who's policing off-topic posts.  Thanks.  I was quoting text from Slice's link https://github.com/CloverHackyColor/CloverBootloader/wiki/Installing that he posted earlier in this thread.  It may help me.  Sorry if that offends you.

[Slice]

19 hours ago, SugoiDogo said:

for any version of DUET, on any operating system, and install to any device. 

Sorry but it is a dream only. Script for Linux/macOS is not same as for Windows. Devices for Linux is not same as for Windows and even for macOS.

DUET is not a common solution it is always specific.

Clover has scripts for macOS and for Windows and explanation what to do in Linux. Just read WiKi.

To make some DUET you need a sources for it, for example sources from Clover that is ported to OpenCore too. But original DUET sources are deprecated and disappeared from EDK2 repository.

Legacy Clover including the modified DUET used by many people to boot Linux, Windows and sometime even macOS.

 

[SugoiDogo]

On 1/27/2023 at 8:30 AM, Slice said:

Sorry but it is a dream only. Script for Linux/macOS is not same as for Windows. Devices for Linux is not same as for Windows and even for macOS.

DUET is not a common solution it is always specific.

Clover has scripts for macOS and for Windows and explanation what to do in Linux. Just read WiKi.

To make some DUET you need a sources for it, for example sources from Clover that is ported to OpenCore too. But original DUET sources are deprecated and disappeared from EDK2 repository.

Legacy Clover including the modified DUET used by many people to boot Linux, Windows and sometime even macOS.

 

 

Command line scripts are OS specific. This is a python script. The python interpreter allows writing scripts that work on all 3 platforms.

 

DUETi uses file access API to open the raw block device on all platforms. The only difference is the access path.

 

DUETi *installs* duet successfully. Whether your chosen bootloader works for your platform is another matter entirely.

 

There is no dream here, it already does what I set out for it to do. From here it simply needs ease of use improvements to be able to fully replace the aforementioned scripts.

Edited February 9, 2023 by SugoiDogo

[SugoiDogo]

Finally got around to updating this. DUETi can now also pull the latest release of clover/opencore, as well as a couple EDK builds of DUET+REFIND from the win-raid forums. The optional flags allow you to provide any github repo or direct download url as a source, change the regex used to identify the chainloader files, and install the mbr and pbr chainloaders in one go.

 

GUI hasn't been added yet, I'm considering including the step of copying over the efi files from the archives but I don't know if it's possible to build a generic opencore/clover install that would work just for getting things booted into macos to run mist or the opencore patcher on the target machine. Would be very helpful for me as I've now got piles of macbooks that need fresh installs and all have different versions of macos they need. Any reference materials or help on that front would be appreciated.

[Captain Flack]

I have a suggestion for your script. Duet's partition boot record is sensitive to what kind of filesystem it is on, so it would be helpful if your script does the following if the user is trying to install Duet in a location where the Duet MBR/PBR combination is unlikely to work:—

 

1) Display a warning

 

2) Offer to install and set up grub4dos such that it can chain load Duet directly without using Duet's MBR/PBR (as described in https://www.insanelymac.com/forum/topic/359685-a-tip-for-anyone-who-wants-to-run-a-uefi-operating-system-on-a-bios-only-commuter/). This would mean either the standalone grub.exe or grldr, depending on what pre-existing capabilities are detected when your script interrogates the existing MBR & PBR (& perhaps also any grub-bios partition). For instance, if grub2 were detected, then loading Duet via a grub2 menu stanza to load grub.exe would be the sanest option, whereas grldr would make sense if the MBR contained no booting code.

 

Spoiler

 

Spoiler

 

This is a confusing subject, especially for anyone experimenting with USB sticks, because the UEFI specification seems to imply that FAT16 is obligatory for EFI system partitions on external disks, but most available 2nd-stage loaders for Duet expect something else, reflecting what the UEFI specification requires for internal disks.

 

 

[SugoiDogo]

This script was only built to install the MBR/PBR boot sectors required for a working DUET install. I'm trying to develop this script in 3 parts: the CLI, the GUI, and the actual functions. The CLI does no checks on purpose, it's meant for people who know what they're doing and just need quick access to the functions. The GUI will be a more user-friendly tool, which would include checks for things like filesystem compatibility. Detecting existing boot code isn't something I had considered, but it seems out of scope for this tool who's sole purpose is to install duet. grub-install for example isn't concerned with whether you're overwriting another bootloader, just getting itself working. But reading your linked post, using grub4dos instead of the DUET MBR/PBR would increase compatibility and allow a selection of DUET binaries, so I think that's definitely worth adding as a function. The comment left by tech128 on github also mentions RefindPlus, which appears to further increase compatibility with different hardware.  I'm considering adding a function to copy over the boot menu files since they come with the DUET binaries anyways, so if I were to go that route, adding RefindPlus to the options also seems like a good idea. But I've only used standard library functions for what I've built so far, and I'm not keen on diving into python's dependency management systems. At the very least, I want to keep the CLI functions limited to standard library so that the script can just be downloaded and run or easily packaged wherever needed. Doing things like detecting filesystems and other bootloaders or configuring them is likely going to involve adding more libraries for those functions. For example, I believe I need psutil to detect what filesystem is on a partition.

Edited July 23, 2024 by SugoiDogo
finish post

[Captain Flack]

On 7/23/2024 at 3:17 AM, SugoiDogo said:

tech128 on github also mentions RefindPlus

Yes, RefindPlus looks interesting too. I hadn't clocked that it was different from Refind until I read your reply 😀.

 

I don't know much python, but presumably there are lightweight ways to test a filesystem's identity. For instance, I'm typing this on a computer that has a FAT16 filesystem on partition on /dev/da0p1, which means that the shell one-liner

dd status=none if=/dev/da0p1 bs=512 count=1|dd status=none bs=1 skip=54 count=5
;echo

returns the string FAT16 — and you're already doing stuff in python that has same outcome as typing dd at the prompt…

 

 

(* The dd on my computer insists of reading raw devices in blocks, hence the pipeline in the code snippet above.)

Edited July 24, 2024 by Captain Flack
spelling mistake

[SugoiDogo]

After doing some testing on my system, the FAT filesystems have their name in the first sector and HFS has its name in the 3rd sector. As far as I know, these are the only supported filesystems, so that function should be easy to implement. If I'm reading everything correctly, the boot1f32 loaders specifically support only fat32 and not fat16, which is where grub4dos would come in?

[SugoiDogo]

So grub4dos appears to be documented in Chinese, which I can't read. Looking at the release archives, there's grldr, grldr.mbr, and glrdr.pbr, as well as grub.exe, but the mbr and pbr files are much larger than I would expect them to be given that the DUET mbr and pbr files are 512 bytes - one sector. Which files am I meant to install into the boot sector, and which ones go on the drive?

[SugoiDogo]

Nevermind, I found the English documentation in the grub4dos release archive. It seems like all I had to do is extend the maximum size for the mbr/pbr source read to 8kb. DUETi can now install grub4dos as well, but even though grub4dos is on github, the release archives are 7zip, which can't be read by the standard library.

[SugoiDogo]

Actually, upon further testing, it seems the pbr has to be written differently per filesystem. The Arch Wiki I got my original instructions from fails to mention this, I just destroyed an exfat partition trying to install grldr on it. The clover archive includes instructions for writing to a fat32 or an hfs partition, but not exfat, and grldr doesn't contain any instructions at all but to use their tool. So for now, DUETi can only write to fat32 formatted partitions.

[Captain Flack]

2 hours ago, SugoiDogo said:

So grub4dos appears to be documented in Chinese, which I can't read.

I can't read Chinese either, but I've tracked down a few English-language documents. Perhaps for versions before 0.4.6a, but with luck some things will stay the same:—

• http://web.archive.org/web/20160410143128/http://diddy.boot-land.net/grub4dos/Grub4dos.htm
• https://rmprepusb.com/tutorials/021-guide-to-grub4dos/
• https://rmprepusb.com/tutorials/021-guide-to-grub4dos/021b/
• https://skamilinux.hu/grub4dos/files/commands.htm

And some of Chennal's documentation is in English:—
https://github.com/chenall/grub4dos/blob/0.4.6a/README_GRUB4DOS.txt

 

2 hours ago, SugoiDogo said:

Which files am I meant to install into the boot sector, and which ones go on the drive?

I haven't tried hand-copying any of those files to a disk boot sector or to a partition boot sector. My experiments, on two separate gpt-partitioned USB sticks, went down two avenues:—

1. Using grub.exe. This file masquerades as a Linux kernel, so I was able to load it from grub2 (already on one of my USB sticks) using the grub2 linux command or linux16 command (in a stanza in grub2's grub.cfg file). According to the internet, you can feed grub.exe a menu when you launch it: either linux grub.exe --config-file=nameOfSomeMenuFile , or some actual grub4dos commands in quotes, as in linux grub.exe --config-file="find --set-root /boot7;chainloader --force --load-segment=0x2000 --load-offset=0x200 --boot-cs=0x2000 --boot-ip=0x200 /boot7" . Unfortunately, the internet is wrong, and grub.exe ignores its argument. What seems to happen is that grub.exe loads up without figuring out its own %root% variables and consequently drops to its command line. But this is just a minor hiccup, since you can (before trying to reboot) open grub.exe in any programmers' text editor and put something sensible as the first item of its built-in menu, which is towards the end of the file. You could probably replace its built-in menu completely (with just the instructions to locate and boot whatever Duet versions you've taken a shine to), but I think it's more convenient to have the built-in menu load a separate menu file (i.e. make sure the first item in the built-in menu has something along the lines of find --set-root /path/to/menuFileThatDefinitelyExists before it tries to run configfile /path/to/menuFileThatDefinitelyExists .
2. Using grldr. This is also a regular file but it does require grub4dos-specific code in the mbr or the partition bootsector (or perhaps both — I haven't checked that carefully). You need to run bootlace.com or bootlace64.com to set it up. If I remember rightly, I booted up a Fedora28 iso image that happened to be on the same USB stick and I ran bootlace64.com from that (yes, it's a Linux executable), taking care to use the grub4dos zero-based disk/partition numbering scheme specify the disk and the partition with grldr on it. Unfortunately I can't check the exact arguments right now as I don't have a Linux to hand, but I'm fairly sure there was also an argument to tell bootlace that the target device was gpt-partioned. I haven't tried running bootlace with the partition boot sector as the target (so I don't know whether that could work too, e.g. via the partition being marked active). My above observations about the built-in menu are true for grldr as well as grub.exe — you may want to fix its built-in menu.

2 hours ago, SugoiDogo said:

the boot1f32 loaders specifically support only fat32 and not fat16, which is where grub4dos would come in?

Yes, exactly. I blundered into this thicket because I have a UEFI-equipped computer that refuses to boot off FAT32 EFI System partitions on USB sticks. Moreover, small partitions sometimes get formatted as FAT16 silently & automatically if the user doesn't expressly ask for FAT32, and the newfs_msdos command on FreeBSD flatly refuses to put FAT32 on small partitions. Hence I ended up with a couple of good USB sticks for one computer and got thoroughly bemused when I tried to add Duet for using them on an older computer.

 

And, as you mentioned, it could be useful to have more than one build of Duet at your disposal owing to ill-documented differences between them. If you want a Duet-bootable external device that works on both i386 and x86_64, then I see no other way.

 

ADDENDUM: I see your point about some filesystems being less easy to identify. I've just had a look at the first few sectors of a UFS-formatted partition: it doesn't give a lot away…

Edited July 24, 2024 by Captain Flack
addendum

[Captain Flack]

1 hour ago, SugoiDogo said:

The clover archive includes instructions for writing to a fat32 or an hfs partition, but not exfat

Clover seems to have an optional 1024-byte bootsector file for exfat, but no instructions for how to modify it before dd'ing it into place.

Edited July 25, 2024 by Captain Flack
bits ≠ bytes

[SugoiDogo]

The instructions for an HFS partition from the Clover archive are rather straightforward, so I've implemented that function in my script as well.

Quote

   - Install boot1h2 to the partition's bootsector:
        sudo dd if=boot1h2 of=/dev/rdisk0s2

 

I seem to have found the origin of boot1x here on the forums, and it comes with the source code to a C program for installing it. 

Quote

installing an exFAT boot record requires a companion program because it needs to compute a checksum of the exFAT VBR.  So I made a program to install all three types of boot records.

This would be very helpful, given that this program contains the meat of my script's implementation, except that I don't know C either, so I'm unable to convert those steps for my script.