After many hours of tweaking and testing a hundred different ideas, I finally gained control of the recovery partition. Most methods of tweaking it end up breaking it, so that "Find My Mac" and Filevault won't work, since they require a valid recovery partition. The few times it didn't break it, the recovery partition would still try to download the installer instead of using the full offline installer. It turns out it is actually not that difficult to change it, but it does require a few preliminary steps.
Because it is a decently long post(I wanted to be detailed as possible), I will link to it. If that is not okay, I can copy the post over here.