250mbps bitrate kind of works. It's strange because sometimes it gives a slow sd card error after 1 second and sometimes it works and continues working for a few minutes which leads me to believe that there's something causing this that isn't the speed of the sd card. If the sd card was really too slow the errors would be much more regular. The exact same behavior happens though much more rarely when I set the bitrate to 200mbps.
One hypothesis might be that the encoder sets the quality too high for the first seconds and then adjusts it after which might explain the higher bitrate of the 1 second files and the camera has trouble writing them fast enough.