Someone on the EAB board mentioned "Gamesmith Development System"
Normally I don't want to use "Frameworks" because I like to dig into it myself, but seeing I was having a hard time getting any kind of graphic performance out of my own code I decided to take a look.

I'm glad I did - it looks like it is the perfect "middleware" between high level C drawing function and the raw amiga specific stuff.
It's a library written in Assembler and C that exposes some basic graphic and audio functions to any other language that can load external libraries.
It comes with both a C and a Assemler compiler, but it also works with Sas/C

It was a commercial product but apparently it's in the public domain now.
I found the download here and the manual here.

The installation barfed on the final steps on my system, so I had to complete it manually: assign the GameSmith: path to the installation dir and rename the correct header file for use with Sas/C.

Some of the examples even are very "bunnymark" like so that looks really promising.
Gamesmith includes some "link" files you can excute, but I could not get them to work.
According to the Sas/C user guide you don't really need to do the compiling and linking in 2 steps, you can just use the sc command with the "link" parameter to do everyrthing in 1 step. 
You can use the "lib" parameter to define the include libraries.
So to compile the example it would be "sc file.c link lib GameSmith:GameSmith.lib to file"

Awesome. Gamesmith includes lot's of stuff I don't really need, I guess, but the base seems really really promising: fast and easy to integrate.

Let's give it a try.