The PS1 version definitely has its own engine, which is not just a port of the Quake 2 engine to the Playstation, but a new engine.

I can't speak on Quake, but I was a level designer on the failed effort to port Unreal to PSX.

My understanding from talking to the coders at the time was that Unreal's software renderer was a huge advantage as a starting point. They were able to reuse a lot of the portal rendering stuff as setup on the R3K cpu, but none of the rasterization. That had to go to the graphics core, which was a post setup 2D engine that in addition to the usual sprites, could do tris and quads.

We had a budget of about 3k polygons post clipping, and having two enemies on screen would burn about half of that. The other huge limit is the texture cache was tiny, so we couldn't do lightmaps. Our lightning was baked in at vertex level and it just was what it was.

There's a bit more info here: https://www.terrygreer.com/unrealpsx.html

I imagine the situation with Quake was comparable. The BSP stuff would carry right over, but I can't imagine they got lightmapping proper working at the time. They'd also need some sort of solution for overdraw, as Quake's PVS was a lot more loose than Unreal's portal clipping.

The PS1 version uses a custom engine based on technology built for the game Shadow Master, the previous title by Hammerhead Studios. It was a technical tour de force for the original PlayStation.