In this paper, we propose a strong alternative to PEFTs, LoReFT. LoReFT achieves strong per- formance across benchmarks from four domains while being