Android proved that the base hardware can be driven by the Linux kernel. There would need to be a much lighter GUI environment if you want reasonable battery life.
I wonder if xfce is light enough. Lol it would need to be reworked. It seems like AOSP is a better starting point than Linux.