The reason I build it on the Galileo is because it was easier for me. I was able to re-build on the board without having to bother with copying the program from the PC over. I expect the cross-compile toolchain should work as well. I am using Sergey's steps on building a LSB compliant image. This image includes GCC and the common libraries.
Since you are using a breadboard, I recommend wiring the Galileo IO10 line over to the touchscreen CS pin (IO8). I didn't do this because I am making hardware modifications directly on the shield and it wasn't physically possible. On a breadboard you have a lot more flexibility. The nice thing about this configuration is you don't have to modify the Clanton platform device driver. It defaults to use IO10 for the user-mode SPI support. The fewer driver modifications you need to do the less that could go wrong. When I made this driver change, I wasn't sure which platform driver the board uses (there are several for Clanton -- possibly development variants), so I just changed them all. If you can avoid having to do this, I expect it will be easier.