Hi
I am not totally sure if you have considered this option but I think you can try a customization to one of the current Linux images by adding the modules that you need, in the following discussions some users have done this for their projects.
Re: Build debian Linux distribution for Galileo board with debootstrap
Re: Building an linux image and modify the kernel - Answered
Here is also a Building a Linux image tutorial
http://www.malinov.com/Home/sergey-s-blog/intelgalileo-buildinglinuximage
Regards,