You are correct much of the gpio are handled natively.
Looking at the schematic the main reason pinMode() takes too long (I measure around 42ms) is because the muxing (IO2_MUX) is controlled by a pin connected to the Cypress on the Galileo.
I verified that there is indeed an I2C transaction by probing the SDA and SCL lines of the Cypress everytime pinMode() is called.
