In a typical pure inertial nav solution, you must cancel out the acceleration due to gravity by subtracting it out. If the vehicle's down vector is incorrect, this error appears as a lateral acceleration that is equal to \$\sin \theta_{error}\$ -- and for small angles is close to \$\theta_{error}\$ itself. So if the gyro has an offset, \$\theta_{error}\$ grows linearly and therefore acceleration error grows linearly (at first).
Since velocity is the integral of acceleration, and position is the integral of velocity, a constant, linear acceleration drift (1st order polynomial) will result in a quadratic velocity error (2nd order polynomial) and cubic position error (3rd order polynomial).
If all you have is a 6-DOF IMU and a model of the earth's gravity, then both the IMU and the gravity model have to be perfect.