[czurnieden]

Somebody insisted on NTT instead of FFT (no non-optional floating points) so I did.
Tried van der Hoevens TFT first, looked really good but had a cutoff/threshold point near 2^30 bits. If we change the lengths to size_t instead of int I can dust it off if you want.
Normal radix-2 NTT was too "steppy", so I went to the middle-ground a mixed radix 2,3 NTT. Still too "steppy". An addition of a radix-5 butterfly was necessary. Which, I found out, is totally different from the 2 and 3 radices. So no radix 7 butterflies from my side.

The final result a 2, 3, 5 mixed radix NTT with Mongomery division and CRT (Garner) with three primes for 60-bit and four primes for 28 bit limbs. Works technically but probably needs some work to adhere to the LTM style. And whatever travis complains about.

MP_64BIT with 60 bit limbs and MP_32BIT with 28 bit limbs work, MP_16BIT could be made to work but the size of an int and therefore the maximum number size is below the NTT thresholds. MP_32BIT with 31 bit limbs does not work as it is organized now (we take all bits of the limb, we do not need to slice them. 31 bit would be too much for the current way NTT is implemented and slicing 31 bits would not only be expensive but also complicated because the slicing would be unbalanced).

Here are some benchmarks (all with MP_64BIT on an i7-2600 @ 3.40GHz and 24 Gibibytes of RAM, only some of the benchmarks have been run under cset isolation. Has not much influence to timing but the noise level is higher and the peaks of 5-smooth input are less visible. All benchmarks are against the full Toom-Cook (2-7) pipeline if not noted otherwise. Base multiplication is always in column 2 unless stated differently. The variable k is the size of the input. The x-axis shows the number of limbs of the first input, the y-axis shows time in seconds.)
My apologies for the botched titles.

NTT multiplication with weights 1,2,6 against Toom-Cook 2-7 pipeline.

NTT squaring with weights 1,2,6 against Toom-Cook 2-7 pipeline and Toom-Cook 2-3 (develop) pipeline.

The three different butterflies have different runtimes, so the computed length of the input arrays sets different weights for the different radices. The difference is visible.

Influence of different weights.

The NTT algorithms works much better with unbalanced input. Our balancer only wins (that is, NTT is at the normal cutoff point) if the difference is small (k + k/100). Once the difference gets larger (k + k/10) NTT wins. That behavior is stable for all nk + k/m with n >= 1 and m >= ~10. Calculating those cutoffs is not yet implemented, but tune.c would have the means to do it.

NTT multiplication k + k/100

NTT multiplication k + k/10

NTT multiplication k + k/2

NTT multiplication 2k - k/100

NTT multiplication 2k - k/10

And with a larger difference:

NTT multiplication 7k

NTT multiplication 7k - k/10