Pretty impressed by this:
https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/
I will not bet in this market. I will look primarily at whatever benchmarks are most prominent at the time to determine which model is better.
The model does not have to be clearly superior to gpt-4, if it's just a bit better I will resolve YES. If anyone has suggestions for less subjective resolution criteria that won't be goodharted I am open.
if the model can be stored on 10b params but takes 100000x more time at inference time, because it searches through lots of possibilities or reprompts itself a bunch of times or similar tricks to improve its output, would that still count? In 2023 it would still not be practical to run, but it would still be smaller in the ‘# of parameters‘ sense