Smart neural network optimization

We wanted to quantize and prune our own models for Tinyfy. What we found is that if we look online, there are a lot of tools you can use to reduce the size and increase the speed of the neural network but everything is quite complicated, you need probably a few days to really read up on what parameters to use, how does everything work and what this will do to the network. What we wanted is something that you can just say, put in your network into this part of the code and then it detects everything automatically and just makes the correct choices for you. So that’s the edge AIminify has over other libraries.  

And concerning what's happened in the recent years, obviously because AI is growing in hyper speed, also a lot is happening in the compression world. I think in terms of quantization, the most interesting thing is that especially for large language models, like ChatGPT and Lama, they are now doing 4-bit quantization, even 2-bit quantization! They quantize to only two bits and still the network is working. You can see that if you go from 32 bits to 2 bits, that's a huge size reduction. This is really interesting and I'm curious to see how small they can go.

When we started with a AIminify we had Tinify in the background, and with Tinify you can just upload your image, we compress it and give it back to you. So, when we started to talk to people, we asked, "Give us your model and then we will compress it and give it back to you". And everybody said that's a secret of the company! Although there were some exceptions, most of them were like "You're not going to touch our model". So, we needed to find a solution for that. We now have an on-premise solution. We run the compression and use your training data on your infrastructure. So please do try it and give us your feedback, because every time we run it somewhere we get some more new information. If something is not working, we need to setup an extra setting and this way we broaden our scope of what kind of models we can handle. So that's also very helpful for us.

No, in the case of AIminify, it is super dependent on network, and we also haven't used it enough on a lot of different neural networks to be confident in what we estimate yet. For the quantization it is easy, because we know beforehand if it saves ~50 or ~75% storage. But for the latency improvement, that comes from the pruning, and can differ from 10x speed increase to 2x speed increase. Fow now, hard to tell why sometimes the speed increases more than other times. This is something we are currently investigating. 

We have tried it on smaller models like VGG and we've seen ~75% size decrease and 3x to 4x latency improvement. On those it works pretty well. We tried lot of open source models to at least be sure that that we can handle those. However, when people used these models, they may need to tweak it a little, or add extra layers and do something to run in their own environments. In such cases the AI tries to prune layers that shouldn't be pruned. So, whenever we get a new model, at least for now, we need to look at it and see if is there anything that we haven't seen before and assess how should we handle that.

If all of you start using it and give us new feedback on your case studies about what works and doesn't work, it will help us speed up the improvement of the product. The more the product is used, the more answers we have.  

We're trying to keep the number of parameters that the user can use as low as possible, because we don't want to complicate it where the user has to put a lot of parameters in and doesn't really understand what it's doing. But we do want to put some easy stuff in there like, "What kind of CPU are you using? What kind of GPU?". Just for the algorithm to be sure to optimize it in the correct way. For some companies, we already noticed that they for example run their known networks on both CPUs and GPUs depending on their customers. So, we want to optimize in a way that they can run multiple different versions of their network at the same time on different hardware. 

That depends a bit on how good your teacher model is. You hope that your Teacher model is already good enough that the Student model just wants to impersonate. So yeah, you do overfit on your teacher model, but that's kind of what you want to do. You just want to make a smaller model that has outputs similar to your teacher model. If your Teacher model has some errors in it, then probably your Student model is also going to have those errors. It's not as much an improvement. It's just an improvement in size and speed. 

I think we would be interested in that. For now, the obvious thing is that if the network does speed up, then there is less GPU hours to use, which saves a lot of energy. We've been talking to people building CPUs, they don't seem much interested in it as then they sell less hardware. However, in the GPU world, it's kind of different because it's more money intense. They are more interested in compressing, also for their end users. I think in five years from now, this will be more seriously considered in the industry.