OK, a bit to unpack here.
Some of your statements appear to be specific to ASICAP only, they are not correct in general. In general, both SER and AVI movie formats are able to hold debayered and undebayered frames, it is not specific to the file type. Firecapture for instance can create both ser and avi files in debayered and undebayered formats with 8 bit resolution. It can only create undebayered frames with 16 bit resolution. It sounds like ASICAP only allows you to create 16 bit undebayered frames in ser format, and debayered avi files in 8 bit resolution. This is unusual.
SER files include information about the bayer pattern used for the frames in the file header, avi files do not. So when Autostakkert reads a ser file, it reads the header to find out which debayer pattern to use and so always gets it right. However in the case of avi files, AS!3 has to make a guess at the bayer pattern, it usually gets it right but not always. This is not an issue, if you know the correct bayer pattern to use for your camera (and it's usually pretty easy to tell if it's wrong by the colour cast), then this can be manually selected and AS!3 will not have a problem from then on.
The best format to use for planetary imaging is to create undebayered 8 bit ser (or avi) files. Unfortunately for you it appears that ASICAP is unable to create this format, Firecapture and SharpCap can. For people using these capture programs this is the recommended format, not 8 bit debayered or 16 bit undebayered.
In terms of your camera pixel size, best practice is to match the focal ratio of your system to the pixel size of the camera. The rule of thumb is to set your focal ratio to be around 5x the pixel size of the camera in microns in good seeing, this allows you to capture the optimal resolution for the camera. So the optimal focal ratio to use with the ASI178 is around f/12, while the ASI224 is around f/19. Capturing at these focal ratio puts the same number of pixels on the target, there is no advantage in using the smaller pixels. Smaller pixels are in general less sensitive then larger ones (simply because they are sample a smaller surface area) so there should be no impact on the amount of light collected. Sampling at too high a focal ratio for the camera means that the image will be dimmer than optimal and you will not be getting any more resolution than the camera will allow. There is a mathematical "proof" of sorts on the Cloudy Nights website, backed up by better and more experienced planetary imagers than I.
https://www.cloudynights.com/topic/4...-which-barlow/
I capture Jupiter at 150fps and Saturn at 100 fps using my ASI224MC at a focal ratio for around f/21, or 5.6x the pixel size of the camera. The size of the OTA doesn't matter, it's all about the camera sensitivity and pixel size.
Andrew