Intitially, when I started JCodec it was easier to use int arrays since integer allows for full 0...255 range representation. This pattern stuck for too long in JCodec. Now there's plenty of code and all the encoders/decoders use 4x memory they are supposed to.
So now I am slowly starting to move classes to byte arrays. For this I will have the inputs shifted by -128 so that 0 will correspond to -128. I am assuming I will need to check in many places to make sure things are working fine.
The first component to change would be H264Encoder, since I figured this one is used the most.