Protocol design question - Journal of Omnifarious
Jan. 27th, 2007
11:58 am - Protocol design question
So, I'm doing some protocol design for CAKE, and I'm wondering about an issue involving very long messages, verification and MAC.
I have a message format that would allow gigantically huge messages, on the order of a tebibyte in size. It's highly unlikely that there will ever be any messages that large. But I feel that in having limits of that nature, it's better to go for overkill in the extreme.
But I also recognize that since messages might be fairly large, it might also be that they are generated on the fly without any idea of how long they will eventually be. HTTP solves this problem with chunked-transfer encoding, and I have a very similar idea. But having this means that the length of the message can't be specified fully at the start.
In previous renditions of the protocol, I decided that the last message chunk was the last chunk less than 1000 bytes. I also had a MAC at the end of each chunk that included the chunk length field in the MAC. This prevented chunks from being maliciously reordered or modified. It also prevented the message from being maliciously shortened. The receiver could easily detect if any of these things had been done and reject the message and request that the sender send it again in the hopes of getting a good copy.
But this time I would also like parties in the middle to be able to collect all the chunks together. Basically if the message consists of a 50000 byte chunk and a 3000 byte chunk and a 5 byte chunk, I would like someone in the middle to be able to modify the message to be a single 53005 byte chunk.
But, I was thinking that it might be nice to have these little checks along the way that a message was still good that were provided by the original versions MACs at the end of the chunks. The only problem is if the length of the chunk isn't included in the MAC it's possible that someone in the middle could modify the message maliciously to be shorter without the receiver being able to detect this.
I can't think of a good way to preserve both properties, and I'm not sure which properties are the most important. So that's my question to all of you. Is there a good way to preserve both things? If there isn't, which is more important? Is it more important that the receiver be able to check each chunk as it's received so it can have a good running idea of the health of the message as it's coming in, or is it more important to allow intermediaries (who may be of the store it for minutes to days variety) to be able to coalesce chunks?