Interesting design problem with serialization and deserialization - Journal of Omnifarious
Feb. 2nd, 2011
02:36 pm - Interesting design problem with serialization and deserialization
I have been working on a serialization framework I'm happy with for Python. I want to be able to describe CAKE protocol messages clearly and succinctly. This will make it easier to tweak the messages without having to rip apart difficult to understand code. It will also make it easier to understand if I drop the project again and then come back to it years later, or if (by some miracle) someone else decides to help me with it.
Here is what I've come up with as the interface, along with one implementation fo that interface for a simple type:
class Serializer(object): """This is class is an abstract base class. Derived classes, when instantiated, create objects that can serialize other objects of a particular type to a sequence of bytes, or alternately deserialize a sequence of bytes into an object of a particular type.""" __slots__ = ('__weakref__',) def __init__(self): super(Serializer, self).__init__() def serialize(self, val): """x.serialize(value) -> b'serialized value' This is implemented in terms of serialize_iter by default. It is suggested that derived classes only implement serialize or serialize_iter and implement one in terms of the other.""" if self.__class__ is Serializer: raise NotImplentedError("This is an abstract class.") return b''.join(x for x in self.serialize_iter(val)) def serialize_iter(self, val): """x.serialize_iter(value) -> an iterator over the bytes sequences making p the seralized version of value.""" if self.__class__ is Serializer: raise NotImplentedError("This is an abstract class.") return iter((self.serialize(val),)) def deserialize(self, data, memo=None): """x.deserialize(data, [memo]) -> (value of the appropriate type, memoryview(remaining_data)) data must be of type 'bytes', or 'memoryview'. The memo must be a value extracted from a previous NotEnoughDataError. It is undefined what happens if you use memo and do not pass the same data (plus some possible extra data on the end) into deserialize that you originally passed in when you got the NotEnoughDataError you extracted the memo from. May raise a ParseError if there is a problem with the data. If the failure was because the parser ran out of data before parsing was finished, this is required to be a NotEnoughDataError.""" return self._deserialize(data if not isinstance(data, bytes) \ else memoryview(data), memo) def _deserialize(self, memview, memo=None): """x._deserialize(memoryview) -> (value of the appropriate type, memoryview(remaining_data)) Exactly like deserialize, except a memoryview object is required. deserialize is implemented in terms of _deserialize. Derived classes are expected to override _deserialize.""" raise NotImplentedError("This is an abstract class.")
class SmallInt(Serializer): """This class is for integers that are 8, 16, 32, or 64 bits long. They may be signed or unsigned. No other sizes are supported. >>> s = SmallInt(2, True) Traceback (most recent call last): ... ValueError: size is 2, must be 8, 16, 32 or 64 >>> s = SmallInt(8, True) >>> b = list(s.serialize_iter(5)) >>> b == [b'\\x05'] True >>> o = s.deserialize(b''.join(b)) >>> o = (o, o.tobytes()) >>> o == (5, b'') True >>> o = s.deserialize(b''.join(b) + b'z') >>> o = (o, o.tobytes()) >>> o == (5, b'z') True >>> s = SmallInt(8, True) >>> b = s.serialize(-5) >>> b == b'\\xfb' True >>> s = SmallInt(8, True) >>> s = s.serialize(128) Traceback (most recent call last): ... ValueError: 128 is out of range for an signed 8 bit integer >>> s = SmallInt(64, False) >>> b = s.serialize(2**64-1) >>> b == b'\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff' True >>> s = SmallInt(64, True) >>> b = s.serialize(-2**63) >>> b == b'\\x80\\x00\\x00\\x00\\x00\\x00\\x00\\x00' True """ _formats = dict(( ((8, True), '>b'), ((8, False), '>B'), ((16, True), '>h'), ((16, False), '>H'), ((32, True), '>i'), ((32, False), '>I'), ((64, True), '>q'), ((64, False), '>Q') )) __slots__ = ('_size', '_signed', '_low', '_high', '_format') def __init__(self, size, signed): if size not in (8, 16, 32, 64): raise ValueError("size is %d, must be 8, 16, 32 or 64" % (size,)) self._size = size self._signed = bool(signed) self._format = self._formats[(size, signed)] def serialize(self, value): if not isinstance(value, (int, long)): raise TypeError("%r must be an int or long" % (value,)) value = int(value) try: ret = _struct.pack(self._format, value) except _struct.error: raise ValueError("%d is out of range for an %ssigned %d bit " "integer" % (value, ("un" if not self._signed else ""), self._size)) return ret def _deserialize(self, memview, memo=None): numbytes = self._size // 8 if len(memview) < numbytes: raise _NotEnoughDataError((self._size // 8) - len(memview)) else: data = memview[0:numbytes].tobytes() remaining = memview[numbytes:] try: result = _struct.unpack(self._format, data) return result, remaining except _struct.error as err: raise ParseErrror(err)
There is also a
CompoundNumbered type for representing
tuples. This allows you to represent structured messages with multiple
fields. Here is example of how you might
represent CAKE new
cake_newsess_v2 = _serial.CompoundNumbered( _serial.Count(), # Version _serial.Count(), # Type _serial.KeyName(), # Destination key _serial.KeyName(), # Source key _serial.SmallInt(64, False), # Session serial # _serial.CountDelimitedByteString(), # Encryption header _serial.CountDelimitedByteString(), # Signature. _serial.FixedLengthByteString(32) # Header HMAC )
There is a problem though. The signature and header HMAC are supposed to be encrypted, but the deserializer can't know the key to use until it's decrypted the encryption header. This means that later parts of the deserialization process need to know about things from previous parts.
I have a way for the deserialization process to save state. This is used so
that if deserialization throws a
NotEnoughDataError because not
enough data is available, the exception may have a
memo field can then be passed in again to resume close to
where deserialization stopped. (Though now I'm sort of wondering if I
shouldn't do something generator based instead...)
But this mechanism does not allow state to be passed forward from a previous deserializer to a new one. And this applies the other way around too. When serializing there is stuff that's not really a part of the data being serialized (like the current HMAC or encryption state) that needs to be known by serializer in order to serialize properly.
I'm thinking of adding an optional
context parameter to the
serialization and deserialization functions that's just an empty dictionary
into which this sort of state can be stuffed. But this seems really messy.
Can anybody think of any better ways to do this that are fairly general?
Please reply to my original post on Dreamwidth. If you don't have an account there you can log in using your LiveJournal account. Just login using OpenID and give
http://<LJ account name>.livejournal.com/ as your OpenID. For example, for the LJ user rosencrantz319 that would be
http://rosencrantz319.livejournal.com/ as their OpenID.