Journal of Omnifarious

Excellent example of market failure

2012-11-19T20:04:56Z

I have an excellent a clear-cut example of the market can fail both workers and consumers in the service of owners.

Target is currently facing a fairly large petition to not open until a reasonable hour on Black Friday so its employees can actually spend Thanksgiving with family. You might think this is just a workers issue, but it isn't. Many consumers have signed this petition, and the general trend of retail outlets opening earlier and earlier on Black Friday is anti-consumer.

And here's why. Basically people show up early to make sure they can get certain gifts before they're sold out. For those people, it's very worth it to show up before the store opens to make sure they get their chosen gift.

But each store then has an incentive to open before the others. The first store that opens gets the lions share of those consumers.

But those consumers don't actually want to wake up at 3am to get to the store before it opens. They'd be much happier waking up at 6am, or even later. But because the stores are now competing with each other to open earlier, they are forced to choose between getting the gift they want or waking up extremely early to get to the store to make sure it's not out of stock.

The 'invisible hand' of the market creates a destructive cycle that's bad for everybody.

I'm not certain what a good solution to this problem is. One would hope a gentleman's agreement on a decent opening time would work. But I doubt it would. Especially since front-line store employees have little input into honoring these sorts of agreements.

But the point is here that the free market creates a situation in which both consumers and employees get the short end of the stick. They both end up in a situation neither of them desire.

Please reply to my original post on Dreamwidth. If you don't have an account there you can log in using your LiveJournal account. Just login using OpenID and give http://<LJ account name>.livejournal.com/ as your OpenID. For example, for the LJ user rosencrantz319 that would be http://rosencrantz319.livejournal.com/ as their OpenID.

The plot is not the story, nor is it the most important part

2012-07-05T12:36:22Z

One interesting phenomena I've noticed recently is a tendency to categorize something (and often dismiss it) based on plot mechanic. "The Hunger Games" has been compared to numerous other 'many enter, one leaves, and everybody watches' stories, especially ones involving children. "Limitless" gets compared to any other story involving medical intelligence enhancement and apparently "Flowers for Algernon" is the canonical example.

I find this sort of distressing. There is a great deal more to a movie than its plot mechanic. Plot is simply the skeleton of a story, not the most important part. It's true that if the skeleton has problems it has a serious negative effect on the whole story, but a story is not its skeleton.

"The Hunger Games", for example, is a story about severe oppression. The games are only a symptom of that oppression. They are certainly not the defining feature of that movie.

Anyway, this is just a minor rant. :-)

The free rider problem

2012-04-24T17:29:50Z

This problem exists in many more contexts than people might otherwise think of. I came to this realization recently while talking with someone about why I really did think her choice to shop around based on price was an admirable one even if I, personally didn't do it.

In that context, I'm a free rider. I reap the benefits of the people who do shop around because they create an incentive for merchants to lower prices. But I do not engage in behavior that will create those incentives myself because it's costly in terms of time and attention.

Similarly, people who use technology that's closed and locked down are free riders on people who consciously make choices to not use such technology. It can be argued that almost every innovative thing we've seen on the Internet in the last 10 years is a direct result of openness and a lack of concern, or even outright hostility towards the idea of 'intellectual property'. Oh, it's true that individual innovators sometimes try to achieve locks. But because of people like me, that generally causes their product to not succeed as well as those who don't. And once the open product achieves critical mass, network effects and the overwhelming advantages of openness do the rest and drive the close product to the fringes of the market.

I don't think free riders are actually necessarily bad. A significant number of them can make markets inefficient. But maybe then people don't actually care about those kinds of inefficiencies. If they did, they would make different choices.

But looking at these things as a free rider problem is a really interesting perspective. And I think the idea is much more broadly applicable than it has been.

It also explains why free riders will not necessarily kill the creation of new and interesting stuff. There are many parts of the market that thrive even when there are a significant number of free riders. When something in the market changes enough that people get upset over the inefficiency created, they stop being free riders.

I watched "Hunger Games" today

2012-03-25T22:39:52Z

I went to watch the much hyped movie today. I was prepared to be revolted, and I was. Not by the movie, or the story. It was well-told, powerful and moving.

I heard people chat idly about the theme appearing in other movies. I saw them smiling as they exited the theater, talking about the finer points of the plot. I saw them wearing nice clothes for an afternoon out.

In the movie I saw the people in The Capitol District chatting idly about the 'contestants'. I saw them smiling and cheering over the 'victory'. I saw them wearing their nicest clothes for the occasion.

I, for a long while, couldn't tell the difference.

That book (and the movie) were written as fiction. But I'm sure the author meant it as a mirror.

Do I vote for that one, or this one? Meaningless choices that we chatter about endlessly, trotting out our best justifications. Few brave enough to make the choice for what they want. The choices are an avoidance of risk, a choice based on fear, not on hope.

That was the choice presented to the two characters at the end of the movie. A choice they were encouraged to make based on fear. The whole system rigged for it. And they made the choice based on hope, the choice the system couldn't tolerate.

I feel like that's what our 'democracy' has degenerated to. A circus, a spectacle geared towards making each of us, individually, make a choice based on fear of what the other guy will do.

I was angry because of the movie. Upset, crying. The happy people around me... I didn't understand. A whole passel of children died on the screen. Horrible deaths, lives shortened needlessly in the service of the subjugation of a whole people.

It's a happy occasion. Time to put on your best stuff and chat idly about it with your friends. There is no mirror. There is no tragedy. The movie has no relevance beyond entertainment. A lie to cover the unbearable truth.

I was angry, I was saddened, and I was revolted.

Yeah, I know, preachy and overbearing. Listen to the message for a change instead of complaining about how it's presented. I too will go back to life as usual. But even a moment of solemnity and understanding of a shared predicament might have been nice.

IPv6 not the peer connectivity panacea that people think

2012-03-13T16:10:14Z

IPv6 is supposed to solve all of the peer connectivity issues introduced by NAT. And, on the surface, it seems to do just that by making it possible to assign a unique, globally routable IP address to every conceivable device that could possibly want one.

But this doesn't really solve the problem of peer connectivity.

My cell phone, for example, may be assigned an address by my carrier. But my carrier may be unwilling to let me have any more addresses. This means that any devices I want to connect to the Internet through my cell phone will not be able to have globally routable addresses because my ISP/cell carrier won't route them. And, of course, under IPv6, nobody is ever supposed to do NAT.

So, peer connectivity is still restrained by network topology. The power to decide who gets to be a router decides what gets to connect. And this is broken.

IMHO, the solution is to have addresses assigned to things that have nothing to do with routing, and allow a routing layer on top of the network layer that can route things to those addresses regardless of the actual topology of the network. Tor is an example of this sort of thing. Tor is basically a routing layer on top of TCP/IP that's designed to obscure which routes any given piece of information takes.

But Tor is a specific example of a larger issue. Routing cannot be left ultimately controlled by anybody except network end-points. Such creates failure modes both physical and political that are significantly less than the best we can do.

Which is one of the biggest advantages to a protocol like CAKE. :-) It divorces routing from addressing and expects end-nodes to have a hand in making routing decisions.

'Religious' issue

2012-03-09T20:10:25Z

Today, a comment I got really rankled me. My affection and desire for technologies that are not freedom hostile was called a 'religious issue'. This trivializes my desire, and makes it seem like someone has to 'drink the kool-aid' to think the issue is real. And that's insulting.

I find this particularly upsetting given how many people rallied to defeat SOPA. Do people not understand the end goal here? Do you really want your technologies to decide for you which websites you're allowed to see, what you can read, what you can hear? Because ignoring freedom when making technology choices is marching down that very road.

Oh, those companies, they'll never do that. But, they will. Maybe they don't even realize they will. But that kind of lockdown and control is so very economically attractive that companies will march there inexorably unless it's clear that's not a direction we want to go in.

And your choices affect me. Whenever you make a choice against freedom, you're affecting my ability to make that choice. It is possible to make technology that works and is convenient, but doesn't rob you of your freedom. But every time you vote with your dollars against such technology, every time you decide this feature or that feature is worth giving up some of your freedom, you're encouraging companies to dangle shiny toys in exchange for your freedom. In fact, you're encouraging them to only provide the shiny toys if you (and I) give up our freedom to get them. It's like giving in to a toddler who throws tantrums.

I recognize that different people make different choices for their own reasons. And I'm fine with them making those choices. But I will not pass up any opportunity to inform them of the effect of their choice on themselves, and on me.

Working on a small library, what should I name it?

2011-11-08T22:13:57Z

I'm working on a small library to express computations in terms of composable trees of dependencies. These dependencies can cross thread boundaries allowing one thread to depend on a result generated in another thread. This is sort of a riff on the whole promise and future concept, but the idea is that you have chains of these with a potential fanout in the chain greater than 1. Kind of like the venerable make utility in which you express what things need to be finished before starting on the particular thing you're talking about.

But I'm not sure what I should call it. Maybe Teleo because it encourages to express your program in terms of a teleology.

I'm writing this basically because I've encountered the same problem on at least two different projects now, and it occurs to me that it would be really good to have a well-defined standard way of launching things in other threads and waiting for the results that suggested an overall program architecture. The projects I worked on were all set to develop a huge mishmash of different techniques that wouldn't necessarily play well together or be easy to debug.

Architecture problem...

2011-10-20T22:43:48Z

I used to have a really good idea of what the architecture of a system that had to respond to multiple different possible sources of input or other reasons to do things (such as some interval of time expiring). My idea was basically to make everything purely event-driven and have big event loops at the heart of the program that dispatched events and got things done.

This solves the vexing problem of how to deal with all these asynchronous occurrences without incurring excessively complex synchronization logic. Nothing gives up control to process another event until the data structures its working with are in a consistent state.

But there are two problems with this model. One is old, and one is relatively new.

The old problem is that such event-driven systems typically exhibit inversion of control, and that makes them confusing and hard to follow. There are ways to structure your program to give people a lot of hints as to what's supposed to happen next when you give up control in the middle of an important operation only to recapture it again at some later point in time in a completely different function. But it's still not the easiest thing in the world to follow.

The 'new' problem is that silicon-based CPUs have not been getting especially faster recently. They've instead been getting more numerous. This is a fairly predictable result. CPUs have a clock. This clock needs to stay synchronized across the entire CPU. Once clock speeds exceed a certain frequency, the clock signal takes longer to propagate across the entire chip than the amount of time before the next pulse is supposed to happen. This means that in order to have an effectively faster CPU on a single chip you need to break it up into independent units that do not need to be strictly synchronized with each other. It's a state horizon problem.

But most programs are not designed to take advantage of several CPUs. If you want a program that's a cohesive whole, but still gets faster as the hardware advances, you need to break it up into several threads.

It seems like maybe it would be simple to do this with a program that had multiple threads. You just have multiple event loops. But then you end up with several interesting problems. How do you decide what things happen in which event loop? What happens if you need to have data shared between things running on different event loops? You run the risk of re-introducing the synchronization issues you avoided when you added the event loops in the first place, all with the cost of inversion of control. It doesn't seem worth it.

Additionally, if you have inter-thread synchronization, what happens if it takes awhile for the other thread to free up the resource you need? How do you prevent deadlocks? Most event systems do allow you to treat the release of a mutex or a semaphore as an event, so you can't just fold waiting for the mutex back into the system as just another event without doing some trick like spawning a thread that waits for the mutex and writes into some sort of IPC mechanism once it's acquired.

And splitting up your program into multiple event threads is not trivial either. How do you detect and prevent the case of one thread being overworked? Also, there is 'state kiting' to consider. Preferably you would prefer one CPU to be handling the same modifiable state for long periods of time. You want to avoid situations where first one CPU cache, then the next have to load up the contents of a particular memory region. Typically, each core will have its own cache. If for no reason other than efficient use of space, it would be good if each core had a disjoint set of memory locations in cache. And to avoid the latency of main memory access, it would be good if that set was relatively static. This means that a single event loop should be working with a fairly small and unchanging set of memory locations.

So simply having several threads, each with its own event loop seems a solution fraught with peril, and it seems like you're throwing away a lot of the advantages you went to an event driven system (with the unpleasant inversion of control side-effect) for in the first place.

So the original idea needs modification, or perhaps a completely new idea is needed.

One modification is embodied in the language Erlang. Erlang still has an event loop and inversion of control. You waiting for messages that come in on a queue. Any other loop can add messages to any queue it knows about. These messages are roughly analogous to events. But the messages themselves convey only information that is immutable. Since it is immutable, shared or not, no synchronization is required since it cannot change.

Erlang also encourages the creation of many such event loops, each of which does a very small job. Hopefully, no individual loop is too overloaded. Modern operating systems are adept at scheduling many jobs, and so this offloads the scheduling of all of these small tasks onto the OS.

I do not think Erlang does overly much to solve the locality of reference problem.

Another approach is the approach taken by the E programming language. It makes extensive use of a concept called a 'future' or 'promise'. This is a promise to deliver the result of some operation at some future point in time. It allows these promises to be chained, so you can build up an elaborate structure of dependencies between promises. In a sense, the programming language handles the inversion of control for you. You specify the program as if control flow were normal, but the language environment automatically launches as many concurrent requests as possible and suspends execution until the results are available.

It is possible to build a set of library-level tools in C++11 to implement this kind of thing somewhat transparently in that language.

I am unsure if there are any major tradeoffs in this approach. Certainly in C++ there is a great deal of implementation complexity, and that complexity cannot be completely hidden from the user as it is in E. I wonder if that implementation complexity introduces unacceptable overhead.

I also suspect that it may be difficult to debug programs that use this sort of a model. They appear to execute sequentially, but in truth they do not. It is possible, for example, to have two outstanding promises for bytes from a file descriptor, but which order those promises will be fulfilled in will not be readily apparent from reading the code. And error conditions can crop up at strange times and propagate to non-obvious places in the control flow of your program.

I also suspect this model will not exhibit the best locality of reference semantics. There will be a tendency to frequently spawn and join threads to handle asynchronous requests. And it will not be immediately apparent to the OS CPU scheduler which threads need to work with which memory objects. And this may lead to active state kiting between CPUs.

Also, those calls to create and destroy threads have a cost, even if that cost is fairly small, it's still likely much more expensive than acquiring an unowned mutex, and probably even more expensive than the call to wait for a file descriptor readability event or waiting for a briefly held mutex to become available.

Of course, it may be possible to implement all of this without creating many threads given a sufficiently clever runtime environment that implements its own queue that folds IO state and semaphore/mutex state events into a single queue. Such an environment would still need a lot of help from the application programmer though to divide up the application to maximize locality of reference within a single thread.

This is a fairly long ramble, and I'm still not really sure what the best approach is. I think I may try to set up some kind of 'smart queue'. This queue will have a priority queue of runnable tasks, and a queue of tasks that could potentially execute given a set of conditions. When a condition is met, the queue will be informed, and if that conditions enables one or more tasks to be run, these tasks will be added to the priority queue.

I envision that the primary thing on which the priority queue will be prioritized is length of time since the task was added to the 'wait for condition' list.

I can then write a C++11 library that will allow you to automatically turn any function that returns a promise into a function that uses these conditions to split up its execution. At least, if you use sufficient care in writing the function.

The conditions (since fulfilling a promise will be a possible condition) will have data associated with them. If this data involves shared mutable state, that will require a great deal of extra care.

Digital signatures and documents

2011-06-24T05:43:42Z

Documents and the digital signatures that apply to them are necessarily separate. Most current cryptographic systems either digitally sign things on the fly (TLS) or send a library of digital signatures with the document they sign (OpenPGP). Though, to be fair, in the OpenPGP case, each of those digital signatures signs a variant document.

In CAKE there are documents to be signed. Examples are documents that say "This public key exists, was created at time X, is valid for new sessions and signatures from times A through B, and is considered invalid at time E.", or "This public key is reachable at this URL from times A through B.", or "Public key I has agreed to store and forward messages for public key J from times A through B.", or "My name for public key J is N.".

For some of these documents there is only one key who's signature is relevant. For others, a specific small set of keys is relevant (the store and forward case, for example). And for others you care about all signatures, but especially signatures by other keys you trust.

Of course, you could consider the document signed to include the name of the signing entity, in which case, each signature would be for a different document.

I'm not completely sure how to handle this. In my system there will be some documents that cannot be considered valid until multiple signatures have been received. So the signature has to be totally detached from the document.

People are so weirdly emotional about money

2011-06-21T19:54:25Z

I've been paying a lot of attention to bitcoin recently. It's a fascinating idea, and I'm really curious as to where it will go. But reading the comments on the Internet about it is even more interesting, though also kind of upsetting. People say the most ridiculous and stupid things, and it's all out of nearly violent emotion. I don't really understand.

Some people say ridiculous things like "It can only go up!" (in reference to the USD/Bitcoin exchange rate) or "It can't fail!". Optimism beyond the point of sanity. Bitcoin can fail. It can fail if it turns out that nobody wants to accept it. Currency that nobody will trade anything for is just as useful as a small piece of paper, and in bitcoin's case, even less useful. And that's a very definite possible future of bitcoin.

People also go through all kinds of logical contortions to declare it a scam. But it doesn't fit the definition of a ponzi scheme any more than any other currency does, nor does it fit the definition of a pyramid scheme at all. The closest it comes to is a hot tech stock. And nobody calls those scams unless they accuse them of 'pump and dump'. But 'pump and dump' doesn't fit the profile of most people who are interested in bitcoins and are trading them either.

And then people declare it valueless, as if any currency (even gold) has any intrinsic value beyond people's willingness to trade stuff for it.

Very few people talk about the worthiness of the cryptography. But even the ones who do paint either incredibly rosy pictures or ridiculous apocalyptic scenarios, neither of which really approach the truth of things.

I just find the way people ignore any reason and base their opinions on pure emotion to be kind of upsetting. And I notice this in a lot of arguments. But the arguments over bitcoin are almost comical in just how incredibly intense this phenomena is. The only thing that makes it not comical is that you realize these people are deadly serious.

I think a lot of people have a lot of unexamined hang-ups about the meaning of money. It's deeply tied to their fundamental beliefs about politics, ethics, morality, and even self-worth. I think most people are terribly unequipped to tease these things apart and examine them separately. Money is 'magic'. People do not see it as the societal cooperation tool that it is.

I think, perhaps, that is one of the most valuable parts of the bitcoin project. Its nature provides a handle or a window for examining money as a societal and organizational tool. I suspect most people won't be able to take advantage of this, but I suspect many will, and our society will become richer for it.

Help! DynDNS has become prohibitively expensive!

2011-06-10T23:48:41Z

They want to charge me $40/yr per domain for secondary DNS! $40/yr! This is completely ridiculous. With the volume of lookups I get, I could probably host all the domains on my own server on a DSL line if I wanted.

Is anybody out there willing to provide secondary DNS for a few domains for me? I'm willing to cough up the equivalent of $10/yr in bitcoins for the service if you really want.

Session properties

2011-05-30T12:48:38Z

I've been puzzling over a minimal and orthogonal set of properties for a session. I at first thought there were 3:

Message boundaries preserved: Whether or not your messages are delivered in discrete units, or whether they are delivered as a stream of bytes in which the original sizes of the send calls bear no relevance to how the bytes are chunked together on the other end.
Ordered: Whether or not data arrives in the order you sent it
Reliable: Well, this has a tricky definition. For TCP it means that failure to deliver is considered a failure of the underlying connection. But after such a failure you can't really be sure about exactly which bytes were delivered and which weren't.

But, as is evidenced by my description of 'reliable', these properties are not as hard-edged as they might seem. I also thought about latency, for example a connection via email is relatively high latency, and a connection between memory and the CPU is generally pretty low latency. But I'm looking for hard-edged, yes/no type properties that are in some sense fundamental. Latency seems like a property that's rather fuzzy. It exists on a continuum, and isn't really a defining feature of a connection, something that would drastically alter how you wrote programs that used the connection. In an object model, it would be an object property, not something you'd make a different class for.

But I find TCP's notion of 'reliability' very curious. It isn't really, in any sense, particularly reliable. I've had ssh connections that died, but when I reconnect to my screen session, I discover that a whole bunch of the stuff I was typing made it through, it just wasn't echoed back.

It also interacts with 'ordered' in an odd way. It might make sense to have an unordered connection that was 'reliable', but what does that really mean then? If it's a TCP notion of reliability, you could just deliver the last message and have the connection drop. Also, what would it mean to have an unreliable, but ordered connection? Would that mean you could send a bunch of messages and have only the first and last ones delivered? And would it make any sense at all to have an unordered, unreliable connection in which message boundaries were not preserved?

So I've come up with a different division...

Message boundaries preserved: Whether or not your messages are delivered in discrete units, or whether they are delivered as a stream of bytes in which the original sizes of the send calls bear no relevance to how the bytes are chunked together on the other end.
Ordered: Whether or not data arrives in the order you sent it
Must not drop: This means that if a message does not make it through, the connection is considered to be in an unrecoverable error state, and no further messages may be sent. Though you may not know which message didn't make it through.
Delivery notification: Whether or not you can know that a message made it to the other side or not.

These are not fully orthogonal. For example, if message boundaries are not preserved, then, in order for a connection to be in the least sensible, it must also have the 'ordered' and 'must not drop' properties. Also, if you must not drop messages, I'm not sure that it would then be sensible to have out-of-order delivery.

One of the rules of the system I'm designing is that any property that is not required may be provided anyway. This makes non-orthogonality much easier to deal with. So the prior cases aren't really a problem.

Can any of you think of a better set of properties, or important properties that I left out?

Some good discussion also happens in this Google Buzz post that mirrors this entry.

The hidden cost of your iPad

2011-05-04T22:04:29Z

Suicide is so common in Chinese iPad factories that the company has taken to forcing prospective employees to sign no-suicide pacts.

Talk about treating the symptom instead of the disease.

A friend of mine has pointed out that this story is made to seem a lot worse than it really is. In particular the suicide at Foxconn plants is much lower than it is at other similar facilities in China. He is also not much of a fan of Apple the company, so he doesn't have a fanboy bias. I'm not completely sure I agree with this way of looking at things, but here is what he wrote, so you can make up your own minds:

This story has been highly sensationalized. The reality is almost exactly the opposite of what you read.

Eighteen Foxconn employees committed suicide in 2010 [1]... out of 920,000 workers [2]. That's a rate much lower than the Chinese average of 66 per million [3], which itself is like half of the American average of 111 per million [3].

Apple is just one of many Foxconn clients. Others include Amazon (Kindle), Intel, Dell, Nintendo, Sony, Samsung, and many others [2]. Apple products are a small minority of Foxconn's output, yet the media calls them the "iPad factory". This is obviously intended to sensationalize the story -- scandal involving Apple is much more interesting that scandal involving Samsung.

I suspect that Foxconn came up with these no-suicide pledges in a desperate attempt to placate the media, and due to cultural differences they don't understand that to the American audience it only makes them look worse.

[1] http://en.wikipedia.org/wiki/Foxconn_suicides
[2] http://en.wikipedia.org/wiki/Foxconn
[3] http://en.wikipedia.org/wiki/List_of_countries_by_suicide_rate

Building codes

2011-04-19T13:29:41Z

Building codes serve a few functions. The most important one is safety. But another is ensuring that your home does not fall to pieces in 10 years (after the builders are long gone) by forcing certain minimum standards of construction.

To the latter end, I think building codes for multi-unit dwellings should require that each and every single unit have a single fiber drop in the unit. I assume there are standards for phone hookups today (and possibly cable), and the fiber standard would have a very similar purpose and structure.

CAKE has reached a small milestone

2011-03-28T16:11:04Z

CAKE reached a new milestone early this morning. It now successfully both generates and parses messages that use the new protocol. It also successfully detected a re-used session id. I also think the code that does this is also a lot better designed than the old code was. It's easier to see how to put it in the context of a larger system that implements a node that speaks the protocol

It's also much more extensively tested at a deeper level with tests that are designed to document the inner workings of the system.

Overall, it's in a much better state than I left it when I sort of stopped working on it much in 2004. And I'm going to handle the hard problems first, how to maintain the relationship between sessions and transports, and having two way realtime conversations between nodes. This rather than concentrating on the messages that will be traded back and forth at a higher level (which will be done using protobuf). That can come later, especially since I'm not likely to get it right the first time anyway.

I also need to think about getting nodes to participate in a DHT to share assertions (like how to reach a particular node) in a distributed way.

Lastly, the protocol has something of a problem with 'liveness' because I designed it with the idea of conversations being able to be initiated without any round trips. There are some mitigation for this problem in session ids, but that mitigation is somewhat problematic because it requires the recipient of a conversation initiation to keep track of some stuff for everybody who tries to talk to it.

I'm not really sure how to handle the 'liveness' problem though and still preserve the lack of round trips property. I could require that session ids contain an 'hour number' or something similar. Though that introduces a requirement for at least very coarse grain time synchronization for all nodes.

Memory

2011-02-25T20:17:55Z

Memory is stored in so many places. A sea shell contains the memory of the organism that made it. Its trials and tribulations are recorded in the layers of material it deposited. Since it was unable to make a meaningful decision based on these memories, we hesitate to call them so, but our scientists eagerly read them, read the memories in whole stratas of seashells, the memories of entire ecosystems.

We implicitly recognize this when we say something like "this house is full of memories". Every nick and change, unnoticeable by some, tells a tale of something that happened there. The patterns of wear on the floor, the neglected dusty corners tell tales as well.

Forensics is the art of reading memories from these structural changes. Reading memory from these things we hesitate to call memory because they are not immediately accessible to a living process. But memories they are.

We have a collective memory too. The most obvious and directly accessible is books. But we have memories in our cities, in our tools, in the structures both great and small. They are like mankind's seashells.

We think of ourselves as relatively self contained. We are divided from the world by the interface of our immediate perceptions. But that division is fuzzy and indistinct. We are much larger than our bodies. And much of our memory lives outside our heads.

Interesting design problem with serialization and deserialization

2011-02-02T22:46:44Z

I have been working on a serialization framework I'm happy with for Python. I want to be able to describe CAKE protocol messages clearly and succinctly. This will make it easier to tweak the messages without having to rip apart difficult to understand code. It will also make it easier to understand if I drop the project again and then come back to it years later, or if (by some miracle) someone else decides to help me with it.

Here is what I've come up with as the interface, along with one implementation fo that interface for a simple type:

class Serializer(object):
    """This is class is an abstract base class.  Derived classes, when
    instantiated, create objects that can serialize other objects of a
    particular type to a sequence of bytes, or alternately deserialize
    a sequence of bytes into an object of a particular type."""

    __slots__ = ('__weakref__',)

    def __init__(self):
        super(Serializer, self).__init__()

    def serialize(self, val):
        """x.serialize(value) -> b'serialized value'

        This is implemented in terms of serialize_iter by default.

        It is suggested that derived classes only implement serialize
        or serialize_iter and implement one in terms of the other."""
        if self.__class__ is Serializer:
            raise NotImplentedError("This is an abstract class.")
        return b''.join(x for x in self.serialize_iter(val))

    def serialize_iter(self, val):
        """x.serialize_iter(value) -> an iterator over the bytes
        sequences making p the seralized version of value."""
        if self.__class__ is Serializer:
            raise NotImplentedError("This is an abstract class.")
        return iter((self.serialize(val),))

    def deserialize(self, data, memo=None):
        """x.deserialize(data, [memo]) ->
        (value of the appropriate type, memoryview(remaining_data))

        data must be of type 'bytes', or 'memoryview'.  The memo must
        be a value extracted from a previous NotEnoughDataError.

        It is undefined what happens if you use memo and do not pass
        the same data (plus some possible extra data on the end) into
        deserialize that you originally passed in when you got the
        NotEnoughDataError you extracted the memo from.

        May raise a ParseError if there is a problem with the data.
        If the failure was because the parser ran out of data before
        parsing was finished, this is required to be a
        NotEnoughDataError."""
        return self._deserialize(data if not isinstance(data, bytes) \
                                     else memoryview(data),
                                 memo)

    def _deserialize(self, memview, memo=None):
        """x._deserialize(memoryview) ->
        (value of the appropriate type, memoryview(remaining_data))

        Exactly like deserialize, except a memoryview object is
        required.  deserialize is implemented in terms of
        _deserialize.  Derived classes are expected to override
        _deserialize."""
        raise NotImplentedError("This is an abstract class.")



class SmallInt(Serializer):
    """This class is for integers that are 8, 16, 32, or 64 bits long.
    They may be signed or unsigned.  No other sizes are supported.

    >>> s = SmallInt(2, True)
    Traceback (most recent call last):
        ...
    ValueError: size is 2, must be 8, 16, 32 or 64
    >>> s = SmallInt(8, True)
    >>> b = list(s.serialize_iter(5))
    >>> b == [b'\\x05']
    True
    >>> o = s.deserialize(b''.join(b))
    >>> o = (o[0], o[1].tobytes())
    >>> o == (5, b'')
    True
    >>> o = s.deserialize(b''.join(b) + b'z')
    >>> o = (o[0], o[1].tobytes())
    >>> o == (5, b'z')
    True
    >>> s = SmallInt(8, True)
    >>> b = s.serialize(-5)
    >>> b == b'\\xfb'
    True
    >>> s = SmallInt(8, True)
    >>> s = s.serialize(128)
    Traceback (most recent call last):
        ...
    ValueError: 128 is out of range for an signed 8 bit integer
    >>> s = SmallInt(64, False)
    >>> b = s.serialize(2**64-1)
    >>> b == b'\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff'
    True
    >>> s = SmallInt(64, True)
    >>> b = s.serialize(-2**63)
    >>> b == b'\\x80\\x00\\x00\\x00\\x00\\x00\\x00\\x00'
    True
    """

    _formats = dict((
            ((8, True),   '>b'),
            ((8, False),  '>B'),
            ((16, True),  '>h'),
            ((16, False), '>H'),
            ((32, True),  '>i'),
            ((32, False), '>I'),
            ((64, True),  '>q'),
            ((64, False), '>Q')
        ))

    __slots__ = ('_size', '_signed', '_low', '_high', '_format')

    def __init__(self, size, signed):
        if size not in (8, 16, 32, 64):
            raise ValueError("size is %d, must be 8, 16, 32 or 64" % (size,))
        self._size = size
        self._signed = bool(signed)
        self._format = self._formats[(size, signed)]

    def serialize(self, value):
        if not isinstance(value, (int, long)):
            raise TypeError("%r must be an int or long" % (value,))
        value = int(value)
        try:
            ret = _struct.pack(self._format, value)
        except _struct.error:
            raise ValueError("%d is out of range for an %ssigned %d bit "
                             "integer" % (value,
                                          ("un" if not self._signed else ""),
                                          self._size))
        return ret

    def _deserialize(self, memview, memo=None):
        numbytes = self._size // 8
        if len(memview) < numbytes:
            raise _NotEnoughDataError((self._size // 8) - len(memview))
        else:
            data = memview[0:numbytes].tobytes()
            remaining = memview[numbytes:]
            try:
                result = _struct.unpack(self._format, data)[0]
                return result, remaining
            except _struct.error as err:
                raise ParseErrror(err)

There is also a CompoundNumbered type for representing tuples. This allows you to represent structured messages with multiple fields. Here is example of how you might represent CAKE new session messages:

cake_newsess_v2 = _serial.CompoundNumbered(
    _serial.Count(), # Version
    _serial.Count(), # Type
    _serial.KeyName(), # Destination key
    _serial.KeyName(), # Source key
    _serial.SmallInt(64, False), # Session serial #
    _serial.CountDelimitedByteString(), # Encryption header
    _serial.CountDelimitedByteString(), # Signature.
    _serial.FixedLengthByteString(32) # Header HMAC
)

There is a problem though. The signature and header HMAC are supposed to be encrypted, but the deserializer can't know the key to use until it's decrypted the encryption header. This means that later parts of the deserialization process need to know about things from previous parts.

I have a way for the deserialization process to save state. This is used so that if deserialization throws a NotEnoughDataError because not enough data is available, the exception may have a memo field. This memo field can then be passed in again to resume close to where deserialization stopped. (Though now I'm sort of wondering if I shouldn't do something generator based instead...)

But this mechanism does not allow state to be passed forward from a previous deserializer to a new one. And this applies the other way around too. When serializing there is stuff that's not really a part of the data being serialized (like the current HMAC or encryption state) that needs to be known by serializer in order to serialize properly.

I'm thinking of adding an optional context parameter to the serialization and deserialization functions that's just an empty dictionary into which this sort of state can be stuffed. But this seems really messy. Can anybody think of any better ways to do this that are fairly general?

Protocol buffers?

2010-12-04T23:28:33Z

I have a problem for which protocol buffers seem like a good solution, but I'm reluctant to use them. First, protocol buffers include facilities for handling the addition of new fields in the future. This adds a small amount to a typical protocol buffer message, but it's a facility I do not need.

Also, I feel the variable sized number encoding is less efficient than it could be, though this is a very minor issue. I also feel like I have a number of special purpose data types that are not adequately represented.

I'm also not completely pleased with the C++ and/or Python APIs. I think they contain too many googlisms. I would like to see public APIs published that were free of adherence to Google coding standards like do-nothing constructors and no exceptions.

I think, maybe, I will be using protocol buffers for some messages that are sent by applications using CAKE as a transport/session layer. These include some of the sub-protocols that are required to be implemented by a conforming CAKE implementation.

On a different note, I think Google's C++ coding standards are lowering the overall quality of Open Source C++ code. This isn't a huge effect, but it's there.

It happens because Google's good name is associated with a set of published standards for C++ coding that include advice that while possibly good for Google internally is of dubious quality as general purpose advice. It also happens because when Google releases code for their internal tools to the Open Source community, these tools follow Google's standards. And some of these standards have the effect of making it hard to use code that doesn't comply with those standards in conjunction with code that does.

Today's XKCD

2010-05-22T00:05:27Z

Normally XKCD is amusing for very positive reasons. But I frequently feel a lot like the guy with the beard in this cartoon. It's really frustrating. So, today's XKCD is darkly amusing to me. Freedom is such a hard sell before people lose it. People choose convenience every time, frequently until it's almost too late to fix the problem all the while berating the people who were worried in the first place.

Eben Moglen Tech Talk at Google

2010-05-18T23:32:09Z

Eben Moglen is one of the principle lawyers behind the GPL. He's also a tireless free software advocate, and significantly more photogenic and diplomatic than Richard Stallman.

He recently gave this interesting tech talk at Google about the perception of Google by entities outside it. It was really well done, and struck a strong chord with me.

I've noticed that people frequently are incapable of believing that some things Google does are for the reasons Google says they're doing them. For example (and I don't really have the time to find references just now) many people seem to think that Google Doodles, those fun, timely modifications to their main search page, are a marketing tool, when in fact they are largely done purely out of whimsy.

I suppose, in one sense there is marketing purpose. Google is projecting their image of themselves out into the world. It's brand building. But, on the other hand, there isn't. I doubt that Google Doodles started as an idea for brand building in some marketing department. I'm betting some random small group of people decided one day that it would be fun to do, and the idea sort of caught on and now it's a tradition.

But people seem to want to analyze doodles for the marketing message they contain, despite the fact there generally isn't one. The more enigmatic the doodle is, the more determined people seem to be to find the marketing message in it.

This means there is a disparity in perception between people outside Google and people inside Google. One that might serve Google very poorly in the future. It's very important that Google understand this and respond appropriately. Perception is reality and people and organizations live up to expectations. Google risks becoming what people perceive them to be unless they act to correct that perception.

Google also frequently doesn't realize how the fact that they are so large and powerful affects people's perceptions of them. Witness the brouhaha over Buzz. Google did do some somewhat wrongheaded things in introducing it, but Buzz was not anywhere near the privacy destroying aggregator that people thought it was. And the fact that people perceived Buzz in this way seemed to mystify people inside Google, even though it was predictable given Google's size and people's perceptions.

Again, this points to a need by Google to better manage people's perceptions of them, and to manage their product releases better in terms of how people perceive them.

Eben Moglen suggests, quite wisely, that one thing Google could do is to change their policy on contributing internal changes back to Open Source projects. I think this is a good idea, but I doubt it will really be enough.

I am a little worried that if Google takes this advice to heart that they will grow a PR arm that does what every other PR arm in the world does, which is to try to make sure that perception stays far more positive than reality instead of simply trying to make perception match reality. But Google should do something, since I think people think far more ill of them than they generally deserve.

Google is, in fact, the only company I know of that has a revenue stream greater than 1 billion dollars a year that I actually have a positive opinion of.

The evils of Flash

2010-05-07T17:54:14Z

This was a Slashdot comment, but I think it deserves a top level post here. It's in response to Apple’s attack on Adobe Flash, it’s all about online video NOT. (I added the 'NOT' because that's the author's conclusion.)

Pot calls kettle black, kettle complains, but it's just as black.

Flash is a despicable disgrace. Most of the time when I talk to a Flash developer, the thing they're the happiest about is the control they get over my computer. This is directly because the Flash player is a piece of garbage closed source tool that purposely caters to developers over end-users. The Open Source gnash (not ganash) player has an option to pause a Flash program. The Adobe player will never, ever end up with that option, ever. Giving me control over my own computer is against Adobe's best interest. That makes Adobe's Flash player is little more than a widely deployed trojan horse that, IMHO, is little better than spyware (Flash cookies anyone? Where's my control over those?).

I wouldn't complain so bitterly about this if the gnash player were actually a decent drop in replacement for the closed source Flash player, but it isn't. I have to either choose my freedom to have my computer do what I want instead of what some random corporation wants with Flash that is broken most of the time, or Flash that works while giving up my freedom. I will choose my freedom, thank you very much, but I will be bitter about the stupid choice I'm forced to make.

So, when one maker of a closed, proprietary platform that steals people's freedom purposely does things to the detriment of another closed proprietary platform that steals people's freedom, I can't help but cheer. And I hope Adobe finds a way to play nasty games with Apple too. The more these two companies can find ways to hurt eachother, the more the rest of us benefit.

If Adobe Open Sourced the Flash player (I could care less about the developer tools, they will end up with Open Source implementations no matter what Adobe does if the player is truly open) my objections to Flash would completely disappear. I could realistically choose a fully functional Flash player and I'm certain I could find one with a pause button, or one that refused to store cookies for longer than a week. I could make it myself if I wanted to.

And lest you tell me that I'm just whining, the majority of large sites out there no longer look right without Flash. By not using Flash, I'm cut off from a significant part of the experience of the web. I shouldn't be forced to give up control of my computer in order to browse the web. That's a completely and utterly ridiculous assertion.

Walking data structures

2010-03-30T16:49:55Z

It's common programmer tech speak to talk about 'walking' data structures, meaning following all the pointers around to put all the data back together again. I think that 'brachiation' is a more apt metaphor, and fits well with the concept of 'code monkey'.

The new health care bill

2010-03-22T15:20:21Z

From what I've heard, the new health care bill will impose fines on people who don't have health insurance. Is this true? If it is, I'm tempted to refuse my employers health insurance plan on general principles.

I'm ambivalent about the idea of such a bill in the first place. I wanted Obama elected in the hopes that he'd fix things like the influence of lobbyists and money and restore civil liberties and have a more open and accountable government. Health care was really low on my list of priorities, and I was nearly certain I wasn't for any kind of solution that involved perpetuating the stupidity that is the current health insurance system.

Unfortunately, Obama seems to be standing still or going backwards on all the things I really care about. For example, his administration has refused significantly more FOIA requests than the previous one.

I'm rather disappointed.

I hate perl

2010-03-18T18:14:06Z

Case in point, the Net::IP module. The documentation looks nice. It handles IPv6 and IPv4 addresses. It looks clean and simple.

Then, I decided I would like to be able to have IPv4 mapped IPv6 addresses match the IPv4 address ranges I'm singling out for special treatment. So I look into its tool for extracting an IPv4 address from an IPv6 address.

The call, ip_get_embedded_ipv4 doesn't seem to work on IPv6 addresses created with 'new'. It only works on IPv6 addresses represented as strings. This leads me to dive into the implementation.

I discover that the is no coherent internal representation. Just a lot of different attributes that are used at different times for different purposes and are converted from one another as needed.

Additionally, there appears to be no way to import particular symbols of certain classes from the module. You have to import them using the import statements specified in the documentation or take your chances on whether or not it will work. This is because the import mechanism and which symbols are global or not is handled in a fairly ad-hoc sort of way and re-implemented in each module according to the whims of the author.

It's really quite surprising the module works at all. And I'm left feeling like I really ought to re-write it if I want something I can count on.

In reality, looking at the module's implementation was a mistake. This is always what happens to me when I look at a perl module. Either it works in a completely mysterious way using language mechanisms I've never seen used before, or it works in a way that's totally broken and practically guaranteed to break for any use that varies from the specific use-cases described in the documentation. Frequently both are the case. Aigh! Run away!

I hope I can convince my new workplace to stop using perl.

I'm on Dreamwidth now!

2010-03-15T00:18:28Z

Thank you foxfirefey, leora, and fallenpegasus. :-) I used foxfirefey's code because she's the first person I learned about the existence of Dreamwidth from.

Please reply to my original post on Dreamwidth. If you don't have an account there you can log in using your LiveJournal account. Just login using OpenID and give http://[Unknown LJ tag].livejournal.com/ as your OpenID. For example, for the LJ user rosencrantz319 that would be http://rosencrantz319.livejournal.com/ as their OpenID.