Journal of Omnifarious

Dec. 31st, 2037

11:59 pm - To all of my friends

A note to all of my friends or those who would friend me

Could you also please friend omnifarious' OpenID account [omnifarious.org] by clicking here: and put that user (who is also me) in all the same groups you put me. If LJ ever lets you use your OpenID identity to log into an existing journal, this may be moot. but I'm not counting on that feature ever happening.

As an aside, if you click on the icon in omnifarious' OpenID account [omnifarious.org], it takes you someplace different than clicking on the hopper [omnifarious.org] does. The takes you to the LJ user info page, and the hopper [omnifarious.org] part takes you to the homepage for the ID.

Tags: ,
Current Mood: [mood icon] working
(29 comments | Leave a comment)

Nov. 19th, 2012

11:55 am - Excellent example of market failure

I have an excellent a clear-cut example of the market can fail both workers and consumers in the service of owners.

Target is currently facing a fairly large petition to not open until a reasonable hour on Black Friday so its employees can actually spend Thanksgiving with family. You might think this is just a workers issue, but it isn't. Many consumers have signed this petition, and the general trend of retail outlets opening earlier and earlier on Black Friday is anti-consumer.

And here's why. Basically people show up early to make sure they can get certain gifts before they're sold out. For those people, it's very worth it to show up before the store opens to make sure they get their chosen gift.

But each store then has an incentive to open before the others. The first store that opens gets the lions share of those consumers.

But those consumers don't actually want to wake up at 3am to get to the store before it opens. They'd be much happier waking up at 6am, or even later. But because the stores are now competing with each other to open earlier, they are forced to choose between getting the gift they want or waking up extremely early to get to the store to make sure it's not out of stock.

The 'invisible hand' of the market creates a destructive cycle that's bad for everybody.

I'm not certain what a good solution to this problem is. One would hope a gentleman's agreement on a decent opening time would work. But I doubt it would. Especially since front-line store employees have little input into honoring these sorts of agreements.

But the point is here that the free market creates a situation in which both consumers and employees get the short end of the stick. They both end up in a situation neither of them desire.

Please reply to my original post on Dreamwidth. If you don't have an account there you can log in using your LiveJournal account. Just login using OpenID and give http://<LJ account name>.livejournal.com/ as your OpenID. For example, for the LJ user rosencrantz319 that would be http://rosencrantz319.livejournal.com/ as their OpenID.

Tags: , ,
Current Location: 424 8th Ave N, 98109
Current Mood: [mood icon] contemplative

Jul. 5th, 2012

05:29 am - The plot is not the story, nor is it the most important part

One interesting phenomena I've noticed recently is a tendency to categorize something (and often dismiss it) based on plot mechanic. "The Hunger Games" has been compared to numerous other 'many enter, one leaves, and everybody watches' stories, especially ones involving children. "Limitless" gets compared to any other story involving medical intelligence enhancement and apparently "Flowers for Algernon" is the canonical example.

I find this sort of distressing. There is a great deal more to a movie than its plot mechanic. Plot is simply the skeleton of a story, not the most important part. It's true that if the skeleton has problems it has a serious negative effect on the whole story, but a story is not its skeleton.

"The Hunger Games", for example, is a story about severe oppression. The games are only a symptom of that oppression. They are certainly not the defining feature of that movie.

Anyway, this is just a minor rant. :-)

Please reply to my original post on Dreamwidth. If you don't have an account there you can log in using your LiveJournal account. Just login using OpenID and give http://<LJ account name>.livejournal.com/ as your OpenID. For example, for the LJ user rosencrantz319 that would be http://rosencrantz319.livejournal.com/ as their OpenID.

Tags: , ,
Current Location: 2464 S Spencer St, 98108
Current Mood: [mood icon] annoyed

Apr. 24th, 2012

10:18 am - The free rider problem

This problem exists in many more contexts than people might otherwise think of. I came to this realization recently while talking with someone about why I really did think her choice to shop around based on price was an admirable one even if I, personally didn't do it.

In that context, I'm a free rider. I reap the benefits of the people who do shop around because they create an incentive for merchants to lower prices. But I do not engage in behavior that will create those incentives myself because it's costly in terms of time and attention.

Similarly, people who use technology that's closed and locked down are free riders on people who consciously make choices to not use such technology. It can be argued that almost every innovative thing we've seen on the Internet in the last 10 years is a direct result of openness and a lack of concern, or even outright hostility towards the idea of 'intellectual property'. Oh, it's true that individual innovators sometimes try to achieve locks. But because of people like me, that generally causes their product to not succeed as well as those who don't. And once the open product achieves critical mass, network effects and the overwhelming advantages of openness do the rest and drive the close product to the fringes of the market.

I don't think free riders are actually necessarily bad. A significant number of them can make markets inefficient. But maybe then people don't actually care about those kinds of inefficiencies. If they did, they would make different choices.

But looking at these things as a free rider problem is a really interesting perspective. And I think the idea is much more broadly applicable than it has been.

It also explains why free riders will not necessarily kill the creation of new and interesting stuff. There are many parts of the market that thrive even when there are a significant number of free riders. When something in the market changes enough that people get upset over the inefficiency created, they stop being free riders.

Please reply to my original post on Dreamwidth. If you don't have an account there you can log in using your LiveJournal account. Just login using OpenID and give http://<LJ account name>.livejournal.com/ as your OpenID. For example, for the LJ user rosencrantz319 that would be http://rosencrantz319.livejournal.com/ as their OpenID.

Tags: , ,
Current Location: 424 8th Ave N, 98109
Current Mood: [mood icon] contemplative

Mar. 25th, 2012

02:59 pm - I watched "Hunger Games" today

I went to watch the much hyped movie today. I was prepared to be revolted, and I was. Not by the movie, or the story. It was well-told, powerful and moving.

I heard people chat idly about the theme appearing in other movies. I saw them smiling as they exited the theater, talking about the finer points of the plot. I saw them wearing nice clothes for an afternoon out.

In the movie I saw the people in The Capitol District chatting idly about the 'contestants'. I saw them smiling and cheering over the 'victory'. I saw them wearing their nicest clothes for the occasion.

I, for a long while, couldn't tell the difference.

That book (and the movie) were written as fiction. But I'm sure the author meant it as a mirror.

Do I vote for that one, or this one? Meaningless choices that we chatter about endlessly, trotting out our best justifications. Few brave enough to make the choice for what they want. The choices are an avoidance of risk, a choice based on fear, not on hope.

That was the choice presented to the two characters at the end of the movie. A choice they were encouraged to make based on fear. The whole system rigged for it. And they made the choice based on hope, the choice the system couldn't tolerate.

I feel like that's what our 'democracy' has degenerated to. A circus, a spectacle geared towards making each of us, individually, make a choice based on fear of what the other guy will do.

I was angry because of the movie. Upset, crying. The happy people around me... I didn't understand. A whole passel of children died on the screen. Horrible deaths, lives shortened needlessly in the service of the subjugation of a whole people.

It's a happy occasion. Time to put on your best stuff and chat idly about it with your friends. There is no mirror. There is no tragedy. The movie has no relevance beyond entertainment. A lie to cover the unbearable truth.

I was angry, I was saddened, and I was revolted.

Yeah, I know, preachy and overbearing. Listen to the message for a change instead of complaining about how it's presented. I too will go back to life as usual. But even a moment of solemnity and understanding of a shared predicament might have been nice.

Please reply to my original post on Dreamwidth. If you don't have an account there you can log in using your LiveJournal account. Just login using OpenID and give http://<LJ account name>.livejournal.com/ as your OpenID. For example, for the LJ user rosencrantz319 that would be http://rosencrantz319.livejournal.com/ as their OpenID.

Tags: , , , ,
Current Location: 2044 Northwest Market Street, 98107
Current Mood: [mood icon] touched

Mar. 13th, 2012

08:55 am - IPv6 not the peer connectivity panacea that people think

IPv6 is supposed to solve all of the peer connectivity issues introduced by NAT. And, on the surface, it seems to do just that by making it possible to assign a unique, globally routable IP address to every conceivable device that could possibly want one.

But this doesn't really solve the problem of peer connectivity.

My cell phone, for example, may be assigned an address by my carrier. But my carrier may be unwilling to let me have any more addresses. This means that any devices I want to connect to the Internet through my cell phone will not be able to have globally routable addresses because my ISP/cell carrier won't route them. And, of course, under IPv6, nobody is ever supposed to do NAT.

So, peer connectivity is still restrained by network topology. The power to decide who gets to be a router decides what gets to connect. And this is broken.

IMHO, the solution is to have addresses assigned to things that have nothing to do with routing, and allow a routing layer on top of the network layer that can route things to those addresses regardless of the actual topology of the network. Tor is an example of this sort of thing. Tor is basically a routing layer on top of TCP/IP that's designed to obscure which routes any given piece of information takes.

But Tor is a specific example of a larger issue. Routing cannot be left ultimately controlled by anybody except network end-points. Such creates failure modes both physical and political that are significantly less than the best we can do.

Which is one of the biggest advantages to a protocol like CAKE. :-) It divorces routing from addressing and expects end-nodes to have a hand in making routing decisions.

Please reply to my original post on Dreamwidth. If you don't have an account there you can log in using your LiveJournal account. Just login using OpenID and give http://<LJ account name>.livejournal.com/ as your OpenID. For example, for the LJ user rosencrantz319 that would be http://rosencrantz319.livejournal.com/ as their OpenID.

Tags: , , ,
Current Location: 2237 NW 62nd ST, 98107
Current Mood: [mood icon] contemplative

Mar. 9th, 2012

12:02 pm - 'Religious' issue

Today, a comment I got really rankled me. My affection and desire for technologies that are not freedom hostile was called a 'religious issue'. This trivializes my desire, and makes it seem like someone has to 'drink the kool-aid' to think the issue is real. And that's insulting.

I find this particularly upsetting given how many people rallied to defeat SOPA. Do people not understand the end goal here? Do you really want your technologies to decide for you which websites you're allowed to see, what you can read, what you can hear? Because ignoring freedom when making technology choices is marching down that very road.

Oh, those companies, they'll never do that. But, they will. Maybe they don't even realize they will. But that kind of lockdown and control is so very economically attractive that companies will march there inexorably unless it's clear that's not a direction we want to go in.

And your choices affect me. Whenever you make a choice against freedom, you're affecting my ability to make that choice. It is possible to make technology that works and is convenient, but doesn't rob you of your freedom. But every time you vote with your dollars against such technology, every time you decide this feature or that feature is worth giving up some of your freedom, you're encouraging companies to dangle shiny toys in exchange for your freedom. In fact, you're encouraging them to only provide the shiny toys if you (and I) give up our freedom to get them. It's like giving in to a toddler who throws tantrums.

I recognize that different people make different choices for their own reasons. And I'm fine with them making those choices. But I will not pass up any opportunity to inform them of the effect of their choice on themselves, and on me.

Please reply to my original post on Dreamwidth. If you don't have an account there you can log in using your LiveJournal account. Just login using OpenID and give http://<LJ account name>.livejournal.com/ as your OpenID. For example, for the LJ user rosencrantz319 that would be http://rosencrantz319.livejournal.com/ as their OpenID.

Tags: , ,
Current Location: 424 8th Ave N, 98109
Current Mood: [mood icon] aggravated

Nov. 8th, 2011

01:59 pm - Working on a small library, what should I name it?

I'm working on a small library to express computations in terms of composable trees of dependencies. These dependencies can cross thread boundaries allowing one thread to depend on a result generated in another thread. This is sort of a riff on the whole promise and future concept, but the idea is that you have chains of these with a potential fanout in the chain greater than 1. Kind of like the venerable make utility in which you express what things need to be finished before starting on the particular thing you're talking about.

But I'm not sure what I should call it. Maybe Teleo because it encourages to express your program in terms of a teleology.

I'm writing this basically because I've encountered the same problem on at least two different projects now, and it occurs to me that it would be really good to have a well-defined standard way of launching things in other threads and waiting for the results that suggested an overall program architecture. The projects I worked on were all set to develop a huge mishmash of different techniques that wouldn't necessarily play well together or be easy to debug.

Please reply to my original post on Dreamwidth. If you don't have an account there you can log in using your LiveJournal account. Just login using OpenID and give http://<LJ account name>.livejournal.com/ as your OpenID. For example, for the LJ user rosencrantz319 that would be http://rosencrantz319.livejournal.com/ as their OpenID.

Tags: , , ,
Current Location: 5819 24th Avenue NW, 98107
Current Mood: [mood icon] creative

Oct. 20th, 2011

03:43 pm - Architecture problem...

I used to have a really good idea of what the architecture of a system that had to respond to multiple different possible sources of input or other reasons to do things (such as some interval of time expiring). My idea was basically to make everything purely event-driven and have big event loops at the heart of the program that dispatched events and got things done.

This solves the vexing problem of how to deal with all these asynchronous occurrences without incurring excessively complex synchronization logic. Nothing gives up control to process another event until the data structures its working with are in a consistent state.

But there are two problems with this model. One is old, and one is relatively new.

The old problem is that such event-driven systems typically exhibit inversion of control, and that makes them confusing and hard to follow. There are ways to structure your program to give people a lot of hints as to what's supposed to happen next when you give up control in the middle of an important operation only to recapture it again at some later point in time in a completely different function. But it's still not the easiest thing in the world to follow.

The 'new' problem is that silicon-based CPUs have not been getting especially faster recently. They've instead been getting more numerous. This is a fairly predictable result. CPUs have a clock. This clock needs to stay synchronized across the entire CPU. Once clock speeds exceed a certain frequency, the clock signal takes longer to propagate across the entire chip than the amount of time before the next pulse is supposed to happen. This means that in order to have an effectively faster CPU on a single chip you need to break it up into independent units that do not need to be strictly synchronized with each other. It's a state horizon problem.

But most programs are not designed to take advantage of several CPUs. If you want a program that's a cohesive whole, but still gets faster as the hardware advances, you need to break it up into several threads.

It seems like maybe it would be simple to do this with a program that had multiple threads. You just have multiple event loops. But then you end up with several interesting problems. How do you decide what things happen in which event loop? What happens if you need to have data shared between things running on different event loops? You run the risk of re-introducing the synchronization issues you avoided when you added the event loops in the first place, all with the cost of inversion of control. It doesn't seem worth it.

Additionally, if you have inter-thread synchronization, what happens if it takes awhile for the other thread to free up the resource you need? How do you prevent deadlocks? Most event systems do allow you to treat the release of a mutex or a semaphore as an event, so you can't just fold waiting for the mutex back into the system as just another event without doing some trick like spawning a thread that waits for the mutex and writes into some sort of IPC mechanism once it's acquired.

And splitting up your program into multiple event threads is not trivial either. How do you detect and prevent the case of one thread being overworked? Also, there is 'state kiting' to consider. Preferably you would prefer one CPU to be handling the same modifiable state for long periods of time. You want to avoid situations where first one CPU cache, then the next have to load up the contents of a particular memory region. Typically, each core will have its own cache. If for no reason other than efficient use of space, it would be good if each core had a disjoint set of memory locations in cache. And to avoid the latency of main memory access, it would be good if that set was relatively static. This means that a single event loop should be working with a fairly small and unchanging set of memory locations.

So simply having several threads, each with its own event loop seems a solution fraught with peril, and it seems like you're throwing away a lot of the advantages you went to an event driven system (with the unpleasant inversion of control side-effect) for in the first place.

So the original idea needs modification, or perhaps a completely new idea is needed.

One modification is embodied in the language Erlang. Erlang still has an event loop and inversion of control. You waiting for messages that come in on a queue. Any other loop can add messages to any queue it knows about. These messages are roughly analogous to events. But the messages themselves convey only information that is immutable. Since it is immutable, shared or not, no synchronization is required since it cannot change.

Erlang also encourages the creation of many such event loops, each of which does a very small job. Hopefully, no individual loop is too overloaded. Modern operating systems are adept at scheduling many jobs, and so this offloads the scheduling of all of these small tasks onto the OS.

I do not think Erlang does overly much to solve the locality of reference problem.

Another approach is the approach taken by the E programming language. It makes extensive use of a concept called a 'future' or 'promise'. This is a promise to deliver the result of some operation at some future point in time. It allows these promises to be chained, so you can build up an elaborate structure of dependencies between promises. In a sense, the programming language handles the inversion of control for you. You specify the program as if control flow were normal, but the language environment automatically launches as many concurrent requests as possible and suspends execution until the results are available.

It is possible to build a set of library-level tools in C++11 to implement this kind of thing somewhat transparently in that language.

I am unsure if there are any major tradeoffs in this approach. Certainly in C++ there is a great deal of implementation complexity, and that complexity cannot be completely hidden from the user as it is in E. I wonder if that implementation complexity introduces unacceptable overhead.

I also suspect that it may be difficult to debug programs that use this sort of a model. They appear to execute sequentially, but in truth they do not. It is possible, for example, to have two outstanding promises for bytes from a file descriptor, but which order those promises will be fulfilled in will not be readily apparent from reading the code. And error conditions can crop up at strange times and propagate to non-obvious places in the control flow of your program.

I also suspect this model will not exhibit the best locality of reference semantics. There will be a tendency to frequently spawn and join threads to handle asynchronous requests. And it will not be immediately apparent to the OS CPU scheduler which threads need to work with which memory objects. And this may lead to active state kiting between CPUs.

Also, those calls to create and destroy threads have a cost, even if that cost is fairly small, it's still likely much more expensive than acquiring an unowned mutex, and probably even more expensive than the call to wait for a file descriptor readability event or waiting for a briefly held mutex to become available.

Of course, it may be possible to implement all of this without creating many threads given a sufficiently clever runtime environment that implements its own queue that folds IO state and semaphore/mutex state events into a single queue. Such an environment would still need a lot of help from the application programmer though to divide up the application to maximize locality of reference within a single thread.

This is a fairly long ramble, and I'm still not really sure what the best approach is. I think I may try to set up some kind of 'smart queue'. This queue will have a priority queue of runnable tasks, and a queue of tasks that could potentially execute given a set of conditions. When a condition is met, the queue will be informed, and if that conditions enables one or more tasks to be run, these tasks will be added to the priority queue.

I envision that the primary thing on which the priority queue will be prioritized is length of time since the task was added to the 'wait for condition' list.

I can then write a C++11 library that will allow you to automatically turn any function that returns a promise into a function that uses these conditions to split up its execution. At least, if you use sufficient care in writing the function.

The conditions (since fulfilling a promise will be a possible condition) will have data associated with them. If this data involves shared mutable state, that will require a great deal of extra care.

Please reply to my original post on Dreamwidth. If you don't have an account there you can log in using your LiveJournal account. Just login using OpenID and give http://<LJ account name>.livejournal.com/ as your OpenID. For example, for the LJ user rosencrantz319 that would be http://rosencrantz319.livejournal.com/ as their OpenID.

Tags: , , ,
Current Mood: [mood icon] contemplative

Jun. 23rd, 2011

12:05 pm - Digital signatures and documents

Random rambling and noodling about a CAKE implementation issueCollapse )

Please reply to my original post on Dreamwidth. If you don't have an account there you can log in using your LiveJournal account. Just login using OpenID and give http://<LJ account name>.livejournal.com/ as your OpenID. For example, for the LJ user rosencrantz319 that would be http://rosencrantz319.livejournal.com/ as their OpenID.

Tags: , ,
Current Location: 2237 NW 62nd ST, 98107
Current Mood: [mood icon] contemplative

Navigate: (Previous 10 Entries)