||[Apr. 23rd, 2008|01:35 am]
Some fun with python (very techy) - a thing called generators
Ok, this is about a lovely lovely thing called generators. It's also about my project. It's about both because I'm writing part of my project in python and I'm using generators in there.
Firstly, let's explain what this part of the project is doing. I'm building a really simple web application that will prod an agent system, and say "Report back to me". So, the agent system will send back messages on a regular basis telling the webserver what's going on. I'll be storing those in a list of messages.
Now, I'm going to have other programs wanting to know what the agent systems said. So, I'll be giving them everything that's on that list. However, that list is going to get quite big, so I'm going to need some way to tell them what's happened since they last visited. So, I need some sort of a mechanic to enable that.
This is when I start talking about generators. The best way to talk about them is that they're functions that can suspend their execution to return a value, so that when you call them again, they pick up from where they left off. You can do all sorts of weird and wonderful things with them. A really good primer on them is the following: Generators for systems programmers, an introduction, it shows some insanely shiny ways to do things.
For my project, my thoughts were to create generators to run over the list of messages. Well, my first thought were iterators, but a few experiments showed that when you create an iterator, it will only go as far as the length of the list when the iterator is created. For example, make a list of 4 elements. Create an iterator, and then append a new value to the end of the list. Now try and get 5 elements out of the iterator. You'll only get 4 then a StopIteration - an event that tells us that the iterator will not go any further, no matter how much you coax it. That isn't good for my purposes; I know my list will grow and I want clients to be able to pick up from where they left off. So, perhaps a generator would sort me out.
It took me ages to try and find a way to make a generator to run over a list and return each element. Finally I did. But my problems didn't end there. If I run to the end of the list, the generator raises a StopIteration and considers its work to be done, and cannot be told otherwise. In other words, it dies. For most cases this is useful behaviour. For me, it's not.
At this point I should probably refer back to my rule 34 of programming - if you can think of something obscure to do in programming there's probably a reason you'd want to do it at some point. This is one of those times. So if anyone asks "Why on earth do you want to do that?", this is the reason, along with the above.
After some playing around, I finally got to making a cool little generator that keeps track of how much of our messages a client has consumed, while keeping it relatively lightweight. The code's pretty cool:
gen = (i for i in li) # An old testament generator
counter = 0 # this will stop us falling off the end of the list
while 1: # this is important - otherwise the generator will stop at some point
ret =  #
while counter < len(li):
counter += 1
yield ret # Return all the updates
This probably sounds really weird, and slightly hackish, but I really like it for my solution because now I can just put those generators into a hash indexed by the session ids that my app distributes.