2
I Use This!
Inactive

News

Analyzed 3 days ago. based on code collected 3 days ago.
Posted almost 14 years ago by Michael
For the past several months the python magazine hasn't sent any new issues out. Indeed, since late last year they ripped out their website and had a banner saying "We're busy building a new python magazine", with a link laughably suggesting that ... [More] there is more information available. This is after last year them getting several months behind with the magazine. They also said " Don't worry—your subscription and back issues are safe and will be available when the new site launches. ". That's fine, in theory. However consider:Whilst they may let the grass grow under their feet they haven't bothered telling their subscribers. Paid subscribers.They haven't bothered updating their website telling their customers what they're doing.Indeed, they appear, from a subscriber point of view, to have simply cut and run.I can't actually think what excuse they can come up with that justifies not actually bothering to contact subscribers for well over 1/2 a year, but I'm sure they have one. On the flip side, they don't have any contact address on their front page, nor on their content free "what we're doing" page. Beyond this, last year they decided, of their own volition to charge my credit card to renew my subscription. Now I was going to renew anyway - it's been a great magazine in the past, but charging my card without upfront consent struck me as rather dodgy.Since they've now reneged on their half of the sale contract and not delivered, and actually have a good reason to need to get in contact with them, I can't. This means I'm left with 2 choices:Either put out a public notice in the hope that it's something that someone there will read, and actually get back in contact to let me know how to contact themContact Visa and say that they're a rogue trader, and that they should be banned from making any further transactions against my card (especially given the last one was done without my explicit consent.Neither is particularly attractive, and hopefully someone knows how to get in contact with them because they sure aren't advertising any contact details right now. Finally, I get that it's a small publication, that it's one borne out of love, rather than profit (at a guess based on guestimates of costs), but if you're having trouble getting things back started, at least have the decency to tell your subscribers, rather than having content free "information" pages.After all, a lot can change in 1/2 a year... (Last issue I have is from August 2009...)(Sorry to anyone who reads this who have nothing to with the python magazine, but if you know someone there, please let me know who is "running" it these days) [Less]
Posted over 14 years ago by Michael
A few months ago, there was a thread on the then google group python-concurrency about some standard forms for showing how some libraries deal with concurrent problems. The specific example chosen looked like this:#!/bin/shtail -f /var/log/system.log ... [More] |grep pantsPete Fein also posted an example of this using generators, based on David Beazley's talk on python generators being used as (limited) coroutines:    import time    import re     def follow(fname):        f = file(fname)        f.seek(0,2) # go to the end        while True:            l = f.readline()            if not l: # no data                time.sleep(.1)            else:                yield l    def grep(lines, pattern):        regex = re.compile(pattern)        for l in lines:            if regex.match(l):                yield l    def printer(lines):        for l in lines:            print l.strip()    f = follow('/var/log/system.log')    g = grep(f, ".*pants.*")    p = printer(g)    for i in p:        passThe question/challenge raised on the list was essentially "what does this look like in your framework or system?". For some reason, someone saw fit to move the mailing list from google groups, and delete the archives, so I can't point at the thread, but I did repost my answer for what was called "99 bottles" for kamaelia on the python wiki .I quite liked the example for describing how to take this and convert it into a collection of kamaelia components, primarily because by doing so we gain a number of reusable components in this way. For me it was able describing how to move from something rather ad-hoc to something somewhat more generally usable.For me, the point about Kamaelia is really that it's a component framework aimed at making concurrent problems more tractable & maintainable. Basically so that I can get stuff done quicker, that won't need rewriting completely to use concurrency, which someone else can hack on without needing to come back to me to understand it. In practice though, this also means that I tend to focus on building stuff, rather than asking "is it concurrent?". (Axon kinda ensures that it either is, or is concurrent friendly) This does sometimes also mean I focus on getting the job done, rather than "does this look nice"... Whilst that does matter to me, I do have deadlines like the next person :-)For example, one thing missing from the above is that when you do something like:    tail -f /var/log/system.log |grep pantsYou aren't interested in the fact this uses 3 processes - tail, grep & parent process - but the fact that by writing it like this you're able to solve a problem quickly and simply. It also isn't particularly pretty, though I personally I view the shell version as rather elegant.Naturally, being pleased with my version, I blogged about it. Much like anyone else, when I write something it seems like a good idea at the time :-). As sometimes happens, it made it onto reddit with some really nice & honest comments.And what were those comments? If I had to summarise in one word "ugh!"Whilst I don't aim for pretty (I aim for safe/correct :), pretty is nice, and pretty is fun. As a result, I've wanted to come back to this.Ugly is no fun :-( . Fun matters :-)There was also a comment that suggested using decorators to achieve the same goal. However, at that point in time I had a mental block about what that would look like in this case. So I just thought "OK, I agree, can't quite see how to do it". I did recognise though that they're right to say that decorators would improve this case.In particular the stumbling block is the way python generators are used in the above example is effectively a one way chaining. printer pulls values from grep. grep pulls values from follow. When one of them exits, they all exit. Essentially this is pull based.In Kamaelia, components can be push, pull or push & pull. Furthermore they can push and pull in as many directions as you need. At the time mapping between the two sensibly it didn't seem tractable to me. Then this morning, as I woke blearily, I realised that the reason why. Essentially the above generator form isn't really directly the same as the shell form - though it is close.Taking grep, for example, if I do this:grep "foo" somefileThen grep will open the file "somefile", read it, and output lines that match the pattern and exit.However, if I do this:bla | grep "foo"Then grep will read values from stdin, and output lines which match the pattern. Furthermore, it will pause outputting values when bla stops pushing values into the chain, and exit when bla exits (after finishing processing stdin). ie It essentially has two modes of operating, based on getting a value or having an absent value.In essence, the differences about what's happening here are subtle - in the shell we pass in a symbol which represents which stream needs opening, whereas in the example above, we pass in, effectively, an open stream. Also the shell is very much a combination of push and pull, whereas the generator pipeline above is essentially pull.This made me realise that rather than activating the generator we want to read from *outside* the generator we're piping into, if we activate the generator *inside* the generator we're piping into, the problem becomes tractable.For example, if we change this:def grep(lines, pattern):    regex = re.compile(pattern)    for l in lines: # Note this requires an activate generator, or another iterable        if regex.match(l):            yield lTo this:def grep(lines, pattern):    "To stop this generator, you need to call it's .throw() method. The wrapper could do this"    regex = re.compile(pattern)    while 1:        for l in lines(): # Note we activate the generator here inside instead            if regex.search(l):                yield l        yieldWe gain something that can operate very much like the command line grep. That is, it reads from its equivalent to stdin until stdin is exhausted. To indicated stdin is exhausted it simply yields - ie yields None. The caller can then go off and get more data to feed grep. Alternatively the caller can shutdown this grep at any point in time by throwing in an exception.Making this small transform allows the above example to be rewritten as kamaelia components like this:import sysimport timeimport reimport Axonfrom Kamaelia.Chassis.Pipeline import Pipelinefrom decorators import blockingProducer, TransformerGenComponent@blockingProducerdef follow(fname):    "To stop this generator, you need to call it's .throw() method. The wrapper could do this"    f = file(fname)    f.seek(0,2) # go to the end    while True:        l = f.readline()        if not l: # no data            time.sleep(.1)        else:            yield l@TransformerGenComponentdef grep(lines, pattern):    "To stop this generator, you need to call it's .throw() method"    regex = re.compile(pattern)    while 1:        for l in lines():            if regex.search(l):                yield l        yield@TransformerGenComponentdef printer(lines):    "To stop this generator, you need to call it's .throw() method"    while 1:        for line in lines():            sys.stdout.write(line)            sys.stdout.flush()        yieldPipeline(    follow('/var/log/system.log'),    grep(None, ".*pants.*"),    printer(None)).run()The implementation for both decorators.py and example.py above can both be found here:http://code.google.com/p/kamaelia/source/browse/trunk/Sketches/MPS/AxonDecorators/Similarly, if we wanted to use multiple processes, we could rewrite that final pipeline like this:    from Axon.experimental.Process import ProcessPipeline    ProcessPipeline(        follow('/var/log/system.log'),        grep(None, ".*pants.*"),        printer(None)    ).run()Specifically the above will use 4 processes. One container process, and 3 subprocesses. (ProcessPipeline would benefit from a rewrite using multiprocess rather than pprocess though)The other nice thing about this approach is that suppose you wanted to define your own generator source like this:def source():    for i in ["hello", "world", "game", "over"]:        yield iYou could use that instead of "follow" above like this:    Pipeline(        grep(source, ".*pants.*"),        printer(None)    ).run()For me, this has a certain symmetry with the change from thistail somefile.txt | grep ".*pants.*" | cat -to this: grep ".*pants.*" source | cat - ie if you pass in an absent value, it processes the standard inbox "inbox", rather than stdin. If you pass in a value, it's assumed to be a generator that needs activating.Stepping back, and answering the "why? What does this give you?" question, it becomes more apparent as to why this might be useful when you start monitoring 5 log files at once for POST requests. For example, putting that all together in a single file would look like this:(assuming you didn't reuse existing components :)import sysimport timeimport reimport Axonfrom Kamaelia.Util.Backplane import Backplane, SubscribeTo, PublishTofrom Kamaelia.Chassis.Pipeline import Pipelinefrom decorators import blockingProducer, TransformerGenComponent@blockingProducerdef follow(fname):    f = file(fname)    f.seek(0,2) # go to the end    while True:        l = f.readline()        if not l: # no data            time.sleep(.1)        else:            yield l@TransformerGenComponentdef grep(lines, pattern):    regex = re.compile(pattern)    while 1:        for l in lines():            if regex.search(l):                yield l        yield@TransformerGenComponentdef printer(lines):    while 1:        for line in lines():            sys.stdout.write(line)            sys.stdout.flush()        yieldBackplane("RESULTS").activate()for logfile in ["com.example.1", "com.example.2", "com.example.3","com.example.4","com.example.5"]:    Pipeline(        follow(logfile+"-access.log"),        grep(None, "POST"),        PublishTo("RESULTS")    ).activate()Pipeline(    SubscribeTo("RESULTS"),    printer(None)).run()Now, I don't particularly like the word pythonic - maybe it is, maybe it isn't - but hopefully this example does look better than perhaps than last time! The biggest area needing work, from my perspective, in this  example is the names of the decorators.Since this will be going into the next release of Axon - any feedback - especially on naming - would be welcome :-).(Incidentally, follow/grep have already been added to kamaelia, so this would really be simpler, but it does make an interesting example IMO :-) [Less]
Posted over 14 years ago by Michael
A few people will have already noticed some small comments about this, but we're plotting to restart python northwest. Specifically, we're restarting this month. Details:When: Thursday 24th September, 6pm Where: Rain Bar, Manchester ... [More] , UK: http://www.rain-bar.co.uk/ 80 Great Bridgewater Street, Manchester, M1 5JG map: http://tinyurl.com/mn4qnl Who: Who can come? If you're reading this YOU can (assuming you're sufficiently close :-)More specifically anyone from beginners, the inexperienced through deeply experienced and all the way back to the plain py-curious. What: Suggestion is to start off with a social meet, and chat about stuff we've found interesting/useful/fun with python recently. Topics likely to include robots and audio generation, the recent unconference, and europython. Twitter: http://twitter.com/pynw / #pythonnorthwest / #pynwMailing list: http://groups.google.com/group/python-north-west How did this happen? I tweeted the idea, a couple of others seconded it, the David Jones pointed out "it easier to arrange for a specific 2 people to meet than it was to e-mail a vague cloud of people and get _any_ 2 to meet anywhere.", so that's where we'll be. If twitter feedback is anything go by, we're hardly going to be alone, so please come along - the more the merrier :-) Better yet, please reply to this post saying you're coming along! More generally, assuming this continues, pynw will probably be every third thursday in the month, maybe alternating between technical meets and social ones. (probably topic for discussion :-) Please forward this to anyone you think may be interested! See you there! [Less]
Posted over 14 years ago by Michael
If this happens this will be awesome. Traffic Server is some really nice code. It's a large codebase, but it's really cool, and it *scales*. (I used to work at Inktomi, so have been inside the code as well). For those that don't know what it is, it's ... [More] a very high performance web caching proxy, with a plugin architecture, allowing for the addition of other protocols. It used to support HTTP (& obvious friends), NNTP, RTSP, RTP, WMV, etc. That's pretty much made my day that has. [Less]
Posted over 14 years ago by Michael
Since I've had a few questions about this, a short status update. At Europython last week I was recording all the talks I was attending. Including the lightning talks this means I have video from 55 talks. The video files from the camera are too ... [More] large for blip.tv, so I'm transcoding them down to a smaller size, before uploading them. Since these 55 talks are spread over nearly 80 files, that naturally takes time. Fortunately/obviously, I'm automating this, and it'll come as no shock to some that I'm automating it using kamaelia. This automation needs to to be stoppable, since I need to only do this overnight, for practicality reasons.Anyway, for those curious, this is the code I'm using to do the transcode & upload. You'll note that it saturates my CPU, keeping both cores busy. Also, it's interleaving an IO bound process (ftp upload) with CPU bound - transcode.import osimport reimport Axonfrom Kamaelia.Chassis.Graphline import Graphline from Kamaelia.Chassis.Pipeline import Pipelineclass Find(Axon.Component.component):    path = "."    walktype = "a"    act_like_find = True    def find(self, path = ".", walktype="a"):        if walktype == "a":            addfiles = True            adddirs = True        elif walktype == "f":            addfiles = True            adddirs = False        elif walktype == "d":            adddirs = True            addfiles = False        deque = []        deque.insert(0,  (os.path.join(path,x) for x in os.listdir(path)) )        while len(deque)>0:            try:                fullentry = deque[0].next()                if os.path.isdir(fullentry):                    if adddirs:                        yield fullentry                    try:                        X= [os.path.join(fullentry,x) for x in os.listdir(fullentry)]                        deque.insert(0, iter(X))                    except OSError:                        if not self.act_like_find:                            raise                elif os.path.isfile(fullentry):                    if addfiles:                        yield fullentry            except StopIteration:                deque.pop(0)    def main(self):        gotShutdown = False        for e in self.find(path = self.path, walktype=self.walktype):            self.send(e, "outbox")            yield 1            if self.dataReady("control"):                gotShutdown = True                break        if not gotShutdown:            self.send(Axon.Ipc.producerFinished(), "signal")        else:            self.send(self.recv("control"), "signal")class Sort(Axon.Component.component):    def main(self):        dataset = []        while 1:            for i in self.Inbox("inbox"):                dataset.append(i)            if self.dataReady("control"):                break            self.pause()            yield 1        dataset.sort()        for i in dataset:            self.send(i, "outbox")            yield 1        self.send(self.recv("control"), "signal")class Grep(Axon.Component.component):    pattern = "."    invert = False    def main(self):        match = re.compile(self.pattern)        while 1:            for i in self.Inbox("inbox"):                if match.search(i):                    if not self.invert:                        self.send(i, "outbox")                else:                    if self.invert:                        self.send(i, "outbox")            if self.dataReady("control"):                break            self.pause()            yield 1        self.send(self.recv("control"), "signal")class TwoWayBalancer(Axon.Component.component):    Outboxes=["outbox1", "outbox2", "signal1","signal2"]    def main(self):        c = 0        while 1:            yield 1            for job in self.Inbox("inbox"):                if c == 0:                    dest = "outbox1"                else:                    dest = "outbox2"                c = (c 1) % 2                self.send(job, dest)                job = None            if not self.anyReady():                self.pause()            if self.dataReady("control"):                break        R=self.recv("control")        self.send(R, "signal1")        self.send(R, "signal2")class Transcoder(Axon.ThreadedComponent.threadedcomponent):    command = 'ffmpeg >transcode.log 2>&1 -i "%(SOURCEFILE)s" -s 640x360 -vcodec mpeg4 -acodec copy -vb 1500000 %(ENCODINGNAME)s'    def main(self):        while 1:            for sourcefile in self.Inbox("inbox"):                shortname = os.path.basename(sourcefile)                encoding_name = shortname.replace(".mp4", ".avi")                finalname = sourcefile.replace(".mp4", ".avi")                # Do the actual transcode                print "TRANSCODING", sourcefile, encoding_name                os.system( self.command % {"SOURCEFILE": sourcefile, "ENCODINGNAME":encoding_name})                # file is transcoded, move to done                print "MOVING DONE FILE", sourcefile, os.path.join("done", sourcefile)                os.rename(sourcefile, os.path.join("done", sourcefile))                # Move encoded version to upload queue                upload_name = os.path.join( "to_upload", encoding_name)                print "MOVING TO UPLOAD QUEUE", encoding_name, upload_name                os.rename(encoding_name, upload_name )                # And tell the encoder to upload it please                print "SETTING OFF UPLOAD",upload_name, finalname                self.send( (upload_name, finalname), "outbox")                print "-----------------"            if self.dataReady("control"):                break        self.send(self.recv("control"), "signal")class Uploader(Axon.ThreadedComponent.threadedcomponent):    command = "ftpput --server=%(HOSTNAME)s --verbose --user=%(USERNAME)s --pass=%(PASSWORD)s --binary --passive %(UPLOADFILE)s"    username = < editted :-) >    password = < editted :-) >    hostname = "ftp.blip.tv"    def main(self):        while 1:            for (upload_name, finalname) in self.Inbox("inbox"):                print "UPLOADING", upload_name                os.system( self.command % {                                        "HOSTNAME":self.hostname,                                        "USERNAME":self.username,                                        "PASSWORD":self.password,                                        "UPLOADFILE":upload_name,                                     } )                print "MOVING", upload_name, "TO", os.path.join("encoded", finalname)                os.rename(upload_name, os.path.join("encoded", finalname))                print "-----------------"            if self.dataReady("control"):                break            if not self.anyReady():                self.pause()        self.send(self.recv("control"), "signal")Graphline(    FILES = Pipeline(                Find(path=".",walktype="f"),                Sort(),                Grep(pattern="(done|encoded|unsorted|transcode.log|to_upload)",                     invert = True),            ),    SPLIT = TwoWayBalancer(), # Would probably be nicer as a customised PAR chassis    CONSUME1 = Pipeline(                     Transcoder(),                    Uploader(),               ),    CONSUME2 = Pipeline(                     Transcoder(),                    Uploader(),               ),    linkages = {        ("FILES","outbox"):("SPLIT","inbox"),        ("SPLIT","outbox1"):("CONSUME1","inbox"),        ("SPLIT","outbox2"):("CONSUME2","inbox"),        ("FILES","signal"):("SPLIT","control"),        ("SPLIT","signal1"):("CONSUME1","control"),        ("SPLIT","signal2"):("CONSUME2","control"),    }).run()It should be fairly clear that this will go as fast as it can, so please be patient :-) [Less]
Posted over 14 years ago by Michael
Before I started using python, I'd used perl for several years, and one thing which I'd liked about perl was its autoload facility. Now in python the closest equivalent that I've seen is __getattr__ for classes, but not __getattr__ for a module. This ... [More] seemed like a real shame since there are times when autoload can be incredibly useful.If it seems chaotic, consider the Unix PATH variable. Any time you type a name, the shell looks in lots of locations and runs the first one it finds. That's effectively the same sort of idea as autoloading. Yes, you can do some really nasty magic if you want, but then you can do that with the shell to, and generally people get along find.Anyway, vaguely curious about it I decided to do some digging around, and came across this post by Leif K Brookes, which suggests this:You could wrap it in an object, but that's a bit of a hack. import sysclass Foo(object):     def __init__(self, wrapped):         self.wrapped = wrapped     def __getattr__(self, name):         try:             return getattr(self.wrapped, name)         except AttributeError:             return 'default'sys.modules[__name__] = Foo(sys.modules[__name__])That looked reasonable, so I created a file mymod.py which looks like this:import sysdef greet(greeting="Hello World"):   print greetingclass mymod_proxy(object):    def __init__(self):        super(mymod_proxy, self).__init__()        self.wrapped = sys.modules["mymod"]    def __getattr__(self, name):        try:            return getattr(self.wrapped, name)        except AttributeError:            def f():                greet(name)            return fsys.modules["mymod"] = mymod()And tried using it like this:~> pythonPython 2.5.1 (r251:54863, Jan 10 2008, 18:01:57)[GCC 4.2.1 (SUSE Linux)] on linux2Type "help", "copyright", "credits" or "license" for more information.>>> import mymod>>> mymod.hello()hello>>> from mymod import Hello_World>>> Hello_World()Hello_WorldAnd as you can see, it seems to work as expected/desired.Now the reason I'd been thinking about this, is because I'd like to retain the hierarchy of components in Kamaelia that we have at the moment (it's useful for navigating what's where), but given we tend to use them in a similar way to Unix pipelines it's natural to want to be able to do something like:from Kamaelia import Pipeline, ConsoleReader, ConsoleWriterPipeline(    ConsoleReader(),    ConsoleWriter(),).run()Rather than the more verbose form specifically pulling them in from particular points. Likewise, we don't really want to import every single module in Kamaelia.py, because of the large number of components there (Kamaelia is really a toolbox IMO where things get wired together, and Axon is the tool for making new tools), the majority of which won't be used in ever application!Now, I haven't done this yet, and wouldn't do it lightly, but the fact that you can actually make autoload functionality work, seems kinda cool, and and a nice opportunity. But I'm also now wondering just how nasty this approach seems to people. After all, Leif describes it as "a bit of a hack", and whilst it's neat, I'm not taking in the positive view. I'm interested in any views on better ways of doing Autoload in python, and also whether people view it as a nice thing at all. (One person's aesthetic is another person's pain after all...) [Less]
Posted almost 15 years ago by Michael
 I've mentioned this in a couple of places, but mentioning on my blog seems appropriate too. I'm giving a tutorial on Kamaelia at Europython '09 this year.Europython details:    Where: Birmingham UK   When:       Tutorial days 28/29th June.      ... [More] Main conference: 30th June - 2nd July      Kamaelia specifically: 28th June, 9am            http://www.europython.eu/talks/timetable/   Cost:       Tutorial days: £100        Conference days: £190   More info:       http://www.europython.eu/       http://www.europython.eu/talks/timetable/Blurb for the Kamaelia tutorial:Kamaelia: Pragmatic ConcurrencyTutorial, Half day (intermediate)Why use concurrency? Since concurrency is viewed as an advanced topic by many developers, this question is often overlooked. However, many real world systems, including transportation, companies, electronics and Unix systems are highly concurrent and accessible by the majority of people. One motivation can be “many hands make light work” but in software this often appears to be false – in no small part due to the tools we use to create systems. Despite this, the need for concurrency often creeps into many systems.Kamaelia is a toolset and mindset aimed at assisting in structuring your code such that you can focus on the problem you want to solve, but in a way that results in naturally reusable code that happens to be painlessly concurrent. It was designed originally to make maintenance of highly concurrent network systems simpler, but has general application in a wider variety of problem domains, including desktop applications, web backend systems (eg video transcode & SMS services), through to tools for teaching a child to read and write.This tutorial will cover:A fast overview in the style of a lightning talk.Kamaelia's core – Axon – which provides the basic tools needed for concurrent systems, followed by a session on implementing your own core.Practical walk throughs of real world Kamaelia systems to demonstrate how to build and design systems of your own.More advanced concepts such as reusing centralised services and constrained data sharing, as well as ongoing open issues will be touched upon.Tips, tricks and rules of thumb when working with concurrent systems.During this highly practical tutorial, where you will create your own version of Axon, your own components and first Kamaelia based system (bring a laptop!). The course expects a level of familiarity with Python but no prior experience of concurrency is assumed.The structure of this in terms of time is 2 x 1.5 hour sessions, with a 15 minute break in the middle, so hopefully enough time to impart enough useful knowledge to help you get started with Kamaelia.Also, if Kamaelia isn't interesting to you (sob :),  Ali Afshar who hangs out on Kamaelia's IRC channel is also giving a tutorial there on PyGTK, along with lots of other people giving interesting tutorials and talks :-) [Less]
Posted almost 15 years ago by Michael
I've added a PythonInterpreter component to Kamaelia in Kamaelia.Experimental.PythonInterpreter. The primary use for this is intended to assist with debugging, but there are other possible uses.Use in a systemThis allows a variety of things, from a ... [More] basic command line console:StandaloneInterpreter().run()Which you can run in the background on any console. For example you could do this:    ServerCore(protocol=myprotocol, port=1234).activate()    StandaloneInterpreter().run()Alternatively, you can use an embeddable component that speaks to inbox/outbox rather than stdin/stdout. Crudely you can do something like this:    Pipeline(        ConsoleReader(),        InterpreterTransformer(),        ConsoleEchoer(),    ).run()But you can also put it inside a pygame application, reading & writing from a Textbox/TextDisplayer:    Pipeline(        Textbox(size = (800, 300), position = (100,380)),        InterpreterTransformer(),        TextDisplayer(size = (800, 300), position = (100,40)),    ).run() This looks like this:The interesting option this opens up of course is the fact that you can add in a networked interpreter (note: this is hideously insecure incidentally - so only safe of a network you completely control!) into any application:    from Kamaelia.Chassis.Pipeline import Pipeline    from Kamaelia.Chassis.ConnectedServer import ServerCore    from Kamaelia.Util.PureTransformer import PureTransformer        from Kamaelia.Experimental.PythonInterpreter import InterpreterTransformer        def NetInterpreter(*args, **argv):        return Pipeline(                    PureTransformer(lambda x: str(x).rstrip()),                    InterpreterTransformer(),                    PureTransformer(lambda x: str(x)+"\r\n>>> "),               )        ServerCore(protocol=NetInterpreter, port=1236).run()In one console:    # ./ServerBasedPythonInterpreter.pyIn another:    ~> telnet 127.0.0.1 1236    Trying 127.0.0.1...    Connected to 127.0.0.1.    Escape character is '^]'.        >>> self    Component Kamaelia.Experimental.PythonInterpreter.InterpreterTransformer_21 [ inboxes : {'control': [], 'inbox': []} outboxes : {'outbox': [], 'signal': []}    >>> "hello world"    hello world    >>> Axon    from '/usr/local/lib/python2.5/site-packages/Axon/__init__.pyc'>    >>> dir(self)    ['Inbox', 'Inboxes', 'Outboxes', 'Usescomponents', '_AxonObject_...    >>> self.scheduler    Axon.Scheduler.scheduler_1 :1 :1 :0 :    >>> self.scheduler.threads{: at 0xb7c92468>, 0xb7ba4acc>: , 0xb7ba6bec>: , : 0xb7c92468>, object at 0xb7b9dc4c>: , : object at 0xb7c92460>, 0xb7b8acec>: , : 0xb7c92468>, 0xb7ba192c>: , : object at 0xb7c92468>, 0xb7b979cc>: }etc.Rummaging Around inside Running SystemsThe idea behind this component really is to assist in debugging live running systems by directly rummaging around inside them, which is why this form is probably most useful:    if interactive_debug:        StandaloneInterpreter().activate()        MyMainApplication().run()And whilst the other forms naturally drop out as useful, this form is probably the safest of the bunch!This form also is slightly more flexible, in that it allows this sort of thing:    ~/> ./BasicPythonInterpreter.py    > from Kamaelia.UI.Pygame.Ticker import Ticker    ok    > X=Ticker()    ok    Component Kamaelia.UI.Pygame.Ticker.Ticker_7 [ inboxes : {'control': [], '_displaycontrol': [], 'unpausebox': [], 'pausebox': [], 'inbox': [], 'alphacontrol': []} outboxes : {'outbox': <>, 'signal': <>, '_displaysignal': <>}> X.activate()    ok    Component Kamaelia.UI.Pygame.Ticker.Ticker_7 [ inboxes : {'control': [], '_displaycontrol': [], 'unpausebox': [], 'pausebox': [], 'inbox': [''], 'alphacontrol': []} outboxes : {'outbox': <>, 'signal': <>, '_displaysignal': <>}> self.link((self, "outbox"), (X,"inbox"))    ok    Link( source:[Kamaelia.Experimental.PythonInterpreter.StandaloneInterpreter_5,outbox], sink:[Kamaelia.UI.Pygame.Ticker.Ticker_7,inbox] )> self.send("Hello", "outbox")    okAs you can guess this then results the text displayer outputting to the display. As another example, if you're rummaging inside a system, you could start up an introspector in another console:   ./AxonVisualiser.py --port 1500And then in the enbedded interpreter do this:> from Kamaelia.Util.Introspector import Introspectorok> from Kamaelia.Chassis.Pipeline import Pipelineok> from Kamaelia.Internet.TCPClient import TCPClientok> Pipeline(    Introspector(),    TCPClient("127.0.0.1", 1500),).activate()> > >okAnd that then adds an Axon visualisation/introspector to a running system - which looks like this:Clearly this can aid in identifying when something has gone wrong.Hope others find it useful too :-) [Less]
Posted almost 15 years ago by Michael
Earlier this evening an announcement by Pete Fein regarding the formation of a python-concurrency mailing list (aka Python Concurrency Special Interest Group) bounced past my inbox in the unladen-swallow mailing list. Naturally, given my work on Axon ... [More] (a concurrency library) and Kamaelia (a bunch of components that use it), it jumped out at me as interesting. (5 minute overview for those that don't know what Kamaelia is...) Pete then posted to the list a small collection of different ways of implementing...#!/bin/shtail -f /var/log/system.log |grep pants... in python.This struck me as a nice example worth comparing to Kamaelia - specifically to compare the generator version. Indeed the Kamaelia version is fairly directly derived from the generator version, which is why I'm picking on it. The full generator version looks like this:import timeimport redef follow(fname):    f = file(fname)    f.seek(0,2) # go to the end    while True:        l = f.readline()        if not l: # no data            time.sleep(.1)        else:            yield ldef grep(lines, pattern):    regex = re.compile(pattern)    for l in lines:        if regex.match(l):            yield ldef printer(lines):    for l in lines:        print l.strip()f = follow('/var/log/system.log')g = grep(f, ".*pants.*")p = printer(g)for i in p:    passNow, this is nice, but the core logic:tail -f /var/log/system.log |grep pantsHas somehow become obfuscated at the end - which is a shame, because it doesn't need to be. Taking the Kamaelia version step by step, let's take that end part:f = follow('/var/log/system.log')g = grep(f, ".*pants.*")p = printer(g)for i in p:    pass... and turn it into something which looks like what you'd do in Kamaelia:from Kamaelia.Chassis.Pipeline import PipelinePipeline(    Follow('tail -f /var/log/system.log'),    Grep(".*pants.*"),    Printer(),).run()(incidentally, to actually use separate processes, ala shell, you'd use ProcessPipeline, not Pipeline)So in order for this to be valid, we now need to adapt follow into Follow, grep into Grep, and printer into Printer.Let's start by adapting follow:def follow(fname):    f = file(fname)    f.seek(0,2) # go to the end    while True:        l = f.readline()        if not l: # no data            time.sleep(.1)        else:            yield lFirst of all, we note that this follow source blocks - specifically calling time.sleep. Now there are other ways of doing this, but the simplest way of keeping this code structure is to just create a threaded component. We also need to capture the argument (fname) which isn't optional and has no logical default, so we need to have an __init__ method. So we grab that, store it somewhere handy, and call the super class initialiser. Pretty standard stuff.import Axonclass Follow(Axon.ThreadedComponent.threadedcomponent):    def __init__(self, fname, **argv):        self.fname = fname        super(Follow,self).__init__(**argv)    def main(self):        f = file(self.fname)        f.seek(0,2) # go to the end        while not self.dataReady("control"):            l = f.readline()            if not l: # no data                time.sleep(.1)            else:                self.send(l, "outbox")        f.close()        self.send(self.recv("control"), "signal")The main() method of this component now follows the same core logic as the follow() generator, as highlighted in green. The logic added in blue, is simply control logic, and is fairly commonly used, which looks like this:        while not self.dataReady("control"):                    self.send(self.recv("control"), "signal")This allows someone to shutdown our component cleanly. Other than that, the other major difference is that this:            yield lHas been replaced with this:            self.send(l, "outbox")ie rather than expecting to be called in a "call me & handle my values" loop, it just passes it's results out an outbox called "outbox". (much like tail -f spits it's results out of stdout)The next part to convert is grep(). Recalling this, this looks like this:def grep(lines, pattern):    regex = re.compile(pattern)    for l in lines:        if regex.match(l):            yield lWith the Kamaelia equivalent looking like this:class Grep(Axon.Component.component):    # Default pattern, override in constructor with pattern="some pattern"    # See below    pattern = "."     def main(self):        regex = re.compile(self.pattern)        while not self.dataReady("control"):           for l in self.Inbox("inbox"):               if regex.match(l):                   self.send(l, "outbox")           self.pause()           yield 1        self.send(self.recv("control"), "signal")Once again the overall logic is unchanged, and again the shutdown logic added is in blue.However, we don't need to explicitly pass in a location for lines to enter this component. Specifically we just expect to be passed lines form out standard inbox "inbox", in the same way normal grep expects data on stdin. Also, because we have a logical default for the pattern (match everything using "."), we don't need to have an __init__ method. The reason for this is because the baseclass for all components does this in its __init__ method:   def __init__(self, *args, **argd):..      self.__dict__.update(argd)So calling Grep(pattern=".*pants.*") will specialise the component value of pattern (The reason for this is it actively enables monkey-patching as reconfiguration, as you can see in this greylisting mail server).Again Grep is actually a special case of a filter, and as a result really just wants to take data from its std inbox and pass matching stuff to its std outbox. With if you look at the core logic, is precisely what it does:           for l in self.Inbox("inbox"):               if regex.match(l):                   self.send(l, "outbox")After clearing its inbox, it pauses, and releases control to the scheduler (Pausing is a component's way of letting the scheduler know "don't call me unless there's some data for me to work on").Finally, we need to implement the Printer() component. Again we can adapt this generator based version, thus:def printer(lines):    for l in lines:        print l.strip()Which transforms into this:class Printer(Axon.Component.component):    def main(self):        while not self.dataReady("control"):            for l in self.Inbox("inbox"):                print l.strip()            self.pause()            yield 1        self.send(self.recv("control"), "signal")Again the core logic is the same (green), and control logic is blue. Again, rather than needing the explicit "lines" iterator, we have a standard place for data to come into the component - the inbox "inbox" - leaving the rest of the logic essentially unchanged.Finally, in the same way the generator version puts the whole thing together:f = follow('/var/log/system.log')g = grep(f, ".*pants.*")p = printer(g)for i in p:    passWe can put the Kamaelia components together:Pipeline(    Follow('tail -f /var/log/system.log'),    Grep(".*pants.*"),    Printer(),).run()This is very different. The generator version creates a chain of iterators, passing them in as the first parameter to the next one in the chain, with the last item in the chain (the for loop) being the one that pulls things along. The Kamaelia version instantiates a Pipeline component which is very clearly piping the outputs from one component into the inputs of the following one.In fact the Pipeline component creates linkages between the components in the pipeline. These are conceptually similar to stackless's channels (though inspired by unix pipes and occam). Implementation whise, a linkage actually collapses an inbox into an outbox. As a result, when Follow sends a message to it's outbox "outbox", that message is instantly delivered into the inbox "inbox" of Grep. Inboxes for generator based components are just plain old lists, and for threaded components they're threadsafe queues.The nice thing about this, is that if you wanted to use this in a network server, you could do this:from Kamaelia.Chassis.ConnectedServer import ServerCoredef FollowPantsInMyLogs(*args):    return Pipeline(             Follow('tail -f /var/log/system.log'),             Grep(".*pants.*"),           )ServerCore(protocol=FollowPantsInMyLogs, port=8000).run()Though that opens a Follow & Grep component for every client. To use just one Follow client, you could just follow and publish the data for all clients:from Kamaelia.Util.Backplane import *Backplane("PANTSINMYLOGS").activate()Pipeline(      Follow('tail -f /var/log/system.log'),      Grep(".*pants.*"),      PublishTo("PANTSINMYLOGS"),).activate()def clientProtocol(*args):    return SubscribeTo("PANTSINMYLOGS")ServerCore(protocol=clientProtocol, port=8000).run()You could also monitor this on the server console by adding in a local printer:Backplane("PANTSINMYLOGS").activate()Pipeline(      Follow('tail -f /var/log/system.log'),      Grep(".*pants.*"),      PublishTo("PANTSINMYLOGS"),).activate()Pipeline(      SubscribeTo("PANTSINMYLOGS"),      Printer(),).activate()def clientProtocol(*args):    return SubscribeTo("PANTSINMYLOGS")ServerCore(protocol=clientProtocol, port=8000).run()... which all fairly naturally describes the higher level co-ordination going on. Now you can do all this from the ground up using plain old generators, but personally I find this approach easier to follow. Some people find others simple :)Anyway, for the curious reader, I hope this is a useful/interesting comparison of a non-kamaelia based approach with a kamaelia based approach, with a visible demonstration at the end of why I prefer the latter :) ... and all this ignores graphlines too :) [Less]
Posted almost 15 years ago by Michael
Bar Camp Leeds (UK) is running for its 3rd year running. It'd be great to see other Python people there if you're around. Details:May 30th, May 31stOld Broadcasting House, Leeds, UKWebsite: http://barcampleeds.com/