» Ad-blocking with Polipo +

Polipo is a fast local proxy that does on-disk caching (by default, at least). Privoxy is another local proxy, with a focus on privacy and ad-blocking. Due to the nature and purpose of Privoxy, it has to buffer portions of the page (to check for content it should block) before serving it to the browser. This makes it a bit slower than Polipo. You could always use Polipo in front of Privoxy (see here, middle of the page), but that is a bit much if you don't need fancy filtering and simply want a domain/regex blocklist.

You could build up the blocklist yourself, by hand, but that would be a pain. Instead, we'll just convert an adblock filterset to a format that Polipo can understand. Since Polipo is blocking by matching the URL only, we don't have the same fine-grained control as Adblock rules, but I personally don't require that level of control.

Firstly, grab an adblock filterset (e.g., easylist.txt). Next, grab either adblock2polipo.py (python) or adblock2polipo.rb (ruby). Then run whichever script you downloaded with the filerset file as the first parameter. The script will dump the re-written rules to the console so they can be inspected or be redirected to the file polipo uses to load its blocking rules from (~/.polipo-forbidden or /etc/polipo/forbidden on *nix systems). Restart Polipo and the new blocking rules should take effect.

PS. If you're like me, you'd rather have Polipo serve up a blank page for blocked URLs rather than a 403 error page. To accomplish this you need to edit the Polipo config file and add a forbiddenUrl option pointing to an empty image, such as this one. One good option for this is the following: make sure localDocumentRoot is either set to a real path (such as /usr/share/polipo/www) or commented out completely. Then create an empty file in that directory:

sudo wget -O /usr/share/polipo/www/empty.gif \
     http://upload.wikimedia.org/wikipedia/commons/4/4b/Empty.gif

Then point forbiddenUrl to http://127.0.0.1:8123/empty.gif. Restart Polipo. That's it. :)

NB. Firefox users will need to add port 8123 to the allowed ports list, or they'll get an equally annoying error message from Firefox instead of a blank page. To do this, you need to open about:config in a new tab/window. Right-click and go to New->String, and for the property name put network.security.ports.banned.override, and for the value put 8123. It should work properly after that.

» Fixing Pulseaudio stutters / pauses / glitches +

It seems that for many AC'97 sound chips, the new scheduler / timer in Pulseaudio >=0.9.11 causes hiccups in sound playback. These are exacerbated when the daemon is not running with higher priority and realtime permissions. The simple fix is to use the older scheduler. To do this, edit /etc/pulse/default.pa and add the tsched=0 option to module-hal-detect (or module-alsa-source if you manually load it). It should look something like this:

load-module module-hal-detect tsched=0

To enable high priority and realtime permissions on the daemon you need to be in the audio, pulse-access and pulse-rt groups:

sudo usermod -a -G audio,pulse-access,pulse-rt

Ps. You'll need to log out and log back in before you're actually in the groups.

Next you (may, depending on your distrbution) need to add the following entries to /etc/security/limits.conf

@pulse-rt       -       rtprio          99
@pulse-rt       -       nice            -20

Then you need to add the following options to /etc/pulse/daemon.conf:

high-priority = yes
nice-level = -11
realtime-scheduling = yes
realtime-priority = 8
no-cpu-limit = yes

Now restart pulseaudio:

pulseaudio -k ; pulseaudio -D

No more stuttering. :)

» FlashBlock WannaBe +

I've been using Midori as my primary browser for a number of months (I have a daily cron job that updates and builds WebKit and Midori). It's a fast little browser using WebKit for it's rendering engine and SFX for it's JS engine, so it's standards compliant and really, really fast. And it implements some handy, commonly used features (e.g., error console, user scripts / styles, element inspector).

One feature I've been missing from FireFox is the FlashBlock extension. For those of you who have never used FlashBlock, it essentially replaces flash elements in the page with a placeholder element, which, when clicked, displays the original flash content. It has some other bells and whistles (such as a custom context menu which allows you to directly copy the location of the flash file, or block flash elements on a per-object or per-domain basis, a toolbar button to enable or disable the extension, &c). However, it's the core behavior that I've been missing.

To remedy this situation I looked for user scripts to the emulate the FlashBlock behavior. To my chagrin, I only found scripts which partially emulate the behavior (with most of them doing so inefficiently). So, in a fit of "Not Created Here," and seeking functionality not yet available, I hacked up my own FlashBlock user script; thus was born FlashBlock WannaBe.

Here is version 0.1.0 of FlashBlock WannaBe, relased under the MIT license.

FlashBlock.user.js.txt (remove the .txt extension)
FlashBlock.css

» Automatic YouTube download with IE7 +

So my mom likes to collect silly little clips from YouTube, and she uses IE7 on Windows XP (I've tried to get her to convert to Linux, but she's not very computer saavy and has learned computing with Windows—even Ubuntu is a bit too much for her to take in). She normally just copies the YouTube url, pastes it to KeepVid, followed by a right-click + save-as, pasting in the video title as the file name in the save dialog instead of the default "video.mp4" (or "video.flv"), and finally clicking "OK." That's a lot of work for what should be a simple operation.

I had some free time over the holidays, so I hacked up a little user script and Windows Scripting Host script to allow automatically downloading files from YouTube with IE7. Now you might be thinking something like: "That's lovely, but you could have just downloaded Firefox, installed GreaseMonkey and FlashGot, writen a user script to re-write YouTube urls to add &fmt=18, and used the FlashGot icon in the statusbar to download the video." That's all very true, but as I said, my mother is used to Windows and IE—it was just easier to KISS than to teach her a new technology (not to mention that she watches Netflix movies online, which requires IE—using two browsers for different things would thoroughly confuse her).

Firstly, you need to install the Trixie plugin for IE (get it here), which in turn requires .NET.

Next you need to place the following user script in C:\Program Files\Bhelpuri\Trixie\Scripts:

YouTube.user.js

// ==UserScript==
// @name          Download YouTube videos
// @description   Creates a link to download YouTube videos directly
// @namespace     MonkeeSage@gmail.com
// @include       *
// ==/UserScript==

(function()
{
    // Trixie's @include directive seems broken using "*youtube.com/watch?*"
    // so we manually filter the url here
    if (location.href.indexOf("youtube.com/watch?") == -1)
        return;

    var match;
    var node;
    var videoId;
    var tParam;
    var videoTitle;

    var videoUrl     = "download://www.youtube.com/get_video?";
    var videoTitleRe = new RegExp("YouTube - (.+)", "m");
    var videoIdRe    = new RegExp("video_id=([^&,\"]+)", "m");
    var tParamRe     = new RegExp("\"t\": \"([^&,\"]+)\"", "m");

    try
    {
        match = document.title.match(videoTitleRe);
        if (match) videoTitle = match[1];

        match = document.body.innerHTML.match(videoIdRe);
        if (match) videoId = match[1];

        match = document.getElementsByTagName("head")[0].innerHTML.match(tParamRe);
        if (match) tParam = match[1];
    }
    catch (ex)
    {
        dump(ex.message);
    }

    if (videoId && tParam)
    {
        videoUrl += "video_id=" + videoId + "&t=" + tParam + "&fmt=18#" +
                    "C:\\Documents and Settings\\mom\\Desktop\\My Music&" +
                    videoTitle + ".mp4";
        a = document.createElement("a");
        a.appendChild(document.createTextNode("Download video"));
        a.href = videoUrl;
        node   = document.getElementById("watch-this-vid");
        node.appendChild(a);
    }

})();

Ps. You obviously need to change the C:\\Documents and Settings\\mom\\Desktop\\My Music to the default location you wish to save to.

Pss.You'll need to enable the script under Tools->Trixie Options (or Trixie Settings, or something similar—I'm going from memory here).

Next, you need to save the following file to C:\Program Files\Common Files\System:

wget.js

(function()
{
    function BrowseForFolder(path, file)
    {
        var objDialog = new ActiveXObject("SAFRCFileDlg.FileSave");
        var fso       = new ActiveXObject("Scripting.FileSystemObject");

        objDialog.FileName = fso.BuildPath(path, file);
        objDialog.FileType = "MPEG4 Video";

        var rv = objDialog.OpenFileSaveDlg();
        if (rv)
        {
            return objDialog.FileName
        }
        else
        {
            return false;
        }
    }

    function DownloadFile(url, file)
    {
        var data;
        var ado;

        try
        {
            var WinHttpReq = new ActiveXObject("WinHttp.WinHttpRequest.5.1");
   
            // ResolveTimeout, ConnectTimeout, SendTimeout, ReceiveTimeout
            WinHttpReq.SetTimeouts(30000, 30000, 30000, 5000);
   
            void(WinHttpReq.Open("GET", url, false));
            WinHttpReq.Send();
            if (WinHttpReq.Status == 404)
            {
                return false;
            }
            data = WinHttpReq.ResponseBody;
        }
        catch (ex)
        {
            WScript.Echo("Error downloading file: " + ex.message);
            return false;
        }

        ado = new ActiveXObject("ADODB.Stream");
        ado.Type = 1; // binary mode
        ado.Open();
        ado.Write(data);
        ado.SaveToFile(file, 2); // 2 = overwrite existing file
        ado.Close();

        return true;
    }

    function SetFileSuffix(filename, suffix)
    {
        var idx = filename.lastIndexOf(".");
        if (idx > -1)
        {
            filename = filename.slice(0, idx);
        }
        return filename + suffix;
    }

    var url;
    var file;
    var path     = "C:\\Documents and Settings\\mom\\Desktop\\My Music";
    var filename = "foo";

    var objArgs = WScript.Arguments;
    if (objArgs.length > 0)
    {
        url = objArgs(0).split("#");
        if (url.length == 2)
        {
            var tmp = url[1].split("&");
            if (tmp.length > 0)
            {
                path = tmp[0];
            }
            if (tmp.length > 1)
            {
                filename = tmp[1];
            }
        }
        url = url[0];
    }
    else
    {
        WScript.Echo("A URL argument is required");
        return;
    }

    filename = SetFileSuffix(filename, ".mp4");

    // replace the download:// pseudo-protocol with http://
    url = "http" + url.slice(8);

    file = BrowseForFolder(path, filename);
    if (file)
    {
        var rv = DownloadFile(url, file);
        if (!rv)
        {
            file = SetFileSuffix(file, ".flv");
            url  = url.replace("&fmt=18", "");
            rv   = DownloadFile(url, file);
        }
        if (rv)
        {
            var objWSH = new ActiveXObject("WScript.Shell");
            objWSH.Popup("Finished downloading " + filename, 10, "Download Complete", 64);
        }
    }

})();

Finally, you need to add a new protocol handler to the registery to handle the download:// pseudo-protocol. Save and run the following regedit file:

YouTubeDL.reg

Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\download]
@="URL:Download Protocol"
"URL Protocol"=""

[HKEY_CLASSES_ROOT\download\shell]

[HKEY_CLASSES_ROOT\download\shell\open]

[HKEY_CLASSES_ROOT\download\shell\open\command]
@="C:\\WINDOWS\\system32\\wscript.exe \"C:\\Program Files\\Common Files\\System\\wget.js\" \"%1\""

Now restart IE, and you should be able to download YouTube videos from a link directly under the video (on the left side).

Caveat: There is no progress dialog and no way to cancel downloads. I could probably figure out a COM dialog to display the progress, and run the HTTP GET asychonously (and cancel the Send operation when requested), but my mom didn't need those features, and I'm lazy, so I didn't. ;P

» String interpolation in python +

In python, the standard way to do string interpolation is to use the % operator. There are some bells and wistles to it, like named substitutions, but overall it behaves much the same as the C printf function on which it was modeled. On the other hand, in ruby you can use inline interpolation with #{} inside a string, which combines perl's ${} and $() into one. I personally prefer the latter method. As of 2.4, python has a similar interpolation method, using the string.Template module, however it is still not as terse (and imo, clean). So here is a simple function to emulate the ruby way (with both expression and variable interpolation).

import sys, re
def interp(string):
  locals  = sys._getframe(1).f_locals
  globals = sys._getframe(1).f_globals
  for item in re.findall(r'#\{([^{]*)\}', string):
    string = string.replace('#{%s}' % item,
                            str(eval(item, globals, locals)))
  return string

test1 = 'example'

def tryit():
  test2 = 1

  # variable interpolation
  print interp('This is an #{test1} (and another #{test1}) and an int (#{test2})')

  # expression interpolation
  print interp('This is an #{test1 + " (and another " + test1 + ")"} and an int (#{test2})')

  # standard way
  print 'This is a %s and a %s and an int (%d)' % (test1, test1, test2)

  # since 2.4
  from string import Template
  map = sys._getframe(0).f_globals
  map.update(sys._getframe(0).f_locals)
  print 'This is a %(test1)s and a %(test1)s and an int (%(test2)d)' % map
  print Template('This is a $test1 and a $test1 and an int ($test2)').substitute(map)

tryit()

I don't mind using the printf style of interpolation, but I prefer the inline method better personally.

[Edit:] Added a cookbook recipe for this.

» New Happy Day! +

Err, Happy Merrynox! Doh. That's not right either. Holiyears Newmas Kwanza-Chana-Peruvian Llama Shedding Merry Days! Oh, I give up. I can't keep all these new joined-together, uber, all-encompassing holidays strait. I celebrate Christmas and the western New Year. So Merry Christmas and Happy New Year! And if you don't celebrate them, well, I hope you enjoyed the days off of work!

So I'ven been taking a bit of a break from the blogs and newsgroups and just kind of vegging out with movies in my free time. I highly recommend Shutter -- a really scary Thai movie in the "J-Horror" genre. Also, a South-Korean flick worth seeing if you're into psychological horror (think Session 9 and The Machinist) is A Tale of Two Sisters, which I very much enjoyed. Or if action and suspense is more your thing, you might check out Sympathy for Mr. Vengeance or Oldboy, the first and second films of the "Vengeance Triology" from writer/director Park Chan-Wook of South Korea. Good stuff.

And with that, I must note that most of the good cinema in the last 5-10 years is coming out of Asia. From Tony Jaa's Ong-Bak and The Protector, to Stephen Chow's A Chinese Oddysey (both parts), Shaolin Soccer and Kung Fu Hustle. Even the more obscure films like Dream of a Warrior put most of the mainline Hollywood flicks to shame. This, of course, is only my inexpert oppinion. But, well, good job Asia! Keep it up!

So anyway. I'm back. And I'm very happy to see YARV land in the trunk! Python 2.5 is working well. GCC 4 doesn't break my machine now. God is in his place in Heaven, and all is right with the world. Yay.

» Scripting Gedit +

Gedit allows you to write python scripts which interface with its backend (and frontend via pygtk). This is very cool, for reasons obvious to VIM and Emacs users. You can write your own plugins to manipulate the document you are editing in many useful ways.

Well I was using various external tools to run my various scripts for test purposes (the External Tools plugin is itself written in python!). So I'd bind F5 to ruby, F6 to python and F10 to perl. Then, depending on the script I was editing, I'd press the corresponding accel key to run it. But I realized that I could just write a plugin that would run any script according to its bang line. The following is the plugin I came up with (place it in ~/.gnome2/gedit/plugins):

First is the plugin definition file (named script_runner.gedit-plugin):

[Gedit Plugin]
Loader=python
Module=script_runner
IAge=2
Name=Script Runner Plugin
Description=This plugin runs scripts by reading the bang line.
Authors=Jordan Callicoat <MonkeeSage@gmail.com>\nJonathan-Marc Lapointe
Copyright=Copyright © Jordan Callicoat, 2006
Website=http://rightfootin.blogspot.com

Then is the actual python script file (named script_runner.py):

#  Script Runner plugin
#  coding: utf-8
#
#  Copyright © 2006 Jordan Callicoat
#  Copyright © 2008 Jonathan-Marc Lapointe
#  Released under the python license.

import re
import os
import gtk
import gedit
from externaltools.functions import capture_menu_action
from externaltools.ElementTree import Element

class RunScriptPlugin(gedit.Plugin):
  """
  A simple plugin that runs the current document
  in the interpreter specified by the bang line
  """

  def run_document(self, action, window):
    """
    Just Do It!
    """
    # read in the first line of the buffer...
    buff  = window.get_active_view().get_buffer()
    siter = buff.get_iter_at_line_offset(0,0)
    eiter = buff.get_iter_at_line_offset(1,0)
    data  = buff.get_text(siter, eiter)
    # now check if it contains a bang line...
    if data:
      text = re.match(r'^#!(.*)$', data)
      if text: # it does...
        text = text.group(1)
        # extract a label (for use in the output
        # panel) from the bang line...
        label = os.path.basename(text).split(' ', 1)
        if label[0].lower() == 'env':
          label = label[1]
        else:
          label = label[0]
        # build an ElementTree Element to feed capture_menu_action...
        elem         = Element('tool')
        elem.command = text
        elem.name    = label # .title()
        elem.input   = 'document'
        elem.output  = 'output-panel'
        # now run the script!
        capture_menu_action(None, window, elem)

  def activate(self, window):
    """
    Setup stuff
    """
    actions = [
      ('Run', gtk.STOCK_EXECUTE, 'Run Script', 'F5', 'Run Script', self.run_document),
    ]
    # store per window data in the window object
    windowdata = {}
    window.set_data('RunScriptPluginWindowDataKey', windowdata)
    windowdata['action_group'] = gtk.ActionGroup('GeditRunScriptPluginActions')
    windowdata['action_group'].add_actions(actions, window)
    manager = window.get_ui_manager()
    manager.insert_action_group(windowdata['action_group'], -1)
    ui_str = """
    <ui>
      <menubar name="MenuBar">
        <menu name="ToolsMenu" action="Tools">
          <placeholder name="ToolsOps_3">
            <menuitem name="Run" action="Run"/>
            <separator/>
          </placeholder>
        </menu>
      </menubar>
    </ui>
    """
    windowdata['ui_id'] = manager.add_ui_from_string(ui_str)
    window.set_data('RunScriptPluginInfo', windowdata)

  def deactivate(self, window):
    """
    Teardown stuff
    """
    windowdata = window.get_data('RunScriptPluginWindowDataKey')
    manager    = window.get_ui_manager()
    manager.remove_ui(windowdata['ui_id'])
    manager.remove_action_group(windowdata['action_group'])

  def update_ui(self, window):
    """
    UI Callback
    """
    view       = window.get_active_view()
    windowdata = window.get_data('RunScriptPluginWindowDataKey')
    windowdata['action_group'].set_sensitive(bool(view)) # and view.get_editable()))

After you add these files, you need to restart Gedit. Then you go to Preferences -> Plugins tab -> check the Script Runner Plugin option, and you're set. Now pressing F5 will automagically run any script with a bang line. Cool stuff! :)

Note: The External Tools plugin needs to be enabled for this script to work! Updated to work with newer gedit versions, thanks Jonathan.