Web utility methods

Utility methods for retrieving and manipulating data from Web resources.

class omnipresence.web.WebCommand

A utility class for writing command plugins that make a single HTTP GET request and do something with the response.

Subclasses should define a url property containing the string %s, and implement the reply() method. When the command is invoked, %s is substituted with the command’s literal argument string, and a deferred request to the resulting URL is made with reply() as its success callback.

An optional property arg_type can be used to indicate the type of argument that your custom command expects. This is used to provide a usage message should no arguments be given; for example, setting arg_type to 'a search term' sets the usage message to “Please specify a search term.” The default value is 'an argument string'.

reply(response, bot, prefix, reply_target, channel, args)

Implement this method in your command subclass. The response argument will contain a (headers, content) response tuple as returned by request(). The other arguments are as passed in to ICommand.execute().

omnipresence.web.decode_html_entities(s)

Convert HTML entities in a string to their Unicode character equivalents. This method is equivalent to:

textify_html(s, format_output=False)

Deprecated since version 2.2: Use textify_html() instead.

omnipresence.web.request(*args, **kwargs)

Make an HTTP request, and return a Deferred that will yield an httplib2-style (headers, content) tuple to its callback.

Arguments are as for a request to a typical Twisted Web agent, with the addition of one keyword argument, max_bytes, that specifies the maximum number of bytes to fetch from the desired resource. If no User-Agent header is specified, one is added before making the request.

Two custom headers are returned in the response, in addition to any set by the HTTP server: X-Omni-Location contains the final location of the request resource after following all redirects, and X-Omni-Length contains the original value of the response’s Content-Length header, which Twisted may overwrite if the actual response exceeds max_bytes in size.

omnipresence.web.textify_html(html, format_output=True)

Convert the contents of html to a Unicode string. html can be either a string containing HTML markup, or a Beautiful Soup tag object. If format_output is True, IRC formatting codes are added to simulate common element styles.