Ivo's blog

I’m knee deep in web development. I also always like to look at websites from a security point of view. This often means I end up looking at http headers, source code, error pages and so on to see what software a site is running and what its vulerabilities might be.

Eventhough this may sound like an odd hobby, I bet there are other people who do something similar, and to save me and those others some time I’ve written two tools to help in figuring out what kind of software a site is running: Fetch and Guess

I hope these tools can be of use to web developers, SEO consultants and security consultants, or just anyone who’s interested in seeing what’s happening behind the (browser) screens

All of this has also allowed me to develop a powerful toolkit to scan/analyze websites which I plan to use for other future projects, SEO and security related, but I can for example also imagine an ISP may find it useful to analyze what kind of software their users are actually running

I’ve blogged about Fetch before so I won’t get into too much detail. It’s basically the web equivalent of telnet host.tld 80 and doing a GET or HEAD by hand, but then a lot more user friendly, verbose and feature-rich.

Guess analyzes a site and tries to figure out what Webserver, language and framework a site is running. If possible, it will also attempt to find the versions used

Guess is learning about new software stacks each day and it already has an impressive hit rate (at least the sites that I usually try it on), though some very obvious systems still aren’t detected.

I’ve also planned much more features such as

• Javascript library detection
• Details about stats tools used
• OS details
• Plugins, options, modules used/installed

Both tools can be used as bookmarklets, simply copy the url to your bookmarks toolbar and clicking it will open the site you have open a new window with either Fetch or Guess

• Fetch bookmarklet
• Guess bookmarklet

I’m also planning on developing simple Chrome and Firefox extensions, but adding more software stacks and features have a higher priority; the bookmarklets actually work really well.

I’ve moved my old blog to a new domain and new software. Popular postings have been migrated, the rest remains at the old blog/site as an archive. As a bonus, comments are now finally supported.

blog.m3r.nl started as a quick hack - I just wanted to share some thoughts and code, but it’s time to give it a more appropriate name and more suitable software. The old blog ran Plone and eventhough there are sufficient blogging products for it it’s always pretty hard to not make it look like Plone anymore.

I wanted the new blog to run some sort of Python blogging software. I have considered Django Mingus, but it looks too alpha for easy deployment. I ended up using Zine which seems pretty mature and feature complete. However, now that I am using it there are some things I’d like to improve, such as:

• I don’t like the included text parsers. I actually like WYSIWYG editors in stead of learning yet another markup language (which Zine-markup can’t actually be considered)
• There’s no way to add images to a posting. You need to upload them elsewhere.
• A modified skin probably, although the default looks fine for now
• Migration options are really limited, migrating to the database directly is not really supported. I’ve hand migrated some articles so that’s not really an issue anymore.
• It uses the retarded US date format M/D/Y in the admin interface, and I see no option to fix this. Probably setting a locale somewhere in the WSGI publication chain.
• The editing workflow is rather primitive, no real preview option.
• It’s too hard to link to another blog entry - you have to copy/craft the url by hand.

Usually I end up considering writing my own variation of the software. Lack of time is what keeps me from reinventing these wheels over and over again :)

All of these things can be accompilished using Zine’s plugin mechanism so it’s worh looking into that. Unfortunately, documentation is seriously lacking.

My old blog is dead, it won’t get updates. You can unsubscribe from the RSS feed and subscribe to this blogs Atom feed in stead.

I just received the iRiver Story EB02 e-reader I blogged about earlier. The first steps of course are:

• hook it up (to load the battery and to upload some books)
• update the firmware (this really is a must)

Hooking it up

You need to explicitly select if it connects as a storage device or if it should just charge. I'm not sure if it charges when used as an external disk.

Disconnecting on OSX (unmounting) seems to be a problem - it immediately reconnects.

Updating the firmware

Make sure you save the firmware in UPPERCASE! It explicitly checks foor EBOOK.HEX, ebook.hex won't work!

It's unclear to me how to properly turn the device off (you don't really need to actually!). Sliding the hold key to the right will simply lock the device.

Other findings

• There's no mention anywhere of GPL'd or otherwise OSS being used/licensed (it uses at least the linux kernel and busybox)
• while reading selecting options->reflow (on->off) will greatly improve readability, at least with the PDF documents I tried.
• I'm positively surprised by the responsiveness and speed of the device. It's snappy enough for me.
  

I recently purchased an iRiver Story EB02 e-reader. I haven't actually received it yet but I was curious about its details and the type of software it would run.

I learned it's an abolute must to download / install the latest firmware so I decided to have a look at that for now as I didn't have a device to install it on anyway.

The firmware is called "ebook.hex" which is a bit mysterious. It turns out to be an ordinary zipfile with some (128 bytes) leading garbage:

0000000   i   r   i   v   e   r       N   e   t   w   o   r   k       T
0000020   e   a   m       Y   o   o       H   y   u   n   g       S   e
0000040   o   u   n   g       a   j   f   e   o   q   n   a   n   g   h
0000060   o   q   p   o   e   a   ]   [   [   k   l   m   .   /   .   .
0000100   ,   m   z   l   l   o   e   =   -   0   j   . 376   _   @ 324
0000120  \b 206 236 351 026 242 036   n 350 335   Q   a   c   1   8  \0
0000140   .   \   _   p   r   e   _   w   o   r   k   .   b   a   t  \0
0000160   .   \   E   B   O   O   K   .   y   h   s  \0   C   r   e   a
0000200   P   K 003 004 024  \0  \t  \0  \b  \0 261   m   3   <   (   G

Unpacking the zipfile will prompt for a password, however. You'll be able to find the password relatively easy using the following command, though:

$ fcrackzip -c "a" -p aaaaaaaa -v ebook.hex

The OS

The OS the iRiver Story appears to be running is (big surprise!) Linux! It appears to be for an ARM based system, using Cairo for rendering the e-books. The most suprising however is that I didn't see any GPL notices or links to source code anywhere. Let's hope that's handled better in the package when it arrives, this is starting to smell a bit like a GPL violation.

It contains a ramfs which you can easily mount

# mount -o loop rootfs.cramfs /mnt/iriver-mnt

Some observations

• on the rootfs, /tmp/status_managers is responsible for handling upgrades (and general system startup probably). Using "strings -a" you will again find the ZIP password. It also appears to try something with "ebook.yhs.
  
• the zipfile itself contains an app/ directory with some "real" binaries (book2pngd, Jmp3_player_copy and mattrib). The other files are ".feb" files and I'm not sure what they are. They're binary with some plaintext strings included. Perhaps native ARM, perhaps interpreted. The binary "/flow_copy" appears to be responsible for starting "start.feb" which in turn may be responsible for starting the other .febs.
  
• booting (or at least starting stuff) from sdcard seems possible. The following snippet comes from rootfs' /etc/init.d/rcS

############ Detect SD Booting Movi Booting ###
echo 157 > /sys/class/gpio/export
if [ "`cat /sys/class/gpio/gpio157/value`" != "0" ]; then
#SD Booting
        mount -t vfat -o shortname=mixed /dev/mmcblk1p1 /mnt/SDFAT
        # SD Card에 elisamake_sh 가 있다면 
        if [ -f /mnt/SDFAT/factory/elisamake_sh ]; then
        echo "##### SD Booting #####"
        echo "##### Run script /mnt/SDFAT/factory/elisamake_sh #####"
        dos2unix /mnt/SDFAT/factory/elisamake_sh
        /mnt/SDFAT/factory/elisamake_sh
        sync
        else
        #MOVI NAND fdisk
        echo "##### SD Booting Start Fdisk MOVI NAND #####"
        fdisk -u -S 16 -H 1 /dev/mmcblk0 < /etc/init.d/sfdisk
        mkfs.vfat -n Story -F 16 /dev/mmcblk0p1
        /tmp/mke2fs /dev/mmcblk0p2
        sync
        fi
fi

• firmware 1.61 uses Linux kernel 2.6.28.6, Linux version 2.6.28.6 (root@jang) (gcc version 4.3.2 (GCC) ) #706 PREEMPT Tue Dec 15 11:59:43 KST 2009
  
• it uses/runs busybox
• it uses SQLite
• There may be WIFI support someday. Perhaps in this device (if it carries the hardware) or else some future device. The following is the contents of the settings.xml file:

<?xml version="1.0" encoding="utf-8"?>
<system_setting>
<setting_info>
  <passwd>NULL</passwd>
  <title1>NULL</title1>
  <title2>NULL</title2>
  <name>iriver</name>
  <phone>0000</phone>
  <time>200901010900am</time>
  <lang>2</lang>
  <shortkey>YURTF</shortkey>
  <font>NULL</font>
  <power>15</power>
  <dic>DD</dic>
  <wifi>NULL</wifi>
</setting_info>
</system_setting>

More may follow once I actually have the device :)

Misc.

A happy blogging iRiver Story user

iRiver story firmware download

One of my toy projects has been running nicely over the past few months so I guess it's time to release it into the wild:

Fetch! 

It fetches url's and provides all kinds of information such as:

• headers
• embedded links, images
• encoding information
• script references

It turns out to be a pretty usefull tool when doing simple SEO analysis, investigating what kind of platform a site uses, testing encoding issues, testing authorization, etc. I've been using it quite alot and I hope it can be useful to others as well.

A bookmarklet is provided - copying it to your bookmark toolbar will allow you to analyze the site you're currently visiting.

This tool is not unique. Besides the obvious tools that provide alot of similar functionality (i.e. Firebug) there are also online services that provide similar functionality. However, most of the online tools aren't as extended as Fetch.

Some improvements I have planned:

• coloured html
• working redirect handling, POST/HEAD support
• cookie decoding
• P3P decoding

Let me know if you find it useful!

I bought a Marmitek IP Robocom 8 wireless (which also supports wired networks) about a year ago. It seems to be discontinued by Marmitek by now.

I've been researching if there are any interesting firmware upgrades or other (OSX) software available, but Marmitek doesn't supply any. However, I found out that the camera is basically a TRENDnet TV IP400W camera in a different encosure (and slightly modified firmware). Or, alternatively, they're both based on the same (unknown) OEM hardware/software.

In case you ever need anything for this camera (software, software support in software, etc), it may be good to know it's essentially the TV-IP400W. I haven't tried the IP400W firmware on my Marmitek but it just might work.

Some stuff I found:

• TV-IP400W emulator. Really similar to the Marmitek. Some pages that are missing on the Marmitek (i.e. /About.htm) can actually be subsituted by hand
• TV IP400W software, including firmwares. More recent than the "3.25" that the Marmitek has.
• GPL sources. The 410W is listed but eventhough similarities (and similar version number) I don't think the IP400W is running any sort of linux.
• Reverse-engineerd details
  
• Another member of the family: NCS-370W
• Get the cam to work with ZoneMinder, by Peter Hulst
  
• Stil no decent OSX support
  

I'm currently looking at some Google IO android talks on youtube and I was annoyed by the fact that eventhough I have a brand new core2duo 2.4ghz laptop, videoplayback was still horrible.

Eventhough this is Flash video playback as I know it I had a look of there are any known solutions. And there are!

1. Create an /etc/adobe/mms.cfg containing 

OverrideGPUValidation=true

2. Install a CPU frequency monitor and adjust your cpu speed settings.

Step 1 was sufficient for me to fix my problems and have an (almost) smooth playback, but forcing your CPU into a higher speed may sometimes also be a good solution.

Links

http://blogs.adobe.com/penguin.swf/2008/08/secrets_of_the_mmscfg_file_1.html

http://allredb.wordpress.com/2009/05/07/speed-up-flash-and-firefox-in-ubuntu-jaunty-904/

http://webupd8.blogspot.com/2009/07/speed-up-flash-and-firefox-in-ubuntu.html

I recently purchased a Samsung Galaxy i7500 when it became available in the Netherlands. It's already been available in Germany and (I think) the UK for a while so it's not even that cutting edge.

In this posting I'll document some of my findings.

First Impression

My first impression which also includes working with Android for the first time was: what an amazing device! The included applications are fantastic, especially with the included hardware and this finally is a device which I can use to chat, mail, consult google maps and surf sites. I've heard people complain about battery life and (keyboard) slowness, but those are okay with me.

I was actually surprised how well the on-screen keyboard works.

However

I was so focussed on the software and gadgetvalue of the device that I didn't pay any real attention to calls. And that's an area where the Galaxy behaves horrible.

• Sometimes the remote party becomes siltent for a fraction of a second during a conversion. Very annoying because you'll miss parts of words. I understand it's both ways, I've heard the remote part complaining I was interrupted by silence
• Sometimes it can't make or receive calls. I can't really define what the problem is - poorer T-Mobile reception perhaps, possibly combined with all the onboard hardware. Or just buggy hardware/software. It's hard to say
• The screen dimms a second after you've made a call which is very confusing. I'm still not sure if it's some feature, just the screensaver/lock kicking in or something else. But if you need to enter numbers or access the menu, it's very confusing

I've attempted to do a firmware upgrade but eventhoug it's reported there are updates, there isn't one for my version. It may be I just don't understand Samsung's horrible "Samsung PC Studio" which, eventhough it's the only available download for the Galaxy, doesn't recorgnize the device without some serious tweaking (and even then I'm not sure). 

Besides the pretty big call issue above, there are some minor issues

• it doesn't always find local wireless networks. Excplicitly searching the network and connecting to it will result in a password not accepted. The fix? Reboot the device :(
• the gallery sometimes shows good pictures as being damaged. Restarting the app fixes this (the pictures really aren't broken), eventhough, when I didn't know the pics were okay, it made me delete pictures that were actually perfectly fine.

Conclusion

The Galaxy may have slightly better specs/looks than the HTC Hero, but for now I'd chose a phone that can actually make calls. Either wait for updates from Samsung, new Android devices or buy a Hero, but at this point I wouldn't advice anyone to buy the Samsung Galaxy i7500

Update

It appears most calling isses are caused by poor T-mobile reception. Switching providers fixed most of these.

After using the device for a couple of months, I find it usable. There still are very annoying bugs (wifi, e-mail, dialpad doesn't respond while calling). Also, by now, Android 1.5 is really old, and it's still not clear if there will be an update.

Plone provides the plone.protect.authenticator view to protect against Cross Site Request Forgery (CSRF).

However, this makes unit-testing such code slightly more difficult. I'm currently using the following hack, if anyone can point me to a better direction.

Imagine a BrowserView Foo with the following __call__

class Foo(BrowserView):
...
    def __call__(self):
        ...
        authenticator = self.context.restrictedTraverse('@@authenticator', None) 
        if not authenticator.verify(): 
            raise Forbidden
        ...

Essentially, this means you need to provide the correct _authenticator formvalue in self.app.REQUEST.form in your unittest. The following method handles this:

    def _getauth(self):
        import re

        authenticator = self.portal.restrictedTraverse("@@authenticator")
        html = authenticator.authenticator()
        handle = re.search('value="(.*)"', html).groups()[0]
        return handle

You can use it in your test as follows:

    def test_foo(self):
        self.app.REQUEST.form['_authenticator'] = self._getauth()
        foo = self.portal.restrictedTraverse("@@foo")
        foo()

Works for me! (at least for now)

I'm not a big fan of wordprocessors like OpenOffice.org or MS Office for writing technical documentation. I always end up focussing on how things are outlined and styled instead of the contents. Especially with technical documentation, content and structure are more important (in my opinion). Also, I like typing text in Vim :)

That's why, when I recently had to write documentation for a project, I went searching for a structured text based solution . After asking around a bit, asciidoc seemed to be a good option. There are many structured text solutions for the web, but only a few that will work standlone from the commandline (and support conversion to multiple target formats).

Coincidentally, I ran into "ditaa" which converts ascii linedrawings to png diagrams. Combining these two would mean I could do everything from within a single document that, by itself, remains perfectly readable but also renders nicely to HTML and PDF (and, therefore, print).

Combining the two required a small pythonscript that acts as a filter. It parses your asciidoc document and scans for '[ASCIIART ..]' and '[TRAIICSA]' markers. Everything in between is fed to ditaa, and an asciidoc image reference is included in the filtered document, which can then be passed to asciidoc for forther processing. You are required to supply a filename which will be created and included, and you can optionally supply arguments between parenthesis which are passed directly to the asciidoc image: markup. By default, this means the title for the document.

I.e. the following markup:

= Sample =

- look at the nifty
- drawings!

[ASCIIART sample.png (Sample drawing)]
/----------\  /--------\  /---------\
| asciidoc |  | mumble |  |         |
|    +     |->| mumble |->| PROFIT! |
|  ditaa   |  | mumble |  |         |
\----------/  \--------/  \---------/
[TRAIICSA]

Will result in the following asiidoc markup after filtering

= Sample =

- look at the nifty
- drawings!

image:sample.png[Sample drawing]

which will render to the following HTML (somewhat scaled)

The script ("handleimages.py") is rather trivial and doesn't handle errors too well (actually, not at all). But for me it gets the job done.

#!/usr/bin/env python
import sys
import re
import subprocess

header = re.compile("\[ASCIIART\s+(\S+?)\s*\((.*?)\)?\]")

def main(document):
    parsed = []
    parsing = False
    filename = ""
    worklist = {}
    for line in document:
        if line.strip().startswith("[ASCIIART"):
            filename, flags = header.match(line).groups()
            if flags:
                flags = "[%s]" % flags
            else:
                flags = ""
            parsed.append("image:%s%s\n\n" % (filename, flags))
            parsing = True
            worklist[filename] = ""
        elif line.strip() == "[TRAIICSA]":
            a = open("/tmp/asciiart.txt", "w")
            a.write(worklist[filename])
            a.close()
            parsing = False
            job = subprocess.Popen(["java", "-jar", "ditaa0_6b.jar", "-o", "/tmp/asciiart.txt", filename], stdout=subprocess.PIPE)
            job.wait()
        elif parsing:
            worklist[filename] += line
        else:
            parsed.append(line)

    print "".join(parsed)

if __name__ == "__main__":
    if len(sys.argv) == 2:
        document = open(sys.argv[1], "r").readlines()
    else:
        document = sys.stdin.readlines()
    main(document)

Invoke it, for example, using the following script ("create.sh")

#!/bin/sh 

if [ "$1" = "" ]; then
  echo "Specify an asciidoc file"
  exit
fi

TARGET=`basename $1 .txt`.html

./handleimages.py $1 | asciidoc -d book -o $TARGET - 

There's probably some plugin/extension system for asciidoc that I can use for a cleaner, tighter integration. But as I said, for now, this gets the job done.