Tuesday, August 9, 2011

Java 7 on Ubuntu 11.04

So after 5+ years, we finally have a new version of Java. In spite of mixed reviews, as a software engineer working a lot on the JVM, I still wanted to take it for a spin. However, being an Ubuntu user, this involves a tad bit of work since there are only RPM and binary packages available at this time.

So until there are official or unofficial .deb packages available, we're going to have to sneak in the raw binaries outselves. The following is how I did this on my 64bit Ubuntu 11.04 system. I take no responsibility in breaking your existing Java installation yada yada, you do this at your own responsibility.

Procedure
Start by downloading the binary.

casper@workstation:~$ wget http://download.oracle.com/otn-pub/java/jdk/7/jdk-7-linux-x64.tar.gz

Then unpack it.

casper@workstation:~$ tar -xzf jdk-7-linux-x64.tar.gz

Remove packed archive and move/rename the extracted folder into something a little more Ubuntu appropiate.

casper@workstation:~$ sudo rm java-7-sun-1.7.0.tar.gz
casper@workstation:~$ sudo mv jdk1.7.0 /usr/lib/jvm/java-7-sun-1.7.0

Follow the Ubuntu tradition, by symlinking a generic major java version to a concrete minor version.

casper@workstation:~$ sudo ln -s /usr/lib/jvm/java-7-sun-1.7.0 /usr/lib/jvm/java-7-sun

Override Ubuntu's standard alternatives mechanism, replacing links to Java 6 with links to Java 7. It would be better to use the update-alternatives tools available in Ubuntu, but I was unable to make this work. If you screw something up here, you may have to uninstall and reinstall your JDK/JRE stuff, in order to revert to a working Java installation.

Delete any existing symlinks.

casper@workstation:~$ sudo rm /etc/alternatives/java
casper@workstation:~$ sudo rm /etc/alternatives/javac
casper@workstation:~$ sudo rm /etc/alternatives/javadoc
casper@workstation:~$ sudo rm /etc/alternatives/javah
casper@workstation:~$ sudo rm /etc/alternatives/javap
casper@workstation:~$ sudo rm /etc/alternatives/javaws
casper@workstation:~$ sudo rm /etc/alternatives/java_vm

Create the new symlinks.

casper@workstation:~$ sudo ln -s /usr/lib/jvm/java-7-sun/jre/bin/java /etc/alternatives/java
casper@workstation:~$ sudo ln -s /usr/lib/jvm/java-7-sun/bin/javac /etc/alternatives/javac
casper@workstation:~$ sudo ln -s /usr/lib/jvm/java-7-sun/bin/javadoc /etc/alternatives/javadoc
casper@workstation:~$ sudo ln -s /usr/lib/jvm/java-7-sun/bin/javah /etc/alternatives/javah
casper@workstation:~$ sudo ln -s /usr/lib/jvm/java-7-sun/bin/javap /etc/alternatives/javap
casper@workstation:~$ sudo ln -s /usr/lib/jvm/java-7-sun/bin/javaws /etc/alternatives/javaws
casper@workstation:~$ sudo ln -s /usr/lib/jvm/java-7-sun/jre/bin/java_vm /etc/alternatives/java_vm

Finally, we may aid various client software in locating where Java in installed, by adding a few environment variables. Do this by editing .bashrc in your home directory.

casper@workstation:~$ gksudo gedit $HOME/.bashrc

Scroll down to the end of the file and add two new lines:

export JDK_HOME=/usr/lib/jvm/java-7-sun
export JAVA_HOME=$JDK_HOME

Restart and that's pretty much it, this worked for me, hope it works for you too. Otherwise, throw me a comment below.

Update: If you are unable to run applets wtihin your browsers, this may be caused by the existing Java Plug-in 1.6 clashing with the new incompatible JRE 1.7 you just installed. To remidy this, you can launch the Java Control Panel and disable the JRE 1.7, making 1.6 the only (and thus default) one.


As you can see below, you still retain Java 1.7, but applets and webstart won't try to use any of this new stuff.


Friday, May 20, 2011

Optware/ipkg on Synology DS211+


Modding
Note that the following covers the DS211+ device, which is characterized by being based on the Marvell Kirkwood mv6282 ARM core (1.6Ghz version of 1.2Ghz mv6281). AFAIK the content of this blog entry should be equally applicable to all DS211 variations, but you should consult the Synology wiki for your device to be certain.

While quite feature rich out-of-the-box, developer types will likely want to make their NAS do even more, either out of a real need or just to tinker. For instance, I eventually plan to connect my USB weather station and generate live charts. Because of the modding desire, Synology has a faily comprehensive wiki about the subject. To make a long story short, ease of modding is greatly assisted by the possibility of installing Optware package manager.

Installing Optware
Optware (ipkg) is an online package system a la what you may know from Debian/Ubuntu. This makes it easy to locate, install and uninstall compiled binaries (*.ipk) through the command-line.

Bootstrapper and Optware packages are maintained by the NSLU2-Linux community for use with various modifiable hardware devices incl. many Synology products. The installation procedure simply involves you downloading and installing the Optware-bootstrapper for your NAS. Here's what I wrote while logged in as root via SSH/telnet, to get it working.

Download the Optware bootstrapper:


NAS> cd /volume1
NAS> wget http://wizjos.endofinternet.net/synology/archief/syno-mvkw-bootstrap_1
.2-7_arm-ds111.xsh
--22:58:07--  http://wizjos.endofinternet.net/synology/archief/syno-mvkw-bootstrap_1.2-7_arm-ds111.xsh
           => `syno-mvkw-bootstrap_1.2-7_arm-ds111.xsh'
Resolving wizjos.endofinternet.net... 94.212.228.25
Connecting to wizjos.endofinternet.net|94.212.228.25|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 251,866 (246K) [text/plain]

100%[====================================>] 251,866      435.14K/s             

22:58:08 (434.50 KB/s) - `syno-mvkw-bootstrap_1.2-7_arm-ds111.xsh' saved [251866/251866]


Start the installation:


NAS> sh syno-mvkw-bootstrap_1.2-7_arm-ds111.xsh 
Optware Bootstrap for syno-mvkw.
Extracting archive... please wait
bootstrap/
bootstrap/optware-bootstrap.ipk
bootstrap/wget.ipk
bootstrap/bootstrap.sh
bootstrap/ipkg.sh
1227+1 records in
1227+1 records out
bootstrap/ipkg-opt.ipk
Creating temporary ipkg repository...
Installing optware-bootstrap package...
Unpacking optware-bootstrap.ipk...Done.
Configuring optware-bootstrap.ipk...Modifying /etc/rc.local
Done.
Installing ipkg...
Unpacking ipkg-opt.ipk...Done.
Configuring ipkg-opt.ipk...WARNING: can't open config file: /usr/syno/ssl/openssl.cnf
Done.
Removing temporary ipkg repository...
Installing wget...
Installing wget (1.12-2) to root...
Configuring wget
Successfully terminated.
Creating /opt/etc/ipkg/cross-feed.conf...
Setup complete.

BusyBox v1.16.1 (2011-04-08 10:13:55 CST) built-in shell (ash)
Enter 'help' for a list of built-in commands.

NAS> 

Note, if you get the message "Error: CPU not Marvell Kirkwood, probably wrong bootstrap.xsh", this is likely due to a too agressive sanity check (mv6281 but not mv6282) in the install script. If the output of the command "cat /proc/cpuinfo" mentions a "Feroceon" CPU, then you may safely open up the file "/volume1/bootstrap/bootstrap.sh" and edit line 21, changing "Feroceon-KW" to just "Feroceon", and run the script manually with "sh /volume1/bootstrap/bootstrap.sh".

The above will install a boot hook and create the directories /volume1/bootstrap and /volume1/@Optware where package manager and installed packages will reside. The content in /volume1/@Optware can also be found in /opt, due to an Optware bind mount requirement you can verify by invoking the mount command:


NAS> mount | grep @optware
/volume1/@optware on /opt type bind (bind)

Now reboot the NAS:


NAS> reboot


Managing packages
After a reboot, you can update the list of packages with:


NAS> ipkg update
Downloading http://ipkg.nslu2-linux.org/feeds/optware/cs08q1armel/cross/unstable/Packages.gz
Inflating http://ipkg.nslu2-linux.org/feeds/optware/cs08q1armel/cross/unstable/Packages.gz
Updated list of available packages in /opt/lib/ipkg/lists/cross
Successfully terminated.
NAS> 
You can list all packages with "ipkg list" or search specific packages with "ipkg list [wildcard]".


NAS> ipkg list cpu* 
cpufrequtils - 006-2 - To make access to the Linux kernel cpufreq subsystem easier for users and cpufreq userspace tools, a cpufrequtils package was c
Successfully terminated.
NAS> 


To install a package, simply type "ipkg install [name]". You may also browse the available packages by accessing the repository directly from within a browser.


NAS> ipkg install iperf
Installing iperf (2.0.4-1) to root...
Downloading http://ipkg.nslu2-linux.org/feeds/optware/cs08q1armel/cross/unstable/iperf_2.0.4-1_arm.ipk
Configuring iperf
Successfully terminated.
NAS> 

Removing packages is just as easy as installing:


NAS> ipkg remove iperf 
Removing package iperf from root...
Successfully terminated.
NAS> 

To make sure all installed Optware packages are up to date:


NAS> ipkg upgrade     
Upgrading grep on root from 2.7-1 to 2.8-1...
Downloading http://ipkg.nslu2-linux.org/feeds/optware/cs08q1armel/cross/unstable/grep_2.8-1_arm.ipk
Configuring grep
update-alternatives: Linking //opt/bin/grep to /opt/bin/grep-grep
update-alternatives: Linking //opt/bin/egrep to /opt/bin/grep-egrep
update-alternatives: Linking //opt/bin/fgrep to /opt/bin/grep-fgrep
Successfully terminated.
NAS> 


Uninstalling Optware
To my knowledge, there isn't a one-liner for uninstalling the Optware system. It involves stopping the Optware service, unmounting the /opt bind mount, deleting the folder/files, editing the Linux service script and rebooting:

NAS> /etc/rc.optware stop
NAS> umount /opt
NAS> rm -rf /volume1/@optware
NAS> rm -f /etc/rc.optware
NAS> echo -e "#!/bin/sh\n" > /etc/rc.local
NAS> reboot

Note that the above wipes the rc.local script. If you have added custom stuff beyond just Optware, you should edit out the Optware stuff manually!

Of course, there aren't as many packages available as one would like, but it's still impressive that you can install Mono, Java, Python, Perl, Lua etc. on a small NAS device just with a single command. Hope this entry helps others get to hacking on their Synology device.

Thursday, May 19, 2011

Home network benchmarking with iperf

Having recently purchased a new NAS device, as well as upgraded network cables in my house to cat6 (1Gbps), I started wondering about just how fast and healthy my network setup was. Are all cables in order? Are any of my switches a bottleneck? Any particular slow endpoints?
performance challenged
There are of course commercial software to assist in this, but after having compiled and loaded up a bunch of Linux software onto my small NAS, it was only natural to give "iperf" a try. It's a small open source client-server utility you start in listening mode on one device, and in transmission mode on another, and have it transfer a bunch of random data over TCP. You can obtain iperf from sourceforge, from your Linux package manager or just Google for precompiled binaries if you are lazy.

Simple unidirectional test
The process is simple, log onto the receiving device and run iperf in listening mode, in the following case it's my Synology NAS:


NAS> iperf -s -p 5555
------------------------------------------------------------
Server listening on TCP port 5555
TCP window size: 85.3 kByte  (default)
------------------------------------------------------------

Then also run iperf, though this time in client mode targeting the server you just started:


casper@laptop:~/$ iperf -c 192.168.0.100 -p 5555
------------------------------------------------------------
Client connecting to 192.168.0.100, TCP port 5555
TCP window size: 19.0 kByte  (default)
------------------------------------------------------------
[  3] local 192.168.0.234 port 52742 connected with 192.168.0.100 port 5555
[ ID] Interval       Transfer     Bandwidth
[  3] 0.0-10.3 sec   26.2MBytes   21.4 Mbit/sec

Note that the above result is quite slow, as a consequence of being on congested WiFi and benchmarking the slowest node in my household, the NAS.

This gives you a bps metric for how fast the client could transfer data to the server. To benchmark both upstream and downstream, you must reverse the client-server pair and repeat the process.

Complex bidirectional test case
By running tests between each and every possible node, it becomes possible to draw some interesting conclusions about the setup in general. The wired setup in my household looks roughly like this (without smartphones, TV, PS3, IP phone etc.):


With Gigabit connections between all major nodes, I expected to see throughput up around 90% of the theoretical maximum bandwidth. What I saw was hardly consistent however, as the following graph shows:


Note that solid edges signify physical cat6 cables, dashed edges logical connections. Edges are labeled with directional throughput in Mbps and nodes contains summerized throughput on the form in/downtream and out/uptream.

Lessons learned
Infrastructure nodes like router and switch would appear to have performed optimal, and there is no hinting of faulty cabling. On several occasions, one can observe throughput beyond 940Mbps, or 94% of the theoretical maximum.
I was a little surprised to see that the overall best performer, was actually my 3 year old sub-notebook. The network stack (hardware+software) is obviously very well implemented on the laptop. The PC's, particularly one of them, would probably benefit from a new dedicated Gigabit adapter rather than relying on the one built into the motherboard.
However, the greatest surprise was to see how the NAS struggled, particularly when receiving data and when having to cross 2 switches. In all fairness, the slow CPU of the NAS (1.6GHz ARM) could very likely be key to the poor performance of iperf - loopback tests on all endpoints incl. the NAS suggests this to be the case.

There are of course many other parameters one could care about; ping time, jitter, packet loss, UDP packets, jumbo frames etc. but I was mostly just concerned about TCP bandwidth throughput. I've learned that perhaps I should flash my DD-WRT router back with the original firmware, or perhaps try another open source image like Tomato, in order to see whether I can mitigate some of the observed bandwidth loss. It also seems abundantly clear, that the network system of my NAS device is more optimized for delivering data than receiving it.

I'd like to do a plugin for Synology's management interface, having it run native iperf against a Java implementation hosted by the client as an applet. Currently looking into whether this is possible. I fear it's too low level for Java, as this was one of the reasons I changed to C# for dealing with raw sockets some years ago.

Thursday, March 24, 2011

SVN checkout without branches and tags

Subversion (SVN) is a decent step up from CVS, and Maven (MVN) likewise from Ant. However one thing continues to bug me about how the two work together. Since MVN requires a fixed directory layout of /trunk, /tags and /branches, it becomes more and more painful to checkout or update a complete corporate source tree from one of the lower roots.

In the perfect world, every project may be checked in at the root of the tree, but we all know the world ain't perfect and it's not unusual to require checkout and build of up to a dozen SNAPSHOT dependencies spread around deeper in the hierarchy. There is no (to my knowledge) easy way around this, either you have to live with insanely long checkout times and the associated waste of disk space, or meticulously checkout each every project trunk individually.

As unimpressed I am by the Java language itself, the hat must come off to the plethora of readily available libraries on this platform. One of these libraries is SVNKit, a complete client API for accessing SVN servers. So I wondered just how hard it would be to put a client together, doing full tree checkouts, *without* /tags and /branches. Turns out it only took a few hours and under 100 lines of code. :)



/* Copyright (C) 2011 Casper Bang (casper.bang@gmail.com) */

package com.blogspot.coffeecokeandcode;

import java.io.File;
import java.util.Collection;
import org.tmatesoft.svn.core.*;
import org.tmatesoft.svn.core.io.*;
import org.tmatesoft.svn.core.wc.*;
import org.tmatesoft.svn.core.internal.io.dav.DAVRepositoryFactory;

public class svntrunks
{
public static void main(String... args) throws SVNException
{
String checkoutPath = null;
String username = null;
String password = null;
String checkoutRootPath = new File("").getAbsolutePath();

if(args.length > 2){
checkoutPath = args[0].trim();
username = args[1].trim();
password = args[2].trim();

if(args.length == 4)
checkoutRootPath = args[3].trim();
}
else
{
System.out.println("Usage: java -jar svntrunks.jar URL USERNAME PASSWORD [PATH]\n");
System.out.println("URL: The mandatory path to the SVN reposatory");
System.out.println("USERNAME: The username to log in as");
System.out.println("PASSWORD: Password for the username");
System.out.println("[PATH] Optional destination folder (defaults to current)\n\n");
System.out.println("Example: java -jar svntrunks.jar https://svn.java.net/svn/jersey~svn david secret jersey-checkout");
System.exit(1);
}

DAVRepositoryFactory.setup();

final SVNRepository repository = SVNRepositoryFactory.create( SVNURL.parseURIDecoded(checkoutPath) );
repository.setAuthenticationManager( SVNWCUtil.createDefaultAuthenticationManager( username , password ) );

final SVNClientManager clientManager = SVNClientManager.newInstance(null, repository.getAuthenticationManager());
final SVNUpdateClient updateClient = clientManager.getUpdateClient();
updateClient.setIgnoreExternals(false);

final SVNNodeKind nodeKind = repository.checkPath( "" , -1 );

if ( nodeKind == SVNNodeKind.NONE ) {
System.err.println( "There is no entry at '" + checkoutPath + "'." );
System.exit( 1 );
} else if ( nodeKind == SVNNodeKind.FILE ) {
System.err.println( "The entry at '" + checkoutPath + "' is a file while a directory was expected." );
System.exit( 1 );
}

System.out.println("Checkout source: "+ checkoutPath );
System.out.println("Checkout destination: "+ checkoutRootPath );
traverse(updateClient, repository , checkoutPath, checkoutRootPath, "");
System.out.println( "Repository latest revision: " + repository.getLatestRevision( ) );
}

public static void traverse(SVNUpdateClient updateClient, SVNRepository repository, String checkoutRootPath, String destRootPath, String repoPath ) throws SVNException {

System.out.println(repoPath);

updateClient.doCheckout(
SVNURL.parseURIDecoded(checkoutRootPath + "/" + repoPath),
new File(destRootPath + (!repoPath.isEmpty() ? "/":"") + repoPath),
SVNRevision.UNDEFINED,
SVNRevision.HEAD,
SVNDepth.FILES, false);

final Collection<SVNDirEntry> entries = repository.getDir( repoPath, -1 , null , (Collection) null );
for(SVNDirEntry entry : entries){
if(!entry.getName().equalsIgnoreCase("branches") && !entry.getName().equalsIgnoreCase("tags")){
if ( entry.getKind() == SVNNodeKind.DIR ) {
traverse(
updateClient,
repository,
checkoutRootPath,
destRootPath,
( repoPath.equals( "" ) ) ? entry.getName( ) : repoPath + "/" + entry.getName( ) );
}
}
}
}
}




Note that I do not subscribe to the legacy policy of 80 columns per line! This is 2011, I have more monitors than I have shoes and none are below 22".

You may download the complete source, or just the compiled binary I have build. The source code is licenced under GPL.



casper@workstation:~/Development/Code/Java/$ java -jar svntrunks.jar
Usage: java -jar svntrunks.jar URL USERNAME PASSWORD [PATH]

URL: The mandatory path to the SVN reposatory
USERNAME: The username to log in as
PASSWORD: Password for the username
[PATH] Optional destination folder (defaults to current)


Example: java -jar svntrunks.jar https://svn.java.net/svn/jersey~svn david secret jersey-checkout
casper@workstation:~/Development/Code/Java/$




Now it's considerably easier to do a new clean checkout of trees, in fact the tool cut down checkout time from the corporate tree I work in, with a factor of about 10! Your mileage may of course vary, depending on the amount of snapshots and branches in the tree.

There are some obvious improvements I'd like to add, i.e. I never understood why the vanilla SVN client doesn't include commands to detach previously checked out source from its repository. Of course, the true solution to the above problems may simply be to switch to a distributed SCM/VCS like Mercurial or Git. :)