Friday, April 6, 2012

Just another Package Manager

The first thing you need to do when you go into the cloud is have a plan. Well, have been brooding for the last three years so there is no lack of ideas. Second thing is to get a cluster running. Obviously it is not that hard to get some process going on a Linux machine and start them. However, I ran into my by far greatest frustration with Java. Java is supposed to be "Write once, Run anywhere," but in practice they leave out this perky little detail of deployment. Java is amazingly poor in bridging that last mile. In the last year I've done some playing with node.js and npm. Let me show you a session that makes me envious:
 ~$ lessc base.less
 -bash: /usr/local/bin/lessc: No such file or directory
 ~$ sudo npm -g install less 
 -> /usr/local/lib/node_modules/less/bin/lessc
 less@1.3.0 /usr/local/lib/node_modules/less 
 ~$ lessc base.less
  #header { ... }
Why on earth can't we do this in Java? Now, installing npm might be painful? Nope:
 curl | sh
It is terribly frustrating that these javascript kiddies (kidding, node.js is very interesting!) can do this and we the enterprise Java developers can't! Something I ran into when I tried to launch my image on Amazon EC2.
At Amazon EC2 you can use a standard image with their variation of Linux. They provide one mechanism to customize the start-up to add your software, this goes through the user-data, that is specified during instance configuration. For the standard Amazon Linux image, this user data can also be a shell script that is run after the end of a reboot; the perfect place to initialize or synchronize your system. How would my user-data shell script look like in a perfect world?
  jpm || (curl | sh)
  jpm libsync
  jpm schedule --hourly mywatchdog 
  jpm start myservice
This jpm (abbreviation of Just another Package Manager) command should store the files in an appropriate place, it should be able to add commands to the shell path, install services, and schedule. Obviously, all these things are different today in MacOS, Linux, IBM mainframes, Windows, and other places where Java is running. The jpm command should therefore know about these differences and act as a good guest in the host environment. For example, on the Mac it should store the repository in /Library/Java/PackageManager but on Linux /var/jpm seems more appropriate. Same as user local files should be stored in Library on the Mac and not in the common Unix ~/.jpm directory. Daemons should follow the local rules and be manageable from the native management tools.
It should provide the following requirements:
  • Applications should not require special knowledge of the platform or jpm 
  • Act, look, and feel native
  • Manage a local repository of jars (either apps or libs)
  • Allow (limited) use without being the admin or root
  • Install command line apps written in Java
  • Install services/daemons written in Java
  • Schedule services or apps in time
  • Monitor the Java services
  • Install as GUI app (with proper icon and clickable)
  • Install from the web
  • Support at least any system with a basic *nix shell, MacOS, and Windows
  • Allow others to contribute new platforms 
  • Support JAR signing
  • Full uninstall
  • Simple & small
A wise man would have given a deep sigh, and spent his time on writing the scripts to run this on Linux. Since I am not a wise man and this is such a great frustration for me of Java I incubated jpm this week. Given the short time, and some of the false directions I took, it is not yet ready for consumption. However, I am using it on Linux and MacOS. It is not yet ready for others to play with it yet but if there are people interested in adapting it to Windows and other environments then I am more than willing to talk if you have the time to work on this. (A student project? I can provide mentoring.) It is ASL 2.0 licensed and it will probably end up at bndtools.
Peter Kriens


  1. what type of repository does jpm use? Maven, osgi bundle repo? How does it know where to deploy? What types of artifacts does it know of?

  2. Local repository is file system based, same as bnd(tools) uses.

    It can make command and services out of any JAR that has a Main-Class manifest header.

    Remote repository is still open but will absolutely support OSGi and maven repos.

  3. Looks awesome!

    Any chance you put it to github or somewhere so we can look, play & contribute?

    I think that jpm is something way "wider" than bndtools. So I think that it would be better if it was an independent project.

  4. Found similar problems at the opposite pole: embedded appliances using standard images. In the project I used a Debian Live boot and Deployment Admin for wrapping up my application specific stuff. But that in between stuff was the hassle - I still needed the prerequisite infrastructure added to the image - Java and OSGi.

    Will jpm be Java based and require a JRE be present?

  5. @iocanel: yes it will be on github, but it is still very immature, need a few weeks to make it presentable. So unless you really wan to spent significant time on it and help me out, wait a bit.

    @John: jpm will be java based. Currently I require java to be installed but can of course install it from the install script. However, nowadays Java being open source it is usually present in *nix envs.

  6. Peter, it's clear to me that jpm is still very immature, but I don't mind having a look at something not very presentable. To be honest I don't have "significant" time, but I'd gladly spend some time playing with it, whenever you feel like sharing it.

  7. @iocanel: I learned the hard way that showing immature products often backfires, but just follow the blog and you'll see when I make it available.

  8. Hi Peter, thanks for the initiative. I make myself available to also help on that.
    Are you considering to enable upgrades like apt?


  9. @B.B: Yes, it should grow into a full blown package manager for java. It should also start supporting full dependency management which it does not yet do.

  10. Neat.

    Why did you start your own project (apart from it being frightfully interesting and rewarding to write); the name alludes to the other package managers you could start with and adapt/decorate, e.g. puppet, Buckminster.

  11. @willr: jpm is focused on doing what npm does and does ... Allowing Java to be used for simple applications that are easy to combine. Puppet seems to be much more ambitious and buckminster was not in this league last time I looked (many years ago)?