IT « Mad Computer Scientist

Archive for the ‘IT’ Category

Setting up Mendelson AS2 HOWTO

Monday, February 8th, 2010

Overview

AS2 is a wire protocol for transferring files between two organizations. This guide explains how to get the mendelson open source as2 server up and running. The instructions are slanted towards set up on a Debian box, though set up on any *NIX system or Windows should be very similar.

The goal in this guide will be to set up two independant machines, one for test and one for production, and get them talking to one another.

Finally, this guide was developed and tested with mendelson AS2 version 1.1 and Debian 5.0.

Installation

Get the zip file from the Mendelson AS2 sourceforge page. No installation is needed (though an installer is provided for Microsoft Windows boxen), just unpack the files in some location. What I did, was:

create a separate user to run the AS2 software (named, cleverly enough, as2user)
Unpack the software in a directory in the as2user's home directory.
Run the software in a GNU Screen session
For the GUI portion, run a lightweight window manager (IceWM was my choice) and a VNC server.

As always, your mileage may vary. Out of the box, the mendelson as2 server is configured to interact with the mendelson test server, and nothing else. The next step is to set up the keys.

Configuring Keys

This process can be done with Portecle, keyman, or the OpenSSL toolchain. The most user-friendly of these is Portecle, which is also the one that Mendelson recommends. Because Portecle is pretty straightforward (and, if you choose one of the other tools, you almost certainly know what you are doing anyway), we will skip the exact sequence of clicks or commands needed for this.

Mendelson AS2 stores its keys in the certificates.p12 file in the root of the install directory. The password for this store is, incidentally, test. The first thing we need to do is recreate the private keys. What we do is delete the keys (or delete the store and create it afresh with new keys with this name) and create new ones. The names of the keys are Key1 and Key2.

After creating the private keys on both machines, we need to export certificates for each, then exchange them. If mendelson AS2 is running, the certificates and keys can always be reloaded by clicking File → reload key store.

Patching the Scripts

Mendelson AS2 comes with DOS Batch files and Bash scripts to launch the AS2 server on Windows and *nix machines, respectively. Not to be degrading or anything, but the bash scripts do not appear to have been well tested. I had to make the following changes to them:

Both the mendelson_as2_start.sh and mendelson_as2_stop.sh files used Windows line endings, instead of UNIX. The dos2unix script (available in the Debian and Ubuntu package managers) fixed this problem.
Make both of the aforementioned files executable
At the top of the mendelson_as2_start.sh file, there is a line setting the CLASSPATH. I had to modify it to CLASSPATH=as2.jar:jetty/start.jar:jetty/lib/servlet-api-2.5-6.1.1.jar

Once the appropriate changes have been made to the start scripts, just run:

./mendelson_as2_start.sh

from the install directory.

Configuring the Local Station

Before the server can receive messages, it must be configured as a local station. By default, a local station will already be set up. The parameters just need to be adapted to match the actual environment. Pretty much, all you will have to change right off the bat is the MDN (URI), which is set to a mendelson domain. While here, you will also need to select the keys you generated above for the local station under the Security tab.

Configuring Partners

After a minute or two, the GUI will pop up. It is here that the AS2 partners must be set up before files can be exchanged. Take the following steps:

Click the button labeled “Partner” (or go to File → Partner).
Fill out the forms. The rest of this should be fairly obvious, but to go over it:
- Misc
  - Name
  - AS2 ID
  - email address – a contact
  - comments
- Keys – if you imported the keys above, the certificate for the trading partner should be available from the drop down. Select it or bad things will happen—I promise.
- MDN – the URI of the recipient.
Click Ok

Sending Messages

This part is easy. Copy a file to the intended recipient’s directory on the server. By default, mendelson is set to poll for new files every 10s (a little inotify support here would rock). In general, from the mendelson install directory, the location will look like:

/messages/<partner name>/outbox

Conversely, the messages will be received on:

/messages/<local station name>/inbox

Copy and run. The main windows on client and server will show their respective progresses.

Configuring HTTPS

At this point, if everything has gone according to plan, messages can be exchanged in plain HTTP. In many situations, however, we want to exchange messages over HTTPS for added security. To do this, we must:

Configure Mendelson AS2 to use HTTPS
Generate new keys for the HTTPS store
On the sender, import the certificates for the recipient

This may sound a little confusing, given that we discussed generating keys above. It turns out that Jetty (the HTTP server & client that Mendelson AS2 uses) has its own separate, independent keystore for sending over HTTPS. Moreover, the keys are expired, which is probably just as well because it makes us generate fresh ones.

Under the main Mendelson AS2 directory, there is a directory named jetty/etc containing Jetty’s configuration files. Jetty itself uses jetty.xml and an example configuration for SSL is in jetty-ssl.xml. Copy the following code from jetty-ssl.xml into jetty.xml:

<Call name="addConnector">
  <Arg>
    <New>
      <Set name="Port">8443</Set>
      <Set name="maxIdleTime">30000</Set>
      <Set name="keystore"><SystemProperty name="jetty.home" default="." />/etc/keystore</Set>
      <Set name="password">OBF:1vny1zlo1x8e1vnw1vn61x8g1zlu1vn4</Set>
      <Set name="keyPassword">OBF:1u2u1wml1z7s1z7a1wnl1u2g</Set>
      <Set name="truststore"><SystemProperty name="jetty.home" default="." />/etc/keystore</Set>
      <Set name="trustPassword">OBF:1vny1zlo1x8e1vnw1vn61x8g1zlu1vn4</Set>
    </New>
  </Arg>
</Call>

You’ll notice that it references the defunct keystore. Create a new keystore, populated with two keys from the following commands:

keytool -genkey -alias Key1 -keypass changeit -keysize 1024 -keystore my.keystore -keyalg RSA -storepass changeit

keytool -genkey -alias Key2 -keypass changeit -keysize 1024 -keystore my.keystore -keyalg RSA -storepass changeit

where my.keystore is the filename of the new keystore and changeit is the password for the store.

For password, keyPassword, and trustPassword, put the values corresponding to those used in the keytool commands.

From the destination, export a certificate and import it into the keystore that was just created.

Conclusion

Once you have the AS2 server up and running, the process of adding real life partners is fairly similar. The only other parting tip I can offer (thanks to the forums) is that if, at any step, something goes wrong the start up script can be patched to provide a lot more debugging information by changing the last line of mendelson_as2_start.sh to read:

java -Xmx192M -Xms92M -classpath $CLASSPATH -Djavax.net.debug=all de.mendelson.comm.as2.AS2

the addition is -Djavax.net.debug=all. As implied this will dump all sorts of goodies to the terminal.

Appendix – Signed MDNs

As of Mendelson AS2 1.1 – build 29, there is a bug that causes verifcation errors with signed MDNs sent by mendelson AS2 to non-mendelson AS2 servers. The solution is to get the b29 source module from CVS (on Sourceforge), change all occurrences of “\n” in the message strings to “\r\n” in MDNText.java, then navigate to de/mendelson/comm/as2/message and run:

javac -classpath /path/to/as2.jar MDNText.java

this will create a new MDNText.class file. Unzip the jar somewhere that will not trample anything, replace the MDNText.class file, and create a new jar. This updated file should solve the problem. At least, it did for me. Your mileage may vary, of course.

Sources

Posted in *NIX, Debian/Ubuntu, IT, Linux, Technical | No Comments »

The “Business Perspective” is a False Canard

Thursday, February 4th, 2010

Flipping through the C++ FQA, the phrase “from a business perspective” popped up a number of times and it occurred to me how often I have heard that phrase or something like it to refer to the needs of the management as opposed to the needs of a programmer. In fairness, I must also add that the author of that fine document is semi-quoting from the C++ FAQ. As I stared at those words, something jumped out at me: when it comes to tech, there really isn’t any such thing as a “business need” because the geeks and the suits ultimately want the same thing.

What kinds of things do we find in the “business perspective”? Well, how about these:

Economy of price – we need to keep costs down in order to increase our margin
Economy of time – closely intertwined with economy of price, but still separate in that we want to get our product to market ASAP, even aside from price, to help grab up marketshare
Capability – it must do whatever it is that we need it to do

That should pretty much cover it. The friction between the two groups does not come from these basic wants. Developers do not want to work longer on a project than is necessary. By and large, they want to do it and move on.

About the only time I see that there is a real collision, is when developers try to make their own jobs a little more interesting. Even here, we see that this is mostly unconscious. The developer trying to interestify the job usually believes consciously that they are solving some problem that is stalking the whole project. This is hardly unique to the developer side of the equation as we (or, at least I) have seen the business types getting all distracted by shiny little trinkets.

So, then, at the end of the day, the friction seems to come less from the core concerns (which are, more or less, shared by both parties) but how they are perceived. But the phrase “business perspective” is a lead-in to a pack of nonsense.

Posted in IT, Programming Languages | No Comments »

MySQL Hatred

Thursday, February 4th, 2010

Further anti-MySQL bile. I hate MySQL, this morning. While doing some nice sysadmin type stuff, I wanted to either lock a database down or (better yet) take it offline completely, while leaving everything else untouched. A quick command in MS SQL Server or a few clicks of the mouse, if you are so inclined. Easy. MySQL does not have this basic, basic admin feature. Hacks from the intrawebs include:

FLUSH TABLES WITH READ LOCK
Changing user permissions.

What the heck? I have to tinker with user permissions to TAKE A DATABASE OFFLINE? And flush tables locks all right–every table in every database. If you’re running one database, that’s fine. Me, I’ve got closer to 30. Sure, in our case, this is because we have a couple of apps that are badly designed. But still. What if I had two? Say, a blog and a wiki? Same problem. Take one down, take the other down. Or fiddle with permissions. I’m sorry, this is just wrong.

Posted in *NIX, Database, IT, MySQL, PHP | No Comments »

QuickBooks Hack

Wednesday, December 23rd, 2009

Recently, at the old work place, I encountered a problem on one of our accountant’s workstations. On this station, starting up QuickBooks Pro 2008 caused the Windows Installer to pop up (usually locking somewhere around “Preparing to install…”, but sometimes making it to the install screen). Going through the installation (again) did not work. Intuit has a knowledgebase article on this (http://support.quickbooks.intuit.com/support/pages/knowledgebasearticle/1005515.html) that makes three suggestions:

Reregister QuickBooks’s DLLs with the reboot.bat script in the QuickBooks directory
Repair the .NET framework versions (1.1 and 2.0, in this case) for the QuickBooks version
Reinstall QuickBooks

In my case, none of these worked. A blog entry, which I cannot seem to find now, suggested uninstalling QuickBooks, removing the .NET framework and reinstalling. For good measure, it also recommended using some tools available for download through MSDN to completely nuke .NET from the system. This also, did not work. What I found did work, however, was running QuickBooks as another user on the same machine. Same permissions, mind you, as the accountant, but someone else. This worked, but was a pain in the neck, as I had to keep logging the user into QuickBooks. The final solution, was to wrap this little bit of hackery in a batch script, create a shortcut to it, and replace the user’s icons (desktop & start menu) with the shortcut to the batch script–oh, and change the icon for good measure.

The batch script follows, with the username expurgiated:

runas /savecred /user:equesada\Administrator “C:\Program Files\Intuit\QuickBooks 2008\QBW32Pro.exe”

runas /savecred /user:<someuser> "C:\Program Files\Intuit\QuickBooks 2008\QBW32Pro.exe"

After trying this several times, it appears to be working fine. My assumption is that the problem lies somewhere in the registry settings for that user’s profile. It isn’t some sort of broad permissions issue, since the user was able to start the program fine before. Even now, the only problem is that it loops into the installer. Several registry scans and cleanups failed to find the problem. Why does QuickBooks have to be such a pain in the neck?

Posted in IT, QuickBooks | 2 Comments »

Unixing away from phpMyAdmin

Tuesday, December 8th, 2009

Here at the ol’ job, we use MySQL (something that I have blogged about before) and, naturally, have phpMyAdmin installed. I seldom use it as I prefer a a nice, CLI interface. It does provide a few amenities that have log me in even when I don’t strictly speaking need it. Some of these are the editing feature (yes, I am lazy enough that sometimes I would rather not sit down and write out an UPDATE query), the printing (which is much nicer than out-of-the-box lp or lpr on *nix machines), and dumping stuff to CSV or Excel (which is nice for one-off reports that I occasionally have to run).

The last couple of days, while working on some reconciliation type reports that get a little involved, I decided to take advantage of the Unix philosophy (a tool for every job, do one thing and do it well, etc.) and make my life quicker and easier from the MySQL command line. So, here is a look at the various tweaks I’ve made.

The first thing to look at is paging. The client doesn’t do any out of the box. After jacking off with a handful of pagers (less, more, most, and w3m) I decided on w3m for reasons that will soon become fairly clear. To make mysql page, simply run the command:

pager w3m

or more, less, most, etc. Whatever command you want to be the pager. This setting can be made permanent

Next up, we have printing. This is why I chose w3m. less and most provided no way that I could see to pass the piped-in text off to a printer. If some pager coniusseur would care to correct me on this score, I am all ears.

Set:

keymap C-p SAVE_SCREEN “| a2ps -r test.txt -1 -r -f 7pts”

keymap q EXIT

keymap C-p SAVE_SCREEN "| a2ps -1 -r -f 7pts"
keymap q EXIT

The second item maps q to exit without confirmation. Out of the box, w3m always prompts. I hate being prompted. Remove at your liking. The first line maps the sequence Control + P to a SAVE_SCREEN command (which is used to dump pages to files) and then pipes it to a2ps. You can look up the options for a2ps, but the end result is that, since no output file was specified, a2ps prettifies the text handed it and sends it off to the printer.

Finally, we have that little problem of dumping to Excel. We do not have to dump straight to .xls or .xlsx format. CSV will do, despite being a poor format in general. MySQL can do this part natively by running a query like so:

select * from foo
into outfile 'someplace.csv'
fields terminated by ',' lines terminated by '\n';

This is nice, but, speaking for myself, I usually review the results before dumping them out, just to be sure it looks roughly the way I want or expect. Another good way to do this, is to simply put the query into a file and run it like this:

mysql -uuser database < query.sql

When in noninteractive mode, the mysql client outputs the records in a tab delimited format. Piping this through sed and into a text file will create a simplistic CSV or opening it in tab-delimited form with a spreadsheet app (like Excel or OpenOffice Calc) will allow it to be exported to a more friendly format.

Posted in *NIX, Database, Debian/Ubuntu, IT, Linux, MySQL, Web Programming | No Comments »

Ruminations on Literate Programming

Thursday, November 12th, 2009

First off, I would like to begin by saying that this post will be a little different than usual. It is not so much an explanation, a tutorial, or the asserting of an opinion (and you all know what an unopinionated fellow I am), as it is a monologue like discussion. Running through possibilities, tossing out ideas, but it is not likely to present any firm conclusions. So, here we go.

I recently read Donald Knuth’s paper on WEB, a literate programming system that he wrote with others for their own use. The paper is listed in the references section. At first glance, literate programming makes perfect sense in academia. Code is not written that is not intended to be published as either a paper or a book. Using literate programming makes the task of doing both easier. As the content of the code changes, the commentary itself is readily changed to match.

The question that comes to mind, though, is whether or not literate programming has potential for the working programmer. In Academia, the real work is generally not the programming. The programming itself is merely a way to try to prove whatever the hypothesis is. It is the equivalent to a test in a physics laboratory. The working programmer is not using his code as a mere test. It is the final product and it has to work. Moreover, it must also be delivered in a reasonable (or, more often, unreasonable) amount of time. In this different atmosphere, does literate programming still have a place? Would it work as well for someone writing code to track truckloads as it does for Knuth when he writes his books? At first glance, it would be easy to say no, but Knuth extols the methodology for reasons that most programmers would find appealing. He intimates the LP makes maintenance easier.

If there is anything the working code monkey would love to see, it is an easier job in maintenance. Most of us have had that experience of looking at a screenful of code and wondering what he (or I) was thinking when this was written. If we are writing down are reasoning with the code, then the questions go away. We may not agree with the reasoning, but at least we would understand the angle from which the problem was hit. Naturally, most people would do as poor a job of maintaining an essay as they would the comments (there are virtually no comments in my production code). As with any methodology, its utility stands on its practitioners, not on its non-practitioners.

On StackOverflow, several users run down the idea as being outdated or outmoded, being suited to the dark days of when we were limited to two-character variable names. While the utility may be increased under such conditions, they have missed the point. Literate programming is not about writing a lot of comments–it is about writing a book or article on the problem, side by side with the problem’s solution. Literate programming is not an idea confined to a specific time. It is not a hack (as intimated). It is a way of looking at programming that turns the whole process on its head. The machine becomes auxiliary, the human audience becomes primary. It may be that this approach does not hold practical utility–but it is not something to be as lightly shoved aside as the idea of starting a completely new and independent piece of software in RPG III.

These rambling thoughts led me to look into some present day tools (even Knuth’s own WEB has been superseded, it seems). The one with the biggest following is noweb, which is language agnostic. My biggest complaint as I fish through the tools I could find, is that they were almost universally using TeX as their typesetting format. Historically speaking, this makes sense. Knuth wrote WEB and TeX and, more specifically, he wrote WEB for TeX. I, however, do not want to compose text in TeX or LaTeX. As I have written before, it is just to cluttering. There are a few out there that rely on something else. I found one that used wiki syntax. At least noweb supports HTML mode which, while still imperfect for composition (as an interchange and basic display format, it is excellent), is at least usable.

Any value that LP has will largely rest on the fact that it forces the programmer to think a little bit more about what he is doing as he is doing it. In this way, it is not unlike Haskell’s type system (which also makes it unsurprising that the Haskell community is one of the more vibrant outposts for LP).

A lot of questions still remain. Most LP tools are usable for standard write-compile-test cycles. For languages like Lisp, a separate tool would have to be created (not that a lot of weekend warrior projects do not already exist). On StackOverflow, a few users expressed concern for how you would use LP in a collaborative environment. Personally, I would suspect that it would work similar to the way that most technical writing team works: divide and conquer. Distributed source control systems like git or darcs make this even easier.

So what is it then? Academian pipe dream or underused tool? There is only one way to find out. Try it.

References

Posted in Functional Programming, IT, Lisps, PHP, Programming Languages | No Comments »

What is the Point of this?

Sunday, November 8th, 2009

I recently stumbled across some articles on WS-BPEL. BPEL stands for Business Process Execution Language. At first this caught my attention because, well, it sounded like some potentially slick DSL that would help describe business rules and execute them. Slapped in front of a good domain-specific API, something like this could help slash development time. Of course, such things are usually little more than pipe dreams, but today’s pipe dream is tomorrow’s brave new world. So, it is always better to keep an eye on things.

Perhaps the first tip off that this had nothing new to offer is that BSPEL is based on XML. Seriously, how can much good come from XML? Even the few times where the end result is cool (like WSDL and SOAP), a better interchange format could have been chosen. Imagine, for example, a YAML or JSON based web services platform? With wider support that would just rock. But I digress.

Here is a tutorial of sports on WS-BPEL. When you get past the buzz words and the fancy terminology, you have an XML based scripting language to tie basic web services together. Pretty disappointing. After looking at the examples, I do not see any way that this wins out over using Java, C#, or PHP. It is quite a stretch to refer to what this thing does as having anything to do with “business processes”. Even an IBM reference on the subject just shows a few simple control mechanisms joined up with the ability to call web services.

So, if you have seen this used in the wild to an efficacy above and beyond typical programming or scripting languages, please drop me a line or a comment—because this looks like buzzword tag soup.

Posted in IT, Programming Languages, Web Programming | No Comments »

Levenshtein Rocks

Friday, October 23rd, 2009

The company I work for is running a project in which various numbers are getting scanned. Often, the barcodes were missing or illegible and had to be typed by hand. On the backend, we found that a great many of them were subtly wrong. For example, O (letter oh) and 0 (number zero) were swapped. Well, it’s pretty easy to drop in a quick AJAX callback that checks the barcode number to make sure it is on file. I thought it would be cool, though, to have the program suggest the correct number to the user. If they were right and it was just something we hadn’t seen yet, then they could just leave it be. If not, the system would give them a much better idea where they were messing up.

Meet the Levenshtein distance. I had heard of it before (it is commonly used in spellcheckers), but never had a reason to use it. A quick googling showed gave a blog post in which the writer implemented the dynamic programming implementation of an algorithm to find the Levenshtein distance as a MySQL UDF. It worked beautifully.

Posted in Algorithms, Database, IT, MySQL, PHP, Programming Languages | No Comments »

Why Am I Still Hacking This?

Sunday, October 18th, 2009

This weekend, I spent a bit of time working on Latrunculi and my wife asked me why I was working on it, rather than one of my more businessey ventures. Well, the short answer was that Latrunculi has been a labor of love for a long time now. It started as an exploratory project, meant as a way to learn some basic AI algorithms, while underemployed (not unemployed–when unemployed, I had no time for Latrunculi) and expanded beyond it to a much larger vision. The goal is for it to have bleeding edge AI with 3D graphics and an excellent user interface.

Latrunculi also represents a lot of growth for me. The initial work was done in Chicken Scheme (an excellent R5RS Scheme implementation, I might add) and is presently happening under CLISP. The switch was done because the SDL bindings were much more mature in Common Lisp. Additionally, I wearied of the ad-hoc way I had to assemble pieces of the Scheme language. Arrays? Vectors? Nope. They’re in a SRFI, though. Modules? Not part of the language. And so on. Common Lisp has a much more practical bent to it as a language. Especially in the very first revisions, there is a lot of code that I am not proud of. The graphics code, especially, relied heavily on side effects and had a lot of poor variable names (brd for board). The number of set!s is simply revolting. It is coming together, though. A lot of the side effects have been cleaned up. On my personal git branch, I am starting to set up some unit tests. After a couple of things are done, we will have something resembling a real game, only with terrible AI (which is where I have been looking forward to being, again; I’ve read some papers and plan on doing a complete overhaul of the AI code).

Since the initial burst, work on Latrunculi has been sporadic. This is the project I do when the boys are napping (like that ever happens) and when all the house is quiet. It has always been that little spot of technical refuge from the grind in IT. Latrunculi is one of those embodiments of why I love this trade so. This idea of building something beautiful and, at the same time, usable is appealing. This week has been one of the longer ones at work, with a new project getting underway. It is after times like pounding on XML dumps and inventory reconciliations that the algorithmic challenges and graphics programming are so soothing.

Posted in IT, Latrunculi, Projects | No Comments »

Piece of CookiePie

Tuesday, October 6th, 2009

When testing a web site with various levels of user permissions, I thought it would be useful to be able to login with different accounts on different tabs of one Firefox session (especially because that’s all you can really have of a given profile). Towards this end, I found CookiePie. It works great and, as advertised, lets you keep multiple sessions of a web application running. However, after a little bit of experimentation today, I found that it interferes with at least three web apps:

VMWare 2.0 web Interface
Facebook
Napster web interface (no, I’m not a pirate; the first time I used Napster was after it had become a legal, paid service).

So, unfortunately, CookiePie must remain disabled on my Firefox profile. I need those apps a lot more on a daily basis than I need CookiePie. Still, it would be nice to see if they can clean up the bugs that make it interfere with other apps. I wasn’t using CookiePie on any of the above, so it shouldn’t have been interfering with them. It is possible that the only way to get this functionality right will be to bake it right into the browser. To the best of my knowledge, this has not been done on any major browser.

Posted in IT | No Comments »

Mad Computer Scientist