I was paging some code through less and accidentally hit the ‘v’ key, and it launched my editor on the file. Unfortunately, it doesn’t work when the file is coming through stdin (though you could rerun the command and redirect the output, launching the editor afterwards). This would be easy to implement. Dump the input until EOF into a file in the /tmp directory, then launch the editor on it. I pulled open the man page and confirmed it. I guess this just falls under live and learn.
Archive for the ‘*NIX’ Category
I didn’t know you could do this…
Thursday, August 13th, 2009Adding Some Color
Tuesday, July 28th, 2009This may sound crazy, but the thing I miss the most about Gentoo is its nice, pretty out of the box terminal. I do a lot of work from a good old shell, no matter what the OS. Even on Windows, I often whip out cmd to do basic file management. It is just much quicker and more efficient.As with any environment you spend a lot of time in, it is nice if it is easy on the eyes. Today, I got utterly and truly sick of Ubuntu’s no-color prompt so I busted Daniel Robbins article “Prompt Magic” and built the following prompt:
PS1=”${debian_chroot:+($debian_chroot)}\e[34;1m\u\e[0;1m@\e[32;1m\h:\e[0;1m\w\e[32;1m \$ \[\e[0m\]“
-
export PS1
The Debian chroot business is lifted from the default /etc/profile. This generates the following:
Imperfect, but much better.
Further Addendums
Thursday, July 2nd, 2009Yet more on “Life is Good..“: This same hardware can be gotten up and running on Ubuntu 9.04 by manually downloading the debs for the 2.6.30 kernel from http://kernel.ubuntu.com/~kernel-ppa/mainline/v2.6.30/ . Things got a little hairy when I hit the nvidia drivers. The theory runs that DKMS should handle all those details automatically. For me, DKMS choked on the nvidia module. One reader on Launchpad suggested installing the drivers first, then the drivers. This didn’t quite work when I tried it. I had to install the new kernel (probably for the source and headers), then run the updated drivers that the reader referenced, then reinstall the kernel (so that DKMS would pick up the new module and use it). I’m sure there’s more rhyme and reason to it all than that, but this is what I had to do to get it running.
Setting up Bridged Networks (in Linux) For Dummies
Monday, January 26th, 2009I set up VirtualBox to do virtualize some odds and ends testing I was doing (specifically, with the intent of freeing up a laptop to be reissued) and encountered the wonderful problem of trying to bridge the connections to the network at large. With VMWare Server, this is delightfully trivial. The bridged connections are already set up and all you have to do is pick it off the list when setting up the virtual adapter. VirtualBox works the same way, but you have to set up the adapters yourself. So, here are my notes on doing it.
At the system level, what you are doing is creating a bridge (which enslaves a hardware interface) and creating virtual adapters off of it. What you get is something like this:
On Debian (and, by extension, Ubuntu) you can create the devices by editing the /etc/network/interfaces file and adding the following sections:
-
auto br0
-
iface br0 inet dhcp
-
bridge_ports all tap0 tap1
-
-
auto tap0
-
iface tap0 inet manual
-
up ifconfig eth0 0.0.0.0 up
-
down ifconfig eth0 down
-
tunctl_user <user>
The first section creates the bridge and enslates eth0 to it. The second section creates a TAP interface that is assigned to that bridged device.
Alternatively, you can manually run commands like the following to accomplish this task. If you go this route, you will want to take your commands, put them in a shell script, and rerun them at boot time so that your interfaces are recreated after booting up. The utilities are brctl (for the bridge) and tunctl (for the TUN interface). Given the sections above, it is actually pretty easy to know what sequence of commands to run, so I’ll leave that as an excercise for the reader (with the help of the man pages/help section).
There you have it. All you need to know to go crazy and create seven million bridged network devices. While my reason and example here is setting up VirtualBox, it is not the only use for it. Other uses include operating system virtualization, running multiple services on different IPs (i.e. bridging Apache to another address than the main system one), and other general tomfoolery.
Packaging: Debian vs. Gentoo
Saturday, April 7th, 2007If you have read this blog, you know that I have been playing with Debian packaging. I have created packages on both Debian and Gentoo systems and so, here and now, I offer my final manifesto on how they compare.
So, what is the difference? Debian packaging basically archives an installed version of the tree (etc/, usr/, usr/bin, etc.) with a couple of text files that describe the package. What it is, what it depends on, etc. Gentoo doesn’t really package anything. Ever. Gentoo’s Portage system is a network of Python and bash scripting that tells the system where to get the parts for the package, how to configure it, how to build it, and how to install it.
There is a deep philosophical divide here. For the package creator, the tasks are almost completely different. In both cases you have to build the package, but in the case of Debian that is almost all you are doing. You are going to build the package, roll the install into one tarball and pass it around. In Gentoo, you are scripting the build for every single user thereafter. You don’t give them a finished product, you give them a machine-readable build manual. From the package creator’s point of view, it can be a pain in the neck either way. From the user’s point of view it’s a case of convenience versus flexibility. Debian packages are nice and easy to install. No fuss, no waiting. It’s just done. Gentoo’s portage system offers endless configurability and flexibility to build your system the way you want it. Ultimately, it comes down to user preference. Do you want the flexibility or do you want it now? There is no right answer here.
There are other concerns for the package creator. With Gentoo, you are more or less recording the build process. Sure, you sometimes have to do some tinkering to get it to play nice in the sandbox, buut those times are relatively rare and for good old fashioned autotools software, it is a piece of cake. Maybe it’s just me, but Debian seems far more fussy. There has been some software that I just said “oh, heck. I’ll just make install.” rather than fiddle any more with the package.
All in all, I like Gentoo’s system better. It is simple, clean, elegant, and flexible. That’s not to say it is without caveat, but my experience with it has been smoother than Debian. But, hey, who knows? I may write here in not to long about the glories of deb packages–but I doubt it.
Chicken Package–Updated
Thursday, April 5th, 2007In my continued travels on the subject of Debian Packages, I came across a discussion of checkinstall. Basically, the high and low of it is that checkinstall is quick and easy–but the packages will not necessarily work all that well in a clean room environment. This combined with the fact that the paths in the previous package are not quite right, prompted me to build a new package “the right way”. It is now available.
Before I talk about the solution, I wanted to mention a couple of things that researching this problem brought into greater perspective for me.
That out of the way, here is what I did: First, I downloaded and extracted the source. Then I cd’d into the base source directory. Then I ran:
$ dh_make -n -e my-email@my-domain.dom
This command generates default build scripts for the Debian package (note: you will need to ensure that the package for dh_make is installed). The next step is to edit the control file. This will, by default, be created under debian/control. The main changes needed are to the description and to the dependencies. The documentation on the specifics can be found here. As it is relatively straight forward I will not go into detail on it here. Once you have finished with that, you would need to edit the various rules in the rules file. If, however, your software is fairly standard automake
then you probably don’t need to do anything. If, on the other hand, you were using NAnt, Ant, HMake, or some other custom build system you would have to modify the rules file to build properly. After all of this is squared away, simply run
$ fakeroot debian/rules binary
If the previous step was completed correctly, dh_make will spit a shiny new Debian package out
in the parent directory. A good old fashioned dpkg run and the package will install on your system.
Sorta-Seamless Virtualization
Saturday, March 31st, 2007I’d seen an article before about using VMWare to run Windows programs “seamlessly”–as though they were being run from the user’s own desktop. I say sorta because you can try to make the themes and widgets match up, but it just won’t always work. Anyway, today I stumbled across another article and so, with little else to do, took the plunge. I won’t regurgitate the article. This is the link:
https://help.ubuntu.com/community/SeamlessVirtualization
Though the one comment I will make is make absolutely sure that you are logged out before trying to launch your app of choice–if you don’t, you will get the whole Windows desktop.
If I won’t regurgitate, then what am I writing about? Well, after the whole thing worked I got to thinking: normally, if I am not using a VM in any way shape or form, I shut it down. I don’t have limitless resources on this Dell laptop and those CPU cycles are precious. Wouldn’t it be nice if we could boot the VM whenever it is not running but we try to run the program? Why yes, it would and we can do make it happen with VMWare Server’s command line tools. I wrote this in Python, as bash does not have a sleep function (which is needed if the VM boots up so that it will wait until the boot process is finished before trying to connect). In all its glory, with some machine-specific redactions, is the nifty little script I wrote:
#!/usr/bin/python
import os
import time
onoffStream = os.popen('vmware-cmd /
[pathto]/Windows XP Professional.vmx getstate | cut -d " " -f 3')
onOff = onoffStream.read().strip()
if onOff == 'off':
os.popen('vmware-cmd /
[pathto]/Windows XP Professional.vmx start')
time.sleep(60) # wait 1 minutes
os.system('rdesktop -A -s "c:seamlessrdpseamlessrdpshell.exe C:
[path]MyExe.exe"
[privateIP]:3389 -u iuser -p ipass')
Haskell as UNIX
Thursday, March 29th, 2007I was continuing work on the tutorial I have mentioned before and one thought occurred to me: Haskell (and functional programming in general) promote a UNIX-ish way of thinking. UNIX’s mantra is “Do one thing and do it well.” Anyone who has done some functional programming knows, you are, very quickly, forced to break the task down into pieces. Then, you write a function that does one and only one piece. Scheme encourages this style, Haskell all but mandates it.
Under the UNIX style, we have a utility (awk/sed or perl, depending on who you ask) to process text files*. We have a utility that dumps a file to stdout (cat), a utility to find files (find), to search text (grep), etc. Indeed, the principle point of shell scripting is to chain multiple utilities together.
Somewhere between this similarity, language fanaticism, genius, and mania someone seems to have come up with the idea of marrying the two and the idea of the Haskell shell was born: h4sh. It looks kind of nifty on paper. I don’t know that I would write my shell scripts with it, though. The more typical ba/sh seem to work well enough and, honestly, I am having a hard time thinking of anything that you could really do with h4sh that you can’t do already. If anything, it’s more stylistic. This isn’t a criticism or a slam of the work. Interesting idea and I am all for new ideas. Maybe I am just not hardcore enough yet.
Is this pattern useful in any way, shape, or form? I am not entirely sure. I suppose one could write a shell for where the difference between user-defined function and a program is practically nil. I could create a function sort (yes, I know that there is a sort utility) and, on the mythical command line, do this:
$ cat foo | grep “bar” | sort
And get the correct result. In bash, you would have to pas the result of grep as a list to sort (barring some magic of which I am not aware). You could store the function definitions (or byte code compiled) versions in the user’s home directory and load it in on startup. Some method to handle potential conflicts between function and program names would have to be put into place. I don’t even know that there would be anything useful about this, but it does have kind of a cool, geeky feel to it. Who knows? Perhaps someday I will try to figure out what the ideal shell would look like and create it. Just not today.
* At least, that was the intention. GNU Awk now includes such functionality as network ports and perl does everything under the sun including, 3D engines, apparently, according to this gem I’d stumbled across in past travels: http://www.perl.com/pub/a/2004/12/01/3d_engine.html)
. What would possess someone to even try a 3D engine in Perl, I have no idea. Still, it all goes to show that feature creep is alive and well in the UNIX world.
Chicken Package
Monday, March 26th, 2007As I was heading out for lunch today, I was thinking about my good old project Latrunculi. I wrote earlier that I was going to put it on hold until I finish Ocean. As Ocean is coming along (at the moment I am rewriting a great deal of the macro expander; the code is coming along cleaner and more elegant than before, but not as quickly as I should long), this could be a while and no wonder. While compilers are not magic, they are not done overnight, either. This leaves Latrunculi hanging. I don’t like to leave projects hanging even though I routinely do it. So I decided to do a quick code audit and see how quickly I could push a first release out the door. I went to do a quick rebuild of the source and then remembered: since switching (for how long, we’ll see) to Ubuntu, I hadn’t installed Chicken Scheme. It is included in the repositories but, like Gentoo, the package was not up to date. The solution was obvious: do what any red-blooded OSS user would do: download the source.
The compilation went down without a hitch and as gobbledy-gook scrolled across my screen, I googled the creation of Debian packages. Why? Well, the whole point of a package manager is to manage your packages. As I quickly learned as a Slackware user (my fault, not Slackware’s) if you do not do this properly, things can quickly become unmanageable, by man or machine. Towards the build’s end, I came across an article on the use of checkinstall (http://www.debian-administration.org/articles/147),
a semiautomated method of generating Debian packages. The basics of Debian packages are easy to understand: a couple of control files and tars containing the actual files for the install–that doesn’t mean that I felt like doing it by hand. Like all programmers, I am fundamentally lazy when it comes to computers. There are bigger, better things to do than haggle with control files. So I decided to give checkinstall a spin. The usage is trivially simple: after building, issue this command as root:
# checkinstall -D installcmd
Where the -D flag instructs checkinstall to make a Debian package (instead of an RPM or tgz) and installcmd is the command to run the install (make install, in most cases). I went ahead and generated the Chicken 2.6 package attached to this blog post. By default, checkinstall automatically installs the package after building it. Sure enough, it worked. The one caveat I did hit was when Chicken tried to load a module that I generated from SWIG it failed, unable to find libpcre. I am not sure of the extent to which this is a problem, but here is the fix. As root, run:
# ln -s /usr/lib/libpcre
That solved the problem on my box.
Your Own Private Database Server with VMWare
Thursday, March 22nd, 2007As the title implies (I hope) this post is a quick guide on how to set up a database server in a virtual machine with the aid of VMWare Server. The nice thing about Server is that it runs as a service, so our virtual server can continue running even after we shut down the console. In this example, I will use FreeBSD though you can use anything you want from Linux distro du jour, to OpenBSD, to Solaris, or even to Windows. On top of this fine OS, you will need to run some database system. I can’t make an example out of nothing, so I chose PostgreSQL. If you have never heard of it, it is, featurewise, the most advanced OSS RDBMS.
What is the motivation behind this little excercise? Well, using VMWare Server in this manner can help with testing so that you can run your apps against a server and get a little closer to real-life. You can also use it to run web apps that you may want to use personally (like a personal bug tracker), but don’t want the world to see. Finally, it is an opportunity to try something new if you haven’t done it before. So, let’s get to it, then!
The first thing you will need to do, if you haven’t done it already, is install VMWare Server. It is a free download, though it will need to register (again, free) for a serial number. So, if applicable, saunter over to http://www.vmware.com/ and let them be your guide.
Done? Okay. Next, we will need to spawn off a new virtual machine. Log into VMWare and, on the home tab, select “Create New Virtual Machine”. While your mileage may vary, I chose the following options:
OS: Other -> FreeBSD
RAM: 160 MB
Hard Drive: 8GB, Not allocated, split into 2GB (mandated by the fact that I put the VM on a FAT32 external hard drive)
Network Card: NAT (this will change later).
and defaults for the rest. The VM we just created is, obviously, completely blank. So, go to http://www.freebsd.org/ and download the disc 1 ISO (we shall not need disc 2 for this tutorial).
Once that is finished, go back to the VMWare Server Console. Select the FreeBSD machine, then click VM->Settings. Set the CD drive to point to the ISO you just downloaded. Click OK and power up the virtual machine. The machine should boot straight up to the FreeBSD installer.
From here, do a standard FreeBSD install. I will leave the explanation of that to the FreeBSD project’s quite good documentation (or, perhaps at a later date, a different tutorial). As an FYI, due to the limited use towards this VM will be put, I elected not to install ports. While useful in general, it is hard drive space wasted here. If you kept it simple, the install should finish pretty quickly. When the prompt shows up asking you if wish to reboot, switch out of the machine (Ctrl+Alt) and switch the CD back to the physical drive so that the VM will not boot to the installer again. Then choose yes. The system will reboot and bring you to a simple login prompt. Login as root with no password. Now the fun begins.
The software that we will want on this box is simple: SSH and Postgres. Postgres was the whole point in doing this and having SSH for “remote” administration is both cool and useful. While, in eventuality, we want to make this a private server, we will need to download the software off the web. Keep the networking card set to NAT (or whatever you use to connect directly to the web). Once logged in, you can determine what device equates to your “card” by looking at /var/run/dmesg.boot In the case of my VM (and therefore, probably yours as well), the card was lnc0 and to get an IP run:
# dhclient /dev/lnc0
In FreeBSD, there are two “software worlds”: ports and packages. Ports is similar to Gentoo’s portage (in fact, it is the progenitor of it) in that it is an automated build system. Packages is more similar to a basic Debian or RPM system: it downloads compressed binaries and installs them on the system. For simplicity and speed, I will use packages here. The basic way to add packages is:
# pkg_add package.tbz
In order for this to work, package.tbz must exist in the current path. Fortunately, for some packages, there is the nifty little short cut
# pkg_add -r package
Which will both download and install the specified package. Like I said, this doesn’t work, out of the box, for everything. So, let’s just use it to get what we need to do the rest. Wget is a cool utility that runs on *NIX systems. It can be used to download files from the command line. So, grab it as above:
# pkg_add -r wget
This will download and install wget in /usr/local/bin. Add this to your path by adding the following to your .cshrc:
set PATH=${PATH}”:/usr/local/bin”
Then run:
source .cshrc
Now we can use wget in all its glory. Go to FreeBSD’s website and go to ports. Search for postgresql. A quick look at the latest version tells us that, in addition to postgresql-server, we will also need gettext, gmake, libiconv, postgresql-client. Fortunately, libiconv and gettext were installed with wget. So, look at the URL for the package and, for each of the remaining, run:
# wget URL
When done, you should have TBZ’s for all of the packages you need. Install gmake, then postgresql-client, and finally postgresql-server.
Almost home free, we just need to do some more configuration on both the client side and in VMWare to finish up. First we need to initialize Postgres’s various settings. We do this with the following command:
# /usr/local/etc/rc.d/postgresql initdb
Then we start the server itself:
# /usr/local/etc/rc.d/postgresql start
We will want Postgres to start up whenever we start the machine, so, using good or not so good old vi add the following line to /etc/rc.conf:
postgresql_enable=”YES”
We will also need to create a database and db users. To create a DB nice and quick, log in as pgsql (a user installed by Postgres) and run
# createdb xyz
and voila! you have a database you can log in to.
Test everything to make sure it is working right, but at this point you should have FreeBSD running PostgreSQL just fine. The next step is to put this on a private subnet and make sure that we can access it from our host system (note: to help test, installing the PostgreSQL client on the host is recommended).
Now for the host-only part. Go to VM->Removable Devices->Ethernet Card 1->Edit and change the setting from NAT to Host Only. Then return to the guest machine and rerun dhclient. If successful, the VM will have acquired an IP on the private subnet. In the prompt, enter this command:
# ifconfig
Your card should be shown on the list with a fresh IP. Next, we need to make sure that PostgreSQL will actually listen on the port. First, edit the file /usr/local/pgsql/data/postgresql.conf. Set the following lines accordingly:
listen_addresses = ‘*’
port = 5432
This tells PostgreSQL to listen on all addresses (note that this does not have to be so, it just makes life easier in this scenario) on port 5432 (which is the client’s default). Next, edit the file /usr/local/pgsql/data/pg_hba.conf (HBA = Host Based Authentication). This file acts as Postgres’s private firewall (kind of). The rules in this file are evaluated to determine whether a given incoming connection will be permitted. I added this line to mine:
host all all 0.0.0.0 0.0.0.0 password
host refers to any SSL or non SSL port, the second field is the databases to be allowed (which can be set to a delimited list), the users that will be allowed (which again, can be specified) the IP address and mask (setting to 0.0.0.0 allows anything; think of a zero in any given portion of the address as a wildcard), and finally the type of authentication (password sets it to the good old fashioned password-based authentication; PostgreSQL has many options). Finally, go ahead and restart the PostgreSQL server with this command:
# /usr/local/etc/rc.d/postgresql restart
That’s it! Fire up a client on the host’s side and run it, specifying the IP address of the VM.
A few parting notes. I intentionally set security quite lax. Why? This is a virtual machine, not a real one, first, and secondly the only permitted connections will be off the private host machine. Be much more conscientious if this is for something remotely related to production! Secondly, while this article goes into detail only on the installation of PostgreSQL, the same lessons and principles can be applied to just about anything else.
References: