Getting sources, source keys, and package lists out of an installed Ubuntu from a Live CD

When reinstalling Ubuntu, you may often find a large quantity of time becomes taken up afterwards as you look for applications that you had installed but failed to install after you finished reinstalling the OS.

There are ways to export and save this data before running reinstalling, backing up /etc/apt/sources.list is most commonly referenced, which gets you all the repositories you had, but you still have to reinstall the individual packages, and obtain the verification keys for them.

From a working system, you'd just

# If you want to preserve deinstalled packages, just omit the grep command 
$ cp /etc/apt/sources.list ~/sources.list
$ dpkg --get-selections | grep -v deinstall > ~/packages.list
$ sudo apt-key exportall > ~/sources.keys

In a Live CD, a situation I recently found myself in (I borked GRUB and decided that halfway through the potential fix was a good time to prepare for a possible reinstall,) it's a little harder. Those commands will get you the sources, packages, and keys from the Live CD rather than your install. Obviously, /etc/apt/sources.list is a simple copy-to-backup-partition away, but the other two are behind applications.

The solution (hijacked from ubuntuforums user 5-HT and pieced together with this Upgrade or Reinstall post by Daniel Bo) is to chroot into the old install, and then run those commands. Here's how to do that:

# /dev/sdb2 and ext3 was my install partition and type, replace those values with your own.
ubuntu@ubuntu:~$ sudo mkdir /mnt/root
ubuntu@ubuntu:~$ sudo mount -t ext3 /dev/sdb2 /mnt/root
ubuntu@ubuntu:~$ sudo mount -t proc none /mnt/root/proc
ubuntu@ubuntu:~$ sudo mount -o bind /dev /mnt/root/dev
ubuntu@ubuntu:~$ sudo chroot /mnt/root /bin/bash

Now you are root in your install partition. Now you need to mount the partition that you will store these on. If your home directory is on a different partition, you can use that/ If it's not, get another drive and back up your home partition to that drive as well as these files. You can't use nautilus for this, so here's what I did as a hint:

# Again, /dev/sda1 is for me, replace that value with the one applicable to your system.
root@ubuntu:~$ sudo mkdir /media/disk
root@ubuntu:~$ sudo mount /dev/sda1 /media/disk

Once you have the partition mounted (I'm going to assume /media/disk as described above,) you can simply run the commands:

root@ubuntu:~$ cp /etc/apt/sources.list /media/disk/sources.list
root@ubuntu:~$ dpkg --get-selections | grep -v deinstall > /media/disk/packages.list
root@ubuntu:~$ sudo apt-key exportall > /media/disk/sources.keys

If you have any other files in the filesystem you need, such as apache2 configs or the /etc/hosts files, now is a good time to get them.

Now, when you have your fresh install, you can restore the data by replacing or adding to the new sources.list file (remember to change the lines if you changed Ubuntu versions,) adding the keys back into apt, and re-installing all your packages. Ironically, the hardest part now (possibly) is the sources file if you need to alter it. If it's the same, just drag it into the Third-Party tab in Software Sources app (System > Administration > Software Sources) and have it add them.

The keys file can be done the same way, except in the Authentication tab.

Now that the repositories and their keys have been added, reinstall all your packages with:

$ dpkg --set-selections < /path/to/packages.list && sudo apt-get dselect-upgrade

Or with Synaptic, File > Read Markings and press Apply.

Simple Extension implemented in Shimmie core

Simple Extension functionality was imported into Shimmie core earlier this week. If you have this extension active, PHP will complain that you can't define SimpleExtension twice.

There were some changes during the import due to some problems with the Shimmie forum (broke during upgrade to phpBB3) and Shish ended up re-implementing it in core rather than performing a true import.

Unlike my version, which merely forwarded execution to functions of the same name as the event, this implementation prepends "on" and removes the trailing "Event." So an event PageRequestEvent will call a function onPageRequest.

This method, however, removes the need for the acrobatics my version was performing to prevent the forwarding from calling the receive_event and new i_am functions. The event_method_callable function is thus not needed and was not used.

I have not yet updated the source of my extensions that used this, though Shish has already made most (if not all) of the core and core-maintained extensions use it.

KFI Internet Stream URL

Update (2010-07-22): Recently KABC also changed their streaming method. Neither station can be played in a dedicated player now.

---

Update (long ago): About a week ago, Clear Channel updated their internet stream again, this time to a flash based thing. Unfortunately, in this update they no longer serve the XML expected by this script to the Sonos user-agent.

If anyone has a way to grab the stream URLs from the new player, please leave a comment.

---

If you've been trying to play the KFI radio stream in an external media player like totem, mplayer, or VLC, you've no doubt suffered annoyance at the number of hoops you have to jump through to acquire it. Further, the url needs to be refreshed every few hours to get the new authorization code.

Luckily, there's a solution. If you set your User-Agent to "Sonos" when you go to the KFI player it will send an XML file containing the stream URL instead of redirecting to the new browser player.

You can change your User-Agent string in Firefox with User Agent Switcher, or you can fetch the url with

wget -U "Sonos" http://www.kfi640.com/cc-common/ondemand/player.html

.

Once you have the file you can just copy the URL into the player of your choice.

Or you can use a Bash script that wget's the file, parses out the url with sed, and loads the results into totem.



I can't claim complete authorship of the bash script, as I found a version on a forum first, and have only subsequently made modifications to keep it working through Clear Channels streaming changes and to add a switch for another station I listen to.

Arrogance and Code Hosting

Arrogance: I'm going to write like I actually have an audience here.

Code hosting: I'd been unsure of how to go about posting the code to my projects, trying sites.google.com, which can host files within a wiki-type application, to Drop.IO, again with file hosting, though they both suffer from making it difficult to push large projects to them. I had to tar.gz them to keep them contained, and that's just not how you do this kind of thing.

And then Ars Technica posts a guide to setting up an account and repository on GitHub. I Already put Simple Extension there and updated the post here to reflect the new code location and changed the drop.io link to point to github. As I work on the other projects, they'll get repositories too.

Shimmie Extension - User Permissions

This is a three-in-one.

User Permissions manages a more complex form of user permissions than can easily be done in Board Config. Basically, in default Shimmie you have three de-facto user groups: Admin, Registered, and Anonymous. Problem is, admin has access to everything, so if you wanted to let some registered users help police images and other user content, you'd have to trust them not to screw you with their admin access, up to and including de-admining your user.

Anyway, it started as an all-in-one that established user groups, default permissions, group-level permissions, and user-specific permissions. This got unwieldy very fast, especially since the user groups part was basically an extension in itself. So I split it in four pieces: User Groups, Global Permissions, Group Permissions, and User Permissions. As posted earlier, User Groups managed groups of users.

Global Permissions was basically a store of all the registered permissions, name, description, and default value.

Group Permissions built on top of Global, if there's a setting for the group level, it uses that, or falls back to global.

User Permissions builds on top of Group. Like Group, it falls back on groups the user is in if user level is unset (multiple settings for one permission are ORed together,) or to global if there are no groups with that permission set.

These were going to be separate extensions, with group and user being agnostic of each other (though depending on global.) Unfortunately, I could not figure out how I'd go about untangling the levels from each other, so I merged the three back together, partially. It has three separate extension definitions, each handling one level, and I could drop the need to have them able to work separately from each other.

So an extension can use this by registering options by hooking the PermissionRegisteringEvent and calling $event->register(String $key, String $name, String $description, Boolean $default), then later access the permissions applied to a user with Permission::user(User $user, String $key).

It's not as clean as the Config object, I admit. I want to have access to it through the User object since it would look like User::permission(String $key), but I also didn't want to require a code patch for core files. I was able to skip requiring the Config and Database objects from being passed in to Permission::user() by cheating and importing globals into the function.

Works well regardless, though it suffers from a bit of disuse. So far there are only 2 extensions that use it.

GitHub Repository
Commit at time of post: 14735d34
Dependencies: Simple Extension, User Groups