The thing with Ubuntu

I've been playing with Ubuntu (Dapper and Edgy) since Dapper was released last summer, and at least the Edgy experience has left me with the impression that the developers behind the project are truly overwhelmed by their task.

The Ubuntu forums are full with people that don't know much about Linux, all screaming for help, yet no developers seem to listen. Launchpad, you say? I can name two critical(I would say) bugs which don't seem to have been even reviewed by any Ubuntu developer.

In bug 78282, somebody broke vnc4server with a security update. That was almost one month ago. Whatever happened to that? Well, nothing. Every once in a while there's a new angry user complaining, and that's all. Oh, at least they confirmed it, and set the severity to 'high', which is a very nice consolation to people who have to still deselect 'vnc4server' from their update list. It would've been really hard to pull back the update, wouldn't it?

In bug 67998, Dapper to Edgy upgrades break the ALSA configuration. In this way, all apps are only able to use OSS output. Critical, you say? Check out the bug report, it is labeled as 'Low importance'! Yet it affects all soundcards without hardware mixing! And most of all, nobody gives a damn!

Whatever happened to 'Linux for human beings'? How can you connect with users if (as far as I can tell) almost none of the devs hang out on the forums, or even look in bug reports? My impression is the team is a bit overwhelmed by the number of users they have, and they focus on stuff reported by people in companies which pay support contracts (check out the latest bug in some libc scripts, for example, quite minor compared to the above - Matt Zimmerman is all over it). However, how can you be encouraged to contribute, if all your bug reports are ignored? If the Ubuntu team is short on people, they should focus on listening to people, not on turning them away to other distros.

Oh, there's also bug 42555, but that's a small potato compared to the others. Despite the fact that sysklogd had been repeatedly reported as broken by people in the know about DNS, ubuntu-minimal still depends on sysklogd, and nothing else. I submitted a patch in that thread so the dependency shifts to 'system-log-daemon', a virtual package that depends on either sysklogd or, for example, syslog-ng, which fixed the BIND/sysklogd problem out of the box.

I can understand that an OSS project depends on volunteers, but maybe they need to try a bit better to give people the impression that somebody's listening. The Feisty section on the Ubuntu forums reflects some quite bitter disappointment at the upcoming Ubuntu release.

"Use Dapper for stability", you say? I'm not looking for rock solid stuff here, just something that doesn't break in the most stupid kind of way, and that developers will fix once they realize it was broken.

Troubleshooting file and printer sharing in (yes, you read right) Windows XP

At my father's house where I'm currently staying there are two computers running Windows (one XP Pro, one XP Home). The XP Pro machine is dual-booting with Linux, but my father needs it to run mostly under Windows. They are connected and some folders are shared. The XP pro machine acts as a server, using ICS , for the other machine.

I've always had problems with the MS way of setting up network shares, because in many cases the errors that come up are obscure and the error messages aren't informative at all.

Since now there is NTFS write support for Linux (see http://www.ntfs-3g.org), I decided to convert the two FAT32 partitions on the server to NTFS.

There were two problems:

Quoting from Knowledge base article 304040:

Remote users may receive an "Access Denied" message on a share that they had connected to successfully before. This behavior occurs after the hard disk is converted to NTFS. This behavior occurs on Windows XP-based computers that have Simple File Sharing turned on that were upgraded from Windows 98, Windows 98 Second Edition, or Windows Millennium Edition. This behavior occurs because the default permissions of a hard disk that is converted to NTFS do not contain the Everyone group. The Everyone group is required for remote users who are using the Guest account to access the files To reset the permissions, unshare and reshare the affected folders.

Well, they could've made the error message more informative, ain't it so?

Then, I couldn't access anything on the XP Home machine, which is my father's laptop. It seems the Guest account is required to be on (from KB 308007, see "Troubleshooting file and printer sharing"):

Note: All network access to either a Windows XP Home Edition-based computer in a workgroup or to a Windows XP Professional-based computer in a workgroup uses the Guest account. Before you continue troubleshooting, make sure that the Guest account is set up for network access.

This can be checked with 'net user guest', as the article goes on to explain. I seem to have needed to turn on the guest account on the XP home machine in the Users dialog, even though it seems that's not needed (also from KB304040):

If you turn off the Guest account in the User Accounts Control Panel tool, only the guest's ability to log on locally is affected. The account is not disabled.

Hmm, that part I might need to go over again, since 'net user guest' was showing the account to be enabled. Hm.

Anyway, now everything seems to work, thank God.

Flash player 9 beta for Linux is out

Finally, there's a new release of Flash on Linux, using more modern APIs - e.g. alsa instead of opening /dev/dsp directly. I was waiting for it, since from this blog (belonging to Mike Melanson, an Adobe developer) it seemed like it was going to release soon.

However, I was a bit scared when Mike came up on the automake list to ask about how to statically link libstdc++. Why scary? Well, it became scary after Ralf Wildenhues's answer - especially the last part. After reading that mail (and trying the test program after a few days) I remembered about Ulrich Drepper's DSO Howto, especially the section about symbol resolution, and the peculiarities of ELF. I won't go too deep into that, but the bottom line is: even if statically linking in libstdc++.so.6, the system could still resolve the relocations with symbols from libstdc++.so.5, if it was loaded beforehand. Also, subsequent plugins loaded by, say, firefox, and linked with version 5 might end up using symbols from version 6 from the Flash plugin (unless the library was linked with -Bsymbolic). Of course, looks like something hard to reproduce, but that might cause random crashes, as I pointed out in that thread.

Oh well. There are other scary parts - the way they use the alsa libraries (dlopening libasound.so?? whatever happened to dynamic linking?), for example. That's the problem with third party developers - most of the time they don't know how to use a platform's development tools right.

So, I downloaded the two versions (standalone and plugin), and indeed, libstdc++ seems to be statically linked in. Oh gosh. Damn, I hate these half-working solutions - it works until proven differently, or until some corner case comes in. Why can't people try to come up with correct solutions, instead of hacks? Well, not that Linux makes compatibility particularly easy :).

They don't even support that many distros, RHEL 3 and 4, SuSE 9.x and 10.1 (no 10.0?). Oops, wait no Ubuntu? Apparently not :(.

Well, it ain't that bad. There are a few browser hangs, one which seems to be always reproducible. The sound and video are almost synchronized, but I guess that's also a problem on Winbloze. No crashes yet, but I'm on Ubuntu, which has mostly stuff linked with libstdc++.so.6. My SuSE 9.1 at work seems more appropriate for triggering bugs.

One thing which got me really confused was that about:plugins would still list version 7.0.68 of the 'Shockwave Flash' plugin, besides the beta 9 version. Thankfully, I looked for 'plugin' in about:config and came up with plugin.expose_full_path, an option which makes 'about:plugins' show the full path to plugin files. Lo and behold:

File name: /home/fane/packages/flash-player-plugin-9.0.21.55/libflashplayer.so
Shockwave Flash 7.0 r68

And then at the end of the list:

File name: /home/fane/packages/flash-player-plugin-9.0.21.55/libflashplayer.so
Shockwave Flash 9.0 d55

Why they make both versions visible is beyond me. Oh well...

Statically linked plugins

While looking for something completely different (isn't that how it always happens?) I found a very interesting discussion from three years ago on the binutils list. It all begins when one guy complains about what he sees as a bug in the linker:

One of the 'issues' he sees is the traditional UNIX linker behaviour, which I found out about on the Sun forums a while ago. The order of the libraries on the command line matters. The linker doesn't help you, as I remember was the case on Windows, and it sucks if you're coming from the Redmond OS to UNIX.

The other main issue is related to the registration of static plugins, which we've also faced in my team at work. Later in the discussion, one guy from ARM sums up the options in this case quite nicely.

Where I work, because of some misunderstood requirements, we have to use static linking and some 'registerable' components - you could call them plugins. (Now even the lead technical guys say: "hey, I never said we should use static linking!". Probably somebody was too lazy to change the old Makefiles, taken over from a different project? But that's a different story). At first the (dumb) approach was to add everything under --whole-archive, which would make the executables incredibly bloated, of course. Nowadays, only the registerable components (which use a mechanism very similar to the one outlined in this guy's first post) libraries are under --whole-archive, while the rest is wrapped around in --start-group/--end-group, because at a certain moment the number of static libs got out of control, and nobody wants to spend times sorting 100 static libraries. Thank God for the *-group options! I did the whole work, since nobody else seemed to be bothered by the binary size or want to do anything about it.

Now, reading the post with the options, there are at least two which look interesting, and I will look into them:

• the one about using the '-u' linker option, but that probably only works for dynamic libs


• the use of the '-R' option to GNU ld

At the moment I haven't checked them out. However, I think that you have to strive to do things properly, otherwise you might end up reinventing the same shit once again. And I'm not looking forward to that.

MandrivaOne 2007 quick tour

I've just installed MandrivaOne 2007 after playing a bit with the LiveCD. OK, long story short: DON'T TRY IT. At least not for upgrading from 2006. If there's an upgrade option, it's hidden somewhere; if I hit F1 for more options, all I'd get would be a grub prompt. That's all. In older versions you'd have a whole list of options. Oh well.

The 3d effects were at least funny to watch (the wobbly windows especially), but not that essential. However, it was quite slow to load, but it might've been the fault of the medium - a slow CD-RW. Hm. Where's the parallel init?

Then, at first I tried the LiveInstall option, assuming that it was maybe smart enough to figure out it needed to upgrade. Oh well, maybe I'm a bit tired - now that I think again, it doesn't make sense. I made sure I copied somewhere safe all the stuff you could call 'important', and then proceeded.

The result was a total fuckup, the old packages database was completely wiped, but the old programs were still there (I had my /home and / on the same partition, since I'm lazy, and since last time I tried sharing /home between distros didn't work anyway). So when trying to run basic stuff like 'ls' I'd get complaints about missing symbols in libc. Oh well...

Then I gave it a second try, this time with a fresh install. It was better, but still, I wouldn't say I'm impressed. The install is quite minimal (2.1 GB, but about 1100 packages), and seems targeted towards the non-technical user. Which is fine and dandy, but what if I'm a technical user? For fsck's sake, man is missing (even though some manual pages get installed - oops). Also info. Fonts are goddamn ugly - try viewing entries from the Debian bug database in Firefox to see my point. And that was at 1024x768 - I don't dare imagine how it would've been at 1280x1024, the normal resolution of my LCD. Ugly, ugly, butt ugly. Same as the old 2006.0.

But there are other bad news. It seems Mandriva has lost quite a bit of the vitality of old - guess it's expected when you decide to release once a year. I barely found the first version of Szaka's ntfs-3g, the one from July, while on Ubuntu they already have the last. Sigh. Yeah, I'm soo lazy. The PLF website is down, so is easy urpmi... Well, at least there are mirrors: on mandrivausers.org (but without PLF, so it's basically useless), on seerofsouls.com (this one has both PLF-free and PLF-nonfree - yipee).

But anyway - I'm kinda disappointed at this release, one year in the making. Of course, they've had their share of troubles, firing the founder and one of their top developers... But still. Too bad, really, too bad.

Afterthought: Or is it that only MandrivaOne sucks? Maybe I'll try the Free version some time, but not that soon.

Getting rid of useless stuff

Since I've been using Redhat/RPM-based distros for the most part of my 'Linux experience', I'm quite used to chkconfig. Especially because of the 'one size fits all' approach of having a zillion unneeded services on bootup taken by some distros. So, every time I'd give a new distro a try I'd inspect the list of services with 'chkconfig --list' and see what doesn't belong there. Usual candidates would be pcmcia and laptop stuff, RAID scripts, but also other things.

So, I was a bit surprised that Ubuntu/Debian's update-rc.d wasn't capable of that much, until I checked the manual page. Here's the relevant excerpt:

Please note that this program was designed for use in package maintainer scripts and, accordingly, has only the very limited functionality required by such scripts. System administrators are not encouraged to use update-rc.d to manage runlevels. They should edit the links directly or use runlevel editors such as sysv-rc-conf and bum instead.

Oh-kay, but then my question is: why is it that neither sysv-rc-conf or bum are installed by default - after all, the number of people coming from RPM-based distros to Ubuntu isn't that negligible, I would say.

Anyway, after googling a bit I found a post that suggested sysv-rc-conf would be a good chkconfig replacement, so I decided to go with it. And it's quite good - so similar to chkconfig that it could be a good idea to make 'chkconfig' a symlink to 'sysv-rc-conf' on installation.

So, after installing sysv-rc-conf (the usual apt-get install sysv-rc-conf) I got to removing useless stuff. NOTE: this list only applies for my particular needs. Study it carefully before applying the changes to your installation.

Here's a list, off the top of my head:

atd

This was deprecated in Mandrake 9.x at least, why Ubuntu enables it is beyond me.

bluez-utils

I don't have any Bluetooth devices; however, this is probably not the case with most people nowadays.

festival

Why the hell would everybody need speech-recognition software? Not to mention that the script in /etc/init.d has 'exit 0' at the beginning, so you have to comment out that line to actually use the package. This is just stupid - the script is being run anyway, I think not enabling the script altogether is more sensible.

laptop-mode, powernowd, pcmcia

I don't have a laptop, so I'm not interested in laptop features like frequency scaling.

mdadm

Software RAID support - I don't use that yet.

ppp

I haven't been using PPP for a while. Apparently others are using it, otherwise it wouldn't be one of the packages ubuntu-standard depends on.

I was tempted to also get rid of rsync, but then realized that it uses a similar scheme to festival: it would check /etc/default/rsync to see if rsync should be enabled, and only after that would run the actual server. This seems a bit stupid to me - why leave the script there if there's nothing for it to do? It's just wasted time on bootup, so after thinking a bit, I've decided to get rid of it.

After this time of playing with startup scripts and realizing that according to recommended usage, most scripts in /etc/init.d should have symlinks to them in /etc/rcX.d which would get run (either with 'start' or 'stop' as arguments), I think it's no wonder that Linux bootup can seem quite slow. Although, I must say, Ubuntu is faster than the SuSE 9.1 (OK, quite old) I have at work or the Mandriva 2006.0 on my othe linux partition. Can't wait for the new init announced for Edgy!

DNS configuration on Ubuntu, part III: enter resolvconf

After I realized that by default the dhcp3-client script on Ubuntu updates /etc/resolv.conf to include the nameservers provided by DHCP, I though about how to get it to write that to the forwarders clause in named.conf. After all, I had set up something custom for me some time ago. But since I'm lazy, I tried to find something already written for the job, and maybe build on that.

Note that if you don't plan to use forwarders, and this is recommended if you don't need to access domains local to your provider, the following part isn't useful to you.

So, the package I found is called resolvconf - Ubuntu package described here. I found it through a link from a bug report against Debian's dhcp3-client script - that was a reference to update-resolv, but it changed name in the meantime. It intercepts the DHCP client hooks (see here for some more details) and, instead of writing the ISP nameservers to resolv.conf, it writes them to named.conf. More precisely, to /var/run/bind/named.options, which you need to include in /etc/bind/named.conf instead of /etc/bind/named.options (NOTE: this is entirely Ubuntu/Debian-specific, SuSE has one named.conf with a named.conf.local where you can put some local settings; it also stores it in /etc/named.conf).

What resolconf does is take /etc/bind/named.conf.options and use it as a template for /var/run/bind/named.options. So, add the parts in my previous post on DNS configuration here. Then, you're all set to go; just make sure you have BIND running and then restart networking to trigger a new DHCP exchange:

sudo /etc/init.d/networking restart

There you go. That's all for today.

DNS configuration on Ubuntu, part II: configuring a local nameserver

So, now that I've explained why it's practical to run a local caching nameserver, let me explain my home setup. I'm currently using BIND-9.3.2 (the bind9 package in Ubuntu), but I'll be soon giving PowerDNS a try. Since I've had to play with BIND at the previous two places I've worked in, it's the one I'm most familiar with.

The Ubuntu package supplies a default configuration file which defines some zones, just so that you won't query nameservers on the internet about them. On SuSE 9.1, this only includes the reverse zone (the one that maps IP addresses to domain names) for the loopback address - 127.x.x.x. On Ubuntu, this includes the zones for 0.x.x.x and 255.x.x.x, besides the loopback. As far as I know, this is useful because a lot of the root nameservers see brain damaged queries for the domain name corresponding to either loopback or RFC1918 private addresses. Come to think of it, Ubuntu doesn't include those RFC1918 zones, because they're not that useful if you don't have a local network at home, using private (RFC1918) IP addresses.

Now, I usually make two changes in the configuration for BIND when used as a local resolver for a desktop machine:

• add listen-on { 127.0.0.1; }; to the options section, since we don't want to reply to queries from outside - the nameserver will be for local use only; if you have a local network set up, you might want to also listen on the interface facing the local network, if you want to use the respective box as a caching nameserver


• especially for BIND 9.2, add listen-on-v6 { none; }; if you don't have IPv6 connectivity; in this case, remember to also disable IPv6 DNS lookups in Firefox


• since there are a lot of idiot DNS admins out there, there are also many lame nameservers; BIND warns you about them by default, but if you hit many domains that are misconfigured like this, you will see many messages about them in the logs; you can disable them by adding the following to the logging section;
  

  
         # Don't log lame server messages.
         category lame-servers { null; };
  

  
  That's the easy way. However, I like to redirect the messages to a different file, so that I could at least notify the lame DNS admins about their problems. As some people say, DNS is a collaborative effort, it being a distributed database, and you know that most things which aren't idiot proof are bound to be broken...
  

Then again, there's another problem: what if I get my IP address and list of nameservers with DHCP? How can I automatically update my named.conf file to include those nameservers? They are always the same, but if you decide to hardcode them and the provider changes them, you're screwed.

DNS configuration on Ubuntu, part I: the case for a local caching nameserver

One thing which is surprisingly lacking from the C resolver library (libresolv, part of glibc) is caching. Support for asynchronous DNS queries is also lacking, but that's less surprising. So, all calls to gethostbyname/_r() and getaddrinfo() are actually hitting the DNS servers in /etc/resolv.conf.

Many people don't distinguish between a stub resolver (the one in the C library) and a full-blown resolver, or caching (and recursing) nameserver, which is what those IP addresses in your /etc/resolv.conf represent. The stub resolver just asks a question to a caching nameserver among the lines "what is the address of www.foo.bar.com?", and then interprets what it receives back. However, nameservers have a more difficult job - you can get a glimpse of how they have to operate by playing with 'dig +trace'. What you will see is something along the lines: query root nameservers for the .com gTLD nameservers, then query one of the gTLDs for www.foo.bar.com, then maybe receive another referrral to the nameserver of bar.com, etc.

On Ubuntu, dig is part of dnsutils, which is unfortunate, since dig is part of the BIND distribution.

However, if all you use is a web browser like Firefox, you won't get hit that hard, because Firefox caches answers to DNS queries (for 15 minutes - see here,
- a google cache link on global server load balancing, which touches on browser DNS caching). However, whenever the cached records expire (or the first time around), firefox will hit your provider's nameservers, generating needless network traffic. Why?

Well, every DNS record has a TTL (time to live) field, which Firefox doesn't honor (because it's not visible in the structure returned by gethostbyname/_r()). If a record has a TTL of 1 hour, that means you hit the network 45 minutes too soon (and 4 times more than necessary).

Now, consider what would happen if the caching nameserver would be running on your computer. All traffic would go on the loopback, which is much faster and doesn't involve the network. Since the nameserver would honor TTLs, it won't make unnecessary queries. Depending on the domains you want to access, you could even bypass your ISP's nameservers completely; however, some ISPs might have some internal domains accesible only to their clients, and in that case you might need to query them sometimes. This can be easily solved with the 'forwarders' feature - you can have your local nameserver forward all its queries further to one of the ISP's nameservers. If the provider's nameservers are down, you can however configure your local caching nameserver to bypass them (although, as we'll see, it might involve disabling the forwarders).

Note that if you ask on bind-users about using forwarders, everybody will tell you that it's a bad idea - there is extra latency involved, and you have one more point of failure in the provider's DNS servers - if they fail, you won't be able to access the internet in the case you use forwarders exclusively ('forward only' option in BIND) or, in the best case, experience significant delays - the DNS query to one of the forwarders must time out, and then the local cache is free to try to resolve the name on its own. So, use it with care, and only if you want to be able to access some domains that you couldn't by having your local nameserver walk the DNS tree from the root nameservers down to your provider.

OK, I don't know how clear I have been, but I can recommend some extra reading: Cricket Liu and Paul Albitz - DNS and BIND, fifth edition. All the DNS veterans on bind-users swear by it.

Ubuntu firewall revisited

A bit of grepping in /etc helped me understand why the numerous runs of ubuntu-firewall.sh (home page here). Ubuntu uses /etc/network/interfaces in order to, well, configure network interfaces and the guys seem to have thought that it's good to put there a number of devices, just in case, all of them made to use DHCP. That's not too good for me, since I only have one network card connected to an ADSL modem. Also, the firewall script is called every time dhclient is called (the dhclient-exit-hooks script called when dhclient exits has '/etc/init.d/ubuntu-firewall.sh reload' in it) - thus, the multiple runs.

So, we have eth0, eth1, eth2, wlan0 and ath0 on by default, and dhclient is run for each of them (setting RUN=yes in /etc/dhcp3/dhclient-exit-hooks.d/debug and examining /tmp/dhclient-script.debug shows this clearly), and also the firewall script. So I got rid of the unneeded interfaces. Now, the debug log also shows that the dhclient exit hooks have been called twice for eth0. Every time the scripts are called there is a number of variables that are passed to it, and the one that caught my eye was 'reason' - the first time around it is 'PREINIT', the second time it's set to 'BOUND' - after the address is acquired.

The reason values are enumerated and explained in the dhclient-script man page. However, even after reading the man page I'm not that sure whether it makes sense to reload the firewall script when reason is 'PREINIT', since that stage is hit even for non-existent interfaces. Maybe I could run 'ifconfig interface' in the script and check whether anything happened with it (if there's no alias for the ethX or whatever other device, I guess it shouldn't show up under ifconfig). Also, I think server programs like sshd should be run *after* dhclient3, so before acquiring an address I don't think there's any danger of a hacker intrusion (unless it's something DHCP related). Oh well, who knows.