Not what it could have beentag:growler.woaf.net,2009-06-06:/not_what_it_could_have_been//12010-04-29T19:46:07ZJon's life and eventual death.Movable Type 4.33-enOil spillstag:growler.woaf.net,2010:/not_what_it_could_have_been//1.72010-04-29T19:17:46Z2010-04-29T19:46:07ZThere's been a fair bit of frothing in the press about the Deepwater Horizon oil rig exploding, mostly focusing on the escape of oil. I noticed a BBC News article today which has some rather suspect claims. I'm sure they're...Growler
article today which has some rather suspect claims. I'm sure they're not the only ones, but even basic fact checking raises a few questions.
]]>
Meanwhile, a firefighting expert said the disaster may become the biggest oil spill ever.
Mike Miller, head of Canadian oil well firefighting company Safety Boss, told the BBC World Service: "Probably the only thing comparable to this is the Kuwait fires [following the Gulf War in 1991].
"The Exxon Valdez is going to pale in comparison to this as it goes on."
See. Terrifying. It's going to be as bad as someone blowing up and setting fire to some 700 oil wells, which then took 8 months to put out... Somehow. Even if we assume he was actually just talking about the Gulf War Oil spill, that's still pretty scary numbers - several oil tankers, and an entire refinery worth of oil were dumped into the Gulf.
Lets do some maths...
In the same article we're told that the rate of leakage is 1000-5000 barrels of oil per day. I'm going to round that up to 10,000 because it's a nice number. According to Wikipedia 1 tonne of oil is about 7.33 barrels, giving us 1364 tonnes of oil per day.
So, to come in first it needs to leak for a mere 997 days. The BBC article claims they'll have it fixed in a matter of weeks, the Wikipedia article on the subject goes for a few months. Either way it's not going to beat the Gulf War oil spill.
What about the Exxon Valdez? That was a comparatively small oil spill, leaking only 35,000 tonnes vs. the 100,000 tonnes of the next one up on the list. The Deepwater Horizon only needs to leak for 25 days or so to exceed that, so there's a good chance it will... not sure it'll cause it to "pale in comparison" though.
Just in case he really was talking about the Kuwait fires lets check the figures: they were leaking some 6 million barrels per day, and left 5% of the country covered in burnt oil by-products. Yes, it's a small country, but that's still an immense environmental disaster.]]>
Moving jobs (again)tag:growler.woaf.net,2010:/not_what_it_could_have_been//1.62010-02-25T18:10:28Z2010-02-25T18:27:59ZYes, it's that time of year again, I'm defecting.Since leaving ADVFN I've been chasing the money and moving increasingly towards pure ops roles. I don't enjoy that sort of work. I don't like supervising releases, I dislike doing deployments, fiddling...Growler Since leaving ADVFN I've been chasing the money and moving increasingly towards pure ops roles. I don't enjoy that sort of work. I don't like supervising releases, I dislike doing deployments, fiddling with config files is tedious, and being called at 2am sucks. The only bits I do enjoy are developing useful tools, and working out what chain of events led to an complex mass of interconnected apps falling over. When there's a limited number of tools that need writing, and a comparatively simple app to support (and with me having no access to the source), it's just not challenging enough to be interesting. So, erm, aren't I supposed to be senior engineering staff? Why am I doing support work? I get the feeling that I'm employed just in case something really bad happens, and the rest of the work is just there to fill the time when things are working.
Fortunately I've been offered the chance to escape into a development role. Sure it's going to mostly be PHP, and yes it's a pay-cut, but I'm actually hoping that I might finally be making a good employment choice. That'll be a first.
I start at the new place on the 12th April, however I've actually only got 3 working weeks left as I'm off skiing for 3 weeks before then. ]]>
New MythTV setuptag:growler.woaf.net,2010:/not_what_it_could_have_been//1.42010-02-25T08:40:39Z2010-02-25T21:12:04ZI decided it'd become time to replace my current MythTV setup. The hardware had irritated me for some time (noisy, and required irqpoll to boot), and the software was now some versions behind and seemed to have mangled its database....Growler
MythTV setup. The hardware had irritated me for some time (noisy, and required irqpoll to boot), and the software was now some versions behind and seemed to have mangled its database. So it was time to purchase some new toys, and lose multiple hours to the headaches of configuring them. It wasn't as easy as expected, and there were some exciting pitfalls that hopefully this will help others avoid.
Yes Phil, that means this is going to be boring.
]]>
So, the hardware:
ASRock NetTop ION 300 HT: this is a tiny atom based system, with external power brick. It's dead quiet, and having an Nvidia ION it has VDPAU support. This gives it the ability to decode 1080p mpeg2 streams despite the lack of CPU power. It even has a built in IR receiver.
Hauppauge WinTV Nova-T 500: a dual DVB-T tuner card. This really turns out to be a USB hub and two of their USB tuners on the same card with a shared antenna. Capable of tuning to two different multiplexes simultaneously, when it works. This was in my old system and could be quite troublesome.
Hauppauge WinTV Nova-HD S2: The only DVB-S2 card I could find. While I don't need S2 yet for FreeSat I decided that it'd be worth future-proofing.
And finally, a random box I'd had lying about for some years. It seems to contain an Athlon T-Bird 1.2Ghz. With the addition of a cheap SATA controller it can take the couple of 500GB SATA drives I had lying about.
And on the software end of things I've chosen to use the Mythbuntu Karmic CDs for install.
So, onto the build.
The first handful of headaches were down to the age of the backend system. Flat BIOS battery had led to it enabling power off on CPU fan fail - not so useful when there's a huge Zalman passive heatsink providing the cooling. CDROM drive had filled with dust and could no longer read disks. The biggest headache by far was down to my choice of cheap SATA controller (purchasing decision based on 0 research), it turns out that the VT-6421 cannot be booted from! This presented a bit of an issue, so I've booted off an old PATA disk that just has grub and /boot on it.
Now something slightly interesting: getting the latest Mythbuntu to do software RAID. They've decided to kill off the alternative install CDs, and the "desktop"one doesn't allow you to do RAID. This isn't a huge problem, but requires a bit of mucking about:
Start with the "try mythbuntu" option. Fire up a terminal and apt-get install mdadm, this will give you the software raid utils. Now partition your disk using fdisk (you might find cfdisk easier if it's installed), creating the partitions you want to use for RAID and setting them to the RAID autodetect type. I decided I wanted a 12GB RAID1 for /, 1GB per drive of swap, a 100GB RAID1 for storage of things I care about, and the rest of the disks as RAID0 for storing recordings (anything I care about I'll move to the RAID1.) Use mdadm to create your RAID arrays. Something like "mdadm -C -n 2 -l1 /dev/sd[ab]1" to create a RAID1 of sda1 and sdb1.
You can now launch the installer. When you get to partitioning you want the "advanced" option, and create the partitions you want under your /dev/md[0-9] devices rather than under the normal disk devices (although don't forget your swap partitions.) Install as normal, however do NOT reboot - it's not going to come up if you do! The Mythbuntu install lacks things from the initrd that are required to boot when / is on RAID.
Go back to your terminal and, if it's not already, mount your / partition. Bind mount proc and dev under it (e.g. if mounted on /target "mount -o bind /proc /target/proc && mount -o bind /dev /target/dev"), and chroot into it ("chroot /target /bin/bash"). Edit /etc/initramfs-tools/modules and add "raid0" and "raid1". Now apt-get install mdadm. It ought to froth about for a bit and build a new initrd. Logout of the chroot, unmount the drive, and you're hopefully safe to reboot. You might need to muck about with grub before you reboot, but I didn't because /boot was on a different disk...
Reet, that's a backend installed. The frontend was nice and painless, although for some reason the Mythbuntu install for a dedicated backend fails to bind various services to the network interface. The myth backend service needed reconfiguring in the settings app to listen on a useful network interface rather than loopback. I also had to reconfigure the mySQL bind-address (in /etc/mysql/my.cnf) to 0.0.0.0. The install onto the Nettop frontend was painless, and it happily came up talking to the backend.. after about an hour of mucking about trying to work out why it couldn't connect due to the backend config issues.
The next thing to do was make it useful! For starters the dvb firmware package needed installing, but after that and reboot dmesg was full of angry messages about the DVB-S card firmware. Looks like the Ubuntu package contains broken firmware for this card :-(
Fortunately, after some experimentation with different versions, I found that the following works fine: wget http://www.wintvcd.co.uk/drivers/88x_2_124_27191_1_WHQL.zip unzip -jo 88x_2_124_27191_1_WHQL.zip Driver88/hcw88bda.sys sudo dd if=hcw88bda.sys of=/lib/firmware/dvb-fe-cx24116.fw skip=105768 bs=1 count=32674
Now after a reboot the firmware loads correctly. I also chose to build and install the latest v4l drivers. This probably isn't actually necessary, but I was desperate due to a problem I had with the DVB-S card...
The DVB-T card was simple, and seems a lot happier in recent kernels
than it used to be. If I start seeing the USB disconnect issues again
I'll post about fixing it.
So it's time to add the cards in the myth backend setup. The DVB-S card needs to be told to talk to a LNB in the DiSEqC. The default (Universal) works fine for me. The other card shows up as two devices (because that's what it really is.) Video sources is where things get a bit odd. I've created two of them (one for DVB-T, one for DVB-S) and set both to use the RadioTimes grabber. Two are required otherwise it'll apparently get utterly confused and assume that all the cards can do the same channels with the same tuning which, obviously, they can't. The two DVB-S cards, being identical, can share the same input.
Under input connections I associated the DVB-T cards with the DVB-T source, and ran a scan (only one needed per source, not one per card.) This took a while but was happy. So, onto DVB-S.
Sadly I ran into a big headache at this point that wasted several hours of my life. Most of the docs seem to recommend using the linuxtv "scan" command to generate a channels.conf, then importing it into Myth to start the scan. Unfortunately this failed for me. While scan produced a sensible looking channels.conf, Myth decided to repeatedly detect the same 7 channels... several hundred times. Not happy, and it led to a big warning about conflicting channels. Initially I decided it must be a bad driver, hence the v4l upgrade, but that didn't fix it.
In the end I gave up and found the scan details for Astra 2D, which seems to have most (all?) of the Freesat channels on it. Frequency: 10788000 Symbol rate: 2200000 Polarity: vertical The rest can be left as default. Started a scan and after a while it started picking up sensible numbers of channels.
Woo, backend is complete... ish.
The frontend, meanwhile, needs some work. The CPU isn't powerful enough to cope with playing BBC HD, however the ION GPU has VDPAU support and is easily able to deal with it. It needs the NVidia proprietary drivers installed, and then under one of the pages (3, Playback profiles) of "settings -> TV -> playback" there's an option to set the system to use VDPAU. That needs to be set to "VDPAU Normal". The hardware doesn't seem to cope with "High Quality." I also had to disable Compositing to fix tearing (sudo nvidia-xconfig --no-composite.)
There's also the matter of the remote control, a driver (with instructions provided) has to be downloaded from the ASrock site: http://www.asrock.com/Nettop/download.asp?Model=ION%20330HT&o=Linux
And there, mostly working. I haven't got round to sorting out the channel listings yet, or configuring the RT grabber. That's all on my todo list for today.
Edit: The odd i2c errors and USB disconnects still happen to the DVB-T card with the latest drivers. To fix I needed to set a 500ms tuning delay and only enable active EIT scanning on one of the capture cards.
]]>
Why I dislike mySQL (an ops perspective)tag:growler.woaf.net,2009:/not_what_it_could_have_been//1.32009-12-31T11:00:00Z2009-12-31T11:15:18ZMuch has been argued over why mySQL may or may not suck. Most arguments on the subject focus on SQL standards compliance and perceived performance. Some will argue that mySQL is really easy to install and make work, and, well,...Growler
]]>
Several
of my issues focus around disaster recovery. Disasters come in many
flavours, ranging from some toss-pot deciding to power cycle your
database server, up to your data centre burning to the ground. Your
mission critical data needs to be recoverable in the shortest possible
time. It needs to be correct, preferably contain ALL of those pesky
transactions that customers have been told were successful, and most
definitely it needs to be consistent. Consistency in this context
requires that those irritating operations that occurred as part of a
single transaction are either all still there, or are all missing.
Everyone likes to pretend that nothing bad will ever happen to their
data, but it's bound to. At some point you'll have a PSU failure, a
breaker trip, motherboard explode, your hosting provider reboot your
database server after getting confused about rack locations, or just
that you've been daft enough to stick it in a VMware ESX cluster and HA
has gone insane.
First off a bit of a rant about myISAM. This
rant contains nothing new, it's all well documented. Unfortunately
people still haven't got the hint and stopped using it for critical
data. As a starter it's really prone to corruption. This is mostly due
to writes to tables not being an atomic operation, and not having any
way to deal with a write operation that only partially completed (due
to power failure or kill -9 or crashing horribly.) Your data is going
to get maimed if something bad happens, no matter what you do. If
you're lucky you'll just lose recent transactions, but sometimes random
chunks of your table will be consumed. But nevermind, because it turns
out that write operations of any sort are horribly bad for performance
and something best avoided. MyISAM tables rely on taking entire table
locks when a complex write operation (most often reuse of space that
has been freed by earlier deletes) is in progress. Any selects
currently running are allowed to complete, but any that come in after
your write now sit waiting for the lock to be released. This makes a
combination of long running selects or lots of concurrent selects plus
any sort of write extremely bad news. The good news is that setting
concurrent_inserts to 2 in your config file will mostly avoid the
locking issues, but this wont save you from the data corruption issues.
An inescapable problem is that after a crash you MUST run a myisamchk
or there's a good chance that things will start behaving in unexpected
ways (selects might just hang, some rows will go missing, etc), and on
a large table this will take forever.
Now something about
Innodb. Until a few versions ago mySQL defaulted to sticking all your
Innodb table data into one giant file. Unfortunately lots of people
still haven't moved to using a file per table. This is a tad evil as it
turns out that free space in Innodb tables isn't reused, and deleting
stuff doesn't reduce the size of the file. If you're using a file per
table you can rebuild each table that you want to shrink, but if
everything is in one file then welcome to a world of pain. This
irritating drawback has caused people who should know better to use
myISAM tables in their applications. Another headache with Innodb is
that by default mySQL doesn't sync writes to the transaction log to
disk. This means that on a crash/power failure/etc, while you wont have
corrupted tables, you might lose the last N transactions. This may not
sound so bad, but it's still a bit of a pain when your app believes
them to have succeeded, and means that by default it's not ACID. A
quick config change can fix this, but there will be significant
performance implications.
There are some additional problems
that come with mixing table types. I've run into problems after a
failure where the Innodb tables all lost the same time frame of data
(since the last transaction log sync), but the myISAM tables lost a
completely different time period. This has left my databases in a
horribly inconsistent state.
So, on to backups. You want to make
a backup of your mySQL database server. This isn't exactly an
unreasonable request, accidents do happen after all.
An obvious
approach to backups is to mysqldump your database. Unfortunately if you
have any myISAM tables then unless you take a lock on all of your
tables while you do this dump it's not going to be consistent: an event
that requires inserts to two tables might end up inserting to table A
after the dump has finished with it, but inserting to table B before
your dump has reached it. This is far from ideal and potentially very
unpleasant if you need to restore from said backup. If all your tables
are Innodb then you can do the dump in a transaction, which will
guarantee consistency (well, as long as no one is going about making
DDL changes while your backup runs, but that's a reasonable assumption
unless your app is utterly broken.) Taking a lock on all your tables is
probably going to lead to an outage, so isn't the best of options. Even
without the lock, and accepting the consistency issues, myISAMs
concurrent access scheme (discussed earlier) means that on a large
table your dump will probably cause issues. A further down side of your
dump is that it's not incremental. You're going to get a giant file
containing all your data every night.
As an alternative you
could do a "hot backup", copying the raw mySQL data files off to your
backup server. This is a really bad idea. Updates to the raw data files
are NOT atomic, so copying a load of data files out from under a
running mySQLd will corrupt tables. If you're lucky it will at least
start up with a restore from your "backup", but any myISAM tables being
written to when you took your "backup" will be horribly corrupted, and
Innodb doesn't like this sort of behaviour too much either. The safe
way to do this is either by shutting down mySQL, or taking a lock on
everything and flushing tables, before doing your backup. This
obviously leads to application issues. As per taking a mysqldump of
your tables, this method also fails to take advantage of incremental
backups as any table that changes will get backed up in full every
backup run.
Finally you could rely on the snapshot feature of
your filesystem. This is an extremely dangerous option if your file
system doesn't support snapshotting multiple files simultaneously. At
present this isn't a particularly appealing option but in the future,
with file systems capable of low cost snapshots, it may become the
norm. It still suffers from the lack of incremental backup
possibilities.
Right, so backups directly off your primary
database server are not a great plan! Lets try this native replication
thing, and run our backups (and inconveniently resource intensive
reporting queries) off a slave. That way we can use any of the above
methods without any risk to our actual service. Hell, we may as well
throw in some nice slaves for our application to use for read-only
purposes (this throws up some interesting problems by itself, but they
are in no way unique to mySQL.)
mySQL native replication is a
feature regularly used to win the PostgreSQL vs mySQL argument. It
seems to have been a major factor in many company's decisions to use
mySQL. It's true that PostgreSQL doesn't have built in replication (at
least, not yet), but it does have several 3rd party replication
solutions, and they all work a hell of a lot better than mySQL's native
option. At this point I must admit that I have not used native row
based replication. This is because I'm yet to come across a replication
set that contains no myISAM tables. Unfortunately myISAM rules out
anything but statement based replication (although more recently
"hybrid" is an option.)
A massive headache with statement based
replication is that a statement run on one server may not produce the
same results on a slave. An obvious example is use of RAND(), a less
obvious one is due to use of NOW(). An extremely unpleasant case occurs
where tables in the master and slave are out of sync (myISAM
corruption, or lost Innodb commits, after a crash being a common
cause), at which point an INSERT INTO...SELECT... will have different
results on different nodes, amplifying the differences (and often
leading to unique constraint violations at a later date.)
Inconsistency
between nodes is something you really don't want, and avoiding it is a
right pain. If you have a crash of your slave you MUST rebuild it from
the master rather than getting it back up and replicating. If you have
a crash of your master you should either fail over to a slave, or
rebuild ALL of your slaves from the master.
Of course it turns
out that after a crash there's a good chance replication just wont
start again anyway. By default the binary log (basically just a
statement log, which is used for replication) is not synced to disk.
This means that in a crash of the master it'll probably lose a load of
writes. When your master comes back up the slaves will have replicated
a load of rows that the master no longer knows exists. The slaves will
refuse to restart replication because the binary log position they
request from the master doesn't exist anymore. While you can manually
set a new bin log position this will leave you in a rather unpleasant
state, with slaves having some rows the master doesn't. To add to the
fun you also can't guarantee that what's in the binary log for
replication after a crash contains everything that has actually been
written to the data files. Your best option is to rebuild the slaves.
Unexpected loss of a slave can leave you in a similar state thanks to
the unpredictable myISAM table corruption that is likely to be visited
upon you.
So, rebuilding a slave... a simple task, right? Yes,
if you have another slave that you can shut down and take a copy from.
If you don't then you're going to need to take a consistent dump from
the master. Hey, doesn't this sound like the backup problem from
earlier? Yep, that's because it's basically the same issue - you MUST
have a consistent snapshot, and, in a mixed table type environment,
that means locking everything (and thus probably downtime.)
The
general headache farm that comes with native mySQL replication leads a
lot of people to try to use drbd to provide a hot-standby capability.
This works surprisingly well if you have only Innodb tables, however
you need to remember that promoting your secondary after a master
exploding is equivalent to dealing with a box that has just been
unexpectedly power cycled - you're going to have exciting problems with
loss of committed transactions and, most irritatingly, myISAM table
corruption. Before your secondary can be started you've got to do a
myisamchk, and this can easily take hours if you have large myISAM
tables. Your hot-standby isn't.
So, isn't all of this normal?
The massive user base, including many massive names, suggests that
mySQL is the best of breed. Surely the alternatives (aside from those
costing lolmoney) have all these problems and more! Fortunately not.
I'll stick to PostgreSQL as that's the alternative that I'm most
familiar with. In the PostgreSQL world hot backups are a nice and
boring affair, have the possibility of continuous archiving of
transaction logs to a remote host, and yet work rather nicely with
incremental backup solutions. There's only one table type and it's well
behaved in crashes. Replication is a bit of a sore spot, but, despite
its complexity, I've had less issues with the nightmare that is Slony
than I have with mySQL native replication. Slony even deals with the
entire initial replication sync all by itself and without taking your
master off-line.
Oh, and mySQL "multi-master" replication IS
NOT MULTI-MASTER REPLICATION. What you are doing is abusing the fact
that mySQL replication has NO sanity checks, and doesn't have any way
to enforce read-only on slave tables. You've just set up
master->slave replication in both directions, and there's nothing to
stop a large number of very, very bad things happening so you're going
to have inconsistencies between your database servers. Additionally
you've probably done it because you think that your app is write heavy
and can benefit from having more than one database to write to... GUESS
WHAT: You're wrong. All of the statements executed on master A are sent
to B, all those executed on B are sent to A. Both database servers have
to execute everything. You've just created yourself a recipe for
disaster with little to no benefit. Even the documentation says this,
stop trusting random blog articles (yes, I know this is a blog article)
by random people who do one query per year on their database.
So,
what is good about mySQL? It's quite simple and there's no pg_hba.conf.
It has nice SQLish commands like "SHOW TABLES", rather than psql's
backslash commands. If you want insane single threaded, sequential,
select performance on a table with absolutely NO writes then myISAM is
the table type for you! On a more serious and useful note, the massive
selling point of mySQL is the plugable storage backends. Sure, for
normal usage these are one of its headaches - why can't the damn
developers pick one storage engine and make it work well, rather than
changing their minds every 30 seconds and leaving yet another in a
half-arsed state. As a developer, however, you've got a SQL front end
and supporting libraries, and can throw together backends that
interface with some obscure, home made, storage systems. You're able to
make various odd services available through the same front end server,
using a common query language that quite a lot of people understand.
This is probably the reason that quite a few big names with specialised
requirements are making use of mySQL.
Oh, and insert a rant about how shit the mySQL query planner is here :-)
Coming soon, a "what I dislike about PostgreSQL."]]>
Why I can't deploy IPv6tag:growler.woaf.net,2009:/not_what_it_could_have_been//1.22009-06-21T16:08:12Z2009-06-21T23:27:41ZI'm one of those people who'll use any excuse to play with the latest or most interesting technology. It should therefore come as no surprise that I'm keen to see IPv6 deployed both at home and at work. Sadly I've...Growler Home is the simplest problem to explain. While my ISP (Enta) are able to support native v6, I own a Draytek 2820 which lacks v6 support. Draytek inform me that they have no interest in implementing support. At $0rkplace our office Internet access provider lacks any interest in providing v6. These two problems mean that I have to resort to tunnelling to have any hope of testing any v6 services that I deploy. Not a huge problem, but at least a little irritating.
At $0rk we're, in theory, in a great situation. We have complete control over our shared web hosting environment, control over the load balancers and firewalls of our enterprise customers, and run DNS for almost all customers. This ought to allow us to make almost any customers' site/service available over v6 without them having to know or care. Unfortunately this isn't the case.
The big problem I have is providers with partial v6 tables and consumers with broken DSL modems. If I were to add an AAAA record for our main web site then hosts that support v6 will prefer that over the A record and attempt to make a v6 connection. If that host is on a network that doesn't have a route to our v6 prefix then they'll hang around for a timeout before attempting a v4 connection using the A record. This timeout is long enough to easily be noticeable, and quite unpleasant, to the user. Even if this problem is limited to a small number of end-users I'm still in trouble: how do I explain to a customer that their web site is "slow" for a user because I've made it available over IPv6? They'll just want me to disable this evil IPv6 thing.
Of course, this problem doesn't just impact the content provider. A v6 enabled user will find that a site that has AAAA records but that lacks v6 connectivity (due to either content or access provider taking only a partial table) to be horribly slow. This encourages the user either not to use the site, or makes them disable IPv6 support. Not ideal.
Providers who are experimenting with providing services over IPv6, but who lack a full table (or a working network), are making IPv6 deployment unpopular from an end-user perspective. A provider with a broken network encourages end users to blame IPv6 and disable support in their OS. Additionally it discourages content providers from making services dual stack.
Until this problem goes away (which it won't) I cannot justify making our services (and our customers' services) dual stack. Yeah, sure, I can do a Google and have new names for my v6 services but who the hell's going to make use of them? Why would anyone make the effort to use http://ipv6.0rkplace/ rather than http://www.0rkplace/? ]]>
I has a blog!tag:growler.woaf.net,2009:/not_what_it_could_have_been//1.12009-06-06T14:54:44Z2009-06-06T19:11:50ZI've decided that it's time to fail to have a blog again. Despite my intention to actually make more than three posts this time round, I expect I'll get fed up and kill it off again in a couple of...Growler I installed Movable Type on dog a week or so ago and prepared a nice long boring entry about how much I hate BT. Dog responded by crashing and burning, completely nuking the file system. We've had to reinstall on new drives, and the last "backups" are actually the drives from the previous incarnation of dog which was replaced with a Dell 1950 in December 2007.
So far we're not really sure what happened to cause the data loss. Due to fail on our part we'd not enabled nproc ulimits, and this allowed a user to (unintentionally) trigger an uncontrollable fork bomb that floored the server. I decided to get the datacentre to power cycle the server, we've needed to do a kernel upgrade for ages anyway, and it seemed to be the quickest way to get dog back. It failed to come back after the power cycle. I headed to the datacentre to take a look, expecting it to be missing the firmware-bnx package since the upgrade to etch and a half (damn you Debian.) It was sitting at a fsck failure. Running a fsck by hand there were a couple of fixable errors, which I wasn't too worried about considering the hard reboot. Dog came back up and looked fine so I headed home.
Sadly all was not fine. About an hour later there was an oooops somewhere in the ext3 code, and the file system became unreadable. We requested another power cycle, and this time dog came back up without additional human intervention. While I was able to log in there were a lot of ext3 errors in dmesg and the root file system had been automagically remounted read-only. Not a happy file system. Suddenly ls decided that the only file on the entire file system was /lost+found, although commands were still executable if you knew the path to them. I decided that another fsck was a good plan, but this led to further corruption and I terminated it once it started doing scary things like zeroing superblocks and trashing the ext3 journal. Now nothing was exeutable anymore (permission denied) and several things that should be files had become directories.
In conclusion the file system is toast. We're hoping that we might be able to recover some data if we're lucky, but... ]]>