Much of medicine is really just boring magic.

However, its not actually miraculous, its just a bit technical and specialized and its got me to thinking about how much time, and effort goes into keeping us ungrateful gits alive.

Fabrice Muamba's remarkable story has been a great piece of positive press for the NHS, and particularly demonstrates how this complex chain of people and technology comes together to really save lives. (It feels a bit corny just writing that, but if there is one thing that deserves it, this is it)



  • 1813 GMT: Muamba collapses on the pitch due to a cardiac arrest 41 minutes into Bolton's FA Cup quarter-final with Tottenham at White Hart Lane 
  • 1819 GMT: Medics move Muamba to an ambulance
  • 1901 GMT: Muamba arrives at the London Chest hospital
  • 1931 GMT: Muamba receives 15 defibrillator shocks in total before his heart starts beating again 
(BBC Sport, http://is.gd/7qBpL2)


total treatment, no detail is too small.

im thinking of something like this for a blog post...

The Quiet Health-Care Revolution - http://is.gd/bkt2b1
"The elderly are prone to falls for many obvious reasons, including weak limbs, impaired vision, and medication side effects. But Dan’s doctors knew that some less obvious causes included shag carpets and long, untrimmed toenails. Because of this, they’d sent someone from the clinic to visit Dan’s apartment and make sure that his daughter replaced the 1980s-vintage carpets with low-pile rugs."

"Dan and his fellow clinic patients are one-fifth as likely as comparable patients elsewhere to suffer falls. "

Models for developer environments


So I have been incentivised to put some thought and research into the various models available for providing platforms for developers in a secure manner, and this is a short summary of what I came out with...

Basically 3 ways to try and mitigate the dangers of un fettered creativity, without putting too much of a sysadmin damper on the use of fucking stupid configurations and settings.


Security!Security!Security!

(I am working on this email as a general send out on policy on securing systems but i guess it is worth sticking on the blog)

Security!Security!Security!

1) I guess the thing to start with is mentioning that "public key cryptography is very secure indeed".

You only distribute the public part of the key, you know.. post it on the internet, attach it to your emails, print it on a tee-shirt. It doesn't matter what you do with it in terms of confidentiality. [1]

However instead of the password, it is the private key that becomes the weak link in the security process. Protect that with your life, well maybe not, but it is very important. Keeps it on your computer and don't copy it to remote machines. Ever!

2) the next thing is that "You are going to get hacked eventually"

Given enough time any non-trivial enterprise is going to have a whole bunch of people, systems, software and screw-ups, eventually one of which is going to let some naughty person have access to something they should not have access to.

Hence be aware of what the effect of the compromise of this system will have on the integrity of the whole system.

If you are writing some tool that requires administrator rights, and runs on the internet, and has passwords for the database and users systems, think twice, think 3 times. Go to the pub get drunk, then think a fourth time. Because you are going to look mighty silly when its your fault!

3) There are 2 golden rules for mitigating the extent of an intrusion

i) The principle of least privilege

Run services as a user with the least possible rights to complete the task. JDBC database connection users should be database specific, if your tomcat is hacked, then your potential losses are limited to that particular database, and not (if you are connecting with root, or dba) the whole 27 databases on that mysqld.

ii) The principle of separation of privileges

If your script needs to install a user, or needs to install a package as well as some other stuff.

Then write 2 scripts. one to install the package or the user, and one to do the normal day-to-day stuff. And protect the admin script as a core part of the system.


4) There are always going to be exceptions. You should know what they are, and why they are there.

Your core infrastructure is an example, your installation tools and scripts, backup tools have to have access to stuff. However this is where you have to be sensible.

Use tools for important tasks when they are available, rather than write your own stuff. If you find yourself chmod'ing 777 then you are an idiot, turn around and walk slowly away from the computer... only joking ;-)









[1] in openssh authorized_keys format the file contains a comment if which is the same as your user-id, gives an attacker a valid account name to work with.

selectively restoring a renamed database from a chunky sql dump file.

with a  multi GB dump.sql.gzip file, and the requirement is to restore one database from the multiple in the file, the first instinct is to break out some grep/sed/awk and generate a sub-set sql file that just contains the "CREATE DATABASE" and insert statements and apply that to the main database.

However if you have some requirement to rename the imported database, or some other import time alterations, this is something of ball-ache.

one way to un fuck this situation is to start another mysqld instance on another port, and then use the mysqldump/mysql tools to do the work of selecting and renaming the databases and tables as required...


things not to type into google (part 2)


"man kill"

to add to my previous list of

"man cut"
"man head"


I think a theme is developing here.




what does java give you? (or jvm type technologies)

It gives you a debugging entry point, at a level which is much deeper than inline print "$myvar" crap, and above that of the pages of syscalls verbiage which is underlying scripting implementations.

And you get all the associated JMX, Heap and thread dumping tools to use whatever the language ie Java, Scala, jruby, jython or groovy etc.


Whats the deal with hosting multiple SSL enabled websites?


So yet more re-purposed content from my serverfault.com procrastinations. This one was about how to stick a second SSL virtual host on apache, but ended up taking a brief look at the state of "Server Name Indication" technology which allows name based hosting with TLS, and also when wildcard SSL certs can be used...


Solution for Reverse Proxying onto URLs that are not amenable to being relocated to a subdirectory

I posted this question on serverfault.com, and it turned into "War & Peace" so I decided to duplicate it here to capture my on going battle with the problem.
I am also motivated to do this because stackexchange have implemented some annoying "community wiki" feature which declares posts to be "community owned" after a certain number of edits, which is most annoying because it steals any reputation points that you might get after that point.

My work-around to the community wiki problem is to maintain the question and answer content for editing elsewhere, so you reduce the number of individual "edits" that serverfault.com sees to below the cut off.

I am starting to believe the old adage that you don't know a system well, until you know something about it that really annoys you. ;-)

wtf is elasticsearch?


elasticsearch is the back-end used in the centralized logging getting started tutorial on the logstash.net site.

So from the front page blurb.. "It is an Open Source (Apache 2), Distributed, RESTful, Search Engine built on top of Apache Lucene. "

Basically you chuck JSON data into elasticsearch, and use lucene queries, or some JSON dsl to request data back. Its all RESTful, so you can look at the stuff in a browser;
http://localhost:9200

or you can use wget or curl, as you prefer...
curl -XGET http://localhost:9200/twitter/tweet/2
 
 
I was a little perplexed by all the multicast fuckery that I was getting when trying to use the non-embedded version of elasticsearch shipped with logstash, but now I have read the docs, I can see why the clustering makes sense.

# ElasticSearch, by default, binds itself to the 0.0.0.0 address, and listens
# on port [9200-9300] for HTTP traffic and on port [9300-9400] for node-to-node
# communication. (the range means that if the port is busy, it will automatically
# try the next port).


Persistence and data
elasticsearch uses the notion of a gateway.
http://www.elasticsearch.org/guide/reference/setup/dir-layout.html
By default elastic search has persistence of data and indexes (??) under the elasticsearch/data/ dir of the unpacked logstash


elastic search uses 9200+ for httpd and 9300+ for rcp..?




rabbitmq-server manager
http://opencirrus-g0803.hpl.hp.com:55672/#/




over riding the JAVA_OPTS
http://www.elasticsearch.org/guide/reference/setup/installation.html



elasticsearch front ends
https://github.com/mobz/elasticsearch-head
This is super cool, whether it is any us, is another question. The install was super easy- elasticsearch/bin/plugin -install mobz/elasticsearch-head

https://github.com/lukas-vlcek/bigdesk


java service wrapper
http://wrapper.tanukisoftware.com/doc/english/download.jsp


service wrapper on github
https://github.com/elasticsearch/elasticsearch-servicewrapper


elasticsearch rpm spec files
https://github.com/tavisto/elasticsearch-rpms

elastic search chef cookbook
http://community.opscode.com/cookbooks/elasticsearch






centralized logging with logstash

Last week I migrated some services off servers in a rack that was being decommissioned.  There was a distinct lack of system documentation, either up to date or otherwise, so I thought configuring nagios, monit, and munin would be a good start to checking the performance and reliability of the service on the new instance.

It had been suggested that the development team have a log monitoring system based around the log4j library, however it turns out that this system is mostly reactive, and there were a few undocumented configurations that didn't make the migration, and caused some problems. So after some hasty "diff -r" and rsync everything seemed to be well.

But I decided that this weekend I am putting together a central logging management system using logstash and chef, that I can deploy zero config style to hosts via a cookbook recipe, to catch 404 and 503 errors and other alerts coming from remote systems in some timely manner.


refreshing a nagios configuration

I've got a nagios box that I setup years ago, and now that there are going to be some substantial platform changes I need to revisit the configuration. Unfortunately what I see there is not a sensible hierarchy of monitors and alerts, but a morass of services, hostgroups, servicegroups and other obscure nagios-language directives.

I haven't got a good reason to try and swap another some other network monitoring tool, so it looks like I am going to have to spend a few hours re-grokking the documentation for nagios (which has upped a version to 3.2.x since I last looked)

Here goes...

Howto: backup to hpcloud.com object store using duplicity

Warning: this is only a demonstration, and the hpcloud is only a beta trial! dont rely on this to backup your important data. However...

I managed to get duplicity working with the hpcloud.com object store without any changes, just some close reading of the hpcloud documentation on legacy authentication support.

"If we want everything to stay as it is, everything will have to change."



The take-away thought from John Mauldin...

http://www.johnmauldin.com/frontlinethoughts/staring-into-the-abyss 


 (original quote)
"If we want everything to stay as it is, everything will have to change."
– from The Leopard by Giuseppe Tomasi di Lamedusa

Religious belief similar to gang colours?


Wearing Gang colours, what does it get you? Shot at? stabbed?

It's certainly not going to get you a job, or fair treatment by the law. It might be worth some street-cred, or engender some respect or access to certain people, but gang members are hardly in the position to provide the sort of business networking opportunities of the masons, unless you are in the crack business.

using tcpwrappers with dovecot on fedora (aka denyhosts for dovecot)

(draft)

I've been planning to move my mail server off a cheap-ole-vps server for quite a while now. Recently I realised that they implemented a 300 thread limit for the vps, and this is killing my glassfish server occasionally, which is seriously heavy weight in its use of threads and dovecot login processes.

However due to laziness, and the fact there there is a big chunk of data in the form of Maildir and /var/lib/mysql is proving a blocker. Maybe a project for the downtime between xmas and new years.

Another limitation of the vps is a lack of access to the underlying iptables kernel module configuration which rules out fail2ban style systems for blocking brute force attacks against ssh.

Go Live Checklist for Web Applications...

I was trying to think of a memorable summation of the go-live check lists that I have variously applied over my various releases. One of my much more competent than I ex-colleagues sent a quite comprehensive list, which I tried to break down into these categories;

  1. Does it work?
  2. Is is reliable?
  3. Is it secure?
  4. Are we legal?

performance dominance... (longest span between world championship victories)


I got into the debate about who were the greatest experts in their area of performance ever this afternoon and I thought I would look up some statistics on unusual records.

The one I am looking for in this post is the longest span of time between world championship victories in any sport or suitable challenging endeavour.

(I am limiting these searched to the modern era, or at least to where there was a well developed championship programme in place)

Fuck the flu

So on Monday I was wandering about in the house making a cup of tea and pottering about, and I got a funny shiver... and I thought that's a bit odd, I hope I'm not coming down with something...

12 hours later, I am curled up in bed, pouring with sweat, shivering cold and running a massive fever. The invaders were not just at the gate, no, the castle walls were down and the keep was being ransacked. ;-) Headache, bone-ache, cough, sore-throat, delirium. etc. etc

Why didn't someone mention frameOS.org before?


for fuck sakes. I probably wasted 3 days last week messing about with various gems, rubies, yum etc.

I just found the frameos repo and all the problems went away...

http://www.frameos.org/FrameOS/Home.html

leaving yum for gems, better late than never.

I've made this particular mistake a couple of times before, which was trying to use the software from my packaging system of choice, rather than the one favoured by the developers.
I love a bit of yum, and eclipse and many of its components are packaged as rpms in the fedora repositories - so I made the mistake of trying to use that version, I kept running into bugs that were fixed in the eclipse update sites version years before, even though fedora releases are every 6 months.

newsflash: Chef much better than Puppet!


After spending a few weeks dealing with Puppet fuckwittery, I decided to deploy some instances into the HP cloud beta using Chef rather than Puppet.

It's like stepping out into the light.