Maturing the organization

I’ve had a number of private conversations about my lightweight development process blog post. Most of these have revolved around what I termed “jargon, unusual diagrams or fancy methodologies”. In fact, some developers and architects took offense and felt I was dissing their favourite development tools, methodologies, and techniques. “How can you think to recommend not using UML? Use cases? <Insert favourite management tool/meth/tech here>?”

Let’s be clear: I wasn’t suggesting not using these things, I was advising keeping them out of customer-facing documents. The value of this advice depends of course on the process maturity of your development organization, your sales organization, and the customer’s procurement or acceptance organization.

The point about jargon, et al, is to keep specialized terminology or diagrams that are unfamiliar to a non-technical audience out of documents intended for that audience. For example, “actors” and “agents” have specific meanings in various contexts, and while developers and architects know which context is which, most customers are going to think Angelina Jolie and Swifty Lazar.

Educating your customers on your terminology may well be counter-productive. Depending on your organization, it may also annoy Sales.

The same thing applies to swim lanes, fish tails, entity-relationship diagrams, etc. They all have their place, absolutely. And in a relatively larger organization with relatively more mature processes, architects and designers would use these, and someone else (marketing? project management? product management?) would translate these to language and format suitable for customers.

(As an aside, which will become quite relevant momentarily, one role of these specialized tools and jargons is to convey meaning accurately and precisely to others who lack context, for reasons of time or space or both.)

My general inclination is that if you have a smaller organization and are just starting on implementing processes, it’s best to just work with the customer presentables – you don’t have the cycles or resources to do both, and you need the customer-facing stuff. The danger is that a lack of precision in your designs may lead to implementation errors.

Or worse, to customer acceptance problems. Hence the “checkpoint and repeat” development cycle (involving the customer if it makes sense). If your organization is relatively small, you gain the benefit of context and conversation: You are not capturing design for later use by people who may not even have been hired yet, but for immediate use by a team of people who are likely all living this project (and possibly nothing else).

One of the big tricks of management is figuring out which organization you are, noticing when you are changing from one to the other, and adding process maturity at the right time (read: Just early enough, and never too late).

That’s a big trick. If you can pull that off repeatedly, you have a solid future ahead of you.

BackupPC – The good, the bad, and the bizarre

Yes, I should have configured backups a long time ago. Mea culpa. At any rate, they are running now. But there were bumps along the way, so allow me to present the good, the bad, and the bizarre of getting BackupPC up and saving my data.

The good

Once configured, it works great. As I type, two backups  are humming right along. The interface for browsing backups is clear and intuitive. I’ll be testing a restore later and I expect it will work flawlessly.

The bad

Not much, really, though the bizarre things had me stuck so long I was thinking they were the bad.

The bizarre

More than a few. One or two in software, several in docware. I’ll cover only the worst ones that slowed me down as I set things up.

All of the documentation – and some of the code – seems to be written from the perspective of a mixed network (Linux and Windows) with considerable use of SMB. For example, if you indicate that a host is configured with DHCP, BackupPC defaults to querying it via nbmlookup.

I beg your pardon? All of our user machines are Linux and all are configured with DHCP. Why on Earth would nmblookup be expected to work? This utterly bizarre default leads to bizarrity number two.

The documentation – and various HowTo’s and intros, etc. – mention that when configuring BackupPC with knowledge of client hosts, one can indicate that hosts are configured with DHCP but that it isn’t really necessary to choose this, even they are, and that it might be better to not choose this.

Point of fact: If you are not using SMB at all, then, as far as I can tell, you MUST NOT indicate that hosts are configured with DHCP. As soon as you indicate DCHP, nmblookup is the default way of looking up hosts – none of the normal Linux/UNIX goodness seems to work. In my case, network history may have played a role, so permit a brief digression.

Our network began as a consumer home network and has evolved over time to reflect my deepening knowledge, my security requirements, and my underlying “if it ain’t broke don’t fix it” philosophy. One effect of this is that we don’t have local DNS – it’s on my list of ToDos, but we haven’t needed it yet. But! A lot of home users are likely in the same position.

What does this mean in practical terms?

You MUST populate /etc/hosts on the BackupPC server, you MUST use static address assignments in your DHCP server (in order for /etc/hosts to remain valid), and you MUST leave the “DHCP” box unchecked for each host.

Otherwise, BackupPC cannot find your clients.

Once BackupPC finds your clients, it still may not be able to connect to them. That is, unless you are a certified Linux guru with long experience with rsync and ssh.

Why? Because the documentation does not describe briefly and succinctly how BackupPC connects to a Linux client and how you have to configure that client. This was the “bizarre” that was almost a “bad”.

This is what the documentation – and the various HowTo’s, etc. – should say:

BackupPC needs to be able to connect to each Linux client automatically, without human intervention. The software defaults to attempting to do this securely using SSH and automatic certificate-based login. This is a good thing – it’s worthwhile doing the work to enable this sound default. This process has to be boot-strapped with a manual login and some copying of certificate files from the BackupPC server to each client machine. And these certificate files have to be created in the first place!

Step zero is to install BackupPC on the server, identify all of your client hosts, choose what data is to be backed up and where, etc. Once BackupPC is installed and seems ready to go, follow this procedure.

1. Become root on the BackupPC host. E.g., “% su”, or “% sudo su”, or whatever gets you there.
2. Become the BackupPC user: “$ su backuppc”
3. Create an RSA key pair for BackupPC: “% ssh-keygen -t rsa” (You may want to read up on SSH – “man ssh” – to learn how to create stronger key pairs or use other algorithms. You will also need to decide whether or not to use passphrases; “no” is more convenient, “yes” is more secure, but requires use of ssh-add and the SSH agent.)
4. Copy the public key to a convenient file, e.g., “% cp ~backuppc/.ssh/id_rsa.pub back.key”. Very important: The next few steps have to repeated on each client machine.
5. Make sure sshd – or equivalent – is installed on the client machine. For example, login to the client machine at its console, and, for Debian-based systems, including Ubuntu, enter this command: sudo apt-get install openssh-server
6. From the backuppc host, copy the file back.key to root’s home directory on the client machine, e.g., “% rcp back.key root@clientMachine:back.key”.   If this is the first time you connect from the backuppc host to the client machine, you will be asked to confirm the client machine’s key. You will be prompted for root’s password. Enter it, and the copy should happen very quickly.
7. Login to the client machine as root: “% ssh root@clientMachine”. You will be prompted for root’s password. Enter it, and proceed to the next and most crucial step.
8. Using the copied RSA certificate, enable the backuppc user to login automatically as root on clientMachine without entering a password. This is done by appending the RSA certificate to the list of authorized hosts: “$ cat back.key >> /.ssh/authorized_keys”. Logout of the client machine.
9. Login to the client machine as root: “% ssh root@clientMachine”.

If steps 5 through 8 worked as they should, you should login automatically, without being prompted for a password. (Refer to comments re passphrases and SSH agent, above.)

Now backuppc is ready to connect to start backups. To verify this, open a browser, connect to your backuppc server, e.g., http://backuppc/backuppc, choose one of your hosts, and click on “Start full backup”. Refresh the page and backuppc should tell you that the backup is running.

If the backup fails, check the logs for errors.

If you ever see the log entry “Got fatal error during xfer (Unable to read 4 bytes)”, then backuppc is not able to connect automatically as root. You need to verify steps 5 through 8 above to make sure that backuppc will not be prompted for root’s password and to make sure the client machine’s sshd certificate has not changed.

Anything else? Well, I had some trouble moving the backuppc home directory from its default of /var/lib/backuppc to /home/backuppc, but that might have been finger trouble. In a nutshell, here is the distillation of the steps I took – that is, this is what I figured out the procedure to be, after poking about for while. This is abbreviated, if you don’t understand what a step is doing, don’t do any of them! (I may update this later with background, but for now, well, caveat emptor.)

$ /etc/init.d/backuppc stop

$ mv /var/lib/backuppc /home

(depending on your OS and filesystem, you may want to use “cp -dpR” instead)

$ usermod –home /home/backuppc backuppc

$ vi /etc/backuppc/config.pl – around about line 315, “change $Conf{TopDir} = ”;” to

“$Conf{TopDir} = ‘/home/backuppc’;” and “$Conf{LogDir} = ”;” to “$Conf{LogDir} = ‘/home/backuppc/log’;”

$ /etc/init.d/backuppc start

I don’t like making the suggestion of editing config.pl, but try as I might, I was unable to move the data directory without doing this. The problem was the LogDir setting: I could override TopDir successfully using the web interface, but not LogDir. Whenever I tried starting BackupPC, I would get errors about not being able to create /var/lib/backuppc/log and the startup script would exit.

Note also that I haven’t gone back and validated this procedure with a clean install. My install works, it ain’t broke, I ain’t touchin’ it unless I need to!

Final bizarrity: If you see the message “because HOST has been on the network at least 7 consecutive times, it will not be backed up from” followed by a schedule, then BackupPC has decided that HOST is always on (possibly a server) and that it backups of this machine should default to “outside of business hours”. The stated intent is to reduce network load during business hours. I’ve disabled this – more on this below – for my own machines and am not sure how I feel about it in general.

For example, if a machine is a server, business hours are when files are changing the most and performing a backup during periods of high change may cause backup inconsistencies. On the other hand, periods of high change are exactly the time when users might need frequent backups all the more. And of course running  a backup on a loaded server may degrade service!

The best approach is likely to take backups when the server is quiet(er) and to ensure that users make use of a revisioning system, preferably one backed by a SAN or a RAID array, etc. This addresses the problems of backup inconsistency and PEBCAD/OOPS during periods of frequent change.

My choice was to disable this blackout period by default (“Edit Config”, “Schedule”, set BlackoutGoodCnt to -1) and to enable for the one machine to which it really applies (a web server; select the host, “Edit Config” at the top of the list, “Schedule”, click override and set BlackoutGoodCnt to 7).

What kind of machine is this, anyway?

As for the server assumption, that’s a little bizzare when you consider that BackupPC is written by Linux geeks and the primary audience is likely Linux geeks (either Geeks@Home, like me, or Geeks@Work who’ve been directed by the PHB to do something about backups). Why bizarre?

Because many geeks are also gamers, many gamers need a Windows box, and many Linux geek gamers use dual-boot to solve this problem. BackupPC was able to ping one of my machines 7 consecutive times because it was being used for gaming (a long day’s gaming :->). It assumed this meant the machine was “always on” and therefore decided it would be backed up only evenings and weekends.

The logic is both understandable and bizarre. Slightly more understandable and less bizarre would have been “machine was pinged AND backups were possible” 7 consecutive times. Backups were NOT possible for this machine, because it is configured for rsync over SSH, which is available only when it runs Linux; when it runs Windows, I’d just as soon it not be on the net at all….

But even with that AND, the decision to back up outside of business hours seems a little heavy handed. After all, when one configures backups, one’s desire is to back up data. IMHO, backing it up as available should be the default, exceptions for schedules or host classes should be just that, exceptions. Decisions about exceptions should be made by humans, not software.

Some design suggestions

• A better alternative would be to implement machine classes, that is, the ability to tag machines as “Linux desktop”, “Linux laptop”, “Linux server”, “Windows desktop”, “LAMP server”, etc., and to then apply intelligent defaults.

For example, default LAMP server to rsync over SSH backups evenings and weekends. Default Windows desktop to nmblookup and smbclient. Etc.

• Flag exceptions or take note of what might exceptional circumstances and present the administrator with choices, instead of implementing an exceptional behaviour, such as the blackout. “This machine appears to always on – would you like to schedule its backups for evenings and weekends only?”

• Provide a simple tool for moving the BackupPC pool or home directory. These are exceptional activities: Exceptional activities require the best documentation and the best tools, because they are the things we do the least, and often the things we most need to get right. That’s why dialing emergency services is so much easier than dialing your friend in across the country, despite the fact that you do the latter much more often the former.

• EITHER rename the “DHCP” column to “SMB” or something else that better reflects what the software does OR change the default behaviour. This requires a bit of thought, but it might be as simple as a little wizard. For example, the administrator enters a hostname and BackupPC attempts a ping, resolving first with gethostbyname then by nmblookup. If the ping works, BackupPC records the host with intelligent defaults, e.g., rsync for gethostbyname, smbclient for nmblookup. If the ping fails, then BackupPC asks a few questions to resolve the situation (“should I be able to connect now”, “how should I resolve the name”, “do you want to enter the IP and have me add this to /etc/hosts”, etc.)

• Display logs most-recent-first. Usually I want to see the most recent item first, so save me the trouble, small as it is, of scrolling to the end….

Lightweight development – good coding practice is good security practice

A recent slashdot discussion concerned a developer who needed to implement some form of management controls for a new project in an organization that had until then not used any software project techniques. Many of the responses had to do with “learning ITIL” or “reading this book”. Well-intended and valuable advice to be sure, but not necessarily applicable. If you are told “implement project controls”, chances are the boss meant “yesterday”. ITIL and the many valuable project and development management books can come later. What do you do when you need it now?

I use to run a number of development teams in a systems integration and custom development shop: We took our employer’s base products and toolkits and integrated them into customer environments. We did a lot of “1.0′s”; typical projects were 2 to 6 weeks in length and if we ever saw them again, we lost money. We could afford one or two moderate bugs (sev 3 – functionality impaired); more than that, we lost money. We could not afford major bugs (sev 1 – all is borked; sev 2 – most is borked). Given the tight timelines, we had to be very sure that what we were developing was what the customer asked for and that what the customer asked for was what the customer wanted.

We almost always made money and our customers were almost always very satisfied. We very rarely lost money, and it was usually on strategic projects (spend integration money to make more license money).

Here’s what we did:

1. Write a high level design document describing the major components and data flows – a system architecture, with enough drill-down for use cases that show who can do what. A mix of diagrams and text. Nothing too technical, because the customer has to understand it. But it has to be enough for a senior developer to either start coding (2 week project) or write an internal-use mid-level design doc (6 week project).

(Since the design has to be fit for the customer, it must not contain jargon, unusual diagrams or fancy methodologies that the customer doesn’t understand. These things have their place, to be sure. But if you cannot describe it in pictures and words, it may be too complicated for you and your organization’s current level of development methodology.)

2. Developer, tester, and writer estimate how long to do their bits based on high-level design. Project management adds some buffer (10% to 50% based on complexity).
3. Customer reviews the high-level design, expected ship date and cost, signs off.
4. Based on the high-level design document, start three simultaneous streams:
   1. Development: Either start coding or write that mid-level design document.
   2. Test: Write the test plan. Not the test cases. Start with the acceptance test plan.
   3. Documentation: Start putting together the major structure of the documentation. (ToC, section headings, text where necessary, etc.)
5. Checkpoint: The developer, tester, and writer meet to ensure that they agree that what they are each working on aligns with the others and with the high-level design. This can be a 30 minute meeting or a three hour meeting, depending on scope, etc. Most important things:
   1. Do we align with the design?
   2. Will we ship on time?
6. Customer signs off the acceptance test plan.
7. Add detail. The developer codes, the tester writes test cases or test scripts, the writer writes documentation.
8. Checkpoint: The developer, tester, and writer meet to ensure alignment.
   1. Do we align with the design?
   2. Will we ship on time?
9. Repeat “add detail” and “checkpoint” steps as necessary. Stop adding detailing when done (e.g., often the writer will finish first, then the tester, then the dev – it’s nice when it goes this way, because the tester can review the docs and make sure test plans and docs really align).
10. Test.
11. Ship.
12. Profit.

Handling exceptions. If at any point things start to drift out of alignment, stop. Figure it out. If the problem was the high-level design, go back to the customer; you may need to reset expectations or eat some profit; get sales involved, they’re good at this. Otherwise, it’s an internal issue you have to identify and correct – careful management is required to stay on track and under budget

VIP: Acceptance test plan. Having the acceptance test plan signed off by the customer is crucial. If they sign it off and everyone codes to it and it aligns with the high-level design and the deliverable passes acceptance, then you are done.

One thing I’ve left out: Change requests. They are the bane of every project under development. You need to dig in your heels and manage them properly. Work collaboratively with the customer and developer to determine whether the change request:

• Can be integrated at no/little cost.
• Can be integrated at the cost of more time, money, and shipping delay.
• Should be punted to a 2.0 or another project.

Always assume the latter and try to involve your developers as little as possible, they are busy developing. Interruptions delay shipping.

In my not so humble opinion, you cannot make do with less than the above. If your management does not accept the need for this level of process, I strongly recommend seeking a change of management or employment.

If you do not have this minimal process – it really is minimal – then in all likelihood, your project will go over budget, you will never ship, and the customer – whoever they are – will never come back for more. This isn’t a slight, this is the reality of most software projects: Over budget and shelved, with 0% satisfaction.

Process, properly managed, is your friend.

Process, out of control, is your second worst nightmare.

No process at all will results in flashbacks, your worst nightmares and a need for therapy for years to come.

More or less. AFAIK. YMMV.

Voices in a vacuum – securing a Facebook account

Social networking presents a special challenge for the security professional. It’s very easy to say “I shall eschew this, it is unproven and likely insecure”. That’s fine, until

1. So much of your personal network is on-line that you are left out: Events are organized, photos are published, jokes are told, all on <insert the social networking site you love to hate here>.
2. Your customers want business advice: Should they or should they not allow employees access to social networking sites and under what circumstances?
3. Your kids want a Facebook account.

1 and 2 are easy: You stick to your principled guns on number one, and retire a friendless grumpy old man who knows he was right all along, dagnabit, now get offa my lawn. For number 2 you simply respond that it is not your area of expertise – an entirely legitimate answer, especially in a high-integrity field like security consulting – and that you know someone who is, perhaps a referral can be arranged?

Number 3 is the tough one. I know parents who say no. Don’t know how well that’s working for them. More importantly, don’t know how well it’s working for their kids.

When I was a teenager, outside of organized sports and activities like scouting, hours on the phone and hanging out at the mall were the social networking activities. Parents complained about both and forbad one or the other or both. I’d argue that the kids who did one or both are – overall – better socially adjusted adults, better able to work in teams, etc. I don’t have any evidence I can cite, I’m just segueing nicely to my next point… …we’re social creatures. Social networking is an inherent part of our makeup. Quite likely literally in our DNA. Deny it and you deny a lot. It’s not insurmountable, but not being able to hang out with your peers can be tough.

So when your teenager says “OK, I’m 14 now, I’m in high school, can I have a Facebook account?”, what do you do?

Why, research, of course! And you learn some very interesting things. And like all good research, what you learn isn’t necessarily what you expected to learn.

Let’s deal with the technical elements first – the stuff I expected to learn, and did. First, kudos to my friend Scott Wright, the Streetwise Security Coach and blogger-in-chief at Security Views.

Scott pointed me at quite a good short paper on securing your Facebook account PDF WARNING available from Spylogic. I won’t repeat or summarize it here, it is brief and to the point. Read it.

Or at least read the first 3/4 of the paper – 5 Tips for Using any Social Network and Privacy Settings – these are the heart of the paper, clear, solid guidance on how to limit potential privacy leaks on Facebook.

Good stuff there, no question.

It’s the last bit of the paper I question, the Profile Information section.

This section makes 5 points:

1. Don’t use your real birthday.
2. Be cautious when posting your real phone number, address, or work information.
3. Watch the pictures you post.
4. Don’t post about your current employer.
5. Treat all photos and profile details as public information.

What’s wrong with these? Well, to be blunt, they’re a mix of dubious advice and advice in a vacuum.

Your birthdate and a lot of other details about you are tombstone information. (That is, data about you that are relatively readily available, e.g., from phone books, voter registration lists, etc.)

I know that some organizations use this information to verify your identity, especially over the phone, but they shouldn’t! Aaargh! Sorry to scream, but as a security professional this vexes me so! Any motivated attacker is going to be able to learn this about you. Easily. With little effort. Dear banking and insurance industries: This information has negligible identity validation value! Please get a grip and stop using it. In all likelihood, it will be cheaper to stop now by choice than later by force of court order (I cannot predict this, of course, but I do expect it: A class action lawsuit over known-to-be-insufficient security measures having led to identity theft,  account pillaging, and credit rating destruction).

I stand to be convinced otherwise, but given how easy I believe it is to obtain this information from other sources, I simply do not see a point in omitting it from a social networking site, especially if you’ve followed the good, clear advice in the paper and restricted profile access appropriately. The people who end up seeing this information are likely people you would want to know it – or, at the very least, you would be somewhere between OK and whatever if they found it out.

As for phone number and work information, well, I suppose my perspective is a bit unusual because I am a self-employed consultant. Independent, that is. I eat what I gather/kill. I want people to know how to get in touch with me! The increased visibility of social networking helps me find billable work – or at least that’s the theory. As for everyone else, well, it’s a social network, right? Refer to comments above re friends knowing your birthdate.

So I question the value of points 1 and 2. Again, I stand to be convinced otherwise, but it seems to me this information is readily available to people who would do harm so there is little point in hiding it – and may be counterproductive to hide it – from people who are supposedly your friends.

Supposedly your friends. That’s the problem with points 3, 4, and 5 in  a nutshell. Not that they may turn out to not be your friends, but that a lot of people – none of them likely to be reading this, sadly – just don’t get it, don’t get the “supposedly” part. “It’s all good”, they say, “we’re just networking, sharing”. With total abandon and a total lack of foresight.

And folks sans foresight (or sin foresight if you prefer your pretension in Spanish) are not likely to read this advice, let alone heed it. Hence advice in a vacuum – points 3, 4, and 5 will be as audible to these folks as screams in space.

As for the rest of us, we fall into two camps.

Most of us are in the first camp. We know better or think we know better than to post drunken photos of ourselves, or to slag our employer in a public forum. We get it. Most of the time – we everyone of us has sent that email we shouldn’t have, and too many of us have written that post we shouldn’t have – but most of the time we get it. Points 3 to 5 aren’t exactly wasted on us, but they’re not exactly news, either.

So what’s the harm with including them? Since neither the foresightless (foreblind?) nor thems of us Camp One are going to read it, it’s extra content that distracts from the main message. A simple recipe – do this to enhance the privacy of your facebook account – is more likely to be followed than an recipe accompanied by debatable content. If the aside can be questioned, the recipe is open to questioning. And questioning leads to non-compliance. That’s why we put subtlety in policy documents – which most people don’t read – and clear directives in procedures that are to be read and followed by everyone.

The other camp gets it all the time. They know. They understand what social networking sites are really all about. They understand public performance. That’s the stuff I didn’t expect to learn, the stuff that seems so obvious in retrospect. That’s the subject of part 2, which you can expect sometime next week. And yes, it does have a bearing on security.