Good Defaults for Technical Decisions

In my experience as a software engineer I've found a few "rules of thumb" for technical decisions. None of these are hard requirements or things that can never be false. However these are good guidelines if you don't have a reason to make a different decision. Unlike most engineering decisions which first present the constraints and then try and find a solution within them, this attempts to document decisions one should make if you didn't have any constraints in the first place.
Its possible you'll disagree with me on some of these and I'd like to understand why. That said, I'm not interested in specific projects where these are a bad idea but for an understanding about why these shouldn't be the default.

1. Be explicit about your requirements. Don't automatically detect features, dependencies, or environment related issues. It is easier to change this later to make things more "magical" than go back and figure out what you need.
2. Namespaces are good: even if you think you'll only ever need one. Its easier to modify in the future, versioning, etc.
3. Errors ought to error. Warning ought to error or not exist. It is generally unhelpful to have noise in your output that you do nothing with. If a warning isn't meaningful, disable it.
4. Keep scope local and private. Prefer hiding functions and information from the outside unless you have to decided to make this an API.
5. Naming your first version as a "v1" and label it as such. During rewrites, migrations, or related issues prefer versions rather than names such as "next" or "new". There will likely be many "nexts" and only one v2.
6. Structured is better than unstructured. Similar to the point about explicitness: it is easier to go from structured to unstructured than vice versa.
7. Fixed is better than editable. Don't let people change things unless there is a reason to. This also applies to code (immutable is better than mutable).
8. Don't rely on people not making mistakes. Even if you have perfect people, they might be tired, have something in their eye, misremember a fact, or otherwise be operating at a sub-optimal state.
9. Name same things the same and different things differently. Use, and accept, the same formats for the same thing at all layers of the stack. As a counter example ruby outputs missing gems as name-version but gem(1) expects name:version.

This is a work in progress document and I'll try and update it over time

Don't Use Timing Functions for Profiling

One common technique for profiling programs is to use the gettimeofday system call (with code that looks something like this):

Example (incorrect) code that uses gettimeofday - click to view

#include <time.h>
#include <stdlib.h>
#include <stdio.h>
void function(void)
{
  struct timeval before;
  struct timeval after;
  gettimeofday(&before, NULL);
  codetoprofile();
  gettimeofday(&after, NULL); 
  time_t delta = after.tv_sec - before.tv_sec;
  printf("%ld\n",delta);
}

However, using gettimeofday(2) or time(3) or any function designed to get a time of day to obtain profiling information is wrong for many reasons:

1. Time can go backwards. In a virtualized environment this can happen quite often. In non-virtualized environments this can happen due to time zones. Even passing CLOCK_MONOTONIC to clock(3) doesn't help as it can go backwards during a leap second expansion.
2. Time can change drastically for no reason. Systems with NTP enabled periodically sync their time with a time source. This can cause the system time to change by minutes, hours, or even days!
3. These functions measure Wall Clock time. Time spent on entirely unrelated processes is going to be included in the profiling data!
4. Even if you have disabled everything else on the system^[1] the delta computed above includes both of User time and System Time. If your algorithm is very fast but the kernel has a slow implementation of some system call you won't learn much.
5. gettimeofday relies on the cpu clock which may differ across cores resulting in time skew.

So what should be used instead?

There isn't a good, portable, function to obtain profiling information. However there are options for those not tied to a particular system (or those willing to maintain multiple implementations for different systems.

The getrusage(2) system call is one option for profiling data. This provides different fields for user time (ru_utime) and system time (ru_stime) at a relatively high level of precision and accuracy.

Using DTraces profiling provider also seems to be a decent choice although I limited experience with it.

Finally, using APIs meant to access hardware specific features such as FreeBSD's hwpmc is likely to provide the best results at the cost of being the least portable. Linux has similar features such as oprofile and perf. Using dedicated profilers such as Intel's vtunes^[2] may also be worthwhile.

1. Including networking, background process swapping, cron, etc.
2. A FreeBSD version is available.

update 2012-11-26: Include note about clock skew across cores.
Update 2013-02-13: Update and fix a massive error I had w.r.t. clock(3)

Reduced Entropy in rand() and random()

TL;DR: Don't rely on undefined behavior, even when you think it should work.

---

I recently reported a minor issue to the FreeBSD security team.

The libc random functions had code^1,2 designed to run when /dev/random is not available. This can easily occur in a chroot or jail environment.

if (!done) {
        struct timeval tv;
        unsigned long junk;

        gettimeofday(&tv, NULL);
        srandom((getpid() << 16) ^ tv.tv_sec ^ tv.tv_usec ^ junk);
        return;
}

This code is designed provide a minimal amount of entropy in the "failure" case. Unfortunately, it doesn't even provide the entropy it claims to. This is a minor issue because getpid, getimeday, and a single long variable don't provide a lot of entropy in the first place: (only $log_2{sizeof(long)}$ bits).

The point of the junk value is to add entropy by using uninitialized memory. This relies on the compiler being "stupid" enough not optimize it away.

Unfortunately clang and newer versions of gcc are smart enough to use the undefined behavior in undesired ways.

clang ³ removes any computation which relies on the undefined behavior and so produces the following object code:

 af0:   e8 5f fc ff ff          callq  754 <gettimeofday@plt>
 af5:   e8 7a fc ff ff          callq  774 <getpid@plt>
 afa:   e8 65 fc ff ff          callq  764 <srandom@plt>

Note that the junk variable is entirely unused and that the xor operation between gettimeofday and getpid is non-existent.

gcc 4.6 ⁴ outputs:

 ce8:   e8 03 fa ff ff          callq  6f0 <gettimeofday@plt>
 ced:   e8 4e fa ff ff          callq  740 <getpid@plt>
 cf2:   48 8b 7c 24 08          mov    0x8(%rsp),%rdi
 cf7:   48 33 3c 24             xor    (%rsp),%rdi
 cfb:   c1 e0 10                shl    $0x10,%eax
 cfe:   48 98                   cltq
 d00:   48 31 c7                xor    %rax,%rdi
 d03:   e8 28 fa ff ff          callq  730 <srandom@plt>

Note that in this case the junk value appears to be (%rsp) which isn't all that random.

gcc 4.2⁵ produces the following code

with the junk variable

 d9f:   e8 18 fa ff ff          callq  7bc <gettimeofday@plt>
 da4:   e8 43 fa ff ff          callq  7ec <getpid@plt>
 da9:   48 8b 3c 24             mov    (%rsp),%rdi
 dad:   48 33 7c 24 08          xor    0x8(%rsp),%rdi
 db2:   c1 e0 10                shl    $0x10,%eax
 db5:   48 98                   cltq
 db7:   48 31 c7                xor    %rax,%rdi
 dba:   48 31 df                xor    %rbx,%rdi
 dbd:   e8 1a fa ff ff          callq  <srandom@plt>

and without:

 d9f:   e8 18 fa ff ff          callq  7bc <gettimeofday@plt>
 da4:   e8 43 fa ff ff          callq  7ec <getpid@plt>
 da9:   48 8b 3c 24             mov    (%rsp),%rdi
 dad:   48 33 7c 24 08          xor    0x8(%rsp),%rdi
 db2:   c1 e0 10                shl    $0x10,%eax
 db5:   48 98                   cltq
 db7:   48 31 c7                xor    %rax,%rdi
 dba:   e8 1d fa ff ff          callq  7dc <srandom@plt>

The base version of gcc isn't vulnerable. However, with the upcoming switch to the clang compiler the FreeBSD does become vulnerable.

The first proposed fix was to add the volatile type qualifier to the junk variable. While this seemed to fix the code generation issue I didn't believe this to be a valid fix as the behavior is still undefined⁶ (I misread text of the standard). Additionally, the value is likely have be very predictable. A preference was expressed to remove the junk variable as using it may leak a small amount of stack data.

I proposed the simple and obvious fix of removing the use of the junk variable⁷

In a brief survey of other libraries I noticed similar issues. I will attempt to notify the vendors

It should be obvious, but undefined behavior is undefined and can't be relied on to ever to give a sensible result.

1. random.c r165903 (line 316)
2. rand.c r241046 (line 131)
3. FreeBSD clang version 3.1 (branches/release_31 156863) 20120523 compiled with -O2 or -O3
4. gcc46 (FreeBSD Ports Collection) 4.6.4 20120608 (prerelease) compiled with -O2
5. gcc (GCC) 4.2.1 20070831 patched [FreeBSD] compiled with -O3
6. sections 5.1.2.2.3 and 6.7.2.4 of ISO9899
7. svn commit r241373

Edit 2010-10-10: update the paragraph referring to undefined behavior of volatile.

Some git-svn notes

When working with a subversion repository I often miss the use git features. However, it is possible for git to speak the subversion protocol:

Cloning the initial repository

$git svn clone svn://svn.freebsd.org/base/head


Resuming an interrupted git svn clone

It is really annoying when you start a git svn clone process overnight and come back to find that it stopped in the middle. Luckily, there is a really simple way to recover - without spending hours to redownload what you already have.

$git svn fetch
$git svn rebase

Committing the final patch

There are a lot of workflows for this, but I prefer the cherry-pick approach:

$git checkout master
$git svn rebase
$git cherry-pick commit-id
$git svn dcommit


Repeating characters in multiple languages

A friend of mine asked me how to repeat a string a specified number of times. There are a few times when ones wants to do this when programing. Here is the "repeating operator" in various languages. I tried to use an operator when possible - but in certain cases I used a function. In all cases I repeat a string followed by a newline.

The BSDs

for i in $(jot 1 5);
do echo -n "Hi";
done;
echo "";
Output:
HiHiHiHiHi

Most Linux distributions

for i in $(seq 1 1 5);
do echo -n "Hi";
done;
echo "";
Output:
HiHiHiHiHi

Perl

print "-" x 10;
print "\n"
Output:
----------

Python

"ab" * 10 Output: 'abababababababababab'

R

paste(rep("Hi",5), collapse='')
Output:
[1] HiHiHiHiHi

Ruby

print "-" * 10;
print "\n"
Output:
----------

Tcl

string repeat "Hi" 5
Output:
HiHiHiHiHi

ZSH

repeat 5 printf 'abc';
echo "";
Output:
abcabcabcabcabc
update 5/30/11: Thanks to Hans I found out that jot is not POSIX. Also fixed formatting.

What language should you learn first?

Kaushal Shriyan on the beginners@perl.org mailing list asked:

"Is it better to learn Perl or Python since i can manage only writing simple bash shell scripts."^[1]

Recently on ##C++-basic there was a discussion that related to learning low level vs high level languages first.

Both of these discussions are based on the premise that there is a right answer.

There are a number of things wrong with this question.

1. Kaushal seems to be saying that he can only learn one language or the other. Why not learn both?
2. Programming language choice depends on the project one plans on working on. Using perl to write a kernel is impossible and using COBOL to write a GUI application is silly. Asking such a question requires more context.
3. For some reason only Perl and Python are the only languages given as choices. What about C++? Why not Haskell? There are many useful languages and they each have their own pros and cons.

More generally, there are many skills have a programmer needs to have and there are different ways of learning each of these skills.

• Code organization and Design Patterns
• Programming constructs
• Abstraction and encapsulation
• Data structures and how to choose between them
• Syscalls and context switching
• and many more

All of these topics can be reasonably broken down into two types: programming topics and computer science topics.

Higher-level languages allow programmers to abstract away the low level details and focus on getting a task done. This is great for completing a task and making a beginner feel proud about what (s)he has created. It also helps the programmer learn programming basics such as variables, loops, arrays, input/output, and functions without getting bogged down with memory management.

On the other hand, choosing between various data structures is confusing unless one understands why things work the way they do. Learning about context switches, data structures, syscalls, pointers, etc is easier in a low level language (like C or C++).
 
So what language should someone learn first?

A good programmer should learn both types of languages. In my experience a lot beginners have trouble wrapping their heads around functions and other organizational patterns such as classes. I usually start people with Python and move quickly to C++. This helps to teach good programming style and techniques before they get to learn the low level concepts as well. 

[1] http://comments.gmane.org/gmane.comp.python.tutor/65413