by Valentin, July 1 2015, in tech

Why Firefox was losing track of my history

A few months ago I encrypted my Firefox profile. It contains my history, bookmarks and most importantly, authentification cookies that I keep for websites I find more conveniant to stay logged in. Those are things I prefer to be encrypted for obvious privacy and security reasons.

The process of encrypting a Firefox profile goes as follow: first, you need to know where the profile is located in the disk. On Windows, it's a directory which sits at {User direcory}\AppData\Roaming\Mozilla\Firefox\Profiles\{the profile directory}. Then you need to use an encryption software such as TrueCrypt to put this directory in an encrypted container. Once you have this container, this is how a software like TrueCrypt works: when you open a container with it, it'll query you with the password and if you enter the correct one it'll mount the container as if it was an external storage device. On Windows, this makes a new disk appear (you know like C: or D: or whatever) and all the content of the container becomes available at this disk. TrueCrypt lets you choose the letter of the disk on which you want to mount the container.

In this case, the content of the container is a Firefox profile, so you have to tell Firefox how to access it once it's mounted. This can be done using the Firefox profile manager. In fact Firefox let's you create an arbitrary number of profiles. A profile has a name, and a path indicating where the actual profile directory is on the computer. Here, the strategy is to choose a letter by convention, which will be the letter of the drive onto which we're always going to mount the Firefox profile container. For example, we can take R:\ (I prefer letters at the end of the alphabets because Windows uses letters at the begining at the alphabet to automatically mount external devices). So we're gonna create a Firefox profile and set its path to R:\

When all this set up is ready, when you want to browse Firefox, first you mount the profile container to R:\ and then you open Firefox with the corresponding profile. When you finish browsing, you close Firefox and you dismount the container. That is a dull process, so you can, like me, write some little batch script that'll make it almost transparent: it just asks me the password when I click on the Firefox icon and does the job of mounting before opening Firefox and dismounting when I close Firefox.

This was not intented to be a tutorial on how to encrypt a Firefox profile, but just an introduction to the rest of this article. This is a step-by-step guide on how to encrypt a Firefox profile.

When you create a container, you need to indicate the size of the container. Before encryption, my profile weighted around 230 MB, so I figured that 500 MB was enough for the container. Well, in fact, Firefox had a little surprise for me.

A few weeks after I had set up my encrypted profile, I noticed that Firefox was losing old history. I never delete my history. It goes (went) all the way to more than six months, actually all the way to when I started my computer for the first time. I was first alerted by auto-completion fails. Having a massive history lets Firefox performs some very helpful and comprehensive auto-completion magic in the address bar; but this time it wasn't even able to auto-complete "". What the hell was going on. Then I discovered in the Library window that there was nothing more than "This month". I was mad. My data had disappeared.

After some googling I found out why.

It turns out that by default, Firefox delete old history based on some expiration criterias. This is the source code of the program doing this slanderous job.

Look at this:

// This percentage of disk size is used to protect against calculating a too
// large database size on disks with tiny quota or available space.

If there is not much space available on the disk, the history database can't take up more than 2% of this available space. My container has a size of 500 MB, but about 230 MB are already taken by other stuff (bookmarks, thumbnails, extensions, etc), so there's 270 MB actually available. So, the maximum size of the database was 5.4 MB. That's not much.

Then there's this:

// This is the average size in bytes of an URI entry in the database.
// Magic numbers are determined through analysis of the distribution of a ratio
// between number of unique URIs and database size among our users.
// Based on these values we evaluate how many unique URIs we can handle before
// starting expiring some.
const URIENTRY_AVG_SIZE = 1600;

Firefox computes the maximum number of URIs it can hold in the database thanks to an average URI size, whose value is set to 1.6 KB. With 5.4 MB available, it means that you can store up to 3375 URIs. I don't know about you, but for me, that's about 15 days of browsing the web.

So, this is why my history was only available for 2 weeks.

The solution to this problem was quite easy. Actually all this hardware calculation for the maximum number of URIs is used only if the maximum size isn't set by the user. There's no regular settings menu to set this preference, I had to go in about:config. According to this, there should be 2 keys relative to the limit of URIs: places.history.expiration.max_pages which can be set by the user to his convenience and places.history.expiration.transient_current_max_pages, a read-only copy of the previous key destined to other components which would want to access this information. When I finally searched for the keys, only the read-only one was there (and its value was 3274, close but not equal to my estimation above, that's because I rounded the numbers). According to this forum post, I simply could create a brand new key called places.history.expiration.max_pages and do my business with it. Which I did, setting its value to one fucking million. I was glad to see that its twin key immediatly updated itself with the new value, indicating that it worked.