Mercurial > hg > Blog

annotate content/Linux/portage-metadata-cache.md @ 112:cf31bf5fce72 default tip

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Author of the blog post as mail header for efficient spam filtering
author Dirk Olmes <dirk.olmes@codedo.de>
date Tue, 06 Sep 2022 07:04:11 +0200
parents 1d9382b0329b
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
58
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
1 Title: speeding up portage's metadata cache
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
2 Date: 2015-02-26
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
3 Lang: en
83
0f9fffa05db1 added the Gentoo tag
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 58
diff changeset
4 Tags: Gentoo
58
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
5
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
6 [Gentoo's](http://www.gentoo.org) portage keeps metadata about installed ebuilds in `/var/cache/edb`. Dependency info for all installed ebuilds is in a `dep` subdirectory which typically looks something like this:
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
7
98
1d9382b0329b Specify the syntax on markdown blocks to avoid broken output that has class=err
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 83
diff changeset
8 :::shell
58
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
9 .
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
10 ├── usr
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
11 │   └── portage
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
12 │   ├── app-admin
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
13 │   │   ├── eselect-1.4.1
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
14 │   │   ├── eselect-lib-bin-symlink-0.1.1
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
15 │   │   ├── eselect-opengl-1.2.7
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
16 │   │   ├── gamin-0.1.10-r1
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
17 │   │   ├── logrotate-3.8.7
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
18 │   │   └── perl-cleaner-2.16
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
19 ...
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
20
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
21 When portage needs to process dependency info, it reads those files. On a normal system, you will have a couple of hundred packages installed. This means that dependency processing does quite a bit of file processing.
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
22
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
23 Now if that dependency info was stored in some kind of database, wouldn't that speed up dependency processing?
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
24
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
25 I stumbled over [a wiki page](http://gentoo-en.vfose.ru/wiki/Portage_SQLite_Cache) describing how to switch portage's metadata cache to sqlite. Even the portage man page talks about this - try running `man portage` and read the section about the `modules` file.
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
26
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
27 So I gave the sqlite metadata cache a try to measure if it really speeds up portage. After configuring the database and rebuilding the metadata cache, `/var/cache/edb/dep` looks a bit different now:
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
28
98
1d9382b0329b Specify the syntax on markdown blocks to avoid broken output that has class=err
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 83
diff changeset
29 :::shell
58
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
30 .
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
31 └── usr
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
32    ├── portage
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
33    └── portage.sqlite
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
34
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
35 Now let's get to the interesting part: does the database really speed up portage? To measure, I ran a couple of `emerge -vp` commands using the normal setup and again using the database. The results are quite disappointing, though:
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
36
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
37 The best improvement was about 6% with the metadata database.
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
38
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
39 57% of the ebuilds did run slower with the metadata database, the worst increase was about 43%
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
40
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
41 So it looks like fiddling with portage's metadata cache is not really worth the hassle.