Mercurial > hg > Blog

annotate content/Linux/portage-metadata-cache.md @ 98:1d9382b0329b

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Specify the syntax on markdown blocks to avoid broken output that has class=err
author Dirk Olmes <dirk@xanthippe.ping.de>
date Thu, 19 Dec 2019 10:04:33 +0100
parents 0f9fffa05db1
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
58
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
1 Title: speeding up portage's metadata cache
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
2 Date: 2015-02-26
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
3 Lang: en
83
0f9fffa05db1 added the Gentoo tag
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 58
diff changeset
4 Tags: Gentoo
58
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
5
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
6 [Gentoo's](http://www.gentoo.org) portage keeps metadata about installed ebuilds in `/var/cache/edb`. Dependency info for all installed ebuilds is in a `dep` subdirectory which typically looks something like this:
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
7
98
1d9382b0329b Specify the syntax on markdown blocks to avoid broken output that has class=err
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 83
diff changeset
8 :::shell
58
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
9 .
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
10 ├── usr
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
11 │   └── portage
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
12 │   ├── app-admin
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
13 │   │   ├── eselect-1.4.1
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
14 │   │   ├── eselect-lib-bin-symlink-0.1.1
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
15 │   │   ├── eselect-opengl-1.2.7
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
16 │   │   ├── gamin-0.1.10-r1
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
17 │   │   ├── logrotate-3.8.7
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
18 │   │   └── perl-cleaner-2.16
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
19 ...
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
20
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
21 When portage needs to process dependency info, it reads those files. On a normal system, you will have a couple of hundred packages installed. This means that dependency processing does quite a bit of file processing.
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
22
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
23 Now if that dependency info was stored in some kind of database, wouldn't that speed up dependency processing?
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
24
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
25 I stumbled over [a wiki page](http://gentoo-en.vfose.ru/wiki/Portage_SQLite_Cache) describing how to switch portage's metadata cache to sqlite. Even the portage man page talks about this - try running `man portage` and read the section about the `modules` file.
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
26
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
27 So I gave the sqlite metadata cache a try to measure if it really speeds up portage. After configuring the database and rebuilding the metadata cache, `/var/cache/edb/dep` looks a bit different now:
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
28
98
1d9382b0329b Specify the syntax on markdown blocks to avoid broken output that has class=err
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 83
diff changeset
29 :::shell
58
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
30 .
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
31 └── usr
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
32    ├── portage
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
33    └── portage.sqlite
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
34
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
35 Now let's get to the interesting part: does the database really speed up portage? To measure, I ran a couple of `emerge -vp` commands using the normal setup and again using the database. The results are quite disappointing, though:
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
36
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
37 The best improvement was about 6% with the metadata database.
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
38
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
39 57% of the ebuilds did run slower with the metadata database, the worst increase was about 43%
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
40
0cd05745be92 add a blog entry about the portage metadata cache
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
41 So it looks like fiddling with portage's metadata cache is not really worth the hassle.