FBSimCity v0.4.0: the backup yard — gbak pins the OIT, nbackup fills the delta

FBSimCity, the explorable isometric city of Firebird internals, is at v0.4.0. This release adds a whole backup yard, built around what gbak and nbackup actually do.

gbak: the backup that pins your OIT

gbak takes a logical backup online: it attaches like any other client and reads every table through a snapshot transaction. That snapshot is the interesting part, because it pins the OIT for the entire run. Garbage collection stalls, cooperative GC refuses to demolish anything, and the record version towers climb until the backup finishes.

This is why a nightly gbak against a busy database and a mysteriously bloating database are so often the same story. Now you can watch it happen instead of inferring it from gstat -h: run the nightly gbak scenario.

nbackup and the difference file

nbackup is the other half: a physical backup, incremental by level. Level 0 copies the whole file, level 1 only the pages changed since level 0, and so on. The chain is enforced in the model just as it is in reality: ask for a level 1 without a level 0 and it refuses, and Restore chain reports which levels a restore would have to apply, in order. Lose level 0 and the rest are waste paper.

Locking the database with nbackup -L freezes the main file so it can be copied safely while the server keeps running. Every page written from that moment lands in the difference file instead, a new orange pit beside the main excavation that fills up visibly and merges back on unlock. Forget to unlock and it grows for as long as you watch: see a locked database filling its delta.

Dirty pages stopped being free

This release also fixes a genuine falsehood in the simulation. Evicting a dirty buffer used to cost nothing, which quietly understated write pressure. It now writes the page out first, so a reader that needs a frame pays for somebody else’s write.

The interesting part is what that does not cause. Because commits flush their page under forced writes, which is Firebird’s default, dirty evictions stay rare on a healthy database at around 1% of evictions, and only start biting when the cache is too small for the working set, reaching roughly 5% at 16 buffers. The honest lesson is “your cache is undersized”, not “writes are bad”, and the new evictions (dirty N) readout shows exactly that.

A knob audit

Since the whole point is intuition rather than emulation, v0.4.0 documents itself. docs/KNOBS.md lists every control and readout, what it does to the model, and whether the mechanism is real, merely scaled, or a plausible modeled stand-in, followed by the deliberate simplifications. Sweep here is time-triggered rather than transaction-gap-triggered; lock contention is a probability rather than a wait-for graph; no SQL is parsed at all. It is all in the table, so nobody has to discover it by reading the source.

Also in this release

  • Subsystem controls now live on the subsystem: start a sweep from the GC depot, run backup levels or lock the database from the nbackup vault, forget to commit a transaction from the Transaction Hall.
  • The screenshot driver no longer leaks browser profiles, and form controls are 16px so iOS Safari stops zooming the page.

City: mariuz.github.io/FBSimCity
Release notes: v0.4.0
Source: github.com/mariuz/FBSimCity (MIT, plain HTML and JavaScript, no build step)

Corrections are very welcome, especially on the backup mechanics, which I modeled from the documentation rather than from the engine source.

FBSimCity is an independent educational project, not affiliated with or endorsed by the Firebird Project. Firebird® is a registered trademark of the Firebird Foundation Incorporated.

Curso: Backups no Firebird (PT-BR)

This is only for Portuguese speakers.

Está cansado de lidar com a constante preocupação de perder informações importantes do seu banco de dados Firebird? Quer ter a certeza de que seus dados estão protegidos e sempre disponíveis, mesmo em situações de emergência? Então você precisa conhecer esse curso especializado em backups de banco de dados Firebird!

Você aprenderá técnicas avançadas e muitas vezes desconhecidas para fazer backups eficientes e confiáveis. Descubra também como fazer backup/restore em um único passo, sem precisar criar arquivos intermediários, economizando tempo, espaço em disco e simplificando o processo.

Saiba também como usar o gbak gerando como resultado final um backup já compactado com o 7zip, otimizando espaço em disco! Você também aprenderá sobre backups incrementais, mais rápidos e que armazenam apenas dos dados que foram adicionados ou alterados desde o último nível de backup, deixando o processo mais eficiente e possibilitando a realização de backups mais frequentes.

Apenas durante o lançamento, até o dia 24/fevereiro, o curso estará sendo vendido de R$ 397 por R$ 250 através do link tinyurl.com/uyuupcez. Todos compradores receberão também gratuitamente meu eBook Guia de Migração para o Firebird 3!

Não perca mais tempo e garanta agora seu acesso a esse curso exclusivo! Proteja seus dados e tenha a certeza de que eles estão sempre seguros. Torne-se um expert em backups de banco de dados Firebird!

A venda e a entrega estão sendo feitas via plataforma Eduzz, e você pode pagar com cartão, pix ou boleto bancário.

Firebird SQL’s backup/restore uses magic words for stdin and stdout – Check the gbak manual update

After An interesting read.

Norman Dunbar updated the gbak manual with the following:

A section on the use of stdin and stdout “file names” and how to clone
a database – on the same server – using a pipe and these “file names” to
avoid the use of a temporary intermediate dump file.

A section on carrying out remote backups and restores using the above
as well as an ssh session. This allows a remote database to be cloned to
a local server, a local database to be cloned to a remote server and a
remote database to be cloned to (another) remote server.

Further updates to document the use of the stdin and stdout file names in backups and restores. A section has been added to Gbak Caveats giving more in depth detail about these two special file names.

Change made to make Faster restores with gbak

There is a simple change which makes restores much faster, tested with Linux 2.5 embedded restore
Two databases, with two different tables filled with 2,000,000 records:
create table t1 (n1 integer);
create table t10 (n1 integer, n2 integer, n3 integer, …, n10 integer);

----------------------
Restore timings:
T1: 14.5s
T10: 20.5s

Improved gbak timings:
T1: 9.7s
T10: 14.7s
----------------------

The good news is there is still room to improve the restore speeds , here is Adriano’s tweet:

I do have a slight different method, to insert 100K records in less than 1s over the same localhost method.

Also confirmed by the ending of the article:

PS: There is other restore improvement capable of take down time from
around 8.5s to 0.5s over TCP, but it still requires some analisys.

How to increase Firebird backup and restore speed (up to 30%)

It seems that many Firebird developers and administrators are not aware about  command line switch -se[rvice] for gbak.exe

-SE[RVICE] <servicename>

This switch causes gbak to backup a remote database via the service manager. This causes the backup file to be created on the remote server, so the path format and filename must be valid on the remote server. The servicename is currently always the text service_mgr.

here is the rest of the article in Eglish

also there is an Italian mention in blogosphere