`borg2 transfer --from-borg1` fails with multiple errors with very old repos

[PhrozenByte]

/kind bug

When running borg2 transfer --from-borg1 with an ancient (some archives were created with Borg 1.0, maybe even Borg 0.x, but I'm not entirely sure) and "beaten" (stored on two HDDs that failed in the past, check --repair had to add all-zero replacement chunks), unencrypted/unauthenticated Borg repo (last used with Borg 1.4.3; I did run check and upgrade beforehand), the transfer fails with three different errors. Each section below describes a different issue.

After fixing all issues the transfer succeeds. borg2 check shows multiple errors though. Besides the expected inconsistencies, some expected ones are gone and some warnings don't indicate an actually all-zero replacement chunk. See the last section below for details.

I'm running Arch Linux in an unprivileged Podman container, Borg 1.4.3 installed from package sources, and Borg 2.0.0b20.dev203+ga38d87b21.d20251203 installed from Git sources.

Output of borg --version, borg2 --version, and borg info (click to expand)

(borg_venv) [root@bad113811dd4 /]# borg --version
borg 1.4.3
(borg_venv) [root@bad113811dd4 /]# borg2 --version
borg2 2.0.0b20.dev203+ga38d87b21.d20251203
(borg_venv) [root@bad113811dd4 /]# borg info -v --show-rc --progress /borg1_repo/
Warning: Attempting to access a previously unknown unencrypted repository!
Do you want to continue? [yN] y
Synchronizing chunks cache...                                                                                           
Archives: 136, w/ cached Idx: 0, w/ outdated Idx: 0, w/o cached Idx: 136.
…
Done.                                                                                                                   
Repository ID: be7bca11684be79c384cd4d08f5f344670864921b8f590c107f6016552ea37c9                                         
Location: /borg1_repo
Encrypted: No
Cache: /root/.cache/borg/be7bca11684be79c384cd4d08f5f344670864921b8f590c107f6016552ea37c9
Security dir: /root/.config/borg/security/be7bca11684be79c384cd4d08f5f344670864921b8f590c107f6016552ea37c9
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
All archives:               36.62 TB             31.75 TB            685.83 GB                                          

                       Unique chunks         Total chunks
Chunk index:                 3635180            149794103
terminating with success status, rc 0
(borg_venv) [root@bad113811dd4 /]# borg2 transfer … (see below)
(borg_venv) [root@bad113811dd4 /]# borg2 check … (see below)
(borg_venv) [root@bad113811dd4 /]# borg2 --repo=/borg2_repo/ repo-info -v --show-rc --progress
Repository ID: 1db630ef5052025edf39b2fb4ddca028bf314404d0bd2d088f520d18be90bdf5
Location: /borg2_repo
Repository version: 3
Encrypted: No
Security directory: /root/.local/share/borg/security/1db630ef5052025edf39b2fb4ddca028bf314404d0bd2d088f520d18be90bdf5
Cache: /root/.cache/borg/1db630ef5052025edf39b2fb4ddca028bf314404d0bd2d088f520d18be90bdf5

Saving cache config

terminating with success status, rc 0

---

AttributeError: attribute comment not found

Running borg2 --repo=/borg2_repo/ transfer --other-repo=/borg1_repo/ --from-borg1 immediately fails with the following error:

AttributeError: attribute comment not found

Offending code:

borg/src/borg/manifest.py

Line 154 in a38d87b

comment=archive_item.comment, # not always present?

This error occurs right away with the first archive created with Borg 1.0 (maybe even Borg 0.x) on 2016-12-31, which didn't have support for --comment yet. Looks like Thomas already had some doubts a year back in 6ce5f2f, so I'm happy to confirm these doubts today 😄

Related: #8491 fixed by #8492

Replacing the offending code with the following fixes the error (maybe None instead of ""?):

comment=archive_item.get("comment", ""),

Full command line output (click to expand)

(borg_venv) [root@039a52d2b3dd /]# borg2 --repo=/borg2_repo/ transfer -v --show-rc --progress --other-repo=/borg1_repo/ --from-borg1
Saving cache config

Local Exception

Error:

AttributeError: attribute comment not found

If reporting bugs, please include the following:

Traceback (most recent call last):
  File "/borg_repo/src/borg/archiver/__init__.py", line 651, in main
    exit_code = archiver.run(args)
  File "/borg_repo/src/borg/archiver/__init__.py", line 545, in run
    rc = func(args)
  File "/borg_repo/src/borg/archiver/_common.py", line 213, in wrapper
    return method(self, args, **kwargs)
  File "/borg_repo/src/borg/archiver/_common.py", line 143, in wrapper
    return method(self, args, repository=repository, cache=cache_, **kwargs)
  File "/borg_repo/src/borg/archiver/transfer_cmd.py", line 137, in do_transfer
    archive_infos = other_manifest.archives.list_considering(args)
  File "/borg_repo/src/borg/manifest.py", line 413, in list_considering
    return self.list(
           ~~~~~~~~~^
        sort_by=args.sort_by.split(","),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<7 lines>...
        deleted=getattr(args, "deleted", False),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/borg_repo/src/borg/manifest.py", line 388, in list
    archive_infos = self._matching_info_tuples(match, match_end, deleted=deleted)
  File "/borg_repo/src/borg/manifest.py", line 176, in _matching_info_tuples
    archive_infos = list(self._info_tuples(deleted=deleted))
  File "/borg_repo/src/borg/manifest.py", line 165, in _info_tuples
    for info in self._infos(deleted=deleted):
                ~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/borg_repo/src/borg/manifest.py", line 162, in _infos
    yield self._get_archive_meta(id)
          ~~~~~~~~~~~~~~~~~~~~~~^^^^
  File "/borg_repo/src/borg/manifest.py", line 154, in _get_archive_meta
    comment=archive_item.comment,  # not always present?
            ^^^^^^^^^^^^^^^^^^^^
  File "src/borg/item.pyx", line 221, in borg.item.PropDictProperty.__get__
AttributeError: attribute comment not found

Platform: Linux 039a52d2b3dd 6.17.9-arch1-1 #1 SMP PREEMPT_DYNAMIC Mon, 24 Nov 2025 15:21:09 +0000 x86_64
Linux: Unknown Linux  
Borg: 2.0.0b20.dev203+ga38d87b21  Python: CPython 3.13.7 msgpack: 1.1.2 fuse: None [pyfuse3,llfuse]
PID: 786  CWD: /
sys.argv: ['/borg_venv/bin/borg2', '--repo=/borg2_repo/', 'transfer', '-v', '--show-rc', '--progress', '--other-repo=/borg1_repo/', '--from-borg1']
SSH_ORIGINAL_COMMAND: None


terminating with error status, rc 2

---

UnboundLocalError: cannot access local variable 'chunk_entry' where it is not associated with a value

Running borg2 --repo=/borg2_repo/ transfer --other-repo=/borg1_repo/ --from-borg1 fails with the following error after fixing the issue above:

UnboundLocalError: cannot access local variable 'chunk_entry' where it is not associated with a value

Offending code:

borg/src/borg/archiver/transfer_cmd.py

Lines 107 to 108 in a38d87b

	cache.repository.async_response(wait=False)
	chunks.append(chunk_entry)

This error occurs as soon as Borg attempts to transfer an all-zero replacement chunk. Following the comment in lines 75-77 I assume that lines 107+108 are supposed to run for the else case only, so it's likely just a simple indentation error (but I'm not entirely sure, I don't know Borg's code base well enough).

Indenting both lines one step further fixes the issue.

Full command line output (click to expand)

(borg_venv) [root@039a52d2b3dd /]# borg2 --repo=/borg2_repo/ transfer -v --show-rc --progress --other-repo=/borg1_repo/ --from-borg1
Dell-Vostro-2016-12-31T12:02:02 2016-12-31T11:02:03.396484+00:00 003625ed17185896fdffab6b0b4c743a6b21873899c12d1ce915181493d1148e: copying archive to destination repo...
Saving chunks cache 58415 N home/daniel/.local/share/tracker/data/tracker-store.journal.2.gz                                                                                                               
Saving cache config

Local Exception

Error:

UnboundLocalError: cannot access local variable 'chunk_entry' where it is not associated with a value

If reporting bugs, please include the following:

Traceback (most recent call last):
  File "/borg_repo/src/borg/archiver/__init__.py", line 651, in main
    exit_code = archiver.run(args)
  File "/borg_repo/src/borg/archiver/__init__.py", line 545, in run
    rc = func(args)
  File "/borg_repo/src/borg/archiver/_common.py", line 213, in wrapper
    return method(self, args, **kwargs)
  File "/borg_repo/src/borg/archiver/_common.py", line 143, in wrapper
    return method(self, args, repository=repository, cache=cache_, **kwargs)
  File "/borg_repo/src/borg/archiver/transfer_cmd.py", line 222, in do_transfer
    chunks, transfer, present = transfer_chunks(
                                ~~~~~~~~~~~~~~~^
        upgrader,
        ^^^^^^^^^
    ...<7 lines>...
        args.chunker_params,
        ^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/borg_repo/src/borg/archiver/transfer_cmd.py", line 108, in transfer_chunks
    chunks.append(chunk_entry)
                  ^^^^^^^^^^^
UnboundLocalError: cannot access local variable 'chunk_entry' where it is not associated with a value

Platform: Linux 039a52d2b3dd 6.17.9-arch1-1 #1 SMP PREEMPT_DYNAMIC Mon, 24 Nov 2025 15:21:09 +0000 x86_64
Linux: Unknown Linux  
Borg: 2.0.0b20.dev203+ga38d87b21.d20251203  Python: CPython 3.13.7 msgpack: 1.1.2 fuse: None [pyfuse3,llfuse]
PID: 1152  CWD: /
sys.argv: ['/borg_venv/bin/borg2', '--repo=/borg2_repo/', 'transfer', '-v', '--show-rc', '--progress', '--other-repo=/borg1_repo/', '--from-borg1']
SSH_ORIGINAL_COMMAND: None


terminating with error status, rc 2

---

AssertionError: info is None, not of type <class 'list'>

Running borg2 --repo=/borg2_repo/ transfer --other-repo=/borg1_repo/ --from-borg1 fails with the following error after fixing both issues above:

AssertionError: info is None, not of type <class 'list'>

Offending code:

borg/src/borg/helpers/fs.py

Line 384 in a38d87b

assert isinstance(info, self.info_type), f"info is {info!r}, not of type {self.info_type}"

Called by:

borg/src/borg/upgrade.py

Line 85 in a38d87b

self.hlm.remember(id=hlid, info=item.get("chunks"))

Absolutely no idea what this issue is about and what causes it. The transfer fails pretty late in the transfer for an archive created with Borg 1.4 (likely Borg 1.4.2) on 2025-11-16. Due to the other errors I resumed the transfer (i.e., I didn't start with a clean borg2 repo).

I was able to circumvent the error by replacing the calling code by the following, but this likely is just a workaround and doesn't actually fix the underlying issue:

self.hlm.remember(id=hlid, info=item.get("chunks", [])) 

Full command line output (click to expand)

(borg_venv) [root@bad113811dd4 /]# borg2 --repo=/borg2_repo/ transfer -v --show-rc --progress --other-repo=/borg1_repo/ --from-borg1
Dell-Vostro-2016-12-31T12:02:02 2016-12-31T11:02:03.396484+00:00: archive is already present in destination repo, skipping.
…
7fe919ca58b429e3c78fb466b5d5ff3447cb6bc30f52b24f4020eb9eca03e021: copying archive to destination repo...
Tuxedo-InfinityBook-14-2025-09-28T14:22:48 2025-09-28T12:22:49.527522+00:00
7fe919ca58b429e3c78fb466b5d5ff3447cb6bc30f52b24f4020eb9eca03e021: finished. transfer_size: 15.74 GB present_size: 306.10 GB                                                                           
Tuxedo-InfinityBook-14-2025-09-30T09:04:58 2025-09-30T07:04:59.320317+00:00
…
53f59dcc117c10fa6fb6f4db25cca0a471c5451a908957a8a7a17b5c0076e5a5: copying archive to destination repo...
Saving chunks cache U 978831 N srv/http/.local/share/containers/storage/overlay/3a242a69d6d2e1e9eba6b028cfbfecae083215a4fe0034ffdf9244094b51a1ee/diff1/sbin/zcip                                                                                                                  
Saving cache config

Local Exception

Error:

AssertionError: info is None, not of type <class 'list'>

If reporting bugs, please include the following:

Traceback (most recent call last):
  File "/borg_repo/src/borg/archiver/__init__.py", line 651, in main
    exit_code = archiver.run(args)
  File "/borg_repo/src/borg/archiver/__init__.py", line 545, in run
    rc = func(args)
  File "/borg_repo/src/borg/archiver/_common.py", line 213, in wrapper
    return method(self, args, **kwargs)
  File "/borg_repo/src/borg/archiver/_common.py", line 143, in wrapper
    return method(self, args, repository=repository, cache=cache_, **kwargs)
  File "/borg_repo/src/borg/archiver/transfer_cmd.py", line 239, in do_transfer
    item = upgrader.upgrade_item(item=item)
  File "/borg_repo/src/borg/upgrade.py", line 85, in upgrade_item
    self.hlm.remember(id=hlid, info=item.get("chunks"))
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/borg_repo/src/borg/helpers/fs.py", line 384, in remember
    assert isinstance(info, self.info_type), f"info is {info!r}, not of type {self.info_type}"
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
AssertionError: info is None, not of type <class 'list'>

Platform: Linux bad113811dd4 6.17.9-arch1-1 #1 SMP PREEMPT_DYNAMIC Mon, 24 Nov 2025 15:21:09 +0000 x86_64
Linux: Unknown Linux  
Borg: 2.0.0b20.dev203+ga38d87b21.d20251203  Python: CPython 3.13.7 msgpack: 1.1.2 fuse: None [pyfuse3,llfuse]
PID: 773  CWD: /
sys.argv: ['/borg_venv/bin/borg2', '--repo=/borg2_repo/', 'transfer', '-v', '--show-rc', '--progress', '--other-repo=/borg1_repo/', '--from-borg1']
SSH_ORIGINAL_COMMAND: None


terminating with error status, rc 2

---

Unexpected inconsistencies found by borg2 check

Running borg2 --repo=/borg2_repo/ check after borg2 transfer (after fixing all issues above) causes check to report a lot of inconsistencies. This is expected for all-zero replacement chunks, but a few things are weird:

• Some expected inconsistencies are gone. For example, borg check (Borg 1.4) reports a few missing chunks for the 2016-12-31 archive also mentioned earlier (it actually caused the second error above), but borg2 check reports no inconsistencies for the same archive. This is true for multiple archives actually, but not all. Even though I don't know the exact Borg versions used to create what archive (related: it might be a good idea to add this info to the archive metadata), I assume (!) that the affected archives were created with Borg 1.0 or Borg 0.x, whereas all other archives were created with Borg 1.1 or later. Anyway, either Borg 1.4 is at fault and fails to repair a chunk that Borg 2 could repair while transferring the archive, or Borg 2 isn't reporting an inconsistent chunk. Any ideas?

• Some inconsistencies are reported in chunks with a size of >0 bytes. For all-zero replacement chunks I'd expect size inconsistency detected: size ???, chunks size 0 warnings only, but there are others as well. However, if comparing the list of affected files per archive, both lists are identical (besides the mentioned archives above). Is this expected (like "all-zero" doesn't really mean "all-zero")?

  Tuxedo-InfinityBook-14-2019-12-31T12:36:48: home/daniel/some/path: size inconsistency detected: size 1829, chunks size 0
  Tuxedo-InfinityBook-14-2020-12-30T15:47:15: var/lib/snapd/snaps/phpstorm_193.snap: size inconsistency detected: size 429719552, chunks size 426630659
  

Full command line output (click to expand)

(borg_venv) [root@bad113811dd4 /]# borg2 --repo=/borg2_repo/ check -v --show-rc --progress
Starting full repository check
Starting from beginning.
Checkpointing at key data/1093d94dda17041328255467f200df24aa5998600b2c4e4f9571114fe1129edb.
…
Checkpointing at key data/f7c0aaa93ad6cb912a714ab636f2166aa7aab6148c044e2471cb1b30ffd6615c.
Finished repository check.
Checked 3658563 repository objects, 0 errors.
Finished full repository check, no problems found.
Starting archive consistency check...
Checking archives 0.0%
Analyzing archive Dell-Vostro-2016-12-31T12:02:02 2016-12-31 11:02:03.396484+00:00 48decb3131eaecf375d2267de209f0e06d259025864e481495c5c645ae8343e9 (1/137)
…
Analyzing archive Tuxedo-InfinityBook-14-2019-12-31T12:36:48 2019-12-31 11:36:53.695344+00:00 fa7d5ded7eaba59fee5bca362b3fbd5850e49e9e1231a8ea3f125d65ec3bb3d8 (4/137)
Tuxedo-InfinityBook-14-2019-12-31T12:36:48: home/daniel/some/path: size inconsistency detected: size 1829, chunks size 0
…
Checking archives 2.9%
Analyzing archive Tuxedo-InfinityBook-14-2020-12-30T15:47:15 2020-12-30 14:47:21.902702+00:00 081ee458d4439cea699517af0c6058f8c39220f386879df1ead10bced817bbea (5/137)
Tuxedo-InfinityBook-14-2020-12-30T15:47:15: var/lib/snapd/snaps/phpstorm_193.snap: size inconsistency detected: size 429719552, chunks size 426630659
…
Checking archives 99.3%
Analyzing archive Tuxedo-InfinityBook-14-2025-12-03T12:01:26 2025-12-03 11:01:27.681778+00:00 5a44bae062ad6b55d13202f307c317c74592940218d3aec815bcca61499e148c (137/137)

Archive consistency check complete, no problems found.
terminating with success status, rc 0

[ThomasWaldmann]

Good catch, thanks for practically testing!

• comment: correct analysis, fixed.

• indentation: correct analysis, fixed.

• "chunks" not present in item borg considers a hardlink master: weird. can you check path and mode / file type of the crashing item?

[ThomasWaldmann]

Inconsistencies:

• weird. borg2 transfer tries to transfer the "healthy" chunkids (not any all-zero replacement chunks). if they are gone at the source repo, it just skips them, so they won't be present at the destination repo either. so borg2 check should discover them as missing. at least until they reappear, if ever.
• size inconsistencies: there might be a computed/cached size in item.size (not sure since which borg version) and there is the sum of chunk sizes from item.chunks, which is a list of (id, size) tuples, both values should always be identical.

[PhrozenByte]

Note: I'm running Borg 2 with your PR from now on, i.e. Borg 2.0.0b20.dev209+ge1c60049c.

"chunks" not present in item borg considers a hardlink master: weird. can you check path and mode / file type of the crashing item?

I just did a borg2 delete of the affected archive followed by borg2 compact (which took ages btw) and another borg2 transfer.

After adding some poor-man's-debugging (i.e. print()), here's hlid and item:

hlid = b'\xc4-\xfe\xa1\xfa\xaa,Yf\xd4\x95x\xf0\xe71\xca\x19B\x90\xa3`\xdd\xc9\n\xf9@\xd41\xd6\nm\x89'fbfecae083215a4fe0034ffdf9244094b51a1ee/diff1/usr/share/perl5/core_perl/Term/Table/LineBreak.pm       
item = Item(internal_dict={'ctime': Timestamp(seconds=1763250430, nanoseconds=525253842), 'gid': 33, 'group': 'http', 'hardlink_master': True, 'mode': 8192, 'mtime': Timestamp(seconds=1763250430, nanoseconds=520253738), 'path': 'srv/http/.local/share/containers/storage/overlay/6077cdaa1b3b59556e831c675db63da9c48249634205b1324bbb50813cf189a0/diff/etc/ssh/ssh_host_ed25519_key.pub', 'rdev': 0, 'uid': 33, 'user': 'http', 'hlid': b'\xc4-\xfe\xa1\xfa\xaa,Yf\xd4\x95x\xf0\xe71\xca\x19B\x90\xa3`\xdd\xc9\n\xf9@\xd41\xd6\nm\x89'})

What exactly is Borg doing there (i.e., what does "considers a hardlink master" mean)?

The reason I'm asking is because the repos /borg1_repo and /borg2_repo are, even though stored on the same physical device, mounted in a way that doesn't allow for cross-mount hardlinks (i.e., one can create hardlinks within /borg1_repo and within /borg2_repo, but not hardlinks between them):

(borg_venv) [root@8959d1db8319 /]# echo foo > /borg1_repo/foo
(borg_venv) [root@8959d1db8319 /]# ln /borg1_repo/foo /borg1_repo/bar
(borg_venv) [root@8959d1db8319 /]# cat /borg1_repo/bar
foo
(borg_venv) [root@8959d1db8319 /]# echo foo > /borg2_repo/foo
(borg_venv) [root@8959d1db8319 /]# ln /borg2_repo/foo /borg2_repo/bar
(borg_venv) [root@8959d1db8319 /]# cat /borg2_repo/bar
foo
(borg_venv) [root@8959d1db8319 /]# ln /borg1_repo/foo /borg2_repo/other_foo
ln: failed to create hard link '/borg2_repo/other_foo' => '/borg1_repo/foo': Invalid cross-device link

By the way, I noticed that the borg1_hardlink_slave case does have a if chunks is not None check. But as I said, I don't know Borg's code base well enough to draw any conclusions, I just noticed it…

weird. borg2 transfer tries to transfer the "healthy" chunkids (not any all-zero replacement chunks). if they are gone at the source repo, it just skips them, so they won't be present at the destination repo either. so borg2 check should discover them as missing. at least until they reappear, if ever.

I just tried to extract home/daniel/Projekte/GitHub/nextcloud/lib/private/Search/Result/File.php from the Dell-Vostro-2018-12-31T12:35:44 archive:

Borg 1.4 reports a backup error and extracts 2.6K of junk:

(borg_venv) [root@8959d1db8319 borg_extract]# borg extract /borg1_repo::Dell-Vostro-2018-12-31T12:35:44 home/daniel/Projekte/GitHub/nextcloud/lib/private/Search/Result/File.php
home/daniel/Projekte/GitHub/nextcloud/lib/private/Search/Result/File.php: backup error
(borg_venv) [root@8959d1db8319 borg_extract]# ls -hl home/daniel/Projekte/GitHub/nextcloud/lib/private/Search/Result/File.php
-rw-r--r-- 1 1000 1000 2.6K Mar  3  2018 home/daniel/Projekte/GitHub/nextcloud/lib/private/Search/Result/File.php
(borg_venv) [root@8959d1db8319 borg_extract]# hexdump home/daniel/Projekte/GitHub/nextcloud/lib/private/Search/Result/File.php
0000000 0000 0000 0000 0000 0000 0000 0000 0000
*
0000a60 0000                                   
0000a62

Borg 2 on the other hand happily extracts the corrupted file with a size of 0 bytes:

(borg_venv) [root@8959d1db8319 borg_extract]# borg2 -r /borg2_repo/ extract Dell-Vostro-2018-12-31T12:35:44 home/daniel/Projekte/GitHub/nextcloud/lib/private/Search/Result/File.php
(borg_venv) [root@8959d1db8319 borg_extract]# ls -hl home/daniel/Projekte/GitHub/nextcloud/lib/private/Search/Result/File.php
-rw-r--r-- 1 1000 1000 0 Mar  3  2018 home/daniel/Projekte/GitHub/nextcloud/lib/private/Search/Result/File.php
(borg_venv) [root@8959d1db8319 borg_extract]# hexdump home/daniel/Projekte/GitHub/nextcloud/lib/private/Search/Result/File.php

The weird thing is that this happens only with my Dell-Vostro-* archives from 2016-2018 (which didn't actually run on a Dell Vostro laptop, but anyway…). All later archives aren't affected by this issue, and all these later archives were created with Borg 1.1 and later (I know this for sure because Tuxedo shipped their laptops with Ubuntu 18.04 back then, which included Borg 1.1). However, I'm not sure what Borg version I was running with the Dell-Vostro-* archives (I don't even remember whether I was using Debian or Ubuntu back then)… However, I'm absolutely sure it wasn't Borg 1.1 (because I remember that the new device was quite a version bump), but likely Borg 1.0, or maybe even Borg 0.x.

Any ideas how to debug this?

size inconsistencies: there might be a computed/cached size in item.size (not sure since which borg version) and there is the sum of chunk sizes from item.chunks, which is a list of (id, size) tuples, both values should always be identical.

Any ideas how to debug this, or can it be considered just a cosmetic issue that needs no fixing?

[ThomasWaldmann]

borg1 (and attic) hardlink handling: if A and B are hardlinked source files and borg backups A first, then B, A will be the hardlink master and B will be the slave. Only the master has the chunks list (for contentful files) and the slave refers back to the master by name via item.source.

That's a sub-optimal and asymmetric approach:

• partial extracts are possible, but if we skip A and only extract B, we have nothing in the extracted files to hardlink to.
• recreate could even drop the hardlink master by an updated exclude list
• storing item.chunks in each hardlinked item would be no big thing, it is just a list of chunkids, so the asymmetrical borg1 approach was an over-optimization (maybe inspired by tar, or by symlinks) causing complications.

borg2 has a simpler symmetric approach:

• stores all hardlinked items in the same way, all include the chunks list for contentful files
• hardlinked items are bound together by having the identical hlid (256bit value) - always using a hlid (and not a path or devno/ino or ...) makes processing always the same.
• considering that the hardlinks in filesystems are also symmetric, this is the better way to represent it.

[ThomasWaldmann]

>>> from stat import *
>>> S_ISCHR(8192)
True

So, that weird file (srv/http/.local/share/containers/storage/overlay/ 6077cdaa1b3b59556e831c675db63da9c48249634205b1324bbb50813cf189a0/diff/etc/ssh/ssh_host_ed25519_key.pub) has the mode of a char device, although the file name indicates it should be a regular file. But I have no idea what that "container overlay" is doing there.

The reason I'm asking is because the repos /borg1_repo and /borg2_repo are,
even though stored on the same physical device, mounted in a way that doesn't
allow for cross-mount hardlinks

The whole discussion only refers to archived hardlinked source files, not to repo storage files.

By the way, I noticed that the borg1_hardlink_slave case does have a if chunks is not None check.

I'll have a look.

About the size inconsistency: it would be interesting to find out the correct size and then how the inconsistent incorrect size got into there.

[PhrozenByte]

borg1 (and attic) hardlink handling: […]
[…]
The whole discussion only refers to archived hardlinked source files, not to repo storage files.

Got it, thanks for the detailed explanation! 👍

>>> from stat import *
>>> S_ISCHR(8192)
True

So, that weird file (srv/http/.local/share/containers/storage/overlay/ 6077cdaa1b3b59556e831c675db63da9c48249634205b1324bbb50813cf189a0/diff/etc/ssh/ssh_host_ed25519_key.pub) has the mode of a char device, although the file name indicates it should be a regular file. But I have no idea what that "container overlay" is doing there.

Yeah, I totally understand the confusion, feels super weird, but this file being a character device is actually correct: It's an overlayfs whiteout file. This path belongs to a Podman container. The filesystem of a container usually consists of multiple layers that are stacked upon each other with OverlayFS. However, what if a file that is stored in one of these layers is deleted at runtime? The layers are immutable, so a whiteout file is created: It tells overlayfs that the file was deleted. Whiteout files are character devices that are created in a special diff directory with major/minor 0:0 and a size of 0 bytes. Why did the kernel devs decide to implement it that way? No idea. But this character device basically just tells overlayfs that the etc/ssh/ssh_host_ed25519_key.pub file that exists in some lower layer was deleted.

However: AFAIK (I'm not 100% sure, but pretty sure) whiteout files are never hardlinks. Is there a way to check whether Borg 1.4 already stored it as hardlink, or whether this originates in Borg 2?

From a practical perspective it's kinda pointless to include these files in backups, but they shouldn't cause any issues. Borg 1.4 has no issues with that file (running as root on the host now because the unprivileged container doesn't have the necessary permissions to extract that file):

# borg extract /run/media/daniel/themisto/Backup/Borg/::Tuxedo-InfinityBook-14-2025-11-16T11:38:11 srv/http/.local/share/containers/storage/overlay/6077cdaa1b3b59556e831c675db63da9c48249634205b1324bbb50813cf189a0/diff/etc/ssh/ssh_host_ed25519_key.pub
# ls -hl srv/http/.local/share/containers/storage/overlay/6077cdaa1b3b59556e831c675db63da9c48249634205b1324bbb50813cf189a0/diff/etc/ssh/ssh_host_ed25519_key.pub 
c--------- 1 http http 0, 0 16. Nov 00:47 srv/http/.local/share/containers/storage/overlay/6077cdaa1b3b59556e831c675db63da9c48249634205b1324bbb50813cf189a0/diff/etc/ssh/ssh_host_ed25519_key.pub

#edit: wrong assumptions removed (click to expand)

Also note that the hid variable references another file (???fbfecae083215a4fe0034ffdf9244094b51a1ee/diff1/usr/share/perl5/core_perl/Term/Table/LineBreak.pm). This incomplete path only matches a single path in the archive: srv/http/.local/share/containers/storage/overlay/3a242a69d6d2e1e9eba6b028cfbfecae083215a4fe0034ffdf9244094b51a1ee/diff1/usr/share/perl5/core_perl/Term/Table/LineBreak.pm. However, that's no whiteout file, but a regular file (i.e., the file was updated at runtime, so overlayfs had to create a copy of that file) and thus can't possibly be a hardlink of the whiteout file:

# borg extract /run/media/daniel/themisto/Backup/Borg/::Tuxedo-InfinityBook-14-2025-11-16T11:38:11 srv/http/.local/share/containers/storage/overlay/3a242a69d6d2e1e9eba6b028cfbfecae083215a4fe0034ffdf9244094b51a1ee/diff1/usr/share/perl5/core_perl/Term/Table/LineBreak.pm
# ls -hl srv/http/.local/share/containers/storage/overlay/3a242a69d6d2e1e9eba6b028cfbfecae083215a4fe0034ffdf9244094b51a1ee/diff1/usr/share/perl5/core_perl/Term/Table/LineBreak.pm
-r--r--r-- 1 200000 200000 2,7K  3. Aug 15:51 srv/http/.local/share/containers/storage/overlay/3a242a69d6d2e1e9eba6b028cfbfecae083215a4fe0034ffdf9244094b51a1ee/diff1/usr/share/perl5/core_perl/Term/Table/LineBreak.pm

Picking up my idea to check whether Borg 1.4 stored these files as hardlinks, I tried to extract both files at the same time, but they show different inodes; but I don't know whether Borg even attempts to restore them as hardlink if they would have been backed up as hardlinks:

# borg extract /run/media/daniel/themisto/Backup/Borg/::Tuxedo-InfinityBook-14-2025-11-16T11:38:11 srv/http/.local/share/containers/storage/overlay/6077cdaa1b3b59556e831c675db63da9c48249634205b1324bbb50813cf189a0/diff/etc/ssh/ssh_host_ed25519_key.pub srv/http/.local/share/containers/storage/overlay/3a242a69d6d2e1e9eba6b028cfbfecae083215a4fe0034ffdf9244094b51a1ee/diff1/usr/share/perl5/core_perl/Term/Table/LineBreak.pm
# ls -hli srv/http/.local/share/containers/storage/overlay/6077cdaa1b3b59556e831c675db63da9c48249634205b1324bbb50813cf189a0/diff/etc/ssh/ssh_host_ed25519_key.pub srv/http/.local/share/containers/storage/overlay/3a242a69d6d2e1e9eba6b028cfbfecae083215a4fe0034ffdf9244094b51a1ee/diff1/usr/share/perl5/core_perl/Term/Table/LineBreak.pm
655483 -r--r--r-- 1 200000 200000 2,7K  3. Aug 15:51 srv/http/.local/share/containers/storage/overlay/3a242a69d6d2e1e9eba6b028cfbfecae083215a4fe0034ffdf9244094b51a1ee/diff1/usr/share/perl5/core_perl/Term/Table/LineBreak.pm
655488 c--------- 1 http   http   0, 0 16. Nov 00:47 srv/http/.local/share/containers/storage/overlay/6077cdaa1b3b59556e831c675db63da9c48249634205b1324bbb50813cf189a0/diff/etc/ssh/ssh_host_ed25519_key.pub

About the size inconsistency: it would be interesting to find out the correct size and then how the inconsistent incorrect size got into there.

Extracting the file with Borg 1.4 naturally prints an backup error, but does the file have the correct size then? Checking the file with hexdump shows that the all-zero replacement block is likely at the end of the file, but it's just a little more than 1 kB.

# borg extract /run/media/daniel/themisto/Backup/Borg/::Tuxedo-InfinityBook-14-2020-12-30T15:47:15 var/lib/snapd/snaps/phpstorm_193.snap
var/lib/snapd/snaps/phpstorm_193.snap: backup error
# ls -l var/lib/snapd/snaps/phpstorm_193.snap 
-rw------- 1 root root 429719552  3. Dez 2020  var/lib/snapd/snaps/phpstorm_193.snap
# hexdump -C var/lib/snapd/snaps/phpstorm_193.snap | grep -E -B3 -A3 '^[0-9a-f]+  (00 ){8} (00 ){8} '
008f7f30  34 3b f7 19 56 42 27 fc  67 59 55 70 b5 e2 78 10  |4;..VB'.gYUp..x.|
008f7f40  9f 05 00 77 4f 46 32 00  01 00 00 00 00 f9 34 00  |...wOF2.......4.|
008f7f50  11 00 00 00 02 7e d8 00  00 f8 d0 00 02 00 00 00  |.....~..........|
008f7f60  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
008f7f70  00 00 00 1b 81 b2 0c 1c  b2 68 06 60 00 a5 48 08  |.........h.`..H.|
008f7f80  56 09 83 3c 11 0c 0a 87  9f 08 86 c2 5e 12 89 6c  |V..<........^..l|
008f7f90  01 36 02 24 03 a7 08 0b  93 46 00 04 20 05 85 54  |.6.$.....F.. ..T|
--
0f985bf0  5a 8d 83 65 95 cc ce 9d  58 5b de 5c 8d 0a e0 17  |Z..e....X[.\....|
0f985c00  98 3b b2 c6 f9 69 64 d9  89 e1 ed 9c fe b7 a2 23  |.;...id........#|
0f985c10  d7 85 b0 4d 06 e3 39 4a  d0 09 47 4e 03 db 85 ab  |...M..9J..GN....|
0f985c20  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
0fc77e10  00 00 00 00 00 00 00 00  00 00 00 00 00 0f b5 24  |...............$|
0fc77e20  40 14 28 b5 08 69 bd 5a  b9 37 08 f8 00 0b 57 dc  |@.(..i.Z.7....W.|
--
199cfbb0  66 e4 ba 3e 30 0d 8b 02  00 00 00 00 01 59 5a a3  |f..>0........YZ.|
199cfbc0  ef 9c 19 00 00 00 00 9d  f5 9c 19 00 00 00 00 04  |................|
199cfbd0  80 00 00 00 00 cf fb 9c  19 00 00 00 00 00 00 00  |................|
199cfbe0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
199d0000

borg2 check reported it as follows:

Tuxedo-InfinityBook-14-2020-12-30T15:47:15: var/lib/snapd/snaps/phpstorm_193.snap: size inconsistency detected: size 429719552, chunks size 426630659

How can I print the chunk size in Borg 1.4? I'm happy to do anything that helps with debugging, just let me know what I need to do.

[ThomasWaldmann]

A hlid should look like b'...(256bits represented as byte string)...'.

What you gave looks like either some mixed-up output or a c&p error:

hlid = b'\xc4-\xfe\xa1\xfa\xaa,Yf\xd4\x95x\xf0\xe71\xca\x19B\x90\xa3`\xdd\xc9\n\xf9@\xd41\xd6\nm\x89'fbfecae083215a4fe0034ffdf9244094b51a1ee/diff1/usr/share/perl5/core_perl/Term/Table/LineBreak.pm 

Note that the closing quote is in the middle of that string.
I don't think the LineBreak.pm path is part of the hlid.

[PhrozenByte]

About the size inconsistency of var/lib/snapd/snaps/phpstorm_193.snap:

I just noticed that I totally missed the actually way bigger NUL byte block 🙈 It starts at 0x0f985c20 and ends at 0x0fc77e1d and thus has a size of 3,089,214 bytes. borg2 check reports a difference of 429,719,552 expected - 426,630,659 actual = 3,088,893 bytes. Could be a coincidence, but seeing that now I wonder whether the message just lead me into making wrong assumptions and everything is fine actually (the issue with the Dell-Vostro-* archives persists though)…

Assuming everything is correct, could Borg 2 print the same warning as Borg 1.4 did? The old warning was way more clear from an user's perspective.

A hlid should look like b'...(256bits represented as byte string)...'.
What you gave looks like either some mixed-up output or a c&p error:

Indeed, it's mixed-up output from --progress. I added a few newlines to the print() debug outputs and it gives me the following now, so yes, that other paths seems to be unrelated:

hlid = b'\xc4-\xfe\xa1\xfa\xaa,Yf\xd4\x95x\xf0\xe71\xca\x19B\x90\xa3`\xdd\xc9\n\xf9@\xd41\xd6\nm\x89'
item = Item(internal_dict={'ctime': Timestamp(seconds=1763250430, nanoseconds=525253842), 'gid': 33, 'group': 'http', 'hardlink_master': True, 'mode': 8192, 'mtime': Timestamp(seconds=1763250430, nanoseconds=520253738), 'path': 'srv/http/.local/share/containers/storage/overlay/6077cdaa1b3b59556e831c675db63da9c48249634205b1324bbb50813cf189a0/diff/etc/ssh/ssh_host_ed25519_key.pub', 'rdev': 0, 'uid': 33, 'user': 'http', 'hlid': b'\xc4-\xfe\xa1\xfa\xaa,Yf\xd4\x95x\xf0\xe71\xca\x19B\x90\xa3`\xdd\xc9\n\xf9@\xd41\xd6\nm\x89'})