IndexError: list index out of range

[brianmay]

[nix-shell:~/private/finances/ledger]$ bh format accounts.beancount
[10:03:57] INFO     Processing file accounts.beancount                             format.py:90
           INFO     Calculate column width                                     formatter.py:646
           INFO     Collecting                                                 formatter.py:130
Traceback (most recent call last):
  File "/nix/store/zjlv2snqfxnf74mrk8yaips6c12m2yqd-python3.12-beanhub-cli-1.4.1/bin/.bh-wrapped", line 9, in <module>
    sys.exit(cli())
             ^^^^^
  File "/nix/store/6k64nlqhg20cky326qhh6282z6zkpr5s-python3.12-click-8.1.7/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/6k64nlqhg20cky326qhh6282z6zkpr5s-python3.12-click-8.1.7/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/nix/store/6k64nlqhg20cky326qhh6282z6zkpr5s-python3.12-click-8.1.7/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/6k64nlqhg20cky326qhh6282z6zkpr5s-python3.12-click-8.1.7/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/6k64nlqhg20cky326qhh6282z6zkpr5s-python3.12-click-8.1.7/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/6k64nlqhg20cky326qhh6282z6zkpr5s-python3.12-click-8.1.7/lib/python3.12/site-packages/click/decorators.py", line 92, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/6k64nlqhg20cky326qhh6282z6zkpr5s-python3.12-click-8.1.7/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/zjlv2snqfxnf74mrk8yaips6c12m2yqd-python3.12-beanhub-cli-1.4.1/lib/python3.12/site-packages/beanhub_cli/format.py", line 93, in main
    formatter.format(tree, output_file)
  File "/nix/store/1l65k28d3za35dchmnvf15y93x0nn4dc-python3.12-beancount-black-1.0.5/lib/python3.12/site-packages/beancount_black/formatter.py", line 696, in format
    sections.append(self.format_statement_group(group))
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/1l65k28d3za35dchmnvf15y93x0nn4dc-python3.12-beancount-black-1.0.5/lib/python3.12/site-packages/beancount_black/formatter.py", line 573, in format_statement_group
    last_entry = entries[-1]
                 ~~~~~~~^^^^
IndexError: list index out of range

The file is valid:

$ bean-check accounts.beancount
[ no output and exit==0 ]

Unfortunately I am not sure which transaction is causing this problem.

Note: while I used beanhub-cli this bug actually looks like a problem with beancount-black,

[fangpenlin]

Hi @brianmay thanks for reporting the issue.

There are two approach I would recommend to help us finding out which line is causing the problem. First would be, could you please run the format command with debug lg level like this?

bh -l debug format accounts.beancount

That way it should generates a tons of debugging logs from the formatter which can better help us find out which line is causing the problem.

Another way is to remove lines like in half (still make it legit), see if the error goes away. Like binary search to narrow down which part of it containing the code with problem.

Finally, beancount-black doesn't support some syntax, like pushtag. Can you please help me double check that you don't have pushtags in your book?

[brianmay]

Thanks.

Here is a minimum file to reproduce:

1970-01-01 commodity AUD

1970-01-01 open Assets:Penguin AUD
1970-01-01 open Liabilities:Evil AUD

2014-10-06 * "Bike"
  * Assets:Penguin                                      1,505.00 AUD
  Liabilities:Evil

I think it doesn't like the fact I have tried to mark the transaction as completed. If I remove that it seems to work.

[fangpenlin]

Yeah, so unfortunately this is part of the rarely used syntax we don't support currently. But interesting, I just checked our parser library:

https://github.com/LaunchPlatform/beancount-parser/blob/e7e9c4b66ce86cfb03421f3290d7b8b14c40f43f/beancount_parser/grammar/beancount.lark#L91-L93

It actually parses FLAG in front of a posting. The beancount-black formatter just doesn't anticipate it. So I guess the only needed fix is to take that optional flag into consideration while performing the format. I think it shouldn't be too hard to fix. Let me see if I can fix it quickly.

[fangpenlin]

Hmm, fixing this problem is actually harder than I expected. Mostly because we handle the emacs section headers

https://github.com/LaunchPlatform/beancount-parser/blob/e7e9c4b66ce86cfb03421f3290d7b8b14c40f43f/beancount_parser/grammar/section_header.lark#L1

And it starts with * happens to be the same as a confirmed posting. This potentially solvable with something like Indentation post processing:

https://lark-parser.readthedocs.io/en/latest/examples/indented_tree.html#parsing-indentation

However, it requires some major overhaul for the Lark grammar, and will then also impact the beancount-black code a lot. I am actually thinking about rewriting beancount-black and beancount-parser with rust at somepoint. So I am not sure if I want to handle this problem right now. Will leave it open for now.