Docker setup error: `AttributeError: 'UTA_postgresql' object has...`

[hotsoupisgood]

rest_variantvalidator-restvv-1   | Traceback (most recent call last):
rest_variantvalidator-restvv-1   |   File "/usr/local/lib/python3.10/site-packages/vvhgvs/dataproviders/uta.py", line 515, in __del__
rest_variantvalidator-restvv-1   |     self.close()
rest_variantvalidator-restvv-1   |   File "/usr/local/lib/python3.10/site-packages/vvhgvs/dataproviders/uta.py", line 519, in close
rest_variantvalidator-restvv-1   |     self._pool.closeall()
rest_variantvalidator-restvv-1   | AttributeError: 'UTA_postgresql' object has no attribute '_pool'```

[hotsoupisgood]

Hi, I am getting this error when running docker compose up. It seems like it could be a version issue of python or postgres. Working them out right now.

[hotsoupisgood]

Ok so I am still getting this same error AttributeError: 'UTA_postgresql' object has no attribute '_pool' even when using a newer version of posgresql

[Peter-J-Freeman]

This is an odd one because I'm building on mac M1 Silicone using the ARM chip method and all works fine.

The error you are reporting we have seen before @hotsoupisgood

https://github.com/openvar/variantValidator/blob/master/docs/COMMON_ERRORS.md

It usually occurs when SeqRepo database cannot be called by the looks of things.

First steps @beboche and @hotsoupisgood .

Have you made sure all the host-machine directories are made?

$ mkdir ~/variantvalidator_data
$ mkdir ~/variantvalidator_data/share
$ mkdir ~/variantvalidator_data/share/seqrepo/
$ mkdir ~/variantvalidator_data/share/logs

These are required for the SeqRepo database to build

Note, this is for the docker build.

[Peter-J-Freeman]

When building on LINUX, the issue is usually that the path to your SeqRepo database is set incorrectly and VV cannot find it.

Have you configured your .variantvalidator config file? it should be located in ~/.variantvalidator.

This might well not be a bug but a config issue.

[Peter-J-Freeman]

@hotsoupisgood. Let me know if you want to set up a call so I can help with the config. Might be quicker then we can post the solution here

[hotsoupisgood]

Hey @Peter-J-Freeman. I was out of the office on Friday and could set up a meeting tomorrow afternoon(PST) if you are free.

I did make those directories. But another thing I should have noted in my original post is that I am making these directories in a different location. I did change the docker compose file to point toward my new location. I am not sure if this could be causing an issue or not.

Additionally I changed some ports being used in the docker compose.

Potentially one of these changes could be causing the error but I am having some trouble figuring it out.

[Peter-J-Freeman]

That might be doing it. If SeqRepo cannot be found then this error will occur. We may need to look at your build files and make sure the settings are adjusted.

An overview of the directories

The container wants to make these internally

  - logs:/usr/local/share/logs
  - seqdata:/usr/local/share/seqrepo
  - share:/usr/local/share

And binds them to the host directories you create

  share:
    driver: local
    driver_opts:
      type: 'none'
      o: 'bind'
      device: '${HOME}/variantvalidator_data/share'
  seqdata:
    driver: local
    driver_opts:
      type: 'none'
      o: 'bind'
      device: '${HOME}/variantvalidator_data/share/seqrepo'
  logs:
    driver: local
    driver_opts:
      type: 'none'
      o: 'bind'
      device: '${HOME}/variantvalidator_data/share/logs'

SeqRepo database Should be being built into /usr/local/share/seqrepo unless you changed anything so the config file should be

version = VV_SR_2022_11/master
location = /usr/local/share/seqrepo
require_threading = True

It looks like the container is building so you should be able to enter the container using docker-compose run restvv bash. Navigate to /usr/local/share/seqrepo and check to see if VV_SR_2022_11/master is present.

Happy to have a call. Send a contact message here https://variantvalidator.org/help/contact/ and we can set something up.

[Peter-J-Freeman]

OK. Testing the following.

Python 3.10 should be fine. I run that on many machines. I have updated the Postgres container to 14.5

I cleared my SeqRepo build in ~/variantvalidator_data/share/seqrepo on my local system.

Pruned all containers

Deleted all volumes from my system

Trying a build now

[Peter-J-Freeman]

I found that vvta fails with postgres 14. I need to remember the steps I had to take to fix this.

It's in here https://github.com/openvar/variantValidator/blob/master/docs/INSTALLATION.md

Postgres < version 14.0

$ wget --output-document=VVTA_202x_xx.noseq.psql.gz https://www528.lamp.le.ac.uk/vvdata/vvta/VVTA_202x_xx.noseq.psql.gz
$ gzip -cdq VVTA_202x_xx.noseq.psql.gz | psql -U <USER> -v ON_ERROR_STOP=0 -d vvta -Eae

Postgres >= version 14.0

$ wget --output-document=VVTA_202x_xx.noseq.psql.gz https://www528.lamp.le.ac.uk/vvdata/vvta/VVTA_202x_xx.noseq.psql.gz
$ gzip -cdq -k VVTA_202x_xx.noseq.psql.gz | sed 's/anyarray/anycompatiblearray/g' | psql -U <USER> -v ON_ERROR_STOP=0 -d vvta -Eae

So the vdb build needs to be updated

[hotsoupisgood]

Hey Peter, Thank you for the help.
So regarding your last message, are you suggesting I replace the

RUN wget --output-document=vvta_2022_11_noseq.sql.gz  https://www528.lamp.le.ac.uk/vvdata/vvta/vvta_2022_11_no_seq.sql.gz  -O /docker-entrypoint-initdb.d/vvta_2022_11_noseq.sql.gz

With something like this because I am running postgres 13.10? I am able to run any version.

RUN wget --output-document=VVTA_202x_xx.noseq.psql.gz https://www528.lamp.le.ac.uk/vvdata/vvta/VVTA_202x_xx.noseq.psql.gz
RUN gzip -cdq VVTA_202x_xx.noseq.psql.gz | psql -U <USER> -v ON_ERROR_STOP=0 -d vvta -Eae

Also, regarding the configuration files. Is the only thing one normally has to add is their email under the Entrez section? I only changed the host location of the variantvalidator_data folder(in docker compose) so don't belive I need to alter the internal locations.

[Peter-J-Freeman]

Basically, I'm having to try and manipulate the download and docker entry point to get the data to loas into postgres 14. So currently, I would leave the vvta.df file as it is except you can upgrade to 13.10.

I would also try placing the directories in the places they were specified in the manual for now so we can debug. It might be that we need to change the docker-compose file to match the locations of your local directories, but one thing at a time.

First we need to try get a build working.

For now, try making the directories as stated in the manual and building. You can also update the postgres version to 13.10.

If you are using postgres 13.10, the vvta df should be

FROM postgres:13.10

ENV POSTGRES_DB=vvta

ENV POSTGRES_USER=uta_admin

ENV POSTGRES_PASSWORD=uta_admin

RUN apt-get update && apt-get install -y wget && rm -rf /var/lib/apt/lists/*

RUN wget --output-document=vvta_2022_11_noseq.sql.gz  https://www528.lamp.le.ac.uk/vvdata/vvta/vvta_2022_11_no_seq.sql.gz  -O /docker-entrypoint-initdb.d/vvta_2022_11_noseq.sql.gz

It's getting complicated trying to move up to pg14

The reason I suspect it might be the directory locations and the bindings is that the error est_variantvalidator-restvv-1 | AttributeError: 'UTA_postgresql' object has no attribute '_pool'``` usually suggests the seqrepo database is not building. I'm wondering whether if the directory bindings are incorrect, it is causing an issue with the build.

[hotsoupisgood]

@Peter-J-Freeman Are the ports in the docker.ini config the host ports? or the container's ports?

Edit: They seem to be the host ports because when I change them I get a bit further in the boot. See comment below...

[hotsoupisgood]

Hey, So I changed the ports in the config docker.ini and I got further in the restvv boot sequence.

Now I have this non specific error:

rest_variantvalidator-restvv-1   | [2023-05-02 21:45:32 +0000] [7] [INFO] Using worker: gthread
rest_variantvalidator-restvv-1   | [2023-05-02 21:45:32 +0000] [8] [INFO] Booting worker with pid: 8
rest_variantvalidator-restvv-1   | [2023-05-02 21:45:33 +0000] [7] [INFO] Shutting down: Master
rest_variantvalidator-restvv-1   | [2023-05-02 21:45:33 +0000] [7] [INFO] Reason: Worker failed to boot.

When I am running this afterward:
docker-compose run restvv bash
This directory is empty: /usr/local/share/seqrepo

Still thinking about ideas...
Edit: I sent contact message to contact above.

[Peter-J-Freeman]

This might help

#91 (comment)

[hotsoupisgood]

Hi, So I got it working using non-docker install.

However I did encounter the same "_pool" issue along the way when running the rest variant validator web server and figured it out in that case.
I am not sure if the issue was exactly the same, but it could have been one of two things.
At first I was forwarding the wrong docker postgres port. Mine looked like 60440:5433 when it should have been 60440:5432. (postgres runs on 5432 by default)
Then the other issue was the version of my VVTA db was wrong. I was using VVTA_2022_11.1_noseq.psql.gz. But in my config I had it set as vvta_2022_11, when it should have been vvta_2022_11_1.

I don't have time to go back and debug the docker version right now, but maybe I can see if the solution is the same when i have time.