Issues with Client initialization from keyfile

[frobnitzem]

I created my keyfile as described in the docs.

$ ls -l ~/.superfacility/
total 4
-rw-------+ 1 username group 1693 Oct 24 08:48 key.pem

However, when I initialize a client, several things go wrong.

1. If no key= argument is given, then client.token is not initialized. I assume the keyfile wasn't found.

2. Using a key= argument, client.token and client._client_id are set, but client._access_token is missing, and there is no "Authorization" header set.

3. Submitting a test job gives a hard-to-use error "sfapi_client.exceptions.SfApiError: sbatch: error: Job request does not match any supported policy." It could be right, but it doesn't give a good hint on how to fix my job script / what policy to check.

import os
from pathlib import Path

from sfapi_client import Client
from sfapi_client.compute import Machine

# with Client() as client: # (1.)
keypath = Path(os.environ["HOME"]) / ".superfacility" / "key.pem"
with Client(key=keypath) as client:
    assert client.token is not None # (1.)

    # (2.) The following vars are not setup on construction using key= initialization.
    client._access_token = client.token
    client._http_client().headers.update(
        {"Authorization": "Bearer " + str(client.token)}
    )
    print(client._client_id, client._access_token)
    assert "Authorization" in client._http_client().headers # (2.)
    machine = client.compute(Machine.perlmutter)
    job = machine.submit_job("""#!/bin/bash
#SBATCH -N 2
#SBATCH -t 2
#SBATCH -A m3792
pwd
""") # (3.)

[tylern4]

Thanks for the issue.

1. It does look like there might be a bug in finding the key so I will take a look into that.

2. The _access_token is only set and used if you have a bearer token you have created outside of the client and set with Client(access_token=bearer_token) If you give the client a client_id/secret pair, or the pem file with client_id then that won't get set ever. We have the OAuth2Client and httpx library handle the authentication parts from there so we don't set that header but the libraries will populate it when the request goes out. Is there a reason for trying to check those?

3. For that error we're just passing back the Slurm error to you sbatch: error: Job request does not match any supported policy. On Perlmutter you must specify a constraint to choose between cpu nodes and gpu nodes and is required which is why you see the error. This script will work:

#!/bin/bash
#SBATCH -N 2
#SBATCH -t 2
#SBATCH -A m3792
# Constraint needs to be set to either cpu or gpu
#SBATCH -C gpu 
pwd

The full python code that works is:

import os
from pathlib import Path

from sfapi_client import Client
from sfapi_client.compute import Machine

keypath = Path(os.environ["HOME"]) / ".superfacility" / "key.pem"
with Client(key=keypath) as client:
    assert client.token is not None
    machine = client.compute(Machine.perlmutter)
    job = machine.submit_job("""#!/bin/bash
#SBATCH -N 2
#SBATCH -t 2
#SBATCH -A m3792
#SBATCH -C gpu
pwd
""")

[frobnitzem]

OK - I can confirm the example above works - so the only issue is finding the default key.pem file.

I was checking _access_token and _client_id because I kept getting authentication / not allowed errors, and couldn't determine whether these were coming from the missing auth header or (as was sometimes the case) from trying to send API requests from a host IP not specifically on the allowed list. The missing 'Authorization' header threw me off there because I didn't realize it could be added at a lower level in the call stack...