Skip to content

[Code scan] Raise cloud download failures before unzip #613

Description

@njzjz

Found by a Codex global repository scan of deepmodeling/dpdispatcher at commit 98a9e08.

Problem
Cloud download_from_url() logs failed HTTP attempts but returns normally if all retries fail and ret remains None. Callers then try to unzip a file that was never written.

Relevant code

def download_from_url(self, url, save_file):
ret = None
for retry_count in range(3):
try:
ret = requests.get(
url, headers={"Authorization": "jwt " + self.token}, stream=True
)
except Exception as e:
dlog.error(f"request error {e}", stack_info=ENABLE_STACK)
continue
if ret.ok:
break
else:
dlog.error(
f"request error status_code:{ret.status_code} reason: {ret.reason} body: \n{ret.text}"
)
time.sleep(retry_count)
ret = None
if ret is not None:
ret.raise_for_status()
with open(save_file, "wb") as f:
for chunk in ret.iter_content(chunk_size=8192):
f.write(chunk)
ret.close()

def _download_job(self, job):
job_url = self.api.get_job_result_url(job.job_id)
if not job_url:
return
job_hash = job.job_hash
result_filename = job_hash + "_back.zip"
target_result_zip = os.path.join(self.context.local_root, result_filename)
self.api.download_from_url(job_url, target_result_zip)
zip_file.unzip_file(target_result_zip, out_dir=self.context.local_root)

def _download_job(self, job):
data = self.job.detail(job.job_id)
job_url = data["resultUrl"]
if not job_url:
return
job_hash = job.job_hash
result_filename = job_hash + "_back.zip"
target_result_zip = os.path.join(self.context.local_root, result_filename)
self.storage.download_from_url(job_url, target_result_zip)
unzip_file(target_result_zip, out_dir=self.context.local_root)

Impact
A network or HTTP failure is reported later as FileNotFoundError or a zip parsing error, hiding the real remote failure and making retries/debugging harder.

Suggested fix
Raise a clear exception after retry exhaustion, including URL, status code, reason, and a short response body when available. Callers should stop before unzip if the download failed.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    Status
    In Progress

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions