Found by a Codex global repository scan of deepmodeling/dpdispatcher at commit 98a9e08.
Problem
Cloud download_from_url() logs failed HTTP attempts but returns normally if all retries fail and ret remains None. Callers then try to unzip a file that was never written.
Relevant code
|
def download_from_url(self, url, save_file): |
|
ret = None |
|
for retry_count in range(3): |
|
try: |
|
ret = requests.get( |
|
url, headers={"Authorization": "jwt " + self.token}, stream=True |
|
) |
|
except Exception as e: |
|
dlog.error(f"request error {e}", stack_info=ENABLE_STACK) |
|
continue |
|
if ret.ok: |
|
break |
|
else: |
|
dlog.error( |
|
f"request error status_code:{ret.status_code} reason: {ret.reason} body: \n{ret.text}" |
|
) |
|
time.sleep(retry_count) |
|
ret = None |
|
if ret is not None: |
|
ret.raise_for_status() |
|
with open(save_file, "wb") as f: |
|
for chunk in ret.iter_content(chunk_size=8192): |
|
f.write(chunk) |
|
ret.close() |
|
|
|
def _download_job(self, job): |
|
job_url = self.api.get_job_result_url(job.job_id) |
|
if not job_url: |
|
return |
|
job_hash = job.job_hash |
|
result_filename = job_hash + "_back.zip" |
|
target_result_zip = os.path.join(self.context.local_root, result_filename) |
|
self.api.download_from_url(job_url, target_result_zip) |
|
zip_file.unzip_file(target_result_zip, out_dir=self.context.local_root) |
|
def _download_job(self, job): |
|
data = self.job.detail(job.job_id) |
|
job_url = data["resultUrl"] |
|
if not job_url: |
|
return |
|
job_hash = job.job_hash |
|
result_filename = job_hash + "_back.zip" |
|
target_result_zip = os.path.join(self.context.local_root, result_filename) |
|
self.storage.download_from_url(job_url, target_result_zip) |
|
unzip_file(target_result_zip, out_dir=self.context.local_root) |
Impact
A network or HTTP failure is reported later as FileNotFoundError or a zip parsing error, hiding the real remote failure and making retries/debugging harder.
Suggested fix
Raise a clear exception after retry exhaustion, including URL, status code, reason, and a short response body when available. Callers should stop before unzip if the download failed.
Found by a Codex global repository scan of deepmodeling/dpdispatcher at commit 98a9e08.
Problem
Cloud
download_from_url()logs failed HTTP attempts but returns normally if all retries fail andretremainsNone. Callers then try to unzip a file that was never written.Relevant code
dpdispatcher/dpdispatcher/utils/dpcloudserver/client.py
Lines 156 to 180 in 98a9e08
dpdispatcher/dpdispatcher/machines/dp_cloud_server.py
Lines 220 to 228 in 98a9e08
dpdispatcher/dpdispatcher/machines/openapi.py
Lines 197 to 206 in 98a9e08
Impact
A network or HTTP failure is reported later as
FileNotFoundErroror a zip parsing error, hiding the real remote failure and making retries/debugging harder.Suggested fix
Raise a clear exception after retry exhaustion, including URL, status code, reason, and a short response body when available. Callers should stop before unzip if the download failed.