Scanning GovCloud for all Services Ends in JSON Serialization Error, Truncated Output

[olivereri]

• Version: 1.1.1
• Local OS Version: Ubuntu 20.04.5 LTS
• Local chip architecture: x86_64
• Reproduces in: 432896750513

Steps to Reproduce:

1. python3 scan.py -s scan/sample/all_services.json -r us-gov-west-1

output:

INFO:__main__:Finished processing for region: us-gov-west-1
ERROR:__main__:'us-gov-west-1' generated an exception: Object of type bytes is not JSON serializable
ERROR:__main__:Traceback (most recent call last):
  File "scan.py", line 334, in main
    json.dump(service_result["result"], f, cls=DateTimeEncoder)
  File "/usr/lib/python3.8/json/__init__.py", line 179, in dump
    for chunk in iterable:
  File "/usr/lib/python3.8/json/encoder.py", line 431, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "/usr/lib/python3.8/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/usr/lib/python3.8/json/encoder.py", line 325, in _iterencode_list
    yield from chunks
  File "/usr/lib/python3.8/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/usr/lib/python3.8/json/encoder.py", line 438, in _iterencode
    o = _default(o)
  File "scan.py", line 44, in default
    return super().default(o)
  File "/usr/lib/python3.8/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type bytes is not JSON serializable

Total elapsed time for scanning: 0h:31m:1s

[olivereri]

For context when this exception is encountered the script is trying to write the output for the below service and function.

  {
    "function": "list_public_keys",
    "service": "cloudtrail"
  },

Output of cloudtrail-list_public_keys.json:

{"PublicKeyList": [{"Value":

As soon as it begins to write the value it throws and exception.

I added a breakpoint after line 325 to inspect what's going on when service information is written to the file system.

This is found in the output of region_results:

{'region': 'us-    gov-west-1', 'service': 'cloudtrail', 'function': 'list_public_keys', 'result': {'PublicKeyList': [{'Value': b'0\x82\x01\n\x02\x82\x0    1\x01\x00\xe9\x85\tr\xed\x15R_\x0f\x17q#\x02 R(\x01l\x8b\x9d"\xeb\xe2\x01\xe9}xF\xcc2\xb0u\xfb\x11\xe8\xdd\xa6\xdc\xb8\xba\xc4\xc6\xb    aw\xb5\xe9\xd2\x0f4\xf0\xff`TR\xd9=\x8b\x02Ub\xa1\xeb\xc6\x05..\x94\xfdN\xa9\x02\xd9\xaf\x0b\xc5b\x12\xa1\x06Y*\x00\xed"\\I\xde\x8f\x    d8, ,\xcd\xcc\xce(QA\xb2B\\BQ\xda\xf7\xa3\x85=/8\xfd\xbf\xfc\xd0w\x91*\xe3T\xfb~\x1cgU\xc6I\xa4\xb3\xc6\x15/\xf1:\xb5\xfc\xf0Xp\xee\x    1b\x88*\xe5\xce\xa4\xc4\xc6\xc7\x10Cu\x04F\x89\r)K\x8c\xf4\xd2\xc3\x96\'$\x95\x82^8\xf2\x1d\x0cn\xcfvF\xeauY\x12\xfd\x8b%\x9f4\xa8\xb    3\x955\xe1\xc5\xde`\x0c5R\x82T\xf6\xc8\xdf\xb6\x9c\x93K,{\xef;\xa2\xf0\x9fr\x022T\x9ekf~\x84\xc3`-\xd3\xe75"S\x052\xbb\xc3u+Y\xbc\x9b    \x1bWo\x94\xf4\xbe`\x83\xb45T\xc1\x08\xec\x06`\xe1F\xd9\x02\x03\x01\x00\x01',  ...

I'll investigate whether script isn't handling byte return types, or isn't handling it correctly.

[olivereri]

According to the boto3 documentation CloudTrail.Client.list_public_keys(**kwargs) returns a value of bytes:

{
    'PublicKeyList': [
        {
            'Value': b'bytes',
            'ValidityStartTime': datetime(2015, 1, 1),
            'ValidityEndTime': datetime(2015, 1, 1),
            'Fingerprint': 'string'
        },
    ],
    'NextToken': 'string'
}

Also the default JSON Encoder does not support bytes:
https://docs.python.org/3/library/json.html#json.JSONEncoder

[Carlos4ndresh]

This is happening to me as well, and outside of GovCloud regions.

[valter-silva-au]

I see that there's an issue with the current implementation of AWS Auto Inventory where it fails to handle byte data returned by some AWS API calls, specifically the CloudTrail.Client.list_public_keys() function. The error occurs because the default JSON encoder cannot serialize bytes objects.

I will work on a fix.