Dataflow AttributeError: module google.cloud has no attribute storage

Today in this article, we will cover below aspects,

Issue Description

Google dataflow runner gives below error when running the command,

File “/usr/local/lib/python3.7/site-packages/dill/_dill.py”, line 827, in _import_module return getattr(__import__(module, None, None, [obj]), obj)AttributeError: module ‘google.cloud’ has no attribute ‘storage’

I had below sample code which makes use of google dataflow storage object references,

def __init__(self, input_path):
    self.input_path = input_path

def start_bundle(self):
    self.client = storage.Client()

Resolution

I was able to resolve the issue by using requirements.txt and running the dataflow commands.

I added the required packages called “google-cloud-storage”.

There could be multiple other issues causing this incompatibility. Please make sure to install the required packages using possible approaches.

The issue can also be resolved using Setup.py for dependencies.

Here is an example of requirements.txt file,

google-cloud-storage==1.28.1
pandas==1.2.2
fsspec==0.8.5

Here is an example of setup.py file,

setup.py

import setuptools

REQUIRED_PACKAGES = [
    'google-cloud-storage==1.35.0',
    'pandas==1.2.2',
    'gcsfs==0.7.1'
]

PACKAGE_NAME = 'TheCodebuzzDataflow'
PACKAGE_VERSION = '0.0.1'

setuptools.setup(
    name=PACKAGE_NAME,
    version=PACKAGE_VERSION,
    description='Set up file for the required Dataflow packages ',
    install_requires=REQUIRED_PACKAGES,
    packages=setuptools.find_packages(),
)

If you get any issue even with the above requirements.txt fix,

• Alternatively closing and opening the project again could be helpful.

References:

• Convert JSON to CSV using Google Data Flow pipeline

• Schedule DataFlow Job with Google Cloud Scheduler

Did I miss anything else in these resolution steps?

Did the above steps resolve your issue? Please sound off your comments below!

Happy Coding !!

Please bookmark this page and share it with your friends. Please Subscribe to the blog to receive notifications on freshly published(2024) best practices and guidelines for software design and development.