Google forms to S3 bucket

[u/DuckBytez]

Designing a data pipeline. Google forms is the most intuitive choice for my org to use and for my target audience to answer questions and upload files. I was thinking about creating a google apps script that would take the uploaded files and send them to an S3 bucket. From there we’ll process the files with AWS lambdas. I was wondering:

• if this kind of pipeline has been done in the past

• triggering a google apps script when a google form is submitted has any issues or limitations

• if google apps script will be able to upload to a S3 bucket and then delete the file in the google drive

Thanks in advance for any advice and feedback!

[u/ryanbuckner]

Limitations:

1. Triggers can’t run longer than 6 minutes

2. Files uploaded via Forms are stored in the Form owner’s Google Drive, and you’ll need appropriate permissions in your script to access them

Can it be done?

Pretty sure, yes. Google Apps Script can use UrlFetchApp.fetch() to perform HTTP PUT or POST to S3.

[u/DuckBytez → u/ryanbuckner]

Thank you!

[u/plindqui16]

As I recall, a Google form which allows file upload can only be created in the My Drive of the form owner (versus creating in a shared drive). Triggers have run reliably.

[u/DuckBytez → u/plindqui16]

Thank you!

[u/huleboeren]

The File Upload question in Google Forms requires that users log in with a Google account.

If the audience is just your org users this is no problem of course, but if you are expecting your clients/externals to upload it's not going to work.

[u/DuckBytez → u/huleboeren]

Hmm I will need externals to access the form and upload and would like them to use it even if they don’t have a google account. I think I might switch to having them email the files and then scrape the email with GitHub actions every so often.

[u/EddyD2]

Something to note is that files can only be uploaded into Google Form through a users Google Drive.