upload pictures to aws and return link api
Then yous're building a REST API and you lot need to add support for uploading files from a web or mobile app. You also need to add a reference to these uploaded files against entities in your database, along with metadata supplied by the customer.
In this article, I'll testify yous how to do this using AWS API Gateway, Lambda and S3. We'll use the example of an event management web app where attendees tin can login and upload photos associated with a specific consequence along with a title and description. We will utilize S3 to shop the photos and an API Gateway API to handle the upload request. The requirements are:
- User can login to the app and view a list of photos for a specific event, along with each photo'due south metadata (date, title, clarification, etc).
- User can only upload photos for the effect if they are registered as having attended that event.
- Use Infrastructure-equally-Code for all deject resources to brand it easy to whorl this out to multiple environments. (No using the AWS Console for mutable operations here š«š¤ )
Considering implementation options
Having built similar functionality in the by using not-serverless technologies (e.g. in Express.js), my initial approach was to investigate how to utilise a Lambda-backed API Gateway endpoint that would handle everything: authentication, potency, file upload and finally writing the S3 location and metadata to the database. While this approach is valid and achievable, it does have a few limitations:
- You need to write lawmaking inside your Lambda to manage the multipart file upload and the edge cases effectually this, whereas the existing S3 SDKs are already optimized for this.
- Lambda pricing is elapsing-based so for larger files your function volition have longer to complete, costing you lot more.
- API Gateway has a payload size hard limit of 10MB. Contrast that to the S3 file size limit of 5GB.
Using S3 presigned URLs for upload
After farther research, I institute a better solution involving uploading objects to S3 using presigned URLs as a means of both providing a pre-upload authorization check and also pre-tagging the uploaded photo with structured metadata.
The diagram below shows the request menstruation from a web app.
The main thing to notice is that from the web client's indicate of view, it's a 2-step process:
- Initiate the upload request, sending metadata related to the photo (e.g. eventId, title, description, etc). The API then does an auth check, executes concern logic (e.g. restricting access merely to users who have attended the event) and finally generates and responds with a secure presigned URL.
- Upload the file itself using the presigned URL.
I'm using Cognito as my user store here but you could easily swap this out for a custom Lambda Authorizer if your API uses a dissimilar auth machinery.
Let'southward dive in…
Step i: Create the S3 bucket
I use the Serverless Framework to manage configuration and deployment of all my cloud resource. For this app, I use ii separate "services" (or stacks), that can be independently deployed:
-
infraservice: this contains the S3 bucket, CloudFront distribution, DynamoDB table and Cognito User Pool resources. -
photos-apiservice: this contains the API Gateway and Lambda functions.
You tin view the full configuration of each stack in the Github repo, but we'll cover the key points below.
The S3 bucket is defined as follows:
resources : Resource : PhotosBucket : Type : AWS: :S3: :Saucepan Backdrop : BucketName : !Sub '${self:custom.photosBucketName}' AccessControl : Private CorsConfiguration : CorsRules : - AllowedHeaders : ['*' ] AllowedMethods : ['PUT'] AllowedOrigins : ['*' ] The CORS configuration is important here every bit without it your web client won't exist able to perform the PUT asking after acquiring the signed URL. I'k likewise using CloudFront as the CDN in guild to minimize latency for users downloading the photos. You can view the config for the CloudFront distribution here. However, this is an optional component and if you'd rather clients read photos directly from S3 then y'all tin alter the AccessControl property in a higher place to be PublicRead.
Step 2: Create "Initiate Upload" API Gateway endpoint
Our side by side step is to add a new API path that the customer endpoint tin can phone call to asking the signed URL. Requests to this volition look like so:
Mail /events/{eventId}/photos/initiate-upload { "title": "Keynote Speech", "description": "Steve walking out on phase", "contentType": "prototype/png" } Responses will comprise an object with a single s3PutObjectUrl field that the customer can use to upload to S3. This URL looks like and then:
https://s3.eu-west-1.amazonaws.com/eventsapp-photos-dev.sampleapps.winterwindsoftware.com/uploads/event_1234/1d80868b-b05b-4ac7-ae52-bdb2dfb9b637.png?AWSAccessKeyId=XXXXXXXXXXXXXXX&Enshroud-Control=max-age%3D31557600&Content-Type=image%2Fpng&Expires=1571396945&Signature=F5eRZQOgJyxSdsAS9ukeMoFGPEA%3D&ten-amz-meta-contenttype=image%2Fpng&x-amz-meta-description=Steve%20walking%20out%20on%20stage&x-amz-meta-eventid=1234&10-amz-meta-photoid=1d80868b-b05b-4ac7-ae52-bdb2dfb9b637&x-amz-meta-title=Keynote%20Speech&x-amz-security-token=XXXXXXXXXX
Notice in detail these fields embedded in the query cord:
-
10-amz-meta-30— These fields contain the metadata values that ourinitiateUploadLambda office will ready. -
x-amz-security-token— this contains the temporary security token used for authenticating with S3 -
Signature— this ensures that the PUT asking cannot be altered past the client (due east.g. by irresolute metadata values)
The following extract from serverless.yml shows the part configuration:
# serverless.yml service : eventsapp-photos-api … custom : appName : eventsapp infraStack : ${self:custom.appName} -infra-${cocky:provider.stage} awsAccountId : ${cf:${self:custom.infraStack}.AWSAccountId} apiAuthorizer : arn : arn:aws:cognito-idp:${self:provider.region} :${self:custom.awsAccountId} :userpool/${cf:${self:custom.infraStack}.UserPoolId} corsConfig : truthful functions : … httpInitiateUpload : handler : src/http/initiate-upload.handler iamRoleStatements : - Effect : Let Activeness : - s3:PutObject Resources : arn:aws:s3: : :${cf:${self:custom.infraStack}.PhotosBucket}* events : - http : path : events/{eventId}/photos/initiate-upload method : post authorizer : ${self:custom.apiAuthorizer} cors : ${cocky:custom.corsConfig} A few things to note hither:
- The
httpInitiateUploadLambda function will handle Post requests to the specified path. - The Cognito user pool (output from the
infrastack) is referenced in the office'sauthorizerbelongings. This makes sure requests without a valid token in theAuthorizationHTTP header are rejected by API Gateway. - CORS is enabled for all API endpoints
- Finally, the
iamRoleStatementsproperty creates an IAM role that this function will run as. This role allowsPutObjectactions against the S3 photos bucket. It is especially of import that this permission set follows the to the lowest degree privilege principle as the signed URL returned to the client contains a temporary access token that allows the token holder to assume all the permissions of the IAM role that generated the signed URL.
Now let's await at the handler code:
import S3 from 'aws-sdk/clients/s3' ; import uuid from 'uuid/v4' ; import { InitiateEventPhotoUploadResponse, PhotoMetadata } from '@common/schemas/photos-api' ; import { isValidImageContentType, getSupportedContentTypes, getFileSuffixForContentType } from '@svc-utils/prototype-mime-types' ; import { s3 as s3Config } from '@svc-config' ; import { wrap } from '@common/middleware/apigw' ; import { StatusCodeError } from '@common/utils/errors' ; const s3 = new S3 ( ) ; export const handler = wrap ( async (outcome) => { // Read metadata from path/body and validate const eventId = result.pathParameters! .eventId; const body = JSON . parse (event.body || '{}' ) ; const photoMetadata: PhotoMetadata = { contentType: body.contentType, title: body.championship, clarification: torso.clarification, } ; if ( ! isValidImageContentType (photoMetadata.contentType) ) { throw new StatusCodeError ( 400 , ` Invalid contentType for epitome. Valid values are: ${ getSupportedContentTypes ( ) . join ( ',' ) } ` ) ; } // TODO: Add whatever further business logic validation here (e.g. that electric current user has write admission to eventId) // Create the PutObjectRequest that will be embedded in the signed URL const photoId = uuid ( ) ; const req: S3 .Types.PutObjectRequest = { Saucepan: s3Config.photosBucket, Key: ` uploads/event_ ${eventId} / ${photoId} . ${ getFileSuffixForContentType (photoMetadata.contentType) ! } ` , ContentType: photoMetadata.contentType, CacheControl: 'max-age=31557600' , // instructs CloudFront to cache for ane yr // Set Metadata fields to exist retrieved post-upload and stored in DynamoDB Metadata: { ... (photoMetadata as whatsoever ) , photoId, eventId, } , } ; // Get the signed URL from S3 and return to client const s3PutObjectUrl = wait s3. getSignedUrlPromise ( 'putObject' , req) ; const result: InitiateEventPhotoUploadResponse = { photoId, s3PutObjectUrl, } ; return { statusCode: 201 , body: JSON . stringify (consequence) , } ; } ) ; The s3.getSignedUrlPromise is the main line of interest hither. Information technology serializes a PutObject request into a signed URL.
I'm using a wrap middleware role in order to handle cross-cutting API concerns such every bit adding CORS headers and uncaught error logging.
Step iii: Uploading file from the web app
Now to implement the client logic. I've created a very basic (read: ugly) create-react-app example (code here). I used Amplify's Auth library to manage the Cognito authentication and then created a PhotoUploader React component which makes apply of the React Dropzone library:
// components/Photos/PhotoUploader.tsx import React, { useCallback } from 'react' ; import { useDropzone } from 'react-dropzone' ; import { uploadPhoto } from '../../utils/photos-api-customer' ; const PhotoUploader: React. FC < { eventId: cord } > = ( { eventId } ) => { const onDrop = useCallback ( async ( files: File[ ] ) => { panel . log ( 'starting upload' , { files } ) ; const file = files[ 0 ] ; attempt { const uploadResult = expect uploadPhoto (eventId, file, { // should enhance this to read title and description from text input fields. championship: 'my title' , clarification: 'my description' , contentType: file. type , } ) ; console . log ( 'upload complete!' , uploadResult) ; render uploadResult; } grab (mistake) { panel . error ( 'Error uploading' , error) ; throw fault; } } , [eventId] ) ; const { getRootProps, getInputProps, isDragActive } = useDropzone ( { onDrop } ) ; return ( <div { ... getRootProps ( ) } > <input { ... getInputProps ( ) } / > { isDragActive ? <p > Drop the files hither ... </p > : <p > Drag and drop some files here, or click to select files </p > } </div > ) ; } ; export default PhotoUploader; // utils/photos-api-client.ts import { API , Auth } from 'aws-amplify' ; import axios, { AxiosResponse } from 'axios' ; import config from '../config' ; import { PhotoMetadata, InitiateEventPhotoUploadResponse, EventPhoto } from '../../../../services/mutual/schemas/photos-api' ; API . configure (config.amplify. API ) ; const API_NAME = 'PhotosAPI' ; async role getHeaders ( ) : Promise < whatever > { // Set auth token headers to be passed in all API requests const headers: any = { } ; const session = await Auth. currentSession ( ) ; if (session) { headers.Authorization = ` ${session. getIdToken ( ) . getJwtToken ( ) } ` ; } return headers; } export async function getPhotos ( eventId: cord ) : Promise <EventPhoto[ ] > { return API . go ( API_NAME , ` /events/ ${eventId} /photos ` , { headers: await getHeaders ( ) } ) ; } export async role uploadPhoto ( eventId: string, photoFile: any, metadata: PhotoMetadata, ) : Promise <AxiosResponse> { const initiateResult: InitiateEventPhotoUploadResponse = await API . post ( API_NAME , ` /events/ ${eventId} /photos/initiate-upload ` , { body: metadata, headers: look getHeaders ( ) } , ) ; return axios. put (initiateResult.s3PutObjectUrl, photoFile, { headers: { 'Content-Type' : metadata.contentType, } , } ) ; } The uploadPhoto part in the photos-api-client.ts file is the key hither. It performs the 2-step process we mentioned before by first calling our initiate-upload API Gateway endpoint and so making a PUT asking to the s3PutObjectUrl information technology returned. Make certain that you prepare the Content-Type header in your S3 put request, otherwise information technology will be rejected as not matching the signature.
Step iv: Pushing photo information into database
Now that the photo has been uploaded, the web app will need a way of listing all photos uploaded for an event (using the getPhotos function in a higher place).
To close this loop and make this query possible, we need to tape the photo data in our database. We do this by creating a second Lambda function processUploadedPhoto that is triggered whenever a new object is added to our S3 bucket.
Let's look at its config:
# serverless.yml service : eventsapp-photos-api … functions : … s3ProcessUploadedPhoto : handler : src/s3/process-uploaded-photograph.handler iamRoleStatements : - Outcome : Allow Action : - dynamodb:Query - dynamodb:Scan - dynamodb:GetItem - dynamodb:PutItem - dynamodb:UpdateItem Resources : arn:aws:dynamodb:${self:provider.region} :${self:custom.awsAccountId} :tabular array/${cf:${cocky:custom.infraStack}.DynamoDBTablePrefix}* - Effect : Allow Action : - s3:GetObject - s3:HeadObject Resources : arn:aws:s3: : :${cf:${self:custom.infraStack}.PhotosBucket}* events : - s3 : saucepan : ${cf:${self:custom.infraStack}.PhotosBucket} result : s3:ObjectCreated:* rules : - prefix : uploads/ existing : true It'southward triggered off the s3:ObjectCreated event and volition merely burn down for files added beneath the uploads/ meridian-level folder. In the iamRoleStatements section, we are assuasive the role to write to our DynamoDB table and read from the S3 Bucket.
Now let's expect at the function lawmaking:
import { S3Event } from 'aws-lambda' ; import S3 from 'aws-sdk/clients/s3' ; import log from '@common/utils/log' ; import { EventPhotoCreate } from '@common/schemas/photos-api' ; import { cloudfront } from '@svc-config' ; import { savePhoto } from '@svc-models/event-photos' ; const s3 = new S3 ( ) ; consign const handler = async (event: S3Event) : Promise < void > => { const s3Record = event.Records[ 0 ] .s3; // Starting time fetch metadata from S3 const s3Object = await s3. headObject ( { Saucepan: s3Record.saucepan.name, Key: s3Record.object.cardinal } ) . promise ( ) ; if ( !s3Object.Metadata) { // Shouldn't get here const errorMessage = 'Cannot procedure photograph as no metadata is set for information technology' ; log. mistake (errorMessage, { s3Object, event } ) ; throw new Error (errorMessage) ; } // S3 metadata field names are converted to lowercase, so need to map them out carefully const photoDetails: EventPhotoCreate = { eventId: s3Object.Metadata.eventid, description: s3Object.Metadata.description, title: s3Object.Metadata.title, id: s3Object.Metadata.photoid, contentType: s3Object.Metadata.contenttype, // Map the S3 bucket key to a CloudFront URL to be stored in the DB url: ` https:// ${cloudfront.photosDistributionDomainName} / ${s3Record.object.primal} ` , } ; // Now write to DDB await savePhoto (photoDetails) ; } ; The event object passed to the Lambda handler function only contains the bucket name and key of the object that triggered it. And then in order to fetch the metadata, we demand to utilize the headObject S3 API call. Once we've extracted the required metadata fields, we and so construct a CloudFront URL for the photo (using the CloudFront distribution's domain name passed in via an surround variable) and save to DynamoDB.
Futurity enhancements
A potential enhancement that could be made to the upload flow is to add in an image optimization step earlier saving it to the database. This would involve a having a Lambda function mind for S3:ObjectCreated events beneath the upload/ key prefix which then reads the prototype file, resizes and optimizes information technology accordingly so saves the new re-create to the aforementioned saucepan simply under a new optimized/ key prefix. The config of our Lambda role that saves to the database should then exist updated to be triggered off this new prefix instead.
Other manufactures yous might bask:
Source: https://serverlessfirst.com/serverless-photo-upload-api/
0 Response to "upload pictures to aws and return link api"
Post a Comment