Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Throw S3FileStorageException when something goes wrong and response is not as expected. #305

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 18 additions & 8 deletions src/Foundatio.AWS/Storage/S3FileStorage.cs
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
using Amazon.S3.Model;
using Amazon.S3.Util;
using Foundatio.AWS.Extensions;
using Foundatio.AWS.Storage;
using Foundatio.Extensions;
using Foundatio.Serializer;
using Microsoft.Extensions.Logging;
Expand Down Expand Up @@ -54,7 +55,7 @@ public S3FileStorage(Builder<S3FileStorageOptionsBuilder, S3FileStorageOptions>
: this(builder(new S3FileStorageOptionsBuilder()).Build()) { }

ISerializer IHaveSerializer.Serializer => _serializer;

public AmazonS3Client Client => _client;
public string Bucket => _bucket;
public S3CannedACL CannedACL => _cannedAcl;
Expand All @@ -69,9 +70,13 @@ public S3FileStorage(Builder<S3FileStorageOptionsBuilder, S3FileStorageOptions>
};

var res = await _client.GetObjectAsync(req, cancellationToken).AnyContext();
if (!res.HttpStatusCode.IsSuccessful())

if (res.HttpStatusCode == System.Net.HttpStatusCode.NotFound)
return null;

if (res.HttpStatusCode != System.Net.HttpStatusCode.OK)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are probably going to want to think about some sort of retry mechanism for things that can be retried. I guess it doesn't need to be part of this change though.

throw new S3FileStorageException($"Invalid status code, expected 200 OK, found {(int)res.HttpStatusCode} {res.HttpStatusCode}.");

return new ActionableStream(res.ResponseStream, () => {
res?.Dispose();
});
Expand All @@ -89,17 +94,22 @@ public async Task<FileSpec> GetFileInfoAsync(string path) {
try {
var res = await _client.GetObjectMetadataAsync(req).AnyContext();

if (!res.HttpStatusCode.IsSuccessful())
if (res.HttpStatusCode == System.Net.HttpStatusCode.NotFound)
return null;

if (res.HttpStatusCode != System.Net.HttpStatusCode.OK)
throw new S3FileStorageException($"Invalid status code, expected 200 OK, found {(int)res.HttpStatusCode} {res.HttpStatusCode}.");
frabe1579 marked this conversation as resolved.
Show resolved Hide resolved

return new FileSpec {
Size = res.ContentLength,
Created = res.LastModified.ToUniversalTime(), // TODO: Need to fix this
Modified = res.LastModified.ToUniversalTime(),
Path = path
};
} catch (AmazonS3Exception) {
return null;
} catch (AmazonS3Exception ex) {
if (ex.StatusCode == System.Net.HttpStatusCode.NotFound)
return null;
throw new S3FileStorageException("Error accessing S3 storage: " + ex.Message, ex);
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be nice to use exception filters here

}

Expand Down Expand Up @@ -209,7 +219,7 @@ public async Task<bool> ExistsAsync(string path) {
continue;

deleteRequest.Objects.AddRange(keys);

var deleteResponse = await _client.DeleteObjectsAsync(deleteRequest, cancellationToken).AnyContext();
if (deleteResponse.DeleteErrors.Count > 0) {
// retry 1 time, continue on.
Expand Down Expand Up @@ -259,9 +269,9 @@ private async Task<NextPageResult> GetFiles(SearchCriteria criteria, int pageSiz
Files = response.S3Objects.MatchesPattern(criteria.Pattern).Select(blob => blob.ToFileInfo()).ToList(),
NextPageFunc = response.IsTruncated ? r => GetFiles(criteria, pageSize, cancellationToken, response.NextContinuationToken) : (Func<PagedFileListResult, Task<NextPageResult>>)null
};
}
}

private class SearchCriteria {
private class SearchCriteria {
public string Prefix { get; set; }
public Regex Pattern { get; set; }
}
Expand Down
16 changes: 16 additions & 0 deletions src/Foundatio.AWS/Storage/S3FileStorageException.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
using System;
using System.Collections.Generic;
using System.Text;

namespace Foundatio.AWS.Storage {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We definitely don't want to have exceptions that are unique to each implementation where a user would need to check for different kinds of exceptions depending on which implementation they are using.

public class S3FileStorageException : Exception {
public S3FileStorageException() : base() {
}

public S3FileStorageException(string message) : base(message) {
}

public S3FileStorageException(string message, Exception innerException) : base(message, innerException) {
}
}
}