cohost2json: transient json errors in /api/v1/project/*/posts

[nex3]

2024-12-30T00:24:23.298003Z  INFO archive_cohost_project{project_name="bethposting" archive_likes=false}: autost::command::cohost2json: GET https://cohost.org/api/v1/project/bethposting/posts?page=10
Error: 
   0: error decoding response body
   1: expected value at line 2 column 1

Location:
   /home/runner/work/autost/autost/src/command/cohost2json.rs:111

Backtrace omitted.
Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.
Warning: SpanTrace capture is Unsupported.
Ensure that you've setup a tracing-error ErrorLayer and the semver versions are compatible

This seems transient, I was able to restart archiving after seeing it and it got through page 10 of bethposting's posts, but it would be nice if there were error recovery for it.

[nex3]

Oh no, it happened on page 11 this time 😰

[delan]

i’ll see what i can do :)

[delan]

@nex3, how about now? i’ve added some retry logic on the main branch. if you’re still getting stuck, we may be able to make autost cohost-archive log an error, move onto the next project, then list the failed projects at the very end.

[nex3]

Unfortunately I don't have an easy way of compiling from source on the machine I'm running this on...

[delan]

which platform/artifact did you need?

[nex3]

Linux GNU

[delan]

binaries hopefully here in ~10 minutes: https://github.com/delan/autost/actions/runs/12553798980

[nex3]

ty!

[docprofsky]

I've encountered a similar issue (a [likely] transient error causing autost cohost-archive to terminate), however this happened while downloading an attachment (error output below). I encountered this while running 1.3.1. I have switched to f9a7a73 and am retrying, though I don't know if the retry logic applies to downloading attachments.

2024-12-31T04:37:31.679127Z  INFO archive_cohost_project{project_name="pendell" archive_likes=false}:convert_chost{entry=DirEntry("chosts/3226762.json")}: autost::command::cohost2autost: writing: RelativePath { inner: "posts/3226762.html", kind: Post { is_markdown: false, in_imported_dir: false } }
Error:
   0: "chosts/1432379.json": failed to convert
   1: error sending request for url (https://cohost.org/rc/attachment-redirect/48c76f34-950d-4cb5-91ae-5d8ed42e99bd)
   2: operation timed out

Location:
   src/attachments.rs:243

Backtrace omitted.
Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.
Warning: SpanTrace capture is Unsupported.
Ensure that you've setup a tracing-error ErrorLayer and the semver versions are compatible

[delan]

I've encountered a similar issue (a [likely] transient error causing autost cohost-archive to terminate), however this happened while downloading an attachment (error output below). I encountered this while running 1.3.1. I have switched to f9a7a73 and am retrying, though I don't know if the retry logic applies to downloading attachments.

@docprofsky, i’ve fixed this in cfab1e8 + ecb074c. try building ecb074c, or grab the binaries from here in a few minutes: https://github.com/delan/autost/actions/runs/12556547445

[docprofsky]

Thank you! It appears to be working.

2024-12-31T07:19:26.021289Z  WARN archive_cohost_project{project_name="nullpat" archive_likes=false}:convert_chost{entry=DirEntry("chosts/3828632.json")}:cache_cohost_resource{cacheable=Attachment { id: "c1c45cbd-04ac-4166-a078-80ad4e3e42
1e", url: None }}: autost::attachments: retrying failed request wait=4s url="https://cohost.org/rc/attachment-redirect/c1c45cbd-04ac-4166-a078-80ad4e3e421e" status=406