Django Community: New/updated Django packages feed ingestion is broken

[jefftriplett]

The latest items from https://www.djangoproject.com/community/packages/ are from October 27, 2017.

The DjangoPackages.org RSS appears to be working, but I'm happy to follow up if something is broken on the website: https://djangopackages.org/feeds/packages/latest/rss/

I'm happy to help if someone can point me to where that should be :)

related to #1137

[carltongibson]

Hey @jefftriplett — yes! I think this is likely related to #1137.

Everything that follows is I think…

The issue is that the superfeedr service that we've been using forever is now unmaintained and it slowly rotting.

We need to replace it with our own. feedparser mapping entries into the FeedItem model, in a management command, that we can run 6hrly (say).

This shouldn't be too ops intensive, such we can't do it ourselves. But if it is I'm minded to spin up a mini-instance myself just so we can get passed this.

The relevant model is in the aggregator app.

If you break ground here, I'm very happy to input as needed. Otherwise I have it down as a possible GSoC idea (but that's still time away)

[jefftriplett]

@carltongibson circling back here. Do we have a dump of feed urls by chance to test with? (I have admin access, but so-many-feeds)

[jefftriplett]

This project might get pulled into Jazzband and might be semi-related. https://github.com/brutasse/django-push

[medmunds]

Now the new/updated packages feed is showing mostly gambling site spam. The last actual Django package updates listed are from 2017.

Maybe just remove it until it can be repaired?

[jefftriplett]

cc @bmispelon since we were both looking into this on Friday. The Django Package feed has newer links, but ingestion isn't picking up on it. I thought we'd removed the other feed that someone must have bought and squatted on.

[bmispelon]

I did remove some spam entries on Friday, and I don't see them anymore on the page at https://www.djangoproject.com/community/packages/

@medmunds Where do you see spam entries?

EDIT: Ah nevermind, I found them in the actual RSS feed. The cause seems to be that the FeedListView and the CommunityAggregatorFeed use different logic. I'll open a new issue for that.

[medmunds]

(Just for reference, what I am seeing is in the page at https://www.djangoproject.com/community/)

[jefftriplett]

I see a new update from August 4th, 2024 this year which ~7 years in between updates. I think there is a chance this is fixed but I will leave this open until we verify it or see a few more trickle in. https://www.djangoproject.com/community/packages/

[nanuxbe]

On Oct 5th, this still seems to be working fine:

[carltongibson]

@nanuxbe the issue here was with adding new feeds. Existing feeds (like Django Packages) have always kept working.

Not sure if it's still the case or not that new feeds can't be OK'd.

[jefftriplett]

@carltongibson I just approved a "pending" feed to test it. Previously, it 500'ed, but it didn't this time.

If you see Django News Jobs show up the next time the "Django jobs" feeds gets refreshed, then I would call it fixed and close the issue: https://www.djangoproject.com/community/

[carltongibson]

@jefftriplett cool. Let's look into it this week. It'll be good if we can add new feeds. (So many new folks posting...)

Still think we need a replacement medium term... 😉

[jefftriplett]

I'm seeing Jobs flowing through the community jobs feed which is a positive sign that things are working again: https://www.djangoproject.com/community/jobs/

Thank you everyone for looking!