PHP to Python plus tests and stuff

[fabianegli]

This PR changes the behaviour:

• ports php code to python -> no more PHP code, one language left
• adds test for the json to txt and md conversion
• removes the action for the php script - everything is now updated at once with the Python script
• adds the action trigger push to the remaining action
• adds some Python specific files and folders to .gitignore

and some file contents slightly:

• removes the space at end of some lines in md table
• adds a newline at end of robots.txt file

The python files have been formatted with black.

[glyn]

Thanks for this. I like the idea of centralising on a single coding language. We don't know if anyone will maintain the code, but at least we'll only need Python skills, which are more common than PHP skills. Also the addition of tests is really appreciated.

I've suggested various improvements. You may regard some of the larger ones as out of scope for this PR, in which case, please log an issue for each of the ones you choose not to address so these can be worked on in future. Alternatively, if you don't mind large PRs, please address all the comments here - your choice.

[glyn]

@cdransf PTAL as this is a substantial PR.

[fabianegli]

Thanks for the review @glyn

[fabianegli]

Why are these configs made here? And should they be changed for the conversion step?

[glyn]

I don't know (x2).

[glyn]

Actually, the purpose of these configs is to make the changes produced by the dark visitors website look as if they were produced by a dark-visitors user. I'm not sure this is entirely necessary. The commit message should be sufficient on its own.

[glyn]

@fabianegli Thanks for the rework. A couple of remaining items for this PR:

1. rationalise the git config user name/email so that updates are not wrongly attributed to dark-visitors. (I haven't worked through to know whether this has already been done, but please take a look.)
2. the tests need to be run each time an action is triggered. Have I missed where that happens?

[glyn]

@fabianegli To expand on remaining item 1: please make sure that updates due to dark visitors changes have an appropriate commit message (and user/email if you decide to keep that) and that updates due to merging a local change also have an appropriate commit message (and user/email). In other words, we need to make sure we differentiate between dark visitors and local changes, especially in the situation where a local change is merged and, coincidentally, dark visitors has changed. If we are not careful, the current code will pick up both changes and put them in a single commit and give that commit an incorrect commit message (and user/email).

It may be necessary to step back from the approach in the current PR and keep the processing of dark visitors separate from the merging of local changes. It should still be possible to (a) replace the PHP by Python and (b) introduce tests of the Python functions.

Hope that helps make the requirements clearer.

[newbold]

Just gonna “Leeroy Jenkins” this and we’ll see what happens!

[newbold]

It looks like the action ran without any issue. Not sure who can see this, but here’s the run that was triggered via the merge: https://github.com/ai-robots-txt/ai.robots.txt/actions/runs/11471373718

No changes to robots.json or robots.txt, though, but I can’t tell if that’s because there truly aren’t any new user agents to add or if something else happened.

[glyn]

Just gonna “Leeroy Jenkins” this and we’ll see what happens!

Will all changes now be attributed to dark visitors?

[fabianegli]

I think there was indeed a change in the attribution logic. PR #58 reverts that.

[fabianegli]

I am sorry it took me so long to come back to this and indeed this probably shouldn't have been merged in this way. FYI that's why I marked it to be a draft before I had to turn my attention to other things for a while.