Strengthen phorge robots.txt
Open, S3 LowPublic
Actions

Assigned To

Authored By

	revi
	Dec 31 2023, 09:28

Description

Setup for 2023-12-31~2024-04-16:

proxy https://phabricator.wikimedia.org/robots.txt + custom text via Cloudflare Workers

Setup as of 2024-04-16: Load extension via rSANDBOX/PhabExt

Possible reinforcement:

Pray to Upstream to implement proper robots.txt management
Host robots.txt file elsewhere and proxy it via cloudflare worker

1 is probably the way to go but too lazy to file a task.

Revisions and Commits

	Restricted Diffusion Commit
	Restricted Diffusion Commit
	Restricted Diffusion Commit
	Restricted Diffusion Commit
	Restricted Diffusion Commit
	Restricted Diffusion Commit
	Restricted Diffusion Commit
	Restricted Diffusion Commit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		revi	T43 Strengthen phorge robots.txt
		Resolved		revi	T126 Cloudflare does not cache robots.txt

Event Timeline

revi created this task.Dec 31 2023, 09:28

revi triaged this task as S3 Low priority.

revi added a project: Phabricator.Jan 2 2024, 18:40

Added…

# No ChatGPT
User-agent: ChatGPT-User
Disallow: /
User-agent: GPTBot
Disallow: /

revi updated the task description. (Show Details)Apr 14 2024, 20:39

revi edited projects, added: Cloudflare Workers, Upstream; removed: Cloudflare.Apr 14 2024, 20:50

cf. https://c4science.ch/source/phab/browse/stable/src/extensions/PhabricatorCustomRobotsController.php

Quasi-extension P49.

revi added a commit: Restricted Diffusion Commit.Apr 16 2024, 20:50

revi added a commit: Restricted Diffusion Commit.

>>> UNRECOVERABLE FATAL ERROR <<<

Class PhabricatorRobotsPlatformController may not inherit from final class (PhabricatorRobotsController)

/var/www/phorge/phorge/src/applications/system/controller/robots/PhabricatorRobotsPlatformController.php:0


┻━┻ ︵ ¯\_(ツ)_/¯ ︵ ┻━┻

revi added a commit: Restricted Diffusion Commit.Apr 16 2024, 20:56

revi added a commit: Restricted Diffusion Commit.Apr 16 2024, 20:58

revi added a commit: Restricted Diffusion Commit.

revi updated the task description. (Show Details)Apr 16 2024, 21:00

revi added a commit: Restricted Diffusion Commit.Apr 16 2024, 21:18

revi updated the task description. (Show Details)Apr 16 2024, 21:24

revi added a commit: Restricted Diffusion Commit.Apr 30 2024, 19:44

revi closed subtask T126: Cloudflare does not cache robots.txt as Resolved.May 2 2024, 20:02

revi added a commit: Restricted Diffusion Commit.Jul 4 2024, 18:55

revi edited projects, added: Milestones (2025~2029); removed: Milestones (2020~2024).Dec 31 2024, 06:08