Looooong lag for back office controls to load - v3.11.5

[PeteDuncanson]

Noticed in the back office the network would hang for a really long time and then would start downloading again. Seems it was this request:

/umbraco/UrlTracker/InfoCaster.Umbraco.UrlTracker.UI.UrlTrackerManager.aspx?culture=en-GB&uiculture=en-GB

Hangs from anything from 14-20 seconds before returning! Not sure what its doing, going to do some digging now.

[PeteDuncanson]

Seems the control is loading straight away in the back office. Ideally it should lazy load itself when visible. I'm not sure how to actually know if a control is "visible" or not at the minute but I've created an issue for it on the Umbraco issues tracker:

http://issues.umbraco.org/issue/U4-11067

Also this is the source file that is being called, if you open it in a new browser (using the url in the original issue description) you can see it loads the full grid so lots of SQL I imagine running and blocking the download.

https://github.com/kipusoep/UrlTracker/blob/master/UI/UrlTrackerManager.aspx

[kipusoep]

Yes you're right, clearing the 404 Not Found entries usually helps. It's been developed by me a long time ago and I would do so much different nowadays, but that doesn't help you ;-)

[daniel-chenery]

Yupp, one day, eventually, I'll probably re-develop the backoffice to use Angular. It's a while away from today though 👍

[PeteDuncanson]

As a work around I've moved it to a new "Tools (Slow)" section in the back office :)

Inspired by a discussion with the evil hacker genius Marc Goodson, it was his smart idea to move it to the Traslation section.

[marcemarc]

Also the Tracker Table gets alot of activity over a period of time, so make sure you regularly check index fragmentation of that table.... https://docs.microsoft.com/en-us/sql/relational-databases/indexes/reorganize-and-rebuild-indexes

Basically any improvement in performance you can squeeze on reading from that table, remove duplicates, 404s (if you not doing anything about them, turn 404 tracking off, (or ignore common requests if you are... /wpadmin.php) etc - will improve the lag time

But yes apologies to any translators in the future who happen to discover UrlTracker hidden there!

[kipusoep]

We've discussed the current status of the UrlTracker at the company I work for and we might pick up where we left off.
Concrete ideas are rewriting the back-office UI to use native angular, fix critical issues, improve performance (404 logging, which slows down the whole tool for example) :-)

Again; the discussion is still in progress, but we've seen how much the UrlTracker is still used and think there's no real replacement yet. Even Umbraco's effort to replace it was a total disaster IMO.

/ping @daniel-chenery

[daniel-chenery]

@kipusoep We are in a similar position where we would love to re-build this in Angular, but our resources are elsewhere at the moment.
I'm trying to keep on top of the critical issues where possible, but if they can't be replicated, they're difficult to fix.

The UrlTracker is excessively used though, and it's a massive shame the Core's version doesn't allow for custom entries.

[kipusoep]

I must say I'm really excited about picking this up again, so I think chances are pretty big that at least myself will be starting development again in the near future :-)

[daniel-chenery]

@kipusoep Excellent! Do ping me an email if you need a hand with anything 😃

[dawoe]

@daniel-chenery
Indeed the core version doesn't allow entries out of the box. But it is possible.

@marcemarc wrote some excellent blogs about it
http://tooorangey.co.uk/posts/301-redirection-in-umbraco-it-s-a-rum-do/
http://tooorangey.co.uk/posts/how-can-it-be-its-a-rum-do-part-two/

Doing something similar for a customer which I released as a package
https://our.umbraco.org/projects/backoffice-extensions/redirects-viewer/

Maybe it's better connect with Umbraco HQ to see which features from this package they would have in the core ?

Dave

[kipusoep]

Maybe it's better connect with Umbraco HQ to see which features from this package they would have in the core ?

I really think it should've been the other way around; Umbraco HQ started developing this feature without talking to me at all. It felt like a backstab when I found out they developed this.
All the effort I've put into this package felt like a total waste when I found out they just introduced a "Disable UrlTracker" button.

[sleslie321]

package is well used - would love to see it being developed further. Cheers

[dawoe]

@kipusoep
All the effort I've put into this package felt like a total waste when I found out they just introduced a "Disable UrlTracker" button.

That button disables the Umbraco Core tracker..not yours if i'm correct

Dave

[dawoe]

The docs even state that they disable core url tracking when your package is installed

https://our.umbraco.org/Documentation/Reference/Routing/URL-Tracking/#enable--disable--configure

[kipusoep]

Alright, my bad, still I would have liked to be part of the Core Url Tracker. I even discussed that with multiple core team members and agreed to cooperate on this, which didn't happen.

Anyway, it's probably bad to start chasing old grudges :-)

[daniel-chenery]

We could always disable the core URL tracker on install too 😆
https://github.com/umbraco/Umbraco-CMS/blob/dev-v7/src/Umbraco.Web/Editors/RedirectUrlManagementController.cs#L70

[ArnoldV]

@kipusoep you can still be part of the Core URL tracker if you create a pull request will all the things missing from the core url tracker. Create an issue on the issue tracker with what is missing and create the pull request. For sure @nul800sebastiaan is willing to add the missing functionality/features if they make sense.

[akeilox]

its great to hear @kipusoep the dev will continue, with better 404 and angular ui

whoever used URL tracker before core tracker introduced, still trying to continue using URL tracker for obvious reasons.

[kipusoep]

I'll discuss the suggestions tomorrow with my colleagues to see which approach we like best.
Downside of adding it to the core with PR's; there's no guarantee it'll be accepted.

[dawoe]

I would first create a issue for it on the tracker to discuss before making a PR

[nul800sebastiaan]

Hello everyone!

Good to see the renewed enthusiasm for updating a plugin that many people have enjoyed over time!

The current implementation of the automatic URL tracker in Umbraco core was the product of a spur of the moment "I'm passionate about this, let's see if I can help HQ make a solid implementation of tracking changes to URLs on nodes". This was kickstarted by @marcemarc at the Umbraco retreat a couple of years ago and the implementation was so simple and workable that we got very excited and after a few rounds of review decided it would be good to put into Umbraco immediately. Unfortunately, there wasn't time to do get your stamp of approval @kipusoep - but we obviously never meant to hurt your feelings. Sometimes, in the heat of the moment oget carried away, sorry about that.

By the time we went from proof of concept to polishing this feature (over the course of 3 days) it was clear that we were taking an entirely different road from the way URLTracker works. When URLTracker started we didn't have such a nice pipeline with IContentFinders and it's impressive that you've been able to build a super successful package out of this without some nice built-in core support.
As you might know, we were scared to interfere with existing URL trackers like this one so we explicitly built in exceptions: if you're already using a solution then we don't want to take over and possibly cause weird results; therefore, the core URL tracker is disabled when another one is installed.

The current implementation was always ONLY meant to be super safe and should not ever be able to cause infinite loops or performance problems. These were the only 2 success criteria.
We've been extremely happy with the implementation that we ended up with in that weekend two years ago, so happy even that we've made only 2 or 3 updates to it since that time.
Of course we realize that this doesn't cater for everyone's needs, but hey: it covers the bases for most people just fine.

Which brings me to the cry for new features that we've been getting from day one. :-) Let me explain the reasons why some things never got implemented even after numerous requests (this is also the reason why the URLTracker package is to this day very popular):

1. Adding manual redirects - since this functionality is exposed to content editors who are not technical and do not know how to solve infinite loops or realize that they've caused them we have never allowed anyone to add a text field where people can type a url to redirect from and a destination node. We have no plans to make this a feature if only for usability reasons and the potential for errors.

   1.1 Additionally we know that people migrating a site to Umbraco need to redirect a large number of URLs to nodes in Umbraco and this is a good task for a developer to set up as IIS redirects. IIS redirects are infinitely faster than anything that has to go through the redirect pipeline in Umbraco and should always be preferred for that reason alone.

2. Manually adding a redirect from one Umbraco node to another Umbraco node - now that we know that the current implementation of redirects works and works really well, we're open to allowing editors to add a new redirect by picking a source node and a target node. Right now @marcemarc is experimenting with this and I think his results should be incorporated in the core at some point: http://tooorangey.co.uk/posts/how-can-it-be-its-a-rum-do-part-two/ and of course @dawoe 's efforts here have been noticed as well at https://our.umbraco.org/projects/backoffice-extensions/redirects-viewer/

3. Show redirects on the Info tab (the artist formerly known as the Generic Properties Tab - TAFKATGPT - hmmm doesn't roll off the tongue very well now does it..) - Yes please, there's a PR: U4-10669 7.8. Show 301 url redirects on info tab too umbraco/Umbraco-CMS#2452

4. Manage redirects on the Info tab (TAFKATGPT) - Yes please, PR: Manage redirects from a content item's Info tab - issues: 8925 & 10669 umbraco/Umbraco-CMS#2511 (though not sure about the ability to add relative URLs to redirect from)

5. 404 logging - very unlikely to happen unless we can aggressively prune this list easily or ban certain classes of URLs from being logged somehow - as we can see from this issue here so much data could potentially be logged and it becomes unmanageable at some point. I think we'd encourage custom solutions instead

I hope this make our well-loved "disaster" a bit more clear and it might also help you decide where to focus your efforts.

Note: All of this is absolutely open for discussion and if we can find a very good way to support some of the features above then we're very happy to listen.

[kipusoep]

Thanks for clarifying these points @nul800sebastiaan.
I didn't want to give any stamp of approval though, it just would've been nice to keep me involved like we discussed in the past.

There are some very valid points in your post about keeping it simple and taking no risks. The UrlTracker has proven that errors and performance issues are so easy to run into. We'll take everything into consideration tomorrow and make a decision on how to move forward.

As far as I understand you guys want to keep the core redirect module lightweight and additional features (created by the community) will be more likely to exist as add-ons outside of the core, right?

[nul800sebastiaan]

While I regret not contacting you after it became clear that we were going to run with the solution we came up with, we also run by the ethos of always assuming positive intent https://our.umbraco.org/terms-and-conditions/ - it's therefore a shame that you felt so slighted by us. So I dropped the communication ball here, and I apologize for that!

Sure, we love add-ons and we also love if add-ons at some point could "graduate" into the core, if they've been proven to be a healthy solution that fits with the goals of Umbraco as a CMS!

[PeteDuncanson]

Just found a long discussion on Our that was had about this some years back this is still valid I think so putting it here for reference.

https://our.umbraco.org/forum/contributing-to-umbraco-cms/80583-is-updating-the-301-redirect-to-evolve-to-full-rewrite-engine-a-good-idea

[marcemarc]

@nul800sebastiaan not kickstarted by me! the driver for it was a general discussion on CMS feature comparison sites that were fashionable at the time, and in a lot of them: 'tracking url changes with 301s - out of the box' was a thing - and Umbraco could only do this with plugins, hence the aim to provide a minimum thing to tick that feature comparison box and nothing else, but if you got UrlTracker installed you'd just use that... I only worked on it cos James M South had done something around redirects already, and he wanted someone to pal up with... and really enjoyed that.... but feel guilty as the dashboard I instigated causes me no end of pain with questions from editors about how to add new entries! and them working out how to do that by cleverly publishing and renaming and removing - hence the blog posts and PRs - but for the record - if I had kickstarted it, then I would definitely only have done so in order to spite @kipusoep who is my arch nemesis, and must be stopped! :-P

[kipusoep]

@marcemarc 🤣

@ALL
We're going to look into the redirect manager built in the umbraco core to see what features it's lacking for us. So for now our plan is to use what's already there and make it better where needed, instead of kickstarting the good ol' UrlTracker.
Maybe we can do this together with everyone involved who wants to be involved and use a slack channel or something like that to keep track of everything and have discussions?

[kipusoep]

Which makes me wonder, what features are you guys missing?
/ping @PeteDuncanson @daniel-chenery @marcemarc @dawoe @sleslie321

[daniel-chenery]

The biggest missing feature for me is the ability to add custom rules, that is what motivated me to fix the initial YSOD bug.

However, @nul800sebastiaan's 1st point makes me wonder if a PR like this would be added. Perhaps if we only limited this feature to those who had access to the "developer" section?
I do agree, there would be a risk for someone taking a whole site down/redirect loop with an incorrect regex.

I don't use Slack daily, but I agree for a channel!

[kipusoep]

Feel free to join me on umbracians slack: https://umbracians.slack.com/messages/C9R54LJRJ/

[nul800sebastiaan]

However, @nul800sebastiaan's 1st point makes me wonder if a PR like this would be added. Perhaps if we only limited this feature to those who had access to the "developer" section?

Yep, we do not want to support this in core, if a developer has to do it then they can go in and update their web.config with a custom redirect rule. That will also force people to do proper testing before just trying to add a redirect on their live site. This will also be much more performant, we don't have to go through the request pipeline in Umbraco at all, we hit IIS first and it can handle the necessary rules.

I only worked on it cos James M South had done something around redirects already

Oh no @marcemarc - I didn't mean to blame you, my memory is apparently faulty! :-)

[kipusoep]

The biggest missing feature for me is the ability to add custom rules, that is what motivated me to fix the initial YSOD bug.

With @dawoe his excellent package you can add custom rules... :-)

[dawoe]

With @dawoe his excellent package you can add custom rules... :-)

What do you mean buy this ?
Regex redirects ? ⛔️
Custom url for a content node ✅

[Matthew-Wise]

Happy new year all.

I am willing to throw my hat in to move away from web forms to angular. Unless someone else has started if not Ill start sometime this month.

Matt

edit
Would be good to get the outstanding PR's merged as I can see me having to make a lot of changes