Support with Nextcloud performance

At Autonomic we’re running a Nextcloud instance for around 50-200 users using the official image (containerised deployment) and a performance tuned configuration (see configs here).

We’re still running into serious issues with speed and performance. The web UI is taking a loonnngggg time to load (20 - 40 seconds in some cases) and it is really killing the usability.

Does anyone here have experience making Nextcloud run faster? Specifically the responsiveness of the web UI. We haven’t seen much issues with the clients.

It would be great to work with someone who has experience with running large Nextcloud instances and what to tweak to get the most out of it. Experience with the container deployment option is a huge plus.

We can figure out some compensation for your time! Hit us up.

3 Likes

We don’t use Docker for this, we use Apache, Coturn, MariaDB, Memcache, PHP-FPM and Redis for Nextcloud and Nginx and PostgreSQL for ONLYOFFICE all deployed to Debian VMs using Ansible and haven’t hit any performance issues, but I don’t think any of our instances have as many users, if you would be interested in using our Ansible to deploy a server for a performance comparison let me know.

2 Likes

Nextcloud is really really CPU heavy. We run an instance with several hundred users on dedicated hardware with the database moved to a different server and still see high loads.

However, within the last 6 months the nextcloud folks have made huge improvements to CPU usage - including a new high performance back end written in rust - which we haven’t tried yet. And also whatever this means. I have a hard time reading through the Nextcloud marketing material to really understand what any of this means, but I suspect if you explore those posts you may find some CPU relief.

3 Likes

I believe that the improvements @jamie has linked to relate to the current notify_push app and future improvements to it, which are not yet available.

These are our Ansible tasks to configure it plus we have this YAML to configure the Apache reverse proxy for it via this template, however note that updated clients are required to use this feature and I’m not sure if the mobile clients support it yet?

2 Likes

@decentral1se do you have the antivirus app enabled? This can bring servers to their knees as it ends up spawning dozens of intensive ClamAV processes, even when MAX_THREADS is set to 1, I haven’t looked at solving this recently as I simply ended up disabling the app on all servers…

2 Likes

Nice one, thanks for weighing in :blush:

No antivirus app no, just took a look, phew. The notify_push app looks like a fine addition but without some clear idea that it is indeed the end-user clients hogging all the resources (which it doesn’t look like, since the web UI is the pain point?).

I can’t see anything majorly different from your Ansible configs, thanks for sharing them.

Ofc there is some growing doubt that “it is just Docker” but I really can’t be sure yet. I will be trying to profile this thing a bit more. Gotta find the bottleneck…

The things I’d suggest checking for the web interface are:

  1. Is OPcache enabled and is opcache.memory_consumption set to at least 64MiB (on the CoTech Nextcloud server the max OPcache memory usage in the last year was 61MiB). This enabled compiled PHP to be called from memory rather than disk.
  2. Is the MariaDB / MySQL InnoDB buffer pool greater than the data size? Again this enables the database to use RAM rather than disk, check for this and other suggestions by running MySQLTuner (it is packaged for Debian).
  3. Is Memcache or Redis installed and enabled?

How many CPUs and of what spec and how much RAM is assigned and what is the usage looking like?

2 Likes

Got that.

Gonna check this out!

Yep, it’s up. We also have the cron switched to background jobs.

8 dedicated vCPUs with 32GB RAM (CCX32 hetzner VPS). We’re not seeing any RAM/CPU overutilisation after doing an upgrade so it seems to be some limit within the app configuration…

Thanks for thinking along!

2 Likes

All the above advice is very similar to what we’ve found in the last couple of years running several NC instances (all of them smaller that yours I think). Some other things that are worth exploring:

Which NC version are you running? We had MySQL performance issues with v20 and v21 (to a lesser degree), apparently caused by the new unified search. https://github.com/nextcloud/server/issues/23835

Have you enabled HTTP2? It should improve Nginx/Apache performance in recent NC versions.

Finally, which PHP version are you running? Apparently OPCache performance is considerably better on v8, PHP 8.0 and 7.4 benchmarks on Symfony: JIT boost up to +9% with OPCache Preload

2 Likes

Cheers for posting that, I wasn’t aware of opcache preloading however I don’t think it can be used with Nextcloud, this issue about supporting it was closed this month:

Currently there a no plans to implement such a feature.

1 Like

That’s a pity, I remembered seeing it mentioned in the release documentation for v21 so I assumed it was already implemented. Yay marketing!

The issue you linked has one person willing to work on it, so maybe it will get done anyway…

2 Likes

Thanks @dvdjaco! We’re running latest stable NC. With the help of @chris we’ve implemented a lot of tweaks especially around the database setup. We’ve seen a number of improvements. We’re using the mainline upstream container, so we take whatever PHP they give us.

It seems Collabora (which is generally the most Google Drive-like experience in terms of features and usability) is one contributor to the slow-down. We’ve seen improvements turning it off :see_no_evil: We may try the new version of OnlyOffice that came out recentlty, it has improved although groups we’ve tried it with have complained about it.

We’re still looking into all this and working on it as we go. Our container configs will remain available over in coop-cloud/nextcloud: Nextcloud Server, a safe home for all your data - nextcloud - Co-op Cloud Code :computer:

2 Likes

This blog has a good summary of the architectural difference between Collabora and ONLYOFFICE — Collabora runs more on the server side so needs more resources:

The way Collabora Online works is:
An embedded version of Libreoffice runs on the server. It reads the document, then ‘streams’ the rendered document as image tiles to the browser client, which shows it to the user. The browser client does some of the menu’s and lots of smart things like showing the cursor, other users, text selection etc, but many other components like pop-up menu’s and sidebars are also streamed from the back-end, giving relatively good feature parity with LibreOffice. This strategy is responsible for giving LibreOffice, for example, desktop-level table style editing, better than any other online office solution.

The way ONLYOFFICE works is:
The document is converted on the server to a JSON file which is streamed to the browser client. The browser client is the full office suite, editing the document. Once done, it sends back the JSON and the server merges and exports it back to a file. A fully html5 canvas based front-end means a relatively pretty user interface and any javascript dev can go hacking.

I’d suggest that if you use LibreOffice on the desktop then Collabora probably makes most sense and if you use Microsoft Office then ONLYOFFICE probably makes more sense.

At the moment Webarchitects only offers a stand alone ONLYOFFICE server for all our Nextcloud clients (I’ve found that this works well when installed using Debian packaging rather than running it in Docker, which was a nightmare) but I intend to have a Collabora option in place when I find time.

3 Likes

Small update: current latest version of NC & Collabora are much more performant!

1 Like

My Nextcloud update is that I’ve added and enabled APCu to our PHP and Nextcloud Ansible roles and this seems to be working well so at some point I’ll probably drop the use of Memcached.

3 Likes