At Autonomic we’re running a Nextcloud instance for around 50-200 users using the official image (containerised deployment) and a performance tuned configuration (see configs here).
We’re still running into serious issues with speed and performance. The web UI is taking a loonnngggg time to load (20 - 40 seconds in some cases) and it is really killing the usability.
Does anyone here have experience making Nextcloud run faster? Specifically the responsiveness of the web UI. We haven’t seen much issues with the clients.
It would be great to work with someone who has experience with running large Nextcloud instances and what to tweak to get the most out of it. Experience with the container deployment option is a huge plus.
We can figure out some compensation for your time! Hit us up.
We don’t use Docker for this, we use Apache, Coturn, MariaDB, Memcache, PHP-FPM and Redis for Nextcloud and Nginx and PostgreSQL for ONLYOFFICE all deployed to Debian VMs using Ansible and haven’t hit any performance issues, but I don’t think any of our instances have as many users, if you would be interested in using our Ansible to deploy a server for a performance comparison let me know.
Nextcloud is really really CPU heavy. We run an instance with several hundred users on dedicated hardware with the database moved to a different server and still see high loads.
However, within the last 6 months the nextcloud folks have made huge improvements to CPU usage - including a new high performance back end written in rust - which we haven’t tried yet. And also whatever this means. I have a hard time reading through the Nextcloud marketing material to really understand what any of this means, but I suspect if you explore those posts you may find some CPU relief.
@decentral1se do you have the antivirus app enabled? This can bring servers to their knees as it ends up spawning dozens of intensive ClamAV processes, even when MAX_THREADS is set to 1, I haven’t looked at solving this recently as I simply ended up disabling the app on all servers…
No antivirus app no, just took a look, phew. The notify_push app looks like a fine addition but without some clear idea that it is indeed the end-user clients hogging all the resources (which it doesn’t look like, since the web UI is the pain point?).
I can’t see anything majorly different from your Ansible configs, thanks for sharing them.
Ofc there is some growing doubt that “it is just Docker” but I really can’t be sure yet. I will be trying to profile this thing a bit more. Gotta find the bottleneck…
The things I’d suggest checking for the web interface are:
Is OPcache enabled and is opcache.memory_consumption set to at least 64MiB (on the CoTech Nextcloud server the max OPcache memory usage in the last year was 61MiB). This enabled compiled PHP to be called from memory rather than disk.
Is the MariaDB / MySQL InnoDB buffer pool greater than the data size? Again this enables the database to use RAM rather than disk, check for this and other suggestions by running MySQLTuner (it is packaged for Debian).
Yep, it’s up. We also have the cron switched to background jobs.
8 dedicated vCPUs with 32GB RAM (CCX32 hetzner VPS). We’re not seeing any RAM/CPU overutilisation after doing an upgrade so it seems to be some limit within the app configuration…
All the above advice is very similar to what we’ve found in the last couple of years running several NC instances (all of them smaller that yours I think). Some other things that are worth exploring:
Thanks @dvdjaco! We’re running latest stable NC. With the help of @chris we’ve implemented a lot of tweaks especially around the database setup. We’ve seen a number of improvements. We’re using the mainline upstream container, so we take whatever PHP they give us.
It seems Collabora (which is generally the most Google Drive-like experience in terms of features and usability) is one contributor to the slow-down. We’ve seen improvements turning it off We may try the new version of OnlyOffice that came out recentlty, it has improved although groups we’ve tried it with have complained about it.
This blog has a good summary of the architectural difference between Collabora and ONLYOFFICE — Collabora runs more on the server side so needs more resources:
The way Collabora Online works is:
An embedded version of Libreoffice runs on the server. It reads the document, then ‘streams’ the rendered document as image tiles to the browser client, which shows it to the user. The browser client does some of the menu’s and lots of smart things like showing the cursor, other users, text selection etc, but many other components like pop-up menu’s and sidebars are also streamed from the back-end, giving relatively good feature parity with LibreOffice. This strategy is responsible for giving LibreOffice, for example, desktop-level table style editing, better than any other online office solution.
The way ONLYOFFICE works is:
The document is converted on the server to a JSON file which is streamed to the browser client. The browser client is the full office suite, editing the document. Once done, it sends back the JSON and the server merges and exports it back to a file. A fully html5 canvas based front-end means a relatively pretty user interface and any javascript dev can go hacking.
I’d suggest that if you use LibreOffice on the desktop then Collabora probably makes most sense and if you use Microsoft Office then ONLYOFFICE probably makes more sense.
At the moment Webarchitects only offers a stand alone ONLYOFFICE server for all our Nextcloud clients (I’ve found that this works well when installed using Debian packaging rather than running it in Docker, which was a nightmare) but I intend to have a Collabora option in place when I find time.
My Nextcloud update is that I’ve added and enabled APCu to our PHP and Nextcloud Ansible roles and this seems to be working well so at some point I’ll probably drop the use of Memcached.