Reorganising and migrating CoTech sites and servers

I’ve been promising to rationalise and reorganise the CoTech hosting services Webarchitects provides for a while and also to configure services on the cotech.uk domain and invoice for this years hosting and I’m going to make a start on this today and use this repo for all the Ansible.

This will result in some downtime for services as they are moved!

4 Likes

Of course this is taking longer than anticipated… :roll_eyes: … however, today I have:

  • Created a new git.coop repo for managing the Bind 9 zone files for coops.tech and cotech.uk , if anyone who knows how to edit Bind 9 zone files and needs or would like access to this let me know your git.coop username and I can add you to the project. This repo is private.
  • Created two new micro servers, one for Discourse and one for the Jekyll site and also all the PHP based sites (Nextcloud, MediaWiki, Lime Survey etc).
  • Configured the new web.cotech.uk server using the Ansible in this repo.
  • Migrated the Nextcloud site to the new web.cotech.uk server.
  • Generated an invoice for Webarchitects services to CoTech for 2022 for £1,553.79 (this is less than the invoice for 2021 which was for £1,683.44 despite the fact that last year we were not VAT registered and this years invoice includes £251.50 for VAT).

TODO:

  • Migrate all the other PHP sites (time.coops.tech, wiki.coops.tech, survey.coops.tech).
  • Update the website repo to install the dev.coops.tech and www.coops.tech sites on the new web.cotech.uk server.
  • Install Matomo (currently the sites use a Matomo instance on a Webarchitects shared hosting server) and configure it to collect stats from the public sites, www.coops.tech, wiki.coops.tech and perhaps community.coops.tech using this plugin.
  • Configure Icinga and Munin monitoring for the new servers, sites and services.
  • Configure email on the cotech.uk domain.
  • Update the Ansible we have for Discourse so that it can be included as a role into the servers repo.
  • Migrate this Discourse site to the new server.
  • Document how other people can use Ansible to update and maintain the servers and add and remove accounts and services.
3 Likes

Well done Chris. That looks like a tremendous amount of work. Brilliant.

When is the next Cooperative Technologists face to face meeting like the one shown in the Nextcloud Login page page. Any idea?

1 Like

15th-16th June in Birmingham, the thread for it is pinned at the top of the front page, further details on the wiki.

Oh, that’s handy. I am booked in to go to the Co-op UK conference so I will see you there.
All the best,

David

1 Like

I’ve copied this forum to https://community.coops.tech/ and I’ll do some testing to check that everything is working there and then later today take a final backup of Discourse on this server and copy it to that server then update the DNS, shutdown the old server and rebuild the Docker container and then we will be up and running on the new VM.

This is a test to see if email is working post-migration…

If this message appears then that means that reply by email is working!

However this is a DKIM issue that needs fixing:

ARC-Authentication-Results: i=1;                                                                                                                          
        mail.webarch.email;                                                                                                                               
        dkim=none;                                                                                                                                        
        spf=temperror (mail.webarch.email: error in processing during lookup of discourse+verp-XXX@community.coops.tech:     
        DNS error) smtp.mailfrom=discourse+verp-XXX@community.coops.tech;                                                    
        dmarc=temperror reason="SPF/DKIM temp error" header.from=coops.tech (policy=temperror)                                                            

This is a reply to check if the DKIM issues have been resolved…

The DKIM issue has been resolved, outgoing email is now DKIM signed, but there is a SPF issue:

ARC-Authentication-Results: i=1;                                                                                                                          
        mail.webarch.email;                                                                                                                               
        dkim=pass header.d=community.coops.tech header.s=20220605 header.b=XXX;                                                                      
        spf=softfail (mail.webarch.email: 81.95.52.58 is neither permitted nor denied by domain of                                                        
        discourse+verp-XXX@community.coops.tech)                                                                             
        smtp.mailfrom=discourse+verp-XXX@community.coops.tech;                                                               
        dmarc=pass (policy=none) header.from=coops.tech                                                        

But hopefully this has been fixed with this DNS update:

dig TXT coops.tech +short
"v=spf1 a mx include:_spf.webarch.email include:office.coops.tech include:community.coops.tech include:fund.coops.tech ip4:81.95.52.58 ~all"

Looks good to me:

ARC-Authentication-Results: i=1;                                                                         
        mail.croome.net;                                                                                 
        dkim=pass header.d=community.coops.tech header.s=20220605 header.b=XXX;                     
        spf=pass (mail.croome.net: domain of                                                             
        discourse+verp-XXX@community.coops.tech designates 81.95.52.58 as   
        permitted sender)                                                                                
        smtp.mailfrom=discourse+verp-XXX@community.coops.tech;              
        dmarc=pass (policy=none) header.from=coops.tech 

There are some internal hyperlinks, for example the one in this post that point to forum.cotech.uk so I’m going to run the last couple of commands documented in this post to sort that out:

discourse remap https://forum.cotech.uk https://community.coops.tech
Rewriting all occurrences of https://forum.cotech.uk to https://community.coops.tech
THIS TASK WILL REWRITE DATA, ARE YOU SURE (type YES)
YES
post_hotlinked_media=2
post_revisions=557
post_search_data=240
posts=5435
site_settings=2
stylesheet_cache=186
topic_links=2380
topic_search_data=117
topics=356
user_histories=37
Done

The above has run OK, now the regenerating all the posts is underway…

And that has now completed:

rake posts:rebake
Rebaking post markdown for 'default'
    13301 / 13301 (100.0%)
13301 posts done!
1 Like

I’ve finally got around to doing some more work on this in anticipation of the Ansible session in Glasgow on Wednesday, so these things have now been done:

  • Migrate all the other PHP sites (time.coops.tech, wiki.coops.tech, survey.coops.tech).
  • Update the website repo to install the dev.coops.tech and www.coops.tech sites on the new web.cotech.uk server.
  • Configure Icinga and Munin monitoring for the new servers, sites and services.
  • Update the Ansible we have for Discourse so that it can be included as a role into the servers repo.
  • Migrate this Discourse site to the new server.

TODO:

  • Install Matomo (currently the sites use a Matomo instance on a Webarchitects shared hosting server) and configure it to collect stats from the public sites, www.coops.tech, wiki.coops.tech and perhaps community.coops.tech using this plugin.
  • Configure email on the cotech.uk domain.
  • Document how other people can use Ansible to update and maintain the servers and add and remove accounts and services.