If all has gone well you will have your old IdPv3 serving authentication requests for your users and a new partially configured  IdPv4, with a little bit of work to complete before is replaces your old IdP.


This guide will take you through the steps to complete the migration. The aim is to create a new IdP that will respond to the metadata already registered in the AAF, i.e without changes to that metadata. This will ensure a a simple roll-forward / roll-back for when you are ready to deploy your new IdP.


TABLE OF CONTENTS


Preparing for later testing

Part of the testing of your IdP migration is ensuring the same user attributes are released before and after the migration. The AAF Attribute validator can help here. While your current IdP is available have a number of users login to the AAF Validator. Preferably users should be of different types, e.g. staff, students, visitors etc. The AAF Validator will record the attribute values released for each user. After migration ask the same users to use the AAF validator again. they can then check the attributes from both old and new IdPs are the same.


Links to the AAF Attribute validatorTest Environment
Production Environment

V3 Export

This guide assume that 

  • you have used export_v3_config.sh on your old IdP to export configuration information and files. 
  • The exported files have then been copied to your new server in the directory ~/V3-export
Please ensure that you change [SERVER-NAME] appropriately in the example code below!


DNS names and IP Addresses

To ensure the IdP metadata registered in the AAF for the IdP does not need to be modified  the new IdP MUST have the same DNS name as the old on completion of the migration process. The last step of the migration is the change the DNS records of the IdP so it points to the new IdP. If a rollback is required, it is a simple reversing of the DNS change.


While testing you can use your local hosts file to have your computer to use the new IdP while all of your users continue to use the old IdP.


Alternatively, if your IdP sits behind a load balancer, you can configure the load balancer to split traffic, general use to the old IdP, testers to the new IdP. When ready, configure to users to the new IdP.


The installer has run

Following these guides you have already run bootstrap-v4.sh using the bootstrap-v4.ini created on your Old IdP. If all has gone well with the install a Jetty process should be running. Verify using the following command;

ps ax | grep jetty


You should see output similar to the following;


345011 ?        Sl     0:12 /usr/lib/jvm/java/bin/java -server -Djetty.home=/opt/jetty/jetty-distribution-9.4.31.v20200723 -jetty.logs=/opt/shibboleth/jetty/logs jetty.state=/opt/shibboleth/jetty/tmp/jetty.state jetty-started.xml Djetty.base=/opt/shibboleth/jetty -Djava.io.tmpdir=/tmp -jar /opt/jetty/jetty-distribution-9.4.31.v20200723/start.jar 
...


If jetty is not running please check the activity.log in the directory /opt/shibboleth-idp4-installer/. Search for any errors are resolve. Re-run bootstrap until it completes successfully.


Re-deploying your IdP

All changes to your IdP should occur in the assets area and then deployed to your running IdP.


The assets area can be found here.

/opt/shibboleth-idp4-installer/repository/assets/[SERVER-NAME]/...


To deploy a change run the deploy script. This can be run from any where on your file system but MUST be run as ROOT.

/opt/shibboleth-idp4-installer/repository/deploy


Migrating LDAP configuration

The export attempted to extract the LDAP connection details from your V3 IdP. These details were then provided to the bootstrap process when creating your new IdP. If your V3 IdP was using a secure connection to your LDAP server (LDAPS, or LDAP over TLS) or your V3 IdP had a more advanced LDAP configuration such as connecting to multiple LDAP instances then you may need additional configuration for your LDAP connection.


The LDAP configuration can be found here.

/opt/shibboleth-idp4-installer/repository/assets/[SERVER-NAME]/idp/conf/ldap.properties


LDAP connection errors in your Shibboleth logs is a good indicator that additional configuration is required.


Check /var/log/shibboleth-idp/idp-warn.log for LDAP connection errors.


Please see the Shibboleth IdP 4 LDAP documentation for a description of all available LDAP options.


Please ensure your Shibboleth IdP can successfully communicate with your LDAP server before continuing.


IdP SAML certificates

Again to ensure the IdP metadata does not need to change, we will migrate the SAML certificate across from the old IdP to the new. The export-v3-config.sh script will have created a credentials directory in which it copied all of the certificates and keys used by the IdP when performing SAML transactions. Copy these certificates into your new IdP repository.


cp ~/V3-export/credentials/* /opt/shibboleth-idp4-installer/repository/assets/[SERVER-NAME]/idp/credentials



Back Channel (Port 8443)

If your IdP needs to use the SAML back channel for older SPs or any of the other back-channel use cases you will also need to copy the keystore password from your old IdP. This password was collected by the export-v3-config.sh so will be available in the V3-export/passwords directory in the shib_idp_keystore file. Copy this file to /opt/shibboleth-idp4-installer/repository/passwords/[SERVER-NAME]


cp ~/V3-export/passwords/shib_idp_keystore /opt/shibboleth-idp4-installer/repository/passwords/[SERVER-NAME]



IdP client facing certificates

The installer creates a pair of self-signed certificates  when it creates a new IdP instance. These need to be replaced with certificates that have been signed by your chosen certificate authority.


The V3 export has copied the certificates from your existing IdP which you can copy using the command below, or you can refresh the certificates using your own local processes for issuing web server certificates. 

cp ~/V3-export/tls/* /opt/shibboleth-idp4-installer/repository/assets/[SERVER-NAME]/tls

 

Bi-Lateral Services

Your old IdP may have had a number of locally connected services configured (bilateral services). The configuration for bilateral services is the same for this version of the installer. The contents of your old IdPs bilateral services area have been includes in the V3-export/bilateral directory.


cp -r V3-export/bilateral/* /opt/shibboleth-idp4-installer/repository/assets/[SERVER-NAME]/idp/bilateral


There may be additional changes required in the relying-party.xml and saml-nameid.xml configuration files to complete the support of your bilateral services. These changes will be discussed later.


Salt Values

The IdP utilizes a number of SALTs to create user's unique identifiers including the auEduPersonSharedToken and eduPersonTargetedId as well as the persistent NameID for some services. To ensure there is no change to the values being created by the IdP the salt values need to be copied from your old IdP. These values have been exported in the V3-export. Copy the files into you new IdP's configuration settings.


cp V3-export/passwords/*_salt /opt/shibboleth-idp4-installer/repository/passwords/[SERVER-NAME]



Resolving Attributes

Attribute resolution configuration has had the most significant changes between version 3 and 4. Much of the configuration has moved out of the attribute-resolver.xml file and have moved into an attribute register directory. This has resulted in a less complex resolver file but now need to maintain some of the attribute information elsewhere.


Migrating your attribute resolver is describers in move detail here.

Attribute Filtering

The installer provides a new version of a attribute filter that should be able to support all services within the AAF. This filter uses information in the federation metadata to determine what attributes to release to each service. 


Attribute filtering for bilateral services

If you have bilateral services, you may have added specific attribute filter rules to release attributes to these services. This should have occurred in the idp/bilateral/filters/[YOUR-FILE-NAME].xml. These filters will be available to copy from the V3-export/bilateral/filters directory.


For each filter file you add you MUST add a reference to each in the services.xml file in the shibboleth.AttributeFilterResources list. This will ensure the filter is loaded when the IdP starts.


When you start the IdP after added a new filter check the log file idp-process.log for any deprecation warnings or errors. The warning should provide an alternative for each deprecated item found. Modify the filters appropriately. 


Populating you local database

Your IdP may have generated some of your users identity information, in particular their auEduPersonSharedToken and eduPersonTargetedId values as well are NameID values for some services. These values would have been created and stored in a local MySQL of Maria database. As part of V3-export the local database contents was dumped so it could be imported into the new IdPs Maria database.


As the table structure has not changed between versions of the IdP, the database contents can simply be imported. This will ensure that user's identity information does not change as a result of the upgrade.


As root execute the following command to populate your database.


mysql idp_db < V3-export/mysql-dump.sql


When complete you should verify that the tables have been populated successfully.

mysql
> select * from StorageRecords;
> select * from shibpid;
> select * from tb_st;


Note: If you are using the external database such as ORACLE, MS SQL Server, etc that you used for your V3 IdP there is no need to migrate. You will however need to modify the database connection information if conf/global.xml and the shibboleth.JPAStorageService.DataSource bean at the end of this file.


Bilateral services metadata

If your old was connected to bilateral services you need to copy the metadata for each into the new IdP. The metadata can be found in the V3-export/bilateral/metadata directory. Copy these files to the equivalent area in your new IdPs assets area. They will then be added to your IdP when you next deploy.


You must also add a reference to each in the metadata-providers.xml file. The original file from your old IdP has been included in the V3-export/config area. We recommended you cut and paste only relevant configuration for your bilateral rather than coping the entire file across.


Bilateral services special configuration

Some bilateral services require some additional configuration to ensure they will work with your IdP. This configuration is generally found in the relying-party.xml file, specifically in the shibboleth.RelyingPartyOverrides area. 


We recommend coping each entry one-by-one and testing by deploying and verifying there are no errors or deprecation warnings before moving onto the next.


The Shibboleth wiki documentation Relying Party Configuration will aid you to resolving issues.


Bilateral services and special NameID

A very few bilateral services will have specific requirements for NameIDs. Most will be dealt with in the relay-party.xml  configuration file, but they may reference additional configuration in the saml-nameid.xml file. As there are very rare you will need to check on your old IdP for any additions.


IdP Branding

The branding on your old IdP has probably not been reviewed since your installed it many years ago. We have deliberately left the old branding back on your old IdP with the hope that you refresh it based on your current organisational branding standards. You will find things mostly the same with the out-of-the-box branding with the most notable addition of the following in most .vm file.


#parse("csrf/csrf.vm")


These have been added to help guard against Cross Site Request Forgery (CSRF) attacks. They are required, otherwise your users will see errors when attempting to use the IdP. If you select to just copy your original branding across you  MUST and csrf code shown above.


Testing

Your IdP migration should almost be complete and well tested. It's time to invite others in to test. If you are running both old and new IdPs in parallel then your testers need to point there computers at the new IdP.  See DNS names and IP Addresses above.


Ask your test users to verify that their attributes have not changed and the services they use regularly are still working as expected with the new IdP. This is particularly important for bilateral services as more can go wrong with these services. 


Enabling your new IdP

Once you get the all clear from your testers and have progressed through your local change processes, it's time to release the IdP onto your user community.


This should be as simple as changing a DNS entry or the configuration of the your load balancer to direct users to the new IdP. If your change is via a DNS change, you should consider lowering the TTL for the IdPs DNS entry so the change is quick and rollback is also quick if required.


Next Step

Once you’ve finalised migration please continue to the operation stage.